draft-ietf-rddp-problem-statement-02.txt   draft-ietf-rddp-problem-statement-03.txt 
Allyn Romanow (Cisco) Allyn Romanow (Cisco)
Internet-Draft Jeff Mogul (HP) Internet-Draft Jeff Mogul (HP)
Expires: December 2003 Tom Talpey (NetApp) Expires: July 2004 Tom Talpey (NetApp)
Stephen Bailey (Sandburst) Stephen Bailey (Sandburst)
RDMA over IP Problem Statement RDMA over IP Problem Statement
draft-ietf-rddp-problem-statement-02 draft-ietf-rddp-problem-statement-03
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 35 skipping to change at page 1, line 35
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved. Copyright (C) The Internet Society (2004). All Rights Reserved.
Abstract Abstract
This draft addresses an IP-based solution to the problem of high This draft addresses an IP-based solution to the problem of high
system costs due to network I/O copying in end-hosts at high system overhead due to end-host copying of user data in the network
speeds. The problem is due to the high cost of memory bandwidth, I/O path at high speeds. The problem is due to the high cost of
and it can be substantially improved using "copy avoidance." The memory bandwidth, and it can be substantially improved using "copy
high overhead has limited the use of TCP/IP in interconnection avoidance." The overhead has limited the use of TCP/IP in
networks especially where high bandwidth, low latency and/or low interconnection networks especially where high bandwidth, low
overhead of end-system data movement are required by the hosted latency and/or low overhead of end-system data movement are
application. required by the hosted application.
Table Of Contents Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 2
2. The high cost of data movement operations in network I/O . 3 2. The high cost of data movement operations in network I/O . 3
2.1. Copy avoidance improves processing overhead . . . . . . . 5 2.1. Copy avoidance improves processing overhead . . . . . . . 5
3. Memory bandwidth is the root cause of the problem . . . . 6 3. Memory bandwidth is the root cause of the problem . . . . 6
4. High copy overhead is problematic for many key Internet 4. High copy overhead is problematic for many key Internet
applications . . . . . . . . . . . . . . . . . . . . . . . 7 applications . . . . . . . . . . . . . . . . . . . . . . . 7
5. Copy Avoidance Techniques . . . . . . . . . . . . . . . . 9 5. Copy Avoidance Techniques . . . . . . . . . . . . . . . . 9
5.1. A Conceptual Framework: DDP and RDMA . . . . . . . . . . . 11 5.1. A Conceptual Framework: DDP and RDMA . . . . . . . . . . . 11
6. Security Considerations . . . . . . . . . . . . . . . . . 11 6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 12
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 12 7. Security Considerations . . . . . . . . . . . . . . . . . 12
Informative References . . . . . . . . . . . . . . . . . . 12 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . 17 Informative References . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . 18
Full Copyright Statement . . . . . . . . . . . . . . . . . 18 Full Copyright Statement . . . . . . . . . . . . . . . . . 18
1. Introduction 1. Introduction
This draft considers the problem of high host processing overhead This draft considers the problem of high host processing overhead
associated with network I/O that occurs under high speed associated with the movement of user data to and from the network
conditions. This problem is often referred to as the "I/O interface under high speed conditions. This problem is often
bottleneck" [CT90]. More specifically, the source of high overhead referred to as the "I/O bottleneck" [CT90]. More specifically, the
that is of interest here is data movement operations - copying. source of high overhead that is of interest here is data movement
This issue is not be confused with TCP offload, which is not operations - copying. The throughput of a system may therefore be
addressed here. High speed refers to conditions where the network limited by the overhead of this copying. This issue is not to be
link speed is high relative to the bandwidths of the host CPU and confused with TCP offload, which is not addressed here. High speed
memory. With today's computer systems, one Gbits/s and over is refers to conditions where the network link speed is high relative
considered high speed. to the bandwidths of the host CPU and memory. With today's
computer systems, one Gbits/s and over is considered high speed.
High costs associated with copying are an issue primarily for large High costs associated with copying are an issue primarily for large
scale systems. Although smaller systems such as rack-mounted PCs scale systems. Although smaller systems such as rack-mounted PCs
and small workstations would benefit from a reduction in copying and small workstations would benefit from a reduction in copying
overhead, the benefit to smaller machines will be primarily in the overhead, the benefit to smaller machines will be primarily in the
next few years as they scale in the amount of bandwidth they next few years as they scale in the amount of bandwidth they
handle. Today it is large system machines with high bandwidth handle. Today it is large system machines with high bandwidth
feeds, usually multiprocessors and clusters, that are adversely feeds, usually multiprocessors and clusters, that are adversely
affected by copying overhead. Examples of such machines include affected by copying overhead. Examples of such machines include
all varieties of servers: database servers, storage servers, all varieties of servers: database servers, storage servers,
skipping to change at page 3, line 41 skipping to change at page 3, line 43
arbitrarily large. arbitrarily large.
This problem statement substantiates the claim that in network I/O This problem statement substantiates the claim that in network I/O
processing, high overhead results from data movement operations, processing, high overhead results from data movement operations,
specifically copying; and that copy avoidance significantly specifically copying; and that copy avoidance significantly
decreases the processing overhead. It describes when and why the decreases the processing overhead. It describes when and why the
high processing overheads occur, explains why the overhead is high processing overheads occur, explains why the overhead is
problematic, and points out which applications are most affected. problematic, and points out which applications are most affected.
In addition, this document introduces an architectural approach to In addition, this document introduces an architectural approach to
solving the problem, which is developed in detail in [BT02]. It solving the problem, which is developed in detail in [BT04]. It
also discusses how the proposed technology may introduce security also discusses how the proposed technology may introduce security
concerns and how they should be addressed. concerns and how they should be addressed.
2. The high cost of data movement operations in network I/O 2. The high cost of data movement operations in network I/O
A wealth of data from research and industry shows that copying is A wealth of data from research and industry shows that copying is
responsible for substantial amounts of processing overhead. It responsible for substantial amounts of processing overhead. It
further shows that even in carefully implemented systems, further shows that even in carefully implemented systems,
eliminating copies significantly reduces the overhead, as eliminating copies significantly reduces the overhead, as
referenced below. referenced below.
skipping to change at page 6, line 15 skipping to change at page 6, line 18
3. Memory bandwidth is the root cause of the problem 3. Memory bandwidth is the root cause of the problem
Data movement operations are expensive because memory bandwidth is Data movement operations are expensive because memory bandwidth is
scarce relative to network bandwidth and CPU bandwidth [PAC+97]. scarce relative to network bandwidth and CPU bandwidth [PAC+97].
This trend existed in the past and is expected to continue into the This trend existed in the past and is expected to continue into the
future [HP97, STREAM], especially in large multiprocessor systems. future [HP97, STREAM], especially in large multiprocessor systems.
With copies crossing the bus twice per copy, network processing With copies crossing the bus twice per copy, network processing
overhead is high whenever network bandwidth is large in comparison overhead is high whenever network bandwidth is large in comparison
to CPU and memory bandwidths. Generally with today's end-systems, to CPU and memory bandwidths. Generally with today's end-systems,
the effects are observable at network speeds over 1 Gbits/s. the effects are observable at network speeds over 1 Gbits/s. In
fact, with multiple bus crossings it is possible to see the bus
bandwidth being the limiting factor for throughput. This prevents
such an end-system from silultaneously achieving full network
bandwidth and full application performance.
A common question is whether increase in CPU processing power A common question is whether increase in CPU processing power
alleviates the problem of high processing costs of network I/O. alleviates the problem of high processing costs of network I/O.
The answer is no, it is the memory bandwidth that is the issue. The answer is no, it is the memory bandwidth that is the issue.
Faster CPUs do not help if the CPU spends most of its time waiting Faster CPUs do not help if the CPU spends most of its time waiting
for memory [CGY01]. for memory [CGY01].
The widening gap between microprocessor performance and memory The widening gap between microprocessor performance and memory
performance has long been a widely recognized and well-understood performance has long been a widely recognized and well-understood
problem [PAC+97]. Hennessy [HP97] shows microprocessor performance problem [PAC+97]. Hennessy [HP97] shows microprocessor performance
skipping to change at page 8, line 46 skipping to change at page 9, line 6
So, excessive overheads on network paths in a "scalable" system So, excessive overheads on network paths in a "scalable" system
both can require the use of more processors than optimal, and can both can require the use of more processors than optimal, and can
reduce the marginal utility of those additional processors. reduce the marginal utility of those additional processors.
Copy avoidance scales a machine upwards by removing at least two- Copy avoidance scales a machine upwards by removing at least two-
thirds the bus bandwidth load from the "very best" 1-copy (on thirds the bus bandwidth load from the "very best" 1-copy (on
receive) implementations, and removes at least 80% of the bandwidth receive) implementations, and removes at least 80% of the bandwidth
overhead from the 2-copy implementations. overhead from the 2-copy implementations.
The removal of bus bandwidth requirement, in turn, removes
bottlenecks from the network processing path and increases the
throughput of the machine. On a machine with limited bus
bandwidth, the advantages of removing this load is immediately
evident, as the host can attain full network bandwidth. Even on a
machine with bus bandwidth adequate to sustain full network
bandwidth, removal of bus bandwidth load serves to increase the
availabilty of the machine for the processing of user applications,
in some cases dramatically.
An example showing poor performance with copies and improved An example showing poor performance with copies and improved
scaling with copy avoidance is illustrative. The IO-Lite work scaling with copy avoidance is illustrative. The IO-Lite work
[PDZ99] shows higher server throughput servicing more clients using [PDZ99] shows higher server throughput servicing more clients using
a zero-copy system. In an experiment designed to mimic real world a zero-copy system. In an experiment designed to mimic real world
web conditions by simulating the effect of TCP WAN connections on web conditions by simulating the effect of TCP WAN connections on
the server, the performance of 3 servers was compared. One server the server, the performance of 3 servers was compared. One server
was Apache, another an optimized server called Flash, and the third was Apache, another an optimized server called Flash, and the third
the Flash server running IO-Lite, called Flash-Lite with zero copy. the Flash server running IO-Lite, called Flash-Lite with zero copy.
The measurement was of throughput in requests/second as a function The measurement was of throughput in requests/second as a function
of the number of slow background clients that could be served. As of the number of slow background clients that could be served. As
skipping to change at page 9, line 35 skipping to change at page 10, line 6
send) and complex Web servers based on SANs or multi-tier systems send) and complex Web servers based on SANs or multi-tier systems
will suffer more from copy overheads than in the example above. will suffer more from copy overheads than in the example above.
5. Copy Avoidance Techniques 5. Copy Avoidance Techniques
There have been extensive research investigation and industry There have been extensive research investigation and industry
experience with two main alternative approaches to eliminating data experience with two main alternative approaches to eliminating data
movement overhead, often along with improving other Operating movement overhead, often along with improving other Operating
System processing costs. In one approach, hardware and/or software System processing costs. In one approach, hardware and/or software
changes within a single host reduce processing costs. In another changes within a single host reduce processing costs. In another
approach, memory-to-memory networking [MAF+02], hosts communicate approach, memory-to-memory networking [MAF+02], the exchange of
via information that allows them to reduce processing costs. explicit data placement information between hosts allows them to
reduce processing costs.
The single host approaches range from new hardware and software The single host approaches range from new hardware and software
architectures [KSZ95, Wa97, DWB+93] to new or modified software architectures [KSZ95, Wa97, DWB+93] to new or modified software
systems [BP96, Ch96, TK95, DP93, PDZ99]. In the approach based on systems [BP96, Ch96, TK95, DP93, PDZ99]. In the approach based on
using a networking protocol to exchange information, the network using a networking protocol to exchange information, the network
adapter, under control of the application, places data directly adapter, under control of the application, places data directly
into and out of application buffers, reducing the need for data into and out of application buffers, reducing the need for data
movement. Commonly this approach is called RDMA, Remote Direct movement. Commonly this approach is called RDMA, Remote Direct
Memory Access. Memory Access.
skipping to change at page 10, line 36 skipping to change at page 11, line 7
of the Operating System such as machine architecture and file I/O, of the Operating System such as machine architecture and file I/O,
and disk I/O [Br99, Ch96, DP93]. and disk I/O [Br99, Ch96, DP93].
For example, the Solaris Zero-Copy TCP work [Ch96], which relies on For example, the Solaris Zero-Copy TCP work [Ch96], which relies on
page remapping, shows that the results are highly interdependent page remapping, shows that the results are highly interdependent
with other systems, such as the file system, and that the with other systems, such as the file system, and that the
particular optimizations are specific for particular architectures, particular optimizations are specific for particular architectures,
meaning for each variation in architecture optimizations must be meaning for each variation in architecture optimizations must be
re-crafted [Ch96]. re-crafted [Ch96].
With RDMA, application I/O buffers are mapped directly, and the
authorized peer may access it without incurring additional
processing overhead. When RDMA is implemented in hardware,
arbitrary data movement can be performed without involving the host
CPU at all.
A number of research projects and industry products have been based A number of research projects and industry products have been based
on the memory-to-memory approach to copy avoidance. These include on the memory-to-memory approach to copy avoidance. These include
U-Net [EBBV95], SHRIMP [BLA+94], Hamlyn [BJM+96], Infiniband [IB], U-Net [EBBV95], SHRIMP [BLA+94], Hamlyn [BJM+96], Infiniband [IB],
Winsock Direct [Pi01]. Several memory-to-memory systems have been Winsock Direct [Pi01]. Several memory-to-memory systems have been
widely used and have generally been found to be robust, to have widely used and have generally been found to be robust, to have
good performance, and to be relatively simple to implement. These good performance, and to be relatively simple to implement. These
include VI [VI], Myrinet [BCF+95], Quadrics [QUAD], Compaq/Tandem include VI [VI], Myrinet [BCF+95], Quadrics [QUAD], Compaq/Tandem
Servernet [SRVNET]. Networks based on these memory-to-memory Servernet [SRVNET]. Networks based on these memory-to-memory
architectures have been used widely in scientific applications and architectures have been used widely in scientific applications and
in data centers for block storage, file system access, and in data centers for block storage, file system access, and
skipping to change at page 11, line 34 skipping to change at page 12, line 11
provided. provided.
While the functions (and potentially protocols) are distinct, While the functions (and potentially protocols) are distinct,
historically both aspects taken together have been referred as historically both aspects taken together have been referred as
"RDMA". The facilities of direct data placement are useful in and "RDMA". The facilities of direct data placement are useful in and
of themselves, and may be employed by other upper layer protocols of themselves, and may be employed by other upper layer protocols
to facilitate data transfer. Therefore, it is often useful to to facilitate data transfer. Therefore, it is often useful to
refer to DDP as the data placement functionality and RDMA as the refer to DDP as the data placement functionality and RDMA as the
control aspect. control aspect.
[BT02] develops an architecture for DDP and RDMA, and is a [BT04] develops an architecture for DDP and RDMA atop the Internet
companion draft to this problem statement. Protocol Suite, and is a companion draft to this problem statement.
6. Security Considerations 6. Conclusions
This Problem Statement concludes that an IP-based, general solution
for reducing processing overhead in end-hosts is desirable.
It has shown that high overhead of the processing of network data
leads to end-host bottlenecks. These bottlenecks are in large part
attributable to the copying of data. The bus bandwidth of machines
has historically been limited, and the bandwidth of high-speed
interconnects taxes it heavily.
An architectural solution to alleviate these bottlenecks best
satisifies the issue. Further, the high speed of today's
interconnects and the deployment of these hosts on Internet
Protocol-based networks leads to the desireability to layer such a
solution on the Internet Protocol Suite. The architecture
described in [BT04] is such a proposal.
7. Security Considerations
Solutions to the problem of reducing copying overhead in high Solutions to the problem of reducing copying overhead in high
bandwidth transfers via one or more protocols may introduce new bandwidth transfers via one or more protocols may introduce new
security concerns. Any proposed solution must be analyzed for security concerns. Any proposed solution must be analyzed for
security threats and any such threats addressed. Potential security threats and any such threats addressed. Potential
security weaknesses due to resource issues that might lead to security weaknesses due to resource issues that might lead to
denial-of-service attacks, overwrites and other concurrent denial-of-service attacks, overwrites and other concurrent
operations, the ordering of completions as required by the RDMA operations, the ordering of completions as required by the RDMA
protocol, the granularity of transfer, and any other identified protocol, the granularity of transfer, and any other identified
threats; need to be examined, described and an adequate solution to threats; need to be examined, described and an adequate solution to
skipping to change at page 12, line 42 skipping to change at page 13, line 37
higher level aspects in addition to the basic formation of higher level aspects in addition to the basic formation of
messages. The semantics associated with each class of error must messages. The semantics associated with each class of error must
be clearly defined, and the expected action to be taken on mismatch be clearly defined, and the expected action to be taken on mismatch
be specified. In some cases, this will result in a catastrophic be specified. In some cases, this will result in a catastrophic
error on the RDMA association, however in others a local or remote error on the RDMA association, however in others a local or remote
error may be signalled. Certain of these errors may require error may be signalled. Certain of these errors may require
consideration of abstract local semantics, which must be carefully consideration of abstract local semantics, which must be carefully
specified so as to provide useful behavior while not constraining specified so as to provide useful behavior while not constraining
the implementation. the implementation.
7. Acknowledgements 8. Acknowledgements
Jeff Chase generously provided many useful insights and Jeff Chase generously provided many useful insights and
information. Thanks to Jim Pinkerton for many helpful discussions. information. Thanks to Jim Pinkerton for many helpful discussions.
8. Informative References 9. Informative References
[BCF+95] [BCF+95]
N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L. N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L.
Seitz, J. N. Seizovic, and W. Su. "Myrinet - A gigabit-per- Seitz, J. N. Seizovic, and W. Su. "Myrinet - A gigabit-per-
second local-area network", IEEE Micro, February 1995 second local-area network", IEEE Micro, February 1995
[BJM+96] [BJM+96]
G. Buzzard, D. Jacobson, M. Mackey, S. Marovich, J. Wilkes, G. Buzzard, D. Jacobson, M. Mackey, S. Marovich, J. Wilkes,
"An implementation of the Hamlyn send-managed interface "An implementation of the Hamlyn send-managed interface
architecture", in Proceedings of the Second Symposium on architecture", in Proceedings of the Second Symposium on
skipping to change at page 13, line 32 skipping to change at page 14, line 27
[BS96] [BS96]
J. C. Brustoloni, P. Steenkiste, "Effects of buffering J. C. Brustoloni, P. Steenkiste, "Effects of buffering
semantics on I/O performance", Proceedings OSDI'96, USENIX, semantics on I/O performance", Proceedings OSDI'96, USENIX,
Seattle, WA October 1996, pp. 277-291 Seattle, WA October 1996, pp. 277-291
RFC Editor note: RFC Editor note:
Replace following architecture draft-ietf- name, status and date Replace following architecture draft-ietf- name, status and date
with appropriate reference when assigned. with appropriate reference when assigned.
[BT02] [BT04]
S. Bailey, T. Talpey, "The Architecture of Direct Data S. Bailey, T. Talpey, "The Architecture of Direct Data
Placement (DDP) And Remote Direct Memory Access (RDMA) On Placement (DDP) And Remote Direct Memory Access (RDMA) On
Internet Protocols", Internet Draft Work in Progress, draft- Internet Protocols", Internet Draft Work in Progress, draft-
ietf-rddp-arch-02, June 2003 ietf-rddp-arch-04, January 2004
[CFF+94] [CFF+94]
C-H Chang, D. Flower, J. Forecast, H. Gray, B. Hawe, A. C-H Chang, D. Flower, J. Forecast, H. Gray, B. Hawe, A.
Nadkarni, K. K. Ramakrishnan, U. Shikarpur, K. Wilde, "High- Nadkarni, K. K. Ramakrishnan, U. Shikarpur, K. Wilde, "High-
performance TCP/IP and UDP/IP networking in DEC OSF/1 for performance TCP/IP and UDP/IP networking in DEC OSF/1 for
Alpha AXP", Proceedings of the 3rd IEEE Symposium on High Alpha AXP", Proceedings of the 3rd IEEE Symposium on High
Performance Distributed Computing, August 1994, pp. 36-42 Performance Distributed Computing, August 1994, pp. 36-42
[CGY01] [CGY01]
J. S. Chase, A. J. Gallatin, and K. G. Yocum, "End system J. S. Chase, A. J. Gallatin, and K. G. Yocum, "End system
skipping to change at page 15, line 38 skipping to change at page 16, line 32
[Ma02] [Ma02]
K. Magoutis, "Design and Implementation of a Direct Access K. Magoutis, "Design and Implementation of a Direct Access
File System (DAFS) Kernel Server for FreeBSD", in Proceedings File System (DAFS) Kernel Server for FreeBSD", in Proceedings
of USENIX BSDCon 2002 Conference, San Francisco, CA, February of USENIX BSDCon 2002 Conference, San Francisco, CA, February
11-14, 2002. 11-14, 2002.
[MAF+02] [MAF+02]
K. Magoutis, S. Addetia, A. Fedorova, M. I. Seltzer, J. S. K. Magoutis, S. Addetia, A. Fedorova, M. I. Seltzer, J. S.
Chase, D. Gallatin, R. Kisley, R. Wickremesinghe, E. Gabber, Chase, D. Gallatin, R. Kisley, R. Wickremesinghe, E. Gabber,
"Structure and Performance of the Direct Access File System "Structure and Performance of the Direct Access File System
(DAFS)", accepted for publication at the 2002 USENIX Annual (DAFS)", in Proceedings of the 2002 USENIX Annual Technical
Technical Conference, Monterey, CA, June 9-14, 2002. Conference, Monterey, CA, June 9-14, 2002.
[Mc95] [Mc95]
J. D. McCalpin, "A Survey of memory bandwidth and machine J. D. McCalpin, "A Survey of memory bandwidth and machine
balance in current high performance computers", IEEE TCCA balance in current high performance computers", IEEE TCCA
Newsletter, December 1995 Newsletter, December 1995
[Ne00] [Ne00]
A. Newman, "IDC report paints conflicted picture of server A. Newman, "IDC report paints conflicted picture of server
market circa 2004", ServerWatch, July 24, 2000 market circa 2004", ServerWatch, July 24, 2000
http://serverwatch.internet.com/news/2000_07_24_a.html http://serverwatch.internet.com/news/2000_07_24_a.html
skipping to change at page 18, line 14 skipping to change at page 18, line 44
Tom Talpey Tom Talpey
Network Appliance Network Appliance
375 Totten Pond Road 375 Totten Pond Road
Waltham, MA 02451 USA Waltham, MA 02451 USA
Phone: +1 781 768 5329 Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com Email: thomas.talpey@netapp.com
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved. Copyright (C) The Internet Society (2004). All Rights Reserved.
This document and translations of it may be copied and furnished to This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain others, and derivative works that comment on or otherwise explain
it or assist in its implementation may be prepared, copied, it or assist in its implementation may be prepared, copied,
published and distributed, in whole or in part, without restriction published and distributed, in whole or in part, without restriction
of any kind, provided that the above copyright notice and this of any kind, provided that the above copyright notice and this
paragraph are included on all such copies and derivative works. paragraph are included on all such copies and derivative works.
However, this document itself may not be modified in any way, such However, this document itself may not be modified in any way, such
as by removing the copyright notice or references to the Internet as by removing the copyright notice or references to the Internet
Society or other Internet organizations, except as needed for the Society or other Internet organizations, except as needed for the
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/