draft-ietf-nfsv4-rpcrdma-02.txt   draft-ietf-nfsv4-rpcrdma-03.txt 
Internet-Draft Brent Callaghan Internet-Draft Tom Talpey
Expires: April 2006 Tom Talpey Expires: December 2006 Brent Callaghan
Document: draft-ietf-nfsv4-rpcrdma-02 October, 2005 Document: draft-ietf-nfsv4-rpcrdma-03 June, 2006
RDMA Transport for ONC RPC RDMA Transport for ONC RPC
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
skipping to change at page 1, line 29 skipping to change at page 1, line 29
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt The list of http://www.ietf.org/ietf/1id-abstracts.txt
Internet-Draft Shadow Directories can be accessed at
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Abstract Abstract
A protocol is described providing RDMA as a new transport for ONC A protocol is described providing RDMA as a new transport for ONC
RPC. The RDMA transport binding conveys the benefits of efficient, RPC. The RDMA transport binding conveys the benefits of efficient,
bulk data transport over high speed networks, while providing for bulk data transport over high speed networks, while providing for
minimal change to RPC applications and with no required revision of minimal change to RPC applications and with no required revision of
the application RPC protocol, or the RPC protocol itself. the application RPC protocol, or the RPC protocol itself.
skipping to change at page 2, line 34 skipping to change at page 2, line 34
5.2. RDMA Write of Long Replies (Reply Chunks) . . . . . . . 22 5.2. RDMA Write of Long Replies (Reply Chunks) . . . . . . . 22
6. Connection Configuration Protocol . . . . . . . . . . . . 23 6. Connection Configuration Protocol . . . . . . . . . . . . 23
6.1. Initial Connection State . . . . . . . . . . . . . . . . 24 6.1. Initial Connection State . . . . . . . . . . . . . . . . 24
6.2. Protocol Description . . . . . . . . . . . . . . . . . . 24 6.2. Protocol Description . . . . . . . . . . . . . . . . . . 24
7. Memory Registration Overhead . . . . . . . . . . . . . . . 25 7. Memory Registration Overhead . . . . . . . . . . . . . . . 25
8. Errors and Error Recovery . . . . . . . . . . . . . . . . 26 8. Errors and Error Recovery . . . . . . . . . . . . . . . . 26
9. Node Addressing . . . . . . . . . . . . . . . . . . . . . 26 9. Node Addressing . . . . . . . . . . . . . . . . . . . . . 26
10. RPC Binding . . . . . . . . . . . . . . . . . . . . . . . 26 10. RPC Binding . . . . . . . . . . . . . . . . . . . . . . . 26
11. Security . . . . . . . . . . . . . . . . . . . . . . . . 27 11. Security . . . . . . . . . . . . . . . . . . . . . . . . 27
12. IANA Considerations . . . . . . . . . . . . . . . . . . . 27 12. IANA Considerations . . . . . . . . . . . . . . . . . . . 27
13. Acknowledgements . . . . . . . . . . . . . . . . . . . . 27 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . 28
14. Normative References . . . . . . . . . . . . . . . . . . 27 14. Normative References . . . . . . . . . . . . . . . . . . 28
15. Informative References . . . . . . . . . . . . . . . . . 28 15. Informative References . . . . . . . . . . . . . . . . . 29
16. Authors' Addresses . . . . . . . . . . . . . . . . . . . 29 16. Authors' Addresses . . . . . . . . . . . . . . . . . . . 29
17. Intellectual Property and Copyright Statements . . . . . 29 17. Intellectual Property and Copyright Statements . . . . . 30
Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 30 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 31
1. Introduction 1. Introduction
RDMA is a technique for efficient movement of data between end RDMA is a technique for efficient movement of data between end
nodes, which becomes increasingly compelling over high speed nodes, which becomes increasingly compelling over high speed
transports. By directing data into destination buffers as it sent transports. By directing data into destination buffers as it is
on a network, and placing it via direct memory access by hardware, sent on a network, and placing it via direct memory access by
the double benefit of faster transfers and reduced host overhead is hardware, the double benefit of faster transfers and reduced host
obtained. overhead is obtained.
ONC RPC [RFC1831] is a remote procedure call protocol that has been ONC RPC [RFC1831] is a remote procedure call protocol that has been
run over a variety of transports. Most RPC implementations today run over a variety of transports. Most RPC implementations today
use UDP or TCP. RPC messages are defined in terms of an eXternal use UDP or TCP. RPC messages are defined in terms of an eXternal
Data Representation (XDR) [RFC1832] which provides a canonical data Data Representation (XDR) [RFC1832] which provides a canonical data
representation across a variety of host architectures. An XDR data representation across a variety of host architectures. An XDR data
stream is conveyed differently on each type of transport. On UDP, stream is conveyed differently on each type of transport. On UDP,
RPC messages are encapsulated inside datagrams, while on a TCP byte RPC messages are encapsulated inside datagrams, while on a TCP byte
stream, RPC messages are delineated by a record marking protocol. stream, RPC messages are delineated by a record marking protocol.
An RDMA transport also conveys RPC messages in a unique fashion An RDMA transport also conveys RPC messages in a unique fashion
skipping to change at page 3, line 24 skipping to change at page 3, line 24
either UDP and TCP alone. They retain message delineations like either UDP and TCP alone. They retain message delineations like
UDP while also providing a reliable, sequenced data transfer like UDP while also providing a reliable, sequenced data transfer like
TCP. And, they provide the new efficient, bulk transfer service of TCP. And, they provide the new efficient, bulk transfer service of
RDMA. RDMA transports are therefore naturally viewed as a new RDMA. RDMA transports are therefore naturally viewed as a new
transport type by ONC RPC. transport type by ONC RPC.
RDMA as a transport will benefit the performance of RPC protocols RDMA as a transport will benefit the performance of RPC protocols
that move large "chunks" of data, since RDMA hardware excels at that move large "chunks" of data, since RDMA hardware excels at
moving data efficiently between host memory and a high speed moving data efficiently between host memory and a high speed
network with little or no host CPU involvement. In this context, network with little or no host CPU involvement. In this context,
the NFS protocol, in all its versions, is an obvious beneficiary of the NFS protocol, in all its versions [RFC1094] [RFC1813] [RFC3530]
RDMA. A complete problem statement is discussed in [NFSRDMAPS], [NFSv4.1], is an obvious beneficiary of RDMA. A complete problem
and related NFSv4 issues are discussed in [NFSSESS]. Many other statement is discussed in [NFSRDMAPS], and related NFSv4 issues are
RPC-based protocols will also benefit. discussed in [NFSv4.1]. Many other RPC-based protocols will also
benefit.
Although the RDMA transport described here provides relatively Although the RDMA transport described here provides relatively
transparent support for any RPC application, the proposal goes transparent support for any RPC application, the proposal goes
further in describing mechanisms that can optimize the use of RDMA further in describing mechanisms that can optimize the use of RDMA
with more active participation by the RPC application. with more active participation by the RPC application.
2. Abstract RDMA Requirements 2. Abstract RDMA Requirements
An RPC transport is responsible for conveying an RPC message from a An RPC transport is responsible for conveying an RPC message from a
sender to a receiver. An RPC message is either an RPC call from a sender to a receiver. An RPC message is either an RPC call from a
skipping to change at page 3, line 50 skipping to change at page 3, line 51
arguments if the message is an RPC call, or an RPC reply header arguments if the message is an RPC call, or an RPC reply header
followed by results if the message is an RPC reply. The call followed by results if the message is an RPC reply. The call
header contains a transaction ID (XID) followed by the program and header contains a transaction ID (XID) followed by the program and
procedure number as well as a security credential. An RPC reply procedure number as well as a security credential. An RPC reply
header begins with an XID that matches that of the RPC call header begins with an XID that matches that of the RPC call
message, followed by a security verifier and results. All data in message, followed by a security verifier and results. All data in
an RPC message is XDR encoded. For a complete description of the an RPC message is XDR encoded. For a complete description of the
RPC protocol and XDR encoding, see [RFC1831] and [RFC1832]. RPC protocol and XDR encoding, see [RFC1831] and [RFC1832].
This protocol assumes the following abstract model for RDMA This protocol assumes the following abstract model for RDMA
transports. Theese terms, common in the RDMA lexicon, are used in transports. These terms, common in the RDMA lexicon, are used in
this document. A more complete glossary of RDMA terms can be found this document. A more complete glossary of RDMA terms can be found
in [RDMA]. in [RDMAP].
o Registered Memory o Registered Memory
All data moved via tagged RDMA operations must be resident in All data moved via tagged RDMA operations must be resident in
registered memory at its destination. This protocol assumes registered memory at its destination. This protocol assumes
that each segment of registered memory may be identified with that each segment of registered memory may be identified with
a steering tag of no more than 32 bits and memory addresses of a steering tag of no more than 32 bits and memory addresses of
up to 64 bits in length. up to 64 bits in length.
o RDMA Send o RDMA Send
The RDMA provider supports an RDMA Send operation with The RDMA provider supports an RDMA Send operation with
skipping to change at page 7, line 39 skipping to change at page 7, line 39
Certain RDMA implementations may impose additional flow control Certain RDMA implementations may impose additional flow control
restrictions, such as limits on RDMA Read operations in progress at restrictions, such as limits on RDMA Read operations in progress at
the responder. Because these operations are outside the scope of the responder. Because these operations are outside the scope of
this protocol, they are not addressed and must be provided for by this protocol, they are not addressed and must be provided for by
other layers. For example, a simple upper layer RPC consumer might other layers. For example, a simple upper layer RPC consumer might
perform single-issue RDMA Read requests, while a more perform single-issue RDMA Read requests, while a more
sophisticated, multithreaded RPC consumer may implement its own sophisticated, multithreaded RPC consumer may implement its own
FIFO queue of such operations. For further discussion of possible FIFO queue of such operations. For further discussion of possible
protocol implementations capable of negotiating these values, see protocol implementations capable of negotiating these values, see
section 6 "Connection Configuration Protocol" of this draft, or section 6 "Connection Configuration Protocol" of this draft, or
[NFSSESS]. [NFSv4.1].
3.4. XDR Encoding with Chunks 3.4. XDR Encoding with Chunks
The data comprising an RPC call or reply message is marshaled or The data comprising an RPC call or reply message is marshaled or
serialized into a contiguous stream by an XDR routine. XDR data serialized into a contiguous stream by an XDR routine. XDR data
types such as integers, strings, arrays and linked lists are types such as integers, strings, arrays and linked lists are
commonly implemented over two very simple functions that encode commonly implemented over two very simple functions that encode
either an XDR data unit (32 bits) or an array of bytes. either an XDR data unit (32 bits) or an array of bytes.
Normally, the separate data items in an RPC call or reply are Normally, the separate data items in an RPC call or reply are
skipping to change at page 27, line 39 skipping to change at page 27, line 39
In setting up a new RDMA connection, the first action by an RPC In setting up a new RDMA connection, the first action by an RPC
client will be to obtain a transport address for the server. The client will be to obtain a transport address for the server. The
mechanism used to obtain this address, and to open an RDMA mechanism used to obtain this address, and to open an RDMA
connection is dependent on the type of RDMA transport, and is the connection is dependent on the type of RDMA transport, and is the
responsibility of each RPC protocol binding and its local responsibility of each RPC protocol binding and its local
implementation. implementation.
10. RPC Binding 10. RPC Binding
RPC services normally register with a portmap or rpcbind service, RPC services normally register with a portmap or rpcbind service,
which associates an RPC program number with a service address. In which associates an RPC program number with a service address. (In
the case of UDP or TCP, the service address for NFS is normally the case of UDP or TCP, the service address for NFS is normally
port 2049. This policy should be no different with RDMA port 2049.) This policy should be no different with RDMA
interconnects. interconnects, although it may require the allocation of port
numbers appropriate to each.
One possibility is to have the server's portmapper register itself When a mapping standard or convention exists for IP ports on an
on the RDMA interconnect at a "well known" service address. On UDP RDMA interconnect, there are two possibilities:
or TCP, this corresponds to port 111. A client could connect to
this service address and use the portmap protocol to obtain a One possibility is to have the server's portmapper register
service address in response to a program number, e.g. an iWARP port itself on the RDMA interconnect at a "well known" service
number, or an Infiniband GID. address. (On UDP or TCP, this corresponds to port 111.) A
client could connect to this service address and use the
portmap protocol to obtain a service address in response to a
program number, e.g. an iWARP port number, or an Infiniband
GID.
Alternatively, the client could simply connect to the mapped
well-known port for the service itself, if it is appropriately
defined.
Historically, different RPC protocols have taken different
approaches to their port assignment, therefore the specific method
is left to each RPC/RDMA-enabled upper layer binding, and not
addressed here.
11. Security 11. Security
ONC RPC provides its own security via the RPCSEC_GSS framework [RFC ONC RPC provides its own security via the RPCSEC_GSS framework
2203]. RPCSEC_GSS can provide message authentication, integrity [RFC2203]. RPCSEC_GSS can provide message authentication,
checking, and privacy. This security mechanism will be unaffected integrity checking, and privacy. This security mechanism will be
by the RDMA transport. The data integrity and privacy features unaffected by the RDMA transport. The data integrity and privacy
alter the body of the message, presenting it as a single chunk. features alter the body of the message, presenting it as a single
For large messages the chunk may be large enough to qualify for chunk. For large messages the chunk may be large enough to qualify
RDMA Read transfer. However, there is much data movement for RDMA Read transfer. However, there is much data movement
associated with computation and verification of integrity, or associated with computation and verification of integrity, or
encryption/decryption, so certain performance advantages may be encryption/decryption, so certain performance advantages may be
lost. lost.
For efficiency, more appropriate security mechanism for RDMA links For efficiency, more appropriate security mechanism for RDMA links
may be link-level protection, such as IPSec, which may be co- may be link-level protection, such as IPSec, which may be co-
located in the RDMA hardware. The use of link-level protection may located in the RDMA hardware. The use of link-level protection may
be negotiated through the use of a new RPCSEC_GSS mechanism like be negotiated through the use of a new RPCSEC_GSS mechanism like
the Credential Cache GSS Mechanism [CCM]. Use of such mechanisms the Credential Cache GSS Mechanism [CCM]. Use of such mechanisms
is recommended where end-to-end integrity and/or privacy is is recommended where end-to-end integrity and/or privacy is
desired, and where efficiency is required. desired, and where efficiency is required.
There are no new issues here with exposed addresses. The only There are no new issues here with exposed addresses. The only
exposed addresses here are in the chunk list and in the transport exposed addresses here are in the chunk list and in the transport
packets transferred via RDMA. The data contained in these packets transferred via RDMA. The data contained in these
addresses continues to be protected by RPCSEC_GSS integrity and addresses continues to be protected by RPCSEC_GSS integrity and
privacy. privacy.
12. IANA Considerations 12. IANA Considerations
The new RPC transport should be assigned a new RPC "netid".
As a new RPC transport, this protocol should have no effect on RPC As a new RPC transport, this protocol should have no effect on RPC
program numbers or registered port numbers. The new RPC transport program numbers or existing registered port numbers. However, new
should be assigned a new RPC "netid". If adopted, the Connection port numbers may be registered for use by RPC/RDMA-enabled
Configuration protocol described herein will require an RPC program services, as appropriate to the new networks over which the
number assignment. services will operate.
If adopted, the Connection Configuration protocol described herein
will require an RPC program number assignment.
13. Acknowledgements 13. Acknowledgements
The authors wish to thank Rob Thurlow, John Howard, Chet Juszczak, The authors wish to thank Rob Thurlow, John Howard, Chet Juszczak,
Alex Chiu, Peter Staubach, Dave Noveck, Brian Pawlowski, Steve Alex Chiu, Peter Staubach, Dave Noveck, Brian Pawlowski, Steve
Kleiman, Mike Eisler, Mark Wittle and Shantanu Mehendale for their Kleiman, Mike Eisler, Mark Wittle, Shantanu Mehendale, David
contributions to this document. Robinson and Mallikarjun Chadalapaka for their contributions to
this document.
14. Normative References 14. Normative References
[RFC1094]
Sun Microsystems, "NFS: Network File System Protocol
Specification", (NFS version 2) Informational RFC,
http://www.ietf.org/rfc/rfc1094.txt
[RFC1831] [RFC1831]
R. Srinivasan, "RPC: Remote Procedure Call Protocol Specification R. Srinivasan, "RPC: Remote Procedure Call Protocol
Version 2", Standards Track RFC, Specification Version 2", Standards Track RFC,
http://www.ietf.org/rfc/rfc1831.txt http://www.ietf.org/rfc/rfc1831.txt
[RFC1832] [RFC1832]
R. Srinivasan, "XDR: External Data Representation Standard", R. Srinivasan, "XDR: External Data Representation Standard",
Standards Track RFC, http://www.ietf.org/rfc/rfc1832.txt Standards Track RFC, http://www.ietf.org/rfc/rfc1832.txt
[RFC1813] [RFC1813]
B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3 Protocol B. Callaghan, B. Pawlowski, P. Staubach, "NFS Version 3
Specification", Informational RFC, Protocol Specification", Informational RFC,
http://www.ietf.org/rfc/rfc1813.txt http://www.ietf.org/rfc/rfc1813.txt
[RFC3530] [RFC3530]
S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame,
Eisler, D. Noveck, "NFS version 4 Protocol", Standards Track RFC, M. Eisler, D. Noveck, "NFS version 4 Protocol", Standards
http://www.ietf.org/rfc/rfc3530.txt Track RFC, http://www.ietf.org/rfc/rfc3530.txt
[RFC2203] [RFC2203]
M. Eisler, A. Chiu, L. Ling, "RPCSEC_GSS Protocol Specification", M. Eisler, A. Chiu, L. Ling, "RPCSEC_GSS Protocol
Standards Track RFC, http://www.ietf.org/rfc/rfc2203.txt Specification", Standards Track RFC,
http://www.ietf.org/rfc/rfc2203.txt
15. Informative References 15. Informative References
[RDMA] [RDMAP]
R. Recio et al, "An RDMA Protocol Specification", Internet Draft R. Recio et al, "An RDMA Protocol Specification", Internet
Work in Progress, draft-ietf-rddp-rdmap Draft Work in Progress, draft-ietf-rddp-rdmap
[CCM] [CCM]
M. Eisler, N. Williams, "CCM: The Credential Cache GSS Mechanism", M. Eisler, N. Williams, "CCM: The Credential Cache GSS
Internet Draft Work in Progress, draft-ietf-nfsv4-ccm Mechanism", Internet Draft Work in Progress, draft-ietf-
nfsv4-ccm
[NFSDDP] [NFSDDP]
B. Callaghan, T. Talpey, "NFS Direct Data Placement" Internet Draft B. Callaghan, T. Talpey, "NFS Direct Data Placement" Internet
Work in Progress, draft-ietf-nfsv4-nfsdirect Draft Work in Progress, draft-ietf-nfsv4-nfsdirect
[RDDP] [RDDP]
Remote Direct Data Placement Working Group Charter, Remote Direct Data Placement Working Group Charter,
http://www.ietf.org/html.charters/rddp-charter.html http://www.ietf.org/html.charters/rddp-charter.html
[NFSRDMAPS] [NFSRDMAPS]
T. Talpey, C. Juszczak, "NFS RDMA Problem Statement", Internet T. Talpey, C. Juszczak, "NFS RDMA Problem Statement", Internet
Draft Work in Progress, draft-ietf-nfsv4-nfs-rdma-problem-statement Draft Work in Progress, draft-ietf-nfsv4-nfs-rdma-problem-
statement
[NFSSESS] [NFSv4.1]
T. Talpey, S. Shepler, J. Bauman, "NFSv4 Session Extensions", S. Shepler, ed., "NFSv4 Minor Version 1" Internet Draft Work
Internet Draft Work in Progress, draft-ietf-nfsv4-nfs-sess in Progress, draft-ietf-nfsv4-minorversion1
[IB] [IB]
Infiniband Architecture Specification, http://www.infinibandta.org Infiniband Architecture Specification,
http://www.infinibandta.org
16. Authors' Addresses 16. Authors' Addresses
Brent Callaghan
1614 Montalto Dr.
Mountain View, California 94040 USA
Phone: +1 650 968 2333
EMail: brent.callaghan@gmail.com
Tom Talpey Tom Talpey
Network Appliance, Inc. Network Appliance, Inc.
375 Totten Pond Road 375 Totten Pond Road
Waltham, MA 02451 USA Waltham, MA 02451 USA
Phone: +1 781 768 5329 Phone: +1 781 768 5329
EMail: thomas.talpey@netapp.com EMail: thomas.talpey@netapp.com
Brent Callaghan
Apple Computer, Inc.
MS: 302-4K
2 Infinite Loop
Cupertino, CA 95014 USA
EMail: brentc@apple.com
17. Intellectual Property and Copyright Statements 17. Intellectual Property and Copyright Statements
Intellectual Property Statement Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed Intellectual Property Rights or other rights that might be claimed
to pertain to the implementation or use of the technology described to pertain to the implementation or use of the technology described
in this document or the extent to which any license under such in this document or the extent to which any license under such
rights might or might not be available; nor does it represent that rights might or might not be available; nor does it represent that
skipping to change at page 31, line 19 skipping to change at page 32, line 7
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. PARTICULAR PURPOSE.
Copyright Statement Copyright Statement
Copyright (C) The Internet Society (2005). This document is Copyright (C) The Internet Society (2006).
subject to the rights, licenses and restrictions contained in BCP
78, and except as set forth therein, the authors retain all their This document is subject to the rights, licenses and restrictions
rights. contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
Acknowledgement Acknowledgement
Funding for the RFC Editor function is currently provided by the Funding for the RFC Editor function is currently provided by the
Internet Society. Internet Society.
 End of changes. 31 change blocks. 
75 lines changed or deleted 107 lines changed or added

This html diff was produced by rfcdiff 1.32. The latest version is available from http://www.levkowetz.com/ietf/tools/rfcdiff/