draft-ietf-nfsv4-sess-01.txt   draft-ietf-nfsv4-sess-02.txt 
INTERNET-DRAFT Tom Talpey INTERNET-DRAFT Tom Talpey
Expires: August 2005 Network Appliance, Inc. Expires: December 2005 Network Appliance, Inc.
Spencer Shepler Spencer Shepler
Sun Microsystems, Inc. Sun Microsystems, Inc.
Jon Bauman Jon Bauman
University of Michigan University of Michigan
February, 2005 July, 2005
NFSv4 Session Extensions NFSv4 Session Extensions
draft-ietf-nfsv4-sess-01 draft-ietf-nfsv4-sess-02
Status of this Memo Status of this Memo
By submitting this Internet-Draft, I certify that any applicable By submitting this Internet-Draft, each author represents that any
patent or other IPR claims of which I am aware have been disclosed, applicable patent or other IPR claims of which he or she is aware
or will be disclosed, and any of which I become aware will be have been or will be disclosed, and any of which he or she becomes
disclosed, in accordance with RFC 3668. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
skipping to change at page 2, line 45 skipping to change at page 2, line 45
2.3.2. Negotiated RDMA Connection Model . . . . . . . . . . . 24 2.3.2. Negotiated RDMA Connection Model . . . . . . . . . . . 24
2.3.3. Automatic RDMA Connection Model . . . . . . . . . . . 24 2.3.3. Automatic RDMA Connection Model . . . . . . . . . . . 24
2.4. Buffer Management, Transfer, Flow Control . . . . . . . 25 2.4. Buffer Management, Transfer, Flow Control . . . . . . . 25
2.5. Retry and Replay . . . . . . . . . . . . . . . . . . . . 28 2.5. Retry and Replay . . . . . . . . . . . . . . . . . . . . 28
2.6. The Back Channel . . . . . . . . . . . . . . . . . . . . 28 2.6. The Back Channel . . . . . . . . . . . . . . . . . . . . 28
2.7. COMPOUND Sizing Issues . . . . . . . . . . . . . . . . . 30 2.7. COMPOUND Sizing Issues . . . . . . . . . . . . . . . . . 30
2.8. Data Alignment . . . . . . . . . . . . . . . . . . . . . 30 2.8. Data Alignment . . . . . . . . . . . . . . . . . . . . . 30
3. NFSv4 Integration . . . . . . . . . . . . . . . . . . . . 31 3. NFSv4 Integration . . . . . . . . . . . . . . . . . . . . 31
3.1. Minor Versioning . . . . . . . . . . . . . . . . . . . . 32 3.1. Minor Versioning . . . . . . . . . . . . . . . . . . . . 32
3.2. Slot Identifiers and Server Duplicate Request Cache . . 32 3.2. Slot Identifiers and Server Duplicate Request Cache . . 32
3.3. COMPOUND and CB_COMPOUND . . . . . . . . . . . . . . . . 35 3.3. COMPOUND and CB_COMPOUND . . . . . . . . . . . . . . . . 36
3.4. eXternal Data Representation Efficiency . . . . . . . . 36 3.4. eXternal Data Representation Efficiency . . . . . . . . 37
3.5. Effect of Sessions on Existing Operations . . . . . . . 36 3.5. Effect of Sessions on Existing Operations . . . . . . . 37
3.6. Authentication Efficiencies . . . . . . . . . . . . . . 37 3.6. Authentication Efficiencies . . . . . . . . . . . . . . 38
4. Security Considerations . . . . . . . . . . . . . . . . . 38 4. Security Considerations . . . . . . . . . . . . . . . . . 39
4.1. Authentication . . . . . . . . . . . . . . . . . . . . . 40 4.1. Authentication . . . . . . . . . . . . . . . . . . . . . 40
5. IANA Considerations . . . . . . . . . . . . . . . . . . . 41 5. IANA Considerations . . . . . . . . . . . . . . . . . . . 41
6. NFSv4 Protocol Extensions . . . . . . . . . . . . . . . . 41 6. NFSv4 Protocol Extensions . . . . . . . . . . . . . . . . 42
6.1. Operation: CREATECLIENTID . . . . . . . . . . . . . . . 41 6.1. Operation: CREATECLIENTID . . . . . . . . . . . . . . . 42
6.2. Operation: CREATESESSION . . . . . . . . . . . . . . . . 46 6.2. Operation: CREATESESSION . . . . . . . . . . . . . . . . 47
6.3. Operation: BIND_BACKCHANNEL . . . . . . . . . . . . . . 51 6.3. Operation: BIND_BACKCHANNEL . . . . . . . . . . . . . . 52
6.4. Operation: DESTROYSESSION . . . . . . . . . . . . . . . 53 6.4. Operation: DESTROYSESSION . . . . . . . . . . . . . . . 54
6.5. Operation: SEQUENCE . . . . . . . . . . . . . . . . . . 54 6.5. Operation: SEQUENCE . . . . . . . . . . . . . . . . . . 55
6.6. Callback operation: CB_RECALLCREDIT . . . . . . . . . . 56 6.6. Callback operation: CB_RECALLCREDIT . . . . . . . . . . 57
6.7. Callback operation: CB_SEQUENCE . . . . . . . . . . . . 56 6.7. Callback operation: CB_SEQUENCE . . . . . . . . . . . . 57
7. NFSv4 Session Protocol Description . . . . . . . . . . . . 58 7. NFSv4 Session Protocol Description . . . . . . . . . . . . 59
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 64 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 65
9. References . . . . . . . . . . . . . . . . . . . . . . . . 64 9. References . . . . . . . . . . . . . . . . . . . . . . . . 65
9.1. Normative References . . . . . . . . . . . . . . . . . . 64 9.1. Normative References . . . . . . . . . . . . . . . . . . 65
9.2. Informative References . . . . . . . . . . . . . . . . . 65 9.2. Informative References . . . . . . . . . . . . . . . . . 66
10. Authors' Addresses . . . . . . . . . . . . . . . . . . . . 67 10. Authors' Addresses . . . . . . . . . . . . . . . . . . . . 68
11. Full Copyright Statement . . . . . . . . . . . . . . . . . 67 11. Full Copyright Statement . . . . . . . . . . . . . . . . . 69
1. Introduction 1. Introduction
This draft proposes extensions to NFS version 4 [RFC3530] enabling This draft proposes extensions to NFS version 4 [RFC3530] enabling
it to support sessions and endpoint management, and to support it to support sessions and endpoint management, and to support
operation atop RDMA-capable RPC over transports such as iWARP. operation atop RDMA-capable RPC over transports such as iWARP.
[RDMAP, DDP] These extensions enable support for exactly-once [RDMAP, DDP] These extensions enable support for exactly-once
semantics by NFSv4 servers, multipathing and trunking of transport semantics by NFSv4 servers, multipathing and trunking of transport
connections, and enhanced security. The ability to operate over connections, and enhanced security. The ability to operate over
RDMA enables greatly enhanced performance. Operation over existing RDMA enables greatly enhanced performance. Operation over existing
skipping to change at page 25, line 20 skipping to change at page 25, line 20
: : : :
: Create Clientid(nfs_client_id4) : : Create Clientid(nfs_client_id4) :
: ------------------------------> : : ------------------------------> :
: : Prepost : : Prepost
: Clientid reply(clientid, ...) : receive : Clientid reply(clientid, ...) : receive
: <------------------------------ : : <------------------------------ :
Prepost : : Prepost : :
receive : Create Session(clientid, size S, : receive : Create Session(clientid, size S, :
: maxreq N, RDMA ...) : : maxreq N, RDMA ...) :
: ------------------------------> : : ------------------------------> :
: : Prepost <=N' receives : : Prepost <=N'
: Session reply(sessionid, size S', : of size S' : Session reply(sessionid, size S', : receives of
: maxreq N') : : maxreq N') : size S'
: <------------------------------ : : <------------------------------ :
: : : :
: <normal operation> : : <normal operation> :
: ------------------------------> : : ------------------------------> :
: <------------------------------ : : <------------------------------ :
: : : : : :
2.4. Buffer Management, Transfer, Flow Control 2.4. Buffer Management, Transfer, Flow Control
Inline operations in NFSv4.1 behave effectively the same as TCP Inline operations in NFSv4.1 behave effectively the same as TCP
skipping to change at page 31, line 28 skipping to change at page 31, line 28
all go into the determination of a maximal NFSv4 request size and all go into the determination of a maximal NFSv4 request size and
therefore minimal buffer size. The client must select its offered therefore minimal buffer size. The client must select its offered
value carefully, so as not to overburden the server, and vice- value carefully, so as not to overburden the server, and vice-
versa. The payoff of an appropriate padding value is higher versa. The payoff of an appropriate padding value is higher
performance. performance.
Sender gather: Sender gather:
|RPC Request|Pad bytes|Length| -> |User data...| |RPC Request|Pad bytes|Length| -> |User data...|
\------+---------------------/ \ \------+---------------------/ \
\ \ \ \
\ Receiver scatter: \--------------+- ... \ Receiver scatter: \-----------+- ...
/-----+----------------\ \ \ /-----+----------------\ \ \
|RPC Request|Pad|Length| -> |FS buffer| -> |FS buffer| -> ... |RPC Request|Pad|Length| -> |FS buffer| -> |FS buffer| -> ...
In the above case, the server may recycle unused buffers to the In the above case, the server may recycle unused buffers to the
next posted receive if unused by the actual received request, or next posted receive if unused by the actual received request, or
may pass the now-complete buffers by reference for normal write may pass the now-complete buffers by reference for normal write
processing. For a server which can make use of it, this removes processing. For a server which can make use of it, this removes
any need for data copies of incoming data, without resorting to any need for data copies of incoming data, without resorting to
complicated end-to-end buffer advertisement and management. This complicated end-to-end buffer advertisement and management. This
includes most kernel-based and integrated server designs, among includes most kernel-based and integrated server designs, among
skipping to change at page 33, line 21 skipping to change at page 33, line 21
When the client issues a new request, it selects a slotid in the When the client issues a new request, it selects a slotid in the
range 0..N-1, where N is the server's current "totalrequests" limit range 0..N-1, where N is the server's current "totalrequests" limit
granted the client on the session over which the request is to be granted the client on the session over which the request is to be
issued. The slotid must be unused by any of the requests which the issued. The slotid must be unused by any of the requests which the
client has already active on the session. "Unused" here means the client has already active on the session. "Unused" here means the
client has no outstanding request for that slotid. Because the client has no outstanding request for that slotid. Because the
slot id is always an integer in the range 0..N-1, client slot id is always an integer in the range 0..N-1, client
implementations can use the slotid from a server response to implementations can use the slotid from a server response to
efficiently match responses with outstanding requests, such as, for efficiently match responses with outstanding requests, such as, for
example, by using the slotid to index into a outstanding request example, by using the slotid to index into a outstanding request
array. array. This can be used to avoid expensive hashing and lookup
functions in the performace-critical receive path.
The sequenceid, which accompanies the slotid in each request, is The sequenceid, which accompanies the slotid in each request, is
important for a second, important check at the server: it must be important for a second, important check at the server: it must be
able to be determined efficiently whether a request using a certain able to be determined efficiently whether a request using a certain
slotid is a retransmit or a new, never-before-seen request. It is slotid is a retransmit or a new, never-before-seen request. It is
not feasible for the client to assert that it is retransmitting to not feasible for the client to assert that it is retransmitting to
implement this, because for any given request the client cannot implement this, because for any given request the client cannot
know the server has seen it unless the server actually replies. Of know the server has seen it unless the server actually replies. Of
course, if the server replied, the client would not need to course, if the client has seen the server's reply, the client would
retransmit! not retransmit!
Therefore, a sequenceid is transmitted along with the slotid in The sequenceid must increase monotonically for each new transmit of
each request. The minimum rule is that the sequenceid must change a given slotid, and must remain unchanged for any retransmission.
for each new transmit of a given slotid, and must remain unchanged The server must in turn compare each newly received request's
for retransmission. This will ensure that the server can detect sequenceid with the last one previously received for that slotid,
any new requests by simply comparing the sequenceid with that to see if the new request is:
previously issued for the slot. However, it is more useful and
logical, particularly for tracing and avoiding possible o A new request, in which the sequenceid is greater than that
implementation errors, to simply increment the sequenceid for each previously seen in the slot (accounting for sequence
new request on any slotid. The sequenceid, in any case, is never wraparound). The server proceeds to execute the new request.
interpreted by the server for anything but a test for equality to
o A retransmitted request, in which the sequenceid is equal to
that last seen in the slot. Note that this request may be
either complete, or in progress. The server performs replay
processing in these cases.
o A misordered duplicate, in which the sequenceid is less than
that previously seen in the slot. The server must drop the
incoming request, which may imply dropping the connection if
the transport is reliable, as dictated by section 3.1.1 of
[RFC3530].
This last condition is possible on any connection, not just
unreliable, unordered transports. Delayed behavior on abandoned
TCP connections which are not yet closed at the server, or
pathological client implementations can cause it, among other
causes. Therefore, the server may wish to harden itself against
certain repeated occurrences of this, as it would for
retransmissions in [RFC3530].
It is recommended, though not necessary for protocol correctness,
that the client simply increment the sequenceid by one for each new
request on each slotid. This reduces the wraparound window to a
minimum, and is useful for tracing and avoidance of possible
implementation errors.
The client may however, for implementation-specific reasons, choose
a different algorithm. For example it might maintain a single
sequence space for all slots in the session - e.g. employing the
RPC XID itself. The sequenceid, in any case, is never interpreted
by the server for anything but to test by comparison with
previously seen values. previously seen values.
The server may thereby use the slotid, in conjunction with the The server may thereby use the slotid, in conjunction with the
session id and sequence id, within the SEQUENCE portion of the session id and sequence id, within the SEQUENCE portion of the
request to maintain its duplicate request cache (DRC) for the request to maintain its duplicate request cache (DRC) for the
session, as opposed to the traditional approach of ONC RPC session, as opposed to the traditional approach of ONC RPC
applications that use the XID along with certain transport applications that use the XID along with certain transport
information [RW96]. information [RW96].
Unlike the XID, the slotid is always within a specific range; this Unlike the XID, the slotid is always within a specific range; this
has two implications. The first implication is that for a given has two implications. The first implication is that for a given
session, the server need only cache the results of a limited number session, the server need only cache the results of a limited number
of COMPOUND requests. The second implication derives from the of COMPOUND requests. The second implication derives from the
first, which is unlike XID indexed DRCs, the slotid DRC by its first, which is unlike XID-indexed DRCs, the slotid DRC by its
nature cannot be overflowed. Through use of the sequenceid to nature cannot be overflowed. Through use of the sequenceid to
identify retransmitted requests, it is notable that the server does identify retransmitted requests, it is notable that the server does
not need to actually cache the request itself, reducing the storage not need to actually cache the request itself, reducing the storage
requirements of the DRC further. These new facilities makes it requirements of the DRC further. These new facilities makes it
practical to maintain all the required entries for an effective practical to maintain all the required entries for an effective
DRC. DRC.
The slotid therefore takes over the traditional role of the port The slotid and sequenceid therefore take over the traditional role
number in the server DRC implementation, and the session replaces of the port number in the server DRC implementation, and the
the IP address. This approach is considerably more portable and session replaces the IP address. This approach is considerably
completely robust - it is not subject to the frequent reassignment more portable and completely robust - it is not subject to the
of ports as clients reconnect over IP networks. In addition, the frequent reassignment of ports as clients reconnect over IP
RPC XID is not used in the reply cache, enhancing robustness of the networks. In addition, the RPC XID is not used in the reply cache,
cache in the face of any rapid reuse of XIDs by the client. enhancing robustness of the cache in the face of any rapid reuse of
XIDs by the client.
It is required to encode the slotid information into each request It is required to encode the slotid information into each request
in a way that does not violate the minor versioning rules of the in a way that does not violate the minor versioning rules of the
NFSv4.0 specification. This is accomplished here by encoding it in NFSv4.0 specification. This is accomplished here by encoding it in
a control operation within each NFSv4.1 COMPOUND and CB_COMPOUND a control operation within each NFSv4.1 COMPOUND and CB_COMPOUND
procedure. The operation easily piggybacks within existing procedure. The operation easily piggybacks within existing
messages. The implementation section of this document describes messages. The implementation section of this document describes
the specific proposal. the specific proposal.
In general, the receipt of a new request using any valid slotid is In general, the receipt of a new sequenced request arriving on any
an indication that the previous DRC contents of that slot may be valid slot is an indication that the previous DRC contents of that
discarded. In order to further assist the server in slot slot may be discarded. In order to further assist the server in
management, the client is required to use the lowest value slotid slot management, the client is required to use the lowest available
when issuing a new request. In this way, the server may be able to slot when issuing a new request. In this way, the server may be
retire additional entries. able to retire additional entries.
However, in the case where the server is actively adjusting its However, in the case where the server is actively adjusting its
granted maximum request count to the client, it may not be able to granted maximum request count to the client, it may not be able to
use receipt of the slotid to retire cache entries. The slotid used use receipt of the slotid to retire cache entries. The slotid used
in an incoming request may not reflect the server's current idea of in an incoming request may not reflect the server's current idea of
the client's session limit, because the request may have been sent the client's session limit, because the request may have been sent
from the client before the update was received. Therefore, in the from the client before the update was received. Therefore, in the
downward adjustment case, the server may have to retain a number of downward adjustment case, the server may have to retain a number of
duplicate request cache entries at least as large as the old value, duplicate request cache entries at least as large as the old value,
until operation sequencing rules allow it to infer that the client until operation sequencing rules allow it to infer that the client
skipping to change at page 44, line 46 skipping to change at page 45, line 29
{ id_arg, *, old_principal_arg, clientid_ret, TRUE } { id_arg, *, old_principal_arg, clientid_ret, TRUE }
Since the value of the clientdesc.id subfield of each client Since the value of the clientdesc.id subfield of each client
record must be unique, there is no modification of the record must be unique, there is no modification of the
server's state, and NFS4ERR_CLID_INUSE is returned to indicate server's state, and NFS4ERR_CLID_INUSE is returned to indicate
the client should retry with a different value for the the client should retry with a different value for the
clientdesc.id subfield of CREATECLIENTID4args. clientdesc.id subfield of CREATECLIENTID4args.
This scenario may also represent a malicious attempt to This scenario may also represent a malicious attempt to
destroy a client's state on the server destroy a client's state on the server. For security reasons,
For security reasons, the server MUST NOT remove the client's the server MUST NOT remove the client's state when there is a
state when there is a principal mismatch. principal mismatch.
4) Replay 4) Replay
If the server has the following unconfirmed record then this If the server has the following unconfirmed record then this
request is likely the result of a client replay due to a request is likely the result of a client replay due to a
network partition or some other connection failure. network partition or some other connection failure.
{ id_arg, verifier_arg, principal_arg, clientid_ret, FALSE } { id_arg, verifier_arg, principal_arg, clientid_ret, FALSE }
Since the response to the CREATECLIENTID request that created Since the response to the CREATECLIENTID request that created
this record may have been lost, it is not acceptable to drop this record may have been lost, it is not acceptable to drop
skipping to change at page 65, line 23 skipping to change at page 66, line 23
9.2. Informative References 9.2. Informative References
[BW87] [BW87]
B. Welch, "The Sprite Remote Procedure Call System", B. Welch, "The Sprite Remote Procedure Call System",
University of California Berkeley Technical Report CSD-87-302, University of California Berkeley Technical Report CSD-87-302,
ftp://sunsite.berkeley.edu/pub/techreps/CSD-87-302.html ftp://sunsite.berkeley.edu/pub/techreps/CSD-87-302.html
[CCM] [CCM]
M. Eisler, N. Williams, "The Channel Conjunction Mechanism M. Eisler, N. Williams, "The Channel Conjunction Mechanism
(CCM) for GSS", Internet-Draft Work in Progress, (CCM) for GSS", Internet-Draft Work in Progress,
http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-ccm-03 http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-ccm
[CJ89] [CJ89]
C. Juszczak, "Improving the Performance and Correctness of an C. Juszczak, "Improving the Performance and Correctness of an
NFS Server," Winter 1989 USENIX Conference Proceedings, USENIX NFS Server," Winter 1989 USENIX Conference Proceedings, USENIX
Association, Berkeley, CA, Februry 1989, pages 53-63. Association, Berkeley, CA, Februry 1989, pages 53-63.
[DAFS] [DAFS]
Direct Access File System, available from Direct Access File System, available from
http://www.dafscollaborative.org http://www.dafscollaborative.org
[DCK+03] [DCK+03]
M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T. M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T.
Talpey, M. Wittle, "The Direct Access File System", in Talpey, M. Wittle, "The Direct Access File System", in
Proceedings of 2nd USENIX Conference on File and Storage Proceedings of 2nd USENIX Conference on File and Storage
Technologies (FAST '03), San Francisco, CA, March 31 - April Technologies (FAST '03), San Francisco, CA, March 31 - April
2, 2003 2, 2003
[DDP] [DDP]
H. Shah, J. Pinkerton, R. Recio, P. Culley, "Direct Data H. Shah, J. Pinkerton, R. Recio, P. Culley, "Direct Data
Placement over Reliable Transports", Placement over Reliable Transports", Internet-Draft Work in
http://www.ietf.org/internet-drafts/draft-ietf-rddp-ddp-03 Progress, http://www.ietf.org/internet-drafts/draft-ietf-rddp-
ddp
[FJDAFS] [FJDAFS]
Fujitsu Prime Software Technologies, "Meet the DAFS Fujitsu Prime Software Technologies, "Meet the DAFS
Performance with DAFS/VI Kernel Implementation using cLAN", Performance with DAFS/VI Kernel Implementation using cLAN",
http://www.pst.fujitsu.com/english/dafsdemo/index.html http://www.pst.fujitsu.com/english/dafsdemo/index.html
[FJNFS] [FJNFS]
Fujitsu Prime Software Technologies, "An Adaptation of VIA to Fujitsu Prime Software Technologies, "An Adaptation of VIA to
NFS on Linux", NFS on Linux",
http://www.pst.fujitsu.com/english/nfs/index.html http://www.pst.fujitsu.com/english/nfs/index.html
[IB] InfiniBand Architecture Specification, Volume 1, Release 1.1. [IB] InfiniBand Architecture Specification, Volume 1, Release 1.1.
available from http://www.infinibandta.org available from http://www.infinibandta.org
[KM02] [KM02]
K. Magoutis, "Design and Implementation of a Direct Access K. Magoutis, "Design and Implementation of a Direct Access
skipping to change at page 66, line 30 skipping to change at page 67, line 32
Proceedings of 2002 USENIX Annual Technical Conference, Proceedings of 2002 USENIX Annual Technical Conference,
Monterey, CA, June 9-14, 2002. Monterey, CA, June 9-14, 2002.
[MIDTAX] [MIDTAX]
B. Carpenter, S. Brim, "Middleboxes: Taxonomy and Issues", B. Carpenter, S. Brim, "Middleboxes: Taxonomy and Issues",
Informational RFC, http://www.ietf.org/rfc/rfc3234 Informational RFC, http://www.ietf.org/rfc/rfc3234
[NFSDDP] [NFSDDP]
B. Callaghan, T. Talpey, "NFS Direct Data Placement", B. Callaghan, T. Talpey, "NFS Direct Data Placement",
Internet-Draft Work in Progress, http://www.ietf.org/internet- Internet-Draft Work in Progress, http://www.ietf.org/internet-
drafts/draft-ietf-nfsv4-nfsdirect-01 drafts/draft-ietf-nfsv4-nfsdirect
[NFSPS] [NFSPS]
T. Talpey, C. Juszczak, "NFS RDMA Problem Statement", T. Talpey, C. Juszczak, "NFS RDMA Problem Statement",
Internet-Draft Work in Progress, http://www.ietf.org/internet- Internet-Draft Work in Progress, http://www.ietf.org/internet-
drafts/draft-ietf-nfsv4-nfs-rdma-problem-statement-02 drafts/draft-ietf-nfsv4-nfs-rdma-problem-statement
[RDDP] [RDDP]
Remote Direct Data Placement Working Group charter, Remote Direct Data Placement Working Group charter,
http://www.ietf.org/html.charters/rddp-charter.html http://www.ietf.org/html.charters/rddp-charter.html
[RDDPPS] [RDDPPS]
A. Romanow, J. Mogul, T. Talpey, S. Bailey, Remote Direct Data A. Romanow, J. Mogul, T. Talpey, S. Bailey, Remote Direct Data
Placement Working Group Problem Statement, Internet-Draft Work Placement Working Group Problem Statement, Standards Track
in Progress, http://www.ietf.org/internet-drafts/draft-ietf- Informational RFC, http://www.ietf.org/internet-drafts/draft-
rddp-problem-statement-05 ietf-rddp-problem-statement
[RDMAP] [RDMAP]
R. Recio, P. Culley, D. Garcia, J. Hilland, "An RDMA Protocol R. Recio, P. Culley, D. Garcia, J. Hilland, "An RDMA Protocol
Specification", Internet-Draft Work in Progress, Specification", Internet-Draft Work in Progress,
http://www.ietf.org/internet-drafts/draft-ietf-rddp-rdmap-03 http://www.ietf.org/internet-drafts/draft-ietf-rddp-rdmap
[RPCRDMA] [RPCRDMA]
B. Callaghan, T. Talpey, "RDMA Transport for ONC RPC" B. Callaghan, T. Talpey, "RDMA Transport for ONC RPC"
Internet-Draft Work in Progress, http://www.ietf.org/internet- Internet-Draft Work in Progress, http://www.ietf.org/internet-
drafts/draft-ietf-nfsv4-rpcrdma-01 drafts/draft-ietf-nfsv4-rpcrdma
[RFC2203] [RFC2203]
M. Eisler, A. Chiu, L. Ling, "RPCSEC_GSS Protocol M. Eisler, A. Chiu, L. Ling, "RPCSEC_GSS Protocol
Specification", Standards Track RFC, Specification", Standards Track RFC,
http://www.ietf.org/rfc/rfc2203 http://www.ietf.org/rfc/rfc2203
[RW96] [RW96]
R. Werme, "RPC XID Issues", Connectathon 1996, San Jose, CA, R. Werme, "RPC XID Issues", Connectathon 1996, San Jose, CA,
http://www.cthon.org/talks96/werme1.pdf http://www.cthon.org/talks96/werme1.pdf
skipping to change at page 68, line 9 skipping to change at page 69, line 9
535 W. William St. Suite 3100 535 W. William St. Suite 3100
Ann Arbor, MI 48103 USA Ann Arbor, MI 48103 USA
Phone: +1 734 615-4782 Phone: +1 734 615-4782
Email: baumanj@umich.edu Email: baumanj@umich.edu
11. Full Copyright Statement 11. Full Copyright Statement
Copyright (C) The Internet Society (2005). This document is Copyright (C) The Internet Society (2005). This document is
subject to the rights, licenses and restrictions contained in BCP subject to the rights, licenses and restrictions contained in BCP
78 and except as set forth therein, the authors retain all their 78, and except as set forth therein, the authors retain all their
rights. rights.
This document and the information contained herein are provided on This document and the information contained herein are provided on
an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. PARTICULAR PURPOSE.
 End of changes. 

This html diff was produced by rfcdiff 1.24, available from http://www.levkowetz.com/ietf/tools/rfcdiff/