draft-ietf-nfsv4-nfs-ulb-v2-00.txt | draft-ietf-nfsv4-nfs-ulb-v2-01.txt | |||
---|---|---|---|---|
Network File System Version 4 C. Lever | Network File System Version 4 C. Lever | |||
Internet-Draft Oracle | Internet-Draft Oracle | |||
Intended status: Standards Track November 17, 2019 | Intended status: Standards Track February 3, 2020 | |||
Expires: May 20, 2020 | Expires: August 6, 2020 | |||
Network File System (NFS) Upper Layer Binding To RPC-Over-RDMA Version 2 | Network File System (NFS) Upper-Layer Binding To RPC-Over-RDMA Version 2 | |||
draft-ietf-nfsv4-nfs-ulb-v2-00 | draft-ietf-nfsv4-nfs-ulb-v2-01 | |||
Abstract | Abstract | |||
This document specifies Upper Layer Bindings of Network File System | This document specifies Upper-Layer Bindings of Network File System | |||
(NFS) protocol versions to RPC-over-RDMA version 2. | (NFS) protocol versions to RPC-over-RDMA version 2. | |||
Status of This Memo | Status of This Memo | |||
This Internet-Draft is submitted in full conformance with the | This Internet-Draft is submitted in full conformance with the | |||
provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
Drafts is at https://datatracker.ietf.org/drafts/current/. | Drafts is at https://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
This Internet-Draft will expire on May 20, 2020. | This Internet-Draft will expire on August 6, 2020. | |||
Copyright Notice | Copyright Notice | |||
Copyright (c) 2019 IETF Trust and the persons identified as the | Copyright (c) 2020 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(https://trustee.ietf.org/license-info) in effect on the date of | (https://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
skipping to change at page 2, line 19 ¶ | skipping to change at page 2, line 19 ¶ | |||
outside the IETF Standards Process, and derivative works of it may | outside the IETF Standards Process, and derivative works of it may | |||
not be created outside the IETF Standards Process, except to format | not be created outside the IETF Standards Process, except to format | |||
it for publication as an RFC or to translate it into languages other | it for publication as an RFC or to translate it into languages other | |||
than English. | than English. | |||
Table of Contents | Table of Contents | |||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 | |||
2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3 | 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3 | |||
3. Reply Size Estimation . . . . . . . . . . . . . . . . . . . . 3 | 3. Reply Size Estimation . . . . . . . . . . . . . . . . . . . . 3 | |||
4. Upper Layer Binding for NFS Versions 2 and 3 . . . . . . . . 3 | 4. Upper-Layer Binding for NFS Versions 2 and 3 . . . . . . . . 3 | |||
4.1. Reply Size Estimation . . . . . . . . . . . . . . . . . . 4 | 4.1. Reply Size Estimation . . . . . . . . . . . . . . . . . . 4 | |||
4.2. RPC Binding Considerations . . . . . . . . . . . . . . . 4 | 4.2. RPC Binding Considerations . . . . . . . . . . . . . . . 4 | |||
5. Upper Layer Bindings for NFS Version 2 and 3 Auxiliary | 4.3. Transport Considerations . . . . . . . . . . . . . . . . 4 | |||
Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . 4 | 5. Upper-Layer Bindings for NFS Version 2 and 3 Auxiliary | |||
5.1. MOUNT, NLM, and NSM Protocols . . . . . . . . . . . . . . 5 | Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . 5 | |||
5.2. NFSACL Protocol . . . . . . . . . . . . . . . . . . . . . 5 | 5.1. MOUNT, NLM, and NSM Protocols . . . . . . . . . . . . . . 6 | |||
6. Upper Layer Binding For NFS Version 4 . . . . . . . . . . . . 5 | 5.2. NFSACL Protocol . . . . . . . . . . . . . . . . . . . . . 6 | |||
6.1. DDP-Eligibility . . . . . . . . . . . . . . . . . . . . . 5 | 6. Upper-Layer Binding For NFS Version 4 . . . . . . . . . . . . 6 | |||
6.2. Reply Size Estimation . . . . . . . . . . . . . . . . . . 6 | 6.1. DDP-Eligibility . . . . . . . . . . . . . . . . . . . . . 7 | |||
6.3. RPC Binding Considerations . . . . . . . . . . . . . . . 7 | 6.2. Reply Size Estimation . . . . . . . . . . . . . . . . . . 7 | |||
6.4. NFS COMPOUND Requests . . . . . . . . . . . . . . . . . . 7 | 6.3. RPC Binding Considerations . . . . . . . . . . . . . . . 8 | |||
6.5. NFS Callback Requests . . . . . . . . . . . . . . . . . . 9 | 6.4. NFS COMPOUND Requests . . . . . . . . . . . . . . . . . . 8 | |||
6.6. Session-Related Considerations . . . . . . . . . . . . . 10 | 6.5. NFS Callback Requests . . . . . . . . . . . . . . . . . . 11 | |||
6.7. Transport Considerations . . . . . . . . . . . . . . . . 11 | 6.6. Session-Related Considerations . . . . . . . . . . . . . 12 | |||
7. Extending NFS Upper Layer Bindings . . . . . . . . . . . . . 13 | 6.7. Transport Considerations . . . . . . . . . . . . . . . . 12 | |||
8. Security Considerations . . . . . . . . . . . . . . . . . . . 13 | 7. Extending NFS Upper-Layer Bindings . . . . . . . . . . . . . 14 | |||
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 | 8. Security Considerations . . . . . . . . . . . . . . . . . . . 14 | |||
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 | 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 | |||
10.1. Normative References . . . . . . . . . . . . . . . . . . 14 | 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 14 | |||
10.2. Informative References . . . . . . . . . . . . . . . . . 15 | 10.1. Normative References . . . . . . . . . . . . . . . . . . 15 | |||
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 15 | 10.2. Informative References . . . . . . . . . . . . . . . . . 16 | |||
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15 | Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 16 | |||
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 16 | ||||
1. Introduction | 1. Introduction | |||
The RPC-over-RDMA version 2 transport may employ direct data | The RPC-over-RDMA version 2 transport may employ direct data | |||
placement to convey data payloads associated with RPC transactions | placement to convey data payloads associated with RPC transactions | |||
[I-D.cel-nfsv4-rpcrdma-version-two]. To enable successful | [RPCRDMA2]. RPC client and server implementations using RPC-over- | |||
interoperation, RPC client and server implementations using RPC-over- | ||||
RDMA version 2 must agree which XDR data items and RPC procedures are | RDMA version 2 must agree which XDR data items and RPC procedures are | |||
eligible to use direct data placement (DDP). | eligible to use direct data placement (DDP) to enable successful | |||
interoperation. | ||||
An Upper Layer Binding specifies this agreement for one or more | An Upper-Layer Binding specifies this agreement for one or more | |||
versions of one RPC program. Other operational details, such as RPC | versions of one RPC program. Other operational details, such as RPC | |||
binding assignments, pairing Write chunks with result data items, and | binding assignments, pairing Write chunks with result data items, and | |||
reply size estimation, are also specified by this Binding. | reply size estimation, are also specified by this Binding. | |||
This document contains material required of Upper Layer Bindings, as | This document contains material required of Upper-Layer Bindings, as | |||
specified in [I-D.cel-nfsv4-rpcrdma-version-two], for the following | specified in [RPCRDMA2], for the following NFS protocol versions: | |||
NFS protocol versions: | ||||
o NFS version 2 [RFC1094] | o NFS version 2 [RFC1094] | |||
o NFS version 3 [RFC1813] | o NFS version 3 [RFC1813] | |||
o NFS version 4.0 [RFC7530] | o NFS version 4.0 [RFC7530] | |||
o NFS version 4.1 [RFC5661] | o NFS version 4.1 [RFC5661] | |||
o NFS version 4.2 [RFC7862] | o NFS version 4.2 [RFC7862] | |||
Upper Layer Bindings are also provided for auxiliary protocols used | The current document also provides Upper-Layer Bindings for auxiliary | |||
with NFS versions 2 and 3 (see Section 5). | protocols used with NFS versions 2 and 3 (see Section 5). | |||
This document assumes the reader is already familiar with concepts | This document assumes the reader is already familiar with concepts | |||
and terminology defined in [I-D.cel-nfsv4-rpcrdma-version-two] and | and terminology defined in [RPCRDMA2] and the documents it | |||
the documents it references. | references. | |||
2. Requirements Language | 2. Requirements Language | |||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | |||
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and | "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and | |||
"OPTIONAL" in this document are to be interpreted as described in BCP | "OPTIONAL" in this document are to be interpreted as described in BCP | |||
14 [RFC2119] [RFC8174] when, and only when, they appear in all | 14 [RFC2119] [RFC8174] when, and only when, they appear in all | |||
capitals, as shown here. | capitals, as shown here. | |||
3. Reply Size Estimation | 3. Reply Size Estimation | |||
During the construction of each RPC Call message, a Requester is | During the construction of each RPC Call message, a Requester is | |||
responsible for allocating appropriate resources for receiving the | responsible for allocating appropriate resources for receiving the | |||
corresponding Reply message. If the Requester expects the RPC Reply | corresponding Reply message. If the Requester expects that the RPC | |||
message will be larger than its inline threshold, it MAY provide | Reply message could be larger than its inline threshold, it MAY | |||
Write and/or Reply chunks wherein the Responder can place results and | provide Write chunks wherein the Responder can place results and | |||
the reply's Payload stream. | Reply chunks wherein the Responder can place the reply's Payload | |||
stream. | ||||
4. Upper Layer Binding for NFS Versions 2 and 3 | 4. Upper-Layer Binding for NFS Versions 2 and 3 | |||
The Upper Layer Binding specification in this section applies to NFS | The Upper-Layer Binding specification in this section applies to NFS | |||
version 2 [RFC1094] and NFS version 3 [RFC1813]. For brevity, in | version 2 [RFC1094] and NFS version 3 [RFC1813]. For brevity, in | |||
this document a "Legacy NFS client" refers to an NFS client using | this document, a "Legacy NFS client" refers to an NFS client using | |||
version 2 or version 3 of the NFS RPC program (100003) to communicate | version 2 or version 3 of the NFS RPC program (100003) to communicate | |||
with an NFS server. Likewise, a "Legacy NFS server" is an NFS server | with an NFS server. Likewise, a "Legacy NFS server" is an NFS server | |||
communicating with clients using NFS version 2 or NFS version 3. | communicating with clients using NFS version 2 or NFS version 3. | |||
The following XDR data items in NFS versions 2 and 3 are DDP- | The following XDR data items in NFS versions 2 and 3 are DDP- | |||
eligible: | eligible: | |||
o The opaque file data argument in the NFS WRITE procedure | o The opaque file data argument in the NFS WRITE procedure | |||
o The pathname argument in the NFS SYMLINK procedure | o The pathname argument in the NFS SYMLINK procedure | |||
o The opaque file data result in the NFS READ procedure | o The opaque file data result in the NFS READ procedure | |||
o The pathname result in the NFS READLINK procedure | o The pathname result in the NFS READLINK procedure | |||
All other argument or result data items in NFS versions 2 and 3 are | All other argument or result data items in NFS versions 2 and 3 are | |||
not DDP-eligible. | not DDP-eligible. | |||
A transport error does not give an indication of whether the server | A transport error does not indicate whether the server has processed | |||
has processed the arguments of the RPC Call, or whether the server | the arguments of the RPC Call, or whether the server has accessed or | |||
has accessed or modified client memory associated with that RPC. | modified client memory associated with that RPC. | |||
4.1. Reply Size Estimation | 4.1. Reply Size Estimation | |||
A Legacy NFS client determines the maximum reply size for each | A Legacy NFS client determines the maximum reply size for each | |||
operation using the criteria outlined in Section 3. | operation using the criteria outlined in Section 3. | |||
4.2. RPC Binding Considerations | 4.2. RPC Binding Considerations | |||
Legacy NFS servers traditionally listen for clients on UDP and TCP | Legacy NFS servers traditionally listen for clients on UDP and TCP | |||
port 2049. Additionally, they register these ports with a local | port 2049. Additionally, they register these ports with a local | |||
portmapper service [RFC1833]. | portmapper service [RFC1833]. | |||
A Legacy NFS server supporting RPC-over-RDMA version 2 on such a | A Legacy NFS server supporting RPC-over-RDMA version 2 on such a | |||
network and registering itself with the RPC portmapper MAY choose an | network and registering itself with the RPC portmapper MAY choose an | |||
arbitrary port, or MAY use the alternative well-known port number for | arbitrary port, or MAY use the alternative well-known port number for | |||
its RPC-over-RDMA service (see Section 9). The chosen port MAY be | its RPC-over-RDMA service (see Section 9). The chosen port MAY be | |||
registered with the RPC portmapper using the netids assigned in | registered with the RPC portmapper using the netids assigned in | |||
[I-D.cel-nfsv4-rpcrdma-version-two]. | [RPCRDMA2]. | |||
5. Upper Layer Bindings for NFS Version 2 and 3 Auxiliary Protocols | 4.3. Transport Considerations | |||
NFS versions 2 and 3 are typically deployed with several other | Legacy NFS client implementations often rely on a transport-layer | |||
protocols, sometimes referred to as "NFS auxiliary protocols." These | keep-alive mechanism to detect when a legacy server has become | |||
are distinct RPC programs that define procedures which are not part | unresponsive. When an NFS server is no longer responsive, client- | |||
of the NFS RPC program (100003). The Upper Layer Bindings in this | side keep-alive terminates the connection, which in turn triggers | |||
section apply to: | reconnection and RPC retransmission. | |||
4.3.1. Keep-Alive | ||||
Some RDMA transports (such as the Reliable Connected QP type on | ||||
InfiniBand) have no keep-alive mechanism. Without a disconnect or | ||||
new RPC traffic, such connections can remain alive long after an NFS | ||||
server has become unresponsive or unreachable. Once an NFS client | ||||
has consumed all available RPC-over-RDMA version 2 credits on that | ||||
transport connection, it awaits a reply indefinitely before sending | ||||
another RPC request. | ||||
Legacy NFS clients SHOULD reserve one RPC-over-RDMA version 2 credit | ||||
to use for periodic server or connection health assessment. Either | ||||
peer can use this credit to drive an RPC request on an otherwise idle | ||||
connection, triggering either an affirmative server response or a | ||||
connection termination. | ||||
4.3.2. Replay Detection | ||||
Legacy NFS servers can employ request replay detection to reduce the | ||||
risk of data corruption that could result when a client retransmits a | ||||
request. A legacy NFS server can choose to send a cached response | ||||
when a replay is detected, rather than executing the request again. | ||||
Replay detection is not perfect, but it is usually adequate. | ||||
For legacy NFS servers, replay detection commonly utilizes heuristic | ||||
indicators such as the IP address of the NFS client, the source port | ||||
of the connection, the transaction ID of the request, and the | ||||
contents of the request's RPC and upper-layer protocol headers. In | ||||
short, replay detection is typically based on a connection tuple and | ||||
the request's XID. A legacy NFS client is careful to re-use the same | ||||
source port, if practical, when reconnecting so that legacy NFS | ||||
servers are better able to detect retransmissions. | ||||
However, a legacy NFS client operating over an RDMA transport has no | ||||
control over connection source ports. It is almost certain that an | ||||
RPC request that is retransmitted on a new connection can never be | ||||
detected as a replay if the legacy NFS server includes the connection | ||||
source port in its replay detection heuristics. | ||||
Therefore a legacy NFS server using an RDMA transport should never | ||||
use a legacy NFS client connection's source port as part of RPC | ||||
request replay detection. | ||||
5. Upper-Layer Bindings for NFS Version 2 and 3 Auxiliary Protocols | ||||
Storage administrators typically deploy NFS versions 2 and 3 with | ||||
several other protocols, sometimes referred to as "NFS auxiliary | ||||
protocols." These are distinct RPC programs that define procedures | ||||
that are not part of the NFS RPC program (100003). The Upper-Layer | ||||
Bindings in this section apply to: | ||||
o Versions 2 and 3 of the MOUNT RPC program (100005) [RFC1813] | o Versions 2 and 3 of the MOUNT RPC program (100005) [RFC1813] | |||
o Versions 1, 3, and 4 of the NLM RPC program (100021) [RFC1813] | o Versions 1, 3, and 4 of the NLM RPC program (100021) [RFC1813] | |||
o Version 1 of the NSM RPC program (100024), described in Chapter 11 | o Version 1 of the NSM RPC program (100024), described in Chapter 11 | |||
of [XNFS] | of [XNFS] | |||
o Version 1 of the NFSACL RPC program (100227), which does not have | o Version 1 of the NFSACL RPC program (100227), which does not have | |||
a public definition. NFSACL is treated in this document as a de | a public definition. NFSACL is treated in this document as a de | |||
facto standard, as there are several interoperating | facto standard, as there are several interoperating | |||
implementations. | implementations. | |||
5.1. MOUNT, NLM, and NSM Protocols | 5.1. MOUNT, NLM, and NSM Protocols | |||
Historically, NFS/RDMA implementations have chosen to convey the | Historically, NFS/RDMA implementations have chosen to convey the | |||
MOUNT, NLM, and NSM protocols via TCP. To enable interoperation of | MOUNT, NLM, and NSM protocols via TCP. A legacy NFS server | |||
these protocols when NFS/RDMA is in use, a legacy NFS server MUST | implementation MUST provide support for these protocols via TCP to | |||
provide support for these protocols via TCP. | enable interoperation of these protocols when NFS/RDMA is in use. | |||
5.2. NFSACL Protocol | 5.2. NFSACL Protocol | |||
Legacy clients and servers that support the NFSACL RPC program | Often legacy clients and servers that support the NFSACL RPC program | |||
typically convey NFSACL procedures on the same connection as the NFS | convey NFSACL procedures on the same connection as the NFS RPC | |||
RPC program (100003). This obviates the need for separate rpcbind | program (100003). Utilizing the same connection obviates the need | |||
queries to discover server support for this RPC program. | for separate rpcbind queries to discover server support for this RPC | |||
program. | ||||
ACLs are typically small, but even large ACLs must be encoded and | ACLs are typically small, but even large ACLs must be encoded and | |||
decoded to some degree. Thus no data item in this Upper Layer | decoded to some degree. Thus no data item in this Upper-Layer | |||
Protocol is DDP-eligible. | Protocol is DDP-eligible. | |||
For procedures whose replies do not include an ACL object, the size | For procedures whose replies do not include an ACL object, the size | |||
of a reply is determined directly from the NFSACL RPC program's XDR | of a reply is determined directly from the NFSACL RPC program's XDR | |||
definition. Legacy client implementations should choose a maximum | definition. Legacy client implementations should choose a maximum | |||
size for ACLs based on their own internal limits. | size for ACLs based on internal limits. | |||
6. Upper Layer Binding For NFS Version 4 | 6. Upper-Layer Binding For NFS Version 4 | |||
The Upper Layer Binding specification in this section applies to | The Upper-Layer Binding specification in this section applies to | |||
versions of the NFS RPC program defined in NFS version 4.0 [RFC7530] | versions of the NFS RPC program defined in NFS version 4.0 [RFC7530] | |||
NFS version 4.1 [RFC5661] and NFS version 4.2 [RFC7862] | NFS version 4.1 [RFC5661] and NFS version 4.2 [RFC7862]. | |||
6.1. DDP-Eligibility | 6.1. DDP-Eligibility | |||
Only the following XDR data items in the COMPOUND procedure of all | Only the following XDR data items in the COMPOUND procedure of all | |||
NFS version 4 minor versions are DDP-eligible: | NFS version 4 minor versions are DDP-eligible: | |||
o The opaque data field in the WRITE4args structure | o The opaque data field in the WRITE4args structure | |||
o The linkdata field of the NF4LNK arm in the createtype4 union | o The linkdata field of the NF4LNK arm in the createtype4 union | |||
o The opaque data field in the READ4resok structure | o The opaque data field in the READ4resok structure | |||
o The linkdata field in the READLINK4resok structure | o The linkdata field in the READLINK4resok structure | |||
6.2. Reply Size Estimation | 6.2. Reply Size Estimation | |||
Within NFS version 4, there are certain variable-length result data | Within NFS version 4, there are certain variable-length result data | |||
items whose maximum size cannot be estimated by clients reliably | items whose maximum size cannot be estimated by clients reliably | |||
skipping to change at page 6, line 24 ¶ | skipping to change at page 7, line 32 ¶ | |||
because there is no protocol-specified size limit on these result | because there is no protocol-specified size limit on these result | |||
arrays. These include: | arrays. These include: | |||
o The attrlist4 field | o The attrlist4 field | |||
o Fields containing ACLs such as fattr4_acl, fattr4_dacl, and | o Fields containing ACLs such as fattr4_acl, fattr4_dacl, and | |||
fattr4_sacl | fattr4_sacl | |||
o Fields in the fs_locations4 and fs_locations_info4 data structures | o Fields in the fs_locations4 and fs_locations_info4 data structures | |||
o Fields opaque to the NFS version 4 protocol which pertain to pNFS | o Fields which pertain to pNFS layout metadata, such as loc_body, | |||
layout metadata, such as loc_body, loh_body, da_addr_body, | loh_body, da_addr_body, lou_body, lrf_body, fattr_layout_types, | |||
lou_body, lrf_body, fattr_layout_types, and fs_layout_types | and fs_layout_types | |||
6.2.1. Reply Size Estimation for Minor Version 0 | 6.2.1. Reply Size Estimation for Minor Version 0 | |||
The NFS version 4.0 protocol itself does not impose any bound on the | The NFS version 4.0 protocol itself does not impose any bound on the | |||
size of NFS calls or responses. | size of NFS calls or responses. | |||
Some of the data items enumerated in Section 6.2 (in particular, the | Some of the data items enumerated in Section 6.2 (in particular, the | |||
items related to ACLs and fs_locations) make it difficult to predict | items related to ACLs and fs_locations) make it difficult to predict | |||
the maximum size of NFS version 4.0 replies that interrogate | the maximum size of NFS version 4.0 replies that interrogate | |||
variable-length fattr4 attributes. Client implementations might rely | variable-length fattr4 attributes. Client implementations might rely | |||
on their own internal architectural limits to constrain the reply | upon internal architectural limits to constrain the reply size, but | |||
size, but such limits are not always guaranteed to be reliable. | such limits are not always guaranteed to be reliable. | |||
When an especially large fattr4 result is expected, an NFS version | When an NFS version 4.0 client expects an especially sizeable fattr4 | |||
4.0 client can provide a Reply chunk to enable a large result to be | result, it can provide a Reply chunk to enable that server to return | |||
returned via explicit RDMA. An NFS version 4.0 client can use short | that result via explicit RDMA. An NFS version 4.0 client can use | |||
Reply chunk retry when an NFS COMPOUND containing a GETATTR operation | short Reply chunk retry when an NFS COMPOUND containing a GETATTR | |||
encounters a transport error. | operation encounters a transport error. | |||
6.2.2. Reply Size Estimation for Minor Version 1 and Newer | 6.2.2. Reply Size Estimation for Minor Version 1 and Newer | |||
In NFS version 4.1 and newer minor versions, the csa_fore_chan_attrs | In NFS version 4.1 and newer minor versions, the csa_fore_chan_attrs | |||
argument of the CREATE_SESSION operation contains a | argument of the CREATE_SESSION operation contains a | |||
ca_maxresponsesize field. The value in this field can be taken as | ca_maxresponsesize field. The value in this field can be taken as | |||
the absolute maximum size of replies generated by an NFS version 4.1 | the absolute maximum size of replies generated by an NFS version 4.1 | |||
server. | server. | |||
This value can be used in cases where it is not possible to estimate | A client can use this value in cases where it is not possible to | |||
a reply size upper bound precisely. In practice, objects such as | estimate a reply size upper bound precisely. In practice, objects | |||
ACLs, named attributes, layout bodies, and security labels are much | such as ACLs, named attributes, layout bodies, and security labels | |||
smaller than this maximum. | are much smaller than this maximum. | |||
6.3. RPC Binding Considerations | 6.3. RPC Binding Considerations | |||
NFS version 4 servers are required to listen on TCP port 2049, and | NFS version 4 servers are required to listen on TCP port 2049, and | |||
they are not required to register with an rpcbind service [RFC7530] | they are not required to register with a rpcbind service [RFC7530]. | |||
Therefore, an NFS version 4 server supporting RPC-over-RDMA version 2 | Therefore, an NFS version 4 server supporting RPC-over-RDMA version 2 | |||
MUST use the alternative well-known port number for its RPC-over-RDMA | MUST use the alternative well-known port number for its RPC-over-RDMA | |||
service (see Section 9 Clients SHOULD connect to this well-known port | service (see Section 9 Clients SHOULD connect to this well-known port | |||
without consulting the RPC portmapper (as for NFS version 4 on TCP | without consulting the RPC portmapper (as for NFS version 4 on TCP | |||
transports). | transports). | |||
6.4. NFS COMPOUND Requests | 6.4. NFS COMPOUND Requests | |||
6.4.1. Multiple DDP-eligible Data Items | 6.4.1. Multiple DDP-eligible Data Items | |||
An NFS version 4 COMPOUND procedure can contain more than one | An NFS version 4 COMPOUND procedure can contain more than one | |||
operation that carries a DDP-eligible data item. An NFS version 4 | operation that carries a DDP-eligible data item. An NFS version 4 | |||
client provides XDR Position values in each Read chunk to | client provides XDR Position values in each Read chunk to | |||
disambiguate which chunk is associated with which argument data item. | disambiguate which chunk is associated with which argument data item. | |||
However NFS version 4 server and client implementations must agree in | However, NFS version 4 server and client implementations must agree | |||
advance on how to pair Write chunks with returned result data items. | in advance on how to pair Write chunks with returned result data | |||
items. | ||||
In the following list, a "READ operation" refers to any NFS version 4 | In the following lists, a "READ operation" refers to any NFS version | |||
operation which has a DDP-eligible result data item. The mechanism | 4 operation that has a DDP-eligible result data item. An NFS version | |||
specified in Section 4.3.2 of [I-D.cel-nfsv4-rpcrdma-version-two] is | 4 client applies the mechanism specified in Section 4.3.2 of | |||
applied to this class of operations: | [RPCRDMA2] is applied to this class of operations as follows: | |||
o If an NFS version 4 client wishes all DDP-eligible items in an NFS | o If an NFS version 4 client wishes all DDP-eligible items in an NFS | |||
reply to be conveyed inline, it leaves the Write list empty. | reply to be conveyed inline, it leaves the Write list empty. | |||
An NFS version 4 server applies that mechanism as follows: | ||||
o The first chunk in the Write list MUST be used by the first READ | o The first chunk in the Write list MUST be used by the first READ | |||
operation in an NFS version 4 COMPOUND procedure. The next Write | operation in an NFS version 4 COMPOUND procedure. The next Write | |||
chunk is used by the next READ operation, and so on. | chunk is used by the next READ operation, and so on. | |||
o If an NFS version 4 client has provided a matching non-empty Write | o If an NFS version 4 client has provided a matching non-empty Write | |||
chunk, then the corresponding READ operation MUST return its DDP- | chunk, then the corresponding READ operation MUST return its DDP- | |||
eligible data item using that chunk. | eligible data item using that chunk. | |||
o If an NFS version 4 client has provided an empty matching Write | o If an NFS version 4 client has provided an empty matching Write | |||
chunk, then the corresponding READ operation MUST return all of | chunk, then the corresponding READ operation MUST return all of | |||
skipping to change at page 8, line 22 ¶ | skipping to change at page 9, line 30 ¶ | |||
return an empty Write chunk in that Write list position. | return an empty Write chunk in that Write list position. | |||
o If there are more READ operations than Write chunks, then | o If there are more READ operations than Write chunks, then | |||
remaining NFS Read operations in an NFS version 4 COMPOUND that | remaining NFS Read operations in an NFS version 4 COMPOUND that | |||
have no matching Write chunk MUST return their results inline. | have no matching Write chunk MUST return their results inline. | |||
6.4.2. Chunk List Complexity | 6.4.2. Chunk List Complexity | |||
The RPC-over-RDMA version 2 protocol does not place any limit on the | The RPC-over-RDMA version 2 protocol does not place any limit on the | |||
number of chunks or segments that may appear in Read or Write lists. | number of chunks or segments that may appear in Read or Write lists. | |||
However, for various reasons NFS version 4 server implementations | However, for various reasons, NFS version 4 server implementations | |||
often have practical limits on the number of chunks or segments they | often have practical limits on the number of chunks or segments they | |||
are prepared to process in a single RPC transaction conveyed via RPC- | can process in a single RPC transaction conveyed via RPC-over-RDMA | |||
over-RDMA version 2. | version 2. | |||
These implementation limits are especially important when Kerberos | These implementation limits are especially important when Kerberos | |||
integrity or privacy is in use [RFC7861]. GSS services increase the | integrity or privacy is in use [RFC7861]. GSS services increase the | |||
size of credential material in RPC headers, potentially requiring the | size of credential material in RPC headers, potentially requiring the | |||
use of Long messages. This can increase the complexity of chunk | use of a Long message, which increases the complexity of chunk lists | |||
lists independent of the NFS version 4 COMPOUND being conveyed. | independent of the particular NFS version 4 COMPOUND being conveyed. | |||
In the absence of explicit knowledge of the server's limits, NFS | In the absence of explicit knowledge of the server's limits, NFS | |||
version 4 clients SHOULD follow the prescriptions listed below when | version 4 clients SHOULD follow the prescriptions listed below when | |||
constructing RPC-over-RDMA version 2 messages. NFS version 4 servers | constructing RPC-over-RDMA version 2 messages. NFS version 4 servers | |||
MUST accept and process such requests. | MUST accept and process all such requests. | |||
o The Read list can contain either a Position-Zero Read chunk, one | o The Read list can contain either a Position-Zero Read chunk, one | |||
Read chunk with a non-zero Position, or both. | Read chunk with a non-zero Position, or both. | |||
o The Write list can contain no more than one Write chunk. | o The Write list can contain no more than one Write chunk. | |||
o Any chunk can contain up to sixteen RDMA segments. | o Any chunk can contain up to sixteen RDMA segments. | |||
NFS version 4 clients wishing to send more complex chunk lists can | NFS version 4 clients wishing to send more complex chunk lists can | |||
provide configuration interfaces to bound the complexity of NFS | provide configuration interfaces to bound the complexity of NFS | |||
version 4 COMPOUNDs, limit the number of elements in scatter-gather | version 4 COMPOUNDs, limit the number of elements in scatter-gather | |||
operations, and avoid other sources of chunk overruns at the | operations, and avoid other sources of chunk overruns at the | |||
receiving peer. | receiving peer. | |||
An NFS version 4 server SHOULD return one of the following responses | If an NFS version 4 server receives an RPC request via RPC-over-RDMA | |||
to a client that has sent an RPC transaction via RPC-over-RDMA | version 2 that it cannot process due to chunk list complexity limits, | |||
version 2 which cannot be processed due to chunk list complexity | it SHOULD return one of the following responses to the client: | |||
limits on the server: | ||||
o A problem is detected by the transport layer while parsing the | o A problem is detected by the transport layer while parsing the | |||
transport header in an RPC Call message. The server responds with | transport header in an RPC Call message. The server responds with | |||
an RDMA2_ERROR message with the err field set to ERR_CHUNK. | an RDMA2_ERROR message with the err field set to ERR_CHUNK. | |||
o A problem is detected during XDR decoding of the RPC Call message | o A problem is detected during XDR decoding of the RPC Call message | |||
while the RPC layer reassembles the call's XDR stream. The server | while the RPC layer reassembles the call's XDR stream. The server | |||
responds with an RPC reply with its "reply_stat" field set to | responds with an RPC reply with its "reply_stat" field set to | |||
MSG_ACCEPTED and its "accept_stat" field set to GARBAGE_ARGS. | MSG_ACCEPTED and its "accept_stat" field set to GARBAGE_ARGS. | |||
skipping to change at page 10, line 28 ¶ | skipping to change at page 11, line 34 ¶ | |||
An NFS version 4.0 client advertises netids and ad hoc port addresses | An NFS version 4.0 client advertises netids and ad hoc port addresses | |||
for contacting its NFS version 4.0 callback service using the | for contacting its NFS version 4.0 callback service using the | |||
SETCLIENTID operation. | SETCLIENTID operation. | |||
6.5.2. NFS Version 4.1 Callback | 6.5.2. NFS Version 4.1 Callback | |||
In NFS version 4.1 and newer minor versions, callback operations may | In NFS version 4.1 and newer minor versions, callback operations may | |||
appear on the same connection as is used for NFS version 4 forward | appear on the same connection as is used for NFS version 4 forward | |||
channel client requests. NFS version 4 clients and servers MUST use | channel client requests. NFS version 4 clients and servers MUST use | |||
the approach described in [RFC8167] when backchannel operations are | the approach described in [RFC8167] to convey backchannel operations | |||
conveyed on RPC-over-RDMA version 2 transports. | on an RPC-over-RDMA version 2 transport. | |||
The csa_back_chan_attrs argument of the CREATE_SESSION operation | The csa_back_chan_attrs argument of the CREATE_SESSION operation | |||
contains a ca_maxresponsesize field. The value in this field can be | contains a ca_maxresponsesize field. The value in this field is the | |||
taken as the absolute maximum size of backchannel replies generated | absolute maximum size of backchannel replies generated by a replying | |||
by a replying NFS version 4 client. | NFS version 4 client. | |||
There are no DDP-eligible data items in callback procedures defined | There are no DDP-eligible data items in callback procedures defined | |||
in NFS version 4.1 or NFS version 4.2. However, some callback | in NFS version 4.1 or NFS version 4.2. However, some callback | |||
operations, such as messages that convey device ID information, can | operations, such as messages that convey device ID information, can | |||
be large. Message Continuation or a Long message might be used in | be sizeable. A sender can use Message Continuation or a Long message | |||
this situation. | in this situation. | |||
When an NFS version 4.1 client can support Long Calls in its | When an NFS version 4.1 client can support Long Calls in its | |||
backchannel, it reports a backchannel ca_maxrequestsize that is | backchannel, it reports a backchannel ca_maxrequestsize that is | |||
larger than the connection's inline thresholds. Otherwise an NFS | larger than the connection's inline thresholds. Otherwise, an NFS | |||
version 4 server MUST use only Short messages to convey backchannel | version 4 server MUST use only Short messages to convey backchannel | |||
operations. | operations. | |||
6.6. Session-Related Considerations | 6.6. Session-Related Considerations | |||
The presence of an NFS session (defined in [RFC5661] has no effect on | The presence of an NFS version 4 session (as defined in [RFC5661]) | |||
the operation of RPC-over-RDMA version 2. None of the operations | does not effect the operation of RPC-over-RDMA version 2. None of | |||
introduced to support NFS sessions (e.g. the SEQUENCE operation) | the operations introduced to support NFS sessions (e.g., the SEQUENCE | |||
contain DDP-eligible data items. There is no need to match the | operation) contain DDP-eligible data items. There is no need to | |||
number of session slots with the number of available RPC-over-RDMA | match the number of session slots with the number of available RPC- | |||
version 2 credits. | over-RDMA version 2 credits. | |||
However, there are a few new cases where an RPC transaction can fail. | However, there are a few new cases where an RPC transaction can fail. | |||
For example, a Requester might receive, in response to an RPC | For example, a Requester might receive, in response to an RPC | |||
request, an RDMA2_ERROR message with an rdma_err value of ERR_CHUNK. | request, an RDMA2_ERROR message with a rdma_err value of ERR_CHUNK. | |||
These situations are not different from existing RPC errors which an | These situations are not different from existing RPC errors, which an | |||
NFS session implementation is already prepared to handle for other | NFS session implementation can already handle for other transport | |||
transports. And as with other transports during such a failure, | types. Moreover, there might be no SEQUENCE result available to the | |||
there might be no SEQUENCE result available to the Requester to | Requester to distinguish whether failure occurred before or after the | |||
distinguish whether failure occurred before or after the requested | Responder executed the requested operations. | |||
operations were executed on the Responder. | ||||
When a transport error occurs (e.g. RDMA2_ERROR), the Requester | When a transport error occurs (e.g., an RDMA2_ERROR type message is | |||
proceeds as usual to match the incoming XID value to a waiting RPC | received), the Requester proceeds, as usual, to match the incoming | |||
Call. The RPC transaction is terminated, and the result status is | XID value to a waiting RPC Call. The Requester terminates the RPC | |||
reported to the Upper Layer Protocol. The Requester's session | transaction and reports the result status to the RPC consumer. The | |||
implementation then determines the session ID and slot for the failed | Requester's session implementation then determines the session ID and | |||
request, and performs slot recovery to make that slot usable again. | slot for the failed request and performs slot recovery to make that | |||
If this were not done, that slot could be rendered permanently | slot usable again. Otherwise, that slot could be rendered | |||
unavailable. | permanently unavailable. | |||
When an NFS session is not present (for example, when NFS version 4.0 | When an NFS session is not present (for example, when NFS version 4.0 | |||
is in use), a transport error does not provide an indication of | is in use), a transport error does not indicate whether the server | |||
whether the server has processed the arguments of the RPC Call, or | has processed the arguments of the RPC Call, or whether the server | |||
whether the server has accessed or modified client memory associated | has accessed or modified client memory associated with that RPC. | |||
with that RPC. | ||||
6.7. Transport Considerations | 6.7. Transport Considerations | |||
6.7.1. Congestion Avoidance | 6.7.1. Congestion Avoidance | |||
Section 3.1 of [RFC7530] states: | Section 3.1 of [RFC7530] states: | |||
Where an NFS version 4 implementation supports operation over the | Where an NFS version 4 implementation supports operation over the | |||
IP network protocol, the supported transport layer between NFS and | IP network protocol, the supported transport layer between NFS and | |||
IP MUST be an IETF standardized transport protocol that is | IP MUST be an IETF standardized transport protocol that is | |||
skipping to change at page 12, line 8 ¶ | skipping to change at page 13, line 11 ¶ | |||
Even if NFS version 4.1 is used over a non-IP network protocol, it | Even if NFS version 4.1 is used over a non-IP network protocol, it | |||
is RECOMMENDED that the transport support congestion control. | is RECOMMENDED that the transport support congestion control. | |||
It is permissible for a connectionless transport to be used under | It is permissible for a connectionless transport to be used under | |||
NFS version 4.1; however, reliable and in-order delivery of data | NFS version 4.1; however, reliable and in-order delivery of data | |||
combined with congestion control by the connectionless transport | combined with congestion control by the connectionless transport | |||
is REQUIRED. As a consequence, UDP by itself MUST NOT be used as | is REQUIRED. As a consequence, UDP by itself MUST NOT be used as | |||
an NFS version 4.1 transport. | an NFS version 4.1 transport. | |||
RPC-over-RDMA version 2 is constructed on a platform of RDMA Reliable | RPC-over-RDMA version 2 utilizes only RDMA Reliable Connected QP type | |||
Connected QP type connections [I-D.cel-nfsv4-rpcrdma-version-two] | connections [RPCRDMA2]. RDMA Reliable Connected QPs are reliable, | |||
[RFC5041]. RDMA Reliable Connected QPs are reliable, connection- | connection-oriented transports that guarantee in-order delivery, | |||
oriented transports that guarantee in-order delivery, meeting all | meeting all the above requirements. | |||
above requirements for NFS version 4 transports. | ||||
6.7.2. Retransmission and Keep-alive | 6.7.2. Retransmission and Keep-alive | |||
NFS version 4 client implementations often rely on a transport-layer | NFS version 4 client implementations often rely on a transport-layer | |||
keep-alive mechanism to detect when an NFS version 4 server has | keep-alive mechanism to detect when an NFS version 4 server has | |||
become unresponsive. When an NFS server is no longer responsive, | become unresponsive. When an NFS server is no longer responsive, | |||
client-side keep-alive terminates the connection, which in turn | client-side keep-alive terminates the connection, which in turn | |||
triggers reconnection and RPC retransmission. | triggers reconnection and RPC retransmission. | |||
Some RDMA transports (such as the Reliable Connected QP type on | Some RDMA transports (such as the Reliable Connected QP type on | |||
InfiniBand) have no keep-alive mechanism. Without a disconnect or | InfiniBand) have no keep-alive mechanism. Without a disconnect or | |||
new RPC traffic, such connections can remain alive long after an NFS | new RPC traffic, such connections can remain alive long after an NFS | |||
server has become unresponsive. Once an NFS client has consumed all | server has become unresponsive. Once an NFS client has consumed all | |||
available RPC-over-RDMA version 2 credits on that transport | available RPC-over-RDMA version 2 credits on that transport | |||
connection, it will forever await a reply before sending another RPC | connection, it indefinitely awaits a reply before sending another RPC | |||
request. | request. | |||
NFS version 4 clients SHOULD reserve one RPC-over-RDMA version 2 | NFS version 4 clients SHOULD reserve one RPC-over-RDMA version 2 | |||
credit to use for periodic server or connection health assessment. | credit to use for periodic server or connection health assessment. | |||
This credit can be used to drive an RPC request on an otherwise idle | Either peer can use this credit to drive an RPC request on an | |||
connection, triggering either a quick affirmative server response or | otherwise idle connection, triggering either a quick affirmative | |||
immediate connection termination. | server response or immediate connection termination. | |||
In addition to network partition and request loss scenarios, RPC- | In addition to network partition and request loss scenarios, RPC- | |||
over-RDMA version 2 transport connections can be terminated when a | over-RDMA version 2 transport connections can be terminated when a | |||
Transport header is malformed, Reply messages are larger than receive | Transport header is malformed, Reply messages exceed receive | |||
resources, or when too many RPC-over-RDMA messages are sent at once. | resources, or when too many RPC-over-RDMA messages are sent at once. | |||
In such cases: | In such cases: | |||
o If there is a transport error indicated (ie, RDMA2_ERROR) before | o If a transport error occurs (e.g., an RDMA2_ERROR type message is | |||
the disconnect or instead of a disconnect, the Requester MUST | received) before the disconnect or instead of a disconnect, the | |||
respond to that error as prescribed by the specification of the | Requester MUST respond to that error as prescribed by the | |||
RPC transport. Then the NFS version 4 rules for handling | specification of the RPC transport. Then the NFS version 4 rules | |||
retransmission apply. | for handling retransmission apply. | |||
o If there is a transport disconnect and the Responder has provided | o If there is a transport disconnect and the Responder has provided | |||
no other response for a request, then only the NFS version 4 rules | no other response for a request, then only the NFS version 4 rules | |||
for handling retransmission apply. | for handling retransmission apply. | |||
7. Extending NFS Upper Layer Bindings | 7. Extending NFS Upper-Layer Bindings | |||
RPC programs such as NFS are required to have an Upper Layer Binding | RPC programs such as NFS are required to have an Upper-Layer Binding | |||
specification to interoperate on RPC-over-RDMA version 2 transports | specification to interoperate on RPC-over-RDMA version 2 transports | |||
[I-D.cel-nfsv4-rpcrdma-version-two]. Via standards action, the Upper | [RPCRDMA2]. Via standards action, the Upper-Layer Binding specified | |||
Layer Binding specified in this document can be extended to cover | in this document can be extended to cover versions of the NFS version | |||
versions of the NFS version 4 protocol specified after NFS version 4 | 4 protocol specified after NFS version 4 minor version 2, or to cover | |||
minor version 2, or to cover separately published extensions to an | separately published extensions to an existing NFS version 4 minor | |||
existing NFS version 4 minor version, as described in [RFC8178]. | version, as described in [RFC8178]. | |||
8. Security Considerations | 8. Security Considerations | |||
RPC-over-RDMA version 2 supports all RPC security models, including | RPC-over-RDMA version 2 supports all RPC security models, including | |||
RPCSEC_GSS security and transport-level security [RFC7861]. The | RPCSEC_GSS security and transport-level security [RFC7861]. The | |||
choice of what Direct Data Placement mechanism to convey RPC argument | choice of what Direct Data Placement mechanism to convey RPC argument | |||
and results does not affect this, since it changes only the method of | and results does not affect this since it changes only the method of | |||
data transfer. Because the current document defines only the binding | data transfer. Because the current document defines only the binding | |||
of the NFS protocols atop [I-D.cel-nfsv4-rpcrdma-version-two], all | of the NFS protocols atop [RPCRDMA2], all relevant security | |||
relevant security considerations are therefore to be described at | considerations are, therefore, described at that layer. | |||
that layer. | ||||
9. IANA Considerations | 9. IANA Considerations | |||
The use of direct data placement in NFS introduces a need for an | The use of direct data placement in NFS introduces a need for an | |||
additional port number assignment for networks that share traditional | additional port number assignment for networks that share traditional | |||
UDP and TCP port spaces with RDMA services. The iWARP protocol is | UDP and TCP port spaces with RDMA services. The iWARP protocol is | |||
such an example [RFC5040] [RFC5041]. | such an example [RFC5040] [RFC5041]. | |||
For this purpose, a set of transport protocol port number assignments | For this purpose, the current document specifies a set of transport | |||
is specified by this document. IANA has assigned the following ports | protocol port number assignments. IANA has assigned the following | |||
for NFS/RDMA in the IANA port registry, according to the guidelines | ports for NFS/RDMA in the IANA port registry, according to the | |||
described in [RFC6335]. | guidelines described in [RFC6335]. | |||
nfsrdma 20049/tcp Network File System (NFS) over RDMA | nfsrdma 20049/tcp Network File System (NFS) over RDMA | |||
nfsrdma 20049/udp Network File System (NFS) over RDMA | nfsrdma 20049/udp Network File System (NFS) over RDMA | |||
nfsrdma 20049/sctp Network File System (NFS) over RDMA | nfsrdma 20049/sctp Network File System (NFS) over RDMA | |||
This document should be listed as a reference for the nfsrdma port | The current document should be added as a reference for the nfsrdma | |||
assignments. This document does not alter these assignments. | port assignments. The current document does not alter these | |||
assignments. | ||||
10. References | 10. References | |||
10.1. Normative References | 10.1. Normative References | |||
[I-D.cel-nfsv4-rpcrdma-version-two] | ||||
Lever, C. and D. Noveck, "RPC-over-RDMA Version 2 | ||||
Protocol", draft-cel-nfsv4-rpcrdma-version-two-09 (work in | ||||
progress), May 2019. | ||||
[RFC1833] Srinivasan, R., "Binding Protocols for ONC RPC Version 2", | [RFC1833] Srinivasan, R., "Binding Protocols for ONC RPC Version 2", | |||
RFC 1833, DOI 10.17487/RFC1833, August 1995, | RFC 1833, DOI 10.17487/RFC1833, August 1995, | |||
<https://www.rfc-editor.org/info/rfc1833>. | <https://www.rfc-editor.org/info/rfc1833>. | |||
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | |||
Requirement Levels", BCP 14, RFC 2119, | Requirement Levels", BCP 14, RFC 2119, | |||
DOI 10.17487/RFC2119, March 1997, | DOI 10.17487/RFC2119, March 1997, | |||
<https://www.rfc-editor.org/info/rfc2119>. | <https://www.rfc-editor.org/info/rfc2119>. | |||
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., | [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., | |||
skipping to change at page 15, line 5 ¶ | skipping to change at page 15, line 47 ¶ | |||
November 2016, <https://www.rfc-editor.org/info/rfc7862>. | November 2016, <https://www.rfc-editor.org/info/rfc7862>. | |||
[RFC8167] Lever, C., "Bidirectional Remote Procedure Call on RPC- | [RFC8167] Lever, C., "Bidirectional Remote Procedure Call on RPC- | |||
over-RDMA Transports", RFC 8167, DOI 10.17487/RFC8167, | over-RDMA Transports", RFC 8167, DOI 10.17487/RFC8167, | |||
June 2017, <https://www.rfc-editor.org/info/rfc8167>. | June 2017, <https://www.rfc-editor.org/info/rfc8167>. | |||
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC | [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC | |||
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, | 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, | |||
May 2017, <https://www.rfc-editor.org/info/rfc8174>. | May 2017, <https://www.rfc-editor.org/info/rfc8174>. | |||
[RPCRDMA2] | ||||
Lever, C. and D. Noveck, "RPC-over-RDMA Version 2 | ||||
Protocol", draft-ietf-nfsv4-rpcrdma-version-two-01 (work | ||||
in progress), Jan 2020. | ||||
10.2. Informative References | 10.2. Informative References | |||
[RFC1094] Nowicki, B., "NFS: Network File System Protocol | [RFC1094] Nowicki, B., "NFS: Network File System Protocol | |||
specification", RFC 1094, DOI 10.17487/RFC1094, March | specification", RFC 1094, DOI 10.17487/RFC1094, March | |||
1989, <https://www.rfc-editor.org/info/rfc1094>. | 1989, <https://www.rfc-editor.org/info/rfc1094>. | |||
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS | [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS | |||
Version 3 Protocol Specification", RFC 1813, | Version 3 Protocol Specification", RFC 1813, | |||
DOI 10.17487/RFC1813, June 1995, | DOI 10.17487/RFC1813, June 1995, | |||
<https://www.rfc-editor.org/info/rfc1813>. | <https://www.rfc-editor.org/info/rfc1813>. | |||
skipping to change at page 15, line 36 ¶ | skipping to change at page 16, line 36 ¶ | |||
[RFC8178] Noveck, D., "Rules for NFSv4 Extensions and Minor | [RFC8178] Noveck, D., "Rules for NFSv4 Extensions and Minor | |||
Versions", RFC 8178, DOI 10.17487/RFC8178, July 2017, | Versions", RFC 8178, DOI 10.17487/RFC8178, July 2017, | |||
<https://www.rfc-editor.org/info/rfc8178>. | <https://www.rfc-editor.org/info/rfc8178>. | |||
[XNFS] The Open Group, "Protocols for Interworking: XNFS, Version | [XNFS] The Open Group, "Protocols for Interworking: XNFS, Version | |||
3W", February 1998. | 3W", February 1998. | |||
Acknowledgments | Acknowledgments | |||
Thanks to Tom Talpey, who contributed the text of Section 6.4.2. | Thanks to Tom Talpey, who contributed the text of Section 6.4.2. | |||
Dave Noveck contributed the text of Section 6.6 and Section 7. | Dave Noveck contributed the text of Section 6.6 and Section 7. The | |||
author also wishes to thank Bill Baker and Greg Marsden for their | ||||
support of this work. | ||||
Special thanks go to Transport Area Director Magnus Westerlund, NFSV4 | Special thanks go to Transport Area Director Magnus Westerlund, NFSV4 | |||
Working Group Chairs Spencer Shepler and Brian Pawlowski, and NFSV4 | Working Group Chairs Spencer Shepler, Brian Pawlowski, and Dave | |||
Working Group Secretary Thomas Haynes for their support. The author | Noveck, and NFSV4 Working Group Secretary Thomas Haynes for their | |||
also wishes to thank Bill Baker and Greg Marsden for their support of | support. | |||
this work. | ||||
Author's Address | Author's Address | |||
Charles Lever | Charles Lever | |||
Oracle Corporation | Oracle Corporation | |||
United States of America | United States of America | |||
Email: chuck.lever@oracle.com | Email: chuck.lever@oracle.com | |||
End of changes. 66 change blocks. | ||||
174 lines changed or deleted | 228 lines changed or added | |||
This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |