--- 1/draft-ietf-nfsv4-mv1-msns-update-03.txt 2019-02-05 10:13:13.439034233 -0800 +++ 2/draft-ietf-nfsv4-mv1-msns-update-04.txt 2019-02-05 10:13:13.643039189 -0800 @@ -1,19 +1,19 @@ NFSv4 D. Noveck, Ed. Internet-Draft NetApp Updates: 5661 (if approved) C. Lever Intended status: Standards Track ORACLE -Expires: May 17, 2019 November 13, 2018 +Expires: August 9, 2019 February 5, 2019 NFS Version 4.1 Update for Multi-Server Namespace - draft-ietf-nfsv4-mv1-msns-update-03 + draft-ietf-nfsv4-mv1-msns-update-04 Abstract This document presents necessary clarifications and corrections concerning features related to the use of attributes in NFSv4.1 related to file system location. These features include migration, which transfers responsibility for a file system from one server to another, and facilities to support trunking by allowing discovery of the set of network addresses to use to access a file system. This document updates RFC5661. @@ -26,189 +26,201 @@ Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on May 17, 2019. + This Internet-Draft will expire on August 9, 2019. Copyright Notice - Copyright (c) 2018 IETF Trust and the persons identified as the + Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. + This document may contain material from IETF Documents or IETF + Contributions published or made publicly available before November + 10, 2008. The person(s) controlling the copyright in some of this + material may not have granted the IETF Trust the right to allow + modifications of such material outside the IETF Standards Process. + Without obtaining an adequate license from the person(s) controlling + the copyright in such materials, this document may not be modified + outside the IETF Standards Process, and derivative works of it may + not be created outside the IETF Standards Process, except to format + it for publication as an RFC or to translate it into languages other + than English. + Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 6 3. Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 6 3.2. Summary of Issues . . . . . . . . . . . . . . . . . . . . 8 3.3. Relationship of this Document to [RFC5661] . . . . . . . 10 4. Changes to Section 11 of [RFC5661] . . . . . . . . . . . . . 11 4.1. Updated introductory material for Section 11 of [RFC5661] entitled "Multi-Server Namespace" . . . . . . . . . . . . 12 4.2. New section to be added as the first sub-section of Section 11 of [RFC5661] to be entitled - "Terminology Related to File System Location" . . . . . . 12 + "Terminology Related to File System Location" . . . . . . 13 4.3. Updated Section 11.1 of [RFC5661] to be retitled - "File System Location Attributes" . . . . . . . . . . . . 14 - 4.4. Re-organization of Sections 11.4 and 11.5 of [RFC5661] . 15 + "File System Location Attributes" . . . . . . . . . . . . 15 + 4.4. Re-organization of Sections 11.4 and 11.5 of [RFC5661] . 16 4.5. Updated Section 11.4 of [RFC5661] to be retitled - "Uses of File System Location Information" . . . . . . . 15 + "Uses of File System Location Information" . . . . . . . 16 4.5.1. New section to be added as the first sub-section of Section 11.4 of [RFC5661] to be entitled - "Combining Multiple Uses in a Single Attribute" . . . 16 + "Combining Multiple Uses in a Single Attribute" . . . 17 4.5.2. New section to be added as the second sub-section of Section 11.4 of [RFC5661] to be entitled - "File System Location Attributes and Trunking" . . . 17 + "File System Location Attributes and Trunking" . . . 18 4.5.3. New section to be added as the third sub-section of Section 11.4 of [RFC5661] to be entitled "File System Location Attributes and Connection Type - Selection" . . . . . . . . . . . . . . . . . . . . . 18 + Selection" . . . . . . . . . . . . . . . . . . . . . 19 4.5.4. Updated Section 11.4.1 of [RFC5661] entitled - "File System Replication" . . . . . . . . . . . . . . 19 + "File System Replication" . . . . . . . . . . . . . . 20 4.5.5. Updated Section 11.4.2 of [RFC5661] entitled - "File System Migration" . . . . . . . . . . . . . . . 19 + "File System Migration" . . . . . . . . . . . . . . . 20 4.5.6. Updated Section 11.4.3 of [RFC5661] entitled - "Referrals" . . . . . . . . . . . . . . . . . . . . . 20 + "Referrals" . . . . . . . . . . . . . . . . . . . . . 21 4.5.7. New section to be added after Section 11.4.3 of [RFC5661] to be entitled - "Changes in a File System Location Attribute" . . . . 22 - 5. Re-organization of Section 11.7 of [RFC5661] . . . . . . . . 23 + "Changes in a File System Location Attribute" . . . . 23 + 5. Re-organization of Material Dealing with File System + Transitions . . . . . . . . . . . . . . . . . . . . . . . . . 24 6. New section to be added after Section 11.6 of [RFC5661] - to be entitled "Overview of File Access Transitions" . . . . 23 + to be entitled "Overview of File Access Transitions" . . . . 26 7. New section to be added second after Section 11.6 of [RFC5661] to be entitled - "Effecting Network Endpoint Transitions" . . . . . . . . . . 24 - + "Effecting Network Endpoint Transitions" . . . . . . . . . . 26 8. Updated Section 11.7 of [RFC5661] entitled - "Effecting File System Transitions" . . . . . . . . . . . . . 25 + "Effecting File System Transitions" . . . . . . . . . . . . . 27 8.1. Updated Section 11.7.1 of [RFC5661] entitled - "File System Transitions and Simultaneous Access" . . . . 25 + "File System Transitions and Simultaneous Access" . . . . 28 8.2. Updated Section 11.7.3 of [RFC5661] entitled - "Filehandles and File System Transitions" . . . . . . . . 26 + "Filehandles and File System Transitions" . . . . . . . . 28 8.3. Updated Section 11.7.4 of [RFC5661] entitled - "Fileids and File System Transitions" . . . . . . . . . . 27 + "Fileids and File System Transitions" . . . . . . . . . . 29 8.4. Updated section 11.7.5 of [RFC5661] entitled - "Fsids and File System Transitions" . . . . . . . . . . . 28 + "Fsids and File System Transitions" . . . . . . . . . . . 30 8.4.1. Updated section 11.7.5.1 of [RFC5661] entitled - "File System Splitting" . . . . . . . . . . . . . . . 28 + "File System Splitting" . . . . . . . . . . . . . . . 31 8.5. Updated Section 11.7.6 of [RFC5661] entitled - "The Change Attribute and File System Transitions" . . . 29 + "The Change Attribute and File System Transitions" . . . 31 8.6. Updated Section 11.7.8 of [RFC5661] entitled - "Write Verifiers and File System Transitions" . . . . . . 29 + "Write Verifiers and File System Transitions" . . . . . . 31 8.7. Updated Section 11.7.9 of [RFC5661] entitled "Readdir Cookies and Verifiers and File System - Transitions)" . . . . . . . . . . . . . . . . . . . . . . 29 + Transitions)" . . . . . . . . . . . . . . . . . . . . . . 32 8.8. Updated Section 11.7.10 of [RFC5661] entitled - "File System Data and File System Transitions" . . . . . 30 + "File System Data and File System Transitions" . . . . . 32 8.9. Updated Section 11.7.7 of [RFC5661] entitled - "Lock State and File System Transitions" . . . . . . . . 31 + "Lock State and File System Transitions" . . . . . . . . 33 9. New section to be added after Section 11.11 of [RFC5661] - to be entitled "Transferring State upon Migration" . . . . . 32 + to be entitled "Transferring State upon Migration" . . . . . 34 9.1. Only sub-section within new section to be added to [RFC5661] to be entitled - "Transparent State Migration and pNFS" . . . . . . . . . 32 + "Transparent State Migration and pNFS" . . . . . . . . . 35 10. New section to be added second after Section 11.11 of [RFC5661] to be entitled - "Client Responsibilities when Access is Transitioned" . . . . 34 + "Client Responsibilities when Access is Transitioned" . . . . 36 10.1. First sub-section within new section to be added to [RFC5661] to be entitled - "Client Transition Notifications" . . . . . . . . . . . 34 + "Client Transition Notifications" . . . . . . . . . . . 37 10.2. Second sub-section within new section to be added to + [RFC5661] to be entitled - "Performing Migration Discovery" . . . . . . . . . . . . 37 + "Performing Migration Discovery" . . . . . . . . . . . . 39 10.3. Third sub-section within new section to be added to [RFC5661] to be entitled - "Overview of Client Response to NFS4ERR_MOVED" . . . . . 39 + "Overview of Client Response to NFS4ERR_MOVED" . . . . . 41 10.4. Fourth sub-section within new section to be added to [RFC5661] to be entitled - "Obtaining Access to Sessions and State after Migration" 41 + "Obtaining Access to Sessions and State after Migration" 43 10.5. Fifth sub-section within new section to be added to [RFC5661] to be entitled "Obtaining Access to Sessions and State after Network - Address Transfer" . . . . . . . . . . . . . . . . . . . 43 + Address Transfer" . . . . . . . . . . . . . . . . . . . 45 11. New section to be added third after Section 11.11 of - [RFC5661] to be entitled - "Server Responsibilities Upon Migration" . . . . . . . . . . 43 + "Server Responsibilities Upon Migration" . . . . . . . . . . 46 11.1. First sub-section within new section to be added to [RFC5661] to be entitled "Server Responsibilities in Effecting State Reclaim - after Migration" . . . . . . . . . . . . . . . . . . . . 44 + after Migration" . . . . . . . . . . . . . . . . . . . . 46 11.2. Second sub-section within new section to be added to [RFC5661] to be entitled "Server Responsibilities in Effecting Transparent State - Migration" . . . . . . . . . . . . . . . . . . . . . . . 45 + Migration" . . . . . . . . . . . . . . . . . . . . . . . 47 11.3. Third sub-section within new section to be added to [RFC5661] to be entitled - "Server Responsibilities in Effecting Session Transfer" 46 - 12. fs_locations_info . . . . . . . . . . . . . . . . . . . . . . 49 - 12.1. Updates to treatment of fs_locations_info . . . . . . . 49 + "Server Responsibilities in Effecting Session Transfer" 49 + 12. fs_locations_info . . . . . . . . . . . . . . . . . . . . . . 51 + 12.1. Updates to treatment of fs_locations_info . . . . . . . 51 12.2. Updated Section 11.10 of [RFC5661] entitled - "The Attribute fs_locations_info" . . . . . . . . . . . 49 + "The Attribute fs_locations_info" . . . . . . . . . . . 52 12.2.1. Updated section 11.10.1 of [RFC5661] entitled - "The fs_locations_server4 Structure" . . . . . . . . 53 + "The fs_locations_server4 Structure" . . . . . . . . 56 12.2.2. Updated Section 11.10.2 of [RFC5661] entitled - "The fs_locations_info4 Structure" . . . . . . . . . 60 + "The fs_locations_info4 Structure" . . . . . . . . . 62 12.2.3. Updated Section 11.10.3 of [RFC5661] entitled - "The fs_locations_item4 Structure" . . . . . . . . . 61 - 13. Changes to [RFC5661] outside Section 11 . . . . . . . . . . . 63 + "The fs_locations_item4 Structure" . . . . . . . . . 63 + 13. Changes to [RFC5661] outside Section 11 . . . . . . . . . . . 65 13.1. Updated section 1.7.3.3 of [RFC5661] to be retitled - "Introduction to Multi-Server Namespace" . . . . . . . . 64 + "Introduction to Multi-Server Namespace" . . . . . . . . 66 13.2. Updated Section 2.10.4 of [RFC5661] entitled - "Server Scope" . . . . . . . . . . . . . . . . . . . . . 65 - 13.3. Revised Treatment of NFS4ERR_MOVED . . . . . . . . . . . 66 - 13.4. Revised Discussion of Server_owner changes . . . . . . . 67 - 13.5. Revision to Treatment of EXCHANGE_ID . . . . . . . . . . 68 - 13.6. Revision to Treatment of RECLAIM_COMPLETE . . . . . . . 69 + "Server Scope" . . . . . . . . . . . . . . . . . . . . . 67 + 13.3. Revised Treatment of NFS4ERR_MOVED . . . . . . . . . . . 69 + 13.4. Revised Discussion of Server_owner changes . . . . . . . 69 + 13.5. Revision to Treatment of EXCHANGE_ID . . . . . . . . . . 71 + 13.6. Revision to Treatment of RECLAIM_COMPLETE . . . . . . . 72 13.7. Updated Section 15.1.9 of [RFC5661] entitled - "Reclaim Errors" . . . . . . . . . . . . . . . . . . . . 70 + "Reclaim Errors" . . . . . . . . . . . . . . . . . . . . 72 13.7.1. Updated Section 15.1.9.1 of [RFC5661] entitled - "NFS4ERR_COMPLETE_ALREADY (Error Code 10054)" . . . 70 + "NFS4ERR_COMPLETE_ALREADY (Error Code 10054)" . . . 72 13.7.2. Updated Section 15.1.9.2 of [RFC5661] entitled - "NFS4ERR_GRACE (Error Code 10013)" . . . . . . . . . 70 + "NFS4ERR_GRACE (Error Code 10013)" . . . . . . . . . 73 13.7.3. Updated Section 15.1.9.3 of [RFC5661] entitled - "NFS4ERR_NO_GRACE (Error Code 10033)" . . . . . . . 70 + "NFS4ERR_NO_GRACE (Error Code 10033)" . . . . . . . 73 13.7.4. Updated Section 15.1.9.4 of [RFC5661] entitled - "NFS4ERR_RECLAIM_BAD (Error Code 10034)" . . . . . . 70 + "NFS4ERR_RECLAIM_BAD (Error Code 10034)" . . . . . . 73 13.7.5. Updated Section 15.1.9.5 of [RFC5661] entitled - "NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)" . . . 71 + "NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)" . . . 73 14. Updated Section 18.35 of [RFC5661] entitled - "Operation 42: EXCHANGE_ID - Instantiate Client ID" . . . . . 71 + "Operation 42: EXCHANGE_ID - Instantiate Client ID" . . . . . 74 15. Updated Section 18.51 of [RFC5661] entitled "Operation 58: RECLAIM_COMPLETE - Indicates Reclaims - Finished" . . . . . . . . . . . . . . . . . . . . . . . . . . 89 - 16. Security Considerations . . . . . . . . . . . . . . . . . . . 93 - 17. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 95 - 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 95 - 18.1. Normative References . . . . . . . . . . . . . . . . . . 95 - 18.2. Informative References . . . . . . . . . . . . . . . . . 96 - Appendix A. Classification of Document Sections . . . . . . . . 97 - Appendix B. Updates to [RFC5661] . . . . . . . . . . . . . . . . 98 - Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 102 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 102 + Finished" . . . . . . . . . . . . . . . . . . . . . . . . . . 92 + 16. Security Considerations . . . . . . . . . . . . . . . . . . . 96 + 17. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 98 + 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 98 + 18.1. Normative References . . . . . . . . . . . . . . . . . . 98 + 18.2. Informative References . . . . . . . . . . . . . . . . . 99 + Appendix A. Classification of Document Sections . . . . . . . . 100 + Appendix B. Updates to [RFC5661] . . . . . . . . . . . . . . . . 101 + Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 105 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 106 1. Introduction This document defines the proper handling, within NFSv4.1, of the attributes related to file system location fs_locations and fs_locations_info and how necessary changes in those attributes are to be dealt with. The necessary corrections and clarifications parallel those done for NFSv4.0 in [RFC7931] and [I-D.cel-nfsv4-mv0-trunking-update]. @@ -223,28 +235,31 @@ Unfortunately [RFC5661], while recognizing that these entries can represent different ways to access the same file system, confuses the matter by treating network access paths as "replicas", making it difficult for these attributes to be used to obtain information about the network addresses to be used to access particular file system instances and engendering confusion between two different sorts of transition: those involving a change of network access paths to the same file system instance and those in which there is a shift between two distinct replicas. - When file system location information is used to determine the set of - network addresses to access a particular file system instance (i.e. - to perform trunking discovery), clarification is needed regarding the - interaction of trunking and transitions between file system replicas, - including migration. Unfortunately [RFC5661], while it provided a - method of determining whether two network addresses were connected to - the same server, did not address the issue of trunking discovery, - making it necessary to address it in this document. + This document supplements facilities related to trunking, introduced + in [RFC5661]. For some important terminology regarding trunking, see + Section 3.1. When file system location information is used to + determine the set of network addresses to access a particular file + system instance (i.e. to perform trunking discovery), clarification + is needed regarding the interaction of trunking and transitions + between file system replicas, including migration. Unfortunately + [RFC5661], while it provided a method of determining whether two + network addresses were connected to the same server, did not address + the issue of trunking discovery, making it necessary to address it in + this document. 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 3. Preliminaries 3.1. Terminology @@ -430,21 +445,21 @@ corrected. These are to be dealt with as described in Sections 13 through 15 of the current document. o A revised introductory section regarding multi-server namespace facilities is provided. o A more realistic treatment of server scope is provided, which reflects the more limited co-ordination of locking state adopted by servers actually sharing a common server scope. - o Some confusing text regarding changes in server_owner needs to be + o Some confusing text regarding changes in server_owner has been clarified. o The description of NFS4ERR_MOVED needs to be updated since two different network access paths to the same file system are no longer considered to be two instances of the same file system. o A new treatment of EXCHANGE_ID is needed, replacing that which appeared in Section 18.35 of [RFC5661]. This is necessary since the existing treatment of client id confirmation does not make sense in the context of transparent state migration, in which @@ -455,21 +470,21 @@ to clarify the function of the one-fs flag and clarify how existing clients, that might not properly use this flag, are to be dealt with. 3.3. Relationship of this Document to [RFC5661] The role of this document is to explain and specify a set of needed changes to [RFC5661]. All of these changes are related to the multi- server namespace features of NFSv4.1. - This document contains sections that propose additions to and other + This document contains sections that provide additions to and other modifications of [RFC5661] as well as others that explain the reasons for modifications but do not directly affect existing specifications. In consequence, the sections of this document can be divided into four groups based on how they relate to the eventual updating of the NFSv4.1 specification. Once the update is published, NFSv4.1 will be specified by two documents that need to be read together, until such time as a consolidated specification is produced. o Explanatory sections do not contain any material that is meant to @@ -702,20 +717,47 @@ Previously, issues related to the fact that multiple location entries directed the client to the same file system instance were dealt with in a separate Section 11.5 of [RFC5661]. Because of the new treatment of trunking, these issues now belong within Section 4.5 below. In this new section of the current document, trunking is dealt with in Section 4.5.2 together with the other uses of file system location information described in Sections 4.5.4, 4.5.5, and 4.5.6. + As a result, Section 4.5 which will replace Section 11.4 of [RFC5661] + is substantially different than the section it replaces in that some + existing sections will be replaced by corresponding sections below + while, at the same time, new sections will be added, resulting in a + replacement containing some renumbered sections, as follows: + + o The material in Section 4.5 of the current document, exclusive of + subsections, replaces the material in Section 11.4 of [RFC5661] + exclusive of subsections. + + o Section 4.5.1 of the current document is a new first subsection of + the overall section. In a consolidated document it would appear + as Section 11.4.1. + + o Section 4.5.2 of the current document is a new second subsection + of the overall section. In a consolidated document it would + appear as Section 11.4.2. + + o Each of the Sections 4.5.4, 4.5.5, and 4.5.6 of the current + document replaces (in order) one of the corresponding Sections + 11.4.1, 11.4.2, and 11.4.3 of [RFC5661]. In a consolidated + document they would appear as Sections 11.4.3, 11.4.4, and 11.4.5. + + o Section 4.5.7 of the current document is a new final subsection of + the overall section. In a consolidated document it would appear + as Section 11.4.6. + 4.5. Updated Section 11.4 of [RFC5661] to be retitled "Uses of File System Location Information" The file system location attributes (i.e. fs_locations and fs_locations_info), together with the possibility of absent file systems, provide a number of important facilities in providing reliable, manageable, and scalable data access. When a file system is present, these attributes can provide o The locations of alternative replicas, to be used to access the @@ -799,26 +841,26 @@ Trunking is the use of multiple connections between a client and server in order to increase the speed of data transfer. A client may determine the set of network addresses to use to access a given file system in a number of ways: o When the name of the server is known to the client, it may use DNS to obtain a set of network addresses to use in accessing the server. - o It may fetch the file system location attribute for the filesystem - which will provide either the name of the server (which can be - turned into a set of network addresses using DNS), or it will find - a set of server-trunkable location entries which can provide the - addresses specified by the server as desirable to use to access - the file system in question. + o The client may fetch the file system location attribute for the + file system. This will provide either the name of the server + (which can be turned into a set of network addresses using DNS), + or a set of server-trunkable location entries. Using the latter + alternative, the server can provide addresses it regards as + desirable to use to access the file system in question. The server can provide location entries that include either names or network addresses. It might use the latter form because of DNS- related security concerns or because the set of addresses to be used might require active management by the server. Locations entries used to discover candidate addresses for use in trunking are subject to change, as discussed in Section 4.5.7 below. The client may respond to such changes by using additional addresses once they are verified or by ceasing to use existing ones. The @@ -841,30 +883,35 @@ client may have to choose a connection type with no possibility of changing it within the scope of a single connection. The two file system location attributes differ as to the information made available in this regard. Fs_locations provides no information to support connection type selection. As a result, clients supporting multiple connection types would need to attempt to establish connections using multiple connection types until the one preferred by the client is successfully established. - Fs_locations_info provides a flag, FSLI4TF_RDMA flag. indicating - that RPC-over-RDMA support is available using the specified location - entry. This flag makes it for a convenient for a client wishing to - use RDMA, to establish a TCP connection and then convert to use of - RDMA. After establishing a TCP connection, the step-up facility, can - be used, if available, to convert that connection to RDMA mode. - Otherwise, if RDMA availability is indicated, a new RDMA connection - can be established and it can be bound to the session already - established by the TCP connection, allowing the TCP connection to be - dropped and the session converted to further use in RDMA node. + Fs_locations_info includes a flag, FSLI4TF_RDMA, which, when set + indicates that RPC-over-RDMA support is available using the specified + location entry, by "stepping up" an existing TCP connection to + include support for RDMA operation. This flag makes it convenient + for a client wishing to use RDMA to establish a TCP connection and + then convert to use of RDMA. After establishing a TCP connection, + the step-up facility can be used, if available, to convert that + connection to RDMA mode. + + Irrespective of the particular attribute used, when there is no + indication that a step-up operation can be performed, a client + supporting RDMA operation can establish a new RDMA connection and it + can be bound to the session already established by the TCP + connection, allowing the TCP connection to be dropped and the session + converted to further use in RDMA node. 4.5.4. Updated Section 11.4.1 of [RFC5661] entitled "File System Replication" The fs_locations and fs_locations_info attributes provide alternative file system locations, to be used to access data in place of or in addition to the current file system instance. On first access to a file system, the client should obtain the set of alternate locations by interrogating the fs_locations or fs_locations_info attribute, with the latter being preferred. @@ -879,26 +926,28 @@ various forms of server clustering in which multiple servers provide alternate ways of accessing the same physical file system. How these different modes of file system transition are represented within the fs_locations and fs_locations_info attributes and how the client deals with file system transition issues will be discussed in detail below. 4.5.5. Updated Section 11.4.2 of [RFC5661] entitled "File System Migration" - When a file system is present and becomes absent, clients can be - given the opportunity to have continued access to their data, at an - alternate location, as specified by a file system location attribute. - This migration of access to another replica includes the ability to - retain locks across the transition, either by using lock reclaim or - by taking advantage of Transparent State Migration. + When a file system is present and becomes absent, the NFSv4.1 + protocol provides a means by which clients can be given the + opportunity to have continued access to their data, using a different + replica. The location of this replica is specified by a file system + location attribute. The ensuing migration of access to another + replica includes the ability to retain locks across the transition, + either by using lock reclaim or by taking advantage of Transparent + State Migration. Typically, a client will be accessing the file system in question, get an NFS4ERR_MOVED error, and then use a file system location attribute to determine the new location of the data. When fs_locations_info is used, additional information will be available that will define the nature of the client's handling of the transition to a new server. Such migration can be helpful in providing load balancing or general resource reallocation. The protocol does not specify how the file @@ -948,22 +997,22 @@ multi-server namespace that encompasses file systems located on a wider range of servers. Some likely uses of this facility include establishment of site-wide or organization-wide namespaces, with the eventual possibility of combining such together into a truly global namespace. Referrals occur when a client determines, upon first referencing a position in the current namespace, that it is part of a new file system and that the file system is absent. When this occurs, typically upon receiving the error NFS4ERR_MOVED, the actual location - or locations of the file system can be determined by fetching the a - locations attribute. attribute. + or locations of the file system can be determined by fetching a + locations attribute. The file system location attribute may designate a single file system location or multiple file system locations, to be selected based on the needs of the client. The server, in the fs_locations_info attribute, may specify priorities to be associated with various file system location choices. The server may assign different priorities to different locations as reported to individual clients, in order to adapt to client physical location or to effect load balancing. When both read-only and read-write file systems are present, some of the read-only locations might not be absolutely up-to-date (as they would @@ -994,23 +1043,25 @@ providing a large set of pure referrals to all of the included file systems. Alternatively, a single multi-server namespace may be administratively segmented with separate referral file systems (on separate servers) for each separately administered portion of the namespace. The top-level referral file system or any segment may use replicated referral file systems for higher availability. Generally, multi-server namespaces are for the most part uniform, in that the same data made available to one client at a given location in the namespace is made available to all clients at that location. - However, as described above, there are facilities provided that allow - different clients to be directed different sets of data, to enable - adaptation to such client characteristics as CPU architecture. + However, there are facilities provided that allow different clients + to be directed to different sets of data, to enable adaptation to + such client characteristics as CPU architecture. These facilities + are described in Section 11.10.3 of [RFC5661] and in Section 12.2.3 + of the current document. 4.5.7. New section to be added after Section 11.4.3 of [RFC5661] to be entitled "Changes in a File System Location Attribute" Although clients will typically fetch a file system location attribute when first accessing a file system and when NFS4ERR_MOVED is returned, a client can choose to fetch the attribute periodically, in which case the value fetched may change over time. For clients not prepared to access multiple replicas simultaneously @@ -1044,52 +1095,103 @@ with one currently in use, the client is free to use the new replica immediately. o When a replica currently in use is deleted from the list, the client need not cease using it immediately. However, since the server may subsequently force such use to cease (by returning NFS4ERR_MOVED), clients might decide to limit the need for later state transfer. For example, new opens might be done on other replicas, rather than on one not present in the list. -5. Re-organization of Section 11.7 of [RFC5661] +5. Re-organization of Material Dealing with File System Transitions - The material in Section 11.7 of [RFC5661] has been reorganized and - augmented as specified below: + The material relating to file system transition, previously contained + in Section 11.7 of [RFC5661] has been reorganized and augmented as + described below: o Because there can be a shift of the network access paths used to access a file system instance without any shift between replicas, a new Section 6 in the current document distinguishes between those cases in which there is a shift between distinct replicas and those involving a shift in network access paths with no shift between replicas. As a result, a new Section 7 in the current document deals with network address transitions while the bulk of the former - Section 11.7 (in [RFC5661]) is replaced by Section 8 in the - current document which is now limited to cases in which there is a - shift between two different sets of replicas. + Section 11.7 (in [RFC5661]) is extensively modified as described + by Section 8 in the current document which is now limited to cases + in which there is a shift between two different sets of replicas. o The additional Section 9 in the current document discusses the case in which a shift to a different replica is made and state is transferred to allow the client the ability to have continues access to the accumulated locking state on the new server. o The additional Section 10 in the current document discusses the client's response to access transitions and how it determines whether migration has occurred, and how it gets access to any transferred locking and session state. o The additional Section 11 in the current document discusses the responsibilities of the source and destination servers when transferring locking and session state. + This re-organization has caused a renumbering of the sections within + Section 11 of [RFC5661] as described below: + + o The new Sections 6 and 7 in the current document would appear as + Sections 11.7 and 11.8 respectively, in an eventual consolidated + document. + + o Section 11.7 of [RFC5661] will be modified as described in + Section 8. The necessary modifications reflect the fact that this + section will only deal with transitions between replicas while + transitions between network addresses are dealt with in other + sections. Details of the reorganization are described later in + this section. The updated section would appear as Section 11.9 in + an eventual consolidated document. + + o The additional Sections 9, 10, and 11 in the current document + would appear as Sections 11.10, 11.11, and 11.12 respectively, in + an eventual consolidated document. + + o Consequently, Sections 11.8, 11.9, 11.10, and 11.11 in [RFC5661] + would appear as Sections 11.13, 11.14, 11.15, and 11.16 + respectively, in an eventual consolidated document. + + As part of this general re-organization, Section 11.7 of [RFC5661] + will be modified as described below: + + o Sections 11.7 and 11.7.1 of [RFC5661] are to be replaced by + Sections 8 8.1 respectively of the current document These sections + would appear as Section 11.9 and 11.9.1 in an eventual + consolidated document. + + o Section 11.7.2 (and included subsections) of [RFC5661] are to be + deleted. + + o Sections 11.7.3, 11.7.4. 11.7.5, 11.7.5.1, and 11.7.6 [RFC5661] + are to be replaced by Sections 8.2, 8.3, 8.4, 8.4.1, and 8.5 + respectively of the current document. These sections would appear + as Sections 11.9.2, 11.9.3 11.9.4, 11.9.4.1 and 11.9.5 in an + eventual consolidated document. + + o Section 11.77 of [RFC5661] is to be replaced by Section 8.9. + Because this sub-section has been moved to the end of the section + dealing with file system transitions, it would appear as + Section 11.9.9 in an eventual consolidated document. + + o Sections 11.7.8, 11.7.9. and 11.7.10 of [RFC5661] are to be + replaced by Sections 8.6, 8.7, and 8.8 respectively of the current + document. These sections would appear as Sections 11.9.6, 11.9.7 + and 11.9.8 in an eventual consolidated document. + 6. New section to be added after Section 11.6 of [RFC5661] to be entitled "Overview of File Access Transitions" File access transitions are of two types: o Those that involve a transition from accessing the current replica to another one in connection with either replication or migration. How these are dealt with is discussed in Section 8 of the current document. @@ -1617,30 +1719,30 @@ system that is no longer accessible at the address previously used to access it, the error NFS4ERR_MOVED is returned. Exceptions are made to allow such file handles to be used when interrogating a file system location attribute. This enables a client to determine a new replica's location or a new network access path. This condition continues on subsequent attempts to access the file system in question. The only way the client can avoid the error - is to cease accessing the filesystem in question at its old server - location and access it instead using a different address at which - it is now available. + is to cease accessing the file system in question at its old + server location and access it instead using a different address at + which it is now available. o Whenever a SEQUENCE operation is sent by a client to a server which generated state held on that client which is associated with a file system that is no longer accessible on the server at which - it was previously available, a lease-migrated indication, in the - form the SEQ4_STATUS_LEASE_MOVED status bit being set, appears in - the response. + it was previously available, the response will contain a lease- + migrated indication, with the SEQ4_STATUS_LEASE_MOVED status bit + being set. This condition continues until the client acknowledges the notification by fetching a file system location attribute for the file system whose network access path is being changed. When there are multiple such file systems, a location attribute for each such file system needs to be fetched. The location attribute for all migrated file system needs to be fetched in order to clear the condition. Even after the condition is cleared, the client needs to respond by using the location information to access the file system at its new location to ensure that leases are not @@ -1979,21 +2081,21 @@ o In the state merger case, it is possible that the server has not attempted Transparent State Migration, in which case state may have been lost without it being reflected in the SEQ4_STATUS bits. To determine whether this has happened, the client can use TEST_STATEID to check whether the stateids created on the source server are still accessible on the destination server. Once a single stateid is found to have been successfully transferred, the client can conclude that Transparent State Migration was begun and any failure to transport all of the stateids will be reflected in - the SEQ4_STATUS bits. Otherwise. Transparent State Migration has + the SEQ4_STATUS bits. Otherwise, Transparent State Migration has not occurred. o In a case in which Transparent State Migration has not occurred, the client can use the per-fs grace period provided by the destination server to reclaim locks that were held on the source server. o In a case in which Transparent State Migration has occurred, and no lock state was lost (as shown by SEQ4_STATUS flags), no lock reclaim is necessary. @@ -2082,24 +2184,24 @@ the client has reclaimed its locks, it indicates the completion of lock reclamation by performing a RECLAIM_COMPLETE specifying rca_one_fs as TRUE. While it is not necessary for source and destination servers to co- operate to transfer information about locks, implementations are well-advised to consider transferring the following useful information: o If information about the set of clients that have locking state - for the transferred file system, the destination server will be - able to terminate the grace period once all such clients have - reclaimed their locks, allowing normal locking activity to resume - earlier than it would have otherwise. + for the transferred file system is made available, the destination + server will be able to terminate the grace period once all such + clients have reclaimed their locks, allowing normal locking + activity to resume earlier than it would have otherwise. o Locking summary information for individual clients (at various possible levels of detail) can detect some instances in which clients do not accurately represent the locks held on the source server. 11.2. Second sub-section within new section to be added to [RFC5661] to be entitled "Server Responsibilities in Effecting Transparent State Migration" @@ -2109,29 +2211,29 @@ the file system being migrated. In addition to client id string and verifier, the source server needs to provide, for each stateid: o The stateid including the current sequence value. o The associated client ID. o The handle of the associated file. o The type of the lock, such as open, byte-range lock, delegation, - layout. + or layout. o For locks such as opens and byte-range locks, there will be information about the owner(s) of the lock. o For recallable/revocable lock types, the current recall status needs to be included. - o For each lock type there will by type-specific information, such + o For each lock type, there will be type-specific information, such as share and deny modes for opens and type and byte ranges for byte-range locks and layouts. A further server responsibility concerns locks that are revoked or otherwise lost during the process of file system migration. Because locks that appear to be lost during the process of migration will be reclaimed by the client, the servers have to take steps to ensure that locks revoked soon before or soon after migration are not inadvertently allowed to be reclaimed in situations in which the continuity of lock possession cannot be assured. @@ -3110,21 +3213,22 @@ When two replies from EXCHANGE_ID, each from two different server network addresses, have the same server scope, there are a number of ways a client can validate that the common server scope is due to two servers cooperating in a group. o If both EXCHANGE_ID requests were sent with RPCSEC_GSS ([RFC2203], [RFC5403], [RFC7861]) authentication and the server principal is the same for both targets, the equality of server scope is validated. It is RECOMMENDED that two servers intending to share - the same server scope also share the same principal name. + the same server scope also share the same principal name, + simplifying the client's task of validating server scope. o The client may accept the appearance of the second server in the fs_locations or fs_locations_info attribute for a relevant file system. For example, if there is a migration event for a particular file system or there are locks to be reclaimed on a particular file system, the attributes for that particular file system may be used. The client sends the GETATTR request to the first server for the fs_locations or fs_locations_info attribute with RPCSEC_GSS authentication. It may need to do this in advance of the need to verify the common server scope. If the client @@ -3173,21 +3277,21 @@ lock reclaim as it relates to such reconfiguration events, see Section 8.4.2.1 While this paragraph is literally true in that such reconfiguration events can happen and clients have to deal with them, it is confusing in that it can be read as suggesting that clients have to deal with them without disruption, which in general is impossible. This has led to confusion especially since the text is not very clear about the actions that might need to be done since: - o The cases of change which are very disruptive (e.g. change if + o The cases of change which are very disruptive (e.g. change of server scope) are not sufficiently distinguished from those that simply involve a change of trunking modes (i.e. change server_owner minor id) o There is an undue focus on the effect of such changes as they affect the comparison with corresponding value from other servers, without fully dealing with the issue that result from value discontinuity within a single server. Because of these issues the paragraph which appears at the end of @@ -4592,52 +4697,52 @@ o Sections 6 and 7 are additional sections. o Sections 8 through 8.9, a total of ten sections, are all replacement sections. o Sections 9 through 11.3, a total of twelve sections, are all additional sections. o Section 12.1 is explanatory. - o Sections 12.2 throuhy 12.2.3, a total of four sections, are all - replacemebt sections. + o Sections 12.2 through 12.2.3, a total of four sections, are all + replacement sections. o Section 13 is explanatory. o Sections 13.1 and 13.2 are replacement sections. o Sections 13.3 and 13.4 are editing sections. o Sections 13.5 and 13.6 is explanatory. - o Section 13.7 is a replcement section, which consists of a total of - six sections. + o Section 13.7 is a replacement section, which consists of a total + of six sections. o Section 14 is a replacement section, which consists of a total of five sections. o Section 15 is a replacement section, which consists of a total of five sections. o Section 16 is an editing section. o Section 17 through Acknowledgments, a total of six sections, are all explanatory. To summarize: o There are seventeen explanatory sections. o There are thirty-seven replacement sections. - o There are eightteen additional sections. + o There are eighteen additional sections. o There are three editing sections. Appendix B. Updates to [RFC5661] In this appendix, we proceed through [RFC5661] identifying sections as unchanged, modified, deleted, or replaced and indicating where additional sections from the current document would appear in an eventual consolidated description of NFSv4.1. In this presentation, when section X is referred to, it denotes that section plus all @@ -4688,63 +4793,87 @@ current document appears next. o Section 11.5 is to be deleted. o Section 11.6 is unchanged. o New sections corresponding to Sections 6 and 7 from the current document appear next. o Section 11.7 is replaced by Section 8 from the current - document. For details regarding subsections see below. + document. For details regarding subsections see below. This + section (with included subsections) would appear as + Section 11.9 in an eventual consolidated document. In addition + to the shift from Section 11.7 to Section 11.9, subsections + within it would be affected by the deletion of Section 11.7.2 + and the move of Section 11.7.7 to be the last sub-section. o Section 11.7.1 is replaced by Section 8.1 from the current - document. + document. In an eventual consolidated document, it would + appear as Section 11.9.1. o Sections 11.7.2, 11.7.2.1, and 11.7.2.2 are deleted. o Section 11.7.3 is replaced by Section 8.2 from the current - document. + document. In an eventual consolidated document, it would + appear as Section 11.9.2. o Section 11.7.4 is replaced by Section 8.3 from the current - document. + document. In an eventual consolidated document, it would + appear as Section 11.9.3. o Sections 11.7.5 and 11.7.5.1 are replaced by Sections 8.4 - and 8.4.1 respectively, from the current document. + and 8.4.1 respectively, from the current document. In an + eventual consolidated document, they would appear as + Sections 11.9.4 and 11.9.4.1. o Section 11.7.6 is replaced by Section 8.5 from the current - document. + document. In an eventual consolidated document, it would + appear as Section 11.9.5. o Section 11.7.7, exclusive of subsections, is replaced by Section 8.9 from the current document. Sections 11.7.7.1 - and 11.7.7.2 are unchanged. + and 11.7.7.2 are unchanged. Because this section will + become the last sub-section of the replacement for + Section 11.7, it would appear as Section 11.9.9 in an + eventual consolidated document. o Section 11.7.8 is replaced by Section 8.6 from the current - document. + document. In an eventual consolidated document, it would + appear as Section 11.9.6. o Section 11.7.9 is replaced by Section 8.7 from the current - document. + document. In an eventual consolidated document, it would + appear as Section 11.9.7. o Section 11.7.10 is replaced by Section 8.8 from the current - document. - - o Sections 11.8, 11.8.1, 11.8.2, and 11.9, are unchanged. - - o Sections 11.10, 11.10.1, 11.10.2, and 11.10.3 are replaced by - Sections 12.2 through 12.2.3 from the current document. - - o Section 11.11 is unchanged. + document. In an eventual consolidated document, it would + appear as Section 11.9.8. o New sections corresponding to Sections 9, 10, and 11 from the current document appear next as additional sub-sections of Section 11. Each of these has subsections, so there is a total - of seventeen sections added. + of seventeen sections added. These sections would appear as + Sections 11.10, 11.11, and 11.12 respectively in an eventual + consolidated document. + + o Sections 11.8, 11.8.1, 11.8.2, and 11.9, are unchanged although + they would be renumber as Sections 11.13 (with included + subsections) and 11.14 in an eventual consolidated document. + + o Sections 11.10, 11.10.1, 11.10.2, and 11.10.3 are replaced by + Sections 12.2 through 12.2.3 from the current document. These + sections would appear as Section 11.15 (with included + subsections) in an eventual consolidated document. + + o Section 11.11 is unchanged, although it would appear as + Section 11.16 in an eventual consolidated document. o Sections 12 through 14 are unchanged. o Section 15 is unmodified except that * The description of NFS4ERR_MOVED in Section 15.1 is revised as described in Section 13.3 of the current document. * The description of the reclaim-related errors in section 15.1.9 is replaced by the revised descriptions in Section 13.7 of the