--- 1/draft-ietf-nfsv4-mv1-msns-update-02.txt 2018-11-13 06:13:15.294029206 -0800 +++ 2/draft-ietf-nfsv4-mv1-msns-update-03.txt 2018-11-13 06:13:15.498034054 -0800 @@ -1,186 +1,245 @@ NFSv4 D. Noveck, Ed. Internet-Draft NetApp Updates: 5661 (if approved) C. Lever Intended status: Standards Track ORACLE -Expires: April 24, 2019 October 21, 2018 +Expires: May 17, 2019 November 13, 2018 NFS Version 4.1 Update for Multi-Server Namespace - draft-ietf-nfsv4-mv1-msns-update-02 + draft-ietf-nfsv4-mv1-msns-update-03 Abstract This document presents necessary clarifications and corrections - concerning features related to the use of location-related attributes - in NFSv4.1. These include migration, which transfers responsibility - for a file system from one server to another, and facilities to - support trunking by allowing discovery of the set of network - addresses to use to access a file system. This document updates - RFC5661. + concerning features related to the use of attributes in NFSv4.1 + related to file system location. These features include migration, + which transfers responsibility for a file system from one server to + another, and facilities to support trunking by allowing discovery of + the set of network addresses to use to access a file system. This + document updates RFC5661. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on April 24, 2019. + This Internet-Draft will expire on May 17, 2019. Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents - 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 - 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4 - 3. Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . 4 - 3.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 - 3.2. Summary of Issues . . . . . . . . . . . . . . . . . . . . 7 - 3.3. Relationship of this Document to RFC5661 . . . . . . . . 9 - 4. Changes to Section 11 of RFC5661 . . . . . . . . . . . . . . 10 - 4.1. Multi-Server Namespace (as updated) . . . . . . . . . . . 11 - 4.2. Location-related Terminology (to be added) . . . . . . . 11 - 4.3. Location Attributes (as updated) . . . . . . . . . . . . 13 - 4.4. Re-organization of Sections 11.4 and 11.5 of RFC5661 . . 14 - 4.5. Uses of Location Information (as updated) . . . . . . . . 14 - 4.5.1. Combining Multiple Uses in a Single Attribute (to be - added) . . . . . . . . . . . . . . . . . . . . . . . 15 - 4.5.2. Location Attributes and Trunking (to be added) . . . 16 - 4.5.3. Location Attributes and Connection Type Selection (to - be added) . . . . . . . . . . . . . . . . . . . . . . 16 - 4.5.4. File System Replication (as updated) . . . . . . . . 17 - 4.5.5. File System Migration (as updated) . . . . . . . . . 17 - 4.5.6. Referrals (as updated) . . . . . . . . . . . . . . . 19 - 4.5.7. Changes in a Location Attribute (to be added) . . . . 20 - 5. Re-organization of Section 11.7 of RFC5661 . . . . . . . . . 21 - 6. Overview of File Access Transitions (to be added) . . . . . . 22 - 7. Effecting Network Endpoint Transitions (to be added) . . . . 22 - 8. Effecting File System Transitions (as updated) . . . . . . . 23 - 8.1. File System Transitions and Simultaneous Access (as - updated) . . . . . . . . . . . . . . . . . . . . . . . . 24 - 8.2. Filehandles and File System Transitions (as updated) . . 24 - 8.3. Fileids and File System Transitions (as updated) . . . . 25 - 8.4. Fsids and File System Transitions (as updated) . . . . . 26 - 8.4.1. File System Splitting (as updated) . . . . . . . . . 26 - 8.5. The Change Attribute and File System Transitions (as - updated) . . . . . . . . . . . . . . . . . . . . . . . . 27 - 8.6. Write Verifiers and File System Transitions (as updated) 27 - 8.7. Readdir Cookies and Verifiers and File System Transitions - (as updated) . . . . . . . . . . . . . . . . . . . . . . 28 - 8.8. File System Data and File System Transitions (as updated) 28 - 8.9. Lock State and File System Transitions (as updated) . . . 29 - 9. Transferring State upon Migration (to be added) . . . . . . . 30 - 9.1. Transparent State Migration and pNFS (to be added) . . . 31 - 10. Client Responsibilities when Access is Transitioned (to be - added) . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 5 + 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 6 + 3. Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . 6 + 3.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 6 + 3.2. Summary of Issues . . . . . . . . . . . . . . . . . . . . 8 + 3.3. Relationship of this Document to [RFC5661] . . . . . . . 10 + 4. Changes to Section 11 of [RFC5661] . . . . . . . . . . . . . 11 + 4.1. Updated introductory material for Section 11 of [RFC5661] + entitled "Multi-Server Namespace" . . . . . . . . . . . . 12 + 4.2. New section to be added as the first sub-section of + Section 11 of [RFC5661] to be entitled + "Terminology Related to File System Location" . . . . . . 12 + 4.3. Updated Section 11.1 of [RFC5661] to be retitled + "File System Location Attributes" . . . . . . . . . . . . 14 + 4.4. Re-organization of Sections 11.4 and 11.5 of [RFC5661] . 15 + 4.5. Updated Section 11.4 of [RFC5661] to be retitled + "Uses of File System Location Information" . . . . . . . 15 + 4.5.1. New section to be added as the first sub-section of + Section 11.4 of [RFC5661] to be entitled + "Combining Multiple Uses in a Single Attribute" . . . 16 + 4.5.2. New section to be added as the second sub-section of + Section 11.4 of [RFC5661] to be entitled + "File System Location Attributes and Trunking" . . . 17 + 4.5.3. New section to be added as the third sub-section of + Section 11.4 of [RFC5661] to be entitled + "File System Location Attributes and Connection Type + Selection" . . . . . . . . . . . . . . . . . . . . . 18 + 4.5.4. Updated Section 11.4.1 of [RFC5661] entitled + "File System Replication" . . . . . . . . . . . . . . 19 + 4.5.5. Updated Section 11.4.2 of [RFC5661] entitled + "File System Migration" . . . . . . . . . . . . . . . 19 + 4.5.6. Updated Section 11.4.3 of [RFC5661] entitled + "Referrals" . . . . . . . . . . . . . . . . . . . . . 20 + 4.5.7. New section to be added after Section 11.4.3 of + [RFC5661] to be entitled + "Changes in a File System Location Attribute" . . . . 22 + 5. Re-organization of Section 11.7 of [RFC5661] . . . . . . . . 23 + 6. New section to be added after Section 11.6 of [RFC5661] + to be entitled "Overview of File Access Transitions" . . . . 23 + 7. New section to be added second after Section 11.6 of + [RFC5661] to be entitled + "Effecting Network Endpoint Transitions" . . . . . . . . . . 24 - 10.1. Client Transition Notifications (to be added) . . . . . 32 - 10.2. Performing Migration Discovery (to be added) . . . . . . 35 - 10.3. Overview of Client Response to NFS4ERR_MOVED (to be - added) . . . . . . . . . . . . . . . . . . . . . . . . . 37 - 10.4. Obtaining Access to Sessions and State after Migration - (to be added) . . . . . . . . . . . . . . . . . . . . . 39 - 10.5. Obtaining Access to Sessions and State after Network - Address Transfer (to be added) . . . . . . . . . . . . . 41 - 11. Server Responsibilities Upon Migration (to be added) . . . . 41 - 11.1. Server Responsibilities in Effecting State Reclaim after - Migration (to be added) . . . . . . . . . . . . . . . . 42 - 11.2. Server Responsibilities in Effecting Transparent State - Migration (to be added) . . . . . . . . . . . . . . . . 42 - 11.3. Server Responsibilities in Effecting Session Transfer - (to be added) . . . . . . . . . . . . . . . . . . . . . 44 - 12. fs_locations_info . . . . . . . . . . . . . . . . . . . . . . 46 - 12.1. Updates to treatment of fs_locations_info . . . . . . . 47 - 12.2. The Attribute fs_locations_info (as updated) . . . . . . 47 - 12.2.1. The fs_locations_server4 Structure (as updated) . . 51 - 12.2.2. The fs_locations_info4 Structure (as updated) . . . 57 - 12.2.3. The fs_locations_item4 Structure (as updated) . . . 59 - 13. Changes to RFC5661 outside Section 11 . . . . . . . . . . . . 61 - 13.1. (Introduction to) Multi-Server Namespace (as updated) . 62 - 13.2. Server Scope (as updated) . . . . . . . . . . . . . . . 62 - 13.3. Revised Treatment of NFS4ERR_MOVED . . . . . . . . . . . 64 - 13.4. Revised Discussion of Server_owner changes . . . . . . . 65 - 13.5. Revision to Treatment of EXCHANGE_ID . . . . . . . . . . 65 - 13.6. Revision to Treatment of RECLAIM_COMPLETE . . . . . . . 67 - 13.7. Reclaim Errors (as updated) . . . . . . . . . . . . . . 67 - 13.7.1. NFS4ERR_COMPLETE_ALREADY (as updated; Error Code - 10054) . . . . . . . . . . . . . . . . . . . . . . . 67 - 13.7.2. NFS4ERR_GRACE (as updated; Error Code 10013) . . . . 67 - 13.7.3. NFS4ERR_NO_GRACE (as updated; Error Code 10033) . . 67 - 13.7.4. NFS4ERR_RECLAIM_BAD (as updated; Error Code 10034) . 68 - 13.7.5. NFS4ERR_RECLAIM_CONFLICT (as updated; Error Code - 10035) . . . . . . . . . . . . . . . . . . . . . . . 68 - 14. Operation 42: EXCHANGE_ID - Instantiate Client ID (as - updated) . . . . . . . . . . . . . . . . . . . . . . . . . . 68 - 15. Operation 58: RECLAIM_COMPLETE - Indicates Reclaims Finished - (as updated) . . . . . . . . . . . . . . . . . . . . . . . . 86 - 16. Security Considerations . . . . . . . . . . . . . . . . . . . 90 - 17. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 92 - 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 92 - 18.1. Normative References . . . . . . . . . . . . . . . . . . 92 - 18.2. Informative References . . . . . . . . . . . . . . . . . 93 - Appendix A. Classification of Document Sections . . . . . . . . 94 - Appendix B. Updates to RFC5661 . . . . . . . . . . . . . . . . . 95 - Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 98 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 99 + 8. Updated Section 11.7 of [RFC5661] entitled + "Effecting File System Transitions" . . . . . . . . . . . . . 25 + 8.1. Updated Section 11.7.1 of [RFC5661] entitled + "File System Transitions and Simultaneous Access" . . . . 25 + 8.2. Updated Section 11.7.3 of [RFC5661] entitled + "Filehandles and File System Transitions" . . . . . . . . 26 + 8.3. Updated Section 11.7.4 of [RFC5661] entitled + "Fileids and File System Transitions" . . . . . . . . . . 27 + 8.4. Updated section 11.7.5 of [RFC5661] entitled + "Fsids and File System Transitions" . . . . . . . . . . . 28 + 8.4.1. Updated section 11.7.5.1 of [RFC5661] entitled + "File System Splitting" . . . . . . . . . . . . . . . 28 + 8.5. Updated Section 11.7.6 of [RFC5661] entitled + "The Change Attribute and File System Transitions" . . . 29 + 8.6. Updated Section 11.7.8 of [RFC5661] entitled + "Write Verifiers and File System Transitions" . . . . . . 29 + 8.7. Updated Section 11.7.9 of [RFC5661] entitled + "Readdir Cookies and Verifiers and File System + Transitions)" . . . . . . . . . . . . . . . . . . . . . . 29 + 8.8. Updated Section 11.7.10 of [RFC5661] entitled + "File System Data and File System Transitions" . . . . . 30 + 8.9. Updated Section 11.7.7 of [RFC5661] entitled + "Lock State and File System Transitions" . . . . . . . . 31 + 9. New section to be added after Section 11.11 of [RFC5661] + to be entitled "Transferring State upon Migration" . . . . . 32 + 9.1. Only sub-section within new section to be added to + [RFC5661] to be entitled + "Transparent State Migration and pNFS" . . . . . . . . . 32 + 10. New section to be added second after Section 11.11 of + [RFC5661] to be entitled + "Client Responsibilities when Access is Transitioned" . . . . 34 + 10.1. First sub-section within new section to be added to + [RFC5661] to be entitled + "Client Transition Notifications" . . . . . . . . . . . 34 + 10.2. Second sub-section within new section to be added to + [RFC5661] to be entitled + "Performing Migration Discovery" . . . . . . . . . . . . 37 + 10.3. Third sub-section within new section to be added to + [RFC5661] to be entitled + "Overview of Client Response to NFS4ERR_MOVED" . . . . . 39 + 10.4. Fourth sub-section within new section to be added to + [RFC5661] to be entitled + "Obtaining Access to Sessions and State after Migration" 41 + 10.5. Fifth sub-section within new section to be added to + [RFC5661] to be entitled + "Obtaining Access to Sessions and State after Network + Address Transfer" . . . . . . . . . . . . . . . . . . . 43 + 11. New section to be added third after Section 11.11 of + + [RFC5661] to be entitled + "Server Responsibilities Upon Migration" . . . . . . . . . . 43 + 11.1. First sub-section within new section to be added to + [RFC5661] to be entitled + "Server Responsibilities in Effecting State Reclaim + after Migration" . . . . . . . . . . . . . . . . . . . . 44 + 11.2. Second sub-section within new section to be added to + [RFC5661] to be entitled + "Server Responsibilities in Effecting Transparent State + Migration" . . . . . . . . . . . . . . . . . . . . . . . 45 + 11.3. Third sub-section within new section to be added to + [RFC5661] to be entitled + "Server Responsibilities in Effecting Session Transfer" 46 + 12. fs_locations_info . . . . . . . . . . . . . . . . . . . . . . 49 + 12.1. Updates to treatment of fs_locations_info . . . . . . . 49 + 12.2. Updated Section 11.10 of [RFC5661] entitled + "The Attribute fs_locations_info" . . . . . . . . . . . 49 + 12.2.1. Updated section 11.10.1 of [RFC5661] entitled + "The fs_locations_server4 Structure" . . . . . . . . 53 + 12.2.2. Updated Section 11.10.2 of [RFC5661] entitled + "The fs_locations_info4 Structure" . . . . . . . . . 60 + 12.2.3. Updated Section 11.10.3 of [RFC5661] entitled + "The fs_locations_item4 Structure" . . . . . . . . . 61 + 13. Changes to [RFC5661] outside Section 11 . . . . . . . . . . . 63 + 13.1. Updated section 1.7.3.3 of [RFC5661] to be retitled + "Introduction to Multi-Server Namespace" . . . . . . . . 64 + 13.2. Updated Section 2.10.4 of [RFC5661] entitled + "Server Scope" . . . . . . . . . . . . . . . . . . . . . 65 + 13.3. Revised Treatment of NFS4ERR_MOVED . . . . . . . . . . . 66 + 13.4. Revised Discussion of Server_owner changes . . . . . . . 67 + 13.5. Revision to Treatment of EXCHANGE_ID . . . . . . . . . . 68 + 13.6. Revision to Treatment of RECLAIM_COMPLETE . . . . . . . 69 + 13.7. Updated Section 15.1.9 of [RFC5661] entitled + "Reclaim Errors" . . . . . . . . . . . . . . . . . . . . 70 + 13.7.1. Updated Section 15.1.9.1 of [RFC5661] entitled + "NFS4ERR_COMPLETE_ALREADY (Error Code 10054)" . . . 70 + 13.7.2. Updated Section 15.1.9.2 of [RFC5661] entitled + "NFS4ERR_GRACE (Error Code 10013)" . . . . . . . . . 70 + 13.7.3. Updated Section 15.1.9.3 of [RFC5661] entitled + "NFS4ERR_NO_GRACE (Error Code 10033)" . . . . . . . 70 + 13.7.4. Updated Section 15.1.9.4 of [RFC5661] entitled + "NFS4ERR_RECLAIM_BAD (Error Code 10034)" . . . . . . 70 + 13.7.5. Updated Section 15.1.9.5 of [RFC5661] entitled + "NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)" . . . 71 + 14. Updated Section 18.35 of [RFC5661] entitled + "Operation 42: EXCHANGE_ID - Instantiate Client ID" . . . . . 71 + 15. Updated Section 18.51 of [RFC5661] entitled + "Operation 58: RECLAIM_COMPLETE - Indicates Reclaims + Finished" . . . . . . . . . . . . . . . . . . . . . . . . . . 89 + 16. Security Considerations . . . . . . . . . . . . . . . . . . . 93 + 17. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 95 + 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 95 + 18.1. Normative References . . . . . . . . . . . . . . . . . . 95 + 18.2. Informative References . . . . . . . . . . . . . . . . . 96 + Appendix A. Classification of Document Sections . . . . . . . . 97 + Appendix B. Updates to [RFC5661] . . . . . . . . . . . . . . . . 98 + Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 102 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 102 1. Introduction This document defines the proper handling, within NFSv4.1, of the - location-related attributes fs_locations and fs_locations_info and - how necessary changes in those attributes are to be dealt with. The - necessary corrections and clarifications parallel those done for - NFSv4.0 in [RFC7931] and [I-D.cel-nfsv4-mv0-trunking-update]. + attributes related to file system location fs_locations and + fs_locations_info and how necessary changes in those attributes are + to be dealt with. The necessary corrections and clarifications + parallel those done for NFSv4.0 in [RFC7931] and + [I-D.cel-nfsv4-mv0-trunking-update]. A large part of the changes to be made are necessary to clarify the handling of Transparent State Migration in NFSv4.1, which was omitted in [RFC5661]. Many of the issues dealt with in [RFC7931] need to be addressed in the context of NFSv4.1. Another important issue to be dealt with concerns the handling of - multiple entries within location-related attributes that represent - different ways to access the same file system. Unfortunately - [RFC5661], while recognizing that these entries can represent - different ways to access the same file system, confuses the matter by - treating network access paths as "replicas", making it difficult for - these attributes to be used to obtain information about the network - addresses to be used to access particular file system instances and - engendering confusion between two different sorts of transition: - those involving a change of network access paths to the same file - system instance and those in which there is a shift between two - distinct replicas. + multiple entries within attributes related to file system locations + that represent different ways to access the same file system. + Unfortunately [RFC5661], while recognizing that these entries can + represent different ways to access the same file system, confuses the + matter by treating network access paths as "replicas", making it + difficult for these attributes to be used to obtain information about + the network addresses to be used to access particular file system + instances and engendering confusion between two different sorts of + transition: those involving a change of network access paths to the + same file system instance and those in which there is a shift between + two distinct replicas. - When location information is used to determine the set of network - addresses to access a particular file system instance (i.e. to - perform trunking discovery), clarification is needed regarding the + When file system location information is used to determine the set of + network addresses to access a particular file system instance (i.e. + to perform trunking discovery), clarification is needed regarding the interaction of trunking and transitions between file system replicas, including migration. Unfortunately [RFC5661], while it provided a method of determining whether two network addresses were connected to the same server, did not address the issue of trunking discovery, making it necessary to address it in this document. 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this @@ -208,22 +268,22 @@ 4.1. o Trunking detection refers to ways of deciding whether two specific network addresses are connected to the same NFSv4 server. The means available to make this determination depends on the protocol version, and, in some cases, on the client implementation. In the case of NFS version 4.1 and later minor versions, the means of trunking detection are as described by [RFC5661] and are available to every client. Two network addresses connected to the - same server are always server-trunkable but are not necessarily - session-trunkable. + same server are always server-trunkable but cannot necessarily be + used together to access a single session. o Trunking discovery is a process by which a client using one network address can obtain other addresses that are connected to the same server. Typically it builds on a trunking detection facility by providing one or more methods by which candidate addresses are made available to the client who can then use trunking detection to appropriately filter them. Despite the support for trunking detection there was no description of trunking discovery provided in [RFC5661]. @@ -250,41 +310,44 @@ over-RDMA as described in [RFC8166]. o The combination of a server network address and a particular connection type to be used by a connection is referred to as a "server endpoint". Although using different connection types may result in different ports being used, the use of different ports by multiple connections to the same network address is not the essence of the distinction between the two endpoints used. o Two network addresses connected to the same server are said to be - server-trunkable. + server-trunkable. Two such addresses support the use of clientid + ID trunking, as described in [RFC5661] o Two network addresses connected to the same server such that those addresses can be used to support a single common session are referred to as session-trunkable. Note that two addresses may be server-trunkable without being session-trunkable and that when two connections of different connection types are made to the same - network address and are based on a single-location entry they are - always session-trunkable, independent of the connection type, as - specified by [RFC5661], since their derivation from the same - location entry assures that both connections are to the same - server. + network address and are based on a single file system location + entry they are always session-trunkable, independent of the + connection type, as specified by [RFC5661], since their derivation + from the same file system location entry together with the + identity of their network addresses assures that both connections + are to the same server and will return server-owner information + allowing session trunking to be used. Discussion of the term "replica" is complicated for a number of reasons: o Even though the term is used in explaining the issues in [RFC5661] that need to be addressed in this document, a full explanation of this term requires explanation of related terms connected to the - location attributes which are provided in Section 4.2 of the - current document. + file system location attributes which are provided in Section 4.2 + of the current document. o The term is also used in [RFC5661], with a meaning different from that in the current document. In short, in [RFC5661] each replica is a identified by a single network access path while, in the current document a set of network access paths which have server- trunkable network addresses and the same root-relative file system pathname are considered to be a single replica with multiple network access paths. 3.2. Summary of Issues @@ -304,21 +367,22 @@ o [RFC5661], while it dealt with situations in which various forms of clustering allowed co-ordination of the state assigned by co- operating servers to be used, made no provisions for Transparent State Migration, as introduced by [RFC7530] and corrected and clarified by [RFC7931]. o Although NFSv4.1 was defined with a clear definition of how trunking detection was to be done, there was no clear specification of how trunking discovery was to be done, despite the fact that the specification clearly indicated that this - information could be made available via the location attributes. + information could be made available via the file system location + attributes. o Because the existence of multiple network access paths to the same file system was dealt with as if there were multiple replicas, issues relating to transitions between replicas could never be clearly distinguished from trunking-related transitions between the addresses used to access a particular file system instance. As a result, in situations in which both migration and trunking configuration changes were involved, neither of these could be clearly dealt with and the relationship between these two features was not seriously addressed. @@ -385,21 +449,21 @@ the existing treatment of client id confirmation does not make sense in the context of transparent state migration, in which client ids are transferred between source and destination servers. o A new treatment of RECLAIM_COMPLETE is needed, replacing that which appeared in Section 18.51 of [RFC5661]. This is necessary to clarify the function of the one-fs flag and clarify how existing clients, that might not properly use this flag, are to be dealt with. -3.3. Relationship of this Document to RFC5661 +3.3. Relationship of this Document to [RFC5661] The role of this document is to explain and specify a set of needed changes to [RFC5661]. All of these changes are related to the multi- server namespace features of NFSv4.1. This document contains sections that propose additions to and other modifications of [RFC5661] as well as others that explain the reasons for modifications but do not directly affect existing specifications. In consequence, the sections of this document can be divided into @@ -409,189 +473,199 @@ time as a consolidated specification is produced. o Explanatory sections do not contain any material that is meant to update the specification of NFSv4.1. Such sections may contain explanations about why and how changes are to be done, without including any text that is to update [RFC5661] or appear in an eventual consolidated document, o Replacement sections contain text that is to replace and thus supersede text within [RFC5661] and then appear in an eventual - consolidated document. Replacement sections have the phrase "(as - updated)" appended to the section title. + consolidated document. The titles of replacement sections + indicate the section(s) within [RFC5661] that is to be replaced. o Additional sections contain text which, although not replacing anything in [RFC5661], will be part of the specification of NFSv4.1 and will be expected to be part of an eventual - consolidated document. Additional sections have the phrase "(to - be added)" appended to the section title. + consolidated document. The titles of additional sections indicate + where, within [RFC5661], the new section would appear. o Editing sections contain some text that replaces text within [RFC5661], although the entire section will not consist of such text and will include other text as well. Such sections make relatively minor adjustments in the existing NFSv4.1 specification which are expected to reflected in an eventual consolidated document. Generally such replacement text appears as a quotation, which may take the form of an indented set of paragraphs. See Appendix A for a classification of the sections of this document according to the categories above. When this document is approved and published, [RFC5661] would be significantly updated with most of the changed sections within the current Section 11 of that document. A detailed discussion of the necessary updates can be found in Appendix B. -4. Changes to Section 11 of RFC5661 +4. Changes to Section 11 of [RFC5661] A number of sections need to be revised, replacing existing sub- sections within section 11 of [RFC5661]: o New introductory material, including a terminology section, replaces the existing material in [RFC5661] ranging from the start of the existing Section 11 up to and including the existing Section 11.1. The new material appears in Sections 4.1 through 4.3 below. o A significant reorganization of the material in the existing Sections 11.4 and 11.5 (of [RFC5661]) is necessary. The reasons for the reorganization of these sections into a single section with multiple subsections are discussed in Section 4.4 below. This replacement appears as Section 4.5 below. - New material relating to the handling of the location attributes - is contained in Sections 4.5.1 and 4.5.7 below. + New material relating to the handling of the file system location + attributes is contained in Sections 4.5.1 and 4.5.7 below. o A major replacement for the existing Section 11.7 of [RFC5661] entitled "Effecting File System Transitions", will appear as Sections 6 through 11 of the current document. The reasons for the reorganization of this section into multiple sections are discussed below in Section 5 of the current document. o A replacement for the existing Section 11.10 of [RFC5661] entitled "The Attribute fs_locations_info", will appear as Section 12.2 of the current document, with Section 12.1 describing the differences between the new section and the treatment within [RFC5661]. A revised treatment is necessary because the existing treatment did not make clear how the added attribute information relates to the case of trunked paths to the same replica. These issues were not addressed in [RFC5661] where the concepts of a replica and a network path used to access a replica were not clearly distinguished. -4.1. Multi-Server Namespace (as updated) +4.1. Updated introductory material for Section 11 of [RFC5661] entitled + "Multi-Server Namespace" NFSv4.1 supports attributes that allow a namespace to extend beyond the boundaries of a single server. It is desirable that clients and servers support construction of such multi-server namespaces. Use of such multi-server namespaces is OPTIONAL however, and for many purposes, single-server namespaces are perfectly acceptable. Use of multi-server namespaces can provide many advantages, by separating a file system's logical position in a namespace from the (possibly changing) logistical and administrative considerations that result in particular file systems being located on particular servers. -4.2. Location-related Terminology (to be added) +4.2. New section to be added as the first sub-section of Section 11 of + [RFC5661] to be entitled "Terminology Related to File System + Location" Regarding terminology relating to the construction of multi-server namespaces out of a set of local per-server namespaces: o Each server has a set of exported file systems which may accessed by NFSv4 clients. Typically, this is done by assigning each file system a name within the pseudo-fs associated with the server, although the pseudo-fs may be dispensed with if there is only a single exported file system. Each such file system is part of the server's local namespace, and can be considered as a file system instance within a larger multi-server namespace. o The set of all exported file systems for a given server constitutes that server's local namespace. o In some cases, a server will have a namespace more extensive than its local namespace, by using features associated with attributes - that provide location information. These features, which allow - construction of a multi-server namespace are all described in - individual sections below and include referrals (described in - Section 4.5.6), migration (described in Section 4.5.5), and - replication (described in Section 4.5.4). + that provide file system location information. These features, + which allow construction of a multi-server namespace are all + described in individual sections below and include referrals + (described in Section 4.5.6), migration (described in + Section 4.5.5), and replication (described in Section 4.5.4). o A file system present in a server's pseudo-fs may have multiple file system instances on different servers associated with it. All such instances are considered replicas of one another. o When a file system is present in a server's pseudo-fs, but there is no corresponding local file system, it is said to be "absent". - In such cases, all associated instances will be accessed on other servers. Regarding terminology relating to attributes used in trunking discovery and other multi-server namespace features: - o Location attributes include the fs_locations and fs_locations_info - attributes. + o File system location attributes include the fs_locations and + fs_locations_info attributes. - o Location entries are the individual file system locations in the - location attributes. Each such entry specifies a server, in the - form of a host name or IP address, and an fs name, which - designates the location of the file system within the server's - pseudo-fs. A location entry designates a set of server endpoints - to which the client may establish connections. There may be - multiple endpoints because a host name may map to multiple network - addresses and because multiple connection types may be used to - communicate with a single network address. However, all such - endpoints MUST provide a way of connecting to a single server. - The exact form of the location entry varies with the particular - location attribute used, as described in Section 4.3. + o File system location entries provide the individual file system + locations within the file system location attributes. Each such + entry specifies a server, in the form of a host name or IP + address, and an fs name, which designates the location of the file + system within the server's pseudo-fs. A file system location + entry designates a set of server endpoints to which the client may + establish connections. There may be multiple endpoints because a + host name may map to multiple network addresses and because + multiple connection types may be used to communicate with a single + network address. However, all such endpoints MUST provide a way + of connecting to a single server. The exact form of the location + entry varies with the particular file system location attribute + used, as described in Section 4.3. - o Location elements are derived from location entries and each - describes a particular network access path, consisting of a - network address and a location within the server's pseudo-fs. - Location elements need not appear within a location attribute, but - the existence of each location element derives from a - corresponding location entry. When a location entry specifies an - IP address there is only a single corresponding location element. - Location entries that contain a host name, are resolved using DNS, - and may result in one or more location elements. All location - elements consist of a location address which is the IP address of - an interface to a server and an fs name which is the location of - the file system within the server's pseudo-fs. The fs name is - empty if the server has no pseudo-fs and only a single exported - file system at the root filehandle. + o File system location elements are derived from location entries + and each describes a particular network access path, consisting of + a network address and a location within the server's pseudo-fs. + Such location elements need not appear within a file system + location attribute, but the existence of each location element + derives from a corresponding location entry. When a location + entry specifies an IP address there is only a single corresponding + location element. File system location entries that contain a + host name, are resolved using DNS, and may result in one or more + location elements. All location elements consist of a location + address which is the IP address of an interface to a server and an + fs name which is the location of the file system within the + server's pseudo-fs. The fs name is empty if the server has no + pseudo-fs and only a single exported file system at the root + filehandle. - o Two location elements are said to be server-trunkable if they - specify the same fs name and the location addresses are such that - the location addresses are server-trunkable. + o Two file system location elements are said to be server-trunkable + if they specify the same fs name and the location addresses are + such that the location addresses are server-trunkable. When the + corresponding network paths are used, the client will always be + able to use client ID trunking, but will only be able to use + session trunking if the paths are also session-trunkable. - o Two location elements are said to be session-trunkable if they - specify the same fs name and the location addresses are such that - the location addresses are session-trunkable. + o Two file system location elements are said to be session-trunkable + if they specify the same fs name and the location addresses are + such that the location addresses are session-trunkable. When the + corresponding network paths are used, the client will be able to + able to use either client ID trunking or session trunking. Each set of server-trunkable location elements defines a set of available network access paths to a particular file system. When there are multiple such file systems, each of which contains the same data, these file systems are considered replicas of one another. Logically, such replication is symmetric, since the fs currently in use and an alternate fs are replicas of each other. Often, in other documents, the term "replica" is not applied to the fs currently in use, despite the fact that the replication relation is inherently symmetric. -4.3. Location Attributes (as updated) +4.3. Updated Section 11.1 of [RFC5661] to be retitled "File System + Location Attributes" NFSv4.1 contains RECOMMENDED attributes that provide information about how (i.e. at what network address and namespace position) a given file system may be accessed. As a result, file systems in the namespace of one server can be associated with one or more instances - of that file system on other servers. These attributes contain - location entries specifying a server address target (either as a DNS - name representing one or more IP addresses or as a specific IP + of that file system on other servers. These attributes contain file + system location entries specifying a server address target (either as + a DNS name representing one or more IP addresses or as a specific IP address) together with the pathname of that file system within the associated single-server namespace. The fs_locations_info RECOMMENDED attribute allows specification of one or more file system instance locations where the data corresponding to a given file system may be found. This attribute provides to the client, in addition to specification of file system instance locations, other helpful information such as: o Information guiding choices among the various file system @@ -599,58 +673,58 @@ etc.). o Information to help the client efficiently effect as seamless a transition as possible among multiple file system instances, when and if that should be necessary. o Information helping to guide the selection of the appropriate connection type to be used when establishing a connection. Within the fs_locations_info attribute, each fs_locations_server4 - entry corresponds to a location entry with the fls_server field - designating the server, with the location pathname within the + entry corresponds to a file system location entry with the fls_server + field designating the server, with the location pathname within the server's pseudo-fs given by the fl_rootpath field of the encompassing fs_locations_item4. The fs_locations attribute defined in NFSv4.0 is also a part of NFSv4.1. This attribute only allows specification of the file system locations where the data corresponding to a given file system may be found. Servers should make this attribute available whenever fs_locations_info is supported, but client use of fs_locations_info is preferable, as it provides more information. - Within the fs_location attribute, each fs_location4 contains a - location entry with the server field designating the server and the - rootpath field giving the location pathname within the server's - pseudo-fs. + Within the fs_location attribute, each fs_location4 contains a file + system location entry with the server field designating the server + and the rootpath field giving the location pathname within the + server's pseudo-fs. -4.4. Re-organization of Sections 11.4 and 11.5 of RFC5661 +4.4. Re-organization of Sections 11.4 and 11.5 of [RFC5661] Previously, issues related to the fact that multiple location entries directed the client to the same file system instance were dealt with in a separate Section 11.5 of [RFC5661]. Because of the new treatment of trunking, these issues now belong within Section 4.5 below. In this new section of the current document, trunking is dealt with - in Section 4.5.2 together with the other uses of location information - described in Sections 4.5.4, 4.5.5, and 4.5.6. + in Section 4.5.2 together with the other uses of file system location + information described in Sections 4.5.4, 4.5.5, and 4.5.6. -4.5. Uses of Location Information (as updated) +4.5. Updated Section 11.4 of [RFC5661] to be retitled "Uses of File + System Location Information" - The location attributes (i.e. fs_locations and fs_locations_info), - together with the possibility of absent file systems, provide a - number of important facilities in providing reliable, manageable, and - scalable data access. + The file system location attributes (i.e. fs_locations and + fs_locations_info), together with the possibility of absent file + systems, provide a number of important facilities in providing + reliable, manageable, and scalable data access. When a file system is present, these attributes can provide - o The locations of alternative replicas, to be used to access the same data in the event of server failures, communications problems, or other difficulties that make continued access to the current replica impossible or otherwise impractical. Provision and use of such alternate replicas is referred to as "replication" and is discussed in Section 4.5.4 below. o The network address(es) to be used to access the current file system instance or replicas of it. Client use of this information is discussed in Section 4.5.2 below. @@ -670,158 +744,168 @@ is discussed in Section 4.5.4 below. Where a file system was previously absent, specification of file system location provides a means by which file systems located on one server can be associated with a namespace defined by another server, thus allowing a general multi-server namespace facility. A designation of such a remote instance, in place of a file system never previously present , is called a "pure referral" and is discussed in Section 4.5.6 below. - Because client support for location-related attributes is OPTIONAL, a - server may (but is not required to) take action to hide migration and - referral events from such clients, by acting as a proxy, for example. - The server can determine the presence of client support from the - arguments of the EXCHANGE_ID operation (see Section 14.3 in the - current document). + Because client support for attributes related to file system location + is OPTIONAL, a server may (but is not required to) take action to + hide migration and referral events from such clients, by acting as a + proxy, for example. The server can determine the presence of client + support from the arguments of the EXCHANGE_ID operation (see + Section 14.3 in the current document). -4.5.1. Combining Multiple Uses in a Single Attribute (to be added) +4.5.1. New section to be added as the first sub-section of Section 11.4 + of [RFC5661] to be entitled "Combining Multiple Uses in a Single + Attribute" - A location attribute will sometimes contain information relating to - the location of multiple replicas which may be used in different - ways. + A file system location attribute will sometimes contain information + relating to the location of multiple replicas which may be used in + different ways. - o Location entries that relate to the file system instance currently - in use provide trunking information, allowing the client to find - additional network addresses by which the instance may be - accessed. + o File system location entries that relate to the file system + instance currently in use provide trunking information, allowing + the client to find additional network addresses by which the + instance may be accessed. - o Location entries that provide information about replicas to which - access is to be transferred. + o File system location entries that provide information about + replicas to which access is to be transferred. - o Other location entries that relate to replicas that are available - to use in the event that access to the current replica becomes - unsatisfactory. + o Other file system location entries that relate to replicas that + are available to use in the event that access to the current + replica becomes unsatisfactory. In order to simplify client handling and allow the best choice of replicas to access, the server should adhere to the following guidelines. - o All location entries that relate to a single file system instance - should be adjacent. + o All file system location entries that relate to a single file + system instance should be adjacent. - o Location entries that relate to the instance currently in use - should appear first. + o File system location entries that relate to the instance currently + in use should appear first. - o Location entries that relate to replica(s) to which migration is - occurring should appear before replicas which are available for - later use if the current replica should become inaccessible. + o File system location entries that relate to replica(s) to which + migration is occurring should appear before replicas which are + available for later use if the current replica should become + inaccessible. -4.5.2. Location Attributes and Trunking (to be added) +4.5.2. New section to be added as the second sub-section of + Section 11.4 of [RFC5661] to be entitled "File System Location + Attributes and Trunking" Trunking is the use of multiple connections between a client and server in order to increase the speed of data transfer. A client may determine the set of network addresses to use to access a given file system in a number of ways: o When the name of the server is known to the client, it may use DNS to obtain a set of network addresses to use in accessing the server. - o It may fetch the location attribute for the filesystem which will - provide either the name of the server (which can be turned into a - set of network addresses using DNS), or it will find a set of - server-trunkable location entries which can provide the addresses - specified by the server as desirable to use to access the file - system in question. + o It may fetch the file system location attribute for the filesystem + which will provide either the name of the server (which can be + turned into a set of network addresses using DNS), or it will find + a set of server-trunkable location entries which can provide the + addresses specified by the server as desirable to use to access + the file system in question. The server can provide location entries that include either names or network addresses. It might use the latter form because of DNS- related security concerns or because the set of addresses to be used might require active management by the server. Locations entries used to discover candidate addresses for use in trunking are subject to change, as discussed in Section 4.5.7 below. The client may respond to such changes by using additional addresses once they are verified or by ceasing to use existing ones. The server can force the client to cease using an address by returning NFS4ERR_MOVED when that address is used to access a file system. This allows a transfer of client access which is similar to migration, although the same file system instance is accessed throughout. -4.5.3. Location Attributes and Connection Type Selection (to be added) +4.5.3. New section to be added as the third sub-section of Section 11.4 + of [RFC5661] to be entitled "File System Location Attributes and + Connection Type Selection" Because of the need to support multiple connections, clients face the issue of determining the proper connection type to use when establishing a connection to a given server network address. In some cases, this issue can be addressed through the use of the connection "step-up" facility described in Section 18.16 of [RFC5661]. However, because there are cases is which that facility is not available, the client may have to choose a connection type with no possibility of changing it within the scope of a single connection. - The two location attributes differ as to the information made - available in this regard. Fs_locations provides no information to - support connection type selection. As a result, clients supporting - multiple connection types would need to attempt to establish - connections using multiple connection types until the one preferred - by the client is successfully established. + The two file system location attributes differ as to the information + made available in this regard. Fs_locations provides no information + to support connection type selection. As a result, clients + supporting multiple connection types would need to attempt to + establish connections using multiple connection types until the one + preferred by the client is successfully established. Fs_locations_info provides a flag, FSLI4TF_RDMA flag. indicating that RPC-over-RDMA support is available using the specified location entry. This flag makes it for a convenient for a client wishing to use RDMA, to establish a TCP connection and then convert to use of RDMA. After establishing a TCP connection, the step-up facility, can be used, if available, to convert that connection to RDMA mode. Otherwise, if RDMA availability is indicated, a new RDMA connection can be established and it can be bound to the session already established by the TCP connection, allowing the TCP connection to be dropped and the session converted to further use in RDMA node. -4.5.4. File System Replication (as updated) +4.5.4. Updated Section 11.4.1 of [RFC5661] entitled "File System + Replication" The fs_locations and fs_locations_info attributes provide alternative - locations, to be used to access data in place of or in addition to - the current file system instance. On first access to a file system, - the client should obtain the set of alternate locations by - interrogating the fs_locations or fs_locations_info attribute, with - the latter being preferred. + file system locations, to be used to access data in place of or in + addition to the current file system instance. On first access to a + file system, the client should obtain the set of alternate locations + by interrogating the fs_locations or fs_locations_info attribute, + with the latter being preferred. In the event that server failures, communications problems, or other difficulties make continued access to the current file system impossible or otherwise impractical, the client can use the alternate locations as a way to get continued access to its data. The alternate locations may be physical replicas of the (typically read-only) file system data, or they may provide for the use of various forms of server clustering in which multiple servers provide alternate ways of accessing the same physical file system. How these different modes of file system transition are represented within the fs_locations and fs_locations_info attributes and how the client deals with file system transition issues will be discussed in detail below. -4.5.5. File System Migration (as updated) +4.5.5. Updated Section 11.4.2 of [RFC5661] entitled "File System + Migration" When a file system is present and becomes absent, clients can be given the opportunity to have continued access to their data, at an - alternate location, as specified by a location attribute. This - migration of access to another replica includes the ability to retain - locks across the transition, either by using lock reclaim or by - taking advantage of Transparent State Migration. + alternate location, as specified by a file system location attribute. + This migration of access to another replica includes the ability to + retain locks across the transition, either by using lock reclaim or + by taking advantage of Transparent State Migration. Typically, a client will be accessing the file system in question, - get an NFS4ERR_MOVED error, and then use a location attribute to - determine the new location of the data. When fs_locations_info is - used, additional information will be available that will define the - nature of the client's handling of the transition to a new server. + get an NFS4ERR_MOVED error, and then use a file system location + attribute to determine the new location of the data. When + fs_locations_info is used, additional information will be available + that will define the nature of the client's handling of the + transition to a new server. Such migration can be helpful in providing load balancing or general resource reallocation. The protocol does not specify how the file system will be moved between servers. It is anticipated that a number of different server-to-server transfer mechanisms might be used with the choice left to the server implementer. The NFSv4.1 protocol specifies the method used to communicate the migration event between client and server. The new location may be, in the case of various forms of server @@ -846,52 +930,52 @@ degree indicated by the fs_locations_info attribute). Where file systems are writable, a change made on the original file system must be visible on all migration targets. Where a file system is not writable but represents a read-only copy (possibly periodically updated) of a writable file system, similar requirements apply to the propagation of updates. Any change visible in the original file system must already be effected on all migration targets, to avoid any possibility that a client, in effecting a transition to the migration target, will see any reversion in file system state. -4.5.6. Referrals (as updated) +4.5.6. Updated Section 11.4.3 of [RFC5661] entitled "Referrals" Referrals allow the server to associate a file system namespace entry located on one server with a file system located on another server. When this includes the use of pure referrals, servers are provided a way of placing a file system in a location within the namespace essentially without respect to its physical location on a particular server. This allows a single server or a set of servers to present a multi-server namespace that encompasses file systems located on a wider range of servers. Some likely uses of this facility include establishment of site-wide or organization-wide namespaces, with the eventual possibility of combining such together into a truly global namespace. Referrals occur when a client determines, upon first referencing a position in the current namespace, that it is part of a new file system and that the file system is absent. When this occurs, typically upon receiving the error NFS4ERR_MOVED, the actual location or locations of the file system can be determined by fetching the a locations attribute. attribute. - The locations attribute may designate a single file system location - or multiple file system locations, to be selected based on the needs - of the client. The server, in the fs_locations_info attribute, may - specify priorities to be associated with various file system location - choices. The server may assign different priorities to different - locations as reported to individual clients, in order to adapt to - client physical location or to effect load balancing. When both - read-only and read-write file systems are present, some of the read- - only locations might not be absolutely up-to-date (as they would have - to be in the case of replication and migration). Servers may also - specify file system locations that include client-substituted + The file system location attribute may designate a single file system + location or multiple file system locations, to be selected based on + the needs of the client. The server, in the fs_locations_info + attribute, may specify priorities to be associated with various file + system location choices. The server may assign different priorities + to different locations as reported to individual clients, in order to + adapt to client physical location or to effect load balancing. When + both read-only and read-write file systems are present, some of the + read-only locations might not be absolutely up-to-date (as they would + have to be in the case of replication and migration). Servers may + also specify file system locations that include client-substituted variables so that different clients are referred to different file systems (with different data contents) based on client attributes such as CPU architecture. When the fs_locations_info attribute is such that that there are multiple possible targets listed, the relationships among them may be important to the client in selecting which one to use. The same rules specified in Section 4.5.5 below regarding multiple migration targets apply to these multiple replicas as well. For example, the client might prefer a writable target on a server that has additional @@ -914,26 +998,27 @@ namespace. The top-level referral file system or any segment may use replicated referral file systems for higher availability. Generally, multi-server namespaces are for the most part uniform, in that the same data made available to one client at a given location in the namespace is made available to all clients at that location. However, as described above, there are facilities provided that allow different clients to be directed different sets of data, to enable adaptation to such client characteristics as CPU architecture. -4.5.7. Changes in a Location Attribute (to be added) +4.5.7. New section to be added after Section 11.4.3 of [RFC5661] to be + entitled "Changes in a File System Location Attribute" - Although clients will typically fetch a location attribute when first - accessing a file system and when NFS4ERR_MOVED is returned, a client - can choose to fetch the attribute periodically, in which case the - value fetched may change over time. + Although clients will typically fetch a file system location + attribute when first accessing a file system and when NFS4ERR_MOVED + is returned, a client can choose to fetch the attribute periodically, + in which case the value fetched may change over time. For clients not prepared to access multiple replicas simultaneously (see Section 8.1 of the current document), the handling of the various cases of change is as follows: o Changes in the list of replicas or in the network addresses associated with replicas do not require immediate action. The client will typically update its list of replicas to reflect the new information. @@ -959,21 +1044,21 @@ with one currently in use, the client is free to use the new replica immediately. o When a replica currently in use is deleted from the list, the client need not cease using it immediately. However, since the server may subsequently force such use to cease (by returning NFS4ERR_MOVED), clients might decide to limit the need for later state transfer. For example, new opens might be done on other replicas, rather than on one not present in the list. -5. Re-organization of Section 11.7 of RFC5661 +5. Re-organization of Section 11.7 of [RFC5661] The material in Section 11.7 of [RFC5661] has been reorganized and augmented as specified below: o Because there can be a shift of the network access paths used to access a file system instance without any shift between replicas, a new Section 6 in the current document distinguishes between those cases in which there is a shift between distinct replicas and those involving a shift in network access paths with no shift between replicas. @@ -991,59 +1076,61 @@ o The additional Section 10 in the current document discusses the client's response to access transitions and how it determines whether migration has occurred, and how it gets access to any transferred locking and session state. o The additional Section 11 in the current document discusses the responsibilities of the source and destination servers when transferring locking and session state. -6. Overview of File Access Transitions (to be added) +6. New section to be added after Section 11.6 of [RFC5661] to be + entitled "Overview of File Access Transitions" File access transitions are of two types: o Those that involve a transition from accessing the current replica to another one in connection with either replication or migration. How these are dealt with is discussed in Section 8 of the current document. o Those in which access to the current file system instance is retained, while the network path used to access that instance is changed. This case is discussed in Section 7 of the current document. -7. Effecting Network Endpoint Transitions (to be added) +7. New section to be added second after Section 11.6 of [RFC5661] to be + entitled "Effecting Network Endpoint Transitions" The endpoints used to access a particular file system instance may change in a number of ways, as listed below. In each of these cases, the same filehandles, stateids, client IDs and session are used to continue access, with a continuity of lock state. o When use of a particular address is to cease and there is also one currently in use which is server-trunkable with it, requests that would have been issued on the address whose use is to be discontinued can be issued on the remaining address(es). When an address is not a session-trunkable one, the request might need to be modified to reflect the fact that a different session will be used. o When use of a particular connection is to cease, as indicated by receiving NFS4ERR_MOVED when using that connection but that address is still indicated as accessible according to the - appropriate location entries, it is likely that requests can be - issued on a new connection of a different connection type, once - that connection is established. Since any two server endpoints - that share a network address are inherently session-trunkable, the - client can use BIND_CONN_TO_SESSION to access the existing session - using the new connection and proceed to access the file system - using the new connection. + appropriate file system location entries, it is likely that + requests can be issued on a new connection of a different + connection type, once that connection is established. Since any + two server endpoints that share a network address are inherently + session-trunkable, the client can use BIND_CONN_TO_SESSION to + access the existing session using the new connection and proceed + to access the file system using the new connection. o When there are no potential replacement addresses in use but there are valid addresses session-trunkable with the one whose use is to be discontinued, the client can use BIND_CONN_TO_SESSION to access the existing session using the new address. Although the target session will generally be accessible, there may be cases in which that session in no longer accessible, in which case a new session can be created to provide the client continued access to the existing instance. @@ -1052,21 +1139,22 @@ to be discontinued, other server-trunkable addresses may be used to provide continued access. Although use of CREATE_SESSION is available to provide continued access to the existing instance, servers have the option of providing continued access to the existing session through the new network access path in a fashion similar to that provided by session migration (see Section 9 of the current document). To take advantage of this possibility, clients can perform an initial BIND_CONN_TO_SESSION, as in the previous case, and use CREATE_SESSION only if that fails. -8. Effecting File System Transitions (as updated) +8. Updated Section 11.7 of [RFC5661] entitled "Effecting File System + Transitions" There are a range of situations in which there is a change to be effected in the set of replicas used to access a particular file system. Some of these may involve an expansion or contraction of the set of replicas used as discussed in Section 8.1 below. For reasons explained in that section, most transitions will involve a transition from a single replica to a corresponding replacement replica. When effecting replica transition, some types of sharing between the replicas may affect handling of the transition as @@ -1091,21 +1179,22 @@ o The client handling of transitions, including determining how to deal with the various means that the server might take to supply effective continuity of locking state are discussed in Section 10 of the current document. o The servers' (source and destination) responsibilities in effecting Transparent Migration of locking and session state are discussed in Section 11 of the current document. -8.1. File System Transitions and Simultaneous Access (as updated) +8.1. Updated Section 11.7.1 of [RFC5661] entitled "File System + Transitions and Simultaneous Access" The fs_locations_info attribute (described in Section 11.10.1 of [RFC5661] and Section 12.2 of this document) may indicate that two replicas may be used simultaneously (see Section 11.7.2.1 of [RFC5661] for details). Although situations in which multiple replicas may be accessed simultaneously are somewhat similar to those in which a single replica is accessed by multiple network addresses, there are important differences, since locking state is not shared among multiple replicas. @@ -1124,21 +1213,22 @@ For example, if one of the replicas become unavailable, access will be transferred to a different replica, also capable of simultaneous access with the one still in use. When there is no such replica, the transition may be to the replica already in use. At this point, the client has a choice between merging the locking state for the two replicas under the aegis of the sole replica in use or treating these separately, until another replica capable of simultaneous access presents itself. -8.2. Filehandles and File System Transitions (as updated) +8.2. Updated Section 11.7.3 of [RFC5661] entitled "Filehandles and File + System Transitions" There are a number of ways in which filehandles can be handled across a file system transition. These can be divided into two broad classes depending upon whether the two file systems across which the transition happens share sufficient state to effect some sort of continuity of file system handling. When there is no such cooperation in filehandle assignment, the two file systems are reported as being in different handle classes. In this case, all filehandles are assumed to expire as part of the file @@ -1147,21 +1237,22 @@ FH4_VOL_MIGRATION bit, which only affects behavior when fs_locations_info is not available. When there is cooperation in filehandle assignment, the two file systems are reported as being in the same handle classes. In this case, persistent filehandles remain valid after the file system transition, while volatile filehandles (excluding those that are only volatile due to the FH4_VOL_MIGRATION bit) are subject to expiration on the target server. -8.3. Fileids and File System Transitions (as updated) +8.3. Updated Section 11.7.4 of [RFC5661] entitled "Fileids and File + System Transitions" In NFSv4.0, the issue of continuity of fileids in the event of a file system transition was not addressed. The general expectation had been that in situations in which the two file system instances are created by a single vendor using some sort of file system image copy, fileids would be consistent across the transition, while in the analogous multi-vendor transitions they would not. This poses difficulties, especially for the client without special knowledge of the transition mechanisms adopted by the server. Note that although fileid is not a REQUIRED attribute, many servers support fileids and @@ -1204,104 +1295,109 @@ are no reliable filehandles across a transition event (either because there is no filehandle continuity or because the filehandles are volatile), the client is in a position where it cannot verify that files it was accessing before the transition are the same objects. It is forced to assume that no object has been renamed, and, unless there are guarantees that provide this (e.g., the file system is read-only), problems for applications may occur. Therefore, use of such configurations should be limited to situations where the problems that this may cause can be tolerated. -8.4. Fsids and File System Transitions (as updated) +8.4. Updated section 11.7.5 of [RFC5661] entitled "Fsids and File + System Transitions" Since fsids are generally only unique on a per-server basis, it is likely that they will change during a file system transition. Clients should not make the fsids received from the server visible to applications since they may not be globally unique, and because they may change during a file system transition event. Applications are best served if they are isolated from such transitions to the extent possible. Although normally a single source file system will transition to a single target file system, there is a provision for splitting a single source file system into multiple target file systems, by specifying the FSLI4F_MULTI_FS flag. -8.4.1. File System Splitting (as updated) +8.4.1. Updated section 11.7.5.1 of [RFC5661] entitled "File System + Splitting" When a file system transition is made and the fs_locations_info indicates that the file system in question might be split into multiple file systems (via the FSLI4F_MULTI_FS flag), the client SHOULD do GETATTRs to determine the fsid attribute on all known objects within the file system undergoing transition to determine the new file system boundaries. Clients might choose to maintain the fsids passed to existing applications by mapping all of the fsids for the descendant file systems to the common fsid used for the original file system. Splitting a file system can be done on a transition between file systems of the same fileid class, since the fact that fileids are unique within the source file system ensure they will be unique in each of the target file systems. -8.5. The Change Attribute and File System Transitions (as updated) +8.5. Updated Section 11.7.6 of [RFC5661] entitled "The Change Attribute + and File System Transitions" Since the change attribute is defined as a server-specific one, change attributes fetched from one server are normally presumed to be invalid on another server. Such a presumption is troublesome since it would invalidate all cached change attributes, requiring refetching. Even more disruptive, the absence of any assured continuity for the change attribute means that even if the same value is retrieved on refetch, no conclusions can be drawn as to whether the object in question has changed. The identical change attribute could be merely an artifact of a modified file with a different change attribute construction algorithm, with that new algorithm just happening to result in an identical change value. When the two file systems have consistent change attribute formats, and this fact is communicated to the client by reporting in the same change class, the client may assume a continuity of change attribute construction and handle this situation just as it would be handled without any file system transition. -8.6. Write Verifiers and File System Transitions (as updated) +8.6. Updated Section 11.7.8 of [RFC5661] entitled "Write Verifiers and + File System Transitions" In a file system transition, the two file systems might be clustered in the handling of unstably written data. When this is the case, and the two file systems belong to the same write-verifier class, write verifiers returned from one system may be compared to those returned by the other and superfluous writes avoided. When two file systems belong to different write-verifier classes, any verifier generated by one must not be compared to one provided by the other. Instead, the two verifiers should be treated as not equal even when the values are identical. -8.7. Readdir Cookies and Verifiers and File System Transitions (as - updated) +8.7. Updated Section 11.7.9 of [RFC5661] entitled "Readdir Cookies and + Verifiers and File System Transitions)" In a file system transition, the two file systems might be consistent in their handling of READDIR cookies and verifiers. When this is the case, and the two file systems belong to the same readdir class, READDIR cookies and verifiers from one system may be recognized by the other and READDIR operations started on one server may be validly continued on the other, simply by presenting the cookie and verifier returned by a READDIR operation done on the first file system to the second. When two file systems belong to different readdir classes, any READDIR cookie and verifier generated by one is not valid on the second, and must not be presented to that server by the client. The client should act as if the verifier was rejected. -8.8. File System Data and File System Transitions (as updated) +8.8. Updated Section 11.7.10 of [RFC5661] entitled "File System Data + and File System Transitions" When multiple replicas exist and are used simultaneously or in succession by a client, applications using them will normally expect that they contain either the same data or data that is consistent with the normal sorts of changes that are made by other clients updating the data of the file system (with metadata being the same to the degree indicated by the fs_locations_info attribute). However, when multiple file systems are presented as replicas of one another, the precise relationship between the data of one and the data of another is not, as a general matter, specified by the NFSv4.1 @@ -1352,21 +1448,22 @@ transition to a replica, will see any reversion in file system state. The specific means of this guarantee varies based on the value of the fss_type field that is reported as part of the fs_status attribute (see Section 11.11 of [RFC5661]). Since these file systems are presumed to be unsuitable for simultaneous use, there is no specification of how locking is handled; in general, locks obtained on one file system will be separate from those on others. Since these are expected to be read-only file systems, this is not likely to pose an issue for clients or applications. -8.9. Lock State and File System Transitions (as updated) +8.9. Updated Section 11.7.7 of [RFC5661] entitled "Lock State and File + System Transitions" While accessing a file system, clients obtain locks enforced by the server which may prevent actions by other clients that are inconsistent with those locks. When access is transferred between replicas, clients need to be assured that the actions disallowed by holding these locks cannot have occurred during the transition. This can be ensured by the methods below. Unless at least one of these is implemented, clients will not be assured of continuity of lock possession across a @@ -1395,57 +1492,66 @@ State Migration is the only available means of providing the needed functionality. It should be noted that these two methods are not mutually exclusive and that a server might well provide both. In particular, if there is some circumstance preventing a specific lock from being transferred transparently, the destination server can allow it to be reclaimed, by implementing a per-fs grace period for the migrated file system. -9. Transferring State upon Migration (to be added) +9. New section to be added after Section 11.11 of [RFC5661] to be + entitled "Transferring State upon Migration" When the transition is a result of a server-initiated decision to transition access and the source and destination servers have implemented appropriate co-operation, it is possible to: o Transfer locking state from the source to the destination server, in a fashion similar to that provided by Transparent State Migration in NFSv4.0, as described in [RFC7931]. Server responsibilities are described in Section 11.2 of the current document. o Transfer session state from the source to the destination server. Server responsibilities in effecting such a transfer are described in Section 11.3 of the current document. The means by which the client determines which of these transfer events has occurred are described in Section 10 of the current document. -9.1. Transparent State Migration and pNFS (to be added) +9.1. Only sub-section within new section to be added to [RFC5661] to be + entitled "Transparent State Migration and pNFS" When pNFS is involved, the protocol is capable of supporting: o Migration of the Metadata Server (MDS), leaving the Data Servers (DS's) in place. o Migration of the file system as a whole, including the MDS and associated DS's. o Replacement of one DS by another. o Migration of a pNFS file system to one in which pNFS is not used. o Migration of a file system not using pNFS to one in which layouts are available. + Note that migration per se is only involved in the transfer of the + MDS function. Although the servicing of a layout may be transferred + from one data server to another, this not done using the file system + location attributes. The MDS can effect such transfers by recalling/ + revoking existing layouts and granting new ones on a different data + server. + Migration of the MDS function is directly supported by Transparent State Migration. Layout state will normally be transparently transferred, just as other state is. As a result, Transparent State Migration provides a framework in which, given appropriate inter-MDS data transfer, one MDS can be substituted for another. Migration of the file system function as a whole can be accomplished by recalling all layouts as part of the initial phase of the migration process. As a result, IO will be done through the MDS during the migration process, and new layouts can be granted once the @@ -1464,21 +1570,22 @@ such but can be effected by an MDS recalling layouts for the DS to be replaced and issuing new ones to be served by the successor DS. Migration may transfer a file system from a server which does not support pNFS to one which does. In order to properly adapt to this situation, clients which support pNFS, but function adequately in its absence should check for pNFS support when a file system is migrated and be prepared to use pNFS when support is available on the destination. -10. Client Responsibilities when Access is Transitioned (to be added) +10. New section to be added second after Section 11.11 of [RFC5661] to + be entitled "Client Responsibilities when Access is Transitioned" For a client to respond to an access transition, it must become aware of it. The ways in which this can happen are discussed in Section 10.1 which discusses indications that a specific file system access path has transitioned as well as situations in which additional activity is necessary to determine the set of file systems that have been migrated. Section 10.2 goes on to complete the discussion of how the set of migrated file systems might be determined. Sections 10.3 through 10.5 discuss how the client should deal with each transition it becomes aware of, either directly or as @@ -1492,56 +1599,58 @@ o "Migration recovery" to that subset of transition recovery which applies when the file system has migrated to a different replica. o "Migration discovery" refers to the process of determining which file system(s) have been migrated. It is necessary to avoid a situation in which leases could expire when a file system is not accessed for a long period of time, since a client unaware of the migration might be referencing an unmigrated file system and not renewing the lease associated with the migrated file system. -10.1. Client Transition Notifications (to be added) +10.1. First sub-section within new section to be added to [RFC5661] to + be entitled "Client Transition Notifications" When there is a change in the network access path which a client is to use to access a file system, there are a number of related status indications with which clients need to deal: o If an attempt is made to use or return a filehandle within a file system that is no longer accessible at the address previously used to access it, the error NFS4ERR_MOVED is returned. Exceptions are made to allow such file handles to be used when - interrogating a location attribute. This enables a client to - determine a new replica's location or a new network access path. + interrogating a file system location attribute. This enables a + client to determine a new replica's location or a new network + access path. This condition continues on subsequent attempts to access the file system in question. The only way the client can avoid the error is to cease accessing the filesystem in question at its old server location and access it instead using a different address at which it is now available. o Whenever a SEQUENCE operation is sent by a client to a server which generated state held on that client which is associated with a file system that is no longer accessible on the server at which it was previously available, a lease-migrated indication, in the form the SEQ4_STATUS_LEASE_MOVED status bit being set, appears in the response. This condition continues until the client acknowledges the - notification by fetching a location attribute for the file system - whose network access path is being changed. When there are - multiple such file systems, a location attribute for each such - file system needs to be fetched. The location attribute for all - migrated file system needs to be fetched in order to clear the - condition. Even after the condition is cleared, the client needs - to respond by using the location information to access the file - system at its new location to ensure that leases are not + notification by fetching a file system location attribute for the + file system whose network access path is being changed. When + there are multiple such file systems, a location attribute for + each such file system needs to be fetched. The location attribute + for all migrated file system needs to be fetched in order to clear + the condition. Even after the condition is cleared, the client + needs to respond by using the location information to access the + file system at its new location to ensure that leases are not needlessly expired. Unlike the case of NFSv4.0, in which the corresponding conditions are both errors and thus mutually exclusive, in NFSv4.1 the client can, and often will, receive both indications on the same request. As a result, implementations need to address the question of how to co- ordinate the necessary recovery actions when both indications arrive in the response to the same request. It should be noted that when processing an NFSv4 COMPOUND, the server will normally decide whether SEQ4_STATUS_LEASE_MOVED is to be set before it determines which file @@ -1601,26 +1710,28 @@ the fact, discussed above, that the two indications are not mutually exclusive, there are number of issues that are important in considering implementation of migration discovery, as discussed in Section 10.2. Because of the absence of NFSV4ERR_LEASE_MOVED, it is possible for file systems whose access path has not changed to be successfully accessed on a given server even though recovery is necessary for other file systems on the same server. As a result, access can go on while, + o The migration discovery process is going on for that server. o The transition recovery process is going on for on other file systems connected to that server. -10.2. Performing Migration Discovery (to be added) +10.2. Second sub-section within new section to be added to [RFC5661] to + be entitled "Performing Migration Discovery" Migration discovery can be performed in the same context as transition recovery, allowing recovery for each migrated file system to be invoked as it is discovered. Alternatively, it may be done in a separate migration discovery thread, allowing migration discovery to be done in parallel with one or more instances of transition recovery. In either case, because the lease-migrated indication does not result in an error. other access to file systems on the server can proceed @@ -1669,28 +1780,28 @@ o completion/verification of migration discovery processing, in which the possible completion of migration discovery processing needs to be verified. Given that framework, migration discovery processing would proceed as follows. o While in the normal-operation state, the thread performing discovery would fetch, for successive file systems known to the - client on the server being worked on, a location attribute plus - the fs_status attribute. + client on the server being worked on, a file system location + attribute plus the fs_status attribute. o If the fs_status attribute indicates that the file system is a migrated one (i.e. fss_absent is true and fss_type != STATUS4_REFERRAL) and thus that it is likely that the fetch of the - location attribute has cleared one the file systems contributing - to the lease-migrated indication. + file system location attribute has cleared one the file systems + contributing to the lease-migrated indication. o In cases in which that happened, the thread cannot know whether the lease-migrated indication has been cleared and so it enters the completion/verification state and proceeds to issue a COMPOUND to see if the LEASE_MOVED indication has been cleared. o When the discovery process is in the completion/verification state, if others request get a lease-migrated indication they note that it was received and the existence of such indications is used when the request completes, as described below. @@ -1714,45 +1825,46 @@ process. It should be noted that the process described above is not guaranteed to terminate, as a long series of new migration events might continually delay the clearing of the LEASE_MOVED indication. To prevent unnecessary lease expiration, it is appropriate for clients to use the discovery of migrations to effect lease renewal immediately, rather than waiting for clearing of the LEASE_MOVED indication when the complete set of migrations is available. -10.3. Overview of Client Response to NFS4ERR_MOVED (to be added) +10.3. Third sub-section within new section to be added to [RFC5661] to + be entitled "Overview of Client Response to NFS4ERR_MOVED" This section outlines a way in which a client that receives NFS4ERR_MOVED can effect transition recovery by using a new server or server endpoint if one is available. As part of that process, it will determine: o Whether the NFS4ERR_MOVED indicates migration has occurred, or whether it indicates another sort of file system access transition as discussed in Section 7 above. o In the case of migration, whether Transparent State Migration has occurred. o Whether any state has been lost during the process of Transparent State Migration. o Whether sessions have been transferred as part of Transparent State Migration. During the first phase of this process, the client proceeds to - examine location entries to find the initial network address it will - use to continue access to the file system or its replacement. For - each location entry that the client examines, the process consists of - five steps: + examine file system location entries to find the initial network + address it will use to continue access to the file system or its + replacement. For each location entry that the client examines, the + process consists of five steps: 1. Performing an EXCHANGE_ID directed at the location address. This operation is used to register the client-owner with the server, to obtain a client ID to be use subsequently to communicate with it, to obtain that client ID's confirmation status, and to determine server_owner and scope for the purpose of determining if the entry is trunkable with that previously being used to access the file system (i.e. that it represents another network access path to the same file system and can share locking state with it). @@ -1803,22 +1915,23 @@ 2. In the case that the network address is session-trunkable with one used previously a BIND_CONN_TO_SESSION is used to access that session using the new network address. Otherwise, or if the bind operation fails, a CREATE_SESSION is done. 3. The verification procedure referred to in step 4 above is used. However, if it fails, the entry is ignored and the next available entry is used. -10.4. Obtaining Access to Sessions and State after Migration (to be - added) +10.4. Fourth sub-section within new section to be added to [RFC5661] to + be entitled "Obtaining Access to Sessions and State after + Migration" In the event that migration has occurred, migration recovery will involve determining whether Transparent State Migration has occurred. This decision is made based on the client ID returned by the EXCHANGE_ID and the reported confirmation status. o If the client ID is an unconfirmed client ID not previously known to the client, then Transparent State Migration has not occurred. o If the client ID is a confirmed client ID previously known to the @@ -1888,43 +2001,45 @@ o In a case in which Transparent State Migration has occurred, and some lock state was lost (as shown by SEQ4_STATUS flags), existing stateids need to be checked for validity using TEST_STATEID, and reclaim used to re-establish any that were not transferred. For all of the cases above, RECLAIM_COMPLETE with an rca_one_fs value of TRUE needs to be done before normal use of the file system including obtaining new locks for the file system. This applies even if no locks were lost and there was no need for any to be reclaimed. -10.5. Obtaining Access to Sessions and State after Network Address - Transfer (to be added) +10.5. Fifth sub-section within new section to be added to [RFC5661] to + be entitled "Obtaining Access to Sessions and State after Network + Address Transfer" The case in which there is a transfer to a new network address without migration is similar to that described in Section 10.4 above in that there is a need to obtain access to needed sessions and locking state. However, the details are simpler and will vary depending on the type of trunking between the address receiving NFS4ERR_MOVED and that to which the transfer is to be made To make a session available for use, a BIND_CONN_TO_SESSION should be used to obtain access to the session previously in use. Only if this fails, should a CREATE_SESSION be done. While this procedure mirrors that in Section 10.4 above, there is an important difference in that preservation of the session is not purely optional but depends on the type of trunking. Access to appropriate locking state should need no actions beyond access to the session. However, the SEQ4_STATUS bits need to be checked for lost locking state, including the need to reclaim locks after a server reboot. -11. Server Responsibilities Upon Migration (to be added) +11. New section to be added third after Section 11.11 of [RFC5661] to + be entitled "Server Responsibilities Upon Migration" In the event of file system migration, when the client connects to the destination server, it needs to be able to provide the client continued to access the files it had open on the source server. There are two ways to provide this: o By provision of an fs-specific grace period, allowing the client the ability to reclaim its locks, in a fashion similar to what would have been done in the case of recovery from a server restart. See Section 11.1 for a more complete discussion. @@ -1940,22 +2055,23 @@ All the features described above can involve transfer of lock-related information between source and destination servers. In some cases this transfer is a necessary part of the implementation while in other cases it is a helpful implementation aid which servers might or might not use. The sub-sections below discuss the information which would transferred but do not define the specifics of the transfer protocol. This is left as an implementation choice although standards in this area could be developed at a later time. -11.1. Server Responsibilities in Effecting State Reclaim after - Migration (to be added) +11.1. First sub-section within new section to be added to [RFC5661] to + be entitled "Server Responsibilities in Effecting State Reclaim + after Migration" In this case, destination server need have no knowledge of the locks held on the source server, but relies on the clients to accurately report (via reclaim operations) the locks previously held, not allowing new locks to be granted on migrated file system until the grace period expires. During this grace period clients have the opportunity to use reclaim operations to obtain locks for file system objects within the migrated file system, in the same way that they do when recovering @@ -1976,22 +2092,23 @@ for the transferred file system, the destination server will be able to terminate the grace period once all such clients have reclaimed their locks, allowing normal locking activity to resume earlier than it would have otherwise. o Locking summary information for individual clients (at various possible levels of detail) can detect some instances in which clients do not accurately represent the locks held on the source server. -11.2. Server Responsibilities in Effecting Transparent State Migration - (to be added) +11.2. Second sub-section within new section to be added to [RFC5661] to + be entitled "Server Responsibilities in Effecting Transparent + State Migration" The basic responsibility of the source server in effecting Transparent State Migration is to make available to the destination server a description of each piece of locking state associated with the file system being migrated. In addition to client id string and verifier, the source server needs to provide, for each stateid: o The stateid including the current sequence value. o The associated client ID. @@ -2058,22 +2175,23 @@ longer exists. o Sequencing of operations is no longer done using owner-based operation sequences numbers. Instead, sequencing is session- based As a result, when sessions are not transferred, the techniques discussed in Section 7.2 of [RFC7931] are adequate and will not be further discussed. -11.3. Server Responsibilities in Effecting Session Transfer (to be - added) +11.3. Third sub-section within new section to be added to [RFC5661] to + be entitled "Server Responsibilities in Effecting Session + Transfer" The basic responsibility of the source server in effecting session transfer is to make available to the destination server a description of the current state of each slot with the session, including: o The last sequence value received for that slot. o Whether there is cached reply data for the last request executed and, if so, the cached reply. @@ -2196,23 +2315,25 @@ rather than to a path to access a replica. o In describing the appropriate value for a server to use for fli_valid_for, it needs to be made clear that there is no need for the client to frequently fetch the fs_locations_info value to be prepared for shifts in trunking patterns. o Clarification of the rules for extensions of the fls_info needs to be provided. The existing treatment reflects the extension model in effect at the time [RFC5661] was written, and need to be - updated in accord with the extension model described [RFC8178]. + updated in accordance with the extension model described in + [RFC8178]. -12.2. The Attribute fs_locations_info (as updated) +12.2. Updated Section 11.10 of [RFC5661] entitled "The Attribute + fs_locations_info" The fs_locations_info attribute is intended as a more functional replacement for the fs_locations attribute which will continue to exist and be supported. Clients can use it to get a more complete set of data about alternative file system locations, including additional network paths to access replicas in use and additional replicas. When the server does not support fs_locations_info, fs_locations can be used to get a subset of the data. A server that supports fs_locations_info MUST support fs_locations as well. @@ -2249,24 +2370,24 @@ The fs_locations_info attribute is structured similarly to the fs_locations attribute. A top-level structure (fs_locations_info4) contains the entire attribute including the root pathname of the file system and an array of lower-level structures that define replicas that share a common rootpath on their respective servers. The lower- level structure in turn (fs_locations_item4) contains a specific pathname and information on one or more individual network access paths. For that last lowest level, fs_locations_info has an fs_locations_server4 structure that contains per-server-replica - information in addition to the location entry. This per-server- - replica information includes a nominally opaque array, fls_info, - within which specific pieces of information are located at the - specific indices listed below. + information in addition to the file system location entry. This per- + server-replica information includes a nominally opaque array, + fls_info, within which specific pieces of information are located at + the specific indices listed below. Two fs_location_server4 entries that are within different fs_location_item4 structures are never trunkable, while two entries within in the same fs_location_item4 structure might or might not be trunkable. Two entries that are trunkable will have identical identity information, although, as noted above, the converse is not the case. The attribute will always contain at least a single fs_locations_server entry. Typically, there will be an entries with @@ -2375,21 +2495,22 @@ just referenced) and its successor location. Servers are strongly urged to support this attribute on all file systems if they support it on any file system. The data presented in the fs_locations_info attribute may be obtained by the server in any number of ways, including specification by the administrator or by current protocols for transferring data among replicas and protocols not yet developed. NFSv4.1 only defines how this information is presented by the server to the client. -12.2.1. The fs_locations_server4 Structure (as updated) +12.2.1. Updated section 11.10.1 of [RFC5661] entitled "The + fs_locations_server4 Structure" The fs_locations_server4 structure consists of the following items in addition to the fls_server field which specifies a network address or set of addresses to be used to access the specified file system. Note that both of these items specify attributes of the file system replica and should not be different when there are multiple fs_locations_server4 structures for the same replica, each specifying a network path to the chosen replica. o An indication of how up-to-date the file system is (fls_currency) @@ -2687,21 +2808,22 @@ o The field at byte index FSLI4BX_WRITEORDER gives the order value to be used for writable access. Depending on the potential need for write access by a given client, one of the pairs of rank and order values is used. The read rank and order should only be used if the client knows that only reading will ever be done or if it is prepared to switch to a different replica in the event that any write access capability is required in the future. -12.2.2. The fs_locations_info4 Structure (as updated) +12.2.2. Updated Section 11.10.2 of [RFC5661] entitled "The + fs_locations_info4 Structure" The fs_locations_info4 structure, encoding the fs_locations_info attribute, contains the following: o The fli_flags field, which contains general flags that affect the interpretation of this fs_locations_info4 structure and all fs_locations_item4 structures within it. The only flag currently defined is FSLI4IF_VAR_SUB. All bits in the fli_flags field that are not defined should always be returned as zero. @@ -2744,21 +2867,22 @@ Note that, because of the ability of the server to return NFS4ERR_MOVED to change to use of different paths, when alternate trunked paths are available, there is generally no need to use low values of fli_valid_for in connection with the management of alternate paths to the same replica. The FSLI4IF_VAR_SUB flag within fli_flags controls whether variable substitution is to be enabled. See Section 12.2.3 for an explanation of variable substitution. -12.2.3. The fs_locations_item4 Structure (as updated) +12.2.3. Updated Section 11.10.3 of [RFC5661] entitled "The + fs_locations_item4 Structure" The fs_locations_item4 structure contains a pathname (in the field fli_rootpath) that encodes the path of the target file system replicas on the set of servers designated by the included fs_locations_server4 entries. The precise manner in which this target location is specified depends on the value of the FSLI4IF_VAR_SUB flag within the associated fs_locations_info4 structure. If this flag is not set, then fli_rootpath simply designates the @@ -2836,21 +2960,21 @@ substituted variables, the result is always a valid successor file system instance to that from which a transition is occurring, i.e., that the data is identical or represents a later image of a writable file system. Note that when fli_rootpath is a null pathname (that is, one with zero components), the file system designated is at the root of the specified server, whether or not the FSLI4IF_VAR_SUB flag within the associated fs_locations_info4 structure is set. -13. Changes to RFC5661 outside Section 11 +13. Changes to [RFC5661] outside Section 11 Beside the major rework of Section 11, there are a number of related changes that are necessary: o The summary that appeared in Section 1.7.3.3 of [RFC5661] needs to be revised to reflect the changes called for in Section 4 of the current document. The updated summary appears as Section 13.1 below. o The discussion of server scope which appeared in Section 2.10.4 of @@ -2890,55 +3014,56 @@ the rca_one_fs and how the server is to deal with inappropriate values of this argument. Because the resulting confusion raises interoperability issues, a new treatment of RECLAIM_COMPLETE is necessary and it appears in Section 15 below while the specific differences between it and the treatment within [RFC5661] are discussed in Section 13.6 below. In addition, the definitions of the reclaim-related errors receive an updated treatment in Section 13.7 to reflect the fact that there are multiple contexts for lock reclaim operations. -13.1. (Introduction to) Multi-Server Namespace (as updated) +13.1. Updated section 1.7.3.3 of [RFC5661] to be retitled "Introduction + to Multi-Server Namespace" NFSv4.1 contains a number of features to allow implementation of namespaces that cross server boundaries and that allow and facilitate a non-disruptive transfer of support for individual file systems between servers. They are all based upon attributes that allow one file system to specify alternate, additional, and new location information which specifies how the client may access to access that file system. These attributes can be used to provide for individual active file systems: o Alternate network addresses to access the current file system instance. o The locations of alternate file system instances or replicas to be used in the event that the current file system instance becomes unavailable. - These attributes may be used together with the concept of absent file - systems, in which a position in the server namespace is associated - with locations on other servers without there being any corresponding - file system instance on the current server. + These file system location attributes may be used together with the + concept of absent file systems, in which a position in the server + namespace is associated with locations on other servers without there + being any corresponding file system instance on the current server. - o Location attributes may be used with absent file systems to - implement referrals whereby one server may direct the client to a - file system provided by another server. This allows extensive - multi-server namespaces to be constructed. + o These attributes may be used with absent file systems to implement + referrals whereby one server may direct the client to a file + system provided by another server. This allows extensive multi- + server namespaces to be constructed. - o Location attributes may be provided when a previously present file + o These attributes may be provided when a previously present file system becomes absent. This allows non-disruptive migration of file systems to alternate servers. -13.2. Server Scope (as updated) +13.2. Updated Section 2.10.4 of [RFC5661] entitled "Server Scope" Servers each specify a server scope value in the form of an opaque string eir_server_scope returned as part of the results of an EXCHANGE_ID operation. The purpose of the server scope is to allow a group of servers to indicate to clients that a set of servers sharing the same server scope value has arranged to use compatible values of otherwise opaque identifiers. Thus, the identifiers generated by two servers within that set can be assumed compatible so that, in some cases, identifiers by one server in that set that set may be presented to another server of the same scope. @@ -3028,55 +3153,86 @@ with it or it might not be present at the server. In the latter case, it might have been relocated or migrated to another server, or it might have never been present. The client may obtain information regarding access to the file system location by obtaining the "fs_locations" or "fs_locations_info" attribute for the current filehandle. For further discussion, refer to Section 11 of [RFC5661], as modified by the current document. 13.4. Revised Discussion of Server_owner changes - Because of likely problems with the treatment of such changes, a - confusing paragraph which appear at the end of Section 2.5.10 if - [RFC5661], which simply says that such changes need to be dealt with, - is to be replaced by the material below. + Section 2.10.5 of [RFC5661] discusses the issue of possible + server_owner changes as follows: + + The client should be prepared for the possibility that + eir_server_owner values may be different on subsequent EXCHANGE_ID + requests made to the same network address, as a result of various + sorts of reconfiguration events. When this happens and the + changes result in the invalidation of previously valid forms of + trunking, the client should cease to use those forms, either by + dropping connections or by adding sessions. For a discussion of + lock reclaim as it relates to such reconfiguration events, see + Section 8.4.2.1 + + While this paragraph is literally true in that such reconfiguration + events can happen and clients have to deal with them, it is confusing + in that it can be read as suggesting that clients have to deal with + them without disruption, which in general is impossible. This has + led to confusion especially since the text is not very clear about + the actions that might need to be done since: + + o The cases of change which are very disruptive (e.g. change if + server scope) are not sufficiently distinguished from those that + simply involve a change of trunking modes (i.e. change + server_owner minor id) + + o There is an undue focus on the effect of such changes as they + affect the comparison with corresponding value from other servers, + without fully dealing with the issue that result from value + discontinuity within a single server. + + Because of these issues the paragraph which appears at the end of + Section 2.10.5 needs to be replaced by the material below. It is always possible that, as a result of various sorts of reconfiguration events, eir_server_scope and eir_server_owner values may be different on subsequent EXCHANGE_ID requests made to the same network address. In most cases such reconfiguration events will be disruptive and indicate that an IP address formerly connected to one server is now connected to an entirely different one. Some guidelines on client handling of such situations follow: * When eir_server_scope changes, the client has no assurance that - any id's it obtained previously (e.g. file handles) can be - validly used on the new server, and, even if the new server - accepts them, there is no assurance that this is not due to - accident. Thus it is best to treat all such state as lost/ - stale although a client may assume that the probability of - inadvertent acceptance is low and treat this situation as + any id's it obtained previously (e.g. file handles, state ids, + client ids) can be validly used on the new server, and, even if + the new server accepts them, there is no assurance that this is + not due to accident. Thus it is best to treat all such state + as lost/stale although a client may assume that the probability + of inadvertent acceptance is low and treat this situation as within the next case. * When eir_server_scope remains the same and - eir_server_owner.so_major_id changes, the client can use - filehandles it has and attempt reclaims. It may find that - these are now stale but if NFS4ERR_STALE is not received, he - can proceed to reclaim his opens. + eir_server_owner.so_major_id changes, the client can use the + filehandles it has, consider its locking state lost, and + attempt to reclaim or otherwise re-obtain its locks. It may + find that its file handle are now stale but if NFS4ERR_STALE is + not received, it can proceed to reclaim or otherwise re-obtain + its open locking state. * When eir_server_scope and eir_server_owner.so_major_id remain the same, the client has to use the now-current values of eir_server_owner.so_minor_id in deciding on appropriate forms - of trunking. + of trunking. This may result in connections being dropped or + new sessions being created. 13.5. Revision to Treatment of EXCHANGE_ID There are a number of issues in the original treatment of EXCHANGE_ID (in [RFC5661]) that cause problems for Transparent State Migration and for the transfer of access between different network access paths to the same file system instance. These issues arise from the fact that this treatment was written: @@ -3135,86 +3291,92 @@ o In a number of places the text is more explicit about the purpose of rca_one_fs and its connection to file system migration. o There is a discussion of situations in which either form of RECLAIM_COMPLETE would need to be done. o There is a discussion of interoperability issues that result from implementations that may have arisen due to the lack of clarity of the previous treatment of RECLAIM_COMPLETE. -13.7. Reclaim Errors (as updated) +13.7. Updated Section 15.1.9 of [RFC5661] entitled "Reclaim Errors" These errors relate to the process of reclaiming locks after a server restart or in connection with the migration of a file system (i.e. in the case in which rca_one_fs is TRUE). -13.7.1. NFS4ERR_COMPLETE_ALREADY (as updated; Error Code 10054) +13.7.1. Updated Section 15.1.9.1 of [RFC5661] entitled + "NFS4ERR_COMPLETE_ALREADY (Error Code 10054)" The client previously sent a successful RECLAIM_COMPLETE operation specifying the same scope, whether that scope is global or for the same file system in the case of a per-fs RECLAIM_COMPLETE. An additional RECLAIM_COMPLETE operation is not necessary and results in this error. -13.7.2. NFS4ERR_GRACE (as updated; Error Code 10013) +13.7.2. Updated Section 15.1.9.2 of [RFC5661] entitled "NFS4ERR_GRACE + (Error Code 10013)" The server was in its recovery or grace period, with regard to the file system object for which the lock was requested. The locking request was not a reclaim request and so could not be granted during that period. -13.7.3. NFS4ERR_NO_GRACE (as updated; Error Code 10033) +13.7.3. Updated Section 15.1.9.3 of [RFC5661] entitled + "NFS4ERR_NO_GRACE (Error Code 10033)" A reclaim of client state was attempted in circumstances in which the server cannot guarantee that conflicting state has not been provided to another client. This can occur because the reclaim has been done outside of a grace period of implemented by the server, after the client has done a RECLAIM_COMPLETE operation which ends its ability to reclaim the requested lock, or because previous operations have created a situation in which the server is not able to determine that a reclaim-interfering edge condition does not exist. -13.7.4. NFS4ERR_RECLAIM_BAD (as updated; Error Code 10034) +13.7.4. Updated Section 15.1.9.4 of [RFC5661] entitled + "NFS4ERR_RECLAIM_BAD (Error Code 10034)" The server has determined that a reclaim attempted by the client is not valid, i.e. the lock specified as being reclaimed could not possibly have existed before the server restart or file system migration event. A server is not obliged to make this determination and will typically rely on the client to only reclaim locks that the client was granted prior to restart or file system migration. However, when a server does have reliable information to enable it make this determination, this error indicates that the reclaim has been rejected as invalid. This is as opposed to the error NFS4ERR_RECLAIM_CONFLICT (see Section 13.7.5) where the server can only determine that there has been an invalid reclaim, but cannot determine which request is invalid. -13.7.5. NFS4ERR_RECLAIM_CONFLICT (as updated; Error Code 10035) +13.7.5. Updated Section 15.1.9.5 of [RFC5661] entitled + "NFS4ERR_RECLAIM_CONFLICT (Error Code 10035)" The reclaim attempted by the client has encountered a conflict and cannot be satisfied. Potentially indicates a misbehaving client, although not necessarily the one receiving the error. The misbehavior might be on the part of the client that established the lock with which this client conflicted. See also Section 13.7.4 for the related error, NFS4ERR_RECLAIM_BAD. -14. Operation 42: EXCHANGE_ID - Instantiate Client ID (as updated) +14. Updated Section 18.35 of [RFC5661] entitled "Operation 42: + EXCHANGE_ID - Instantiate Client ID" The EXCHANGE_ID exchanges long-hand client and server identifiers (owners), and provides access to a client ID, creating one if necessary. This client ID becomes associated with the connection on which the operation is done, so that it is available when a CREATE_SESSION is done or when the connection is used to issue a request on an existing session associated with the current client. -14.1. ARGUMENT +14.1. Updated Section 18.35.1 of [RFC5661] entitled "ARGUMENT" const EXCHGID4_FLAG_SUPP_MOVED_REFER = 0x00000001; const EXCHGID4_FLAG_SUPP_MOVED_MIGR = 0x00000002; const EXCHGID4_FLAG_BIND_PRINC_STATEID = 0x00000100; const EXCHGID4_FLAG_USE_NON_PNFS = 0x00010000; const EXCHGID4_FLAG_USE_PNFS_MDS = 0x00020000; @@ -3254,21 +3416,21 @@ struct EXCHANGE_ID4args { client_owner4 eia_clientowner; uint32_t eia_flags; state_protect4_a eia_state_protect; nfs_impl_id4 eia_client_impl_id<1>; }; -14.2. RESULT +14.2. Updated Section 18.35.2 of [RFC5661] entitled "RESULT" struct ssv_prot_info4 { state_protect_ops4 spi_ops; uint32_t spi_hash_alg; uint32_t spi_encr_alg; uint32_t spi_ssv_len; uint32_t spi_window; gsshandle4_t spi_handles<>; }; @@ -3295,21 +3457,21 @@ union EXCHANGE_ID4res switch (nfsstat4 eir_status) { case NFS4_OK: EXCHANGE_ID4resok eir_resok4; default: void; }; -14.3. DESCRIPTION +14.3. Updated Section 18.35.3 of [RFC5661] entitled "DESCRIPTION" The client uses the EXCHANGE_ID operation to register a particular client_owner with the server. However, when the client_owner has been already been registered by other means (e.g. Transparent State Migration), the client may still use EXCHANGE_ID to obtain the client ID assigned previously. The client ID returned from this operation will be associated with the connection on which the EXHANGE_ID is received and will serve as a parent object for sessions created by the client on this connection @@ -3776,21 +3938,21 @@ peer's manifesting a particular allowed behavior based on an implementation identifier but are required to interoperate as specified elsewhere in the protocol specification. Because it is possible that some implementations might violate the protocol specification and interpret the identity information, implementations MUST provide facilities to allow the NFSv4 client and server be configured to set the contents of the nfs_impl_id structures sent to any specified value. -14.4. IMPLEMENTATION +14.4. Updated Section 18.35.4 of [RFC5661] entitled "IMPLEMENTATION" A server's client record is a 5-tuple: 1. co_ownerid The client identifier string, from the eia_clientowner structure of the EXCHANGE_ID4args structure. 2. co_verifier: A client-specific value used to indicate incarnations (where a @@ -4050,50 +4212,50 @@ If EXCHGID4_FLAG_UPD_CONFIRMED_REC_A is set, and the server has the following confirmed record, then this request is an illegal attempt at an update by an unauthorized principal. { ownerid_arg, verifier_arg, old_principal_arg, clientid_ret, confirmed } The server returns NFS4ERR_PERM and leaves the client record intact. -15. Operation 58: RECLAIM_COMPLETE - Indicates Reclaims Finished (as - updated) +15. Updated Section 18.51 of [RFC5661] entitled "Operation 58: + RECLAIM_COMPLETE - Indicates Reclaims Finished" -15.1. ARGUMENT +15.1. Updated Section 18.51.1 of [RFC5661] entitled "ARGUMEBNT" struct RECLAIM_COMPLETE4args { /* * If rca_one_fs TRUE, * * CURRENT_FH: object in * file system reclaim is * complete for. */ bool rca_one_fs; }; -15.2. RESULTS +15.2. Updated Section 18.51.2 of [RFC5661] entitled "RESULTS" struct RECLAIM_COMPLETE4res { nfsstat4 rcr_status; }; -15.3. DESCRIPTION +15.3. Updated Section 18.51.3 of [RFC5661] entitled "DESCRIPTION" A RECLAIM_COMPLETE operation is used to indicate that the client has reclaimed all of the locking state that it will recover using reclaim, when it is recovering state due to either a server restart or the migration of a file system to another server. There are two types of RECLAIM_COMPLETE operations: o When rca_one_fs is FALSE, a global RECLAIM_COMPLETE is being done. This indicates that recovery of all locks that the client held on the previous server instance have been completed. The current @@ -4146,21 +4308,21 @@ locks that it does not own, and so has no right to reclaim. See Section 8.4.3 of [RFC5661] for a discussion of edge conditions related to lock reclaim. By sending a RECLAIM_COMPLETE, the client indicates readiness to proceed to do normal non-reclaim locking operations. The client should be aware that such operations may temporarily result in NFS4ERR_GRACE errors until the server is ready to terminate its grace period. -15.4. IMPLEMENTATION +15.4. Updated Section 18.51.4 of [RFC5661] entitled "IMPLEMENTATION" Servers will typically use the information as to when reclaim activity is complete to reduce the length of the grace period. When the server maintains in persistent storage a list of clients that might have had locks, it is able to use the fact that all such clients have done a RECLAIM_COMPLETE to terminate the grace period and begin normal operations (i.e., grant requests for new locks) sooner than it might otherwise. Latency can be minimized by doing a RECLAIM_COMPLETE as part of the @@ -4223,71 +4385,72 @@ In light of this, the following considerations should be taken note of: o When DNS is used to convert server named to addresses and DNSSEC [RFC4033] is not available, the validity of the network addresses returned cannot be relied upon. However, when the client uses RPCSEC_GSS to access the designated server, it is possible for mutual authentication to discover invalid server addresses provided. - o The fetching of attributes containing location information - SHOULD be performed using RPCSEC_GSS with integrity protection, - as previously explained in the Security Considerations section - of [RFC5661]. It is important to note here that a client - making a request of this sort without using RPCSEC_GSS - including integrity protection needs be aware of the negative - consequences of doing so, which can lead to invalid host names - or network addresses being returned. In light of this, the - client needs to recognize that using such returned location - information to access an NFSv4 server without use of RPCSEC_GSS - (i.e. by using AUTH_SYS) poses dangers as it can result in the - client interacting with an unverified network address posing as - an NFSv4 server. + o The fetching of attributes containing file system location + information SHOULD be performed using RPCSEC_GSS with integrity + protection, as previously explained in the Security + Considerations section of [RFC5661]. It is important to note + here that a client making a request of this sort without using + RPCSEC_GSS including integrity protection needs be aware of the + negative consequences of doing so, which can lead to invalid + host names or network addresses being returned. In light of + this, the client needs to recognize that using such returned + location information to access an NFSv4 server without use of + RPCSEC_GSS (i.e. by using AUTH_SYS) poses dangers as it can + result in the client interacting with an unverified network + address posing as an NFSv4 server. o Despite the fact that it is a REQUIREMENT (of [RFC5661]) that "implementations" provide "support" for use of RPCSEC_GSS, it cannot be assumed that use of RPCSEC_GSS is always available between any particular client-server pair. o When a client has the network addresses of a server but not the associated host names, that would interfere with its ability to use RPCSEC_GSS. - In light of the above, a server should present location entries - that correspond to file systems on other servers using a host - name. This would allow the client to interrogate the fs_locations - on the destination server to obtain trunking information (as well - as replica information) using RPCSEC_GSS with integrity, - validating the name provided while assuring that the response has - not been corrupted. + In light of the above, a server should present file system + location entries that correspond to file systems on other servers + using a host name. This would allow the client to interrogate the + fs_locations on the destination server to obtain trunking + information (as well as replica information) using RPCSEC_GSS with + integrity, validating the name provided while assuring that the + response has not been corrupted. When RPCSEC_GSS is not available on a server, the client needs to be aware of the fact that the location entries are subject to corruption and cannot be relied upon. In the case of a client being directed to another server after NFS4ERR_MOVED, this could vitiate the authentication provided by the use of RPCSEC_GSS on the destination. Even when RPCSEC_GSS authentication is available on the destination, the server might validly represent itself as the server to which the client was erroneously directed. Without a way to decide whether the server is a valid one, the client can only determine, using RPCSEC_GSS, that the server corresponds to the name provided, with no basis for trusting that server. As a result, the client should not use such unverified location entries as a basis for migration, even though RPCSEC_GSS might be available on the destination. - When a location attribute is fetched upon connecting with an NFS - server, it SHOULD, as stated above, be done using RPCSEC_GSS with - integrity protection. When this not possible, it is generally - best for the client to ignore trunking and replica information or - simply not fetch the location information for these purposes. + When a file system location attribute is fetched upon connecting + with an NFS server, it SHOULD, as stated above, be done using + RPCSEC_GSS with integrity protection. When this not possible, it + is generally best for the client to ignore trunking and replica + information or simply not fetch the location information for these + purposes. When location information cannot be verified, it can be subjected to additional filtering to prevent the client from being inappropriately directed. For example, if a range of network addresses can be determined that assure that the servers and clients using AUTH_SYS are subject to the appropriate set of constrains (e.g. physical network isolation, administrative controls on the operating systems used), then network addresses in the appropriate range can be used with others discarded or restricted in their use of AUTH_SYS. @@ -4464,21 +4627,21 @@ To summarize: o There are seventeen explanatory sections. o There are thirty-seven replacement sections. o There are eightteen additional sections. o There are three editing sections. -Appendix B. Updates to RFC5661 +Appendix B. Updates to [RFC5661] In this appendix, we proceed through [RFC5661] identifying sections as unchanged, modified, deleted, or replaced and indicating where additional sections from the current document would appear in an eventual consolidated description of NFSv4.1. In this presentation, when section X is referred to, it denotes that section plus all included subsections. When it is necessary to refer to the part of a section outside any included subsections, the exclusion is noted explicitly. @@ -4505,64 +4668,76 @@ document. o Sections 11.2, 11.3, 11.3.1, and 11.3.2 are unchanged. o Section 11.4 is replaced by Section 4.5 from the current document. For details regarding subsections see below. o New sections corresponding to Sections 4.5.1 through 4.5.3 from the current document appear next. - o Section 11.4.1 is replaced by Section 4.5.4 + o Section 11.4.1 is replaced by Section 4.5.4 from the current + document. - o Section 11.4.2 is replaced by Section 4.5.5 + o Section 11.4.2 is replaced by Section 4.5.5 from the current + document. - o Section 11.4.3 is replaced by Section 4.5.6 + o Section 11.4.3 is replaced by Section 4.5.6 from the current + document. o A new section corresponding to Section 4.5.7 from the current document appears next. o Section 11.5 is to be deleted. o Section 11.6 is unchanged. o New sections corresponding to Sections 6 and 7 from the current document appear next. o Section 11.7 is replaced by Section 8 from the current document. For details regarding subsections see below. - o Section 11.7.1 is replaced by Section 8.1 + o Section 11.7.1 is replaced by Section 8.1 from the current + document. o Sections 11.7.2, 11.7.2.1, and 11.7.2.2 are deleted. - o Section 11.7.3 is replaced by Section 8.2 + o Section 11.7.3 is replaced by Section 8.2 from the current + document. + + o Section 11.7.4 is replaced by Section 8.3 from the current + document. - o Section 11.7.4 is replaced by Section 8.3 o Sections 11.7.5 and 11.7.5.1 are replaced by Sections 8.4 - and 8.4.1 respectively. + and 8.4.1 respectively, from the current document. - o Section 11.7.6 is replaced by Section 8.5 + o Section 11.7.6 is replaced by Section 8.5 from the current + document. o Section 11.7.7, exclusive of subsections, is replaced by - Section 8.9. Sections 11.7.7.1 and 11.7.72 are unchanged. + Section 8.9 from the current document. Sections 11.7.7.1 + and 11.7.7.2 are unchanged. - o Section 11.7.8 is replaced by Section 8.6 + o Section 11.7.8 is replaced by Section 8.6 from the current + document. - o Section 11.7.9 is replaced by Section 8.7 + o Section 11.7.9 is replaced by Section 8.7 from the current + document. - o Section 11.7.10 is replaced by Section 8.8 + o Section 11.7.10 is replaced by Section 8.8 from the current + document. o Sections 11.8, 11.8.1, 11.8.2, and 11.9, are unchanged. o Sections 11.10, 11.10.1, 11.10.2, and 11.10.3 are replaced by - Sections 12.2 through 12.2.3. + Sections 12.2 through 12.2.3 from the current document. o Section 11.11 is unchanged. o New sections corresponding to Sections 9, 10, and 11 from the current document appear next as additional sub-sections of Section 11. Each of these has subsections, so there is a total of seventeen sections added. o Sections 12 through 14 are unchanged. @@ -4570,21 +4745,21 @@ * The description of NFS4ERR_MOVED in Section 15.1 is revised as described in Section 13.3 of the current document. * The description of the reclaim-related errors in section 15.1.9 is replaced by the revised descriptions in Section 13.7 of the current document. o Sections 16 and 17 are unchanged. - o Section 18 is unmodified except the + o Section 18 is unmodified except for the following: * Section 18.35 is replaced by Section 14 in the current document. * Section 18.51 is replaced by Section 15 in the current document. o Sections 19 through 23 are unchanged. In terms of top-level sections, exclusive of appendices: @@ -4620,22 +4795,22 @@ | Deleted | 0 | 1 | 4 | 0 | 4 | | Modified | 5 | 3 | 0 | 2 | 2 | | Unchanged | 18 | 210 | 12 | 910 | 922 | | in RFC5661 | 23 | 220 | 37 | 927 | 964 | +------------+------+------+--------+------------+--------+ Acknowledgments The authors wish to acknowledge the important role of Andy Adamson of Netapp in clarifying the need for trunking discovery functionality, - and exploring the role of the location attributes in providing the - necessary support. + and exploring the role of the file system location attributes in + providing the necessary support. The authors also wish to acknowledge the work of Xuan Qi of Oracle with NFSv4.1 client and server prototypes of transparent state migration functionality. The authors wish to thank others that brought attention to important issues. The comments of Trond Myklebust of Primary Data related to trunking helped to clarify the role of DNS in trunking discovery. Rick Macklem's comments brought attention to problems in the handling of the per-fs version of RECLAIM_COMPLETE.