draft-ietf-nfsv4-minorversion2-13.txt   draft-ietf-nfsv4-minorversion2-14.txt 
NFSv4 T. Haynes NFSv4 T. Haynes
Internet-Draft Editor Internet-Draft Editor
Intended status: Standards Track July 11, 2012 Intended status: Standards Track September 30, 2012
Expires: January 12, 2013 Expires: April 3, 2013
NFS Version 4 Minor Version 2 NFS Version 4 Minor Version 2
draft-ietf-nfsv4-minorversion2-13.txt draft-ietf-nfsv4-minorversion2-14.txt
Abstract Abstract
This Internet-Draft describes NFS version 4 minor version two, This Internet-Draft describes NFS version 4 minor version two,
focusing mainly on the protocol extensions made from NFS version 4 focusing mainly on the protocol extensions made from NFS version 4
minor version 0 and NFS version 4 minor version 1. Major extensions minor version 0 and NFS version 4 minor version 1. Major extensions
introduced in NFS version 4 minor version two include: Server-side introduced in NFS version 4 minor version two include: Server-side
Copy, Application I/O Advise, Space Reservations, Sparse Files, Copy, Application I/O Advise, Space Reservations, Sparse Files,
Application Data Blocks, and Labeled NFS. Application Data Blocks, and Labeled NFS.
skipping to change at page 1, line 41 skipping to change at page 1, line 41
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 12, 2013. This Internet-Draft will expire on April 3, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 19 skipping to change at page 3, line 19
1.2. Scope of This Document . . . . . . . . . . . . . . . . . 5 1.2. Scope of This Document . . . . . . . . . . . . . . . . . 5
1.3. NFSv4.2 Goals . . . . . . . . . . . . . . . . . . . . . . 5 1.3. NFSv4.2 Goals . . . . . . . . . . . . . . . . . . . . . . 5
1.4. Overview of NFSv4.2 Features . . . . . . . . . . . . . . 6 1.4. Overview of NFSv4.2 Features . . . . . . . . . . . . . . 6
1.4.1. Sparse Files . . . . . . . . . . . . . . . . . . . . . 6 1.4.1. Sparse Files . . . . . . . . . . . . . . . . . . . . . 6
1.4.2. Application I/O Advise . . . . . . . . . . . . . . . . 6 1.4.2. Application I/O Advise . . . . . . . . . . . . . . . . 6
1.5. Differences from NFSv4.1 . . . . . . . . . . . . . . . . 6 1.5. Differences from NFSv4.1 . . . . . . . . . . . . . . . . 6
2. NFS Server-side Copy . . . . . . . . . . . . . . . . . . . . . 6 2. NFS Server-side Copy . . . . . . . . . . . . . . . . . . . . . 6
2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 6 2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 7 2.2. Protocol Overview . . . . . . . . . . . . . . . . . . . . 7
2.2.1. Overview of Copy Operations . . . . . . . . . . . . . 7 2.2.1. Overview of Copy Operations . . . . . . . . . . . . . 7
2.2.2. Intra-Server Copy . . . . . . . . . . . . . . . . . . 8 2.2.2. Locking the Files . . . . . . . . . . . . . . . . . . 8
2.2.3. Inter-Server Copy . . . . . . . . . . . . . . . . . . 9 2.2.3. Intra-Server Copy . . . . . . . . . . . . . . . . . . 8
2.2.4. Server-to-Server Copy Protocol . . . . . . . . . . . . 12 2.2.4. Inter-Server Copy . . . . . . . . . . . . . . . . . . 10
2.3. Requirements for Operations . . . . . . . . . . . . . . . 14 2.2.5. Server-to-Server Copy Protocol . . . . . . . . . . . . 14
2.3.1. netloc4 - Network Locations . . . . . . . . . . . . . 14 2.3. Requirements for Operations . . . . . . . . . . . . . . . 15
2.3.2. Copy Offload Stateids . . . . . . . . . . . . . . . . 15 2.3.1. netloc4 - Network Locations . . . . . . . . . . . . . 16
2.4. Security Considerations . . . . . . . . . . . . . . . . . 16 2.3.2. Copy Offload Stateids . . . . . . . . . . . . . . . . 16
2.4.1. Inter-Server Copy Security . . . . . . . . . . . . . . 16 2.4. Security Considerations . . . . . . . . . . . . . . . . . 17
3. Support for Application IO Hints . . . . . . . . . . . . . . . 24 2.4.1. Inter-Server Copy Security . . . . . . . . . . . . . . 17
4. Sparse Files . . . . . . . . . . . . . . . . . . . . . . . . . 24 3. Support for Application IO Hints . . . . . . . . . . . . . . . 25
4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 24 4. Sparse Files . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 25 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 25
5. Space Reservation . . . . . . . . . . . . . . . . . . . . . . 25 4.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 26
5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 26 5. Space Reservation . . . . . . . . . . . . . . . . . . . . . . 26
6. Application Data Block Support . . . . . . . . . . . . . . . . 28 5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 27
6. Application Data Hole Support . . . . . . . . . . . . . . . . 29
6.1. Generic Framework . . . . . . . . . . . . . . . . . . . . 29 6.1. Generic Framework . . . . . . . . . . . . . . . . . . . . 29
6.1.1. Data Block Representation . . . . . . . . . . . . . . 29 6.1.1. Data Hole Representation . . . . . . . . . . . . . . . 30
6.1.2. Data Content . . . . . . . . . . . . . . . . . . . . . 30 6.1.2. Data Content . . . . . . . . . . . . . . . . . . . . . 30
6.2. pNFS Considerations . . . . . . . . . . . . . . . . . . . 30 6.2. An Example of Detecting Corruption . . . . . . . . . . . 31
6.3. An Example of Detecting Corruption . . . . . . . . . . . 30 6.3. Example of READ_PLUS . . . . . . . . . . . . . . . . . . 32
6.4. Example of READ_PLUS . . . . . . . . . . . . . . . . . . 32 7. Labeled NFS . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.5. Zero Filled Holes . . . . . . . . . . . . . . . . . . . . 32
7. Labeled NFS . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 33 7.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 33
7.2. Definitions . . . . . . . . . . . . . . . . . . . . . . . 34 7.2. Definitions . . . . . . . . . . . . . . . . . . . . . . . 34
7.3. MAC Security Attribute . . . . . . . . . . . . . . . . . 34 7.3. MAC Security Attribute . . . . . . . . . . . . . . . . . 34
7.3.1. Delegations . . . . . . . . . . . . . . . . . . . . . 35 7.3.1. Delegations . . . . . . . . . . . . . . . . . . . . . 35
7.3.2. Permission Checking . . . . . . . . . . . . . . . . . 35 7.3.2. Permission Checking . . . . . . . . . . . . . . . . . 35
7.3.3. Object Creation . . . . . . . . . . . . . . . . . . . 35 7.3.3. Object Creation . . . . . . . . . . . . . . . . . . . 36
7.3.4. Existing Objects . . . . . . . . . . . . . . . . . . . 36 7.3.4. Existing Objects . . . . . . . . . . . . . . . . . . . 36
7.3.5. Label Changes . . . . . . . . . . . . . . . . . . . . 36 7.3.5. Label Changes . . . . . . . . . . . . . . . . . . . . 36
7.4. pNFS Considerations . . . . . . . . . . . . . . . . . . . 37 7.4. pNFS Considerations . . . . . . . . . . . . . . . . . . . 37
7.5. Discovery of Server Labeled NFS Support . . . . . . . . . 37 7.5. Discovery of Server Labeled NFS Support . . . . . . . . . 37
7.6. MAC Security NFS Modes of Operation . . . . . . . . . . . 37 7.6. MAC Security NFS Modes of Operation . . . . . . . . . . . 38
7.6.1. Full Mode . . . . . . . . . . . . . . . . . . . . . . 38 7.6.1. Full Mode . . . . . . . . . . . . . . . . . . . . . . 38
7.6.2. Guest Mode . . . . . . . . . . . . . . . . . . . . . . 39 7.6.2. Guest Mode . . . . . . . . . . . . . . . . . . . . . . 39
7.7. Security Considerations . . . . . . . . . . . . . . . . . 39 7.7. Security Considerations . . . . . . . . . . . . . . . . . 39
8. Sharing change attribute implementation details with NFSv4 8. Sharing change attribute implementation details with NFSv4
clients . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 clients . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 40 8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 40
9. Security Considerations . . . . . . . . . . . . . . . . . . . 40 9. Security Considerations . . . . . . . . . . . . . . . . . . . 40
10. Error Values . . . . . . . . . . . . . . . . . . . . . . . . . 40 10. Error Values . . . . . . . . . . . . . . . . . . . . . . . . . 41
10.1. Error Definitions . . . . . . . . . . . . . . . . . . . . 41 10.1. Error Definitions . . . . . . . . . . . . . . . . . . . . 41
10.1.1. General Errors . . . . . . . . . . . . . . . . . . . . 41 10.1.1. General Errors . . . . . . . . . . . . . . . . . . . . 41
10.1.2. Server to Server Copy Errors . . . . . . . . . . . . . 41 10.1.2. Server to Server Copy Errors . . . . . . . . . . . . . 41
10.1.3. Labeled NFS Errors . . . . . . . . . . . . . . . . . . 42 10.1.3. Labeled NFS Errors . . . . . . . . . . . . . . . . . . 42
11. New File Attributes . . . . . . . . . . . . . . . . . . . . . 42 11. New File Attributes . . . . . . . . . . . . . . . . . . . . . 42
11.1. New RECOMMENDED Attributes - List and Definition 11.1. New RECOMMENDED Attributes - List and Definition
References . . . . . . . . . . . . . . . . . . . . . . . 42 References . . . . . . . . . . . . . . . . . . . . . . . 42
11.2. Attribute Definitions . . . . . . . . . . . . . . . . . . 43 11.2. Attribute Definitions . . . . . . . . . . . . . . . . . . 43
12. Operations: REQUIRED, RECOMMENDED, or OPTIONAL . . . . . . . . 46 12. Operations: REQUIRED, RECOMMENDED, or OPTIONAL . . . . . . . . 46
13. NFSv4.2 Operations . . . . . . . . . . . . . . . . . . . . . . 49 13. NFSv4.2 Operations . . . . . . . . . . . . . . . . . . . . . . 50
13.1. Operation 59: COPY - Initiate a server-side copy . . . . 49 13.1. Operation 59: COPY - Initiate a server-side copy . . . . 50
13.2. Operation 60: COPY_ABORT - Cancel a server-side copy . . 57 13.2. Operation 60: OFFLOAD_ABORT - Cancel a server-side
copy . . . . . . . . . . . . . . . . . . . . . . . . . . 57
13.3. Operation 61: COPY_NOTIFY - Notify a source server of 13.3. Operation 61: COPY_NOTIFY - Notify a source server of
a future copy . . . . . . . . . . . . . . . . . . . . . . 58 a future copy . . . . . . . . . . . . . . . . . . . . . . 58
13.4. Operation 62: COPY_REVOKE - Revoke a destination 13.4. Operation 62: OFFLOAD_REVOKE - Revoke a destination
server's copy privileges . . . . . . . . . . . . . . . . 59 server's copy privileges . . . . . . . . . . . . . . . . 60
13.5. Operation 63: COPY_STATUS - Poll for status of a 13.5. Operation 63: OFFLOAD_STATUS - Poll for status of a
server-side copy . . . . . . . . . . . . . . . . . . . . 60 server-side copy . . . . . . . . . . . . . . . . . . . . 61
13.6. Modification to Operation 42: EXCHANGE_ID - 13.6. Modification to Operation 42: EXCHANGE_ID -
Instantiate Client ID . . . . . . . . . . . . . . . . . . 62 Instantiate Client ID . . . . . . . . . . . . . . . . . . 62
13.7. Operation 64: INITIALIZE . . . . . . . . . . . . . . . . 63 13.7. Operation 64: INITIALIZE . . . . . . . . . . . . . . . . 63
13.8. Operation 67: IO_ADVISE - Application I/O access 13.8. Operation 67: IO_ADVISE - Application I/O access
pattern hints . . . . . . . . . . . . . . . . . . . . . . 66 pattern hints . . . . . . . . . . . . . . . . . . . . . . 67
13.9. Changes to Operation 51: LAYOUTRETURN . . . . . . . . . . 72 13.9. Changes to Operation 51: LAYOUTRETURN . . . . . . . . . . 72
13.10. Operation 65: READ_PLUS . . . . . . . . . . . . . . . . . 75 13.10. Operation 65: READ_PLUS . . . . . . . . . . . . . . . . . 75
13.11. Operation 66: SEEK . . . . . . . . . . . . . . . . . . . 80 13.11. Operation 66: SEEK . . . . . . . . . . . . . . . . . . . 80
14. NFSv4.2 Callback Operations . . . . . . . . . . . . . . . . . 81 14. NFSv4.2 Callback Operations . . . . . . . . . . . . . . . . . 81
14.1. Procedure 16: CB_ATTR_CHANGED - Notify Client that 14.1. Procedure 16: CB_ATTR_CHANGED - Notify Client that
the File's Attributes Changed . . . . . . . . . . . . . . 81 the File's Attributes Changed . . . . . . . . . . . . . . 81
14.2. Operation 15: CB_COPY - Report results of a 14.2. Operation 15: CB_COPY - Report results of a
server-side copy . . . . . . . . . . . . . . . . . . . . 82 server-side copy . . . . . . . . . . . . . . . . . . . . 82
15. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 83 15. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 84
16. References . . . . . . . . . . . . . . . . . . . . . . . . . . 83 16. References . . . . . . . . . . . . . . . . . . . . . . . . . . 84
16.1. Normative References . . . . . . . . . . . . . . . . . . 83 16.1. Normative References . . . . . . . . . . . . . . . . . . 84
16.2. Informative References . . . . . . . . . . . . . . . . . 84 16.2. Informative References . . . . . . . . . . . . . . . . . 85
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . . 85 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . . 86
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 86 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 87
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 86 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 87
1. Introduction 1. Introduction
1.1. The NFS Version 4 Minor Version 2 Protocol 1.1. The NFS Version 4 Minor Version 2 Protocol
The NFS version 4 minor version 2 (NFSv4.2) protocol is the third The NFS version 4 minor version 2 (NFSv4.2) protocol is the third
minor version of the NFS version 4 (NFSv4) protocol. The first minor minor version of the NFS version 4 (NFSv4) protocol. The first minor
version, NFSv4.0, is described in [10] and the second minor version, version, NFSv4.0, is described in [10] and the second minor version,
NFSv4.1, is described in [2]. It follows the guidelines for minor NFSv4.1, is described in [2]. It follows the guidelines for minor
versioning that are listed in Section 11 of [10]. versioning that are listed in Section 11 of [10].
skipping to change at page 8, line 9 skipping to change at page 8, line 9
compatible with the traditional copy authentication approach. The compatible with the traditional copy authentication approach. The
client and user are authorized at the source for reading. Then they client and user are authorized at the source for reading. Then they
are authorized at the destination for writing. are authorized at the destination for writing.
2.2.1. Overview of Copy Operations 2.2.1. Overview of Copy Operations
COPY_NOTIFY: For inter-server copies, the client sends this COPY_NOTIFY: For inter-server copies, the client sends this
operation to the source server to notify it of a future file copy operation to the source server to notify it of a future file copy
from a given destination server for the given user. from a given destination server for the given user.
(Section 13.3) (Section 13.3)
COPY_REVOKE: Also for inter-server copies, the client sends this OFFLOAD_REVOKE: Also for inter-server copies, the client sends this
operation to the source server to revoke permission to copy a file operation to the source server to revoke permission to copy a file
for the given user. (Section 13.4) for the given user. (Section 13.4)
COPY: Used by the client to request a file copy. (Section 13.1) COPY: Used by the client to request a file copy. (Section 13.1)
COPY_ABORT: Used by the client to abort an asynchronous file copy. OFFLOAD_ABORT: Used by the client to abort an asynchronous file
(Section 13.2) copy. (Section 13.2)
COPY_STATUS: Used by the client to poll the status of an OFFLOAD_STATUS: Used by the client to poll the status of an
asynchronous file copy. (Section 13.5) asynchronous file copy. (Section 13.5)
CB_COPY: Used by the destination server to report the results of an CB_COPY: Used by the destination server to report the results of an
asynchronous file copy to the client. (Section 14.2) asynchronous file copy to the client. (Section 14.2)
2.2.2. Intra-Server Copy 2.2.2. Locking the Files
Both the source and destination file may need to be locked to protect
the content during the copy operations. A client can achieve this by
a combination of OPEN and LOCK operations. I.e., either share or
byte range locks might be desired.
2.2.3. Intra-Server Copy
To copy a file on a single server, the client uses a COPY operation. To copy a file on a single server, the client uses a COPY operation.
The server may respond to the copy operation with the final results The server may respond to the copy operation with the final results
of the copy or it may perform the copy asynchronously and deliver the of the copy or it may perform the copy asynchronously and deliver the
results using a CB_COPY operation callback. If the copy is performed results using a CB_COPY operation callback. If the copy is performed
asynchronously, the client may poll the status of the copy using asynchronously, the client may poll the status of the copy using
COPY_STATUS or cancel the copy using COPY_ABORT. OFFLOAD_STATUS or cancel the copy using OFFLOAD_ABORT.
A synchronous intra-server copy is shown in Figure 1. In this A synchronous intra-server copy is shown in Figure 1. In this
example, the NFS server chooses to perform the copy synchronously. example, the NFS server chooses to perform the copy synchronously.
The copy operation is completed, either successfully or The copy operation is completed, either successfully or
unsuccessfully, before the server replies to the client's request. unsuccessfully, before the server replies to the client's request.
The server's reply contains the final result of the operation. The server's reply contains the final result of the operation.
Client Server Client Server
+ + + +
| | | |
|--- OPEN ---------------------------->| Client opens
|<------------------------------------/| the source file
| |
|--- OPEN ---------------------------->| Client opens
|<------------------------------------/| the destination file
| |
|--- COPY ---------------------------->| Client requests |--- COPY ---------------------------->| Client requests
|<------------------------------------/| a file copy |<------------------------------------/| a file copy
| | | |
|--- CLOSE --------------------------->| Client closes
|<------------------------------------/| the destination file
| |
|--- CLOSE --------------------------->| Client closes
|<------------------------------------/| the source file
| |
| | | |
Figure 1: A synchronous intra-server copy. Figure 1: A synchronous intra-server copy.
An asynchronous intra-server copy is shown in Figure 2. In this An asynchronous intra-server copy is shown in Figure 2. In this
example, the NFS server performs the copy asynchronously. The example, the NFS server performs the copy asynchronously. The
server's reply to the copy request indicates that the copy operation server's reply to the copy request indicates that the copy operation
was initiated and the final result will be delivered at a later time. was initiated and the final result will be delivered at a later time.
The server's reply also contains a copy stateid. The client may use The server's reply also contains a copy stateid. The client may use
this copy stateid to poll for status information (as shown) or to this copy stateid to poll for status information (as shown) or to
cancel the copy using a COPY_ABORT. When the server completes the cancel the copy using a OFFLOAD_ABORT. When the server completes the
copy, the server performs a callback to the client and reports the copy, the server performs a callback to the client and reports the
results. results.
Client Server Client Server
+ + + +
| | | |
|--- OPEN ---------------------------->| Client opens
|<------------------------------------/| the source file
| |
|--- OPEN ---------------------------->| Client opens
|<------------------------------------/| the destination file
| |
|--- COPY ---------------------------->| Client requests |--- COPY ---------------------------->| Client requests
|<------------------------------------/| a file copy |<------------------------------------/| a file copy
| | | |
| | | |
|--- COPY_STATUS --------------------->| Client may poll |--- OFFLOAD_STATUS ------------------>| Client may poll
|<------------------------------------/| for status |<------------------------------------/| for status
| | | |
| . | Multiple COPY_STATUS | . | Multiple OFFLOAD_STATUS
| . | operations may be sent. | . | operations may be sent.
| . | | . |
| | | |
|<-- CB_COPY --------------------------| Server reports results |<-- CB_COPY --------------------------| Server reports results
|\------------------------------------>| |\------------------------------------>|
| | | |
|--- CLOSE --------------------------->| Client closes
|<------------------------------------/| the destination file
| |
|--- CLOSE --------------------------->| Client closes
|<------------------------------------/| the source file
| |
| |
Figure 2: An asynchronous intra-server copy. Figure 2: An asynchronous intra-server copy.
2.2.3. Inter-Server Copy 2.2.4. Inter-Server Copy
A copy may also be performed between two servers. The copy protocol A copy may also be performed between two servers. The copy protocol
is designed to accommodate a variety of network topologies. As shown is designed to accommodate a variety of network topologies. As shown
in Figure 3, the client and servers may be connected by multiple in Figure 3, the client and servers may be connected by multiple
networks. In particular, the servers may be connected by a networks. In particular, the servers may be connected by a
specialized, high speed network (network 192.168.33.0/24 in the specialized, high speed network (network 192.168.33.0/24 in the
diagram) that does not include the client. The protocol allows the diagram) that does not include the client. The protocol allows the
client to setup the copy between the servers (over network client to setup the copy between the servers (over network
10.11.78.0/24 in the diagram) and for the servers to communicate on 10.11.78.0/24 in the diagram) and for the servers to communicate on
the high speed network if they choose to do so. the high speed network if they choose to do so.
skipping to change at page 11, line 8 skipping to change at page 12, line 8
the destination server chooses to perform the copy before responding the destination server chooses to perform the copy before responding
to the client's COPY request. to the client's COPY request.
An asynchronous copy is shown in Figure 5. In this case, the An asynchronous copy is shown in Figure 5. In this case, the
destination server chooses to respond to the client's COPY request destination server chooses to respond to the client's COPY request
immediately and then perform the copy asynchronously. immediately and then perform the copy asynchronously.
Client Source Destination Client Source Destination
+ + + + + +
| | | | | |
|--- OPEN --->| | Returns os1
|<------------------/| |
| | |
|--- COPY_NOTIFY --->| | |--- COPY_NOTIFY --->| |
|<------------------/| | |<------------------/| |
| | | | | |
|--- OPEN ---------------------------->| Returns os2
|<------------------------------------/|
| | | | | |
|--- COPY ---------------------------->| |--- COPY ---------------------------->|
| | | | | |
| | | | | |
| |<----- read -----| | |<----- read -----|
| |\--------------->| | |\--------------->|
| | | | | |
| | . | Multiple reads may | | . | Multiple reads may
| | . | be necessary | | . | be necessary
| | . | | | . |
| | | | | |
| | | | | |
|<------------------------------------/| Destination replies |<------------------------------------/| Destination replies
| | | to COPY | | | to COPY
| | |
|--- CLOSE --------------------------->| Release open state
|<------------------------------------/|
| | |
|--- CLOSE --->| | Release open state
|<------------------/| |
Figure 4: A synchronous inter-server copy. Figure 4: A synchronous inter-server copy.
Client Source Destination Client Source Destination
+ + + + + +
| | | | | |
|--- COPY_NOTIFY --->| | |--- OPEN --->| | Returns os1
|<------------------/| | |<------------------/| |
| | | | | |
| | | |--- LOCK --->| | Optional, could be done
|--- COPY ---------------------------->| |<------------------/| | with a share lock
|<------------------------------------/| | | |
| | | |--- COPY_NOTIFY --->| | Need to pass in
| | | |<------------------/| | os1 or lock state
| |<----- read -----| | | |
| |\--------------->| | | |
| | | | | |
| | . | Multiple reads may |--- OPEN ---------------------------->| Returns os2
| | . | be necessary |<------------------------------------/|
| | . | | | |
| | | |--- LOCK ---------------------------->| Optional ...
| | | |<------------------------------------/|
|--- COPY_STATUS --------------------->| Client may poll | | |
|<------------------------------------/| for status |--- COPY ---------------------------->| Need to pass in
| | | |<------------------------------------/| os2 or lock state
| | . | Multiple COPY_STATUS | | |
| | . | operations may be sent | | |
| | . | | |<----- read -----|
| | | | |\--------------->|
| | | | | |
| | | | | . | Multiple reads may
|<-- CB_COPY --------------------------| Destination reports | | . | be necessary
|\------------------------------------>| results | | . |
| | | | | |
| | |
|--- OFFLOAD_STATUS ------------------>| Client may poll
|<------------------------------------/| for status
| | |
| | . | Multiple OFFLOAD_STATUS
| | . | operations may be sent
| | . |
| | |
| | |
| | |
|<-- CB_COPY --------------------------| Destination reports
|\------------------------------------>| results
| | |
|--- LOCKU --------------------------->| Only if LOCK was done
|<------------------------------------/|
| | |
|--- CLOSE --------------------------->| Release open state
|<------------------------------------/|
| | |
|--- LOCKU --->| | Only if LOCK was done
|<------------------/| |
| | |
|--- CLOSE --->| | Release open state
|<------------------/| |
| | |
Figure 5: An asynchronous inter-server copy. Figure 5: An asynchronous inter-server copy.
2.2.4. Server-to-Server Copy Protocol 2.2.5. Server-to-Server Copy Protocol
The source server and destination server are not required to use a The source server and destination server are not required to use a
specific protocol to transfer the file data. The choice of what specific protocol to transfer the file data. The choice of what
protocol to use is ultimately the destination server's decision. protocol to use is ultimately the destination server's decision.
2.2.4.1. Using NFSv4.x as a Server-to-Server Copy Protocol 2.2.5.1. Using NFSv4.x as a Server-to-Server Copy Protocol
The destination server MAY use standard NFSv4.x (where x >= 1) to The destination server MAY use standard NFSv4.x (where x >= 1) to
read the data from the source server. If NFSv4.x is used for the read the data from the source server. If NFSv4.x is used for the
server-to-server copy protocol, the destination server can use the server-to-server copy protocol, the destination server can use the
filehandle contained in the COPY request with standard NFSv4.x filehandle contained in the COPY request with standard NFSv4.x
operations to read data from the source server. Specifically, the operations to read data from the source server. Specifically, the
destination server may use the NFSv4.x OPEN operation's CLAIM_FH destination server may use the NFSv4.x OPEN operation's CLAIM_FH
facility to open the file being copied and obtain an open stateid. facility to open the file being copied and obtain an open stateid.
Using the stateid, the destination server may then use NFSv4.x READ Using the stateid, the destination server may then use NFSv4.x READ
operations to read the file. operations to read the file.
2.2.4.2. Using an alternative Server-to-Server Copy Protocol 2.2.5.2. Using an alternative Server-to-Server Copy Protocol
In a homogeneous environment, the source and destination servers In a homogeneous environment, the source and destination servers
might be able to perform the file copy extremely efficiently using might be able to perform the file copy extremely efficiently using
specialized protocols. For example the source and destination specialized protocols. For example the source and destination
servers might be two nodes sharing a common file system format for servers might be two nodes sharing a common file system format for
the source and destination file systems. Thus the source and the source and destination file systems. Thus the source and
destination are in an ideal position to efficiently render the image destination are in an ideal position to efficiently render the image
of the source file to the destination file by replicating the file of the source file to the destination file by replicating the file
system formats at the block level. Another possibility is that the system formats at the block level. Another possibility is that the
source and destination might be two nodes sharing a common storage source and destination might be two nodes sharing a common storage
skipping to change at page 14, line 13 skipping to change at page 15, line 25
this are given in Section 2.4.1.2.4 and Section 2.4.1.4. this are given in Section 2.4.1.2.4 and Section 2.4.1.4.
2.3. Requirements for Operations 2.3. Requirements for Operations
The implementation of server-side copy is OPTIONAL by the client and The implementation of server-side copy is OPTIONAL by the client and
the server. However, in order to successfully copy a file, some the server. However, in order to successfully copy a file, some
operations MUST be supported by the client and/or server. operations MUST be supported by the client and/or server.
If a client desires an intra-server file copy, then it MUST support If a client desires an intra-server file copy, then it MUST support
the COPY and CB_COPY operations. If COPY returns a stateid, then the the COPY and CB_COPY operations. If COPY returns a stateid, then the
client MAY use the COPY_ABORT and COPY_STATUS operations. client MAY use the OFFLOAD_ABORT and OFFLOAD_STATUS operations.
If a client desires an inter-server file copy, then it MUST support If a client desires an inter-server file copy, then it MUST support
the COPY, COPY_NOTICE, and CB_COPY operations, and MAY use the the COPY, COPY_NOTICE, and CB_COPY operations, and MAY use the
COPY_REVOKE operation. If COPY returns a stateid, then the client OFFLOAD_REVOKE operation. If COPY returns a stateid, then the client
MAY use the COPY_ABORT and COPY_STATUS operations. MAY use the OFFLOAD_ABORT and OFFLOAD_STATUS operations.
If a server supports intra-server copy, then the server MUST support If a server supports intra-server copy, then the server MUST support
the COPY operation. If a server's COPY operation returns a stateid, the COPY operation. If a server's COPY operation returns a stateid,
then the server MUST also support these operations: CB_COPY, then the server MUST also support these operations: CB_COPY,
COPY_ABORT, and COPY_STATUS. OFFLOAD_ABORT, and OFFLOAD_STATUS.
If a source server supports inter-server copy, then the source server If a source server supports inter-server copy, then the source server
MUST support all these operations: COPY_NOTIFY and COPY_REVOKE. If a MUST support all these operations: COPY_NOTIFY and OFFLOAD_REVOKE.
destination server supports inter-server copy, then the destination If a destination server supports inter-server copy, then the
server MUST support the COPY operation. If a destination server's destination server MUST support the COPY operation. If a destination
COPY operation returns a stateid, then the destination server MUST server's COPY operation returns a stateid, then the destination
also support these operations: CB_COPY, COPY_ABORT, COPY_NOTIFY, server MUST also support these operations: CB_COPY, OFFLOAD_ABORT,
COPY_REVOKE, and COPY_STATUS. COPY_NOTIFY, OFFLOAD_REVOKE, and OFFLOAD_STATUS.
Each operation is performed in the context of the user identified by Each operation is performed in the context of the user identified by
the ONC RPC credential of its containing COMPOUND or CB_COMPOUND the ONC RPC credential of its containing COMPOUND or CB_COMPOUND
request. For example, a COPY_ABORT operation issued by a given user request. For example, a OFFLOAD_ABORT operation issued by a given
indicates that a specified COPY operation initiated by the same user user indicates that a specified COPY operation initiated by the same
be canceled. Therefore a COPY_ABORT MUST NOT interfere with a copy user be canceled. Therefore a OFFLOAD_ABORT MUST NOT interfere with
of the same file initiated by another user. a copy of the same file initiated by another user.
An NFS server MAY allow an administrative user to monitor or cancel An NFS server MAY allow an administrative user to monitor or cancel
copy operations using an implementation specific interface. copy operations using an implementation specific interface.
2.3.1. netloc4 - Network Locations 2.3.1. netloc4 - Network Locations
The server-side copy operations specify network locations using the The server-side copy operations specify network locations using the
netloc4 data type shown below: netloc4 data type shown below:
enum netloc_type4 { enum netloc_type4 {
skipping to change at page 15, line 35 skipping to change at page 16, line 43
When netloc4 values are used for an inter-server copy as shown in When netloc4 values are used for an inter-server copy as shown in
Figure 3, their values may be evaluated on the source server, Figure 3, their values may be evaluated on the source server,
destination server, and client. The network environment in which destination server, and client. The network environment in which
these systems operate should be configured so that the netloc4 values these systems operate should be configured so that the netloc4 values
are interpreted as intended on each system. are interpreted as intended on each system.
2.3.2. Copy Offload Stateids 2.3.2. Copy Offload Stateids
A server may perform a copy offload operation asynchronously. An A server may perform a copy offload operation asynchronously. An
asynchronous copy is tracked using a copy offload stateid. Copy asynchronous copy is tracked using a copy offload stateid. Copy
offload stateids are included in the COPY, COPY_ABORT, COPY_STATUS, offload stateids are included in the COPY, OFFLOAD_ABORT,
and CB_COPY operations. OFFLOAD_STATUS, and CB_COPY operations.
Section 8.2.4 of [2] specifies that stateids are valid until either Section 8.2.4 of [2] specifies that stateids are valid until either
(A) the client or server restart or (B) the client returns the (A) the client or server restart or (B) the client returns the
resource. resource.
A copy offload stateid will be valid until either (A) the client or A copy offload stateid will be valid until either (A) the client or
server restarts or (B) the client returns the resource by issuing a server restarts or (B) the client returns the resource by issuing a
COPY_ABORT operation or the client replies to a CB_COPY operation. OFFLOAD_ABORT operation or the client replies to a CB_COPY operation.
A copy offload stateid's seqid MUST NOT be 0. In the context of a A copy offload stateid's seqid MUST NOT be 0. In the context of a
copy offload operation, it is ambiguous to indicate the most recent copy offload operation, it is ambiguous to indicate the most recent
copy offload operation using a stateid with seqid of 0. Therefore a copy offload operation using a stateid with seqid of 0. Therefore a
copy offload stateid with seqid of 0 MUST be considered invalid. copy offload stateid with seqid of 0 MUST be considered invalid.
2.4. Security Considerations 2.4. Security Considerations
The security considerations pertaining to NFSv4 [10] apply to this The security considerations pertaining to NFSv4 [10] apply to this
chapter. chapter.
skipping to change at page 21, line 37 skipping to change at page 22, line 37
When the client sends a COPY request to the destination server, it When the client sends a COPY request to the destination server, it
uses the privileged "copy_to_auth" RPCSEC_GSSv3 handle. uses the privileged "copy_to_auth" RPCSEC_GSSv3 handle.
ca_source_server in COPY MUST be the same as the name of the source ca_source_server in COPY MUST be the same as the name of the source
server specified in copy_to_auth_priv. Otherwise, COPY will fail server specified in copy_to_auth_priv. Otherwise, COPY will fail
with NFS4ERR_ACCESS. The destination server verifies that the with NFS4ERR_ACCESS. The destination server verifies that the
privilege <"copy_to_auth", user id, source> exists, and annotates it privilege <"copy_to_auth", user id, source> exists, and annotates it
with the source and destination filehandles. If the client has with the source and destination filehandles. If the client has
failed to establish the "copy_to_auth" policy it will reject the failed to establish the "copy_to_auth" policy it will reject the
request with NFS4ERR_PARTNER_NO_AUTH. request with NFS4ERR_PARTNER_NO_AUTH.
If the client sends a COPY_REVOKE to the source server to rescind the If the client sends a OFFLOAD_REVOKE to the source server to rescind
destination server's copy privilege, it uses the privileged the destination server's copy privilege, it uses the privileged
"copy_from_auth" RPCSEC_GSSv3 handle and the cra_destination_server "copy_from_auth" RPCSEC_GSSv3 handle and the cra_destination_server
in COPY_REVOKE MUST be the same as the name of the destination server in OFFLOAD_REVOKE MUST be the same as the name of the destination
specified in copy_from_auth_priv. The source server will then delete server specified in copy_from_auth_priv. The source server will then
the <"copy_from_auth", user id, destination> privilege and fail any delete the <"copy_from_auth", user id, destination> privilege and
subsequent copy requests sent under the auspices of this privilege fail any subsequent copy requests sent under the auspices of this
from the destination server. privilege from the destination server.
2.4.1.2.3. Securing ONC RPC Server-to-Server Copy Protocols 2.4.1.2.3. Securing ONC RPC Server-to-Server Copy Protocols
After a destination server has a "copy_to_auth" privilege established After a destination server has a "copy_to_auth" privilege established
on it, and it receives a COPY request, if it knows it will use an ONC on it, and it receives a COPY request, if it knows it will use an ONC
RPC protocol to copy data, it will establish a "copy_confirm_auth" RPC protocol to copy data, it will establish a "copy_confirm_auth"
privilege on the source server, using nfs@<destination> as the privilege on the source server, using nfs@<destination> as the
initiator principal, and nfs@<source> as the target principal. initiator principal, and nfs@<source> as the target principal.
The value of the field ccap_shared_secret_mic is a GSS_VerifyMIC() of The value of the field ccap_shared_secret_mic is a GSS_VerifyMIC() of
skipping to change at page 28, line 5 skipping to change at page 29, line 5
that are allocated to the given file that would be freed on its that are allocated to the given file that would be freed on its
deletion. In the example, both A and B would report space_freed as 4 deletion. In the example, both A and B would report space_freed as 4
* BLOCK_SIZE and space_used as 10 * BLOCK_SIZE. If A is deleted, B * BLOCK_SIZE and space_used as 10 * BLOCK_SIZE. If A is deleted, B
will report space_freed as 10 * BLOCK_SIZE as the deletion of B would will report space_freed as 10 * BLOCK_SIZE as the deletion of B would
result in the deallocation of all 10 blocks. result in the deallocation of all 10 blocks.
The addition of this problem doesn't solve the problem of space being The addition of this problem doesn't solve the problem of space being
over-reported. However, over-reporting is better than under- over-reported. However, over-reporting is better than under-
reporting. reporting.
6. Application Data Block Support 6. Application Data Hole Support
At the OS level, files are contained on disk blocks. Applications At the OS level, files are contained on disk blocks. Applications
are also free to impose structure on the data contained in a file and are also free to impose structure on the data contained in a file and
we can define an Application Data Block (ADB) to be such a structure. we can define an Application Data Block (ADB) to be such a structure.
From the application's viewpoint, it only wants to handle ADBs and From the application's viewpoint, it only wants to handle ADBs and
not raw bytes (see [16]). An ADB is typically comprised of two not raw bytes (see [16]). An ADB is typically comprised of two
sections: a header and data. The header describes the sections: a header and data. The header describes the
characteristics of the block and can provide a means to detect characteristics of the block and can provide a means to detect
corruption in the data payload. The data section is typically corruption in the data payload. The data section is typically
initialized to all zeros. initialized to all zeros.
The format of the header is application specific, but there are two The format of the header is application specific, but there are two
main components typically encountered: main components typically encountered:
1. An ADB Number (ADBN), which allows the application to determine 1. A logical block number which allows the application to determine
which data block is being referenced. The ADBN is a logical which data block is being referenced. This is useful when the
block number and is useful when the client is not storing the client is not storing the blocks in contiguous memory.
blocks in contiguous memory.
2. Fields to describe the state of the ADB and a means to detect 2. Fields to describe the state of the ADB and a means to detect
block corruption. For both pieces of data, a useful property is block corruption. For both pieces of data, a useful property is
that allowed values be unique in that if passed across the that allowed values be unique in that if passed across the
network, corruption due to translation between big and little network, corruption due to translation between big and little
endian architectures are detectable. For example, 0xF0DEDEF0 has endian architectures are detectable. For example, 0xF0DEDEF0 has
the same bit pattern in both architectures. the same bit pattern in both architectures.
Applications already impose structures on files [16] and detect Applications already impose structures on files [16] and detect
corruption in data blocks [17]. What they are not able to do is corruption in data blocks [17]. What they are not able to do is
efficiently transfer and store ADBs. To initialize a file with ADBs, efficiently transfer and store ADBs. To initialize a file with ADBs,
the client must send the full ADB to the server and that must be the client must send the full ADB to the server and that must be
stored on the server. When the application is initializing a file to stored on the server.
have the ADB structure, it could compress the ADBs to just the
information to necessary to later reconstruct the header portion of
the ADB when the contents are read back. Using sparse file
techniques, the disk blocks described by would not be allocated.
Unlike sparse file techniques, there would be a small cost to store
the compressed header data.
In this section, we are going to define a generic framework for an In this section, we are going to define an Application Data Hole
ADB, present one approach to detecting corruption in a given ADB (ADH), which is a generic framework for transfering the ADB, present
implementation, and describe the model for how the client and server one approach to detecting corruption in a given ADH implementation,
can support efficient initialization of ADBs, reading of ADB holes, and describe the model for how the client and server can support
punching holes in ADBs, and space reservation. efficient initialization of ADHs, reading of ADH holes, punching ADH
holes in a file, and space reservation. We define the ADHN to be the
Application Data Hole Number, which is the logical block number
discussed earlier.
6.1. Generic Framework 6.1. Generic Framework
We want the representation of the ADB to be flexible enough to We want the representation of the ADH to be flexible enough to
support many different applications. The most basic approach is no support many different applications. The most basic approach is no
imposition of a block at all, which means we are working with the raw imposition of a block at all, which means we are working with the raw
bytes. Such an approach would be useful for storing holes, punching bytes. Such an approach would be useful for storing holes, punching
holes, etc. In more complex deployments, a server might be holes, etc. In more complex deployments, a server might be
supporting multiple applications, each with their own definition of supporting multiple applications, each with their own definition of
the ADB. One might store the ADBN at the start of the block and then the ADH. One might store the ADHN at the start of the block and then
have a guard pattern to detect corruption [18]. The next might store have a guard pattern to detect corruption [18]. The next might store
the ADBN at an offset of 100 bytes within the block and have no guard the ADHN at an offset of 100 bytes within the block and have no guard
pattern at all. I.e., existing applications might already have well pattern at all, i.e., existing applications might already have well
defined formats for their data blocks. defined formats for their data blocks.
The guard pattern can be used to represent the state of the block, to The guard pattern can be used to represent the state of the block, to
protect against corruption, or both. Again, it needs to be able to protect against corruption, or both. Again, it needs to be able to
be placed anywhere within the ADB. be placed anywhere within the ADH.
We need to be able to represent the starting offset of the block and We need to be able to represent the starting offset of the block and
the size of the block. Note that nothing prevents the application the size of the block. Note that nothing prevents the application
from defining different sized blocks in a file. from defining different sized blocks in a file.
6.1.1. Data Block Representation 6.1.1. Data Hole Representation
struct app_data_block4 { struct app_data_hole4 {
offset4 adb_offset; offset4 adh_offset;
length4 adb_block_size; length4 adh_block_size;
length4 adb_block_count; length4 adh_block_count;
length4 adb_reloff_blocknum; length4 adh_reloff_blocknum;
count4 adb_block_num; count4 adh_block_num;
length4 adb_reloff_pattern; length4 adh_reloff_pattern;
opaque adb_pattern<>; opaque adh_pattern<>;
}; };
The app_data_block4 structure captures the abstraction presented for The app_data_hole4 structure captures the abstraction presented for
the ADB. The additional fields present are to allow the transmission the ADH. The additional fields present are to allow the transmission
of adb_block_count ADBs at one time. We also use adb_block_num to of adh_block_count ADHs at one time. We also use adh_block_num to
convey the ADBN of the first block in the sequence. Each ADB will convey the ADHN of the first block in the sequence. Each ADH will
contain the same adb_pattern string. contain the same adh_pattern string.
As both adb_block_num and adb_pattern are optional, if either As both adh_block_num and adh_pattern are optional, if either
adb_reloff_pattern or adb_reloff_blocknum is set to NFS4_UINT64_MAX, adh_reloff_pattern or adh_reloff_blocknum is set to NFS4_UINT64_MAX,
then the corresponding field is not set in any of the ADB. then the corresponding field is not set in any of the ADH.
6.1.2. Data Content 6.1.2. Data Content
/* /*
* Use an enum such that we can extend new types. * Use an enum such that we can extend new types.
*/ */
enum data_content4 { enum data_content4 {
NFS4_CONTENT_DATA = 0, NFS4_CONTENT_DATA = 0,
NFS4_CONTENT_APP_BLOCK = 1, NFS4_CONTENT_APP_DATA_HOLE = 1,
NFS4_CONTENT_HOLE = 2 NFS4_CONTENT_HOLE = 2
}; };
New operations might need to differentiate between wanting to access New operations might need to differentiate between wanting to access
data versus an ADB. Also, future minor versions might want to data versus an ADH. Also, future minor versions might want to
introduce new data formats. This enumeration allows that to occur. introduce new data formats. This enumeration allows that to occur.
6.2. pNFS Considerations 6.2. An Example of Detecting Corruption
While this document does not mandate how sparse ADBs are recorded on
the server, it does make the assumption that such information is not
in the file. I.e., the information is metadata. As such, the
INITIALIZE operation is defined to be not supported by the DS - it
must be issued to the MDS. But since the client must not assume a
priori whether a read is sparse or not, the READ_PLUS operation MUST
be supported by both the DS and the MDS. I.e., the client might
impose on the MDS to asynchronously read the data from the DS.
Furthermore, each DS MUST not report to a client a sparse ADB which
belongs to another DS. One implication of this requirement is that
the app_data_block4's adb_block_size MUST be either be the stripe
width or the stripe width must be an even multiple of it. The second
implication here is that the DS must be able to use the Control
Protocol to determine from the MDS where the sparse ADBs occur.
6.3. An Example of Detecting Corruption
In this section, we define an ADB format in which corruption can be In this section, we define an ADH format in which corruption can be
detected. Note that this is just one possible format and means to detected. Note that this is just one possible format and means to
detect corruption. detect corruption.
Consider a very basic implementation of an operating system's disk Consider a very basic implementation of an operating system's disk
blocks. A block is either data or it is an indirect block which blocks. A block is either data or it is an indirect block which
allows for files to be larger than one block. It is desired to be allows for files to be larger than one block. It is desired to be
able to initialize a block. Lastly, to quickly unlink a file, a able to initialize a block. Lastly, to quickly unlink a file, a
block can be marked invalid. The contents remain intact - which block can be marked invalid. The contents remain intact - which
would enable this OS application to undelete a file. would enable this OS application to undelete a file.
The application defines 4k sized data blocks, with an 8 byte block The application defines 4k sized data blocks, with an 8 byte block
counter occurring at offset 0 in the block, and with the guard counter occurring at offset 0 in the block, and with the guard
pattern occurring at offset 8 inside the block. Furthermore, the pattern occurring at offset 8 inside the block. Furthermore, the
guard pattern can take one of four states: guard pattern can take one of four states:
0xfeedface - This is the FREE state and indicates that the ADB 0xfeedface - This is the FREE state and indicates that the ADH
format has been applied. format has been applied.
0xcafedead - This is the DATA state and indicates that real data 0xcafedead - This is the DATA state and indicates that real data
has been written to this block. has been written to this block.
0xe4e5c001 - This is the INDIRECT state and indicates that the 0xe4e5c001 - This is the INDIRECT state and indicates that the
block contains block counter numbers that are chained off of this block contains block counter numbers that are chained off of this
block. block.
0xba1ed4a3 - This is the INVALID state and indicates that the block 0xba1ed4a3 - This is the INVALID state and indicates that the block
contains data whose contents are garbage. contains data whose contents are garbage.
Finally, it also defines an 8 byte checksum [19] starting at byte 16 Finally, it also defines an 8 byte checksum [19] starting at byte 16
which applies to the remaining contents of the block. If the state which applies to the remaining contents of the block. If the state
is FREE, then that checksum is trivially zero. As such, the is FREE, then that checksum is trivially zero. As such, the
application has no need to transfer the checksum implicitly inside application has no need to transfer the checksum implicitly inside
the ADB - it need not make the transfer layer aware of the fact that the ADH - it need not make the transfer layer aware of the fact that
there is a checksum (see [17] for an example of checksums used to there is a checksum (see [17] for an example of checksums used to
detect corruption in application data blocks). detect corruption in application data blocks).
Corruption in each ADB can be detected thusly: Corruption in each ADH can be detected thusly:
o If the guard pattern is anything other than one of the allowed o If the guard pattern is anything other than one of the allowed
values, including all zeros. values, including all zeros.
o If the guard pattern is FREE and any other byte in the remainder o If the guard pattern is FREE and any other byte in the remainder
of the ADB is anything other than zero. of the ADH is anything other than zero.
o If the guard pattern is anything other than FREE, then if the o If the guard pattern is anything other than FREE, then if the
stored checksum does not match the computed checksum. stored checksum does not match the computed checksum.
o If the guard pattern is INDIRECT and one of the stored indirect o If the guard pattern is INDIRECT and one of the stored indirect
block numbers has a value greater than the number of ADBs in the block numbers has a value greater than the number of ADHs in the
file. file.
o If the guard pattern is INDIRECT and one of the stored indirect o If the guard pattern is INDIRECT and one of the stored indirect
block numbers is a duplicate of another stored indirect block block numbers is a duplicate of another stored indirect block
number. number.
As can be seen, the application can detect errors based on the As can be seen, the application can detect errors based on the
combination of the guard pattern state and the checksum. But also, combination of the guard pattern state and the checksum. But also,
the application can detect corruption based on the state and the the application can detect corruption based on the state and the
contents of the ADB. This last point is important in validating the contents of the ADH. This last point is important in validating the
minimum amount of data we incorporated into our generic framework. minimum amount of data we incorporated into our generic framework.
I.e., the guard pattern is sufficient in allowing applications to I.e., the guard pattern is sufficient in allowing applications to
design their own corruption detection. design their own corruption detection.
Finally, it is important to note that none of these corruption checks Finally, it is important to note that none of these corruption checks
occur in the transport layer. The server and client components are occur in the transport layer. The server and client components are
totally unaware of the file format and might report everything as totally unaware of the file format and might report everything as
being transferred correctly even in the case the application detects being transferred correctly even in the case the application detects
corruption. corruption.
6.4. Example of READ_PLUS 6.3. Example of READ_PLUS
The hypothetical application presented in Section 6.3 can be used to The hypothetical application presented in Section 6.2 can be used to
illustrate how READ_PLUS would return an array of results. A file is illustrate how READ_PLUS would return an array of results. A file is
created and initialized with 100 4k ADBs in the FREE state: created and initialized with 100 4k ADHs in the FREE state:
INITIALIZE {0, 4k, 100, 0, 0, 8, 0xfeedface} INITIALIZE {0, 4k, 100, 0, 0, 8, 0xfeedface}
Further, assume the application writes a single ADB at 16k, changing Further, assume the application writes a single ADH at 16k, changing
the guard pattern to 0xcafedead, we would then have in memory: the guard pattern to 0xcafedead, we would then have in memory:
0 -> (16k - 1) : 4k, 4, 0, 0, 8, 0xfeedface 0 -> (16k - 1) : 4k, 4, 0, 0, 8, 0xfeedface
16k -> (20k - 1) : 00 00 00 05 ca fe de ad XX XX ... XX XX 16k -> (20k - 1) : 00 00 00 05 ca fe de ad XX XX ... XX XX
20k -> 400k : 4k, 95, 0, 6, 0xfeedface 20k -> 400k : 4k, 95, 0, 6, 0xfeedface
And when the client did a READ_PLUS of 64k at the start of the file, And when the client did a READ_PLUS of 64k at the start of the file,
it would get back a result of an ADB, some data, and a final ADB: it would get back a result of an ADH, some data, and a final ADH:
ADB {0, 4, 0, 0, 8, 0xfeedface} ADH {0, 4, 0, 0, 8, 0xfeedface}
data 4k data 4k
ADB {20k, 4k, 59, 0, 6, 0xfeedface} ADH {20k, 4k, 59, 0, 6, 0xfeedface}
6.5. Zero Filled Holes
As applications are free to define the structure of an ADB, it is
trivial to define an ADB which supports zero filled holes. Such a
case would encompass the traditional definitions of a sparse file and
hole punching. For example, to punch a 64k hole, starting at 100M,
into an existing file which has no ADB structure:
INITIALIZE {100M, 64k, 1, NFS4_UINT64_MAX,
0, NFS4_UINT64_MAX, 0x0}
7. Labeled NFS 7. Labeled NFS
7.1. Introduction 7.1. Introduction
Access control models such as Unix permissions or Access Control Access control models such as Unix permissions or Access Control
Lists are commonly referred to as Discretionary Access Control (DAC) Lists are commonly referred to as Discretionary Access Control (DAC)
models. These systems base their access decisions on user identity models. These systems base their access decisions on user identity
and resource ownership. In contrast Mandatory Access Control (MAC) and resource ownership. In contrast Mandatory Access Control (MAC)
models base their access control decisions on the label on the models base their access control decisions on the label on the
subject (usually a process) and the object it wishes to access [7]. subject (usually a process) and the object it wishes to access [7].
These labels may contain user identity information but usually These labels may contain user identity information but usually
contain additional information. In DAC systems users are free to contain additional information. In DAC systems users are free to
skipping to change at page 47, line 40 skipping to change at page 47, line 47
follows: follows:
pNFS Parallel NFS pNFS Parallel NFS
FDELG File Delegations FDELG File Delegations
DDELG Directory Delegations DDELG Directory Delegations
COPY Server Side Copy COPY Server Side Copy
ADB Application Data Blocks ADH Application Data Holes
Operations Operations
+----------------------+--------------------+-----------------------+ +----------------------+--------------------+-----------------------+
| Operation | REQ, REC, OPT, or | Feature (REQ, REC, or | | Operation | REQ, REC, OPT, or | Feature (REQ, REC, or |
| | MNI | OPT) | | | MNI | OPT) |
+----------------------+--------------------+-----------------------+ +----------------------+--------------------+-----------------------+
| ACCESS | REQ | | | ACCESS | REQ | |
| BACKCHANNEL_CTL | REQ | | | BACKCHANNEL_CTL | REQ | |
| BIND_CONN_TO_SESSION | REQ | | | BIND_CONN_TO_SESSION | REQ | |
| CLOSE | REQ | | | CLOSE | REQ | |
| COMMIT | REQ | | | COMMIT | REQ | |
| COPY | OPT | COPY (REQ) | | COPY | OPT | COPY (REQ) |
| COPY_ABORT | OPT | COPY (REQ) | | OFFLOAD_ABORT | OPT | COPY (REQ) |
| COPY_NOTIFY | OPT | COPY (REQ) | | COPY_NOTIFY | OPT | COPY (REQ) |
| COPY_REVOKE | OPT | COPY (REQ) | | OFFLOAD_REVOKE | OPT | COPY (REQ) |
| COPY_STATUS | OPT | COPY (REQ) | | OFFLOAD_STATUS | OPT | COPY (REQ) |
| CREATE | REQ | | | CREATE | REQ | |
| CREATE_SESSION | REQ | | | CREATE_SESSION | REQ | |
| DELEGPURGE | OPT | FDELG (REQ) | | DELEGPURGE | OPT | FDELG (REQ) |
| DELEGRETURN | OPT | FDELG, DDELG, pNFS | | DELEGRETURN | OPT | FDELG, DDELG, pNFS |
| | | (REQ) | | | | (REQ) |
| DESTROY_CLIENTID | REQ | | | DESTROY_CLIENTID | REQ | |
| DESTROY_SESSION | REQ | | | DESTROY_SESSION | REQ | |
| EXCHANGE_ID | REQ | | | EXCHANGE_ID | REQ | |
| FREE_STATEID | REQ | | | FREE_STATEID | REQ | |
| GETATTR | REQ | | | GETATTR | REQ | |
| GETDEVICEINFO | OPT | pNFS (REQ) | | GETDEVICEINFO | OPT | pNFS (REQ) |
| GETDEVICELIST | OPT | pNFS (OPT) | | GETDEVICELIST | OPT | pNFS (OPT) |
| GETFH | REQ | | | GETFH | REQ | |
| INITIALIZE | OPT | ADB (REQ) | | INITIALIZE | OPT | ADH (REQ) |
| GET_DIR_DELEGATION | OPT | DDELG (REQ) | | GET_DIR_DELEGATION | OPT | DDELG (REQ) |
| LAYOUTCOMMIT | OPT | pNFS (REQ) | | LAYOUTCOMMIT | OPT | pNFS (REQ) |
| LAYOUTGET | OPT | pNFS (REQ) | | LAYOUTGET | OPT | pNFS (REQ) |
| LAYOUTRETURN | OPT | pNFS (REQ) | | LAYOUTRETURN | OPT | pNFS (REQ) |
| LINK | OPT | | | LINK | OPT | |
| LOCK | REQ | | | LOCK | REQ | |
| LOCKT | REQ | | | LOCKT | REQ | |
| LOCKU | REQ | | | LOCKU | REQ | |
| LOOKUP | REQ | | | LOOKUP | REQ | |
| LOOKUPP | REQ | | | LOOKUPP | REQ | |
skipping to change at page 48, line 45 skipping to change at page 49, line 5
| OPEN | REQ | | | OPEN | REQ | |
| OPENATTR | OPT | | | OPENATTR | OPT | |
| OPEN_CONFIRM | MNI | | | OPEN_CONFIRM | MNI | |
| OPEN_DOWNGRADE | REQ | | | OPEN_DOWNGRADE | REQ | |
| PUTFH | REQ | | | PUTFH | REQ | |
| PUTPUBFH | REQ | | | PUTPUBFH | REQ | |
| PUTROOTFH | REQ | | | PUTROOTFH | REQ | |
| READ | OBS | | | READ | OBS | |
| READDIR | REQ | | | READDIR | REQ | |
| READLINK | OPT | | | READLINK | OPT | |
| READ_PLUS | OPT | ADB (REQ) | | READ_PLUS | OPT | ADH (REQ) |
| RECLAIM_COMPLETE | REQ | | | RECLAIM_COMPLETE | REQ | |
| RELEASE_LOCKOWNER | MNI | | | RELEASE_LOCKOWNER | MNI | |
| REMOVE | REQ | | | REMOVE | REQ | |
| RENAME | REQ | | | RENAME | REQ | |
| RENEW | MNI | | | RENEW | MNI | |
| RESTOREFH | REQ | | | RESTOREFH | REQ | |
| SAVEFH | REQ | | | SAVEFH | REQ | |
| SECINFO | REQ | | | SECINFO | REQ | |
| SECINFO_NO_NAME | REC | pNFS file layout | | SECINFO_NO_NAME | REC | pNFS file layout |
| | | (REQ) | | | | (REQ) |
skipping to change at page 50, line 4 skipping to change at page 50, line 8
| CB_RECALLABLE_OBJ_AVAIL | OPT | DDELG, pNFS (REQ) | | CB_RECALLABLE_OBJ_AVAIL | OPT | DDELG, pNFS (REQ) |
| CB_SEQUENCE | OPT | FDELG, DDELG, pNFS | | CB_SEQUENCE | OPT | FDELG, DDELG, pNFS |
| | | (REQ) | | | | (REQ) |
| CB_WANTS_CANCELLED | OPT | FDELG, DDELG, pNFS | | CB_WANTS_CANCELLED | OPT | FDELG, DDELG, pNFS |
| | | (REQ) | | | | (REQ) |
+-------------------------+-------------------+---------------------+ +-------------------------+-------------------+---------------------+
13. NFSv4.2 Operations 13. NFSv4.2 Operations
13.1. Operation 59: COPY - Initiate a server-side copy 13.1. Operation 59: COPY - Initiate a server-side copy
13.1.1. ARGUMENT 13.1.1. ARGUMENT
const COPY4_GUARDED = 0x00000001; const COPY4_GUARDED = 0x00000001;
const COPY4_METADATA = 0x00000002; const COPY4_METADATA = 0x00000002;
struct COPY4args { struct COPY4args {
/* SAVED_FH: source file */ /* SAVED_FH: source file */
/* CURRENT_FH: destination file or */ /* CURRENT_FH: destination file or */
/* directory */ /* directory */
stateid4 ca_src_stateid;
stateid4 ca_dst_stateid;
offset4 ca_src_offset; offset4 ca_src_offset;
offset4 ca_dst_offset; offset4 ca_dst_offset;
length4 ca_count; length4 ca_count;
uint32_t ca_flags; uint32_t ca_flags;
component4 ca_destination; component4 ca_destination;
netloc4 ca_source_server<>; netloc4 ca_source_server<>;
}; };
13.1.2. RESULT 13.1.2. RESULT
skipping to change at page 51, line 19 skipping to change at page 51, line 24
either PUTFH or SAVEFH checked the validity of the filehandle, the either PUTFH or SAVEFH checked the validity of the filehandle, the
operation would likely fail and return NFS4ERR_STALE. operation would likely fail and return NFS4ERR_STALE.
If a server supports the server-to-server COPY feature, a PUTFH If a server supports the server-to-server COPY feature, a PUTFH
followed by a SAVEFH MUST NOT return NFS4ERR_STALE for either followed by a SAVEFH MUST NOT return NFS4ERR_STALE for either
operation. These restrictions do not pose substantial difficulties operation. These restrictions do not pose substantial difficulties
for servers. The CURRENT_FH and SAVED_FH may be validated in the for servers. The CURRENT_FH and SAVED_FH may be validated in the
context of the operation referencing them and an NFS4ERR_STALE error context of the operation referencing them and an NFS4ERR_STALE error
returned for an invalid file handle at that point. returned for an invalid file handle at that point.
For an intra-server copy, both the ca_src_stateid and ca_dst_stateid
MUST refer to either open or locking states provided earlier by the
server. If either stateid is invalid, then the operation MUST fail.
If the request is for a inter-server copy, then the ca_src_stateid
can be ignored. If ca_dst_stateid is invalid, then the operation
MUST fail.
The CURRENT_FH and ca_destination together specify the destination of The CURRENT_FH and ca_destination together specify the destination of
the copy operation. If ca_destination is of 0 (zero) length, then the copy operation. If ca_destination is of 0 (zero) length, then
CURRENT_FH specifies the target file. In this case, CURRENT_FH MUST CURRENT_FH specifies the target file. In this case, CURRENT_FH MUST
be a regular file and not a directory. If ca_destination is not of 0 be a regular file and not a directory. If ca_destination is not of 0
(zero) length, the ca_destination argument specifies the file name to (zero) length, the ca_destination argument specifies the file name to
which the data will be copied within the directory identified by which the data will be copied within the directory identified by
CURRENT_FH. In this case, CURRENT_FH MUST be a directory and not a CURRENT_FH. In this case, CURRENT_FH MUST be a directory and not a
regular file. regular file.
If the file named by ca_destination does not exist and the operation If the file named by ca_destination does not exist and the operation
skipping to change at page 52, line 30 skipping to change at page 52, line 43
If the ca_source_server list is specified, then this is an inter- If the ca_source_server list is specified, then this is an inter-
server copy operation and the source file is on a remote server. The server copy operation and the source file is on a remote server. The
client is expected to have previously issued a successful COPY_NOTIFY client is expected to have previously issued a successful COPY_NOTIFY
request to the remote source server. The ca_source_server list MUST request to the remote source server. The ca_source_server list MUST
be the same as the COPY_NOTIFY response's cnr_source_server list. If be the same as the COPY_NOTIFY response's cnr_source_server list. If
the client includes the entries from the COPY_NOTIFY response's the client includes the entries from the COPY_NOTIFY response's
cnr_source_server list in the ca_source_server list, the source cnr_source_server list in the ca_source_server list, the source
server can indicate a specific copy protocol for the destination server can indicate a specific copy protocol for the destination
server to use by returning a URL, which specifies both a protocol server to use by returning a URL, which specifies both a protocol
service and server name. Server-to-server copy protocol service and server name. Server-to-server copy protocol
considerations are described in Section 2.2.4 and Section 2.4.1. considerations are described in Section 2.2.5 and Section 2.4.1.
The ca_flags argument allows the copy operation to be customized in The ca_flags argument allows the copy operation to be customized in
the following ways using the guarded flag (COPY4_GUARDED) and the the following ways using the guarded flag (COPY4_GUARDED) and the
metadata flag (COPY4_METADATA). metadata flag (COPY4_METADATA).
If the guarded flag is set and the destination exists on the server, If the guarded flag is set and the destination exists on the server,
this operation will fail with NFS4ERR_EXIST. this operation will fail with NFS4ERR_EXIST.
If the guarded flag is not set and the destination exists on the If the guarded flag is not set and the destination exists on the
server, the behavior is implementation dependent. server, the behavior is implementation dependent.
skipping to change at page 57, line 4 skipping to change at page 57, line 15
o NFS4ERR_FBIG o NFS4ERR_FBIG
o NFS4ERR_NOTDIR o NFS4ERR_NOTDIR
o NFS4ERR_WRONG_TYPE o NFS4ERR_WRONG_TYPE
o NFS4ERR_ISDIR o NFS4ERR_ISDIR
o NFS4ERR_INVAL o NFS4ERR_INVAL
o NFS4ERR_DELAY o NFS4ERR_DELAY
o NFS4ERR_METADATA_NOTSUPP o NFS4ERR_METADATA_NOTSUPP
o NFS4ERR_WRONGSEC o NFS4ERR_WRONGSEC
13.2. Operation 60: COPY_ABORT - Cancel a server-side copy 13.2. Operation 60: OFFLOAD_ABORT - Cancel a server-side copy
13.2.1. ARGUMENT 13.2.1. ARGUMENT
struct COPY_ABORT4args { struct OFFLOAD_ABORT4args {
/* CURRENT_FH: destination file */ /* CURRENT_FH: destination file */
stateid4 caa_stateid; stateid4 oaa_stateid;
}; };
13.2.2. RESULT 13.2.2. RESULT
struct COPY_ABORT4res { struct OFFLOAD_ABORT4res {
nfsstat4 car_status; nfsstat4 oar_status;
}; };
13.2.3. DESCRIPTION 13.2.3. DESCRIPTION
COPY_ABORT is used for both intra- and inter-server asynchronous OFFLOAD_ABORT is used for both intra- and inter-server asynchronous
copies. The COPY_ABORT operation allows the client to cancel a copies. The OFFLOAD_ABORT operation allows the client to cancel a
server-side copy operation that it initiated. This operation is sent server-side copy operation that it initiated. This operation is sent
in a COMPOUND request from the client to the destination server. in a COMPOUND request from the client to the destination server.
This operation may be used to cancel a copy when the application that This operation may be used to cancel a copy when the application that
requested the copy exits before the operation is completed or for requested the copy exits before the operation is completed or for
some other reason. some other reason.
The request contains the filehandle and copy stateid cookies that act The request contains the filehandle and copy stateid cookies that act
as the context for the previously initiated copy operation. as the context for the previously initiated copy operation.
The result's car_status field indicates whether the cancel was The result's oar_status field indicates whether the cancel was
successful or not. A value of NFS4_OK indicates that the copy successful or not. A value of NFS4_OK indicates that the copy
operation was canceled and no callback will be issued by the server. operation was canceled and no callback will be issued by the server.
A copy operation that is successfully canceled may result in none, A copy operation that is successfully canceled may result in none,
some, or all of the data and/or metadata copied. some, or all of the data and/or metadata copied.
If the server supports asynchronous copies, the server is REQUIRED to If the server supports asynchronous copies, the server is REQUIRED to
support the COPY_ABORT operation. support the OFFLOAD_ABORT operation.
The COPY_ABORT operation may fail for the following reasons (this is The OFFLOAD_ABORT operation may fail for the following reasons (this
a partial list): is a partial list):
o NFS4ERR_NOTSUPP o NFS4ERR_NOTSUPP
o NFS4ERR_RETRY o NFS4ERR_RETRY
o NFS4ERR_COMPLETE_ALREADY o NFS4ERR_COMPLETE_ALREADY
o NFS4ERR_SERVERFAULT o NFS4ERR_SERVERFAULT
13.3. Operation 61: COPY_NOTIFY - Notify a source server of a future 13.3. Operation 61: COPY_NOTIFY - Notify a source server of a future
copy copy
13.3.1. ARGUMENT 13.3.1. ARGUMENT
struct COPY_NOTIFY4args { struct COPY_NOTIFY4args {
/* CURRENT_FH: source file */ /* CURRENT_FH: source file */
stateid4 cna_src_stateid;
netloc4 cna_destination_server; netloc4 cna_destination_server;
}; };
13.3.2. RESULT 13.3.2. RESULT
struct COPY_NOTIFY4resok { struct COPY_NOTIFY4resok {
nfstime4 cnr_lease_time; nfstime4 cnr_lease_time;
netloc4 cnr_source_server<>; netloc4 cnr_source_server<>;
}; };
skipping to change at page 58, line 44 skipping to change at page 59, line 12
void; void;
}; };
13.3.3. DESCRIPTION 13.3.3. DESCRIPTION
This operation is used for an inter-server copy. A client sends this This operation is used for an inter-server copy. A client sends this
operation in a COMPOUND request to the source server to authorize a operation in a COMPOUND request to the source server to authorize a
destination server identified by cna_destination_server to read the destination server identified by cna_destination_server to read the
file specified by CURRENT_FH on behalf of the given user. file specified by CURRENT_FH on behalf of the given user.
The cna_src_stateid MUST refer to either open or locking states
provided earlier by the server. If it is invalid, then the operation
MUST fail.
The cna_destination_server MUST be specified using the netloc4 The cna_destination_server MUST be specified using the netloc4
network location format. The server is not required to resolve the network location format. The server is not required to resolve the
cna_destination_server address before completing this operation. cna_destination_server address before completing this operation.
If this operation succeeds, the source server will allow the If this operation succeeds, the source server will allow the
cna_destination_server to copy the specified file on behalf of the cna_destination_server to copy the specified file on behalf of the
given user as long as both of the following conditions are met: given user as long as both of the following conditions are met:
o The destination server begins reading the source file before the o The destination server begins reading the source file before the
cnr_lease_time expires. If the cnr_lease_time expires while the cnr_lease_time expires. If the cnr_lease_time expires while the
skipping to change at page 59, line 42 skipping to change at page 60, line 11
The COPY_NOTIFY operation may fail for the following reasons (this is The COPY_NOTIFY operation may fail for the following reasons (this is
a partial list): a partial list):
o NFS4ERR_MOVED o NFS4ERR_MOVED
o NFS4ERR_NOTSUPP o NFS4ERR_NOTSUPP
o NFS4ERR_WRONGSEC o NFS4ERR_WRONGSEC
13.4. Operation 62: COPY_REVOKE - Revoke a destination server's copy 13.4. Operation 62: OFFLOAD_REVOKE - Revoke a destination server's copy
privileges privileges
13.4.1. ARGUMENT 13.4.1. ARGUMENT
struct COPY_REVOKE4args { struct OFFLOAD_REVOKE4args {
/* CURRENT_FH: source file */ /* CURRENT_FH: source file */
netloc4 cra_destination_server; netloc4 ora_destination_server;
}; };
13.4.2. RESULT 13.4.2. RESULT
struct COPY_REVOKE4res { struct OFFLOAD_REVOKE4res {
nfsstat4 crr_status; nfsstat4 orr_status;
}; };
13.4.3. DESCRIPTION 13.4.3. DESCRIPTION
This operation is used for an inter-server copy. A client sends this This operation is used for an inter-server copy. A client sends this
operation in a COMPOUND request to the source server to revoke the operation in a COMPOUND request to the source server to revoke the
authorization of a destination server identified by authorization of a destination server identified by
cra_destination_server from reading the file specified by CURRENT_FH ora_destination_server from reading the file specified by CURRENT_FH
on behalf of given user. If the cra_destination_server has already on behalf of given user. If the ora_destination_server has already
begun copying the file, a successful return from this operation begun copying the file, a successful return from this operation
indicates that further access will be prevented. indicates that further access will be prevented.
The cra_destination_server MUST be specified using the netloc4 The ora_destination_server MUST be specified using the netloc4
network location format. The server is not required to resolve the network location format. The server is not required to resolve the
cra_destination_server address before completing this operation. ora_destination_server address before completing this operation.
The client uses COPY_ABORT to inform the destination to stop the The client uses OFFLOAD_ABORT to inform the destination to stop the
active transfer and COPY_REVOKE to inform the source to not allow any active transfer and OFFLOAD_REVOKE to inform the source to not allow
more copy requests from the destination. The COPY_REVOKE operation any more copy requests from the destination. The OFFLOAD_REVOKE
is also useful in situations in which the source server granted a operation is also useful in situations in which the source server
very long or infinite lease on the destination server's ability to granted a very long or infinite lease on the destination server's
read the source file and all copy operations on the source file have ability to read the source file and all copy operations on the source
been completed. file have been completed.
For a copy only involving one server (the source and destination are For a copy only involving one server (the source and destination are
on the same server), this operation is unnecessary. on the same server), this operation is unnecessary.
If the server supports COPY_NOTIFY, the server is REQUIRED to support If the server supports COPY_NOTIFY, the server is REQUIRED to support
the COPY_REVOKE operation. the OFFLOAD_REVOKE operation.
The COPY_REVOKE operation may fail for the following reasons (this is The OFFLOAD_REVOKE operation may fail for the following reasons (this
a partial list): is a partial list):
o NFS4ERR_MOVED o NFS4ERR_MOVED
o NFS4ERR_NOTSUPP o NFS4ERR_NOTSUPP
13.5. Operation 63: COPY_STATUS - Poll for status of a server-side copy 13.5. Operation 63: OFFLOAD_STATUS - Poll for status of a server-side
copy
13.5.1. ARGUMENT 13.5.1. ARGUMENT
struct COPY_STATUS4args { struct OFFLOAD_STATUS4args {
/* CURRENT_FH: destination file */ /* CURRENT_FH: destination file */
stateid4 csa_stateid; stateid4 osa_stateid;
}; };
13.5.2. RESULT 13.5.2. RESULT
struct COPY_STATUS4resok { struct OFFLOAD_STATUS4resok {
length4 csr_bytes_copied; length4 osr_bytes_copied;
nfsstat4 csr_complete<1>; nfsstat4 osr_complete<1>;
}; };
union COPY_STATUS4res switch (nfsstat4 csr_status) { union OFFLOAD_STATUS4res switch (nfsstat4 osr_status) {
case NFS4_OK: case NFS4_OK:
COPY_STATUS4resok resok4; OFFLOAD_STATUS4resok resok4;
default: default:
void; void;
}; };
13.5.3. DESCRIPTION 13.5.3. DESCRIPTION
COPY_STATUS is used for both intra- and inter-server asynchronous OFFLOAD_STATUS is used for both intra- and inter-server asynchronous
copies. The COPY_STATUS operation allows the client to poll the copies. The OFFLOAD_STATUS operation allows the client to poll the
destination server to determine the status of an asynchronous copy destination server to determine the status of an asynchronous copy
operation. operation.
If this operation is successful, the number of bytes copied are If this operation is successful, the number of bytes copied are
returned to the client in the csr_bytes_copied field. The returned to the client in the osr_bytes_copied field. The
csr_bytes_copied value indicates the number of bytes copied but not osr_bytes_copied value indicates the number of bytes copied but not
which specific bytes have been copied. which specific bytes have been copied.
If the optional csr_complete field is present, the copy has If the optional osr_complete field is present, the copy has
completed. In this case the status value indicates the result of the completed. In this case the status value indicates the result of the
asynchronous copy operation. In all cases, the server will also asynchronous copy operation. In all cases, the server will also
deliver the final results of the asynchronous copy in a CB_COPY deliver the final results of the asynchronous copy in a CB_COPY
operation. operation.
The failure of this operation does not indicate the result of the The failure of this operation does not indicate the result of the
asynchronous copy in any way. asynchronous copy in any way.
If the server supports asynchronous copies, the server is REQUIRED to If the server supports asynchronous copies, the server is REQUIRED to
support the COPY_STATUS operation. support the OFFLOAD_STATUS operation.
The COPY_STATUS operation may fail for the following reasons (this is The OFFLOAD_STATUS operation may fail for the following reasons (this
a partial list): is a partial list):
o NFS4ERR_NOTSUPP o NFS4ERR_NOTSUPP
o NFS4ERR_BAD_STATEID o NFS4ERR_BAD_STATEID
o NFS4ERR_EXPIRED o NFS4ERR_EXPIRED
13.6. Modification to Operation 42: EXCHANGE_ID - Instantiate Client ID 13.6. Modification to Operation 42: EXCHANGE_ID - Instantiate Client ID
13.6.1. ARGUMENT 13.6.1. ARGUMENT
skipping to change at page 63, line 15 skipping to change at page 63, line 27
o The NFS server SHOULD support client ID trunking, and if it does o The NFS server SHOULD support client ID trunking, and if it does
and the EXCHGID4_FLAG_SUPP_FENCE_OPS capability is enabled, then a and the EXCHGID4_FLAG_SUPP_FENCE_OPS capability is enabled, then a
session ID created on one node of the storage cluster MUST be session ID created on one node of the storage cluster MUST be
destroyable via DESTROY_SESSION. In addition, DESTROY_CLIENTID destroyable via DESTROY_SESSION. In addition, DESTROY_CLIENTID
and an EXCHANGE_ID with a new verifier affects all sessions and an EXCHANGE_ID with a new verifier affects all sessions
regardless what node the sessions were created on. regardless what node the sessions were created on.
13.7. Operation 64: INITIALIZE 13.7. Operation 64: INITIALIZE
This operation can be used to initialize the structure imposed by an This operation can be used to initialize the structure imposed by an
application onto a file, i.e., ADBs, and to punch a hole into a file. application onto a file, i.e., ADHs, and to punch a hole into a file.
13.7.1. ARGUMENT 13.7.1. ARGUMENT
struct data_info4 {
offset4 di_offset;
length4 di_length;
bool di_allocated;
};
/* /*
* We use data_content4 in case we wish to * We use data_content4 in case we wish to
* extend new types later. Note that we * extend new types later. Note that we
* are explicitly disallowing data. * are explicitly disallowing data.
*/ */
union initialize_arg4 switch (data_content4 content) { union initialize_arg4 switch (data_content4 content) {
case NFS4_CONTENT_APP_BLOCK: case NFS4_CONTENT_APP_DATA_HOLE:
app_data_block4 ia_adb; app_data_hole4 ia_adh;
case NFS4_CONTENT_HOLE: case NFS4_CONTENT_HOLE:
data_info4 ia_hole; data_info4 ia_hole;
default: default:
void; void;
}; };
struct INITIALIZE4args { struct INITIALIZE4args {
/* CURRENT_FH: file */ /* CURRENT_FH: file */
stateid4 ia_stateid; stateid4 ia_stateid;
stable_how4 ia_stable; stable_how4 ia_stable;
skipping to change at page 64, line 24 skipping to change at page 64, line 44
union INITIALIZE4res switch (nfsstat4 status) { union INITIALIZE4res switch (nfsstat4 status) {
case NFS4_OK: case NFS4_OK:
INITIALIZE4resok resok4; INITIALIZE4resok resok4;
default: default:
void; void;
}; };
13.7.3. DESCRIPTION 13.7.3. DESCRIPTION
Using the data_content4 (Section 6.1.2), INITIALIZE can be used Using the data_content4 (Section 6.1.2), INITIALIZE can be used
either to punch holes or to impose ADB structure on a file. either to punch holes or to impose ADH structure on a file.
13.7.3.1. Hole punching 13.7.3.1. Hole punching
Whenever a client wishes to zero the blocks backing a particular Whenever a client wishes to zero the blocks backing a particular
region in the file, it calls the INITIALIZE operation with the region in the file, it calls the INITIALIZE operation with the
current filehandle set to the filehandle of the file in question, and current filehandle set to the filehandle of the file in question, and
the equivalent of start offset and length in bytes of the region set the equivalent of start offset and length in bytes of the region set
in ia_hole.di_offset and ia_hole.di_length respectively. If the in ia_hole.di_offset and ia_hole.di_length respectively. If the
ia_hole.di_allocated is set to TRUE, then the blocks will be zeroed ia_hole.di_allocated is set to TRUE, then the blocks will be zeroed
and if it is set to FALSE, then they will be deallocated. All and if it is set to FALSE, then they will be deallocated. All
skipping to change at page 65, line 32 skipping to change at page 66, line 12
NFS4ERR_NOTSUPP The Hole punch operations are not supported by the NFS4ERR_NOTSUPP The Hole punch operations are not supported by the
NFS server receiving this request. NFS server receiving this request.
NFS4ERR_DIR The current filehandle is of type NF4DIR. NFS4ERR_DIR The current filehandle is of type NF4DIR.
NFS4ERR_SYMLINK The current filehandle is of type NF4LNK. NFS4ERR_SYMLINK The current filehandle is of type NF4LNK.
NFS4ERR_WRONG_TYPE The current filehandle does not designate an NFS4ERR_WRONG_TYPE The current filehandle does not designate an
ordinary file. ordinary file.
13.7.3.2. ADBs 13.7.3.2. ADHs
If the server supports ADBs, then it MUST support the If the server supports ADHs, then it MUST support the
NFS4_CONTENT_APP_BLOCK arm of the INITIALIZE operation. The server NFS4_CONTENT_APP_DATA_HOLE arm of the INITIALIZE operation. The
has no concept of the structure imposed by the application. It is server has no concept of the structure imposed by the application.
only when the application writes to a section of the file does order It is only when the application writes to a section of the file does
get imposed. In order to detect corruption even before the order get imposed. In order to detect corruption even before the
application utilizes the file, the application will want to application utilizes the file, the application will want to
initialize a range of ADBs using INITIALIZE. initialize a range of ADHs using INITIALIZE.
For ADBs, when the client invokes the INITIALIZE operation, it has For ADHs, when the client invokes the INITIALIZE operation, it has
two desired results: two desired results:
1. The structure described by the app_data_block4 be imposed on the 1. The structure described by the app_data_block4 be imposed on the
file. file.
2. The contents described by the app_data_block4 be sparse. 2. The contents described by the app_data_block4 be sparse.
If the server supports the INITIALIZE operation, it still might not If the server supports the INITIALIZE operation, it still might not
support sparse files. So if it receives the INITIALIZE operation, support sparse files. So if it receives the INITIALIZE operation,
then it MUST populate the contents of the file with the initialized then it MUST populate the contents of the file with the initialized
ADBs. ADHs.
If the data was already initialized, there are two interesting If the data was already initialized, there are two interesting
scenarios: scenarios:
1. The data blocks are allocated. 1. The data blocks are allocated.
2. Initializing in the middle of an existing ADB. 2. Initializing in the middle of an existing ADH.
If the data blocks were already allocated, then the INITIALIZE is a If the data blocks were already allocated, then the INITIALIZE is a
hole punch operation. If INITIALIZE supports sparse files, then the hole punch operation. If INITIALIZE supports sparse files, then the
data blocks are to be deallocated. If not, then the data blocks are data blocks are to be deallocated. If not, then the data blocks are
to be rewritten in the indicated ADB format. to be rewritten in the indicated ADH format.
Since the server has no knowledge of ADBs, it should not report Since the server has no knowledge of ADHs, it should not report
misaligned creation of ADBs. Even while it can detect them, it misaligned creation of ADHs. Even while it can detect them, it
cannot disallow them, as the application might be in the process of cannot disallow them, as the application might be in the process of
changing the size of the ADBs. Thus the server must be prepared to changing the size of the ADHs. Thus the server must be prepared to
handle an INITIALIZE into an existing ADB. handle an INITIALIZE into an existing ADH.
This document does not mandate the manner in which the server stores This document does not mandate the manner in which the server stores
ADBs sparsely for a file. It does assume that if ADBs are stored ADHs sparsely for a file. However, if an INITIALIZE arrives that
sparsely, then the server can detect when an INITIALIZE arrives that will force a new ADH to start inside an existing ADH then the server
will force a new ADB to start inside an existing ADB. For example, will have three ADHs instead of two. It will have one up to the new
assume that ADBi has a adb_block_size of 4k and that an INITIALIZE one for the INITIALIZE, one for the INITIALIZE, and one for after the
starts 1k inside ADBi. The server should [[Comment.3: Need to flesh INITIALIZE. Note that depending on server specific policies for
this out. --TH]] block allocation, there may also be some physical blocks allocated to
align the boundaries.
13.8. Operation 67: IO_ADVISE - Application I/O access pattern hints 13.8. Operation 67: IO_ADVISE - Application I/O access pattern hints
13.8.1. ARGUMENT 13.8.1. ARGUMENT
enum IO_ADVISE_type4 { enum IO_ADVISE_type4 {
IO_ADVISE4_NORMAL = 0, IO_ADVISE4_NORMAL = 0,
IO_ADVISE4_SEQUENTIAL = 1, IO_ADVISE4_SEQUENTIAL = 1,
IO_ADVISE4_SEQUENTIAL_BACKWARDS = 2, IO_ADVISE4_SEQUENTIAL_BACKWARDS = 2,
IO_ADVISE4_RANDOM = 3, IO_ADVISE4_RANDOM = 3,
IO_ADVISE4_WILLNEED = 4, IO_ADVISE4_WILLNEED = 4,
IO_ADVISE4_WILLNEED_OPPORTUNISTIC = 5, IO_ADVISE4_WILLNEED_OPPORTUNISTIC = 5,
IO_ADVISE4_DONTNEED = 6, IO_ADVISE4_DONTNEED = 6,
skipping to change at page 75, line 9 skipping to change at page 75, line 21
The MDS is not required to indefinitely retain per-client storage The MDS is not required to indefinitely retain per-client storage
device error information. An MDS is also not required to device error information. An MDS is also not required to
automatically reinstate use of a previously problematic storage automatically reinstate use of a previously problematic storage
device; administrative intervention may be required instead. device; administrative intervention may be required instead.
13.10. Operation 65: READ_PLUS 13.10. Operation 65: READ_PLUS
READ_PLUS is a new variant of the NFSv4.1 READ operation [2]. READ_PLUS is a new variant of the NFSv4.1 READ operation [2].
Besides being able to support all of the data semantics of READ, it Besides being able to support all of the data semantics of READ, it
can also be used by the server to return either holes or ADBs to the can also be used by the server to return either holes or ADHs to the
client. For holes, READ_PLUS extends the response to avoid returning client. For holes, READ_PLUS extends the response to avoid returning
data for portions of the file which are either initialized and data for portions of the file which are either initialized and
contain no backing store or if the result would appear to be so. contain no backing store or if the result would appear to be so.
I.e., if the result was a data block composed entirely of zeros, then I.e., if the result was a data block composed entirely of zeros, then
it is easier to return a hole. Returning data blocks of unitialized it is easier to return a hole. Returning data blocks of
data wastes computational and network resources, thus reducing uninitialized data wastes computational and network resources, thus
performance. For ADBs, READ_PLUS is used to return the metadata reducing performance. For ADHs, READ_PLUS is used to return the
describing the portions of the file which are either initialized and metadata describing the portions of the file which are either
contain no backing store. initialized and contain no backing store.
If the client sends a READ operation, it is explicitly stating that If the client sends a READ operation, it is explicitly stating that
it is neither supporting sparse files nor ADBs. So if a READ occurs it is neither supporting sparse files nor ADHs. So if a READ occurs
on a sparse ADB or file, then the server must expand such data to be on a sparse ADH or file, then the server must expand such data to be
raw bytes. If a READ occurs in the middle of a hole or ADB, the raw bytes. If a READ occurs in the middle of a hole or ADH, the
server can only send back bytes starting from that offset. In server can only send back bytes starting from that offset. In
contrast, if a READ_PLUS occurs in the middle of a hole or ADB, the contrast, if a READ_PLUS occurs in the middle of a hole or ADH, the
server can send back a range which starts before the offset and server can send back a range which starts before the offset and
extends past the range. extends past the range.
READ is inefficient for transfer of sparse sections of the file. As READ is inefficient for transfer of sparse sections of the file. As
such, READ is marked as OBSOLETE in NFSv4.2. Instead, a client such, READ is marked as OBSOLETE in NFSv4.2. Instead, a client
should issue READ_PLUS. Note that as the client has no a priori should issue READ_PLUS. Note that as the client has no a priori
knowledge of whether either an ADB or a hole is present or not, it knowledge of whether either an ADH or a hole is present or not, it
should always use READ_PLUS. should always use READ_PLUS.
13.10.1. ARGUMENT 13.10.1. ARGUMENT
struct READ_PLUS4args { struct READ_PLUS4args {
/* CURRENT_FH: file */ /* CURRENT_FH: file */
stateid4 rpa_stateid; stateid4 rpa_stateid;
offset4 rpa_offset; offset4 rpa_offset;
count4 rpa_count; count4 rpa_count;
}; };
13.10.2. RESULT 13.10.2. RESULT
union read_plus_content switch (data_content4 content) { union read_plus_content switch (data_content4 content) {
case NFS4_CONTENT_DATA: case NFS4_CONTENT_DATA:
opaque rpc_data<>; opaque rpc_data<>;
case NFS4_CONTENT_APP_BLOCK: case NFS4_CONTENT_APP_DATA_HOLE:
app_data_block4 rpc_block; app_data_hole4 rpc_adh;
case NFS4_CONTENT_HOLE: case NFS4_CONTENT_HOLE:
data_info4 rpc_hole; data_info4 rpc_hole;
default: default:
void; void;
}; };
/* /*
* Allow a return of an array of contents. * Allow a return of an array of contents.
*/ */
struct read_plus_res4 { struct read_plus_res4 {
skipping to change at page 76, line 48 skipping to change at page 77, line 10
The client provides a rpa_offset of where the READ_PLUS is to start The client provides a rpa_offset of where the READ_PLUS is to start
and a rpa_count of how many bytes are to be read. A rpa_offset of and a rpa_count of how many bytes are to be read. A rpa_offset of
zero means to read data starting at the beginning of the file. If zero means to read data starting at the beginning of the file. If
rpa_offset is greater than or equal to the size of the file, the rpa_offset is greater than or equal to the size of the file, the
status NFS4_OK is returned with di_length (the data length) set to status NFS4_OK is returned with di_length (the data length) set to
zero and eof set to TRUE. zero and eof set to TRUE.
The READ_PLUS result is comprised of an array of rpr_contents, each The READ_PLUS result is comprised of an array of rpr_contents, each
of which describe a data_content4 type of data (Section 6.1.2). For of which describe a data_content4 type of data (Section 6.1.2). For
NFSv4.2, the allowed values are data, ADB, and hole. A server is NFSv4.2, the allowed values are data, ADH, and hole. A server is
required to support the data type, but neither ADB nor hole. Both an required to support the data type, but neither ADH nor hole. Both an
ADB and a hole must be returned in its entirety - clients must be ADH and a hole must be returned in its entirety - clients must be
prepared to get more information than they requested. prepared to get more information than they requested. Both the start
and the end of the hole may execeed what was requested.
READ_PLUS has to support all of the errors which are returned by READ READ_PLUS has to support all of the errors which are returned by READ
plus NFS4ERR_UNION_NOTSUPP. If the client asks for a hole and the plus NFS4ERR_UNION_NOTSUPP. If the client asks for a hole and the
server does not support that arm of the discriminated union, but does server does not support that arm of the discriminated union, but does
support one or more additional arms, it can signal to the client that support one or more additional arms, it can signal to the client that
it supports the operation, but not the arm with it supports the operation, but not the arm with
NFS4ERR_UNION_NOTSUPP. NFS4ERR_UNION_NOTSUPP.
If the data to be returned is comprised entirely of zeros, then the If the data to be returned is comprised entirely of zeros, then the
server may elect to return that data as a hole. The server server may elect to return that data as a hole. The server
differentiates this to the client by setting di_allocated to TRUE in differentiates this to the client by setting di_allocated to TRUE in
this case. Note that in such a scenario, the server is not required this case. Note that in such a scenario, the server is not required
to determine the full extent of the "hole" - it does not need to to determine the full extent of the "hole" - it does not need to
determine where the zeros start and end. determine where the zeros start and end.
The server may elect to return adjacent elements of the same type. The server may elect to return adjacent elements of the same type.
For example, the guard pattern or block size of an ADB might change, For example, the guard pattern or block size of an ADH might change,
which would require adjacent elements of type ADB. Likewise if the which would require adjacent elements of type ADH. Likewise if the
server has a range of data comprised entirely of zeros and then a server has a range of data comprised entirely of zeros and then a
hole, it might want to return two adjacent holes to the client. hole, it might want to return two adjacent holes to the client.
If the client specifies a rpa_count value of zero, the READ_PLUS If the client specifies a rpa_count value of zero, the READ_PLUS
succeeds and returns zero bytes of data. In all situations, the succeeds and returns zero bytes of data. In all situations, the
server may choose to return fewer bytes than specified by the client. server may choose to return fewer bytes than specified by the client.
The client needs to check for this condition and handle the condition The client needs to check for this condition and handle the condition
appropriately. appropriately.
If the client specifies an rpa_offset and rpa_count value that is If the client specifies an rpa_offset and rpa_count value that is
skipping to change at page 78, line 40 skipping to change at page 78, line 50
13.10.4. IMPLEMENTATION 13.10.4. IMPLEMENTATION
In general, the IMPLEMENTATION notes for READ in Section 18.22.4 of In general, the IMPLEMENTATION notes for READ in Section 18.22.4 of
[2] also apply to READ_PLUS. One delta is that when the owner has a [2] also apply to READ_PLUS. One delta is that when the owner has a
locked byte range, the server MUST return an array of rpr_contents locked byte range, the server MUST return an array of rpr_contents
with values inside that range. with values inside that range.
13.10.4.1. Additional pNFS Implementation Information 13.10.4.1. Additional pNFS Implementation Information
With pNFS, the semantics of using READ_PLUS remains the same. Any With pNFS, the semantics of using READ_PLUS remains the same. Any
data server MAY return a hole or ADB result for a READ_PLUS request data server MAY return a hole or ADH result for a READ_PLUS request
that it receives. When a data server chooses to return such a that it receives. When a data server chooses to return such a
result, it has the option of returning information for the data result, it has the option of returning information for the data
stored on that data server (as defined by the data layout), but it stored on that data server (as defined by the data layout), but it
MUST not return results for a byte range that includes data managed MUST not return results for a byte range that includes data managed
by another data server. by another data server.
A data server should do its best to return as much information about A data server should do its best to return as much information about
a hole ADB as is feasible without having to contact the metadata a ADH as is feasible without having to contact the metadata server.
server. If communication with the metadata server is required, then If communication with the metadata server is required, then every
every attempt should be taken to minimize the number of requests. attempt should be taken to minimize the number of requests.
If mandatory locking is enforced, then the data server must also If mandatory locking is enforced, then the data server must also
ensure that to return only information that is within the owner's ensure that to return only information that is within the owner's
locked byte range. locked byte range.
13.10.5. READ_PLUS with Sparse Files Example 13.10.5. READ_PLUS with Sparse Files Example
The following table describes a sparse file. For each byte range, The following table describes a sparse file. For each byte range,
the file contains either non-zero data or a hole. In addition, the the file contains either non-zero data or a hole. In addition, the
server in this example uses a Hole Threshold of 32K. server in this example uses a Hole Threshold of 32K.
skipping to change at page 80, line 27 skipping to change at page 81, line 10
stateid4 sa_stateid; stateid4 sa_stateid;
offset4 sa_offset; offset4 sa_offset;
data_content4 sa_what; data_content4 sa_what;
}; };
13.11.2. RESULT 13.11.2. RESULT
union seek_content switch (data_content4 content) { union seek_content switch (data_content4 content) {
case NFS4_CONTENT_DATA: case NFS4_CONTENT_DATA:
data_info4 sc_data; data_info4 sc_data;
case NFS4_CONTENT_APP_BLOCK: case NFS4_CONTENT_APP_DATA_HOLE:
app_data_block4 sc_block; app_data_hole4 sc_adh;
case NFS4_CONTENT_HOLE: case NFS4_CONTENT_HOLE:
data_info4 sc_hole; data_info4 sc_hole;
default: default:
void; void;
}; };
struct seek_res4 { struct seek_res4 {
bool sr_eof; bool sr_eof;
seek_content sr_contents; seek_content sr_contents;
}; };
skipping to change at page 81, line 8 skipping to change at page 81, line 33
union SEEK4res switch (nfsstat4 status) { union SEEK4res switch (nfsstat4 status) {
case NFS4_OK: case NFS4_OK:
seek_res4 resok4; seek_res4 resok4;
default: default:
void; void;
}; };
13.11.3. DESCRIPTION 13.11.3. DESCRIPTION
From the given sa_offset, find the next data_content4 of type sa_what From the given sa_offset, find the next data_content4 of type sa_what
in the file. For either a hole or ADB, this must return the in the file. For either a hole or ADH, this must return the
data_content4 in its entirety. For data, it must not return the data_content4 in its entirety. For data, it must not return the
actual data. actual data.
SEEK must follow the same rules for stateids as READ_PLUS SEEK must follow the same rules for stateids as READ_PLUS
(Section 13.10.3). (Section 13.10.3).
If the server could not find a corresponding sa_what, then the status If the server could not find a corresponding sa_what, then the status
would still be NFS4_OK, but sr_eof would be TRUE. The sr_contents would still be NFS4_OK, but sr_eof would be TRUE. The sr_contents
would contain a zero-ed out content of the appropriate type. would contain a zero-ed out content of the appropriate type.
 End of changes. 142 change blocks. 
301 lines changed or deleted 361 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/