draft-ietf-nfsv4-flex-files-05.txt   draft-ietf-nfsv4-flex-files-06.txt 
NFSv4 B. Halevy NFSv4 B. Halevy
Internet-Draft Internet-Draft
Intended status: Standards Track T. Haynes Intended status: Standards Track T. Haynes
Expires: August 13, 2015 Primary Data Expires: January 22, 2016 Primary Data
February 09, 2015 July 21, 2015
Parallel NFS (pNFS) Flexible File Layout Parallel NFS (pNFS) Flexible File Layout
draft-ietf-nfsv4-flex-files-05.txt draft-ietf-nfsv4-flex-files-06.txt
Abstract Abstract
The Parallel Network File System (pNFS) allows a separation between The Parallel Network File System (pNFS) allows a separation between
the metadata (onto a metadata server) and data (onto a storage the metadata (onto a metadata server) and data (onto a storage
device) for a file. The Flexible File Layout Type is defined in this device) for a file. The Flexible File Layout Type is defined in this
document as an extension to pNFS to allow the use of storage devices document as an extension to pNFS to allow the use of storage devices
in a fashion such that they require only a quite limited degree of in a fashion such that they require only a quite limited degree of
interaction with the metadata server, using already existing interaction with the metadata server, using already existing
protocols. Client side mirroring is also added to provide protocols. Client side mirroring is also added to provide
skipping to change at page 1, line 38 skipping to change at page 1, line 38
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 13, 2015. This Internet-Draft will expire on January 22, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 17 skipping to change at page 2, line 17
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Difference Between a Data Server and a Storage Device . . 5 1.2. Difference Between a Data Server and a Storage Device . . 5
1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6 1.3. Requirements Language . . . . . . . . . . . . . . . . . . 6
2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6 2. Coupling of Storage Devices . . . . . . . . . . . . . . . . . 6
2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6 2.1. LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . 6
2.2. Security Models . . . . . . . . . . . . . . . . . . . . . 6 2.2. Fencing Clients from the Data Server . . . . . . . . . . 6
2.2.1. Implementation Notes for Synthetic uids/gids . . . . 7 2.2.1. Implementation Notes for Synthetic uids/gids . . . . 7
2.2.2. Example of using Synthetic uids/gids . . . . . . . . 7 2.2.2. Example of using Synthetic uids/gids . . . . . . . . 7
2.3. State and Locking Models . . . . . . . . . . . . . . . . 8 2.3. State and Locking Models . . . . . . . . . . . . . . . . 8
3. XDR Description of the Flexible File Layout Type . . . . . . 9 3. XDR Description of the Flexible File Layout Type . . . . . . 9
3.1. Code Components Licensing Notice . . . . . . . . . . . . 9 3.1. Code Components Licensing Notice . . . . . . . . . . . . 9
4. Device Addressing and Discovery . . . . . . . . . . . . . . . 11 4. Device Addressing and Discovery . . . . . . . . . . . . . . . 11
4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 11 4.1. ff_device_addr4 . . . . . . . . . . . . . . . . . . . . . 11
4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 12 4.2. Storage Device Multipathing . . . . . . . . . . . . . . . 12
5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 13 5. Flexible File Layout Type . . . . . . . . . . . . . . . . . . 13
5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 14 5.1. ff_layout4 . . . . . . . . . . . . . . . . . . . . . . . 14
5.2. Interactions Between Devices and Layouts . . . . . . . . 17 5.2. Interactions Between Devices and Layouts . . . . . . . . 17
5.3. Handling Version Errors . . . . . . . . . . . . . . . . . 17 5.3. Handling Version Errors . . . . . . . . . . . . . . . . . 18
6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 18 6. Striping via Sparse Mapping . . . . . . . . . . . . . . . . . 18
7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 18 7. Recovering from Client I/O Errors . . . . . . . . . . . . . . 19
8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 19 8. Mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . 19
8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 20 8.1. Selecting a Mirror . . . . . . . . . . . . . . . . . . . 20
8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 20 8.2. Writing to Mirrors . . . . . . . . . . . . . . . . . . . 20
8.3. Metadata Server Resilvering of the File . . . . . . . . . 21 8.3. Metadata Server Resilvering of the File . . . . . . . . . 21
9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 21 9. Flexible Files Layout Type Return . . . . . . . . . . . . . . 21
9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 22 9.1. I/O Error Reporting . . . . . . . . . . . . . . . . . . . 22
9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 22 9.1.1. ff_ioerr4 . . . . . . . . . . . . . . . . . . . . . . 22
9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 23 9.2. Layout Usage Statistics . . . . . . . . . . . . . . . . . 23
9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 23 9.2.1. ff_io_latency4 . . . . . . . . . . . . . . . . . . . 23
9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 23 9.2.2. ff_layoutupdate4 . . . . . . . . . . . . . . . . . . 24
9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 24 9.2.3. ff_iostats4 . . . . . . . . . . . . . . . . . . . . . 24
9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 25 9.3. ff_layoutreturn4 . . . . . . . . . . . . . . . . . . . . 25
10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 25 10. Flexible Files Layout Type LAYOUTERROR . . . . . . . . . . . 26
11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 25 11. Flexible Files Layout Type LAYOUTSTATS . . . . . . . . . . . 26
12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 26 12. Flexible File Layout Type Creation Hint . . . . . . . . . . . 26
12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 26 12.1. ff_layouthint4 . . . . . . . . . . . . . . . . . . . . . 27
13. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 27 13. Recalling Layouts . . . . . . . . . . . . . . . . . . . . . . 27
13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 27 13.1. CB_RECALL_ANY . . . . . . . . . . . . . . . . . . . . . 27
14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 28 14. Client Fencing . . . . . . . . . . . . . . . . . . . . . . . 28
15. Security Considerations . . . . . . . . . . . . . . . . . . . 28 15. Security Considerations . . . . . . . . . . . . . . . . . . . 29
15.1. Kerberized File Access . . . . . . . . . . . . . . . . . 29 15.1. Kerberized File Access . . . . . . . . . . . . . . . . . 29
15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 29 15.1.1. Loosely Coupled . . . . . . . . . . . . . . . . . . 30
15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 29 15.1.2. Tightly Coupled . . . . . . . . . . . . . . . . . . 30
16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30
17. References . . . . . . . . . . . . . . . . . . . . . . . . . 30 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 30
17.1. Normative References . . . . . . . . . . . . . . . . . . 30 17.1. Normative References . . . . . . . . . . . . . . . . . . 30
17.2. Informative References . . . . . . . . . . . . . . . . . 31 17.2. Informative References . . . . . . . . . . . . . . . . . 31
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 31 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 31
Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 31 Appendix B. RFC Editor Notes . . . . . . . . . . . . . . . . . . 32
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32
1. Introduction 1. Introduction
In the parallel Network File System (pNFS), the metadata server In the parallel Network File System (pNFS), the metadata server
returns Layout Type structures that describe where file data is returns Layout Type structures that describe where file data is
located. There are different Layout Types for different storage located. There are different Layout Types for different storage
systems and methods of arranging data on storage devices. This systems and methods of arranging data on storage devices. This
document defines the Flexible File Layout Type used with file-based document defines the Flexible File Layout Type used with file-based
data servers that are accessed using the Network File System (NFS) data servers that are accessed using the Network File System (NFS)
protocols: NFSv3 [RFC1813], NFSv4.0 [RFCNFSv4], NFSv4.1 [RFC5661], protocols: NFSv3 [RFC1813], NFSv4.0 [RFCNFSv4], NFSv4.1 [RFC5661],
skipping to change at page 6, line 33 skipping to change at page 6, line 33
LAYOUTCOMMIT (see Section 18.42 of [RFC5661]), the semantics of the LAYOUTCOMMIT (see Section 18.42 of [RFC5661]), the semantics of the
File Layout Type MUST be met (see Section 12.5.4 of [RFC5661]). With File Layout Type MUST be met (see Section 12.5.4 of [RFC5661]). With
a loosely coupled system, a LAYOUTCOMMIT to the metadata server MUST a loosely coupled system, a LAYOUTCOMMIT to the metadata server MUST
be proceeded with a COMMIT to the storage device. It is the be proceeded with a COMMIT to the storage device. It is the
responsibility of the client to make sure the data file is stable responsibility of the client to make sure the data file is stable
before the metadata server begins to query the storage devices about before the metadata server begins to query the storage devices about
the changes to the file. Note that if the client has not done a the changes to the file. Note that if the client has not done a
COMMIT to the storage device, then the LAYOUTCOMMIT might not be COMMIT to the storage device, then the LAYOUTCOMMIT might not be
synchronized to the last WRITE operation to the storage device. synchronized to the last WRITE operation to the storage device.
2.2. Security Models 2.2. Fencing Clients from the Data Server
With loosely coupled storage devices, the metadata server uses With loosely coupled storage devices, the metadata server uses
synthetic uids and gids for the data file, where the uid owner of the synthetic uids and gids for the data file, where the uid owner of the
data file is allowed read/write access and the gid owner is allowed data file is allowed read/write access and the gid owner is allowed
read only access. As part of the layout (see ffds_user and read only access. As part of the layout (see ffds_user and
ffds_group in Section 5.1), the client is provided with the user and ffds_group in Section 5.1), the client is provided with the user and
group to be used in the Remote Procedure Call (RPC) [RFC5531] group to be used in the Remote Procedure Call (RPC) [RFC5531]
credentials needed to access the data file. Fencing off of clients credentials needed to access the data file. Fencing off of clients
is achieved by the metadata server changing the synthetic uid and/or is achieved by the metadata server changing the synthetic uid and/or
gid owners of the data file on the storage device to implicitly gid owners of the data file on the storage device to implicitly
skipping to change at page 8, line 34 skipping to change at page 8, line 34
The client is thus solely responsible for enforcing file permissions The client is thus solely responsible for enforcing file permissions
in a loosely coupled model. To allow loghyr write access, it will in a loosely coupled model. To allow loghyr write access, it will
send an RPC to the storage device with a credential of 1066:1067. To send an RPC to the storage device with a credential of 1066:1067. To
allow garbo read access, it will send an RPC to the storage device allow garbo read access, it will send an RPC to the storage device
with a credential of 1067:1067. The value of the uid does not matter with a credential of 1067:1067. The value of the uid does not matter
as long as it is not the synthetic uid granted it when getting the as long as it is not the synthetic uid granted it when getting the
layout. layout.
While pushing the enforcement of permission checking onto the client While pushing the enforcement of permission checking onto the client
may seem to weaken security, the client may already be responsible may seem to weaken security, the client may already be responsible
for enforcing permissions before modificaations are sent to a server. for enforcing permissions before modifications are sent to a server.
With cached writes, the client is always responsible for tracking who With cached writes, the client is always responsible for tracking who
is modifying a file and making sure to not coalesce requests from is modifying a file and making sure to not coalesce requests from
multiple users into one request. multiple users into one request.
2.3. State and Locking Models 2.3. State and Locking Models
Metadata file OPEN, LOCK, and DELEGATION operations are always Metadata file OPEN, LOCK, and DELEGATION operations are always
executed only against the metadata server. executed only against the metadata server.
The metadata server responds to state changing operations by The metadata server responds to state changing operations by
skipping to change at page 14, line 38 skipping to change at page 14, line 38
LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an LAYOUT4_FLEX_FILES. [RFC5661] specifies the loc_body structure as an
XDR type "opaque". The opaque layout is uninterpreted by the generic XDR type "opaque". The opaque layout is uninterpreted by the generic
pNFS client layers, but obviously must be interpreted by the Flexible pNFS client layers, but obviously must be interpreted by the Flexible
File Layout Type implementation. This section defines the structure File Layout Type implementation. This section defines the structure
of this opaque value, ff_layout4. of this opaque value, ff_layout4.
5.1. ff_layout4 5.1. ff_layout4
<CODE BEGINS> <CODE BEGINS>
/// const FF_FLAGS_NO_LAYOUTCOMMIT = 1;
/// typedef uint32_t ff_flags4;
///
/// struct ff_data_server4 { /// struct ff_data_server4 {
/// deviceid4 ffds_deviceid; /// deviceid4 ffds_deviceid;
/// uint32_t ffds_efficiency; /// uint32_t ffds_efficiency;
/// stateid4 ffds_stateid; /// stateid4 ffds_stateid;
/// nfs_fh4 ffds_fh_vers<>; /// nfs_fh4 ffds_fh_vers<>;
/// fattr4_owner ffds_user; /// fattr4_owner ffds_user;
/// fattr4_owner_group ffds_group; /// fattr4_owner_group ffds_group;
/// }; /// };
/// ///
/// struct ff_mirror4 { /// struct ff_mirror4 {
/// ff_data_server4 ffm_data_servers<>; /// ff_data_server4 ffm_data_servers<>;
/// }; /// };
/// ///
/// struct ff_layout4 { /// struct ff_layout4 {
/// length4 ffl_stripe_unit; /// length4 ffl_stripe_unit;
/// ff_mirror4 ffl_mirrors<>; /// ff_mirror4 ffl_mirrors<>;
/// ff_flags4 ffl_flags;
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
The ff_layout4 structure specifies a layout over a set of mirrored The ff_layout4 structure specifies a layout over a set of mirrored
copies of that portion of the data file described in the current copies of that portion of the data file described in the current
layout segment. This mirroring protects against loss of data in layout segment. This mirroring protects against loss of data in
layout segments. Note that while not explicitly shown in the above layout segments. Note that while not explicitly shown in the above
XDR, each layout4 element returned in the logr_layout array of XDR, each layout4 element returned in the logr_layout array of
skipping to change at page 17, line 30 skipping to change at page 17, line 30
that if using Kerberos for security, the expectation is that these that if using Kerberos for security, the expectation is that these
values will be a name@domain string. values will be a name@domain string.
ffds_efficiency describes the metadata server's evaluation as to the ffds_efficiency describes the metadata server's evaluation as to the
effectiveness of each mirror. Note that this is per layout and not effectiveness of each mirror. Note that this is per layout and not
per device as the metric may change due to perceived load, per device as the metric may change due to perceived load,
availability to the metadata server, etc. Higher values denote availability to the metadata server, etc. Higher values denote
higher perceived utility. The way the client can select the best higher perceived utility. The way the client can select the best
mirror to access is discussed in Section 8.1. mirror to access is discussed in Section 8.1.
ffl_flags is a bitmap that allows the metadata server to inform the
client of particular conditions that may result from the more or less
tight coupling of the storage devices. FF_FLAGS_NO_LAYOUTCOMMIT, can
be set to indicate that the client is not required to send
LAYOUTCOMMIT to the metadata server.
5.2. Interactions Between Devices and Layouts 5.2. Interactions Between Devices and Layouts
In [RFC5661], the File Layout Type is defined such that the In [RFC5661], the File Layout Type is defined such that the
relationship between multipathing and filehandles can result in relationship between multipathing and filehandles can result in
either 0, 1, or N filehandles (see Section 13.3). Some rationals for either 0, 1, or N filehandles (see Section 13.3). Some rationals for
this are clustered servers which share the same filehandle or this are clustered servers which share the same filehandle or
allowing for multiple read-only copies of the file on the same allowing for multiple read-only copies of the file on the same
storage device. In the Flexible File Layout Type, while there is an storage device. In the Flexible File Layout Type, while there is an
array of filehandles, they are independent of the multipathing being array of filehandles, they are independent of the multipathing being
used. If the metadata server wants to provide multiple read-only used. If the metadata server wants to provide multiple read-only
skipping to change at page 23, line 12 skipping to change at page 23, line 23
Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4
operations. operations.
9.2. Layout Usage Statistics 9.2. Layout Usage Statistics
9.2.1. ff_io_latency4 9.2.1. ff_io_latency4
<CODE BEGINS> <CODE BEGINS>
/// struct ff_io_latency4 { /// struct ff_io_latency4 {
/// nfstime4 ffil_min; /// uint64_t ffil_ops_requested;
/// nfstime4 ffil_max; /// uint64_t ffil_bytes_requested;
/// nfstime4 ffil_avg; /// uint64_t ffil_ops_completed;
/// uint32_t ffil_count; /// uint64_t ffil_bytes_completed;
/// uint64_t ffil_bytes_not_delivered;
/// nfstime4 ffil_total_busy_time;
/// nfstime4 ffil_aggregate_completion_time;
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
When determining latencies, the client can collect the minimum via Both operation counts and bytes transferred are kept in the
ffil_min, the maximum via ffil_max, and the average via ffil_avg. ff_io_latency4. READ operations are used for read latencies. Both
Further, ffil_count relates how many data points were collected in WRITE and COMMIT operations are used for write latencies.
the reported period. "Requested" counters track what the client is attempting to do and
"completed" counters track what was done. Note that there is no
requirement that the client only report completed results that have
matching requested results from the reported period.
ffil_bytes_not_delivered is used to track the aggregate number of
bytes requested by not fulfilled due to error conditions.
ffil_total_busy_time is the aggregate time spent with outstanding RPC
calls, ffil_aggregate_completion_time is the sum of all latencies for
completed RPC calls.
Note that LAYOUTSTATS are cummulative, i.e., not reset each time the
operation is sent. If two RPC calls originating from the same NFS
client are processed at the same time by the metdata server, then the
call containing the larger values contains the most recent time
series data.
9.2.2. ff_layoutupdate4 9.2.2. ff_layoutupdate4
<CODE BEGINS> <CODE BEGINS>
/// struct ff_layoutupdate4 { /// struct ff_layoutupdate4 {
/// netaddr4 ffl_addr; /// netaddr4 ffl_addr;
/// nfs_fh4 ffl_fhandle; /// nfs_fh4 ffl_fhandle;
/// ff_io_latency4 ffl_read; /// ff_io_latency4 ffl_read;
/// ff_io_latency4 ffl_write; /// ff_io_latency4 ffl_write;
skipping to change at page 24, line 27 skipping to change at page 25, line 8
/// }; /// };
/// ///
<CODE ENDS> <CODE ENDS>
Recall that [NFSv42] defines io_info4 as: Recall that [NFSv42] defines io_info4 as:
<CODE BEGINS> <CODE BEGINS>
struct io_info4 { struct io_info4 {
uint32_t ii_count; uint64_t ii_count;
uint64_t ii_bytes; uint64_t ii_bytes;
}; };
<CODE ENDS> <CODE ENDS>
With pNFS, the data transfers are performed directly between the pNFS With pNFS, the data transfers are performed directly between the pNFS
client and the storage devices. Therefore, the metadata server has client and the storage devices. Therefore, the metadata server has
no visibility to the I/O stream and cannot use any statistical no visibility to the I/O stream and cannot use any statistical
information about client I/O to optimize data storage location. information about client I/O to optimize data storage location.
ff_iostats4 MAY be used by the client to report I/O statistics back ff_iostats4 MAY be used by the client to report I/O statistics back
 End of changes. 22 change blocks. 
28 lines changed or deleted 58 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/