[Docs] [txt|pdf] [Tracker] [Email] [Nits]



Network File System Version 4                                 S. Faibish
Internet-Draft January 1, 2020                                  D. Black
Intended status: Informational                                  Dell EMC
Expires: January 6, 2021                                      C. Hellwig
                                                            July 6, 2020


          Using the Parallel NFS (pNFS) SCSI/NVMe Layout
              draft-faibish-nfsv4-scsi-nvme-layout-00

Abstract

   This document explains how to use the Parallel Network File System
   (pNFS) SCSI NVMe Layout Types with transports using the
   NVMe over Fabrics protocols. This draft picks the previous SCSI
   over NVMe draft of C. Hellwig and extended it to support all the
   types of transport protocols supported by NVMe transport over fabrics
   additional to the SCSI transport protocol introduced in pNFS SCSI
   Layout. The proposed transport protocols include support for
   several transports and fabrics and support the RDMA transports.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of Internet-Draft Shadow Directories can be accessed at
   https://www.ietf.org/standards/ids/internet-draft-mirror-sites/.

   This Internet-Draft will expire on January 6, 2021.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.









Faibish                   Expires  January 6, 2021              [Page 1]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 2
     1.1. Conventions Used in This Document . . . . . . . . . . . . .  2
     1.2. General Definitions . . . . . . . . . . . . . . . . . . . .  2
   2. SCSI Layout mapping to NVMe . . . . . . . . . . . . . . . . . .  3
     2.1. Volume Identification . . . . . . . . . . . . . . . . . . .  7
     2.2. Client Fencing  . . . . . . . . . . . . . . . . . . . . . .  7
       2.2.1. Reservation Key Generation  . . . . . . . . . . . . . .  8
       2.2.2. MDS Registration and Reservation  . . . . . . . . . . .  8
       2.2.3. Client Registration . . . . . . . . . . . . . . . . . .  8
       2.2.4. Fencing Action  . . . . . . . . . . . . . . . . . . . .  8
       2.2.5. Client Recovery after a Fence Action  . . . . . . . . .  9
     2.3. Volatile write caches . . . . . . . . . . . . . . . . . . . 10
   3. Security Considerations . . . . . . . . . . . . . . . . . . . . 10
   4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 10
   5. Normative References  . . . . . . . . . . . . . . . . . . . . . 11
   Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 11

1. Introduction

   The pNFS Small Computer System Interface (SCSI) layout [RFC8154] is
   layout type that allows NFS clients to directly perform I/O to block
   storage devices while bypassing the MDS. It is specified by using
   concepts from the SCSI protocol family for the data path to the
   storage devices. This documents explains how to access PCI Express,
   RDMA or Fibre Channel devices using the NVM Express protocol [NVME]
   using the SCSI layout. This document does not amend the pNFS SCSI
   layout document in any way, instead of explains how to map the SCSI
   constructs used in the pNFS SCSI layout document to NVMe concepts
   using the NVMe SCSI translation reference.


1.1. Conventions Used in This Document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].



Faibish                 Expires  January 6, 2021                [Page 2]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

1.2. General Definitions

   The following definitions are provided for the purpose of providing
   an appropriate context for the reader.

   Client The "client" is the entity that accesses the NFS server's
   resources. The client may be an application that contains the logic
   to access the NFS server directly. The client may also be the
   traditional operating system client that provides remote file
   system services for a set of applications.

   Server/Controller The "server" is the entity responsible for
   coordinating client access to a set of file systems and is
   identified by a server owner.

2. SCSI Layout mapping to NVMe

   The SCSI layout definition [RFC8154] only references few SCSI
   specific concepts directly.

   NVM Express [NVME] Base Specification revision 1.4 and prior
   revisions define a register level interface for host software to
   communicate with a non-volatile memory subsystem over PCI Express
   (NVMe over PCIe). This specification defines extensions to NVMe
   that enable operation over other interconnects (NVMe over Fabrics).
   The NVM Express Base Specification revision 1.4 is referred to as
   the NVMe Base specification.

   The goal for this draft is to enable an implementer who is
   familiar with the pNFS SCSI layout (RFC 8154) and the NVMe
   standards (both NVMe-oF 1.1 and NVMe 1.4) to implement the
   pNFS SCSI layout over NVMe-oF. The mapping of extensions defined
   in this document refers to a specific NVMe Transport defined in
   an NVMe Transport binding specification. This document refers to
   NVMe Transport binding specification for FC, RDMA and TCP [RFC7525].
   The NVMe Transport binding specification for Fibre
   Channel is defined in INCITS 540 Fibre Channel - Non-Volatile
   Memory Express [FC-NVMe].

   NVMe over Fabrics has the following differences from the NVMe Base
   specification used with SCSI:
   - There is a one-to-one mapping between I/O Submission Queues and
     I/O Completion Queues. NVMe over Fabrics does not support multiple
     I/O Submission Queues being mapped to a single I/O Completion
     Queue;







Faibish                 Expires  January 6, 2021                [Page 3]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

   - NVMe over Fabrics does not define an interrupt mechanism that
     allows a controller to generate a host interrupt. It is the
     responsibility of the host fabric interface (e.g., Host Bus
     Adapter) to generate host interrupts;

   - NVMe over Fabrics does not use the Create I/O Completion Queue,
     Create I/O Submission Queue, Delete I/O Completion Queue, and
     Delete I/O Submission Queue commands. NVMe over Fabrics does
     not use the Admin Submission Queue Base Address (ASQ), Admin
     Completion Queue Base Address (ACQ), and Admin Queue Attributes
     (AQA) properties (i.e., registers in PCI Express). Queues are
     created using the Connect command;
   - NVMe over Fabrics uses the Disconnect command to delete an I/O
     Submission Queue and corresponding I/O Completion Queue;
   - Metadata, if supported, shall be transferred as a contiguous part
     of the logical block. NVMe over Fabrics does not support
     transferring metadata from a separate buffer;
   - NVMe over Fabrics does not support PRPs but requires use of SGLs
     for Admin, I/O, and Fabrics commands. This differs from NVMe over
     PCIe where SGLs are not supported for Admin commands and are
     optional for I/O commands;
   - NVMe over Fabrics does not support Completion Queue flow control.
     This requires that the host ensures there are available Completion
     Queue slots before submitting new commands; and
   - NVMe over Fabrics allows Submission Queue flow control to be
     disabled if the host and controller agree to disable it. If
     Submission Queue flow control is disabled, the host is required
     to ensure that there are available Submission Queue slots before
     submitting new commands.

   NVMe over Fabrics requires the underlying NVMe Transport to provide
   reliable NVMe command and data delivery. An NVMe Transport is an
   abstract protocol layer independent of any physical interconnect
   properties. An NVMe Transport may expose a memory model, a message
   model, or a combination of the two. A memory model is one in which
   commands, responses and data are transferred between fabric nodes
   by performing explicit memory read and write operations while a
   message model is one in which only messages containing command
   capsules, response capsules, and data are sent between fabric nodes.

   The only memory model NVMe Transport supported by NVMe [NVME] is
   PCI Express, as defined in the NVMe Base specification. While
   differences exist between NVMe over Fabrics and NVMe over PCIe
   implementations, both implement the same architecture and command
   sets. But NVMe SCSI Translation reference is only using the NVMe
   over Fabrics not the memory model. NVMe over Fabrics utilizes the
   protocol layering shown in Figure 1. The native fabric
   communication services and the Fabric Protocol and Physical Fabric
   layers in Figure 1 are outside the scope of this specification.



Faibish                 Expires  January 6, 2021                [Page 4]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

                         +-------------------+
                         |  pNFS host SCSI   |
                         | layout over NVMe  |
                         +---------+---------+
                                   |
                                   v
                         +-------------------+
                         | NVMe over Fabrics |
                         +---------+---------+
                                   |
                                   v
                         +-------------------+
                         | Transport Binding |
                         +---------+---------+
                                   |
                                   v
                         +--------------------+
                         | NVMe Transport svc |
                         +---------+----------+
                                   |
                                   v
                         +-------------------+
                         |    NVMe Transport |
                         +---------+---------+
                                   |
                                   v
                        +--------------------+
                        | Fabric Protocol    |
                        +---------+----------+
                                  |
                                  v
                         +-------------------+
                         | Physical Fabric   |
                         +---------+---------+
                                   |
                                   v
                      +------------------------+
                      | pNFS SCSI layout       |
                      | server/NVMe controller |
                      +------------------------+
            Figure 1: pNFS SCSI over NVMe over Fabrics Layering

   An NVM subsystem port may support multiple NVMe Transports if more
   than one NVMe Transport binding specifications exist for the
   underlying fabric (e.g., an NVM subsystem port identified by a
   Port ID may support both iWARP and RoCE). This draft is also
   defining NVMe binding implementation that uses the Transport
   type of RDMA Transport. The RDMA Transport is RDMA Provider
   agnostic. The diagram in Figure 2 illustrates the layering of
   the RDMA Transport and common RDMA providers (iWARP, InfiniBand,
   and RoCE) within the host and NVM subsystem.

Faibish                 Expires  January 6, 2021                [Page 5]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

                         +-------------------+
                         |      NVMe Host    |
                         +---------+---------+
                         |  RDMA Transport   |
                +--------+---+------------+--+---------+
                |    iWARP   | Infiniband |    RoCE    |
                +------------+-----++-----+------------+
                                   || RDMA Fabric
                                   vv
                +------------+------------+--+---------+
                |    iWARP   | Infiniband |    RoCE    |
                +---------+--+------------+---+--------+
                          |  RDMA Transport   |
                          +-------------------+
                          |   NVM Subsystem   |
                          +-------------------+
                Figure 2: RDMA Transport Protocol Layers

   NVMe over Fabrics allows multiple hosts to connect to different
   controllers in the NVM subsystem through the same port. All other
   aspects of NVMe over Fabrics multi-path I/O and namespace sharing
   are equivalent to that defined in the NVMe Base specification.

   An association is established between a host and a controller when
   the host connects to a controller's Admin Queue using the Fabrics
   Connect command. Within the Connect command, the host specifies
   the Host NQN, NVM Subsystem NQN, Host Identifier, and may request a
   specific Controller ID or may request a connection to any available
   controller. The host is the pNFS client and the controller is the
   NFSv4 server. The pNFS clients connect to the server using different
   network protocols and different transports excluding PCIe direct
   connection. While an association exists between a host and
   a controller, only that host may establish connections with I/O
   Queues of that controller.

   NVMe over Fabrics supports both fabric secure channel and NVMe
   in-band authentication. An NVM subsystem may require a host to
   use fabric secure channel, NVMe in-band authentication, or both.
   The Discovery Service indicates if fabric secure channel shall be
   used for an NVM subsystem. The Connect response indicates if NVMe
   in-band authentication shall be used with that controller. For
   SCSI over NVMe over Fabrics only the in-band authentication model
   will be used as the fabric secure channel is only supported with
   PCIe transport memory model not supported by SCSI layout protocol.

   The pNFS SCSI layout uses the Device Identification VPD page (page
   code 0x83) from [SPC4] to identify the devices used by a layout.
   There are several ways to build SCSI Device




Faibish                 Expires  January 6, 2021                [Page 6]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

2.1. Volume Identification


   Identification descriptors from NVMe Identify data included in the
   Controller Identify Attributes specific to NVMe over Fabrics
   specified in the Identify Controller fields in Section 4.1 of
   [NVMEoF]. This document uses a subset of this information to
   identify LUs backing pNFS SCSI layouts.

   To be used as storage devices for the pNFS SCSI layout, NVMe
   devices MUST support the EUI-64 [RFC8154] value in the
   Identify Namespace data, as the methods based on the Serial
   Number for legacy devices might not be suitable for unique
   addressing needs and thus MUST NOT be used. UUID identification
   can be added by
   using a large enough enum value to avoid conflict with whatever
   T10 might do in a future version of the SCSI [SBC3] standard (the
   underlying SCSI field in SPC is 4 bits, so an enum value of 32
   in this draft MUST be used). For NVMe, these identifiers need to
   be obtained via the Namespace Identification Descriptors in NVMe
   1.4 (returned by the Identify command with the CNS field set to
   03h).

2.2. Client Fencing

   The SCSI layout uses Persistent Reservations to provide client
   fencing. For this both the MDS and the Clients have to register a
   key with the storage device, and the MDS has to create a reservation
   on the storage device. The pNFS SCSI protocol implements fencing
   using persistent reservations (PRs), similar to the fencing method
   used by existing shared disk file systems. To allow fencing
   individual systems, each system MUST use a unique persistent
   reservation key. The following is a full mapping of the required
   PR IN and PR OUT SCSI command to NVMe commands which MUST be used
   when using NVMe devices as storage devices for the pNFS SCSI layout.

2.2.1. Reservation Key Generation

   Prior to establishing a reservation on a namespace, a host shall
   become a registrant of that namespace by registering a reservation
   key. This reservation key may be used by the host as a means of
   identifying the registrant (host), authenticating the registrant,
   and preempting a failed or uncooperative registrant. This document
   assigns the burden to generate unique keys to the MDS, which MUST
   generate a key for itself before exporting a volume and a key for
   each client that accesses SCSI layout volumes.

   One important difference between SCSI Persistent Reservations
   and NVMe Reservations is that NVMe reservation keys always apply
   to all controllers used by a host (as indicated by the NVMe Host
   Identifier)

Faibish                 Expires  January 6, 2021                [Page 7]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

   This behavior is somewhat similar to setting the ALL_TG_PT bit when
   registering a SCSI Reservation key, but actually guaranteed to
   work reliably.

2.2.2. MDS Registration and Reservation

   Before returning a PNFS_SCSI_VOLUME_BASE volume to the client, the
   MDS needs to prepare the volume for fencing using NVMeReservations.
   Registering
   a reservation key with a namespace creates an association between
   a host and a namespace. A host that is a registrant of a namespace
   may use any controller with which that host is associated (i.e.,
   that has the same Host Identifier, refer to [NVME]
   section-5.21.1.26) to access that namespace as a registrant.

2.2.3. Client Registration

2.2.3.1 SCSI client

   Before performing the first I/O to a device returned from a
   GETDEVICEINFO operation, the client will register the
   reservation key returned by the MDS with the storage device
   by issuing a "PERSISTENT RESERVE OUT" command with a service
   action of REGISTER with the "SERVICE ACTION RESERVATION KEY" set
   to the reservation key.

2.2.3.2 NVMe Client

   A client registers a reservation key by executing a Reservation
   Register command (refer to [NVME] section 6.11) on the namespace
   with the Reservation Register Action (RREGA) field cleared to
   000b (i.e., Register Reservation Key) and supplying a reservation
   key in the New Reservation Key (NRKEY) field. A client that is a
   registrant of a namespace may register the same reservation key
   value multiple times with the namespace on the same or different
   controllers. There are no restrictions on the reservation key
   value used by hosts with different Host Identifiers.

2.2.4. Fencing Action

2.2.4.1 SCSI client

   In case of a non-responding client, the MDS fences the client by
   issuing a "PERSISTENT RESERVE OUT" command with the service action
   set to "PREEMPT" or "PREEMPT AND ABORT", the "RESERVATION KEY" field
   set to the server's reservation key, the service action "RESERVATION
   KEY" field set to the reservation key associated with the non-
   responding client, and the "TYPE" field set to 8h (Exclusive Access
   - Registrants Only).



Faibish                 Expires  January 6, 2021                [Page 8]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

2.2.4.2 NVMe Client

   A host that is a registrant may preempt a reservation and/or
   registration by executing a Reservation Acquire command (refer to
   section 6.10), setting the Reservation Acquire Action (RACQA) field
   to 001b (Preempt), and supplying the current reservation key
   associated with the host in the Current Reservation Key (CRKEY)
   field. The CRKEY value shall match that used by the registrant to
   register with the namespace. If the CRKEY value does not match,
   then the command is aborted with status Reservation Conflict.

   If the PRKEY field value does not match that of the current
   reservation holder and is equal to 0h, then the command is aborted
   with status Invalid Field in Command. A reservation preempted
   notification occurs on all controllers in the NVM subsystem that
   are associated with hosts that have their registrations removed as
   a result of actions taken in this section except those associated
   with the host that issued the Reservation Release command. After
   the MDS preempts a client, all client I/O to the LU fails.
   The client SHOULD at this point return any layout that refers to
   the device ID that points to the LU.

2.2.5. Client Recovery after a Fence Action

   A client that detects a NVMe status codes (I/O
   error) on the storage devices MUST commit all layouts that use the
   storage device through the MDS, return all outstanding layouts for
   the device, forget the device ID, and unregister the reservation
   key.

   Future GETDEVICEINFO calls MAY refer to the storage device again,
   in which case the client will perform a new registration based on
   the key provided. If a reservation holder attempts to obtain a
   reservation of a different type on a namespace for which that host
   already is the reservation holder, then the command is aborted with
   status Reservation Conflict. It is not an error if a reservation
   holder attempts to obtain a reservation of the same type on a
   namespace for which that host already is the reservation holder.

   NVMe over Fabrics [NVMEoF] utilizes the same controller
   architecture as that defined in the NVMe Base specification [NVME].











Faibish                 Expires  January 6, 2021                [Page 9]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020


   This includes using Submission and Completion Queues to execute
   commands between a host and a controller. Section 8.20 of [NVME]
   base specifications describes the relationship between a controller
   (MDS) and a namespace associated to the Clients. In a static
   controller model used by SCSI layout, controllers that may be
   allocated to a particular Client may have different state at the
   time the association is established.

2.3. Volatile write caches

   The Volatile Write Cache Enable (WCE) bit (i.e., bit 00) in
   the Volatile Write Cache Feature (Feature Identifier 06h)
   is the Write Cache Enable field in the NVMe Get Features command,
   see Section-5.21.1.6 of [NVME]. If a write cache is enable on a
   NVMe device used as a storage device for the pNFS SCSI layout, the
   MDS  MUST ensure to use the NVMe FLUSH command to flush the
   volatile write cache. If there is no volatile write cache on the
   server, then attempts to access this NVMe Feature cause errors.
   The Get Features command specifying the Volatile Write Cache feature
   identifier is expected to fail with Invalid Field in Command status.

3. Security Considerations

   Since no protocol changes are proposed here, no security
   considerations apply. But the protocol is assuming that NVMe
   Authentication commands are implemented in the NVMe
   Security Protocol as the format of the data to be transferred is
   dependent on the Security Protocol. Authentication Receive/Response
   commands return the appropriate data corresponding to an
   Authentication Send command as defined by the rules of the
   Security Protocol. As the current draft is only supporting
   MVMe over fabric In-band protocol the Authentication requirements
   for security commands are based on the security protocol indicated
   by the SECP field in the command and DO NOT require authentication
   when used for NVMe in-band authentication. When used for other
   purposes, in-band authentication of the commands is required.


4. IANA Considerations
   The document does not require any actions by IANA.











Faibish                 Expires  January 6, 2021               [Page 10]


Internet-Draft      pNFS SCSI/NVMe Layout over Fabrics         July 2020

5. Normative References

   [NVME] NVM Express, Inc., "NVM Express Revision 1.4", June 10, 2019.

   [NVMEoF] "NVM Express over Fabrics Revision 1.1", July 26, 2019

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", March 1997.

   [RFC8154] Hellwig, C., "Parallel NFS (pNFS) Small Computer System
             Interface (SCSI) Layout", May 2017.

   [SBC3] INCITS Technical Committee T10, "SCSI Block Commands-3",
          ANSI INCITS INCITS 514-2014, ISO/IEC 14776-323, 2014.

   [SPC4] INCITS Technical Committee T10, "SCSI Primary Commands-4",
          ANSI INCITS 513-2015, 2015.

   [FC-NVMe] INCITS Technical Committee T10, "Fibre Channel -
             Non-Volatile Memory Express", ANSI INCITS 540, 2018.

   [RFC7525] Sheffer, Y., "Recommendations for Secure Use of Transport
             Layer Security (TLS) and Datagram Transport Layer Security
             (DTLS)" alsa known as BCP 195.

Author's Address

   Sorin Faibish
   Dell EMC
   228 South Street
   Hopkinton, MA  01774
   United States of America

   Phone: +1 508-249-5745
   Email: faibish.sorin@dell.com

   David Black
   Dell EMC
   176 South Street
   Hopkinton, MA  01748
   United States of America

   Phone: +1 774-350-9323
   Email: david.black@dell.com

   Christoph Hellwig
   Email: hch@lst.de





Faibish                 Expires  January 6, 2021               [Page 11]

Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/