draft-ietf-ips-iwarp-da-05.txt   rfc5047.txt 
INTERNET DRAFT Mallikarjun Chadalapaka Network Working Group M. Chadalapaka
draft-ietf-ips-iwarp-da-05.txt HP Request for Comments: 5047 HP
John Hufferd Category: Informational J. Hufferd
IBM Brocade Inc.
Julian Satran J. Satran
IBM IBM
Hemal Shah H. Shah
Intel Broadcom Corporation
October 2007
Expires
Datamover Architecture for iSCSI (DA)
Status of this Memo
By submitting this Internet-Draft, each author represents
that any applicable patent or other IPR claims of which he or
she is aware have been or will be disclosed, and any of which
he or she becomes aware will be disclosed, in accordance with
Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet
Engineering Task Force (IETF), its areas, and its working
groups. Note that other groups may also distribute working
documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of DA: Datamover Architecture for
six months and may be updated, replaced, or obsoleted by the Internet Small Computer System Interface (iSCSI)
other documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other
than as "work in progress."
The list of current Internet-Drafts can be accessed at Status of This Memo
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed This memo provides information for the Internet community. It does
at http://www.ietf.org/shadow.html. not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Abstract Abstract
iSCSI is a SCSI transport protocol that maps the SCSI family
of application protocols onto TCP/IP. Datamover Architecture The Internet Small Computer System Interface (iSCSI) is a SCSI
for iSCSI (DA) defines an abstract model in which the transport protocol that maps the SCSI family of application protocols
movement of data between iSCSI end nodes is logically onto TCP/IP. Datamover Architecture for iSCSI (DA) defines an
separated from the rest of the iSCSI protocol in order to abstract model in which the movement of data between iSCSI end nodes
allow iSCSI to adapt to innovations available in new IP is logically separated from the rest of the iSCSI protocol in order
transports. While DA defines the architectural functions to allow iSCSI to adapt to innovations available in new IP
required of the class of Datamover protocols, it does not transports. While DA defines the architectural functions required of
define any specific Datamover protocols. Each such Datamover the class of Datamover protocols, it does not define any specific
protocol, to be defined in a separate document, provides a Datamover protocols. Each such Datamover protocol, defined in a
reliable transport for all iSCSI PDUs, but actually moves the separate document, provides a reliable transport for all iSCSI PDUs,
data required for certain iSCSI PDUs without involving the but actually moves the data required for certain iSCSI PDUs without
remote iSCSI layer itself. This document begins with an involving the remote iSCSI layer itself. This document begins with
introduction of a few new abstractions, defines a layered an introduction of a few new abstractions, defines a layered
architecture for iSCSI and Datamover protocols, and then architecture for iSCSI and Datamover protocols, and then models the
models the interactions within an iSCSI end node between the interactions within an iSCSI end node between the iSCSI layer and the
iSCSI layer and the Datamover layer that happen in order to Datamover layer that happen in order to transparently perform remote
transparently perform remote data movement within an IP data movement within an IP fabric. It is intended that this
fabric. It is intended that this definition would help map definition will help map iSCSI to generic Remote Direct Memory Access
iSCSI to generic RDMA-capable IP fabrics in the future (RDMA)-capable IP fabrics in the future comprising TCP, the Stream
comprising TCP, SCTP, and possibly other underlying network Control Transmission Protocol (SCTP), and possibly other underlying
transport layers such as InfiniBand. network transport layers, such as InfiniBand.
Table of Contents Table of Contents
1 Definitions and acronyms ...............................5 1. Motivation ......................................................4
1.1 Definitions ............................................5 1.1. Intent .....................................................4
1.2 Acronyms ...............................................5 1.2. Interpretation of Requirements .............................5
2 Motivation .............................................7 2. Definitions and Acronyms ........................................5
2.1 Intent .................................................7 2.1. Definitions ................................................5
2.2 Interpretation of Requirements .........................8 2.2. Acronyms ...................................................6
3 Architectural layering of iSCSI and Datamover layers ...9 3. Architectural Layering of iSCSI and Datamover Layers ............7
4 Design Overview .......................................11 4. Design Overview .................................................9
5 Architectural Concepts ................................13 5. Architectural Concepts .........................................10
5.1 iSCSI PDU types .......................................13 5.1. iSCSI PDU Types ...........................................10
5.1.1 iSCSI data-type PDUs.................................13 5.1.1. iSCSI Data-Type PDUs ...............................10
5.1.2 iSCSI control-type PDUs..............................14 5.1.2. iSCSI Control-Type PDUs ............................11
5.2 Data_Descriptor .......................................14 5.2. Data_Descriptor ...........................................11
5.3 Connection_Handle .....................................14 5.3. Connection_Handle .........................................11
5.4 Operational Primitive .................................15 5.4. Operational Primitive .....................................12
5.5 Transport Connection ..................................16 5.5. Transport Connection ......................................13
6 Datamover layer and Datamover protocol ................17 6. Datamover Layer and Datamover Protocol .........................13
7 Functional Overview ...................................19 7. Functional Overview ............................................14
7.1 Startup ...............................................19 7.1. Startup ...................................................14
7.2 Full Feature Phase ....................................19 7.2. Full Feature Phase ........................................15
7.3 Wrapup ................................................20 7.3. Wrap-up ...................................................15
8 Operational Primitives provided by the Datamover layer 22 8. Operational Primitives Provided by the Datamover Layer .........16
8.1 Send_Control ..........................................22 8.1. Send_Control ..............................................16
8.2 Put_Data ..............................................23 8.2. Put_Data ..................................................17
8.3 Get_Data ..............................................24 8.3. Get_Data ..................................................17
8.4 Allocate_Connection_Resources .........................24 8.4. Allocate_Connection_Resources .............................18
8.5 Deallocate_Connection_Resources .......................25 8.5. Deallocate_Connection_Resources ...........................19
8.6 Enable_Datamover ......................................26 8.6. Enable_Datamover ..........................................19
8.7 Connection_Terminate ..................................26 8.7. Connection_Terminate ......................................20
8.8 Notice_Key_Values .....................................27 8.8. Notice_Key_Values .........................................20
8.9 Deallocate_Task_Resources .............................27 8.9. Deallocate_Task_Resources .................................20
9 Operational Primitives provided by the iSCSI layer ....29 9. Operational Primitives Provided by the iSCSI Layer .............21
9.1 Control_Notify ........................................29 9.1. Control_Notify ............................................21
9.2 Connection_Terminate_Notify ...........................30 9.2. Connection_Terminate_Notify ...............................22
9.3 Data_Completion_Notify ................................30 9.3. Data_Completion_Notify ....................................22
9.4 Data_ACK_Notify .......................................31 9.4. Data_ACK_Notify ...........................................23
10 Datamover Interface (DI) ..............................33 10. Datamover Interface (DI) ......................................23
10.1 Overview.............................................33 10.1. Overview .................................................23
10.2 Interactions for handling asynchronous notifications.33 10.2. Interactions for Handling Asynchronous Notifications .....24
10.2.1 Connection termination .............................33 10.2.1. Connection Termination ............................24
10.2.2 Data transfer completion ...........................33 10.2.2. Data Transfer Completion ..........................24
10.2.3 Data acknowledgement ...............................34 10.2.3. Data Acknowledgement ..............................25
10.3 Interactions for sending an iSCSI PDU................35 10.3. Interactions for Sending an iSCSI PDU ....................25
10.3.1 SCSI Command .......................................35 10.3.1. SCSI Command ......................................26
10.3.2 SCSI Response ......................................36 10.3.2. SCSI Response .....................................26
10.3.3 Task Management Function Request ...................36 10.3.3. Task Management Function Request ..................26
10.3.4 Task Management Function Response ..................37 10.3.4. Task Management Function Response .................27
10.3.5 SCSI Data-out & SCSI Data-in .......................37 10.3.5. SCSI Data-Out and SCSI Data-In ....................27
10.3.6 Ready To Transfer (R2T) ............................37 10.3.6. Ready To Transfer (R2T) ...........................28
10.3.7 Asynchronous Message ...............................38 10.3.7. Asynchronous Message ..............................28
10.3.8 Text Request .......................................38 10.3.8. Text Request ......................................28
10.3.9 Text Response ......................................38 10.3.9. Text Response .....................................28
10.3.10 Login Request ....................................39 10.3.10. Login Request ....................................29
10.3.11 Login Response ...................................39 10.3.11. Login Response ...................................29
10.3.12 Logout Command ...................................40 10.3.12. Logout Command ...................................29
10.3.13 Logout Response ..................................40 10.3.13. Logout Response ..................................30
10.3.14 SNACK Request ....................................40 10.3.14. SNACK Request ....................................30
10.3.15 Reject ...........................................41 10.3.15. Reject ...........................................30
10.3.16 NOP-Out ..........................................41 10.3.16. NOP-Out ..........................................30
10.3.17 NOP-In ...........................................41 10.3.17. NOP-In ...........................................30
10.4 Interactions for receiving an iSCSI PDU..............41 10.4. Interactions for Receiving an iSCSI PDU ..................31
10.4.1 General Control-type PDU notification ..............42 10.4.1. General Control-Type PDU Notification .............31
10.4.2 SCSI Data Transfer PDUs ............................42 10.4.2. SCSI Data Transfer PDUs ...........................31
10.4.3 Login Request ......................................43 10.4.3. Login Request .....................................32
10.4.4 Login Response .....................................44 10.4.4. Login Response ....................................32
11 Security Considerations ...............................45 11. Security Considerations .......................................33
11.1 Architectural Considerations.........................45 11.1. Architectural Considerations .............................33
11.2 Wire Protocol Considerations.........................46 11.2. Wire Protocol Considerations .............................33
12 IANA Considerations ...................................47 12. References ....................................................34
13 References and Bibliography ...........................48 12.1. Normative References .....................................34
13.1 Normative References.................................48 12.2. Informative References ...................................34
13.2 Informative References...............................48 Appendix A. Design Considerations and Examples ....................35
14 Authors' Addresses ....................................49 A.1. Design Considerations for a Datamover Protocol ............35
15 Acknowledgements ......................................50 A.2. Examples of Datamover Interactions ........................35
16 Appendix ..............................................54 Acknowledgements ..................................................44
16.1 Design considerations for a Datamover protocol.......54
16.2 Examples of Datamover interactions...................54
17 Full Copyright Statement ..............................64
18 Intellectual Property Statement .......................65
Table of Figures Table of Figures
Figure 1 Datamover Architecture diagram, with the RDMAP Figure 1. Datamover Architecture Diagram, with the RDMAP Example ...8
example......................................................9 Figure 2. A Successful iSCSI Login on Initiator ...................37
Figure 2 A successful iSCSI login on initiator..............56 Figure 3. A Successful iSCSI Login on Target ......................37
Figure 3 A successful iSCSI login on target.................56 Figure 4. A Failed iSCSI Login on Initiator .......................38
Figure 4 A failed iSCSI login on initiator..................57 Figure 5. A Failed iSCSI Login on Target ..........................38
Figure 5 A failed iSCSI login on target.....................57 Figure 6. iSCSI Does Not Enable the Datamover .....................39
Figure 6 iSCSI does not enable the Datamover................58 Figure 7. A Normal iSCSI Connection Termination ...................40
Figure 7 A normal iSCSI connection termination..............59 Figure 8. An Abnormal iSCSI Connection Termination ................40
Figure 8 An abnormal iSCSI connection termination...........59 Figure 9. A SCSI Write Data Transfer ..............................41
Figure 9 A SCSI Write data transfer.........................60 Figure 10. A SCSI Read Data Transfer ..............................42
Figure 10 A SCSI Read data transfer.........................61 Figure 11. A SCSI Read Data Acknowledgement .......................43
Figure 11 A SCSI Read data acknowledgement..................62 Figure 12. Task Resource Cleanup on Abort .........................44
Figure 12 Task resource cleanup on abort...................63
1 Definitions and acronyms 1. Motivation
1.1 Definitions 1.1. Intent
I/O Buffer - A buffer that is used in a SCSI Read or Write There are relatively new standard protocols that enable Remote Direct
operation so SCSI data may be sent from or received into Memory Access (RDMA) and Remote Direct Data Placement (RDDP)
that buffer. technologies to work over IP fabrics. The principal value
proposition of these technologies is that they enable one end node to
place data in the final intended buffer on the remote end node, thus
eliminating the need for a receive path data copy that moves the data
to its final location. The data copy avoidance in turn eliminates
unnecessary memory bandwidth consumption, substantially decreases the
reassembly buffer size requirements, and preserves CPU cycles that
would otherwise be spent in copying.
Datamover protocol - A Datamover protocol is a data transfer The iSCSI specification [RFC3720] defines a very detailed data
wire protocol for iSCSI that meets the requirements transfer model that employs SCSI Data-In PDUs, SCSI Data-Out PDUs,
stated in section 6. and R2T PDUs, in addition to the SCSI Command and SCSI Response PDUs
that respectively create and conclude the task context for the data
transfer. In the traditional iSCSI model, the iSCSI protocol layer
plays the central role in pacing the data transfer and carrying out
the ensuing data transfer itself. An alternative architecture would
be for iSCSI to delegate a large part of this data transfer role to a
separate protocol layer exclusively designed to move data, which in
turn is possibly aided by a data movement and placement technology
such as RDMA.
Datamover layer - A Datamover layer is a protocol layer If iSCSI were operating in such RDMA environments, iSCSI would be
within an end node that implements the Datamover shielded from the low-level data transfer mechanics but would only be
protocol. privy to the conclusion of the requested data transfer. Thus, there
would be an effective "off-loading" of the work that an iSCSI
protocol layer is expected to perform, compared to today's iSCSI end
nodes. For such RDMA environments, it is highly desirable that there
be a standard architecture to separate the data movement part of the
iSCSI protocol definition from the rest of the iSCSI functionality.
This architecture precisely defines what a Datamover layer is and
also describes the model of interactions between the iSCSI layer and
the Datamover layer (Section 6). In order to satisfy this need, this
document presents a Datamover Architecture for iSCSI (DA) and
summarizes a reasonable model for interactions between the iSCSI
layer and the Datamover layer for each of the iSCSI PDUs that are
defined in [RFC3720]. Note that while DA is motivated by the advent
of RDMA over TCP/IP technology, the architecture is not dependent on
RDMA in its design. DA is intended to be a generic architectural
framework for allowing different types of Datamovers based on
different types of RDMA and transport protocols. Adoption of this
model will help iSCSI proliferate into more environments.
Datamover-assisted - An iSCSI connection is said to be 1.2. Interpretation of Requirements
"Datamover-assisted" when a Datamover layer is enabled
for moving control and data information on that iSCSI
connection.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL This document introduces certain architectural abstractions and
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and builds an abstract functional interface model between iSCSI and
"OPTIONAL" in this document are to be interpreted as Datamover protocol layers based on those abstractions. This
described in [RFC2119]. architectural style is motivated by the following desires:
1.2 Acronyms a) Provide guidance to Datamover protocol designers with respect
to the functional boundary between iSCSI and the Datamover
protocols. This guidance is critical since a significant part
of the [RFC3720] protocol definition is left unchanged by DA
architecture and the iSCSI notions from [RFC3720] (e.g., tasks,
ITTs) are leveraged by the Datamover protocol.
Acronym Definition b) Aid existing iSCSI implementations to rapidly adapt to DA
architecture, largely by leveraging the architectural
abstractions into implementation constructs -- e.g., functions,
APIs, modules.
However, note that DA architecture does not intend to impose any
implementation specifics per se. When a DA architectural concept
(e.g., Operational Primitive) is described as mandatory ("MUST") or
recommended ("SHOULD") of a layer (iSCSI or Datamover) in this
document, the intent is that an implementation respectively MUST or
SHOULD produce the same protocol action as what the model describes.
Specifically, no implementation compliance in terms of names, modules
or API arguments etc. is implied by this Architecture by such use of
[RFC2119] terms, only a functional compliance is sought.
2. Definitions and Acronyms
2.1. Definitions
I/O Buffer - A buffer that is used in a SCSI Read or Write operation
so that SCSI data may be sent from or received by the buffer.
Datamover protocol - A Datamover protocol is a data transfer wire
protocol for iSCSI that meets the requirements stated in Section
6.
Datamover layer - A Datamover layer is a protocol layer within an end
node that implements the Datamover protocol.
Datamover-assisted - An iSCSI connection is said to be "Datamover-
assisted" when a Datamover layer is enabled for moving control and
data information on that iSCSI connection.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
2.2. Acronyms
Acronym Definition
------------------------------------------------------------- -------------------------------------------------------------
DA Datamover Architecture for iSCSI DA Datamover Architecture for iSCSI
DDP Direct Data Placement Protocol DDP Direct Data Placement Protocol
DI Datamover Interface DI Datamover Interface
IANA Internet Assigned Numbers Authority IANA Internet Assigned Numbers Authority
skipping to change at page 6, line 25 skipping to change at page 7, line 4
R2T Ready To Transfer R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number R2TSN Ready To Transfer Sequence Number
RDMA Remote Direct Memory Access RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments RFC Request For Comments
SAM SCSI Architecture Model SAM SCSI Architecture Model
SCSI Small Computer Systems Interface SCSI Small Computer Systems Interface
SN Sequence Number SN Sequence Number
SNACK Selective Negative Acknowledgment - also SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for Data
Sequence Number Acknowledgement for data
TCP Transmission Control Protocol TCP Transmission Control Protocol
TTT Target Transfer Tag TTT Target Transfer Tag
2 Motivation 3. Architectural Layering of iSCSI and Datamover Layers
2.1 Intent
There are relatively new standard protocols that enable
Remote Direct Memory Access (RDMA) and Remote Direct Data
Placement (RDDP) technologies to work over IP fabrics. The
principal value proposition of these technologies is that
they enable one end node to place data in the final intended
buffer on the remote end node, thus eliminating the data copy
that traditionally happens in the receive path to move the
data to the final buffer. The data copy avoidance in turn
eliminates unnecessary memory bandwidth consumption, substan-
tially decreases the reassembly buffer size requirements, and
preserves CPU cycles that would otherwise be spent in
copying.
The iSCSI specification ([RFC3720]) defines a very detailed
data transfer model that employs SCSI Data-In PDUs, SCSI
Data-Out PDUs, and R2T PDUs, in addition to the SCSI Command
and SCSI Response PDUs that respectively create and conclude
the task context for the data transfer. In the traditional
iSCSI model, the iSCSI protocol layer plays the central role
in pacing the data transfer and carrying out the ensuing data
transfer itself. An alternative architecture would be for
iSCSI to delegate a large part of this data transfer role to
a separate protocol layer exclusively designed to move data,
which in turn is possibly aided by a data movement and
placement technology such as RDMA.
If iSCSI were operating in such RDMA environments, iSCSI
would be shielded from the low-level data transfer mechanics
but would only be privy to the conclusion of the requested
data transfer Thus, there would be an effective "off-
loading" of the work that an iSCSI protocol layer is expected
to perform, compared to today's iSCSI end nodes. For such
RDMA environments, it is highly desirable that there be a
standard architecture to separate the data movement part of
the iSCSI protocol definition from the rest of the iSCSI
functionality. This architecture precisely defines what a
Datamover layer is and also describes the model of
interactions between the iSCSI layer and the Datamover layer
(section 6). In order to satisfy this need, this document
presents a Datamover Architecture for iSCSI(DA) and also
summarizes a reasonable model for interactions between the
iSCSI layer and the Datamover layer for each of the iSCSI
PDUs that are defined in [RFC3720]. Note that while DA is
motivated by the advent of RDMA over TCP/IP technology, the
architecture is not dependent on RDMA in its design. DA is
intended to be a generic architectural framework for allowing
different types of Datamovers based on different types of
RDMA and transport protocols. Adoption of this model will
help iSCSI proliferate into more environments.
2.2 Interpretation of Requirements
This draft introduces certain architectural abstractions and
builds an abstract functional interface model between iSCSI
and Datamover protocol layers based on those abstractions.
This architectural style is motivated by the following
desires:
a) Provide guidance to Datamover protocol designers
with respect to the functional boundary between
iSCSI and the Datamover protocols. This guidance is
critical since a significant part of the [RFC3720]
protocol definition is left unchanged by DA
architecture and the iSCSI notions from [RFC3720]
(e.g., tasks, ITTs) are leveraged by the Datamover
protocol.
b) Aid existing iSCSI implementations to rapidly adapt
to DA architecture, largely by leveraging the
architectural abstractions also into implementation
constructs - e.g., functions, APIs, modules.
However, note that DA architecture does not intend to impose
any implementation specifics per se. When a DA architectural
concept (e.g., Operational Primitive) is described as
mandatory ("MUST") or recommended ("SHOULD") of a layer
(iSCSI or Datamover) in this document, the intent is that an
implementation respectively MUST or SHOULD produce the same
protocol action as what the model describes. Specifically,
no implementation compliance in terms of names, modules or
API arguments etc. is implied by this Architecture by such
use of [RFC2119] terms, only a functional compliance is
sought.
3 Architectural layering of iSCSI and Datamover layers
Figure 1 illustrates an example of the architectural layering Figure 1 illustrates an example of the architectural layering of
of iSCSI and Datamover layers, in conjunction with a TCP/IP iSCSI and Datamover layers, in conjunction with a TCP/IP
implementation of RDMAP/DDP ([DDP]) layers in an iSCSI end implementation of RDMAP/DDP ([DDP]) layers in an iSCSI end node.
node. Note that RDMAP/DDP/MPA, and TCP protocol layers are Note that RDMAP/DDP/MPA and TCP protocol layers are shown here only
shown here only as an example and in reality, DA is as an example, and in reality, DA is completely oblivious to protocol
completely oblivious to protocol layers below the Datamover layers below the Datamover layer. The RDMAP/DDP/MPA protocol stack
layer. The RDMAP/DDP/MPA protocol stack provides a generic provides a generic transport service with direct data placement.
transport service with direct data placement. There is no There is no need to tailor the implementation of this protocol stack
need to tailor the implementation of this protocol stack to to the specific ULP to benefit from these services.
the specific ULP to benefit from these services.
Initiator stack Target stack Initiator stack Target stack
+----------------+ SCSI application +----------------+ +----------------+ SCSI application +----------------+
| SCSI Layer | protocols | SCSI Layer | | SCSI Layer | protocols | SCSI Layer |
+----------------+ +----------------+ +----------------+ +----------------+
^ ^ ^ ^
| | | |
v v v v
+----------------+ iSCSI protocol +----------------+ +----------------+ iSCSI protocol +----------------+
skipping to change at page 9, line 53 skipping to change at page 8, line 40
| | network | | | network | | | network | | | network |
| | transport| | | transport | | | transport| | | transport |
| v | | v | | v | | v |
|+---------------+ | | +----------------+ | |+---------------+ | | +----------------+ |
|| TCP Layer | | TCP protocol | | TCP Layer | | || TCP Layer | | TCP protocol | | TCP Layer | |
|+---------------+ | | +----------------+ | |+---------------+ | | +----------------+ |
| ^ | | ^ | | ^ | | ^ |
+-------+----------+ +---------+-----------+ +-------+----------+ +---------+-----------+
+------------------------------------------+ +------------------------------------------+
Figure 1 Datamover Architecture diagram, with the Figure 1. Datamover Architecture Diagram,
RDMAP example with the RDMAP Example
The scope of this document is limited to: The scope of this document is limited to:
1. Defining the notion of a Datamover layer and a Datamover 1. Defining the notion of a Datamover layer and a Datamover
protocol (section 6), protocol (Section 6).
2. Defining the functionality distribution between the 2. Defining the functionality distribution between the iSCSI layer
iSCSI layer and the Datamover layer along with the and the Datamover layer, along with the communication model
communication model between the two (Operational between the two (Operational Primitives).
Primitives), and,
3. Modeling the interactions between the blocks labeled as 3. Modeling the interactions between the blocks labeled as "iSCSI
"iSCSI Layer" and "Datamover Layer" in Figure 1 - i.e. Layer" and "Datamover Layer" in Figure 1 -- i.e., defining the
defining the interface labeled as "DI" in the figure - interface labeled "DI" in the figure -- for each defined iSCSI
for each defined iSCSI PDU, based on the Operational PDU, based on the Operational Primitives.
Primitives.
4 Design Overview 4. Design Overview
This document discusses and defines a model for interactions This document discusses and defines a model for interactions between
between the iSCSI layer and a "Datamover layer" (see section the iSCSI layer and a "Datamover layer" (see Section 6) operating
6) operating within an iSCSI end node, presumably within an iSCSI end node, presumably communicating with one or more
communicating with one or more iSCSI end nodes with similar iSCSI end nodes with similar layering. The model for interactions
layering. The model for interactions for handling different for handling different iSCSI operations is called the "Datamover
iSCSI operations is called the "Datamover Interface" (DI, Interface" (DI, Section 10), while the architecture itself is called
section 10), while the architecture itself is called the "Datamover Architecture for iSCSI" (DA). It is likely that the
"Datamover Architecture for iSCSI" (DA). It is likely that architecture will have implications on the Datamover wire protocols
the architecture will have implications on the Datamover wire as DA places certain requirements and functionality expectations on
protocols as DA places certain requirements and functionality the Datamover layer. However, this document itself neither defines
expectations on the Datamover layer. However, this document any new wire protocol for the Datamover layer, nor any potential
itself neither defines any new wire protocol for the modifications to the iSCSI wire protocol to employ the Datamover
Datamover layer, nor any potential modifications to the iSCSI layer. The scope of this document is strictly limited to specifying
wire protocol to employ the Datamover layer. The scope of the architectural framework and the minimally required interactions
this document is strictly limited to specifying the that happen within an iSCSI end node to leverage the Datamover layer.
architectural framework and the minimally required
interactions that happen within an iSCSI end node to leverage
the Datamover layer.
The design ideas behind DA can be summarized thus - The design ideas behind DA can be summarized as follows:
1) DA defines an abstract functional interface model of iSCSI 1) DA defines an abstract functional interface model of the iSCSI
layer's interactions with a Datamover layer below - i.e. DA layer's interactions with a Datamover layer below -- i.e., DA
models the interactions between the logical "bottom" models the interactions between the logical "bottom" interface
interface of iSCSI and the logical "top" interface of a of iSCSI and the logical "top" interface of a Datamover.
Datamover.
2) DA guides the wire protocol for a Datamover layer by 2) DA guides the wire protocol for a Datamover layer by defining
defining the iSCSI knowledge that the Datamover layer may the iSCSI knowledge that the Datamover layer may utilize in its
utilize in its protocol definition (as an example, this protocol definition (as an example, this document completely
draft completely limits the notion of "iSCSI session" to limits the notion of "iSCSI session" to the iSCSI layer).
the iSCSI layer).
3) DA is designed to allow implementing the Datamover layer 3) DA is designed to allow implementation of the Datamover layer
either in hardware or in software. either in hardware or in software.
4) DA is not a wire protocol spec, but an architecture that 4) DA is not a wire protocol spec, but an architecture that also
also models the interactions between iSCSI and Datamover models the interactions between iSCSI and Datamover layers
layers operating within an iSCSI end node. operating within an iSCSI end node.
5) DA by design seeks to model the iSCSI-Datamover 5) DA by design seeks to model the iSCSI-Datamover interactions in
interactions in a way that the modeling is independent of a way that the modeling is independent of the specifics of
the specifics of either a particular iSCSI revision, or a either a particular iSCSI revision or an instantiation of a
specific instantiation of a Datamover layer. Datamover layer.
6) DA introduces and relies on the notion of a defined set of 6) DA introduces and relies on the notion of a defined set of
Operational Primitives (could be seen as entry point Operational Primitives (could be seen as entry point
definitions in implementation terms) provided by each layer definitions in implementation terms) provided by each layer to
to the other to carry out the request-response the other to carry out the request-response interactions.
interactions.
7) DA is intended to allow Datamover protocol definitions with 7) DA is intended to allow Datamover protocol definitions with
minimal changes to existing iSCSI implementations. minimal changes to existing iSCSI implementations.
8) DA is designed to allow the iSCSI layer to completely rely 8) DA is designed to allow the iSCSI layer to completely rely on
on the Datamover layer for all the data transport needs. the Datamover layer for all data transport needs.
9) DA models the architecturally required minimal interactions 9) DA models the architecturally required minimal interactions
between an operational iSCSI layer and a Datamover layer to between an operational iSCSI layer and a Datamover layer to
realize the iSCSI-transparent data movement. There may be realize the iSCSI-transparent data movement. There may be
several other interactions in a typical implementation in several other interactions in a typical implementation in order
order to bootstrap a Datamover layer (or an iSCSI layer) to bootstrap a Datamover layer (or an iSCSI layer) into
into operation, and they are outside the scope of this operation, but they are outside the scope of this document.
document.
Note that in summary, DA is architected to support many Note that in summary, DA is architected to support many different
different Datamover protocols operating under the iSCSI Datamover protocols operating under the iSCSI layer. One such
layer. One such example of a Datamover protocol is iSER example of a Datamover protocol is iSER [iSER].
([iSER]).
5 Architectural Concepts 5. Architectural Concepts
5.1 iSCSI PDU types 5.1. iSCSI PDU Types
This section defines the iSCSI PDU classification This section defines the iSCSI PDU classification terminology, as
terminology, as defined and used in this document. Out of defined and used in this document. Out of the set of legal iSCSI
the set of legal iSCSI PDUs defined in [RFC3720], as we will PDUs defined in [RFC3720], as we will see in Section 5.1.1, the iSCSI
see in section 5.1.1, the iSCSI layer does not request a SCSI layer does not request a SCSI Data-Out PDU carrying solicited data
Data-Out PDU carrying solicited data for transmission across for transmission across the Datamover Interface per this
the Datamover Interface per this architecture. For this architecture. For this reason, the SCSI Data-Out PDU carrying
reason, the SCSI Data-Out PDU carrying solicited data is solicited data is excluded in the iSCSI PDU classification we
excluded in the iSCSI PDU classification we introduce in this introduce in this section (for SCSI Data-Out PDUs for unsolicited
section (for SCSI Data-Out PDUs for unsolicited Data, see Data, see Section 5.1.2). The rest of the legal iSCSI PDUs that may
section 5.1.2). The rest of the legal iSCSI PDUs that may be be exchanged across the Datamover Interface are defined to consist of
exchanged across the Datamover Interface are defined to two classes:
consist of two classes:
1) iSCSI data-type PDUs 1) iSCSI data-type PDUs
2) iSCSI control-type PDUs 2) iSCSI control-type PDUs
5.1.1 iSCSI data-type PDUs 5.1.1. iSCSI Data-Type PDUs
An iSCSI data-type PDU is defined as an iSCSI PDU that causes An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
data transfer, transparent to the remote iSCSI layer, to take transfer, transparent to the remote iSCSI layer, to take place
place between the peer iSCSI nodes on a full feature phase between the peer iSCSI nodes on a Full Feature Phase iSCSI
iSCSI connection. A data-type PDU, when requested for connection. A data-type PDU, when requested for transmission by the
transmission by the sender iSCSI layer, results in the sender iSCSI layer, results in the associated data transfer without
associated data transfer without the participation of the the participation of the remote iSCSI layer, i.e., the PDU itself is
remote iSCSI layer, i.e. the PDU itself is not delivered as- not delivered as-is to the remote iSCSI layer. The following iSCSI
is to the remote iSCSI layer. The following iSCSI PDUs PDUs constitute the set of iSCSI data-type PDUs:
constitute the set of iSCSI data-type PDUs -
1) SCSI Data-In PDU 1) SCSI Data-In PDU
2) R2T PDU 2) R2T PDU
In an iSCSI end node structured as an iSCSI layer and a In an iSCSI end node structured as an iSCSI layer and a Datamover
Datamover layer as defined in this document, the solicitation layer as defined in this document, the solicitation for Data-Out
for Data-out (i.e. R2T PDU) is not delivered to the initiator (i.e., R2T PDU) is not delivered to the initiator iSCSI layer, per
iSCSI layer, per the definition of an iSCSI data-type PDU. the definition of an iSCSI data-type PDU. The data transfer is
The data transfer is instead performed via the mechanisms instead performed via the mechanisms known to the Datamover layer
known to the Datamover layer (e.g. RDMA Read). This in turn (e.g., RDMA Read). This in turn implies that a SCSI Data-Out PDU for
implies that a SCSI Data-Out PDU for solicited data is never solicited data is never requested for transmission across the
requested for transmission across the Datamover Interface at Datamover Interface at the initiator.
the initiator.
5.1.2 iSCSI control-type PDUs 5.1.2. iSCSI Control-Type PDUs
Any iSCSI PDU that is not an iSCSI data-type PDU and also not Any iSCSI PDU that is not an iSCSI data-type PDU and also not a
a solicited SCSI Data-out PDU is defined as an iSCSI control- solicited SCSI Data-Out PDU is defined as an iSCSI control-type PDU.
type PDU. Specifically, it is to be noted that SCSI Data-Out Specifically, note that SCSI Data-Out PDUs for unsolicited Data are
PDUs for unsolicited Data are defined as iSCSI control-type defined as iSCSI control-type PDUs.
PDUs.
5.2 Data_Descriptor 5.2. Data_Descriptor
A Data_Descriptor is an information element that describes an A Data_Descriptor is an information element that describes an
iSCSI/SCSI data buffer, provided by the iSCSI layer to its iSCSI/SCSI data buffer, provided by the iSCSI layer to its local
local Datamover layer or by the Datamover layer to its local Datamover layer or provided by the Datamover layer to its local iSCSI
iSCSI layer for identifying the data associated respectively layer for identifying the data associated respectively with the
with the requested or completed operation. requested or completed operation.
In implementation terms, a Data_Descriptor may be a scatter- In implementation terms, a Data_Descriptor may be a scatter-gather
gather list describing a local buffer, the exact structure of list describing a local buffer, the exact structure of which is
which is subject to the constraints imposed by the operating subject to the constraints imposed by the operating environment on
environment on the local iSCSI node. the local iSCSI node.
5.3 Connection_Handle 5.3. Connection_Handle
A Connection_Handle is an information element that identifies A Connection_Handle is an information element that identifies the
the particular iSCSI connection for which an inbound or particular iSCSI connection for which an inbound or outbound iSCSI
outbound iSCSI PDU is intended. A connection handle is unique PDU is intended. A connection handle is unique for a given pair of
for a given pair of an iSCSI layer instance and a Datamover an iSCSI layer instance and a Datamover layer instance. The
layer instance. The Connection_Handle qualifier is used in Connection_Handle qualifier is used in all invocations of any
all invocations of any Operational Primitive for connection Operational Primitive for connection identification.
identification.
Note that the Connection_Handle is conceptually different Note that the Connection_Handle is conceptually different from the
from the Connection Identifier (CID) defined by the iSCSI Connection Identifier (CID) defined by the iSCSI specification.
specification. While the CID is a unique identifier of an While the CID is a unique identifier of an iSCSI connection within an
iSCSI connection within an iSCSI session, the uniqueness of iSCSI session, the uniqueness of the Connection_Handle extends to the
the Connection_Handle extends to the entire iSCSI layer entire iSCSI layer instance coupled with the Datamover layer
instance coupled with the Datamover layer instance, across instance, across possibly multiple iSCSI sessions.
possibly multiple iSCSI sessions.
In implementation terms, a Connection_Handle could be an In implementation terms, a Connection_Handle could be an opaque
opaque identifier exchanged between the iSCSI layer and the identifier exchanged between the iSCSI layer and the Datamover layer
Datamover layer at the connection login time. One may also at the connection login time. One may also consider it to be similar
consider it to be similar in scope of uniqueness to a socket in scope of uniqueness to a socket identifier. The exact structure
identifier. The exact structure and modalities of exchange and modalities of exchange of a Connection_Handle between the two
of a Connection_Handle between the two layers is layers is implementation-specific.
implementation-specific.
5.4 Operational Primitive 5.4. Operational Primitive
An Operational Primitive, in this document, is an abstract An Operational Primitive, in this document, is an abstract functional
functional interface procedure that requests another layer to interface procedure that requests another layer perform a specific
perform a specific action on the requestor's behalf or action on the requestor's behalf or notifies the other layer of some
notifies the other layer of some event. The Datamover event. The Datamover Interface between an iSCSI layer instance and a
Interface between an iSCSI layer instance and a Datamover Datamover layer instance within an iSCSI end node uses a set of
layer instance within an iSCSI end node uses a set of Operational Primitives to define the functional interface between the
Operational Primitives to define the functional interface two layers. Note that not every invocation of an Operational
between the two layers. Note that not every invocation of an Primitive may elicit a response from the requested layer. This
Operational Primitive may elicit a response from the document describes the types of Operational Primitives that are
requested layer. This document describes the types of implicitly required and provided by the iSCSI protocol layer as
Operational Primitives that are implicitly required and defined in [RFC3720], and the semantics of these Primitives.
provided by the iSCSI protocol layer as defined in [RFC3720],
and the semantics of these Primitives.
Note that ownership of buffers and data structures is likely Note that ownership of buffers and data structures is likely to be
to be exchanged between the iSCSI layer and its local exchanged between the iSCSI layer and its local Datamover layer in
Datamover layer in invoking the Operational Primitives invoking the Operational Primitives defined in this architecture.
defined in this architecture. The buffer management details, The buffer management details, including how buffers are allocated
including how buffers are allocated and released, are and released, are implementation-specific and thus are outside the
implementation-specific and thus are outside the scope of scope of this document.
this document.
Each Operational Primitive invocation needs a certain Each Operational Primitive invocation needs a certain "information
"information context" (e.g., Connection_Handle) for context" (e.g., Connection_Handle) for performing the specific action
performing the specific action being requested of it. The being requested. The required information context is described in
required information context is described in this document by this document by a listing of "qualifiers" on each invocation, in the
a listing of "qualifiers" on each invocation - in the style style of function call arguments. There is no specific
of function call arguments. No implementation specific is implementation implied in this notation. The "qualifiers" of any
however implied in this notation. The "qualifiers" of any Operational Primitive invocation specified in this document thus
Operational Primitive invocation specified in this document represent the mandatory information context that the Operational
thus represent the mandatory information context that the Primitive invocation MUST consider in performing the action. While
Operational Primitive invocation MUST consider in performing the qualifiers are required, the method of realizing the qualifiers
the action. While the qualifiers are required, the method of (passed synchronously with invocation, or retrieved from task
realizing the qualifiers (passed synchronously with context, or retrieved from shared memory etc.) is really up to the
invocation, or retrieved from task context, or retrieved from implementations.
shared memory etc.) is really up to the implementations.
When an Operational Primitive implementation is described as When an Operational Primitive implementation is described as
mandatory ("MUST") or recommended ("SHOULD") of a layer mandatory ("MUST") or recommended ("SHOULD") of a layer (iSCSI or
(iSCSI or Datamover) in this document, the intent is that an Datamover) in this document, the intent is that an implementation
implementation respectively MUST or SHOULD produce the same respectively MUST or SHOULD produce the same protocol action as what
protocol action as what the model describes. the model describes.
5.5 Transport Connection 5.5. Transport Connection
The term "Transport Connection" is used in this document as a The term "Transport Connection" is used in this document as a generic
generic term to represent the end-to-end logical connection term to represent the end-to-end logical connection as defined by the
as defined by the underlying reliable transport protocol. underlying reliable transport protocol. For this document, all
For this revision of this document, a Transport Connection instances of Transport Connection refer to a TCP connection.
means only a TCP connection.
6 Datamover layer and Datamover protocol 6. Datamover Layer and Datamover Protocol
This section introduces the notion of a "Datamover layer" and This section introduces the notion of a "Datamover layer" and
"Datamover protocol" as meant in this document, and defines "Datamover protocol" as meant in this document, and defines the
the requirements on a Datamover protocol. requirements on a Datamover protocol.
A Datamover layer is the implementation component that A Datamover layer is the implementation component that realizes a
realizes a Datamover protocol functionality in an iSCSI- Datamover protocol functionality in an iSCSI-capable end node in
capable end node, in communicating with other iSCSI end nodes communicating with other iSCSI end nodes with similar capabilities.
with similar capabilities. More specifically, a "Datamover More specifically, a "Datamover layer" MUST provide the following
layer" MUST provide the following functionality and the functionality and the "Datamover protocol" MUST consist of the wire
"Datamover protocol" MUST consist of the wire protocol protocol required to realize the following functionality:
required to realize the following functionality -
1) guarantee that all the necessary data transfers take place 1) guarantee that all the necessary data transfers take place when
when the local iSCSI layer requests transmitting a command the local iSCSI layer requests transmitting a command (in order
(in order to complete a SCSI command, for an initiator),or to complete a SCSI command, for an initiator), or
sending/receiving an iSCSI data sequence (in order to sending/receiving an iSCSI data sequence (in order to complete
complete part of a SCSI command, for a target). part of a SCSI command for a target).
2) transport an iSCSI control-type PDU as-is to the peer 2) transport an iSCSI control-type PDU as-is to the peer Datamover
Datamover layer when requested to do so by the local iSCSI layer when requested to do so by the local iSCSI layer.
layer.
3) provide notification and delivery to the iSCSI layer upon 3) provide notification and delivery to the iSCSI layer upon
arrival of an iSCSI control-type PDU. arrival of an iSCSI control-type PDU.
4) provide an initiator-to-target data acknowledgement of SCSI 4) provide an initiator-to-target data acknowledgement of SCSI
read data back to the target iSCSI layer, when requested. read data back to the target iSCSI layer, when requested.
5) provide an asynchronous notification upon completion of a 5) provide an asynchronous notification upon completion of a
requested data transfer operation that moved data without requested data transfer operation that moved data without
involving the iSCSI layer. involving the iSCSI layer.
6) place the SCSI data into the I/O buffers or pick up the 6) place the SCSI data into the I/O buffers or pick up the SCSI
SCSI data for transmission out of the data buffers that the data for transmission out of the data buffers that the iSCSI
iSCSI layer had requested to be used for a SCSI I/O. layer had requested to be used for a SCSI I/O.
7) provide an error-free (i.e. must have at least the same 7) provide an error-free (i.e., must have at least the same level
level of assurance of data integrity as the CRC32C iSCSI of assurance of data integrity as the CRC32C iSCSI data
data digest), reliable, in-order delivery transport digest), reliable, in-order delivery transport mechanism over
mechanism over IP networks in performing the data transfer, IP networks in performing the data transfer, and asynchronously
and asynchronously notify the iSCSI layer upon iSCSI notify the iSCSI layer upon iSCSI connection termination.
connection termination.
Note that this architecture expects that each compliant Note that this architecture expects that each compliant Datamover
Datamover protocol will define the precise means of protocol will define the precise means of satisfying the requirements
satisfying the requirements specified in this section. specified in this section.
In order to meet the functional requirements listed in this In order to meet the functional requirements listed in this section,
section, certain Datamover protocols may require pre-posted certain Datamover protocols may require pre-posted buffers from the
buffers from the local iSCSI protocol layer via mechanisms local iSCSI protocol layer via mechanisms outside the scope of this
outside the scope of this document and in some document. In some implementations, the absence of such buffers may
implementations, the absence of such buffers may result in a result in a connection failure. Datamover protocols may also realize
connection failure. Datamover protocols may also realize these functional requirements via methods not explicitly listed in
these functional requirements via methods not explicitly this document.
listed in this document.
7 Functional Overview 7. Functional Overview
This section presents an overview of the functional This section presents an overview of the functional interactions
interactions between the iSCSI layer and the Datamover layer between the iSCSI layer and the Datamover layer as intended by this
as intended by this Architecture. Architecture.
7.1 Startup 7.1. Startup
The iSCSI Login Phase on an iSCSI connection occurs as The iSCSI Login Phase on an iSCSI connection occurs as defined in
defined in [RFC3720]. The Architecture assumes that at the [RFC3720]. The Architecture assumes that at the end of the Login
end of the Login Phase, both the initiator and target, if Phase, both the initiator and target, if they had so decided,
they had so decided, transition the connection to being transition the connection to being Datamover-assisted. The precise
Datamover-assisted. The precise means of how an iSCSI means of how an iSCSI initiator and an iSCSI target agree on having
initiator and an iSCSI target agree on having the connection the connection Datamover-assisted is defined by the Datamover
Datamover-assisted is defined by the Datamover protocol. The protocol. The only architectural requirement is that all iSCSI
only architectural requirement is that all iSCSI interactions interactions in the iSCSI Full Feature Phase MUST be Datamover-
in the iSCSI Full Feature Phase MUST be Datamover-assisted assisted subject to the prior agreement, meaning that the Datamover
subject to the prior agreement, meaning that Datamover protocol is in the iSCSI-to-iSCSI communication path below the iSCSI
protocol is in the iSCSI-to-iSCSI communication path below layer on either side as shown in Figure 1. DA defines the
the iSCSI layer on either side as shown in Figure 1. DA Enable_Datamover Operational Primitive (Section 8.6) to bring about
defines the Enable_Datamover Operational Primitive (section this transition to a Datamover-assisted connection.
8.6) to bring about this transition to a Datamover-assisted
connection.
The Architecture also assumes that the Datamover layer may The Architecture also assumes that the Datamover layer may require a
require a certain number of opaque local resources for making certain number of opaque local resources for making a connection
a connection Datamover-assisted. DA thus defines the Datamover-assisted. DA thus defines the
Allocate_Connection_Resources Operational Primitive (section Allocate_Connection_Resources Operational Primitive (Section 8.4) to
8.4) to model this interaction. This Primitive is intended model this interaction. This Primitive is intended to be invoked on
to be invoked on each side once the two sides decide (as each side once the two sides decide (as previously noted) to have the
previously noted) to have the connection Datamover-assisted. connection be Datamover-assisted. The expected sequence of Primitive
The expected sequence of Primitive invocations is depicted in invocations is depicted in Figures 2 and 3 in Section 13.2. Figures
Figure 2 and Figure 3 in section 16.2. Figure 4, Figure 5, 4, 5, and 6 illustrate how the Primitives may be employed to deal
and Figure 6 illustrate how the Primitives may be employed to with various legal login outcomes.
deal with various legal login outcomes.
7.2 Full Feature Phase 7.2. Full Feature Phase
All iSCSI peer communication in the Full Feature Phase All iSCSI peer communication in the Full Feature Phase happens
happens through the Datamover layers if the iSCSI connection through the Datamover layers if the iSCSI connection is Datamover-
is Datamover-assisted. The Architecture assumes that a assisted. The Architecture assumes that a Datamover layer may
Datamover layer may require a certain number of opaque local require a certain number of opaque local resources for each new iSCSI
resources for each new iSCSI task. In the normal course of task. In the normal course of execution, these task-level resources
execution, these task-level resources in the Datamover layer in the Datamover layer are assumed to be transparently allocated on
are assumed to be transparently allocated on each task each task initiation and deallocated on the conclusion of each task
initiation and deallocated on the conclusion of each task as as appropriate. In exception scenarios however -- scenarios that do
appropriate. In exception scenarios however - in scenarios not yield a SCSI Response for each task such as ABORT TASK operation
that do not yield a SCSI Response for each task such as ABORT -- the Architecture assumes that the Datamover layer needs to be
TASK operation - the Architecture assumes that the Datamover notified of the individual task terminations to aid its task-level
layer needs to be notified of the individual task resource management. DA thus defines the Deallocate_Task_Resources
terminations to aid its task-level resource management. DA Operational Primitive (Section 8.9) to model this task-resource
thus defines the Deallocate_Task_Resources Operational
Primitive (section 8.9) to model this task-resource
management. In specifying the ITT qualifier for the management. In specifying the ITT qualifier for the
Deallocate_Task_Resources Primitive, the Architecture further Deallocate_Task_Resources Primitive, the Architecture further assumes
assumes that the Datamover layer tracks its opaque task-level that the Datamover layer tracks its opaque task-level local resources
local resources by the iSCSI ITT. DA also defines by the iSCSI ITT. DA also defines Send_Control (Section 8.1),
Send_Control (section 8.1), Put_Data (section 8.2), Get_Data Put_Data (Section 8.2), Get_Data (Section 8.3),
(section 8.3), Data_Completion_Notify(section 9.3), Data_Completion_Notify (Section 9.3), Data_ACK_Notify (Section 9.4),
Data_ACK_Notify (section 9.4), and Control_Notify (section and Control_Notify (Section 9.1) Operational Primitives to model the
9.1) Operational Primitives to model the various Full Feature various Full Feature Phase interactions.
Phase interactions.
Figure 9, Figure 10, and Figure 11 in section 16.2 show some Figures 9, 10, and 11 in Section 13.2 show some Full Feature Phase
Full Feature Phase interactions - SCSI Write task, SCSI Read interactions -- SCSI Write task, SCSI Read task, and a SCSI Read Data
task, and a SCSI Read Data acknowledgement respectively. acknowledgement, respectively. Figure 12 in Section 13.2 illustrates
Figure 12 in section 16.2 illustrates how an ABORT TASK how an ABORT TASK operation can be modeled leading to deterministic
operation can be modeled leading to deterministic resource resource cleanup on the Datamover layer.
cleanup on the Datamover layer.
7.3 Wrapup 7.3. Wrap-up
Once an iSCSI connection becomes Datamover-assisted, the Once an iSCSI connection becomes Datamover-assisted, the connection
connection continues in that state till the end of the Full continues in that state until the end of the Full Feature Phase,
Feature Phase, i.e. the termination of the connection. The i.e., the termination of the connection. The Architecture assumes
Architecture assumes that when a connection is normally that when a connection is normally logged out, the Datamover layer
logged out, the Datamover layer needs to be notified so that needs to be notified so that its connection-level opaque resources
its connection-level opaque resources (see section 7.1) may (see Section 7.1) may be freed up. DA thus defines a
now be freed up. DA thus defines a Connection_Terminate Connection_Terminate Operational Primitive (Section 8.7) to model
Operational Primitive (section 8.7) to model this this interaction. The Architecture further assumes that when a
interaction. The Architecture further assumes that when a connection termination happens without iSCSI layer's involvement
connection termination happens without iSCSI layer's (e.g., TCP RST), the Datamover layer is capable of locally cleaning
involvement (e.g., TCP RST), the Datamover layer is capable up its task-level and connection-level resources before notifying the
of locally cleaning up its task-level and connection-level iSCSI layer of the fact. DA thus defines the
resources before notifying the iSCSI layer of the fact. DA Connection_Terminate_Notify Operational Primitive (Section 9.2) to
thus defines the Connection_Terminate_Notify Operational model this interaction.
Primitive (section 9.2) to model this interaction.
Figure 7 and Figure 8 in section 16.2 illustrate the Figures 7 and 8 in Section 13.2 illustrate the interactions between
interactions between the iSCSI and Datamover layers in normal the iSCSI and Datamover layers in normal and unexpected connection
and unexpected connection termination scenarios. termination scenarios.
8 Operational Primitives provided by the Datamover layer 8. Operational Primitives Provided by the Datamover Layer
While the iSCSI specification itself does not have a notion While the iSCSI specification itself does not have a notion of
of Operational Primitives, any iSCSI layer implementing the Operational Primitives, any iSCSI layer implementing the iSCSI
iSCSI specification functionally requires the following specification functionally requires the following Operational
Operational Primitives from its Datamover layer. Thus, any Primitives from its Datamover layer. Thus, any Datamover protocol
Datamover protocol compliant with this architecture MUST compliant with this architecture MUST implement the Operational
implement the Operational Primitives described in this Primitives described in this section. These Operational Primitives
section. These Operational Primitives are invoked by the are invoked by the iSCSI layer as appropriate. Unless otherwise
iSCSI layer as appropriate. Unless otherwise stated, all the stated, all the following Operational Primitives may be used both on
following Operational Primitives may be used both on the the initiator side and the target side. In general programming
initiator side and the target side. In general programming terminology, this set of Operational Primitives may be construed as
terminology, this set of Operational Primitives may be "down calls".
construed as "down calls".
1) Send_Control 1) Send_Control
2) Put_Data 2) Put_Data
3) Get_Data 3) Get_Data
4) Allocate_Connection_Resources 4) Allocate_Connection_Resources
5) Deallocate_Connection_Resources 5) Deallocate_Connection_Resources
6) Enable_Datamover 6) Enable_Datamover
7) Connection_Terminate 7) Connection_Terminate
8) Notice_Key_Values 8) Notice_Key_Values
9) Deallocate_Task_Resources 9) Deallocate_Task_Resources
8.1 Send_Control 8.1. Send_Control
Input qualifiers: Connection_Handle, iSCSI PDU-specific Input qualifiers: Connection_Handle, iSCSI PDU-specific qualifiers
qualifiers
Return Results: Not specified. Return Results: Not specified.
An iSCSI layer requests its local Datamover layer to transmit An iSCSI layer requests that its local Datamover layer transmit an
an iSCSI control-type PDU to the peer iSCSI layer operating iSCSI control-type PDU to the peer iSCSI layer operating in the
in the remote iSCSI node by this Operational Primitive. The remote iSCSI node by this Operational Primitive. The Datamover layer
Datamover layer performs the requested operation, and may add performs the requested operation, and may add its own protocol
its own protocol headers in doing so. The iSCSI layer MUST headers in doing so. The iSCSI layer MUST NOT invoke the
NOT invoke the Send_Control Operational Primitive on an iSCSI Send_Control Operational Primitive on an iSCSI connection that is not
connection that is not yet Datamover-assisted. yet Datamover-assisted.
An initiator iSCSI layer requesting the transfer of a SCSI An initiator iSCSI layer requesting the transfer of a SCSI Command
command PDU or a target iSCSI layer requesting the transfer PDU or a target iSCSI layer requesting the transfer of a SCSI
of a SCSI response PDU are examples of invoking the response PDU are examples of invoking the Send_Control Operational
Send_Control Operational Primitive. As section 10.3.1 Primitive. As Section 10.3.1 illustrates later on, the iSCSI PDU-
illustrates later on, the iSCSI PDU-specific qualifiers in specific qualifiers in this example are: BHS and AHS,
this example are: BHS and AHS, DataDescriptorOut, DataDescriptorOut, DataDescriptorIn, ImmediateDataSize, and
DataDescriptorIn, ImmediateDataSize, and UnsolicitedDataSize UnsolicitedDataSize.
8.2 Put_Data 8.2. Put_Data
Input qualifiers: Connection_Handle, contents of a SCSI Data- Input qualifiers: Connection_Handle, contents of a SCSI Data-In PDU
In PDU header, Data_Descriptor, Notify_Enable header, Data_Descriptor, Notify_Enable
Return Results: Not specified. Return Results: Not specified.
An iSCSI layer requests its local Datamover layer to transmit An iSCSI layer requests that its local Datamover layer transmit the
the data identified by the Data_Descriptor for the SCSI Data- data identified by the Data_Descriptor for the SCSI Data-In PDU to
In PDU to the peer iSCSI layer on the remote iSCSI node by the peer iSCSI layer on the remote iSCSI node by this Operational
this Operational Primitive. The Datamover layer performs the Primitive. The Datamover layer performs the operation by using its
operation by using its own protocol means, completely own protocol means, completely transparent to the remote iSCSI layer.
transparent to the remote iSCSI layer. The iSCSI layer MUST The iSCSI layer MUST NOT invoke the Put_Data Operational Primitive on
NOT invoke the Put_Data Operational Primitive on an iSCSI an iSCSI connection that is not yet Datamover-assisted.
connection that is not yet Datamover-assisted.
The Notify_Enable qualifier is used to request the local The Notify_Enable qualifier is used to request the local Datamover
Datamover layer to generate or to not generate the eventual layer to generate or not generate the eventual local completion
local completion notification to the iSCSI layer for this notification to the iSCSI layer for this Put_Data invocation. For
Put_Data invocation. For detailed semantics of this detailed semantics of this qualifier, see Section 9.3.
qualifier, see section 9.3.
A Put_Data Primitive may only be invoked by an iSCSI layer on A Put_Data Primitive may only be invoked by an iSCSI layer on the
the target to its local Datamover layer. target to its local Datamover layer.
A target iSCSI layer requesting the transfer of an iSCSI read A target iSCSI layer requesting the transfer of an iSCSI read data
data sequence (also known as a read burst) is an example of sequence (also known as a read burst) is an example of invoking the
invoking the Put_Data Operational Primitive. Put_Data Operational Primitive.
8.3 Get_Data 8.3. Get_Data
Input qualifiers: Connection_Handle, contents of an R2T PDU, Input qualifiers: Connection_Handle, contents of an R2T PDU,
Data_Descriptor, Notify_Enable Data_Descriptor, Notify_Enable
Return Results: Not specified. Return Results: Not specified.
An iSCSI layer requests its local Datamover layer to retrieve An iSCSI layer requests that its local Datamover layer retrieve
certain data identified by the R2T PDU from the peer iSCSI certain data identified by the R2T PDU from the peer iSCSI layer on
layer on the remote iSCSI node into the buffer identified by the remote iSCSI node and place it into the buffer identified by the
the Data_Descriptor by invoking this Operational Primitive. Data_Descriptor by invoking this Operational Primitive. The
The Datamover layer performs the operation by using its own Datamover layer performs the operation by using its own protocol
protocol means, completely transparent to the remote iSCSI means, completely transparent to the remote iSCSI layer. The iSCSI
layer. The iSCSI layer MUST NOT invoke the Get_Data layer MUST NOT invoke the Get_Data Operational Primitive on an iSCSI
Operational Primitive on an iSCSI connection that is not yet connection that is not yet Datamover-assisted.
Datamover-assisted.
The Notify_Enable qualifier is used to request the local The Notify_Enable qualifier is used to request that the local
Datamover layer to generate or to not generate the eventual Datamover layer generate or not generate the eventual local
local completion notification to the iSCSI layer for this completion notification to the iSCSI layer for this Get_Data
Get_Data invocation. For detailed semantics of this invocation. For detailed semantics of this qualifier, see Section
qualifier, see section 9.3. 9.3.
A Get_Data Primitive may only be invoked by an iSCSI layer on A Get_Data Primitive may only be invoked by an iSCSI layer on the
the target to its local Datamover layer. target to its local Datamover layer.
A target iSCSI layer requesting the transfer of an iSCSI A target iSCSI layer requesting the transfer of an iSCSI write data
write data sequence (also known as a write burst) is an sequence (also known as a write burst) is an example of invoking the
example of invoking the Get_Data Operational Primitive. Get_Data Operational Primitive.
8.4 Allocate_Connection_Resources 8.4. Allocate_Connection_Resources
Input qualifiers: Connection_Handle[, Resource_Descriptor ] Input qualifiers: Connection_Handle[, Resource_Descriptor ]
Return Results: Status. Return Results: Status.
By invoking this Operational Primitive, an iSCSI layer By invoking this Operational Primitive, an iSCSI layer requests that
requests its local Datamover layer to perform all the its local Datamover layer perform all the Datamover-specific resource
Datamover-specific resource allocations required for the full allocations required for the Full Feature Phase of an iSCSI
feature phase of an iSCSI connection. The Connection_Handle connection. The Connection_Handle identifies the connection for
identifies the connection the iSCSI layer is requesting the which the iSCSI layer is requesting resources to be allocated.
resource allocation for in order to eventually transition the Allocation of these resources is a step towards eventually
connection to be a Datamover-assisted iSCSI connection. Note transitioning the connection to become a Datamover-assisted iSCSI
that the Datamover layer however does not allocate any connection. Note that the Datamover layer however does not allocate
Datamover-specific task-level resources upon invocation of any Datamover-specific task-level resources upon invocation of this
this Primitive. Primitive.
An iSCSI layer, in addition, optionally specifies the An iSCSI layer, in addition, optionally specifies the
implementation-specific resource requirements for the iSCSI implementation-specific resource requirements for the iSCSI
connection to the Datamover layer, by passing an input connection to the Datamover layer by passing an input qualifier
qualifier called Resource_Descriptor. The exact structure of called Resource_Descriptor. The exact structure of a
a Resource_Descriptor is implementation-dependent, and hence Resource_Descriptor is implementation-dependent, and hence
structurally opaque to DA. structurally opaque to DA.
A return result of Status=success means that the A return result of Status=success means that the
Allocate_Connection_Resources invocation corresponding to Allocate_Connection_Resources invocation corresponding to that
that Connection_Handle succeeded. If an Connection_Handle succeeded. If an Allocate_Connection_Resources
Allocate_Connection_Resources invocation is made for a invocation is made for a Connection_Handle for which an earlier
Connection_Handle for which an earlier invocation succeeded, invocation succeeded, the return Status must be success and the
the return Status must be success and the request will be request will be ignored by the Datamover layer. A return result of
ignored by the Datamover layer. A return result of
Status=failure means that the Allocate_Connection_Resources Status=failure means that the Allocate_Connection_Resources
invocation corresponding to that Connection_Handle failed. invocation corresponding to that Connection_Handle failed. There
There MUST NOT be more than one Allocate_Connection_Resources MUST NOT be more than one Allocate_Connection_Resources Primitive
Primitive invocation outstanding for a given invocation outstanding for a given Connection_Handle at any time.
Connection_Handle at any time.
The iSCSI layer must invoke the Allocate_Connection_Resources The iSCSI layer must invoke the Allocate_Connection_Resources
Primitive before the invocation of the Enable_Datamover Primitive before the invocation of the Enable_Datamover Primitive.
Primitive.
8.5 Deallocate_Connection_Resources 8.5. Deallocate_Connection_Resources
Input qualifiers: Connection_Handle Input qualifiers: Connection_Handle
Return Results: Not specified. Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer By invoking this Operational Primitive, an iSCSI layer requests that
requests its local Datamover layer to deallocate all the its local Datamover layer deallocate all the Datamover-specific
Datamover-specific resources that may have been allocated resources that may have been allocated earlier for the Transport
earlier for the Transport Connection identified by the Connection identified by the Connection_Handle. The iSCSI layer may
Connection_Handle. The iSCSI layer may invoke this invoke this Operational Primitive when the Datamover-specific
Operational Primitive when the Datamover-specific resources resources associated with the Connection_Handle are no longer
associated with the Connection_Handle are no longer necessary necessary (such as the Login failure of the corresponding iSCSI
(such as the Login failure of the corresponding iSCSI
connection). connection).
8.6 Enable_Datamover 8.6. Enable_Datamover
Input qualifiers: Connection_Handle, Input qualifiers: Connection_Handle, Transport_Connection_Descriptor
Transport_Connection_Descriptor [, Final_Login_Response_PDU] [, Final_Login_Response_PDU]
Return Results: Not specified. Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer By invoking this Operational Primitive, an iSCSI layer requests that
requests its local Datamover layer to assist all further its local Datamover layer assist all further iSCSI exchanges on the
iSCSI exchanges on the iSCSI connection (i.e. to make the iSCSI connection (i.e., to make the connection Datamover-assisted)
connection Datamover-assisted) identified by the identified by the Connection_Handle, for which the Datamover-specific
Connection_Handle, for which the Datamover-specific resource resource allocation was earlier made. The iSCSI layer MUST NOT
allocation was earlier made. The iSCSI layer MUST NOT invoke invoke the Enable_Datamover Operational Primitive for an iSCSI
the Enable_Datamover Operational Primitive for an iSCSI connection unless there is a corresponding prior resource allocation.
connection unless there was a corresponding prior resource
allocation.
The Final_Login_Response_PDU input qualifier is applicable The Final_Login_Response_PDU input qualifier is applicable only for a
only for a target, and contains the final Login Response that target, and contains the final Login Response that concludes the
concludes the iSCSI Login phase and which must be sent as a iSCSI Login Phase and which must be sent as a byte stream as expected
byte stream as expected by the initiator iSCSI layer. When by the initiator iSCSI layer. When this qualifier is used, the
this qualifier is used, the target-Datamover layer MUST target-Datamover layer MUST transmit this final Login Response before
transmit this final Login Response before Datamover Datamover assistance is enabled for the Transport Connection.
assistance is enabled for the Transport Connection.
The iSCSI layer identifies the specific Transport Connection The iSCSI layer identifies the specific Transport Connection
associated with the Connection_Handle to the Datamover layer associated with the Connection_Handle to the Datamover layer by
by specifying the Transport_Connection_Descriptor. The exact specifying the Transport_Connection_Descriptor. The exact structure
structure of this Descriptor is implementation-dependent. of this Descriptor is implementation-dependent.
8.7 Connection_Terminate 8.7. Connection_Terminate
Input qualifiers: Connection_Handle Input qualifiers: Connection_Handle
Return Results: Not specified. Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer By invoking this Operational Primitive, an iSCSI layer requests that
requests its local Datamover layer to terminate the Transport its local Datamover layer terminate the Transport Connection and
Connection and deallocate all the connection and task deallocate all the connection and task resources associated with the
resources associated with the Connection_Handle. When this Connection_Handle. When this Operational Primitive invocation
Operational Primitive invocation returns to the iSCSI layer, returns to the iSCSI layer, the iSCSI layer may assume the full
the iSCSI layer may assume the full ownership of all the ownership of all the iSCSI-level resources, e.g., I/O Buffers,
iSCSI-level resources, e.g. I/O Buffers, associated with the associated with the connection. This Operational Primitive may be
connection. This Operational Primitive may be invoked only invoked only with a valid Connection_Handle, and the Transport
with a valid Connection_Handle and the Transport Connection Connection associated with the Connection_Handle must already be
associated with the Connection_Handle must already be
Datamover-assisted. Datamover-assisted.
8.8 Notice_Key_Values 8.8. Notice_Key_Values
Input qualifiers: Connection_Handle, Number of keys, a list Input qualifiers: Connection_Handle, Number of keys, a list of Key-
of Key-Value pairs Value pairs.
Return Results: Not specified. Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer By invoking this Operational Primitive, an iSCSI layer requests that
requests its local Datamover layer to take note of the its local Datamover layer take note of the negotiated values of the
negotiated values of the listed keys for the Transport listed keys for the Transport Connection. This Operational Primitive
Connection. This Operational Primitive may be invoked only may be invoked only with a valid Connection_Handle, and the Key-Value
with a valid Connection_Handle and the Key-Value pairs MUST pairs MUST be the current values that were successfully agreed upon
be the current values that were successfully agreed upon by by the iSCSI peers for the connection. The Datamover layer may use
the iSCSI peers for the connection. The Datamover layer may the values of the keys to aid the Datamover operation as it deems
use the values of the keys to aid the Datamover operation as appropriate. The specific keys to be passed as input qualifiers and
it deems appropriate. The specific keys to be passed in as the point(s) in time this Operational Primitive is invoked are
input qualifiers and the point(s) in time this Operational implementation-dependent.
Primitive is invoked are implementation-dependent.
8.9 Deallocate_Task_Resources 8.9. Deallocate_Task_Resources
Input qualifiers: Connection_Handle, ITT Input qualifiers: Connection_Handle, ITT
Return Results: Not specified. Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer By invoking this Operational Primitive, an iSCSI layer requests that
requests its local Datamover Layer to deallocate all its local Datamover layer deallocate all Datamover-specific resources
Datamover-specific resources that earlier may have been that earlier may have been allocated for the task identified by the
allocated for the task identified by the ITT qualifier. The ITT qualifier. The iSCSI layer uses this Operational Primitive
iSCSI layer uses this Operational Primitive during exception during exception processing when one or more active tasks are to be
processing when one or more active tasks are to be terminated terminated without corresponding SCSI Response PDUs. This Primitive
without corresponding SCSI Response PDUs. This Primitive MUST be invoked for each active task terminated without a SCSI
MUST be invoked for each active task terminated without a Response PDU. This Primitive MUST NOT be invoked by the iSCSI layer
SCSI Response PDU. This Primitive MUST NOT be invoked by the when a SCSI Response PDU normally concludes a task. When a SCSI
iSCSI layer when a SCSI Response PDU normally concludes a Response PDU normally concludes a task (even if the SCSI Status was
task. When a SCSI Response PDU normally concludes a task not a success), the Datamover layer is assumed to have automatically
(even if the SCSI Status was not a success), the Datamover deallocated all Datamover-specific task resources for that task.
layer is assumed to have automatically deallocated all Refer to Section 7.2 for a related discussion on the Architectural
Datamover-specific task resources for that task. Refer to
section 7.2 for a related discussion on the Architectural
assumptions on the task-level Datamover resource management, assumptions on the task-level Datamover resource management,
especially with respect to when the resources are assumed to especially with respect to when the resources are assumed to be
be allocated. allocated.
9 Operational Primitives provided by the iSCSI layer 9. Operational Primitives Provided by the iSCSI Layer
While the iSCSI specification itself does not have a notion While the iSCSI specification itself does not have a notion of
of Operational Primitives, any iSCSI layer implementing the Operational Primitives, any iSCSI layer implementing the iSCSI
iSCSI specification would have to provide the following specification would have to provide the following Operational
Operational Primitives to its local Datamover layer. Thus, Primitives to its local Datamover layer. Thus, any iSCSI protocol
any iSCSI protocol implementation compliant with this implementation compliant with this architecture MUST implement the
architecture MUST implement the Operational Primitives Operational Primitives described in this section. These Operational
described in this section. These Operational Primitives are Primitives are invoked by the Datamover layer as appropriate and when
invoked by the Datamover layer as appropriate and when the the iSCSI connection is Datamover-assisted. Unless otherwise stated,
iSCSI connection is Datamover-assisted. Unless otherwise all the following Operational Primitives may be used both on the
stated, all the following Operational Primitives may be used initiator side and the target side. In general programming
both on the initiator side and the target side. In general terminology, this set of Operational Primitives may be construed as
programming terminology, this set of Operational Primitives "up calls".
may be construed as "up calls".
1) Control_Notify 1) Control_Notify
2) Connection_Terminate_Notify 2) Connection_Terminate_Notify
3) Data_Completion_Notify 3) Data_Completion_Notify
4) Data_ACK_Notify 4) Data_ACK_Notify
9.1 Control_Notify 9.1. Control_Notify
Input qualifiers: Connection_Handle, an iSCSI control-type Input qualifiers: Connection_Handle, an iSCSI control-type PDU.
PDU.
Return Results: Not specified. Return Results: Not specified.
A Datamover layer notifies its local iSCSI layer, via this A Datamover layer notifies its local iSCSI layer, via this
Operational Primitive, of the arrival of an iSCSI control- Operational Primitive, of the arrival of an iSCSI control-type PDU
type PDU from the peer Datamover layer on the remote iSCSI from the peer Datamover layer on the remote iSCSI node. The iSCSI
node. The iSCSI layer processes the control-type PDU as layer processes the control-type PDU as defined in [RFC3720].
defined in [RFC3720].
A target iSCSI layer being notified of the arrival of a SCSI A target iSCSI layer being notified of the arrival of a SCSI command
Command is an example of invoking the Control_Notify is an example of invoking the Control_Notify Operational Primitive.
Operational Primitive.
Note that implementations may choose to describe the "iSCSI Note that implementations may choose to describe the "iSCSI control-
control-type PDU" qualifier in this notification using a type PDU" qualifier in this notification using a Data_Descriptor
Data_Descriptor (section 5.2) and not necessarily one (Section 5.2) and not necessarily one contiguous buffer.
contiguous buffer.
9.2 Connection_Terminate_Notify 9.2. Connection_Terminate_Notify
Input qualifiers: Connection_Handle Input qualifiers: Connection_Handle
Return Results: Not specified. Return Results: Not specified.
A Datamover layer notifies its local iSCSI layer on an A Datamover layer notifies its local iSCSI layer on an unsolicited
unsolicited termination or failure of an iSCSI connection termination or failure of an iSCSI connection providing the
providing the Connection_Handle associated with the iSCSI Connection_Handle associated with the iSCSI Connection. The iSCSI
Connection. The iSCSI Layer MUST consider the layer MUST consider the Connection_Handle to be invalid upon being so
Connection_Handle to be invalid upon being so notified. The notified. The iSCSI layer processes the connection termination as
iSCSI layer processes the connection termination as defined defined in [RFC3720]. The Datamover layer MUST deallocate the
in [RFC3720]. The Datamover layer MUST deallocate the
connection and task resources associated with the terminated connection and task resources associated with the terminated
connection before notifying the iSCSI layer of the connection before notifying the iSCSI layer of the termination via
termination via this Operational Primitive. this Operational Primitive.
A target iSCSI layer being notified of an ungraceful A target iSCSI layer is notified of an ungraceful connection
connection termination by the Datamover layer when the termination by the Datamover layer when the underlying Transport
underlying Transport Connection is torn down. Such a Connection is torn down. Such a Connection_Terminate_Notify
Connection_Terminate_Notify Operational Primitive may be Operational Primitive may be triggered, for example, by a TCP RESET
triggered, for example, by a TCP RESET in cases where the in cases where the underlying Transport Connection uses TCP.
underlying Transport Connection uses TCP.
9.3 Data_Completion_Notify 9.3. Data_Completion_Notify
Input qualifiers: Connection_Handle, ITT, SN Input qualifiers: Connection_Handle, ITT, SN
Return Results: Not specified. Return Results: Not specified.
A Datamover layer notifies its local iSCSI layer on A Datamover layer notifies its local iSCSI layer on completing the
completing the retrieval of the data or upon sending the retrieval of the data or upon sending the data, as requested in a
data, as requested in a prior iSCSI data-type PDU, from/to prior iSCSI data-type PDU, from/to the peer Datamover layer on the
the peer Datamover layer on the remote iSCSI node via this remote iSCSI node via this Operational Primitive. The iSCSI layer
Operational Primitive. The iSCSI layer processes the processes the operation as defined in [RFC3720].
operation as defined in [RFC3720].
SN may be either the DataSN associated with the SCSI Data-In SN may be either the DataSN associated with the SCSI Data-In PDU or
PDU or R2TSN associated with the R2T PDU depending on the R2TSN associated with the R2T PDU depending on the SCSI operation.
SCSI operation. Note that, for targets, a TTT (see Note that, for targets, a TTT (see [RFC3720]) could have been
[RFC3720]) could have been specified instead of an SN. specified instead of an SN. However, the considered choice was to
However, the considered choice was to leave the SN to be the leave the SN to be the qualifier for two reasons -- a) it is generic
qualifier for two reasons - a) it is generic and applicable and applicable to initiators and targets as well as Data-In and
to initiators and targets as well as Data-in and Data-out, Data-Out, and b) having both SN and TTT qualifiers for the
and b) having both SN and TTT qualifiers for the notification notification is considered onerous on the Datamover layer, in terms
was considered onerous on the Datamover layer, in terms of of state maintenance for each completion notification. The
state maintenance for each completion notification. The implication of this choice is that iSCSI target implementations will
implication of this choice is that iSCSI target have to adapt to using the ITT-SN tuple in associating the solicited
implementations will have to adapt to using the ITT-SN tuple data to the appropriate task, rather than the ITT-TTT tuple for doing
in associating the solicited data to the appropriate task, the same.
rather than the ITT-TTT tuple for doing the same.
If Notify_Enable was set in either a Put_Data or a Get_Data If Notify_Enable is set in either a Put_Data or a Get_Data
invocation, the Datamover layer MUST invoke the invocation, the Datamover layer MUST invoke the
Data_Completion_Notify Operational Primitive upon completing Data_Completion_Notify Operational Primitive upon completing that
that requested data transfer. If the Notify_Enable was requested data transfer. If the Notify_Enable was cleared in either
cleared in either a Put_Data or a Get_Data invocation, the a Put_Data or a Get_Data invocation, the Datamover layer MUST NOT
Datamover layer MUST NOT invoke the Data_Completion_Notify invoke the Data_Completion_Notify Operational Primitive upon
Operational Primitive upon completing that requested data completing that requested data transfer.
transfer.
A Data_Completion_Notify invocation serves to notify the A Data_Completion_Notify invocation serves to notify the iSCSI layer
iSCSI layer of the Put_Data or Get_Data completion of the Put_Data or Get_Data completion, respectively. As earlier
respectively. As earlier noted in sections 8.2 and 8.3, noted in Sections 8.2 and 8.3, specific Datamover protocol
specific Datamover protocol definitions may restrict the definitions may restrict the usage scope of Put_Data and Get_Data,
usage scope of Put_Data and Get_Data, and thus implicitly the and thus implicitly the usage scope of Data_Completion_Notify.
usage scope of Data_Completion_Notify.
A target iSCSI layer being notified of the retrieval of a A target iSCSI layer being notified of the retrieval of a write data
write data sequence is an example of invoking the sequence is an example of invoking the Data_Completion_Notify
Data_Completion_Notify Operational Primitive. Operational Primitive.
9.4 Data_ACK_Notify 9.4. Data_ACK_Notify
Input qualifiers: Connection_Handle, ITT, DataSN Input qualifiers: Connection_Handle, ITT, DataSN
Return Results: Not specified. Return Results: Not specified.
A target Datamover layer notifies its local iSCSI layer of A target Datamover layer notifies its local iSCSI layer of the
the arrival of a previously requested data acknowledgement arrival of a previously requested data acknowledgement from the peer
from the peer Datamover layer on the remote (initiator) iSCSI Datamover layer on the remote (initiator) iSCSI node via this
node via this Operational Primitive. The iSCSI layer Operational Primitive. The iSCSI layer processes the data
processes the data acknowledgement notification as defined in acknowledgement notification as defined in [RFC3720].
[RFC3720].
A target iSCSI layer being notified of the arrival of a data A target iSCSI layer being notified of the arrival of a data
acknowledgement for a certain SCSI Read data PDU is the only acknowledgement for a certain SCSI Read data PDU is the only example
example of invoking the Data_ACK_Notify Operational of invoking the Data_ACK_Notify Operational Primitive.
Primitive.
10 Datamover Interface (DI) 10. Datamover Interface (DI)
10.1 Overview 10.1. Overview
This chapter describes the interactions model between iSCSI This section describes the model of interactions between iSCSI and
and Datamover layers when the iSCSI connection is Datamover- Datamover layers when the iSCSI connection is Datamover-assisted so
assisted so the iSCSI layer may carry out the following - the iSCSI layer may carry out the following:
- send iSCSI data-type PDUs and exchange iSCSI control-type - send iSCSI data-type PDUs and exchange iSCSI control-type PDUs,
PDUs, and and
- handle asynchronous notifications such as completion of - handle asynchronous notifications such as completion of data
data sequence transfer, and connection failure. sequence transfer and connection failure.
This chapter relies on the notion of Operational Primitives This chapter relies on the notion of Operational Primitives (Section
(section 5.4) to define DI. 5.4) to define DI.
10.2 Interactions for handling asynchronous notifications 10.2. Interactions for Handling Asynchronous Notifications
10.2.1 Connection termination 10.2.1. Connection Termination
As stated in section 9.2, the Datamover layer notifies the As stated in Section 9.2, the Datamover layer notifies the iSCSI
iSCSI layer of a failed or terminated connection via the layer of a failed or terminated connection via the
Connection_Terminate_Notify Operational Primitive. The iSCSI Connection_Terminate_Notify Operational Primitive. The iSCSI layer
layer MUST consider the connection as unusable upon the MUST consider the connection unusable upon the invocation of this
invocation of this Primitive and handle the connection Primitive and handle the connection termination as specified in
termination as specified in [RFC3720]. [RFC3720].
10.2.2 Data transfer completion 10.2.2. Data Transfer Completion
As stated in section 9.3, the Datamover layer notifies the As stated in Section 9.3, the Datamover layer notifies the iSCSI
iSCSI layer of a completed data transfer operation via the layer of a completed data transfer operation via the
Data_Completion_Notify Operational Primitive. The iSCSI Data_Completion_Notify Operational Primitive. The iSCSI layer
layer processes the transfer completion as specified in processes the transfer completion as specified in [RFC3720].
[RFC3720].
10.2.2.1 Completion of a requested SCSI Data transfer 10.2.2.1. Completion of a Requested SCSI Data Transfer
The Datamover layer, to notify the iSCSI layer of the To notify the iSCSI layer of the completion of a requested iSCSI
completion of a requested iSCSI data-type PDU transfer, uses data-type PDU transfer, the Datamover layer uses the
the Data_Completion_Notify Operational Primitive with the Data_Completion_Notify Operational Primitive with the following input
following input qualifiers. qualifiers.
a) Connection_Handle a) Connection_Handle.
b) ITT: Initiator Task Tag semantics as defined in
[RFC3720]
c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
for an iSCSI R2T PDU. The semantics for both types of
sequence numbers are as defined in [RFC3720].
The rationale for choosing SN is explained in section 9.3. c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN for an
iSCSI R2T PDU. The semantics for both types of sequence
numbers are as defined in [RFC3720].
Every invocation of the Data_Completion_Notify Operational The rationale for choosing SN is explained in Section 9.3.
Primitive MUST be preceded by an invocation of the Put_Data
or Get_Data Operational Primitive with the Notify_Enable
qualifier set by the iSCSI layer at an earlier point in time.
10.2.3 Data acknowledgement Every invocation of the Data_Completion_Notify Operational Primitive
MUST be preceded by an invocation of the Put_Data or Get_Data
Operational Primitive with the Notify_Enable qualifier set by the
iSCSI layer at an earlier point in time.
10.2.3. Data Acknowledgement
[RFC3720] allows the iSCSI targets to optionally solicit data [RFC3720] allows the iSCSI targets to optionally solicit data
acknowledgement from the initiator for one or more Data-in acknowledgement from the initiator for one or more Data-In PDUs, via
PDUs, via setting of the A-bit on a Data-in PDU. The setting of the A-bit on a Data-In PDU. The Data_ACK_Notify
Data_ACK_Notify Operational Primitive with the following Operational Primitive with the following input qualifiers is used by
input qualifiers is used by the target Datamover layer to the target Datamover layer to notify the local iSCSI layer of the
notify the local iSCSI layer of the arrival of data arrival of data acknowledgement of a previously solicited iSCSI read
acknowledgement of a previously solicited iSCSI read data data acknowledgement. This Operational Primitive thus is applicable
acknowledgement. This Operational Primitive thus is appli- only to iSCSI targets.
cable only to iSCSI targets.
a) Connection_Handle a) Connection_Handle.
b) ITT: Initiator Task Tag semantics as defined in [RFC3720] b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
c) DataSN: of the next SCSI Data-in PDU which immediately c) DataSN: of the next SCSI Data-In PDU, which immediately follows
follows the SCSI Data-in PDU with the A-bit set to which the SCSI Data-In PDU with the A-bit set to which this
this notification corresponds, with semantics as defined in notification corresponds, with semantics as defined in
[RFC3720]. [RFC3720].
Every invocation of the Data_ACK_Notify Operational Primitive Every invocation of the Data_ACK_Notify Operational Primitive MUST be
MUST be preceded by an invocation of the Put_Data Operational preceded by an invocation of the Put_Data Operational Primitive by
Primitive by the iSCSI target layer with the A-bit set to 1 the iSCSI target layer with the A-bit set to 1 at an earlier point in
at an earlier point in time. time.
10.3 Interactions for sending an iSCSI PDU 10.3. Interactions for Sending an iSCSI PDU
This section discusses the interactions model for sending This section discusses the model of interactions for sending each of
each of the iSCSI PDUs defined in [RFC3720]. A the iSCSI PDUs defined in [RFC3720]. A Connection_Handle (see
Connection_Handle (see section 5.3) is assumed to qualify Section 5.3) is assumed to qualify each of these interactions so that
each of these interactions so that the Datamover layer can the Datamover layer can route it to the appropriate Transport
route it to the appropriate Transport Connection. The Connection. The qualifying Connection_Handle is not explicitly
qualifying Connection_Handle is not explicitly listed in the listed in the subsequent sections.
subsequent sections.
Note that the defined list of input qualifiers represents the Note that the defined list of input qualifiers represents the
semantically required set for the Datamover layer to consider semantically required set for the Datamover layer to consider in
in implementing the Primitive in each interaction described implementing the Primitive in each interaction described in this
in this section (see section 5.4 for an elaboration). section (see Section 5.4 for an elaboration). Implementations may
Implementations may choose to deduce the qualifiers in ways choose to deduce the qualifiers in ways that are optimized for the
that are optimized for the implementation specifics. Two implementation specifics. Two examples of this are:
examples of this are:
1. For SCSI Command (section 10.3.1), deducing the 1. For SCSI command (Section 10.3.1), deducing the
ImmediateDataSize input qualifier from the ImmediateDataSize input qualifier from the DataSegmentLength
DataSegmentLength field of the SCSI Command PDU. field of the SCSI Command PDU.
2. For SCSI Data-Out (section 10.3.5.1), deducing the 2. For SCSI Data-Out (Section 10.3.5.1), deducing the
DataDescriptorOut input qualifier from the associated DataDescriptorOut input qualifier from the associated SCSI
SCSI Command invocation qualifiers (assuming such state command invocation qualifiers (assuming such state is
is maintained) in conjunction with BHS fields of the maintained) in conjunction with BHS fields of the SCSI Data-Out
SCSI Data-out PDU. PDU.
10.3.1 SCSI Command 10.3.1. SCSI Command
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a SCSI Command
SCSI Command PDU. PDU.
a) BHS and AHS, if any, of the SCSI Command PDU as defined in a) BHS and AHS, if any, of the SCSI Command PDU as defined in
[RFC3720] [RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer meant for b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
Data-out for the entire command, in the case of a write or Out for the entire command, in the case of a write or
bidirectional command bidirectional command.
c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
for the entire command, in the case of a read or bidirectional
command.
c) DataDescriptorIn: that defines the I/O Buffer meant for
Data-in for the entire command, in the case of a read or
bidirectional command
d) ImmediateDataSize: that defines the number of octets of d) ImmediateDataSize: that defines the number of octets of
immediate unsolicited data for a write/bidirectional immediate unsolicited data for a write/bidirectional command.
command
e) UnsolicitedDataSize: that defines the number of octets of e) UnsolicitedDataSize: that defines the number of octets of
immediate and non-immediate unsolicited data for a immediate and non-immediate unsolicited data for a
write/bidirectional command. write/bidirectional command.
10.3.2 SCSI Response 10.3.2. SCSI Response
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a SCSI Response
SCSI Response PDU. PDU.
a) BHS of the SCSI Response PDU as defined in [RFC3720] a) BHS of the SCSI Response PDU as defined in [RFC3720].
b) DataDescriptorStatus: that defines the iSCSI buffer which b) DataDescriptorStatus: that defines the iSCSI buffer that
contains the sense and response information for the command contains the sense and response information for the command.
10.3.3 Task Management Function Request 10.3.3. Task Management Function Request
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a Task
Task Management Function Request PDU. Management Function Request PDU.
a) BHS of the Task Management Function Request PDU as defined a) BHS of the Task Management Function Request PDU as defined in
in [RFC3720] [RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer meant for b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
Data-out for the entire command, in the case of a write or Out for the entire command, in the case of a write or
bidirectional command (Only valid if Function="TASK bidirectional command. (Only valid if Function="TASK REASSIGN"
REASSIGN" - [RFC3720] ] - [RFC3720].)
c) DataDescriptorIn: that defines the I/O Buffer meant for c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
Data-in for the entire command, in the case of a read or for the entire command, in the case of a read or bidirectional
bidirectional command (Only valid if Function="TASK command. (Only valid if Function="TASK REASSIGN" - [RFC3720].)
REASSIGN" - [RFC3720] )
10.3.4 Task Management Function Response 10.3.4. Task Management Function Response
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifier is used for requesting the transmission of a qualifier is used for requesting the transmission of a Task
Task Management Function Response PDU. Management Function Response PDU.
a) BHS of the Task Management Function Response PDU as defined a) BHS of the Task Management Function Response PDU as defined in
in [RFC3720] [RFC3720].
10.3.5 SCSI Data-out & SCSI Data-in 10.3.5. SCSI Data-Out and SCSI Data-In
10.3.5.1 SCSI Data-out 10.3.5.1. SCSI Data-Out
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used by the initiator iSCSI layer for qualifiers is used by the initiator iSCSI layer for requesting the
requesting the transmission of a SCSI Data-out PDU carrying transmission of a SCSI Data-Out PDU carrying the non-immediate
the non-immediate unsolicited data. unsolicited data.
a) BHS of the SCSI Data-out PDU as defined in [RFC3720] a) BHS of the SCSI Data-Out PDU as defined in [RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer with the b) DataDescriptorOut: that defines the I/O Buffer with the Data-
Data-out to be carried in the iSCSI data segment of the PDU Out to be carried in the iSCSI data segment of the PDU.
10.3.5.2 SCSI Data-in 10.3.5.2. SCSI Data-In
The Put_Data Operational Primitive with the following input The Put_Data Operational Primitive with the following input
qualifiers is used by the target iSCSI layer for requesting qualifiers is used by the target iSCSI layer for requesting the
the transmission of the data carried by a SCSI Data-in PDU. transmission of the data carried by a SCSI Data-In PDU.
a) BHS of the SCSI Data-in PDU as defined in [RFC3720] a) BHS of the SCSI Data-In PDU as defined in [RFC3720].
b) DataDescriptorIn: that defines the I/O Buffer with the b) DataDescriptorIn: that defines the I/O Buffer with the Data-In
Data-in being requested for transmission being requested for transmission.
10.3.6 Ready To Transfer (R2T) 10.3.6. Ready To Transfer (R2T)
The Get_Data Operational Primitive with the following input The Get_Data Operational Primitive with the following input
qualifiers is used by the target iSCSI layer for requesting qualifiers is used by the target iSCSI layer for requesting the
the retrieval of the data as specified by the semantic retrieval of the data as specified by the semantic content of an R2T
content of an R2T PDU. PDU.
a) BHS of the Ready To Transfer PDU as defined in [RFC3720] a) BHS of the Ready To Transfer PDU as defined in [RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer for the b) DataDescriptorOut: that defines the I/O Buffer for the Data-Out
Data-out being requested for retrieval being requested for retrieval.
10.3.7 Asynchronous Message 10.3.7. Asynchronous Message
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of qualifiers is used for requesting the transmission of an Asynchronous
an Asynchronous Message PDU. Message PDU.
a) BHS of the Asynchronous Message PDU as defined in [RFC3720] a) BHS of the Asynchronous Message PDU as defined in [RFC3720].
b) DataDescriptorSense: that defines an iSCSI buffer which b) DataDescriptorSense: that defines an iSCSI buffer that contains
contains the sense and iSCSI Event information. the sense and iSCSI Event information.
10.3.8 Text Request 10.3.8. Text Request
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a Text Request
Text Request PDU. PDU.
a) BHS of the Text Request PDU as defined in [RFC3720] a) BHS of the Text Request PDU as defined in [RFC3720].
b) DataDescriptorTextOut: that defines the iSCSI Text Request b) DataDescriptorTextOut: that defines the iSCSI Text Request
buffer buffer.
10.3.9 Text Response 10.3.9. Text Response
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a Text Response
Text Response PDU. PDU.
a) BHS of the Text Response PDU as defined in [RFC3720].
a) BHS of the Text Response PDU as defined in [RFC3720]
b) DataDescriptorTextIn: that defines the iSCSI Text Response b) DataDescriptorTextIn: that defines the iSCSI Text Response
buffer buffer.
10.3.10 Login Request 10.3.10. Login Request
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a Login Request
Login Request PDU. PDU.
a) BHS of the Login Request PDU as defined in [RFC3720] a) BHS of the Login Request PDU as defined in [RFC3720].
b) DataDescriptorLoginRequest: that defines the iSCSI Login b) DataDescriptorLoginRequest: that defines the iSCSI Login
Request buffer Request buffer.
Note that specific Datamover protocols may choose to disallow Note that specific Datamover protocols may choose to disallow the
the standard DA Primitives from being used for the iSCSI standard DA Primitives from being used for the iSCSI Login Phase.
Login phase. When used in conjunction with such Datamover When used in conjunction with such Datamover protocols, an attempt to
protocols, an attempt to send a Login Request via the send a Login Request via the Send_Control Operational Primitive
Send_Control Operational Primitive invocation is clearly an invocation is clearly an error scenario, as the Login Request PDU is
error scenario, as the Login Request PDU is being sent while being sent while the connection is in the iSCSI Full Feature Phase.
the connection is in the iSCSI full feature phase. It is It is outside the scope of this document to specify the resulting
outside the scope of this document to specify the resulting implementation behavior in this case -- [RFC3720] already defines the
implementation behavior in this case - [RFC3720] already error handling for this error scenario.
defines the error handling for this error scenario.
10.3.11 Login Response 10.3.11. Login Response
The Send_Control Operational Primitive with the following The Send_Control Operational Primitive with the following input
input qualifiers is used for requesting the transmission of a qualifiers is used for requesting the transmission of a Login
Login Response PDU. Response PDU.
a) BHS of the Login Response PDU as defined in [RFC3720] a) BHS of the Login Response PDU as defined in [RFC3720].
b) DataDescriptorLoginResponse: that defines the iSCSI Login b) DataDescriptorLoginResponse: that defines the iSCSI Login
Response buffer Response buffer.
Note that specific Datamover protocols may choose to disallow
the standard DA Primitives from being used for the iSCSI
Login phase. When used in conjunction with such Datamover
protocols, an attempt to send a Login Response via the
Send_Control Operational Primitive invocation is clearly an
error scenario, as the Login Response PDU is being sent while
in the iSCSI full feature phase. It is outside the scope of
this document to specify the resulting implementation
behavior in this case - [RFC3720] already defines the error
handling for this error scenario.
10.3.12 Logout Command
The Send_Control Operational Primitive with the following Note that specific Datamover protocols may choose to disallow the
input qualifier is used for requesting the transmission of a standard DA Primitives from being used for the iSCSI Login Phase.
Logout Command PDU. When used in conjunction with such Datamover protocols, an attempt to
send a Login Response via the Send_Control Operational Primitive
invocation is clearly an error scenario, as the Login Response PDU is
being sent while in the iSCSI Full Feature Phase. It is outside the
scope of this document to specify the resulting implementation
behavior in this case -- [RFC3720] already defines the error handling
for this error scenario.
a) BHS of the Logout Command PDU as defined in [RFC3720] 10.3.12. Logout Command
10.3.13 Logout Response The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a Logout Command
PDU.
The Send_Control Operational Primitive with the following a) BHS of the Logout Command PDU as defined in [RFC3720].
input qualifier is used for requesting the transmission of a
Logout Response PDU.
a) BHS of the Logout Response PDU as defined in [RFC3720] 10.3.13. Logout Response
10.3.14 SNACK Request The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a Logout
Response PDU.
The Send_Control Operational Primitive with the following a) BHS of the Logout Response PDU as defined in [RFC3720].
input qualifier is used for requesting the transmission of a
SNACK Request PDU.
a) BHS of the SNACK Request PDU as defined in [RFC3720] 10.3.14. SNACK Request
10.3.15 Reject The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a SNACK Request
PDU.
The Send_Control Operational Primitive with the following a) BHS of the SNACK Request PDU as defined in [RFC3720].
input qualifiers is used for requesting the transmission of a
Reject PDU.
a) BHS of the Reject PDU as defined in [RFC3720] 10.3.15. Reject
b) DataDescriptorReject: that defines the iSCSI Reject buffer The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Reject PDU.
10.3.16 NOP-Out a) BHS of the Reject PDU as defined in [RFC3720].
The Send_Control Operational Primitive with the following b) DataDescriptorReject: that defines the iSCSI Reject buffer.
input qualifiers is used for requesting the transmission of a
NOP-Out PDU.
a) BHS of the NOP-Out PDU as defined in [RFC3720] 10.3.16. NOP-Out
b) DataDescriptorNOPOut: that defines the iSCSI Ping data The Send_Control Operational Primitive with the following input
buffer qualifiers is used for requesting the transmission of a NOP-Out PDU.
10.3.17 NOP-In a) BHS of the NOP-Out PDU as defined in [RFC3720].
The Send_Control Operational Primitive with the following b) DataDescriptorNOPOut: that defines the iSCSI Ping data buffer.
input qualifiers is used for requesting the transmission of a
NOP-In PDU.
a) BHS of the NOP-In PDU as defined in [RFC3720] 10.3.17. NOP-In
b) DataDescriptorNOPIn: that defines the iSCSI Return Ping The Send_Control Operational Primitive with the following input
data buffer qualifiers is used for requesting the transmission of a NOP-In PDU.
10.4 Interactions for receiving an iSCSI PDU a) BHS of the NOP-In PDU as defined in [RFC3720].
The only PDUs that are received by an iSCSI layer operating b) DataDescriptorNOPIn: that defines the iSCSI Return Ping data
on a Datamover layer are the iSCSI control-type PDUs. The buffer.
Datamover layer delivers the iSCSI control-type PDUs as they
arrive, qualifying each with the Connection_Handle (see
section 5.3) that identifies the iSCSI connection the PDU is
meant for. The subsequent processing of the iSCSI control-
type PDUs proceeds as defined in [RFC3720].
10.4.1 General Control-type PDU notification 10.4. Interactions for Receiving an iSCSI PDU
This sub-section describes the general mechanics applicable The only PDUs that are received by an iSCSI layer operating on a
to several control-type PDUs. The following sub-sections Datamover layer are the iSCSI control-type PDUs. The Datamover layer
note additional considerations for control-type PDUs not delivers the iSCSI control-type PDUs as they arrive, qualifying each
covered in this sub-section. with the Connection_Handle (see Section 5.3) that identifies the
iSCSI connection for which the PDU is meant. The subsequent
processing of the iSCSI control-type PDUs proceeds as defined in
[RFC3720].
The Control_Notify Operational Primitive is used for 10.4.1. General Control-Type PDU Notification
notifying the arrival of the following iSCSI control-type
PDUs: SCSI Command, SCSI Response, Task Management Function
Request, Task Management Function Response, Asynchronous
Message, Text Request, Text Response, Logout command, Logout
Response, SNACK, Reject, NOP-Out, NOP-In.
10.4.2 SCSI Data Transfer PDUs This sub-section describes the general mechanics applicable to
several control-type PDUs. The following sub-sections note
additional considerations for control-type PDUs that are not covered
in this sub-section.
10.4.2.1 SCSI Data-out The Control_Notify Operational Primitive is used to notify the iSCSI
layer of the arrival of the following iSCSI control-type PDUs: SCSI
Command, SCSI Response, Task Management Function Request, Task
Management Function Response, Asynchronous Message, Text Request,
Text Response, Logout Command, Logout Response, SNACK, Reject, NOP-
Out, NOP-In.
The Control_Notify Operational Primitive is used for 10.4.2. SCSI Data Transfer PDUs
notifying the iSCSI layer of the arrival of a SCSI Data-out
PDU carrying the non-immediate unsolicited data. Note
however that the solicited SCSI Data-out arriving on the
target is not notified to the iSCSI layer using the
Control_Notify Primitive because the solicited SCSI Data-out
was not sent by the initiator iSCSI layer as control-type
PDUs.
10.4.2.2 SCSI Data-in 10.4.2.1. SCSI Data-Out
The arrival of the SCSI Data-in is not notified to the iSCSI The Control_Notify Operational Primitive is used to notify the iSCSI
layer by the Datamover layer at the initiator, because SCSI layer of the arrival of a SCSI Data-Out PDU carrying the non-
Data-in is an iSCSI data-type PDU (see section 5.1). The immediate unsolicited data. Note however that the solicited SCSI
iSCSI layer at the initiator however may infer the arrival of Data-Out arriving on the target does not cause a notification to the
the SCSI Data-in when it receives a subsequent notification iSCSI layer using the Control_Notify Primitive because the solicited
of the SCSI Response PDU via a Control_Notify invocation. SCSI Data-Out was not sent by the initiator iSCSI layer as control-
type PDUs.
While this document does not contemplate the possibility of a 10.4.2.2. SCSI Data-In
Data-in PDU being received at the initiator iSCSI layer,
specific Datamover protocols may define how to deal with an
unexpected inbound SCSI Data-in PDU that may result in the
initiator iSCSI layer receiving the Data-in PDU. This
document leaves the details of handling this error scenario
to the specific Datamover protocols, so each may define the
appropriate error handling specific to the Datamover
environment.
10.4.2.3 Ready To Transfer (R2T) The Datamover layer does not notify the iSCSI layer of the arrival of
the SCSI Data-in at the initiator, because SCSI Data-in is an iSCSI
data-type PDU (see section 5.1). The iSCSI layer at the initiator
however may infer the arrival of the SCSI Data-In when it receives a
subsequent notification of the SCSI Response PDU via a Control_Notify
invocation.
Because an R2T PDU is an iSCSI data-type PDU (see section While this document does not contemplate the possibility of a Data-In
5.1) that is not delivered as-is to the initiator iSCSI PDU being received at the initiator iSCSI layer, specific Datamover
layer, the arrival of an R2T PDU is not notified to the iSCSI protocols may define how to deal with an unexpected inbound SCSI
layer by the Datamover layer. When an iSCSI node sends an Data-In PDU that may result in the initiator iSCSI layer receiving
R2T PDU to its local Datamover layer, the local and remote the Data-In PDU. This document leaves the details of handling this
Datamover layers transparently bring about the data transfer error scenario to the specific Datamover protocols, so each may
requested by the R2T PDU. define the appropriate error handling specific to the Datamover
environment.
While this document does not contemplate the possibility of 10.4.2.3. Ready To Transfer (R2T)
an R2T PDU being received at the initiator iSCSI layer,
specific Datamover protocols may define how to deal with an
unexpected inbound R2T PDU that may result in the initiator
iSCSI layer receiving the R2T PDU. This document leaves the
details of handling this error scenario to the specific
Datamover protocols, so each may define the appropriate error
handling specific to the Datamover environment.
10.4.3 Login Request Because an R2T PDU is an iSCSI data-type PDU (see Section 5.1) that
is not delivered as-is to the initiator iSCSI layer, the Datamover
layer does not notify the iSCSI layer of the arrival of an R2T PDU.
When an iSCSI node sends an R2T PDU to its local Datamover layer, the
local and remote Datamover layers transparently bring about the data
transfer requested by the R2T PDU.
The Control_Notify Operational Primitive is used for While this document does not contemplate the possibility of an R2T
notifying the target iSCSI layer of the arrival of a Login PDU being received at the initiator iSCSI layer, specific Datamover
Request PDU. Note that specific Datamover protocols may protocols may define how to deal with an unexpected inbound R2T PDU
choose to disallow the standard DA Primitives from being used that may result in the initiator iSCSI layer receiving the R2T PDU.
for the iSCSI Login phase. When used in conjunction with This document leaves the details of handling this error scenario to
such Datamover protocols, the arrival of a Login Request the specific Datamover protocols, so each may define the appropriate
necessitating the Control_Notify Operational Primitive error handling specific to the Datamover environment.
invocation is clearly an error scenario, as the Login Request
PDU is arriving in the iSCSI full feature phase. It is
outside the scope of this document to specify the resulting
implementation behavior in this case - [RFC3720] already
defines the error handling in this error scenario.
10.4.4 Login Response 10.4.3. Login Request
The Control_Notify Operational Primitive is used for The Control_Notify Operational Primitive is used for notifying the
notifying the initiator iSCSI layer of the arrival of a Login target iSCSI layer of the arrival of a Login Request PDU. Note that
Response PDU. Note that specific Datamover protocols may specific Datamover protocols may choose to disallow the standard DA
choose to disallow the standard DA Primitives from being used Primitives from being used for the iSCSI Login Phase. When used in
for the iSCSI Login phase. When used in conjunction with conjunction with such Datamover protocols, the arrival of a Login
such Datamover protocols, the arrival of a Login Response Request necessitating the Control_Notify Operational Primitive
necessitating the Control_Notify Operational Primitive invocation is clearly an error scenario, as the Login Request PDU is
invocation is clearly an error scenario, as the Login arriving in the iSCSI Full Feature Phase. It is outside the scope of
Response PDU is arriving in the iSCSI full feature phase. It this document to specify the resulting implementation behavior in
is outside the scope of this document to specify the this case -- [RFC3720] already defines the error handling in this
resulting implementation behavior in this case - [RFC3720] error scenario.
already defines the error handling in this error scenario.
11 Security Considerations 10.4.4. Login Response
11.1 Architectural Considerations The Control_Notify Operational Primitive is used to notify the
initiator iSCSI layer of the arrival of a Login Response PDU. Note
that specific Datamover protocols may choose to disallow the standard
DA Primitives from being used for the iSCSI Login Phase. When used
in conjunction with such Datamover protocols, the arrival of a Login
Response necessitating the Control_Notify Operational Primitive
invocation is clearly an error scenario, as the Login Response PDU is
arriving in the iSCSI Full Feature Phase. It is outside the scope of
this document to specify the resulting implementation behavior in
this case -- [RFC3720] already defines the error handling in this
error scenario.
DA enables compliant iSCSI implementations to realize a 11. Security Considerations
control and data separation in the way they interact with
their Datamover protocols. Note however that this separation
does not imply a separation in transport mediums between
control traffic and data traffic - basic iSCSI architecture
with respect to tasks and PDU relationships to tasks remains
unchanged. [RFC3720] defines several MUST requirements on
ordering relationships across control and data for a given
task besides a mandatory deterministic task allegiance model
- DA does not change this basic architecture (DA has a
normative reference on [RFC3720]) nor allow any additional
flexibility in compliance in this area. To summarize,
sending bulk data transfers (prompted by Put_Data and
Get_Data Primitive invocations) on a different transport
medium would be as ill-advised as sending just the Data-
out/Data-in PDUs on a different TCP connection in RFC 3720-
based iSCSI implementations. Consequently, all the iSCSI-
related security text in [RFC3723] is directly applicable to
a DA-enabled iSCSI implementation.
Another area with security implications is the Datamover 11.1. Architectural Considerations
connection resource management model which DA defines -
particularly the Allocate_Connection_Resources Primitive. An
inadvertent realization of this model could leave an iSCSI
implementation exposed to denial of service attacks. As
Figure 2 and Figure 3 in section 16.2 illustrate, the most
effective countermeasure to this potential attack consists of
performing the Datamover resource allocation when the iSCSI
layer is sufficiently far along in the iSCSI Login Phase that
it is reasonably certain that the peer side is not an
attacker. In particular, if the Login Phase includes a
SecurityNegotiation stage, an iSCSI end node MUST defer the
Datamover connection resource allocation (i.e. invoking the
Allocate_Connection_Resources Primitive) to the
LoginOperationalNegotiation stage ([RFC3720]) so that the
resource allocation happens post-authentication. This
considerably minimizes the potential for a denial of service
attack.
11.2 Wire Protocol Considerations DA enables compliant iSCSI implementations to realize a control and
data separation in the way they interact with their Datamover
protocols. Note however that this separation does not imply a
separation in transport mediums between control traffic and data
traffic -- the basic iSCSI architecture with respect to tasks and PDU
relationships to tasks remains unchanged. [RFC3720] defines several
MUST requirements on ordering relationships across control and data
for a given task besides a mandatory deterministic task allegiance
model -- DA does not change this basic architecture (DA has a
normative reference to [RFC3720]) for allow any additional
flexibility in compliance in this area. To summarize, sending bulk
data transfers (prompted by Put_Data and Get_Data Primitive
invocations) on a different transport medium would be as ill-advised
as sending just the Data-Out/Data-In PDUs on a different TCP
connection in RFC 3720-based iSCSI implementations. Consequently,
all the iSCSI-related security text in [RFC3723] is directly
applicable to a DA-enabled iSCSI implementation.
In view of the fact that the DA architecture itself does not Another area with security implications is the Datamover connection
define any new wire protocol nor propose modifications to the resource management model, which DA defines -- particularly the
existing protocols, there are no additional wire protocol Allocate_Connection_Resources Primitive. An inadvertent realization
security considerations in employing DA itself. However, a of this model could leave an iSCSI implementation exposed to denial-
DA-compliant iSCSI implementation MUST comply with all the of-service attacks. As Figures 2 and 3 in Section 13.2 illustrate,
iSCSI-related requirements stipulated in [RFC3723] and the most effective countermeasure to this potential attack consists
[RFC3720]. Note further that in realizing DA, each Datamover of performing the Datamover resource allocation when the iSCSI layer
protocol must define and elaborate as appropriate on any is sufficiently far along in the iSCSI Login Phase that it is
additional security considerations resulting from the use of reasonably certain that the peer side is not an attacker. In
that Datamover protocol. particular, if the Login Phase includes a SecurityNegotiation stage,
an iSCSI end node MUST defer the Datamover connection resource
allocation (i.e., invoking the Allocate_Connection_Resources
Primitive) to the LoginOperationalNegotiation stage [RFC3720] so that
the resource allocation happens post-authentication. This
considerably minimizes the potential for a denial-of service attack.
All Datamover protocol designers are strongly recommended to 11.2. Wire Protocol Considerations
refer to [RDDPSEC] for the types of security issues to
consider. While [RDDPSEC] elaborates on the security
considerations applicable to an RDDP-based Datamover
([iSER]), the document is representative of the type of
analysis of resource exhaustion and the application of
countermeasures that needs to be done for any Datamover
protocol.
12 IANA Considerations In view of the fact that the DA architecture itself does not define
any new wire protocol or propose modifications to the existing
protocols, there are no additional wire protocol security
considerations in employing DA itself. However, a DA-compliant iSCSI
implementation MUST comply with all the iSCSI-related requirements
stipulated in [RFC3723] and [RFC3720]. Note further that in
realizing DA, each Datamover protocol must define and elaborate as
appropriate on any additional security considerations resulting from
the use of that Datamover protocol.
DA architecture does not have any IANA considerations. All Datamover protocol designers are strongly recommended to refer to
[RDDPSEC] for the types of security issues to consider. While
[RDDPSEC] elaborates on the security considerations applicable to an
RDDP-based Datamover [iSER], the document is representative of the
type of analysis of resource exhaustion and the application of
countermeasures that need to be done for any Datamover protocol.
13 References and Bibliography 12. References
13.1 Normative References 12.1. Normative References
[RFC3720] J. Satran, K. Meth, C. Sapuntzakis, M. Chadalapaka, [RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and
E. Zeidner, "Internet Small Computer Systems Interface E. Zeidner, "Internet Small Computer Systems Interface
(iSCSI)", RFC 3720, April 2004. (iSCSI)", RFC 3720, April 2004.
[RFC3723] B. Aboba, J. Tseng, J. Walker, V. Rangan, F. [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
Travostino, "Securing Block Storage Protocols over IP", Travostino, "Securing Block Storage Protocols over IP", RFC
RFC 3723, April 2004. 3723, April 2004.
[RFC2119] S. Bradner, "Key words for use in RFCs to Indicate
Requirement Levels", March 1997.
13.2 Informative References
[DDP] H. Shah et al., "Direct Data Placement over Reliable
Transports", IETF Internet Draft draft-ietf-rddp-ddp-
06.txt (work in progress), June 2006.
[iSER] M. Ko et al., "iSCSI Extensions for RDMA", IETF
Internet Draft draft-ietf-ips-iser-03.txt (work in
progress), April 2005.
[RDDPSEC] J. Pinkerton et al., "DDP/RDMAP Security", IETF
Internet Draft draft-ietf-rddp-security-07.txt (work in
14 Authors' Addresses
Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA
Phone: +1-916-785-5621
E-mail: cbm@rose.hp.com
John L. Hufferd
IBM
San Jose CA, USA
Phone: +1-408-256-0403
E-mail: hufferd@us.ibm.com
Julian Satran
IBM, Haifa Research Lab
Haifa University Campus - Mount Carmel
Haifa 31905, Israel
Phone +972-4-829-6264
E-mail: Julian_Satran@il.ibm.com
Hemal Shah
Intel Corporation
MS PTL1
1501 South Mopac Expressway, #400
Austin, TX 78746 USA
Phone: +1 (512) 732-3963
Email: hemal.shah@intel.com
Comments may be sent to Mallikarjun Chadalapaka.
15 Acknowledgements
The IP Storage (ips) Working Group in the Transport Area of
IETF has been responsible for defining the iSCSI protocol
(apart from a host of other relevant IP Storage protocols).
The authors are grateful to the entire working group, whose
work allowed this document to build on the concepts and
details of the iSCSI protocol.
In addition, the following individuals had reviewed and
contributed to the improvement of this document. The authors
are grateful for their contribution.
John Carrier
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035 USA
Phone: +1 (360) 378-8526
Email: john_carrier@adaptec.com
Hari Ghadia
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035 USA
Phone: +1 (408) 957-5608
Email: hari_ghadia@adaptec.com
Hari Mudaliar
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035 USA
Phone: +1 (408) 957-6012
Email: hari_mudaliar@adaptec.com
Patricia Thaler
Agilent Technologies, Inc.
1101 Creekside Ridge Drive, #100, M/S-RG10,
Roseville, CA 95678
Phone: +1-916-788-5662
email: pat_thaler@agilent.com
Uri Elzur
Broadcom Corporation
16215 Alton Parkway, Irvine, CA 92619-7013 USA
Phone: +1 (949) 585-6432
Email: Uri@Broadcom.com
Mike Penna
Broadcom Corporation
16215 Alton Parkway,Irvine, CA 92619-7013 USA
Phone: +1 (949) 926-7149
Email: MPenna@Broadcom.com
David Black
EMC Corporation
176 South St., Hopkinton, MA 01748, USA
Phone: +1 (508) 293-7953
Email: black_david@emc.com
Ted Compton
EMC Corporation
Research Triangle Park, NC 27709, USA
Phone: +1-919-248-6075
Email: compton_ted@emc.com
Dwight Barron
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698 USA
Phone: +1 (281) 514-2769
Email: Dwight.Barron@Hp.com
Paul R. Culley
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698 USA
Phone: +1 (281) 514-5543
Email: paul.culley@hp.com
Dave Garcia
Hewlett-Packard Company
19333 Vallco Parkway, Cupertino, Ca. 95014 USA
Phone: +1 (408) 285-6116
Email: dave.garcia@hp.com
Randy Haagens
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA
Phone: +1-916-785-4578
email: randy_haagens@hp.com
Jeff Hilland
Hewlett-Packard Company
20555 SH 249, Houston, Tx. 77070-2698 USA
Phone: +1 (281) 514-9489
Email: jeff.hilland@hp.com
Mike Krause
Hewlett-Packard Company, 43LN
19410 Homestead Road, Cupertino, CA 95014 USA
Phone: +1 (408) 447-3191
Email: krause@cup.hp.com
Jim Wendt
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA
Phone: +1-916-785-5198
email: jim_wendt@hp.com
Mike Ko [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
IBM Requirement Levels", BCP 14, RFC 2119, March 1997.
650 Harry Rd, San Jose, CA 95120
Phone: +1 (408) 927-2085
Email: mako@us.ibm.com
Renato Recio 12.2. Informative References
IBM Corporation
11501 Burnett Road, Austin, TX 78758 USA
Phone: +1 (512) 838-1365
Email: recio@us.ibm.com
Howard C. Herbert
Intel Corporation
MS CH7-404,5000 West Chandler Blvd., Chandler, AZ 85226 USA
Phone: +1 (480) 554-3116
Email: howard.c.herbert@intel.com
Dave Minturn [DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
Intel Corporation Data Placement over Reliable Transports", RFC 5041, October
MS JF1-210, 5200 North East Elam Young Parkway 2007.
Hillsboro, OR 97124 USA
Phone: +1 (503) 712-4106
Email: dave.b.minturn@intel.com
James Pinkerton [iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H.,
Microsoft Corporation and P. Thaler, "Internet Small Computer System Interface
One Microsoft Way, Redmond, WA 98052 USA (iSCSI) Extensions for Remote Direct Memory Access (RDMA)",
Phone: +1 (425) 705-5442 RFC 5046, October 2007.
Email: jpink@microsoft.com
Tom Talpey [RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement
Network Appliance Protocol (DDP) / Remote Direct Memory Access Protocol
375 Totten Pond Road, Waltham, MA 02451 USA (RDMAP) Security", RFC 5042, October 2007.
Phone: +1 (781) 768-5329
EMail: thomas.talpey@netapp.com
16 Appendix Appendix A. Design Considerations and Examples
16.1 Design considerations for a Datamover protocol A.1. Design Considerations for a Datamover Protocol
This section discusses the specific considerations for RDMA- This section discusses the specific considerations for RDMA-based and
based and RDDP-based Datamover protocols. RDDP-based Datamover protocols.
a) Note that the modeling of interactions for SCSI Data-Out a) Note that the modeling of interactions for SCSI Data-Out
(section 10.3.5.1) is only used for unsolicited data (Section 10.3.5.1) is only used for unsolicited data transfer.
transfer.
b) The modeling of interactions for SNACK (section 10.3.14, b) The modeling of interactions for SNACK (Sections 10.3.14 and
and section 10.4.1) is not expected to be used given that 10.4.1) is not expected to be used given that one of the design
one of the design requirements on the Datamover is that it requirements on the Datamover is that it "guarantees an error-
"guarantees an error-free, reliable, in-order transport free, reliable, in-order transport mechanism" (Section 6). The
mechanism" (section 6). The interactions for sending and interactions for sending and receiving a SNACK are nevertheless
receiving a SNACK are nevertheless modeled in this document modeled in this document because the receiving iSCSI layer can
because the receiving iSCSI layer can deterministically deterministically deal with an inadvertent SNACK. This also
deal with an inadvertent SNACK. This also shows the DA shows the DA designers' intent that DI is not meant to filter
designers' intent that DI is not meant to filter certain certain types of PDUs.
types of PDUs.
c) The onus is on a reliable Datamover (per requirements c) The onus is on a reliable Datamover (per requirements stated in
stated in section 6) to realize end-to-end data Section 6) to realize end-to-end data acknowledgements via
acknowledgements via Datamover-specific means. In view of Datamover-specific means. In view of this, even use of data-
this, even data-ACK-type SNACKs are unnecessary to be used. ACK-type SNACKs are unnecessary. Consequently, an initiator
Consequently, an initiator may never request sending a may never request sending a SNACK Request in this model
SNACK Request in this model assuming that the proactive assuming that the proactive (timeout-driven) SNACK
(timeout-driven) SNACK functionality is turned off in the functionality is turned off in the legacy iSCSI code.
legacy iSCSI code.
d) Note that the current DA model for bootstrapping a d) Note that the current DA model for bootstrapping a
Connection_Handle into service - i.e. associating a new Connection_Handle into service -- i.e., associating a new iSCSI
iSCSI connection with a Connection_Handle - clearly implies connection with a Connection_Handle -- clearly implies that the
that the iSCSI connection must already be in full feature iSCSI connection must already be in Full Feature Phase when the
phase when the Datamover layer comes into the stack. This Datamover layer comes into the stack. This further implies
further implies that the iSCSI login phase must be carried that the iSCSI Login Phase must be carried out in the
out in the traditional "Byte streaming mode" with no traditional "Byte streaming mode" with no assistance or
assistance or involvement from the Datamover layer. involvement from the Datamover layer.
16.2 Examples of Datamover interactions A.2. Examples of Datamover Interactions
The figures described in this section provide some examples The figures described in this section provide some examples of the
of the usage of Operational Primitives in interactions usage of Operational Primitives in interactions between the iSCSI
between the iSCSI layer and the Datamover layer. The layer and the Datamover layer. The following abbreviations are used
following abbreviations are used in this section. in this section.
Avail - Available Avail - Available
Abted - Aborted Abted - Aborted
Buf - I/O Buffer Buf - I/O Buffer
Cmd - Command Cmd - Command
Compl - Complete Compl - Complete
Conn - Connection Conn - Connection
Ctrl_Ntfy - Control_Notify Ctrl_Ntfy - Control_Notify
skipping to change at page 56, line 23 skipping to change at page 37, line 23
| L | Final Login Response (success) v succeeds | L | Final Login Response (success) v succeeds
| a |<----------------------------------------^ | a |<----------------------------------------^
| y | | L | | iSCSI | y | | L | | iSCSI
| e | Enable_Datamover | a | | Full | e | Enable_Datamover | a | | Full
| r |------------------------------->| y | | Feature | r |------------------------------->| y | | Feature
| | Datamover is enabled | e | | Phase | | Datamover is enabled | e | | Phase
| | | r | | | | | r | |
| | Full Feature Phase | | | | | Full Feature Phase | | |
| | control and data Transfer | | v | | control and data Transfer | | v
Figure 2 A successful iSCSI login on initiator Figure 2. A Successful iSCSI Login on Initiator
| | Notice_Key_Values | | | | | Notice_Key_Values | | |
| |------------------------------->| | | | |------------------------------->| | |
| | Datamover layer is notified | | | | | Datamover layer is notified | | |
| | of the negotiated key values | | | | | of the negotiated key values | | |
| | | | | | | | | |
| | Allocate_Connection_Resources | | | | | Allocate_Connection_Resources | | |
| |------------------------------->| D | | | |------------------------------->| D | |
| | Connection resources are | a | | | | Connection resources are | a | |
| i | successfully allocated | t | | iSCSI | i | successfully allocated | t | | iSCSI
skipping to change at page 56, line 49 skipping to change at page 37, line 49
| L | Datamover is enabled | r | ^ succeeds | L | Datamover is enabled | r | ^ succeeds
| a | | | | | a | | | |
| y | | L | | iSCSI | y | | L | | iSCSI
| e | | a | | Full | e | | a | | Full
| r | | y | | Feature | r | | y | | Feature
| | | e | | Phase | | | e | | Phase
| | Full Feature Phase | r | | | | Full Feature Phase | r | |
| | control and data Transfer | | | | | control and data Transfer | | |
| | | | v | | | | v
Figure 3 A successful iSCSI login on target Figure 3. A Successful iSCSI Login on Target
| | Allocate_Connection_Resources | D | ^ | | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | | | |------------------------------->| a | |
| | Connection resources are | t | | | | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI | i | successfully allocated | a | | iSCSI
| S | | m | | Login | S | | m | | Login
| C | | o | | Phase | C | | o | | Phase
| S | | v | | | S | | v | |
| I | | e | | | I | | e | |
| | | r | | Login | | | r | | Login
| | | | | Phase | | | | | Phase
skipping to change at page 57, line 26 skipping to change at page 38, line 26
| y | | L | | y | | L |
| e | Deallocate_Connection_Resources| a | | e | Deallocate_Connection_Resources| a |
| r |------------------------------->| y | | r |------------------------------->| y |
| | Datamover-specific | e | | | Datamover-specific | e |
| | connection resources freed | r | | | connection resources freed | r |
| | | | | | | |
| | | |
| | Connection terminated by standard means | | Connection terminated by standard means
| |---------------------------------------------> | |--------------------------------------------->
Figure 4 A failed iSCSI login on initiator Figure 4. A Failed iSCSI Login on Initiator
| | Allocate_Connection_Resources | D | ^ | | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | | | |------------------------------->| a | |
| | Connection resources are | t | | | | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI | i | successfully allocated | a | | iSCSI
| S | | m | | Login | S | | m | | Login
| C | | o | | Phase | C | | o | | Phase
| S | | v | | | S | | v | |
| I | | e | | | I | | e | |
| | | r | | Login | | | r | | Login
skipping to change at page 57, line 50 skipping to change at page 38, line 50
| y | | L | | y | | L |
| e | Deallocate_Connection_Resources| a | | e | Deallocate_Connection_Resources| a |
| r |------------------------------->| y | | r |------------------------------->| y |
| | Datamover-specific | e | | | Datamover-specific | e |
| | connection resources freed | r | | | connection resources freed | r |
| | | | | | | |
| | | |
| | Connection terminated by standard means | | Connection terminated by standard means
| |--------------------------------------------> | |-------------------------------------------->
Figure 5 A failed iSCSI login on target Figure 5. A Failed iSCSI Login on Target
| | Allocate_Connection_Resources | D | ^ | | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | | | |------------------------------->| a | |
| | Connection resources are | t | | | | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI | i | successfully allocated | a | | iSCSI
| S | | m | | Login | S | | m | | Login
| C | | o | | Phase | C | | o | | Phase
| S | | v | | | S | | v | |
| I | | e | | | I | | e | |
| | | r | | | | | r | |
| L | Login non-Final Request/Response | | L | Login non-Final Request/Response |
skipping to change at page 58, line 32 skipping to change at page 39, line 32
| | resources deallocated | | | | | resources deallocated | | |
| | | | | Login | | | | | Login
| | | | | Phase | | | | | Phase
| | | continues | | | continues
| | Regular Login negotiation continues | | | Regular Login negotiation continues |
| |<---------------------------------------->| | |<---------------------------------------->|
| | . | | .
| | . | | .
| | . | | .
Figure 6 iSCSI does not enable the Datamover Figure 6. iSCSI Does Not Enable the Datamover
| | | | ^ | | | | ^
| | Full Feature Phase Control & | | | | | Full Feature Phase Control & | | |
| | Data Transfer Using DM | D | | iSCSI | | Data Transfer Using DM | D | | iSCSI
| | | a | | Full Feature | | | a | | Full Feature
| i | | t | | Phase | i | | t | | Phase
| S | | a | | (DM Enabled) | S | | a | | (DM Enabled)
| C | | m | | | C | | m | |
| S | Successful iSCSI Logout | o | | | S | Successful iSCSI Logout | o | |
| I | | v | v | I | | v | v
| | Connection_Terminate | e | | | Connection_Terminate | e |
| L |------------------------------->| r | | L |------------------------------->| r |
| a | Connection is terminated | | | a | Connection is terminated | |
| y | Datamover-specific resources | L | Transport | y | Datamover-specific resources | L | Transport
| e | deallocated, both connection | a | Connection | e | deallocated, both connection | a | Connection
| r | level & task level | y | is terminated | r | level & task level | y | is terminated
| | | e | | | | e |
| | | r | | | | r |
| | | | | | | |
| | | | | | | |
Figure 7 A normal iSCSI connection termination
Figure 7. A Normal iSCSI Connection Termination
| | | | ^ | | | | ^
| | Full Feature Phase Control & | D | | iSCSI | | Full Feature Phase Control & | D | | iSCSI
| | Data Transfer Using DM | a | | Full Feature | | Data Transfer Using DM | a | | Full Feature
| i | | t | | Phase | i | | t | | Phase
| S | | a | | (DM Enabled) | S | | a | | (DM Enabled)
| C | | m | v | C | | m | v
| S | | o |<--Transport | S | | o |<--Transport
| I | Datamover-specific resources | v | Connection | I | Datamover-specific resources | v | Connection
| | deallocated, both connection | e | Terminated (e.g. | | deallocated, both connection | e | Terminated (e.g.
| L | level & task level | r | unexpected | L | level & task level | r | unexpected
| a | | | FIN/RESET) | a | | | FIN/RESET)
| y | | L | | y | | L |
| e | Connection_Terminate_Notify | a | | e | Connection_Terminate_Notify | a |
| r |<-------------------------------| y | | r |<-------------------------------| y |
| | | e | | | | e |
| | | r | | | | r |
| | | | | | | |
Figure 8 An abnormal iSCSI connection termination Figure 8. An Abnormal iSCSI Connection Termination
<-----Initiator-----> <-------Target-------> <-----Initiator-----> <-------Target------->
| | | | DM Msg holding | | | | | | | | DM Msg holding | | | |
SCSI | | | | SCSI Cmd PDU & | | | |SCSI SCSI | | | | SCSI Cmd PDU & | | | |SCSI
Cmd | | Snd_Ctrl | |Unsol Imm Data | |Ctrl_Notify | |Cmd Cmd | | Snd_Ctrl | |Unsol Imm Data | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |---> ---->| |--------->| |--------------->| |----------->| |--->
| | | | | | | | | | | | | | | |
| | | | DM Msg holding | | | | | | | | DM Msg holding | | | |
| | Snd_Ctrl | |SCSI Dataout PDU| |Ctrl_Notify | | | | Snd_Ctrl | |SCSI Dataout PDU| |Ctrl_Notify | |
| |--------->| |--------------->| |----------->| | | |--------->| |--------------->| |----------->| |
skipping to change at page 60, line 40 skipping to change at page 41, line 40
| | | || Transfer | | | | | | | || Transfer | | | |
| | | |--------------->| |Data_Cmp_Nfy| |Data | | | |--------------->| |Data_Cmp_Nfy| |Data
| | | | | |----------->| |Trans | | | | | |----------->| |Trans
| | | | | | | |Compl | | | | | | | |Compl
| | | | DM Msg holding | | | | | | | | DM Msg holding | | | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<---- <----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | | | | | | | | | |
Figure 9 A SCSI Write data transfer Figure 9. A SCSI Write Data Transfer
<-----Initiator-----> <-------Target-------> <-----Initiator-----> <-------Target------->
| | | | | | | | | | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |---> ---->| |--------->| |--------------->| |----------->| |--->
| | | | | | | | | | | | | | | |
| | | D| SCSI Read | D| | |Buf | | | D| SCSI Read | D| | |Buf
| | | a| Data Transfer | a| Put_Data | |Avail | | | a| Data Transfer | a| Put_Data | |Avail
| i| | t|<---------------| t|<-----------| i|<---- | i| | t|<---------------| t|<-----------| i|<----
skipping to change at page 61, line 31 skipping to change at page 42, line 31
| y| | L| | L| | y| | y| | L| | L| | y|
| e| | a| | a|Data_Cmp_Nfy| e|Data | e| | a| | a|Data_Cmp_Nfy| e|Data
| r| | y| | y|----------->| r|Trans | r| | y| | y|----------->| r|Trans
| | | e| | e| | |Compl | | | e| | e| | |Compl
| | | r| DM Msg holding | r| | | | | | r| DM Msg holding | r| | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<---- <----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | | | | | | | | | |
Figure 10 A SCSI Read data transfer Figure 10. A SCSI Read Data Transfer
<-----Initiator-----> <-------Target-------> <-----Initiator-----> <-------Target------->
| | | | | | | | | | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |----> ---->| |--------->| |--------------->| |----------->| |---->
| | | | | | | | | | | | | | | |
| | | D| SCSI Read | D| Put_Data | |Buf | | | D| SCSI Read | D| Put_Data | |Buf
| | | a| Data Transfer | a|Data_in.A=1 | |Avail | | | a| Data Transfer | a|Data_in.A=1 | |Avail
| i| | t|<---------------| t|<-----------| i|<---- | i| | t|<---------------| t|<-----------| i|<----
skipping to change at page 62, line 31 skipping to change at page 43, line 31
| y| | L| | L| | y| | y| | L| | L| | y|
| e| | a| | a| | e|Data | e| | a| | a| | e|Data
| r| | y| | y| | r|Trans | r| | y| | y| | r|Trans
| | | e| | e| | |Compl | | | e| | e| | |Compl
| | | r| DM Msg holding | r| | | | | | r| DM Msg holding | r| | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<---- <----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | | | | | | | | | |
Figure 11 A SCSI Read data acknowledgement Figure 11. A SCSI Read Data Acknowledgement
<-----Initiator-----> <-------Target-------> <-----Initiator-----> <-------Target------->
| | | | | | | | | | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |----> ---->| |--------->| |--------------->| |----------->| |---->
| | | | | | | | | | | | | | | |
| | | D| SCSI Read | D| | |Buf | | | D| SCSI Read | D| | |Buf
| | | a| Data Transfer | a| Put_Data | |Avail | | | a| Data Transfer | a| Put_Data | |Avail
| i| | t|<---------------| t|<-----------| i|<---- | i| | t|<---------------| t|<-----------| i|<----
skipping to change at page 63, line 31 skipping to change at page 44, line 31
<----| y|<---------| L|<---------------| L|<-----------| y|<---- <----| y|<---------| L|<---------------| L|<-----------| y|<----
| e| | a| | a| | e| | e| | a| | a| | e|
| r| | y| | y| | r| | r| | y| | y| | r|
| | | e| | e| | | | | | e| | e| | |
| | | r| | r| | | | | | r| | r| | |
| | | | | | | | | | | | | | | |
| |Dal_Tk_Res| | | |Dal_Tk_Res | | | |Dal_Tk_Res| | | |Dal_Tk_Res | |
| |--------->| | | |<-----------| | | |--------->| | | |<-----------| |
| | | | | | | | | | | | | | | |
Figure 12 Task resource cleanup on abort Figure 12. Task Resource Cleanup on Abort
17 Full Copyright Statement Acknowledgements
Copyright (C) The IETF Trust (2006). This document is The IP Storage (IPS) Working Group in the Transport Area of
subject to the rights, licenses and restrictions contained in IETF has been responsible for defining the iSCSI protocol
BCP 78, and except as set forth therein, the authors retain (apart from a host of other relevant IP Storage protocols).
all their rights. The authors are grateful to the entire working group, whose
work allowed this document to build on the concepts and
details of the iSCSI protocol.
This document and the information contained herein are In addition, the following individuals reviewed and
provided on an "AS IS" basis and THE CONTRIBUTOR, THE contributed to the improvement of this document. The authors
ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), are grateful for their contribution.
THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE
DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES
OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
18 Intellectual Property Statement John Carrier
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (360) 378-8526
EMail: john_carrier@adaptec.com
Hari Ghadia
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (408) 957-5608
EMail: hari_ghadia@adaptec.com
The IETF takes no position regarding the validity or scope of Hari Mudaliar
any Intellectual Property Rights or other rights that might Adaptec, Inc.
be claimed to pertain to the implementation or use of the 691 S. Milpitas Blvd., Milpitas, CA 95035, USA
technology described in this document or the extent to which Phone: +1 (408) 957-6012
any license under such rights might or might not be EMail: hari_mudaliar@adaptec.com
available; nor does it represent that it has made any
independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can
be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and Patricia Thaler
any assurances of licenses to be made available, or the Agilent Technologies, Inc.
result of an attempt made to obtain a general license or 1101 Creekside Ridge Drive, #100, M/S-RG10,
permission for the use of such proprietary rights by Roseville, CA 95678, USA
implementers or users of this specification can be obtained Phone: +1-916-788-5662
from the IETF on-line IPR repository at EMail: pat_thaler@agilent.com
Uri Elzur
Broadcom Corporation
16215 Alton Parkway, Irvine, CA 92619-7013, USA
Phone: +1 (949) 585-6432
EMail: Uri@Broadcom.com
Mike Penna
Broadcom Corporation
16215 Alton Parkway,Irvine, CA 92619-7013, USA
Phone: +1 (949) 926-7149
EMail: MPenna@Broadcom.com
David Black
EMC Corporation
176 South St., Hopkinton, MA 01748, USA
Phone: +1 (508) 293-7953
EMail: black_david@emc.com
Ted Compton
EMC Corporation
Research Triangle Park, NC 27709, USA
Phone: +1-919-248-6075
EMail: compton_ted@emc.com
Dwight Barron
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-2769
EMail: Dwight.Barron@Hp.com
Paul R. Culley
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-5543
EMail: paul.culley@hp.com
Dave Garcia
Hewlett-Packard Company
19333 Vallco Parkway, Cupertino, CA 95014, USA
Phone: +1 (408) 285-6116
EMail: dave.garcia@hp.com
Randy Haagens
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA, USA
Phone: +1-916-785-4578
EMail: randy_haagens@hp.com
Jeff Hilland
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-9489
EMail: jeff.hilland@hp.com
Mike Krause
Hewlett-Packard Company, 43LN
19410 Homestead Road, Cupertino, CA 95014, USA
Phone: +1 (408) 447-3191
EMail: krause@cup.hp.com
Jim Wendt
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA, USA
Phone: +1-916-785-5198
EMail: jim_wendt@hp.com
Mike Ko
IBM
650 Harry Rd, San Jose, CA 95120, USA
Phone: +1 (408) 927-2085
EMail: mako@us.ibm.com
Renato Recio
IBM Corporation
11501 Burnett Road, Austin, TX 78758, USA
Phone: +1 (512) 838-1365
EMail: recio@us.ibm.com
Howard C. Herbert
Intel Corporation
MS CH7-404,5000 West Chandler Blvd., Chandler, AZ 85226, USA
Phone: +1 (480) 554-3116
EMail: howard.c.herbert@intel.com
Dave Minturn
Intel Corporation
MS JF1-210, 5200 North East Elam Young Parkway
Hillsboro, OR 97124, USA
Phone: +1 (503) 712-4106
EMail: dave.b.minturn@intel.com
James Pinkerton
Microsoft Corporation
One Microsoft Way, Redmond, WA 98052, USA
Phone: +1 (425) 705-5442
EMail: jpink@microsoft.com
Tom Talpey
Network Appliance
375 Totten Pond Road, Waltham, MA 02451, USA
Phone: +1 (781) 768-5329
EMail: thomas.talpey@netapp.com
Authors' Addresses
Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA
Phone: +1-916-785-5621
EMail: cbm@rose.hp.com
John L. Hufferd
Brocade, Inc.
1745 Technology Drive
San Jose, CA 95110, USA
Phone: +1-408-333-5244
EMail: jhufferd@brocade.com
Julian Satran
IBM, Haifa Research Lab
Haifa University Campus - Mount Carmel
Haifa 31905, Israel
Phone +972-4-829-6264
EMail: Julian_Satran@il.ibm.com
Hemal Shah
Broadcom Corporation
5300 California Avenue
Irvine, California 92617, USA
Phone: +1-949-926-6941
EMail: hemal@broadcom.com
Comments may be sent to Mallikarjun Chadalapaka.
Full Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr. http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its The IETF invites any interested party to bring to its attention any
attention any copyrights, patents or patent applications, or copyrights, patents or patent applications, or other proprietary
other proprietary rights that may cover technology that may rights that may cover technology that may be required to implement
be required to implement this standard. Please address the this standard. Please address the information to the IETF at
information to the IETF at ietf-ipr@ietf.org. ietf-ipr@ietf.org.
 End of changes. 304 change blocks. 
1468 lines changed or deleted 1338 lines changed or added

This html diff was produced by rfcdiff 1.34. The latest version is available from http://tools.ietf.org/tools/rfcdiff/