draft-ietf-rddp-ddp-01.txt   draft-ietf-rddp-ddp-02.txt 
INTERNET-DRAFT Hemal Shah INTERNET-DRAFT Hemal Shah
draft-ietf-rddp-ddp-01.txt Intel Corporation draft-ietf-rddp-ddp-02.txt Intel Corporation
James Pinkerton Expires: August, 2004 James Pinkerton
Microsoft Corporation Microsoft Corporation
Renato Recio Renato Recio
IBM Corporation IBM Corporation
Paul Culley Paul Culley
Hewlett-Packard Company Hewlett-Packard Company
Expires: April, 2004 February, 2004
Direct Data Placement over Reliable Transports Direct Data Placement over Reliable Transports
1 Status of this Memo 1 Status of this Memo
This document is an Internet-Draft and is subject to all provisions This document is an Internet-Draft and is subject to all provisions
of Section 10 of RFC2026. of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
skipping to change at line 43 skipping to change at line 43
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
2 Abstract 2 Abstract
The Direct Data Placement protocol provides information to Place the The Direct Data Placement protocol provides information to Place the
incoming data directly into an upper layer protocol's receive buffer incoming data directly into an upper layer protocol's receive buffer
without intermediate buffers. This removes excess CPU and memory without intermediate buffers. This removes excess CPU and memory
utilization associated with transferring data through the utilization associated with transferring data through the
intermediate buffers. intermediate buffers.
shah, et. al. Expires April 2004 1 Shah, et. al. Expires August 2004 1
Table of Contents Table of Contents
1 Status of this Memo..........................................1 1 Status of this Memo.........................................1
2 Abstract.....................................................1 2 Abstract....................................................1
3 Introduction.................................................4 3 Introduction................................................4
3.1 Architectural Goals..........................................4 3.1 Architectural Goals.........................................4
3.2 Protocol Overview............................................5 3.2 Protocol Overview...........................................5
3.3 DDP Layering.................................................6 3.3 DDP Layering................................................6
4 Glossary.....................................................9 4 Glossary....................................................9
4.1 General......................................................9 4.1 General.....................................................9
4.2 LLP.........................................................10 4.2 LLP........................................................10
4.3 Direct Data Placement (DDP).................................10 4.3 Direct Data Placement (DDP)................................10
5 Reliable Delivery LLP Requirements..........................13 5 Reliable Delivery LLP Requirements.........................13
6 Header Format...............................................15 6 Header Format..............................................15
6.1 DDP Control Field...........................................15 6.1 DDP Control Field..........................................15
6.2 DDP Tagged Buffer Model Header..............................16 6.2 DDP Tagged Buffer Model Header.............................16
6.3 DDP Untagged Buffer Model Header............................17 6.3 DDP Untagged Buffer Model Header...........................17
6.4 DDP Segment Format..........................................18 6.4 DDP Segment Format.........................................18
7 Data Transfer...............................................19 7 Data Transfer..............................................19
7.1 DDP Tagged or Untagged Buffer Models........................19 7.1 DDP Tagged or Untagged Buffer Models.......................19
7.1.1 Tagged Buffer Model.......................................19 7.1.1 Tagged Buffer Model......................................19
7.1.2 Untagged Buffer Model.....................................19 7.1.2 Untagged Buffer Model....................................19
7.2 Segmentation and Reassembly of a DDP Message................19 7.2 Segmentation and Reassembly of a DDP Message...............19
7.3 Ordering Among DDP Messages.................................21 7.3 Ordering Among DDP Messages................................21
7.4 DDP Message Completion & Delivery...........................22 7.4 DDP Message Completion & Delivery..........................22
8 DDP Stream Setup & Teardown.................................23 8 DDP Stream Setup & Teardown................................23
8.1 DDP Stream Setup............................................23 8.1 DDP Stream Setup...........................................23
8.2 DDP Stream Teardown.........................................23 8.2 DDP Stream Teardown........................................23
8.2.1 DDP Graceful Teardown.....................................23 8.2.1 DDP Graceful Teardown....................................23
8.2.2 DDP Abortive Teardown.....................................24 8.2.2 DDP Abortive Teardown....................................24
9 Error Semantics.............................................25 9 Error Semantics............................................25
9.1 Errors detected at the Data Sink............................25 9.1 Errors detected at the Data Sink...........................25
9.2 DDP Error Numbers...........................................26 9.2 DDP Error Numbers..........................................26
10 Security Considerations.....................................27 10 Security Considerations....................................27
10.1 Protocol-specific Security Considerations.................27 10.1 Protocol-specific Security Considerations................27
10.2 Using IPSec with DDP......................................27 10.2 Using IPSec with DDP.....................................27
10.3 Association of an STag and a DDP Stream...................27 10.3 Association of an STag and a DDP Stream..................27
10.4 Other Security Considerations.............................28 10.4 Other Security Considerations............................28
11 IANA Considerations.........................................30 11 IANA Considerations........................................30
12 References..................................................31 12 References.................................................31
12.1 Normative References......................................31 12.1 Normative References.....................................31
12.2 Informative References....................................31 12.2 Informative References...................................31
13 Appendix....................................................32 13 Appendix...................................................32
13.1 Receive Window sizing.....................................32 13.1 Receive Window sizing....................................32
14 Author's Addresses..........................................33 14 Author's Addresses.........................................33
15 Acknowledgments.............................................34 15 Acknowledgments............................................34
16 Full Copyright Statement....................................36 16 Full Copyright Statement...................................37
Table of Figures Table of Figures
Figure 1 DDP Layering.............................................7 Figure 1 DDP Layering.............................................7
Figure 2 MPA, DDP, and RDMAP Header Alignment.....................8 Figure 2 MPA, DDP, and RDMAP Header Alignment.....................8
shah, et. al. Expires April 2004 2 Shah, et. al. Expires August 2004 2
Figure 3 DDP Control Field.......................................15 Figure 3 DDP Control Field.......................................15
Figure 4 Tagged Buffer DDP Header................................16 Figure 4 Tagged Buffer DDP Header................................16
Figure 5 Untagged Buffer DDP Header..............................17 Figure 5 Untagged Buffer DDP Header..............................17
Figure 6 DDP Segment Format......................................18 Figure 6 DDP Segment Format......................................18
shah, et. al. Expires April 2004 3 Shah, et. al. Expires August 2004 3
3 Introduction 3 Introduction
Direct Data Placement Protocol (DDP) enables an Upper Layer Protocol Direct Data Placement Protocol (DDP) enables an Upper Layer Protocol
(ULP) to send data to a Data Sink without requiring the Data Sink to (ULP) to send data to a Data Sink without requiring the Data Sink to
Place the data in an intermediate buffer - thus when the data Place the data in an intermediate buffer - thus when the data
arrives at the Data Sink, the network interface can Place the data arrives at the Data Sink, the network interface can Place the data
directly into the ULP's buffer. This can enable the Data Sink to directly into the ULP's buffer. This can enable the Data Sink to
consume substantially less memory bandwidth than a buffered model consume substantially less memory bandwidth than a buffered model
because the Data Sink is not required to move the data from the because the Data Sink is not required to move the data from the
skipping to change at line 156 skipping to change at line 156
without a need for a copy, even if incoming DDP Segments arrive without a need for a copy, even if incoming DDP Segments arrive
out of order. This requires the protocol to separate Data out of order. This requires the protocol to separate Data
Placement of ULP Payload contained in an incoming DDP Segment Placement of ULP Payload contained in an incoming DDP Segment
from Data Delivery of completed ULP Messages. from Data Delivery of completed ULP Messages.
* If the LLP supports multiple LLP streams within a LLP * If the LLP supports multiple LLP streams within a LLP
Connection, provide the above capabilities independently on Connection, provide the above capabilities independently on
each LLP stream and enable the capability to be exported on a each LLP stream and enable the capability to be exported on a
per LLP stream basis to the ULP. per LLP stream basis to the ULP.
shah, et. al. Expires April 2004 4 Shah, et. al. Expires August 2004 4
3.2 Protocol Overview 3.2 Protocol Overview
DDP supports two basic data transfer models - a Tagged Buffer data DDP supports two basic data transfer models - a Tagged Buffer data
transfer model and an Untagged Buffer data transfer model. transfer model and an Untagged Buffer data transfer model.
The Tagged Buffer data transfer model requires the Data Sink to send The Tagged Buffer data transfer model requires the Data Sink to send
the Data Source an identifier for the ULP buffer, referred to as a the Data Source an identifier for the ULP buffer, referred to as a
Steering Tag (STag). The STag is transferred to the Data Source Steering Tag (STag). The STag is transferred to the Data Source
using a ULP defined method. Once the Data Source ULP has an STag for using a ULP defined method. Once the Data Source ULP has an STag for
skipping to change at line 212 skipping to change at line 212
Buffer Model, a DDP Message can only start at offset 0. Buffer Model, a DDP Message can only start at offset 0.
* The Tagged Buffer Model allows multiple DDP Messages targeted * The Tagged Buffer Model allows multiple DDP Messages targeted
to a Tagged Buffer with a single ULP buffer Advertisement. The to a Tagged Buffer with a single ULP buffer Advertisement. The
Untagged Buffer Model requires associating a receive ULP buffer Untagged Buffer Model requires associating a receive ULP buffer
for each DDP Message targeted to an Untagged Buffer. for each DDP Message targeted to an Untagged Buffer.
Either data transfer model Places a ULP Message into a DDP Message. Either data transfer model Places a ULP Message into a DDP Message.
Each DDP Message is then sliced into DDP Segments that are intended Each DDP Message is then sliced into DDP Segments that are intended
shah, et. al. Expires April 2004 5 Shah, et. al. Expires August 2004 5
to fit within a lower-layer-protocol's (LLP) Maximum Upper Layer to fit within a lower-layer-protocol's (LLP) Maximum Upper Layer
Protocol Data Unit (MULPDU). Thus the ULP can post arbitrary size Protocol Data Unit (MULPDU). Thus the ULP can post arbitrary size
ULP Messages, containing up to 2^32 - 1 octets of ULP Payload, and ULP Messages, containing up to 2^32 - 1 octets of ULP Payload, and
DDP slices the ULP message into DDP Segments which are reassembled DDP slices the ULP message into DDP Segments which are reassembled
transparently at the Data Sink. transparently at the Data Sink.
DDP provides in-order Delivery for the ULP. However, DDP DDP provides in-order Delivery for the ULP. However, DDP
differentiates between Data Delivery and Data Placement. DDP differentiates between Data Delivery and Data Placement. DDP
provides enough information in each DDP Segment to allow the ULP provides enough information in each DDP Segment to allow the ULP
Payload in each inbound DDP Segment payloads to be directly Placed Payload in each inbound DDP Segment payloads to be directly Placed
skipping to change at line 263 skipping to change at line 263
DDP is intended to be LLP independent, subject to the requirements DDP is intended to be LLP independent, subject to the requirements
defined in section 5. However, DDP was specifically defined to be defined in section 5. However, DDP was specifically defined to be
part of a family of protocols that were created to work well part of a family of protocols that were created to work well
together, as shown in Figure 1 DDP Layering. For LLP protocol together, as shown in Figure 1 DDP Layering. For LLP protocol
definitions of each LLP, see [MPA], [TCP], and [SCTP]. definitions of each LLP, see [MPA], [TCP], and [SCTP].
DDP enables direct data Placement capability for any ULP, but it has DDP enables direct data Placement capability for any ULP, but it has
been specifically designed to work well with RDMAP (see [RDMA]), and been specifically designed to work well with RDMAP (see [RDMA]), and
is part of the iWARP protocol suite. is part of the iWARP protocol suite.
shah, et. al. Expires April 2004 6 Shah, et. al. Expires August 2004 6
+-------------------+ +-------------------+
| | | |
| RDMA ULP | | RDMA ULP |
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | |
| ULP | RDMAP | | ULP | RDMAP |
| | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | |
skipping to change at line 294 skipping to change at line 294
| | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1 DDP Layering Figure 1 DDP Layering
If DDP is layered below RDMAP and on top of MPA and TCP, then the If DDP is layered below RDMAP and on top of MPA and TCP, then the
respective headers and payload are arranged as follows (Note: For respective headers and payload are arranged as follows (Note: For
clarity, MPA header and CRC are included but framing markers are not clarity, MPA header and CRC are included but framing markers are not
shown.): shown.):
shah, et. al. Expires April 2004 7 Shah, et. al. Expires August 2004 7
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | |
// TCP Header // // TCP Header //
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | | | MPA Header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| | | |
skipping to change at line 321 skipping to change at line 321
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | |
// RDMAP ULP Payload // // RDMAP ULP Payload //
| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC | | MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2 MPA, DDP, and RDMAP Header Alignment Figure 2 MPA, DDP, and RDMAP Header Alignment
shah, et. al. Expires April 2004 8 Shah, et. al. Expires August 2004 8
4 Glossary 4 Glossary
4.1 General 4.1 General
Advertisement (Advertised, Advertise, Advertisements, Advertises) - Advertisement (Advertised, Advertise, Advertisements, Advertises) -
the act of informing a Remote Peer that a local RDMA Buffer is The act of informing a Remote Peer that a local RDMA Buffer is
available to it. A Node makes available an RDMA Buffer for available to it. A Node makes available an RDMA Buffer for
incoming RDMA Read or RDMA Write access by informing its incoming RDMA Read or RDMA Write access by informing its
RDMA/DDP peer of the Tagged Buffer identifiers (STag, base RDMA/DDP peer of the Tagged Buffer identifiers (STag, base
address, length). This advertisement of Tagged Buffer address, length). This advertisement of Tagged Buffer
information is not defined by RDMA/DDP and is left to the ULP. A information is not defined by RDMA/DDP and is left to the ULP. A
typical method would be for the Local Peer to embed the Tagged typical method would be for the Local Peer to embed the Tagged
Buffer's Steering Tag, address, and length in a Send message Buffer's Steering Tag, address, and length in a Send message
destined for the Remote Peer. destined for the Remote Peer.
Data Delivery (Delivery, Delivered, Delivers) - Delivery is defined Data Delivery (Delivery, Delivered, Delivers) - Delivery is defined
as the process of informing the ULP or consumer that a as the process of informing the ULP or consumer that a
particular Message is available for use. This is specifically particular message is available for use. This is specifically
different from "Placement", which may generally occur in any different from "Placement", which may generally occur in any
order, while the order of "Delivery" is strictly defined. See order, while the order of "Delivery" is strictly defined. See
"Data Placement". "Data Placement".
Data Sink - The peer receiving a data payload. Note that the Data Data Sink - The peer receiving a data payload. Note that the Data
Sink can be required to both send and receive RDMA/DDP Messages Sink can be required to both send and receive RDMA/DDP Messages
to transfer a data payload. to transfer a data payload.
Data Source - The peer sending a data payload. Note that the Data Data Source - The peer sending a data payload. Note that the Data
Source can be required to both send and receive RDMA/DDP Source can be required to both send and receive RDMA/DDP
skipping to change at line 362 skipping to change at line 362
iWARP - A suite of wire protocols comprised of RDMAP [RDMAP], DDP iWARP - A suite of wire protocols comprised of RDMAP [RDMAP], DDP
[DDP], and MPA [MPA]. The iWARP protocol suite may be layered [DDP], and MPA [MPA]. The iWARP protocol suite may be layered
above TCP, SCTP, or other transport protocols. above TCP, SCTP, or other transport protocols.
Local Peer - The RDMA/DDP protocol implementation on the local end Local Peer - The RDMA/DDP protocol implementation on the local end
of the connection. Used to refer to the local entity when of the connection. Used to refer to the local entity when
describing a protocol exchange or other interaction between two describing a protocol exchange or other interaction between two
Nodes. Nodes.
Node - A computing device attached to one or more links of network. Node - A computing device attached to one or more links of a
A Node in this context does not refer to a specific application network. A Node in this context does not refer to a specific
or protocol instantiation running on the computer. A Node may application or protocol instantiation running on the computer. A
consist of one or more RNICs installed in a host computer. Node may consist of one or more RNICs installed in a host
computer.
Remote Peer - The RDMA/DDP protocol implementation on the opposite Remote Peer - The RDMA/DDP protocol implementation on the opposite
end of the connection. Used to refer to the remote entity when end of the connection. Used to refer to the remote entity when
describing protocol exchanges or other interactions between two describing protocol exchanges or other interactions between two
Nodes. Nodes.
ULP - Upper Layer Protocol. The protocol layer above the protocol ULP - Upper Layer Protocol. The protocol layer above the protocol
layer currently being referenced. The ULP for RDMA/DDP is layer currently being referenced. The ULP for RDMA/DDP is
expected to be an OS, Application, adaptation layer, or expected to be an OS, application, adaptation layer, or
proprietary device. The RDMA/DDP documents do not specify a ULP
shah, et. al. Expires April 2004 9 Shah, et. al. Expires August 2004 9
proprietary device. The RDMA/DDP documents do not specify a ULP
- they provide a set of semantics that allow a ULP to be - they provide a set of semantics that allow a ULP to be
designed to utilize RDMA/DDP. designed to utilize RDMA/DDP.
ULP Message - the ULP data that is handed to a specific protocol ULP Message - The ULP data that is handed to a specific protocol
layer for transmission. Data boundaries are preserved as they layer for transmission. Data boundaries are preserved as they
are transmitted through iWARP. are transmitted through iWARP.
ULP Payload - The ULP data that is contained within a single ULP Payload - The ULP data that is contained within a single
protocol segment or packet (e.g. a DDP Segment). protocol segment or packet (e.g. a DDP Segment).
4.2 LLP 4.2 LLP
LLP - Lower Layer Protocol. The protocol layer beneath the protocol LLP - Lower Layer Protocol. The protocol layer beneath the protocol
layer currently being referenced. For example, for DDP the LLP layer currently being referenced. For example, for DDP the LLP
skipping to change at line 405 skipping to change at line 406
LLP Connection - Corresponds to an LLP transport-level connection LLP Connection - Corresponds to an LLP transport-level connection
between the peer LLP layers on two nodes. between the peer LLP layers on two nodes.
LLP Stream - Corresponds to a single LLP transport-level stream LLP Stream - Corresponds to a single LLP transport-level stream
between the peer LLP layers on two Nodes. One or more LLP between the peer LLP layers on two Nodes. One or more LLP
Streams may map to a single transport-level LLP Connection. For Streams may map to a single transport-level LLP Connection. For
transport protocols that support multiple streams per connection transport protocols that support multiple streams per connection
(e.g. SCTP), a LLP Stream corresponds to one transport-level (e.g. SCTP), a LLP Stream corresponds to one transport-level
stream. stream.
MULPDU - Maximum ULPDU. The current maximum size of the record that MULPDU - Maximum Upper Layer Protocol Data Unit. The current maximum
is acceptable for DDP to pass to the LLP for transmission. size of the record that is acceptable for DDP to pass to the LLP
for transmission.
ULPDU - Upper Layer Protocol Data Unit. The data record defined by
the layer above MPA.
4.3 Direct Data Placement (DDP) 4.3 Direct Data Placement (DDP)
DDP Graceful Teardown - The act of closing a DDP Stream such that DDP Graceful Teardown - The act of closing a DDP Stream such that
all in-progress and pending DDP Messages are allowed to complete all in-progress and pending DDP Messages are allowed to complete
successfully. successfully.
DDP Abortive Teardown - The act of closing a DDP Stream without DDP Abortive Teardown - The act of closing a DDP Stream without
attempting to complete in-progress and pending DDP Messages. attempting to complete in-progress and pending DDP Messages.
Data Placement (Placement, Placed, Places) - For DDP, this term is Data Placement (Placement, Placed, Places) - For DDP, this term is
specifically used to indicate the process of writing to a data specifically used to indicate the process of writing to a data
buffer by a DDP implementation. DDP Segments carry Placement buffer by a DDP implementation. DDP Segments carry Placement
information, which may be used by the receiving DDP information, which may be used by the receiving DDP
implementation to perform Data Placement of the DDP Segment ULP implementation to perform Data Placement of the DDP Segment ULP
Payload. See "Data Delivery". Payload. See "Data Delivery" and “Direct Data Placement”.
DDP Control Field - a fixed 8-bit field in the DDP Header. DDP Control Field - A fixed 8-bit field in the DDP Header.
shah, et. al. Expires April 2004 10
DDP Header - The header present in all DDP Segments. The DDP Header DDP Header - The header present in all DDP Segments. The DDP Header
contains control and Placement fields that are used to define contains control and Placement fields that are used to define
Shah, et. al. Expires August 2004 10
the final Placement location for the ULP Payload carried in a the final Placement location for the ULP Payload carried in a
DDP Segment. DDP Segment.
DDP Message - A ULP defined unit of data interchange, which is DDP Message - A ULP defined unit of data interchange, which is
subdivided into one or more DDP Segments. This segmentation may subdivided into one or more DDP Segments. This segmentation may
occur for a variety of reasons, including segmentation to occur for a variety of reasons, including segmentation to
respect the maximum segment size of the underlying transport respect the maximum segment size of the underlying transport
protocol. protocol.
DDP Segment - The smallest unit of data transfer for the DDP DDP Segment - The smallest unit of data transfer for the DDP
skipping to change at line 483 skipping to change at line 483
and an STag. Under this mechanism, the use of an STag is valid and an STag. Under this mechanism, the use of an STag is valid
on a DDP Stream if the STag has the same Protection Domain on a DDP Stream if the STag has the same Protection Domain
Identifier (PD ID) as the DDP Stream. Identifier (PD ID) as the DDP Stream.
Protection Domain Identifier (PD ID) – An identifier for the Protection Domain Identifier (PD ID) – An identifier for the
Protection Domain. Protection Domain.
Queue Number (QN) - For the DDP Untagged Buffer Model, identifies a Queue Number (QN) - For the DDP Untagged Buffer Model, identifies a
destination Data Sink queue for a DDP Segment. destination Data Sink queue for a DDP Segment.
shah, et. al. Expires April 2004 11
Steering Tag - An identifier of a Tagged Buffer on a Node, valid as Steering Tag - An identifier of a Tagged Buffer on a Node, valid as
defined within a protocol specification. defined within a protocol specification.
Shah, et. al. Expires August 2004 11
STag - Steering Tag STag - Steering Tag
Tagged Buffer - A buffer that is explicitly Advertised to the Remote Tagged Buffer - A buffer that is explicitly Advertised to the Remote
Peer through exchange of an STag, Target Offset, and length. Peer through exchange of an STag, Tagged Offset, and length.
Tagged Buffer Model - A DDP data transfer model used to transfer Tagged Buffer Model - A DDP data transfer model used to transfer
Tagged Buffers from the Local Peer to the Remote Peer. Tagged Buffers from the Local Peer to the Remote Peer.
Tagged DDP Message - A DDP Message that targets a Tagged Buffer. Tagged DDP Message - A DDP Message that targets a Tagged Buffer.
Target Offset (TO) - The offset within a Tagged Buffer on a Node. Tagged Offset (TO) - The offset within a Tagged Buffer on a Node.
ULP Buffer - A buffer owned above the DDP Layer and advertised to ULP Buffer - A buffer owned above the DDP Layer and advertised to
the DDP Layer either as a Tagged Buffer or an Untagged ULP the DDP Layer either as a Tagged Buffer or an Untagged ULP
Buffer. Buffer.
ULP Message Length - is the total length of the ULP Payload contained ULP Message Length - The total length, in octets, of the ULP Payload
in a DDP Message. contained in a DDP Message.
Untagged Buffer - A buffer that is not explicitly Advertised to the Untagged Buffer - A buffer that is not explicitly Advertised to the
Remote Peer. Remote Peer.
Untagged Buffer Model - A DDP data transfer model used to transfer Untagged Buffer Model - A DDP data transfer model used to transfer
Untagged Buffers from the Local Peer to the Remote Peer. Untagged Buffers from the Local Peer to the Remote Peer.
Untagged DDP Message - A DDP Message that targets an Untagged Untagged DDP Message - A DDP Message that targets an Untagged
Buffer. Buffer.
shah, et. al. Expires April 2004 12 Shah, et. al. Expires August 2004 12
5 Reliable Delivery LLP Requirements 5 Reliable Delivery LLP Requirements
1. LLPs MUST expose MULPDU & MULPDU Changes. This is required so 1. LLPs MUST expose MULPDU & MULPDU Changes. This is required so
that the DDP layer can perform segmentation aligned with the that the DDP layer can perform segmentation aligned with the
MULPDU and can adapt as MULPDU changes come about. The corner MULPDU and can adapt as MULPDU changes come about. The corner
case of how to handle outstanding requests during a MULPDU case of how to handle outstanding requests during a MULPDU
change is covered by the requirements below. change is covered by the requirements below.
2. In the event of a MULPDU change, DDP MUST NOT be required by the 2. In the event of a MULPDU change, DDP MUST NOT be required by the
skipping to change at line 571 skipping to change at line 571
DDP Stream to be torn down. DDP Stream to be torn down.
9. For a specific LLP Stream, the LLP MUST provide a mechanism to 9. For a specific LLP Stream, the LLP MUST provide a mechanism to
indicate that the LLP Stream has been gracefully torn down. For indicate that the LLP Stream has been gracefully torn down. For
a specific LLP Connection, the LLP MUST provide a mechanism to a specific LLP Connection, the LLP MUST provide a mechanism to
indicate that the LLP Connection has been gracefully torn down. indicate that the LLP Connection has been gracefully torn down.
Note that if the LLP does not allow an LLP Stream to be torn Note that if the LLP does not allow an LLP Stream to be torn
down independently of the LLP Connection, the above requirements down independently of the LLP Connection, the above requirements
allow the LLP to notify DDP of both events at the same time. allow the LLP to notify DDP of both events at the same time.
shah, et. al. Expires April 2004 13 Shah, et. al. Expires August 2004 13
10. For a specific LLP Connection, when all LLP Streams are either 10. For a specific LLP Connection, when all LLP Streams are either
gracefully torn down or are labeled as erroneous LLP streams, gracefully torn down or are labeled as erroneous LLP streams,
the LLP Connection MUST be torn down. the LLP Connection MUST be torn down.
11. The LLP MUST NOT pass a duplicate DDP Segment to the DDP Layer 11. The LLP MUST NOT pass a duplicate DDP Segment to the DDP Layer
after it has passed all the previous DDP Segments to the DDP after it has passed all the previous DDP Segments to the DDP
Layer and the associated ordering information for the previous Layer and the associated ordering information for the previous
DDP Segments and the current DDP Segment. DDP Segments and the current DDP Segment.
shah, et. al. Expires April 2004 14 Shah, et. al. Expires August 2004 14
6 Header Format 6 Header Format
DDP has two different header formats: one for Data Placement into DDP has two different header formats: one for Data Placement into
Tagged Buffers, and the other for Data Placement into Untagged Tagged Buffers, and the other for Data Placement into Untagged
Buffers. See Section 7.1 for a description of the two models. Buffers. See Section 7.1 for a description of the two models.
6.1 DDP Control Field 6.1 DDP Control Field
The first 8 bits of the DDP Header carry a DDP Control Field that is The first 8 bits of the DDP Header carry a DDP Control Field that is
skipping to change at line 635 skipping to change at line 635
Delivered to the ULP after: Delivered to the ULP after:
. Placement of all DDP Segments of this DDP Message and all . Placement of all DDP Segments of this DDP Message and all
prior DDP Messages, and prior DDP Messages, and
. Delivery of each prior DDP Message. . Delivery of each prior DDP Message.
If the Last flag is set to zero, the DDP Segment is an If the Last flag is set to zero, the DDP Segment is an
intermediate DDP Segment. intermediate DDP Segment.
shah, et. al. Expires April 2004 15 Shah, et. al. Expires August 2004 15
Rsvd - Reserved: 4 bits. Rsvd - Reserved: 4 bits.
Reserved for future use by the DDP protocol. This field MUST be Reserved for future use by the DDP protocol. This field MUST be
set to zero on transmit, and not checked on receive. set to zero on transmit, and not checked on receive.
DV - Direct Data Placement Protocol Version: 2 bits. DV - Direct Data Placement Protocol Version: 2 bits.
The version of the DDP Protocol in use. This field MUST be set The version of the DDP Protocol in use. This field MUST be set
to one to indicate the version of the specification described to one to indicate the version of the specification described
in this document. The value of DV MUST be the same for all the in this document. The value of DV MUST be the same for all the
skipping to change at line 681 skipping to change at line 681
RsvdULP - Reserved for use by the ULP: 8 bits. RsvdULP - Reserved for use by the ULP: 8 bits.
The RsvdULP field is opaque to the DDP protocol and can be The RsvdULP field is opaque to the DDP protocol and can be
structured in any way by the ULP. At the Data Source, DDP MUST structured in any way by the ULP. At the Data Source, DDP MUST
set RsvdULP Field to the value specified by the ULP. It is set RsvdULP Field to the value specified by the ULP. It is
transferred unmodified from the Data Source to the Data Sink. transferred unmodified from the Data Source to the Data Sink.
At the Data Sink, DDP MUST provide the RsvdULP field to the ULP At the Data Sink, DDP MUST provide the RsvdULP field to the ULP
when the DDP Message is delivered. Each DDP Segment within a when the DDP Message is delivered. Each DDP Segment within a
specific DDP Message MUST contain the same value for this specific DDP Message MUST contain the same value for this
field. field. The Data Source MUST ensure that each DDP Segment within
a specific DDP Message contains the same value for this field.
STag - Steering Tag: 32 bits. STag - Steering Tag: 32 bits.
The Steering Tag identifies the Data Sink's Tagged Buffer. The The Steering Tag identifies the Data Sink's Tagged Buffer. The
STag MUST be valid for this DDP Stream. The STag is associated STag MUST be valid for this DDP Stream. The STag is associated
with the DDP Stream through a mechanism that is outside the with the DDP Stream through a mechanism that is outside the
scope of the DDP Protocol specification. At the Data Source,
shah, et. al. Expires April 2004 16 Shah, et. al. Expires August 2004 16
scope of the DDP Protocol specification. At the Data Source,
DDP MUST set the STag field to the value specified by the ULP. DDP MUST set the STag field to the value specified by the ULP.
At the Data Sink, the DDP MUST provide the STag field when the At the Data Sink, the DDP MUST provide the STag field when the
ULP Message is delivered. Each DDP Segment within a specific ULP Message is delivered. Each DDP Segment within a specific
DDP Message MUST contain the same value for this field and MUST DDP Message MUST contain the same value for this field and MUST
be the value supplied by the ULP. be the value supplied by the ULP. The Data Source MUST ensure
that each DDP Segment within a specific DDP Message contains
the same value for this field.
TO - Tagged Offset: 64 bits. TO - Tagged Offset: 64 bits.
The Tagged Offset specifies the offset, in octets, within the The Tagged Offset specifies the offset, in octets, within the
Data Sink's Tagged Buffer, where the Placement of ULP Payload Data Sink's Tagged Buffer, where the Placement of ULP Payload
contained in the DDP Segment starts. A DDP Message MAY start at contained in the DDP Segment starts. A DDP Message MAY start at
an arbitrary TO within a Tagged Buffer. an arbitrary TO within a Tagged Buffer.
6.3 DDP Untagged Buffer Model Header 6.3 DDP Untagged Buffer Model Header
skipping to change at line 739 skipping to change at line 742
RsvdULP - Reserved for use by the ULP: 40 bits. RsvdULP - Reserved for use by the ULP: 40 bits.
The RsvdULP field is opaque to the DDP protocol and can be The RsvdULP field is opaque to the DDP protocol and can be
structured in any way by the ULP. At the Data Source, DDP MUST structured in any way by the ULP. At the Data Source, DDP MUST
set RsvdULP Field to the value specified by the ULP. It is set RsvdULP Field to the value specified by the ULP. It is
transferred unmodified from the Data Source to the Data Sink. transferred unmodified from the Data Source to the Data Sink.
At the Data Sink, DDP MUST provide RsvdULP field to the ULP At the Data Sink, DDP MUST provide RsvdULP field to the ULP
when the ULP Message is Delivered. Each DDP Segment within a when the ULP Message is Delivered. Each DDP Segment within a
specific DDP Message MUST contain the same value for the specific DDP Message MUST contain the same value for the
Shah, et. al. Expires August 2004 17
RsvdULP field. At the Data Sink, the DDP implementation is NOT RsvdULP field. At the Data Sink, the DDP implementation is NOT
REQUIRED to verify that the same value is present in the REQUIRED to verify that the same value is present in the
RsvdULP field of each DDP Segment within a specific DDP Message RsvdULP field of each DDP Segment within a specific DDP Message
shah, et. al. Expires April 2004 17
and MAY provide the value from any one of the received DDP and MAY provide the value from any one of the received DDP
Segment to the ULP when the ULP Message is Delivered. Segment to the ULP when the ULP Message is Delivered.
QN - Queue Number: 32 bits. QN - Queue Number: 32 bits.
The Queue Number identifies the Data Sink's Untagged Buffer The Queue Number identifies the Data Sink's Untagged Buffer
queue referenced by this header. Each DDP segment within a queue referenced by this header. Each DDP segment within a
specific DDP message MUST contain the same value for this field specific DDP message MUST contain the same value for this field
and MUST be the value supplied by the ULP at the Data Source. and MUST be the value supplied by the ULP at the Data Source.
The Data Source MUST ensure that each DDP Segment within a
specific DDP Message contains the same value for this field.
MSN - Message Sequence Number: 32 bits. MSN - Message Sequence Number: 32 bits.
The Message Sequence Number specifies a sequence number that The Message Sequence Number specifies a sequence number that
MUST be increased by one (modulo 2^32) with each DDP Message MUST be increased by one (modulo 2^32) with each DDP Message
targeting the specific Queue Number on the DDP Stream targeting the specific Queue Number on the DDP Stream
associated with this DDP Segment. The initial value for MSN associated with this DDP Segment. The initial value for MSN
MUST be one. The MSN value MUST wrap to 0 after a value of MUST be one. The MSN value MUST wrap to 0 after a value of
0xFFFFFFFF. 0xFFFFFFFF. Each DDP segment within a specific DDP message MUST
contain the same value for this field. The Data Source MUST
ensure that each DDP Segment within a specific DDP Message
contains the same value for this field.
MO - Message Offset: 32 bits. MO - Message Offset: 32 bits.
The Message Offset specifies the offset, in octets, from the The Message Offset specifies the offset, in octets, from the
start of the DDP Message represented by the MSN and Queue start of the DDP Message represented by the MSN and Queue
Number on the DDP Stream associated with this DDP Segment. The Number on the DDP Stream associated with this DDP Segment. The
MO referencing the first octet of the DDP Message MUST be set MO referencing the first octet of the DDP Message MUST be set
to zero by the DDP layer. to zero by the DDP layer.
6.4 DDP Segment Format 6.4 DDP Segment Format
skipping to change at line 783 skipping to change at line 791
Each DDP Segment MUST contain a DDP Header. Each DDP Segment may Each DDP Segment MUST contain a DDP Header. Each DDP Segment may
also contain ULP Payload. Following is the DDP Segment format: also contain ULP Payload. Following is the DDP Segment format:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP | | | DDP | |
| Header| ULP Payload (if any) | | Header| ULP Payload (if any) |
| | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6 DDP Segment Format Figure 6 DDP Segment Format
shah, et. al. Expires April 2004 18 Shah, et. al. Expires August 2004 18
7 Data Transfer 7 Data Transfer
DDP supports multi-segment DDP Messages. Each DDP Message is DDP supports multi-segment DDP Messages. Each DDP Message is
composed of one or more DDP Segments. Each DDP Segment contains a composed of one or more DDP Segments. Each DDP Segment contains a
DDP Header. The DDP Header contains the information required by the DDP Header. The DDP Header contains the information required by the
receiver to Place any ULP Payload included in the DDP Segment. receiver to Place any ULP Payload included in the DDP Segment.
7.1 DDP Tagged or Untagged Buffer Models 7.1 DDP Tagged or Untagged Buffer Models
skipping to change at line 839 skipping to change at line 847
communicate how many buffers have been queued is outside the scope communicate how many buffers have been queued is outside the scope
of this specification. Similarly, the exact implementation of the of this specification. Similarly, the exact implementation of the
buffer queue is outside the scope of this specification. buffer queue is outside the scope of this specification.
7.2 Segmentation and Reassembly of a DDP Message 7.2 Segmentation and Reassembly of a DDP Message
At the Data Source, the DDP layer MUST segment the data contained in At the Data Source, the DDP layer MUST segment the data contained in
a ULP message into a series of DDP Segments, where each DDP Segment a ULP message into a series of DDP Segments, where each DDP Segment
contains a DDP Header and ULP Payload, and MUST be no larger than contains a DDP Header and ULP Payload, and MUST be no larger than
shah, et. al. Expires April 2004 19 Shah, et. al. Expires August 2004 19
the MULPDU value advertised by the LLP. The ULP Message Length MUST the MULPDU value advertised by the LLP. The ULP Message Length MUST
be less than 2^32. At the Data Source, the DDP layer MUST send all be less than 2^32. At the Data Source, the DDP layer MUST send all
the data contained in the ULP message. At the Data Sink, the DDP the data contained in the ULP message. At the Data Sink, the DDP
layer MUST Place the ULP Payload contained in all valid incoming DDP layer MUST Place the ULP Payload contained in all valid incoming DDP
Segments associated with a DDP Message into the ULP Buffer. Segments associated with a DDP Message into the ULP Buffer.
DDP Message segmentation at the Data Source is accomplished by DDP Message segmentation at the Data Source is accomplished by
identifying a DDP Message (which corresponds one-to-one with a ULP identifying a DDP Message (which corresponds one-to-one with a ULP
Message) uniquely and then, for each associated DDP Segment of a DDP Message) uniquely and then, for each associated DDP Segment of a DDP
Message, by specifying an octet offset for the portion of the ULP Message, by specifying an octet offset for the portion of the ULP
skipping to change at line 894 skipping to change at line 902
of the STag effectively enables the ULP to implement of the STag effectively enables the ULP to implement
segmentation and reassembly due to ULP specific constraints. segmentation and reassembly due to ULP specific constraints.
See [RDMAP] for details of how this is done. See [RDMAP] for details of how this is done.
A different implementation of a ULP could use an Untagged DDP A different implementation of a ULP could use an Untagged DDP
Message sent after the Tagged DDP Message which details the Message sent after the Tagged DDP Message which details the
initial TO for the STag that was used in the Tagged DDP initial TO for the STag that was used in the Tagged DDP
Message. And finally, another implementation of a ULP could Message. And finally, another implementation of a ULP could
choose to always use an initial TO of zero such that no choose to always use an initial TO of zero such that no
shah, et. al. Expires April 2004 20 Shah, et. al. Expires August 2004 20
additional message is required to convey the initial TO used in additional message is required to convey the initial TO used in
a Tagged DDP Message. a Tagged DDP Message.
Regardless of whether the ULP chooses to recover the original ULP Regardless of whether the ULP chooses to recover the original ULP
Message boundary at the Data Sink for a Tagged DDP Message, DDP Message boundary at the Data Sink for a Tagged DDP Message, DDP
supports segmentation and reassembly of the Tagged DDP Message. The supports segmentation and reassembly of the Tagged DDP Message. The
STag is used to identify the ULP Buffer at the Data Sink and the TO STag is used to identify the ULP Buffer at the Data Sink and the TO
is used to identify the octet-offset within the ULP Buffer is used to identify the octet-offset within the ULP Buffer
referenced by the STag. The ULP at the Data Source MUST specify the referenced by the STag. The ULP at the Data Source MUST specify the
STag and the initial TO when the ULP Message is handed to DDP. STag and the initial TO when the ULP Message is handed to DDP.
skipping to change at line 921 skipping to change at line 929
For example, if the ULP Tagged Message was 2048 octets with an For example, if the ULP Tagged Message was 2048 octets with an
initial TO of 16384, and the MULPDU was 1500 octets, the Data Source initial TO of 16384, and the MULPDU was 1500 octets, the Data Source
would generate two DDP Segments, one with TO = 16384, containing the would generate two DDP Segments, one with TO = 16384, containing the
first 1486 octets of ULP payload, and a second with TO = 17870, first 1486 octets of ULP payload, and a second with TO = 17870,
containing 562 octets of ULP payload. In this example, the amount of containing 562 octets of ULP payload. In this example, the amount of
ULP payload for the first DDP Segment was calculated as: ULP payload for the first DDP Segment was calculated as:
1486 = 1500 (MULPDU) - 14 (for the DDP Header) 1486 = 1500 (MULPDU) - 14 (for the DDP Header)
A zero-length Tagged DDP Message is allowed and MUST consume exactly A zero-length DDP Message is allowed and MUST consume exactly one
one DDP Segment. Only the DDP Control and RsvdULP Fields MUST be DDP Segment. Only the DDP Control and RsvdULP Fields MUST be valid
valid for a zero length Tagged DDP Segment. The STag and TO fields for a zero length Tagged DDP Segment. The STag and TO fields MUST
MUST NOT be checked for a zero-length Tagged DDP Message. NOT be checked for a zero-length Tagged DDP Message.
For either Untagged or Tagged DDP Messages, the Data Sink is not For either Untagged or Tagged DDP Messages, the Data Sink is not
required to verify that the entire ULP Message has been received. required to verify that the entire ULP Message has been received.
7.3 Ordering Among DDP Messages 7.3 Ordering Among DDP Messages
Messages passed through the DDP MUST conform to the ordering rules Messages passed through the DDP MUST conform to the ordering rules
defined in this section. defined in this section.
At the Data Source, DDP: At the Data Source, DDP:
skipping to change at line 948 skipping to change at line 956
* SHOULD transmit DDP Segments within a DDP Message in increasing * SHOULD transmit DDP Segments within a DDP Message in increasing
MO order for Untagged DDP Messages and in increasing TO order MO order for Untagged DDP Messages and in increasing TO order
for Tagged DDP Messages. for Tagged DDP Messages.
At the Data Sink, DDP (Note: The following rules are motivated by At the Data Sink, DDP (Note: The following rules are motivated by
LLP implementations that separate Placement and Delivery.): LLP implementations that separate Placement and Delivery.):
* MAY perform Placement of DDP Segments out of order, * MAY perform Placement of DDP Segments out of order,
shah, et. al. Expires April 2004 21 Shah, et. al. Expires August 2004 21
* MAY perform Placement of a DDP Segment more than once, * MAY perform Placement of a DDP Segment more than once,
* MUST Deliver a DDP Message to the ULP at most once, * MUST Deliver a DDP Message to the ULP at most once,
* MUST Deliver DDP Messages to the ULP in the order they were * MUST Deliver DDP Messages to the ULP in the order they were
sent by the Data Source. sent by the Data Source.
7.4 DDP Message Completion & Delivery 7.4 DDP Message Completion & Delivery
At the Data Source, DDP Message transfer is considered completed At the Data Source, DDP Message transfer is considered completed
skipping to change at line 985 skipping to change at line 993
At the Data Sink, DDP MUST provide the ULP Message Length to the ULP At the Data Sink, DDP MUST provide the ULP Message Length to the ULP
when an Untagged DDP Message is Delivered. The ULP Message Length when an Untagged DDP Message is Delivered. The ULP Message Length
may be calculated by adding the MO and the ULP Payload length in the may be calculated by adding the MO and the ULP Payload length in the
last DDP Segment (with the Last flag set) of an Untagged DDP last DDP Segment (with the Last flag set) of an Untagged DDP
Message. Message.
At the Data Sink, DDP MUST provide the RsvdULP Field of the DDP At the Data Sink, DDP MUST provide the RsvdULP Field of the DDP
Message to the ULP when the DDP Message is delivered. Message to the ULP when the DDP Message is delivered.
shah, et. al. Expires April 2004 22 Shah, et. al. Expires August 2004 22
8 DDP Stream Setup & Teardown 8 DDP Stream Setup & Teardown
This section describes LLP independent issues related to DDP Stream This section describes LLP independent issues related to DDP Stream
setup and teardown. setup and teardown.
8.1 DDP Stream Setup 8.1 DDP Stream Setup
It is expected that the ULP will use a mechanism outside the scope It is expected that the ULP will use a mechanism outside the scope
of this specification to establish an LLP Connection, and that the of this specification to establish an LLP Connection, and that the
skipping to change at line 1039 skipping to change at line 1047
torn down. torn down.
If the Local Peer LLP supports a half-closed LLP Stream, on the If the Local Peer LLP supports a half-closed LLP Stream, on the
receipt of a LLP graceful teardown request of the DDP Stream, DDP receipt of a LLP graceful teardown request of the DDP Stream, DDP
SHOULD indicate the half-closed state to the ULP, and continue to SHOULD indicate the half-closed state to the ULP, and continue to
process outbound data transfer requests normally. Following this process outbound data transfer requests normally. Following this
event, when the Local Peer ULP requests graceful teardown, DDP MUST event, when the Local Peer ULP requests graceful teardown, DDP MUST
indicate to the LLP that it SHOULD perform a graceful close of the indicate to the LLP that it SHOULD perform a graceful close of the
other half of the LLP Stream. other half of the LLP Stream.
shah, et. al. Expires April 2004 23 Shah, et. al. Expires August 2004 23
If the Local Peer LLP supports a half-closed LLP Stream, on the If the Local Peer LLP supports a half-closed LLP Stream, on the
receipt of a ULP graceful half-close teardown request of the DDP receipt of a ULP graceful half-close teardown request of the DDP
Stream, DDP SHOULD keep data reception enabled on the other half of Stream, DDP SHOULD keep data reception enabled on the other half of
the LLP stream. the LLP stream.
8.2.2 DDP Abortive Teardown 8.2.2 DDP Abortive Teardown
As previously mentioned, DDP does not independently terminate a DDP As previously mentioned, DDP does not independently terminate a DDP
Stream. Thus any of the following fatal errors on a DDP Stream MUST Stream. Thus any of the following fatal errors on a DDP Stream MUST
cause DDP to indicate to the ULP that a fatal error has occurred: cause DDP to indicate to the ULP that a fatal error has occurred:
skipping to change at line 1069 skipping to change at line 1077
Error Numbers) and complete all outstanding ULP requests with an Error Numbers) and complete all outstanding ULP requests with an
error. If the underlying LLP Stream is still intact, DDP SHOULD error. If the underlying LLP Stream is still intact, DDP SHOULD
continue to allow the ULP to transfer additional DDP Messages on the continue to allow the ULP to transfer additional DDP Messages on the
outgoing half connection after the fatal error was indicated to the outgoing half connection after the fatal error was indicated to the
ULP. This enables the ULP to transfer an error syndrome to the ULP. This enables the ULP to transfer an error syndrome to the
Remote Peer. After indicating to the ULP a fatal error has occurred, Remote Peer. After indicating to the ULP a fatal error has occurred,
the DDP Stream MUST NOT be terminated until the Local Peer ULP the DDP Stream MUST NOT be terminated until the Local Peer ULP
indicates to the DDP layer that the DDP Stream should be abortively indicates to the DDP layer that the DDP Stream should be abortively
torndown. torndown.
shah, et. al. Expires April 2004 24 Shah, et. al. Expires August 2004 24
9 Error Semantics 9 Error Semantics
All LLP errors reported to DDP SHOULD be passed up to the ULP. All LLP errors reported to DDP SHOULD be passed up to the ULP.
9.1 Errors detected at the Data Sink 9.1 Errors detected at the Data Sink
For non-zero length Untagged DDP Segments, the DDP Segment MUST be For non-zero length Untagged DDP Segments, the DDP Segment MUST be
validated before Placement by verifying: validated before Placement by verifying:
1. The QN is valid for this stream. 1. The QN is valid for this stream.
2. The QN and MSN have an associated buffer that allows Placement 2. The QN and MSN have an associated buffer that allows Placement
of the payload. of the payload.
Implementers note: DDP implementations SHOULD consider lack of Implementers note: DDP implementations SHOULD consider lack of
an associated buffer as a system fault. DDP implementations MAY an associated buffer as a system fault. DDP implementations MAY
try to recover from the system fault using LLP means in a ULP- try to recover from the system fault using LLP means in a ULP-
transparent way. DDP implementations SHOULD NOT permit system transparent way. DDP implementations SHOULD NOT permit system
faults to occur repeatedly or frequently. faults to occur repeatedly or frequently. If there is not an
associated buffer, DDP implementations MAY choose to disable
the stream for the reception and report an error to the ULP at
the Data Sink.
3. The MO falls in the range of legal offsets associated with the 3. The MO falls in the range of legal offsets associated with the
Untagged Buffer. Untagged Buffer.
4. The sum of the DDP Segment payload length and the MO falls in 4. The sum of the DDP Segment payload length and the MO falls in
the range of legal offsets associated with the Untagged Buffer. the range of legal offsets associated with the Untagged Buffer.
5. For DDP Messages using Untagged Buffer model, the Message 5. The Message Sequence Number falls in the range of legal Message
Sequence Number falls in the range of legal Message Sequence Sequence Numbers, for the queue defined by the QN. The legal
Numbers, for the queue defined by the QN. The legal range is range is defined as being between the MSN value assigned to the
defined as being between the MSN value assigned to the first first available buffer for a specific QN and the MSN value
available buffer for a specific QN and the MSN value assigned to assigned to the last available buffer for a specific QN.
the last available buffer for a specific QN.
Implementers note: for a typical Queue Number, the lower limit Implementers note: for a typical Queue Number, the lower limit
of the Message Sequence Number is defined by whatever DDP of the Message Sequence Number is defined by whatever DDP
Messages have already been Completed. The upper limit is Messages have already been Completed. The upper limit is
defined by however many message buffers are currently available defined by however many message buffers are currently available
for that queue. Both numbers change dynamically as new DDP for that queue. Both numbers change dynamically as new DDP
Messages are received and Completed, and new buffers are added. Messages are received and Completed, and new buffers are added.
It is up to the ULP to ensure that sufficient buffers are It is up to the ULP to ensure that sufficient buffers are
available to handle the incoming DDP Segments. available to handle the incoming DDP Segments.
For non-zero length Tagged DDP Segments, the segment MUST be For non-zero length Tagged DDP Segments, the segment MUST be
validated before Placement by verifying: validated before Placement by verifying:
1. The STag is valid for this stream. 1. The STag is valid for this stream.
2. The STag has an associated buffer that allows Placement of the 2. The STag has an associated buffer that allows Placement of the
payload. payload.
Shah, et. al. Expires August 2004 25
3. The TO falls in the range of legal offsets registered for the 3. The TO falls in the range of legal offsets registered for the
STag. STag.
shah, et. al. Expires April 2004 25
4. The sum of the DDP Segment payload length and the TO falls in 4. The sum of the DDP Segment payload length and the TO falls in
the range of legal offsets registered for the STag. the range of legal offsets registered for the STag.
5. A 64-bit unsigned sum of the DDP Segment payload length and the 5. A 64-bit unsigned sum of the DDP Segment payload length and the
TO does not wrap. TO does not wrap.
If the DDP layer detects any of the receive errors listed in this If the DDP layer detects any of the receive errors listed in this
section, it MUST cease placing the remainder of the DDP Segment and section, it MUST cease placing the remainder of the DDP Segment and
report the error(s) to the ULP. The DDP layer SHOULD include in the report the error(s) to the ULP. The DDP layer SHOULD include in the
error report the DDP Header, the type of error, and the length of error report the DDP Header, the type of error, and the length of
the DDP segment, if available. DDP MUST silently drop any subsequent the DDP segment, if available. DDP MUST silently drop any subsequent
incoming DDP Segments. Since each of these errors represents a incoming DDP Segments. Since each of these errors represents a
failure of the sending ULP or protocol, DDP SHOULD enable the ULP to failure of the sending ULP or protocol, DDP SHOULD enable the ULP to
send one additional DDP Message before terminating the DDP Stream. send one additional DDP Message before terminating the DDP Stream.
9.2 DDP Error Numbers 9.2 DDP Error Numbers
The following error numbers MUST be used when reporting receive The following error numbers MUST be used when reporting errors to
errors to the ULP. They correspond to the checks enumerated in the ULP. They correspond to the checks enumerated in section 9.1.
section 9.1. Each error is subdivided into a 4-bit Error Type and an Each error is subdivided into a 4-bit Error Type and an 8 bit Error
8 bit Error Code. Code.
Error Error Error Error
Type Code Description Type Code Description
---------------------------------------------------------- ----------------------------------------------------------
0x0 0x00 Local Catastrophic 0x0 0x00 Local Catastrophic
0x1 Tagged Buffer Error 0x1 Tagged Buffer Error
0x00 Invalid STag 0x00 Invalid STag
0x01 Base or bounds violation 0x01 Base or bounds violation
0x02 STag not associated with DDP Stream 0x02 STag not associated with DDP Stream
skipping to change at line 1169 skipping to change at line 1179
0x2 Untagged Buffer Error 0x2 Untagged Buffer Error
0x01 Invalid QN 0x01 Invalid QN
0x02 Invalid MSN - no buffer available 0x02 Invalid MSN - no buffer available
0x03 Invalid MSN - MSN range is not valid 0x03 Invalid MSN - MSN range is not valid
0x04 Invalid MO 0x04 Invalid MO
0x05 DDP Message too long for available buffer 0x05 DDP Message too long for available buffer
0x06 Invalid DDP version 0x06 Invalid DDP version
0x3 Rsvd Reserved for the use by the LLP 0x3 Rsvd Reserved for the use by the LLP
shah, et. al. Expires April 2004 26 Shah, et. al. Expires August 2004 26
10 Security Considerations 10 Security Considerations
This section discusses both protocol-specific considerations and the This section discusses both protocol-specific considerations and the
implications of using DDP with existing security mechanisms. A more implications of using DDP with existing security mechanisms. A more
detailed analysis of the security issues around the implementation detailed analysis of the security issues around the implementation
and the use of the DDP can be found in [RDMASEC]. and the use of the DDP can be found in [RDMASEC].
10.1 Protocol-specific Security Considerations 10.1 Protocol-specific Security Considerations
skipping to change at line 1208 skipping to change at line 1218
packet streams, including streams where packets are lost, DDP can packet streams, including streams where packets are lost, DDP can
run on top of IPsec without any change. IPsec packets are processed run on top of IPsec without any change. IPsec packets are processed
(e.g., integrity checked and possibly decrypted) in the order they (e.g., integrity checked and possibly decrypted) in the order they
are received, and a DDP Data Sink will process the decrypted DDP are received, and a DDP Data Sink will process the decrypted DDP
Segments contained in these packets in the same manner as DDP Segments contained in these packets in the same manner as DDP
Segments contained in unsecured IP packets. Segments contained in unsecured IP packets.
10.3 Association of an STag and a DDP Stream 10.3 Association of an STag and a DDP Stream
There are several mechanisms for associating an STag and a DDP There are several mechanisms for associating an STag and a DDP
Stream. Two reasonable mechanisms for this association are a Stream. Two required mechanisms for this association are a
Protection Domain (PD) association and a DDP Stream association. Protection Domain (PD) association and a DDP Stream association.
Under the Protection Domain (PD) association, a unique Protection Under the Protection Domain (PD) association, a unique Protection
Domain Identifier (PD ID) is created and used locally to associate Domain Identifier (PD ID) is created and used locally to associate
an STag with a set of DDP Streams. Under this mechanism, the use of an STag with a set of DDP Streams. Under this mechanism, the use of
the STag is only permitted on the DDP Streams that have the same PD the STag is only permitted on the DDP Streams that have the same PD
ID as the STag. For an incoming DDP Segment of a Tagged DDP Message ID as the STag. For an incoming DDP Segment of a Tagged DDP Message
on a DDP Stream, if the PD ID of the DDP Stream is not the same as on a DDP Stream, if the PD ID of the DDP Stream is not the same as
the PD ID of the STag targeted by the Tagged DDP Message, then the the PD ID of the STag targeted by the Tagged DDP Message, then the
DDP Segment is not placed and the DDP layer MUST surface a local DDP Segment is not placed and the DDP layer MUST surface a local
error to the ULP. Note that the PD ID is locally defined, and cannot error to the ULP. Note that the PD ID is locally defined, and cannot
be directly manipulated by the Remote Peer. be directly manipulated by the Remote Peer.
Under the DDP Stream association, a DDP Stream is identified locally Under the DDP Stream association, a DDP Stream is identified locally
by a unique DDP Stream identifier (ID). An STag is associated with a by a unique DDP Stream identifier (ID). An STag is associated with a
shah, et. al. Expires April 2004 27 Shah, et. al. Expires August 2004 27
DDP Stream by using a DDP Stream ID. In this case, for an incoming DDP Stream by using a DDP Stream ID. In this case, for an incoming
DDP Segment of a Tagged DDP Message on a DDP Stream, if the DDP DDP Segment of a Tagged DDP Message on a DDP Stream, if the DDP
Stream ID of the DDP Stream is not the same as the DDP Stream ID of Stream ID of the DDP Stream is not the same as the DDP Stream ID of
the STag targeted by the Tagged DDP Message, then the DDP Segment is the STag targeted by the Tagged DDP Message, then the DDP Segment is
not placed and the DDP layer MUST surface a local error to the ULP. not placed and the DDP layer MUST surface a local error to the ULP.
Note that the DDP Stream ID is locally defined, and cannot be Note that the DDP Stream ID is locally defined, and cannot be
directly manipulated by the Remote Peer. directly manipulated by the Remote Peer.
A ULP SHOULD associate an STag and a DDP Stream. DDP MUST support A ULP SHOULD associate an STag and a DDP Stream. DDP MUST support
Protection Domain association and DDP Stream association mechanisms Protection Domain association and DDP Stream association mechanisms
skipping to change at line 1276 skipping to change at line 1286
1. STags are only valid over the exact byte range established by the 1. STags are only valid over the exact byte range established by the
ULP. DDP MUST provide a mechanism for the ULP to establish and ULP. DDP MUST provide a mechanism for the ULP to establish and
revoke the TO range associated with the ULP Buffer referenced by revoke the TO range associated with the ULP Buffer referenced by
the STag. the STag.
2. STags are only valid for the duration established by the ULP. The 2. STags are only valid for the duration established by the ULP. The
ULP may revoke them at any time, in accordance with its own upper ULP may revoke them at any time, in accordance with its own upper
layer protocol requirements. DDP MUST provide a mechanism for the layer protocol requirements. DDP MUST provide a mechanism for the
ULP to establish and revoke STag validity. ULP to establish and revoke STag validity.
shah, et. al. Expires April 2004 28 Shah, et. al. Expires August 2004 28
3. DDP MUST provide a mechanism for the ULP to communicate the 3. DDP MUST provide a mechanism for the ULP to communicate the
association between a STag and a specific DDP Stream. association between a STag and a specific DDP Stream.
4. A ULP may only expose memory to remote access to the extent that 4. A ULP may only expose memory to remote access to the extent that
it already had access to that memory itself. it already had access to that memory itself.
5. If an STag is not valid on a DDP Stream, DDP MUST pass the invalid 5. If an STag is not valid on a DDP Stream, DDP MUST pass the invalid
access attempt to the ULP. The ULP may provide a mechanism for access attempt to the ULP. The ULP may provide a mechanism for
terminating the DDP Stream. terminating the DDP Stream.
Further, DDP provides a mechanism that directly Places incoming Further, DDP provides a mechanism that directly Places incoming
payloads in user-mode ULP Buffers. This avoids the risks of prior payloads in user-mode ULP Buffers. This avoids the risks of prior
solutions that relied upon exposing system buffers for incoming solutions that relied upon exposing system buffers for incoming
payloads. payloads.
shah, et. al. Expires April 2004 29 Shah, et. al. Expires August 2004 29
11 IANA Considerations 11 IANA Considerations
If DDP was enabled a priori for a ULP by connecting to a well-known If DDP was enabled a priori for a ULP by connecting to a well-known
port, this well-known port would be registered for the DDP with port, this well-known port would be registered for the DDP with
IANA. IANA.
shah, et. al. Expires April 2004 30 Shah, et. al. Expires August 2004 30
12 References 12 References
12.1 Normative References 12.1 Normative References
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision [RFC2026] Bradner, S., "The Internet Standards Process -- Revision
3", BCP 9, RFC 2026, October 1996. 3", BCP 9, RFC 2026, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[MPA] P. Culley et al., "Markers with PDU Alignment", RDMA [MPA] Culley, P., Elzur, U., Recio, R., Bailey, S., Carrier, J.,
Consortium Draft Specification draft-cully-iwarp-mpa-01.doc, "Marker PDU Aligned Framing for TCP Specification", Internet
October 2002 Draft draft-ietf-rddp-mpa-00.txt (work in progress), September
2003
[RDMAP] R. Recio et al., "RDMA Protocol Specification", RDMA [RDMAP] Recio, R., Culley, P., Garcia, D., Hilland, J., "An RDMA
Consortium Draft Specification draft-recio-iwarp-01, October Protocol Specification", Internet Draft draft-ietf-rddp-rdmap-
2002 01.txt (work in progress), October 2003
[SCTP] R. Stewart et al., "Stream Control Transmission Protocol", [SCTP] Stewart, R. et al., "Stream Control Transmission Protocol",
RFC 2960, October 2000. RFC 2960, October 2000.
[TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793,
September 1981. September 1981.
12.2 Informative References 12.2 Informative References
[TLS] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0", RFC [TLS] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0", RFC
2246, November 1998. 2246, November 1998.
[IPSEC] Atkinson, R., Kent, S., "Security Architecture for the [IPSEC] Atkinson, R., Kent, S., "Security Architecture for the
Internet Protocol", RFC 2401, November 1998. Internet Protocol", RFC 2401, November 1998.
[RDMASEC] J. Pinkerton et al., "DDP/RDMAP Security", draft- [RDMASEC] Pinkerton J., Deleganes E., Romanow A., Bitan S.,
pinkerton-rddp-security-00.txt, June 2003. "DDP/RDMAP Security", draft-ietf-rddp-security-01.txt (work in
progress), February 2004.
shah, et. al. Expires April 2004 31 Shah, et. al. Expires August 2004 31
13 Appendix 13 Appendix
13.1 Receive Window sizing 13.1 Receive Window sizing
Reliable, sequenced, LLPs include a mechanism to advertise the Reliable, sequenced, LLPs include a mechanism to advertise the
amount of receive buffer space a sender may consume. This is amount of receive buffer space a sender may consume. This is
generally called a "receive window". generally called a "receive window".
DDP allows data to be transferred directly to predefined buffers at DDP allows data to be transferred directly to predefined buffers at
skipping to change at line 1368 skipping to change at line 1380
the rate that DDP Segments can be retired; there may be some cases the rate that DDP Segments can be retired; there may be some cases
where segment processing cannot keep up with the incoming packet where segment processing cannot keep up with the incoming packet
rate. If this occurs, one reasonable way to slow the incoming packet rate. If this occurs, one reasonable way to slow the incoming packet
rate is to reduce the receive window. rate is to reduce the receive window.
Note that the LLP should take care to comply with the applicable Note that the LLP should take care to comply with the applicable
RFCs; for instance, for TCP, receivers are highly discouraged from RFCs; for instance, for TCP, receivers are highly discouraged from
"shrinking" the receive window (reducing the right edge of the "shrinking" the receive window (reducing the right edge of the
window after it has been advertised). window after it has been advertised).
shah, et. al. Expires April 2004 32 Shah, et. al. Expires August 2004 32
14 Author's Addresses 14 Author's Addresses
Hemal Shah Hemal Shah
Intel Corporation Intel Corporation
MS AN1-PTL1 MS AN1-PTL1
1501 South Mopac Expressway, #400 1501 South Mopac Expressway, #400
Austin, TX 78746 USA Austin, TX 78746 USA
Phone: +1 (512) 732-3963 Phone: +1 (512) 732-3963
Email: hemal.shah@intel.com Email: hemal.shah@intel.com
skipping to change at line 1401 skipping to change at line 1413
Phone: +1 (512) 838-1365 Phone: +1 (512) 838-1365
Email: recio@us.ibm.com Email: recio@us.ibm.com
Paul R. Culley Paul R. Culley
Hewlett-Packard Company Hewlett-Packard Company
20555 SH 249 20555 SH 249
Houston, TX 77070-2698 USA Houston, TX 77070-2698 USA
Phone: +1 (281) 514-5543 Phone: +1 (281) 514-5543
Email: paul.culley@hp.com Email: paul.culley@hp.com
shah, et. al. Expires April 2004 33 Shah, et. al. Expires August 2004 33
15 Acknowledgments 15 Acknowledgments
John Carrier John Carrier
Adaptec, Inc. Adaptec, Inc.
691 S. Milpitas Blvd. 691 S. Milpitas Blvd.
Milpitas, CA 95035 USA Milpitas, CA 95035 USA
Phone: +1 (360) 378-8526 Phone: +1 (360) 378-8526
Email: john_carrier@adaptec.com Email: john_carrier@adaptec.com
skipping to change at line 1457 skipping to change at line 1469
Jim Wendt Jim Wendt
Hewlett-Packard Company Hewlett-Packard Company
8000 Foothills Boulevard 8000 Foothills Boulevard
Roseville, CA 95747-5668 USA Roseville, CA 95747-5668 USA
Phone: +1 (916) 785-5198 Phone: +1 (916) 785-5198
Email: jim_wendt@hp.com Email: jim_wendt@hp.com
Mike Krause Mike Krause
Hewlett-Packard Company, 43LN Hewlett-Packard Company, 43LN
shah, et. al. Expires April 2004 34 Shah, et. al. Expires August 2004 34
19410 Homestead Road 19410 Homestead Road
Cupertino, CA 95014 USA Cupertino, CA 95014 USA
Phone: +1 (408) 447-3191 Phone: +1 (408) 447-3191
Email: krause@cup.hp.com Email: krause@cup.hp.com
Dave Minturn Dave Minturn
Intel Corporation Intel Corporation
MS JF1-210 MS JF1-210
5200 North East Elam Young Parkway 5200 North East Elam Young Parkway
Hillsboro, OR 97124 USA Hillsboro, OR 97124 USA
skipping to change at line 1507 skipping to change at line 1519
Phone: +1 (408) 285-6116 Phone: +1 (408) 285-6116
Email: dave.garcia@hp.com Email: dave.garcia@hp.com
Jeff Hilland Jeff Hilland
Hewlett-Packard Company Hewlett-Packard Company
20555 SH 249 20555 SH 249
Houston, Tx. 77070-2698 USA Houston, Tx. 77070-2698 USA
Phone: +1 (281) 514-9489 Phone: +1 (281) 514-9489
Email: jeff.hilland@hp.com Email: jeff.hilland@hp.com
shah, et. al. Expires April 2004 35 Shah, et. al. Expires August 2004 35
Barry Reinhold
Lamprey Networks
Durham, NH 03824 USA
Phone: +1 (603) 868-8411
Email: bbr@LampreyNetworks.com
Shah, et. al. Expires August 2004 36
16 Full Copyright Statement 16 Full Copyright Statement
This document and the information contained herein is provided on an This document and the information contained herein is provided on an
"AS IS" basis and ADAPTEC INC., AGILENT TECHNOLOGIES INC., BROADCOM "AS IS" basis and ADAPTEC INC., AGILENT TECHNOLOGIES INC., BROADCOM
CORPORATION, CISCO SYSTEMS INC., EMC CORPORATION, HEWLETT-PACKARD CORPORATION, CISCO SYSTEMS INC., EMC CORPORATION, HEWLETT-PACKARD
COMPANY, INTERNATIONAL BUSINESS MACHINES CORPORATION, INTEL COMPANY, INTERNATIONAL BUSINESS MACHINES CORPORATION, INTEL
CORPORATION, MICROSOFT CORPORATION, NETWORK APPLIANCE INC., THE CORPORATION, MICROSOFT CORPORATION, NETWORK APPLIANCE INC., THE
INTERNET SOCIETY, AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM INTERNET SOCIETY, AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM
ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE
ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. FOR A PARTICULAR PURPOSE.
Copyright (c) 2002 ADAPTEC INC., BROADCOM CORPORATION, CISCO SYSTEMS Copyright (c) 2002 ADAPTEC INC., BROADCOM CORPORATION, CISCO SYSTEMS
INC., EMC CORPORATION, HEWLETT-PACKARD COMPANY, INTERNATIONAL INC., EMC CORPORATION, HEWLETT-PACKARD COMPANY, INTERNATIONAL
BUSINESS MACHINES CORPORATION, INTEL CORPORATION, MICROSOFT BUSINESS MACHINES CORPORATION, INTEL CORPORATION, MICROSOFT
CORPORATION, NETWORK APPLIANCE INC., All Rights Reserved. CORPORATION, NETWORK APPLIANCE INC., All Rights Reserved.
shah, et. al. Expires April 2004 36 Shah, et. al. Expires August 2004 37
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/