Remote Direct Data Placement R. Stewart Working Group Cisco Systems, Inc. Internet-Draft C. Bestler Expires:
February 15,December 2, 2006 Broadcom Corporation J. Pinkerton Microsoft S. Ganguly Consultant H. Shah IntelBroadcom Corporation V. Kashyap IBM S. Ganguly Consultant August 14, 2005May 31, 2006 Stream Control Transmission Protocol (SCTP) Remote Direct Memory Access (RDMA) Direct Data Placement (DDP) Adaptation draft-ietf-rddp-sctp-02.txtdraft-ietf-rddp-sctp-03.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on February 15,December 2, 2006. Copyright Notice Copyright (C) The Internet Society (2005).(2006). Abstract This document describes a methodspecifies an Adaptation Layer to adaptprovide a Lower Layer Protocol (LLP) service for Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) tousing the Stream Control Transmission Protocol (SCTP) RFC2960  using a generic description found in [RDMA-Draft]  and [DDP-Draft] .(SCTP). Table of Contents 1. IntroductionDefinitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Definitions2. Introduction . . . . . . . . . . . . . . . . . . . . . . . 3 1.2. . 4 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.4. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . 4 4.2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 4 5. Data Formats . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.15.1. Adaptation Layer Indicator . . . . . . . . . . . . . . . . 5 2.25.2. Payload Data Chunks . . . . . . . . . . . . . . . . . . . 5 2.2.16 5.2.1. DDP Source Sequence Number (DDP-SSN) . . . . . . . . . 6 126.96.36.199.2. DDP Segment Payload DataChunk . . . . . . . . . . . . . . . . . . 6 188.8.131.52.3. DDP Stream Session Control Data Chunk. . . . . . . . . . . . . . 7 3.6. DDP Stream Sessions . . . . . . . . . . . . . . . . . . . . . 8 3.16.1. Sequencing . . . . . . . . . . . . . . . . . . . . . . . . 8 3.26.2. Legal Sequence: Active/Passive Session Accepted . . . . . 8 3.36.3. Legal Sequence: Active/Passive Session Rejected . . . . . 9 3.48 6.4. Legal Sequence: Active/Passive Session Non-ULP Rejected . 9 3.56.5. ULP Specific Sequencing . . . . . . . . . . . . . . . . . 9 3.66.6. Other Sequencing Rules . . . . . . . . . . . . . . . . . . 9 4.7. SCTP Endpoints . . . . . . . . . . . . . . . . . . . . . . . . 11 4.110 7.1. Adaptation Layer Indication Restriction . . . . . . . . . 11 4.210 7.2. Multihoming Implications . . . . . . . . . . . . . . . . . 11 5.10 8. Number of Streams . . . . . . . . . . . . . . . . . . . . . . 12 6.11 9. Fragmentation . . . . . . . . . . . . . . . . . . . . . . . . 13 7.11 10. Sequenced Unordered Operation . . . . . . . . . . . . . . . . 14 8.12 11. Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . 15 8.112 11.1. Association Initialization . . . . . . . . . . . . . . . . 15 8.212 11.2. Chunk Bundling . . . . . . . . . . . . . . . . . . . . . . 15 8.313 11.3. Association Termination . . . . . . . . . . . . . . . . . 16 9.13 12. IANA considerations . . . . . . . . . . . . . . . . . . . . . 17 10.14 13. Security Considerations . . . . . . . . . . . . . . . . . . 18 11.. 14 14. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 19 12. Normative. 14 15. References . . . . . . . . . . . . . . . . . . . . 19. . . . . . 14 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 19. . 16 Intellectual Property and Copyright Statements . . . . . . . . 21. . 18 1. Introduction This document describes a method to adapt Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) toDefinitions DDP Endpoint - The logical sender/receiver of DDP Segments. An SCTP Stream Control Transmission Protocol (SCTP) RFC2960  using a generic description found in [RDMA-Draft]  and [DDP-Draft]  This adaption provides a method for two peers to know that each side is performing DDP or RDMA thus enabling hardware acceleration if available. Some implementations may include this adaptation layer within their SCTP implementations to obtain maximum performance but the behavior of SCTP will be unaffected. In order to accomplish this we specify the use of the new adaptation layer indication as defined in [ADDIP- Draft]  1.1 Definitions DDP Endpoint - The logical sender/receiver of DDP Segments. An SCTP Stream pair is not assumed to havepair is not assumed to have a DDP Endpoint. DDP Segments may only be sent once a DDP Endpoint has been assigned to an SCTP Stream pair by a local interface. DDP Source Stream Sequence (DDP-SSN) - A stream specific sequence number assigned by the DDP layerAdaptation Layer for each SCTP Data Chunk sent. Use of the SCTP Stream Sequence Number (SSN) could result in ordered delivery at the receiving end. Use of unordered Data Chunks indicates that the receiving SCTP layerThis is to deliver them without delay. The DDP-SSN retainsthe originalorder the Data Chunksthat chunks were generated in,submitted to SCTP, no matter what order they wereare actually sent or received.received in. The smallest unit of data transfer for the DDP protocol. It includes a DDP Header and ULP Payload (if present). A DDP Segment should be sized to fit within the Lower Layer Protocol MULPDU. An SCTP Payload Data Chunk that encapsulates the DDP Stream Sequence (DDP-SSN) and a DDP Segment. DDP Stream - Aa sequence of DDP Segments whose ordering is defined by the LLP. For SCTP, a DDP Stream maps directly to a bi-directional pair of SCTP streams which havewith the same SCTP stream identifier.Stream IDs. Note that DDP has no ordering guarantees between DDP Streams. DDP Stream Session - A single pairing of DDP Endpoints over a DDP Stream that lasts from a Initiation message through the Termination message(s). DDP Stream Session Control Message - DDP Stream Session Control messages are used to control the association of the DDP Endpoint with the DDP Stream. A common local interface convention to control which Steering Tags (STags) are valid with which DDP Endpoints. Under this convention both the Steering Tag and DDP Endpoint are created within the context of a Protection Domain, and the Steering Tag may only be enabled for DDP Endpoints created under the same Protection Domain. RDMA - Remote Direct Memory Access. RNIC - RDMA Network Interface Card. SCTP association - A protocol relationship between two SCTP endpoints. An SCTP association supports multiple SCTP streams. SCTP Data Chunk - An SCTP Chunk used to convey Payload Data. There can be multiple Chunks within each SCTP packet. Other Chunks are used to control the SCTP Association. SCTP endpoint - The logical sender/receiver of SCTP packets. On a multi-homed host, an SCTP endpoint is represented to its peers as a combination of a SCTP port number and a set of eligible destination transport addresses to which SCTP packets can be sent. SCTP Stream - A uni-directional logical channel established from one to another associated SCTP endpoint. There can be multiple SCTP Streams within each SCTP association. An SCTP Stream is used to form one direction of a DDP stream. Transmission Sequence Number (TSN) - A 32-bit sequence number used internally by SCTP. One TSN is attached to each chunk containing user data to permit the receiving SCTP endpoint to acknowledge its receipt and detect duplicate deliveries. 1.22. Introduction This document describes a method to adapt Direct Data Placement [DDP] [I-D.ietf-rddp-ddp] to Stream Control Transmission Protocol (SCTP) [RFC2960]. Some implementations may include this adaptation layer within their SCTP implementations to obtain maximum performance but the behavior of SCTP will be unaffected. An SCTP Layer used solely by this adaptation layer is able to take certain optimizations based on the limited subset of SCTP capabilities used. In order to allow optimization for these implementations we specify the use of the new adaptation layer indication as defined in [ADDIP-Draft] [I-D.ietf- tsvwg-addip-sctp] 3. Conventions The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL, when they appear in this document, are to be interpreted as described in RFC2119 . 2.[RFC2119]. 4. Introduction 4.1. Motivation This document specifies an Adaptation Layer which fulfills the requirements of a Lower Layer Protocol (LLP) for DDP using a specific subset of SCTP capabilities. The defined protocol is intended to be implementable over existing SCTP stacks, while clearly defining what portions of SCTP are required to enable an implementation to be optimized specifically to support DDP. 4.2. Overview The Adaptation Layer uses a pair of like-numbered SCTP Streams within an SCTP Association to provide a reliable DDP Stream between two DDP Endpoints. Except as specifically noted, each DDP Segment submitted by the DDP Layer is encoded as a single unordered SCTP Data Chunk. In addition to the DDP Segment the Data Chunk also contains a sequence number (DDP-SSN) that reflects the order that DDP submitted the segments in for that stream. A DDP Stream Session is defined by DDP Stream Session Control Chunks that manage the state of the DDP Stream Session. These Chunks dynamically bind DDP Endpoints to the DDP Stream Session, and DDP Segment Chunks are used to reliably deliver DDP Segments with the session. 5. Data Formats 2.15.1. Adaptation Layer Indicator This mapping placesThe DDP/SCTP Adaptation Layer uses all streams within an entireSCTP association into a specific DDP mode:association. An SCTP Association that has had the DDP or DDP+RDMA.Adaptation Indication negotiated will carry only SCTP Data Chunks as defined in this document. It is presumed that the handling of incoming data chunks for DDP enabled associations is sufficiently different than for routine SCTP associations that it is undesirable to mixrequire support for mixing DDP and non-DDP streams in a single association. AnMore than a single association is required if an application that needsdesires to mixutilize both DDP and non-DDP traffic must use use different associationswith different adaptation indications forthe DDP traffic and non-DDP traffic.same remote host. We define a adaptionadaptation indication which MUST appear in the INIT or INIT-ACK with the following format as defined in [ADDIP-Draft] [I-D.ietf-tsvwg-addip-sctp] 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type =0xC006 | Length = Variable | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Adaptation Indication | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Adaptation Indication: The following values are defined for DDP in this document: DDP - 0x00000001 DDP+RDMA - 0x00000002 2.25.2. Payload Data Chunks After the SCTP association has been established, all DDP relevant messages are encoded as Payload Data Chunks. Each includes a SCTP Stream identifier, a Transmissions Sequence Number (TSN), a Payload Protocol Identifier, the chunk length and the payload data bytes.The DDP SCTP adaptation uses two types of SCTP Payload Data Chunks, differentiated by the Payload Protocol Identifier: DDP SegmentsSegment Chunks are useused to for messagesreliably deliver DDP Segments sent between DDP Endpoints. Each DDP Segment is exactly contained in one SCTP payload data chunk with the payload protocol identifier 0x00000001DDP Stream Session messagesControl Messages are used to controlestablish and teardown DDP Stream Sessions, specifically by controlling the binding of DDP endpoints with SCTP streams. Payload Protocol Identifier: The following value are defined for DDP in this document: DDP Segment Chunk - 0x000000010x00000010 DDP Stream Session Control - 0x00000002 2.2.10x00000011 5.2.1. DDP Source Sequence Number (DDP-SSN) All SCTP Payload Data Chunks used by this Adaptation layer include a DDP Source Sequence Number (DDP- SSN) that(DDP-SSN). the DDP-SSN tracks the sequence the messages were submitted to the SCTP layer. This field MUST be maintained by the adaptation layer. It is initialized to 1layer for eachthe SCTP stream atin use. The DDP-SSN MUST have the beginning of each DDPsame value that the SCTP Stream Session. DDP-SSN is increased by one (modulo 65536)Sequence Number (SSN) would have been assigned had ordered SCTP Payload Data Chunks been used rather than unordered. The rationale for each DDP segment submitted tospecifying the SCTP layerDDP-SSN is as follows: The SCTP Stream Sequence Number (SSN) is not suitable for this purpose, because all messages defined by this document use unordered Payload Data Chunks to ensure prompt delivery from the receiving SCTP layer. The SCTP Transmission Sequence Number (TSN) is not suitable for determine the original order of Data Chunks within a stream. The sending SCTP layer is allowed to optimize the transmission sequence of unordered Data Chunks to encourage Chunk Bundling, or other purposes. DDP requires that an LLP deliver ordering information with each DDP Segment. The SCTP Adaptation presents the DDP-SSN for this purpose. 184.108.40.206.2. DDP Segment Payload DataChunk 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP-SSN | DDP Segment | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ DDP Segments are as defined in [DDP-Draft]. 2.2.3[DDP] [I-D.ietf-rddp-ddp]. The DDP Segment Chunk serves the same purpose as the MPA Upper Layer PDU (MULPDU) in that it carries DDP Segments over a reliable protocol with added sequencing information. 5.2.3. DDP Stream Session Control Data Chunk0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP-SSN | Function Code | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Private Data (Dependent on Function Code) | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The following function code values are defined for DDP in this document: DDP Stream Session Initiate - 0x001 DDP Stream Session Accept - 0x002 DDP Stream Session Reject - 0x003 DDP Stream Session Terminate - 0x004 ULP supplied Private Data MUST be included for DDP Stream Session Initiate DDP Stream Session Accept and DDP Stream Session Reject messages. However, the ULP supplied Private DATA MAY be of zero length. Private Data length MUST NOT exceed 512 bytes in any message. Private Data MUST NOT be included for the DDP Stream Session Terminate message. The length of private data is derived from the length of the Data ChunkReceived DDP Stream Session Control messages SHOULD be reported to the ULP. If reported, any supplied Private Data MUST be available for the ULP to examine. ThereThe DDP/SCTP adaptation layer MAY be alimit onthe rate at which Streamnumber of Session Control message can be reportedInitiate requests that it has submitted to the ULP. When this rate is exceeded, or when other factors prevent the message from being reporteda DDP Stream Session Initiate cannot be forwarded to the ULP,ULP due to such a limit the sessionadaptation layer MUST be terminated. 3.respond with a DDP Stream Session Terminate message. 6. DDP Stream Sessions A DDP Endpoint is the logical sender/receiver of DDP Segments. A DDP Stream connects two DDP Endpoints using a matched pair of SCTP Streams. A DDP Stream Session defines the sequence of Data Chunks exchanged between two DDP Endpoints over a DDP Stream that has a distinct beginning and end. Data Chunks from one DDP Stream Session are never carried over to the next session. The local interface MAY associate a DDP Endpoint with the DDP Stream based upon the initial exchanges of a DDP Session, and terminate that association at the session's end. A DDP Stream is associated with at most one Protection Domain during a single DDP Stream Session. 3.16.1. Sequencing The DDP Source Sequence Number(DDP-SSN) is reset to onezero at the beginning of each DDP Stream Session. The Payload Data Chunks for a given session, when sequenced by their DDP-SSN, MUST follow one of the patterns defined in this section. If the adaptation layer receives a Payload Data Chunk that conforms to none of the enumerated legal patterns the DDP Stream Session MUST be terminated. 3.26.2. Legal Sequence: Active/Passive Session Accepted In this DDP Stream Session sequence one DDP Endpoint assumes the active role in requesting a DDP Stream Session, which the other side accepts. Active Side sends a DDP Stream Session Initiate message. Passive Side sends a DDP Stream Session Accept mesage.message. Each side may then send zero or more DDP Segments with increasing DDP-SSNs, subject to various layers of flow control. The final User Data Chunk for each side MAY be a DDP Stream Terminate. At least one side MUST send a DDP Stream Terminate. Note that this would follow any RDMAP Terminate message, which to thisthe Adaptation layer is simply another Payload Data Chunk. 3.3DDP Segment. 6.3. Legal Sequence: Active/Passive Session Rejected DDP Stream Sessions allow each party to send a single non-payload message before the other end commits specific resources to the session. This allow each end to determine which resources are to be used, and how they are to be configured, or even if the session should be granted. These decision MAY be influenced by the need to assign a specific Protection Domain, to determine how many RDMA Read Credits are required, or to determine now many receive operations the ULP should enable. Because of these, or other, factors the Passive side MAY choose to reject a DDP Stream Session Request. This results in the following legal sequence: Active Side sends a DDP Stream Session Initiate message. Passive Side sends a DDP Stream Session Reject mesage.message. An DDP Stream Session Reject message MUST NOT be sent unless the rejection is at the direction of the ULP. 3.46.4. Legal Sequence: Active/Passive Session Non-ULP Rejected Acceptance or rejection of DDP Stream Session Initiate messages SHOULD be under the control of the ULP. This MAY require passing an event to the ULP. There MUST be a finite limit on the number of such requests that are pending a ULP decision. When more session requests are received the passive side MUST respond to the Initiate message with a DDP Stream Terminate Message. 3.56.5. ULP Specific Sequencing An implementation MAY choose to support additional ULP specific sequences, but MUST NOT do so unless requested to do so by the ULP. A defined ULP MUST be able to operate using only the defined mandatory session sequences. Any additional sequences must be used only for optional optimizations. 3.66.6. Other Sequencing Rules A DDP Stream Session Control message MUST NOT be sent if it may be received before a prior DDP Stream Session Control message within the same DDP Stream Session. An active side of a DDP Stream Session MUST NOT send a DDP Segment that might be received before the DDP Stream Session Initiate message. This MAY be determined by SCTP ackingacknowledgment of the Data Chunk used to carry the DDP Stream Session Initiate message, or by receipt of a responsive DDP Stream Session Control message. A DDP Stream MUST NOT be re-used for another DDP Stream Session while any Data Chunk from a prior session might be outstanding. 4.7. SCTP Endpoints 4.17.1. Adaptation Layer Indication Restriction The local interface MUST allow the ULP to specify an SCTP endpoint to use a specific Adaptation Indication. It MAY require the ULP to do so. Once an endpoint decides on its acceptable Adaptation Indication(s), it SHOULD terminate all requests to establish an association with any different Adaptation Indication. An SCTP implementation MAY choose to accept association requests for a given SCTP endpoint only until one association for the endpoint has been established. At that point it MAY choose to restrict all further associations for the same endpoint to use the same Adaptation Indication. 4.27.2. Multihoming Implications SCTP allows an SCTP endpoint to be associated with multiple IP addresses, potentially representing different interface devices. Distribution of the logic for a single DDP stream across multiple input devices can be very undesirable, resulting in complex cache coherency challenges. Therefore the local interface MAY restrict DDP-enabled SCTP endpoints to a single IP address, or to a set of IP addresses that are all assigned to the same input device ("RNIC"). The default binding of a DDP enabled SCTP endpoint SHOULD NOT cover more than a single IP address unless doing so results in no additional bus traffic or duplication of memory registration resources. This will frequently result in a different default than for SCTP endpoints that are not DDP enabled. Even when multi-homing is supported, ULPs are cautioned that they SHOULD NOT use ULP control of the source address in attempt to load- balance a stream across multiple paths. A receiving DDP/SCTP implementation that chooses to support multi-homing SHOULD optimize its design on the assumption that multi-homing will be used for network fault tolerance, and not to load-balance between paths. This is consistent with recommended SCTP practices. 5.8. Number of Streams DDP Streams are bidirectional. They are always composed by pairing the inbound and outbound SCTP streams with the same SCTP Stream Identifier. DDPThe adaptation layer should request the maximum number of SCTP stream it will wish to use over the lifetime of the association. SCTP streams must still be bound to DDP Endpoints, and a DDP or DDP+RDMA enabled SCTP association does not support ordered Data Chunks. Therefore the mere existence of an SCTP stream is unlikely to require signifigantsignificant supporting resources. This mapping uses an SCTP association to carry one or more DDP Steams. Each DDP Stream will be mapped to a pair of SCTP streams with the same SCTP stream number. DDPThe adaptation MUST initialize all of its SCTP associations with the same number of inbound and outbound streams. 6.9. Fragmentation A DDP/SCTP Receiver already must dealdeals with fragementationfragmentation at both the IP and DDP Layers. Therefore use of SCTP layer segmenting will be avoided for most cases. As a Lower Layer Protocol (LLP) for DDP, the SCTP adaptation layer MUST inform the DDP layer of the maximum DDP Segment size that will be supported. This should be the largest value that can be supported without use of IP or SCTP fragmention,fragmentation, or 516 bytes, whichever is larger. A minimum of 516 bytes is required to allow a DDP Stream Session Control Message with 512 bytes of Private Data. SCTP Data Chunkdata chunk fragmentation MUST NOT be used forunless the cases where IP fragmentationalternative is not required. SCTP data chunk fragmentation MAY be used to avoidIP fragmentationfragmentation. The SCTP adaptation layer SHOULD set the maximum DDP Segment size below the theoretical maximum in order to allow bundling of Control Chunks in the same SCTP packet. The SCTP adaptation layer MUST reject user messagesDDP Segments that are larger than the maximum size specified. 7.10. Sequenced Unordered Operation DDPThe Adaptation layer MUST use the Unordered option on all Data Chunks (U Flag set to one). The SCTP Layer is expected to deliver unordered Data Chunks to the DDP layerwithout delay. Because DDP employs unordered SCTP delivery, the receiver MUST NOT rely upon the SCTP Transmission Sequence Number (TSN) to imply ordering of DDP Segments. The fact that the SCTP Data Chunk for a DDP Segment is prior the cumulative ack point does not guarantee that all prior DDP segments have been placed. The SCTP sender is not obligated to transmit unordered Data Chunks in the order presented. The DDP-SSN can be used without special logic to determine the submission sequence when the maximum number of in-flight messages is less than 32768. This also applies if the sending SCTP accepts no more than 32767 Data Chunks for a single stream without assigning TSNs. If SCTP does accept more than 32768 Data chunks for a single stream without assigning TSNs, the sending DDP must simply refrain from sending more than 32767 Data Chunks for a single stream without acknowledgement.acknowledgment. Note that it MUST NOT rely upon ULP flow control for this purpose. Typical ULP flow control will deal exclusively with untagged DDP Messages,messages, not with DDP segments. The receiving DDP implementationreceiver MAY perform a validity check on received DDP-SSNs to ensure that any gap could be accounted for by unreceived Data Chunks. Implementations are advised against allocating resources on the assumption that DDP-SSNs are valid without first performing such a validtityvalidity check. An invalid DDP-SSN MAY result in termination of the DDP Stream. 8.11. Procedures 8.111.1. Association Initialization At the startup of an association, an endpoint wishing to perform DDP, RDMA, or DDP+RDMA placementa DDP/SCTP Adaptation Layer MUST include an adaptation layer indication in its INIT or INIT-ACK (as defined in Section 220.127.116.11. After the exchange of the initial first two SCTP chunks (INIT and INIT- ACK),INIT-ACK), an endpoint MUST verify and inspect the adaptation indication and compare it to the following table to determine proper action. Indication | Action type | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | This indicates that the peer DOES NOT NONE | support ANY DDP or RDMA adaptionadaptation and thus | RDMA and DDP procedures MUST NOT be | performed upon this association. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | This indicates that the peer DOES support DDP | DDP (but not RDMA). Procedures outlined in | [DDP-Draft] MUST be followed. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | This indicates that the peer supports BOTH DDP+RDMA | RDMA and DDP. If the receiving endpoint | indicated the same, then the procedures in | both [RDMA-Draft] and [DDP-Draft] | MUST be followed. If the local endpoint | only indicated DDP, then ONLYthe | procedures in [DDP-Draft] MUST be followed.DDP/SCTP Adaptation Layer defined here. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | This indicates that the peer DOES NOT ANY-OTHER | support ANYthe DDP or RDMA adaptionadaptation and thus Indication | RDMA andDDP procedures MUST NOT be |performed | upon this association. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ An implementation MAY require that all associations for a given SCTP endpoint be placed in the same mode. The local interface MAY allow the ULP to accept only requests to establish an association in a specified mode. 8.211.2. Chunk Bundling SCTP allows multiple Data Chunks to be bundled in a single SCTP packet. Data chunks containing DDP Segments with untagged messages SHOULD NOT be delayed to facilitate bundling. Data chunks containing DDP Segments with tagged messages will generally be full sized, and hence not subject to bundling. However partial size tagged messages MAY be delayed, as that they are frequently followed by a short untagged message. 8.311.3. Association Termination Termination of an SCTP Association due to errors should be handled at the SCTP layer. The RDMAP defined RDMAP Terminate Message SHOULD NOT be sent on each DDP Stream when a determination has been made to terminate an SCTP association. Sending that message on each SCTP stream could severely delay the termination of the association. The local interface SHOULD notify all consumers of DDP streams when the underlying SCTP stream has been terminated. Other RDMAP defined Terminate Messages MUST be generated as specified when a DDP Stream is terminated. Note that with the SCTP mapping, termination of a DDP Stream does not mandate termination of the Association. 9.12. IANA considerations This document defines twoone new Adaptation Layer Indication codepoints:codepoints with the recommended value of: DDP - 0x00000001 DDP+RDMA - 0x00000002This document also defines two new Payload Protocol Identifier (PPIDs): DDP Segment Chunk - 0x000000010x00000010 DDP Stream Session Control - 0x00000002 10.0x00000011 IANA is requested to assign the next two Payload Protocol Identifiers. 13. Security Considerations Any direct placement of memory could pose a significant security risk if adequate local controls are not provided. These threats should be addressed in the appropriate DDP [DDP-Draft]  or[I-D.ietf-rddp-ddp], RDMA [RDMA-Draft] [I-D.ietf-rddp-rdmap] or Security [RDMA-Security] [I-D.ietf-rddp-security] drafts. This document does not add any additional security risks over those found in RFC2960 . 11.[RFC2960]. 14. Acknowledgments Special Acknowledgment to Sukanta Ganguly for his extra efforts in reading and reviewing this document.The authors would like to thank the following people that have provided comments and input: Stephen Bailey, David Black, Douglas Otis, Allyn Romanow and Jim Williams. 12. Normative15. References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2960] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang, L., and V. Paxson, "Stream Control Transmission Protocol", RFC 2960, October 2000. [I-D.ietf-rddp-ddp] Shah, H., "Direct Data Placement over Reliable Transports", draft-ietf-rddp-ddp-05 (work in progress), July 2005. [I-D.ietf-rddp-rdmap] Recio, R., "An RDMA Protocol Specification", draft-ietf-rddp-rdmap-05 (work in progress), July 2005. [I-D.ietf-rddp-security] Pinkerton, J., "DDP/RDMAP Security", draft-ietf-rddp-security-09 (work in progress), May 2006. [I-D.ietf-tsvwg-sctpsocket] Stewart, R., "Sockets API Extensions for Stream Control Transmission Protocol (SCTP)", draft-ietf-tsvwg-sctpsocket-10draft-ietf-tsvwg-sctpsocket-12 (work in progress), February 2005. 2006. [I-D.ietf-tsvwg-addip-sctp] Stewart, R., "Stream Control Transmission Protocol (SCTP) Dynamic Address Reconfiguration", draft-ietf-tsvwg-addip-sctp-12draft-ietf-tsvwg-addip-sctp-14 (work in progress), June 2005.March 2006. Authors' Addresses Randall R. Stewart Cisco Systems, Inc. Forest Drive Columbia, SC 29036 USA Phone: +1-815-342-5222 Email: firstname.lastname@example.org Caitlin Bestler Broadcom 49 DiscoveryCorporation 16215 Alton Parkway P.O. Box 57013 Irvine, CA 9261892619-7013 USA Phone: 949-926-6383 Email: email@example.com Jim Pinkerton Microsoft One Microsoft Way Redmond, WA 98052 USA Phone: +1-425-705-5442 Email: firstname.lastname@example.org Sukanta Ganguly Consultant Phone: +1-858-748-5268 Email: email@example.com@yahoo.com Hemal V. Shah IntelBroadcom Corporation Mailstop: PTL1 1501 S. Mopac Expressway, #400 Austin, TX 7874616215 Alton Parkway P.O. Box 57013 Irvine, CA 92619-7013 USA Phone: 949-926-6023 Email: firstname.lastname@example.org@broadcom.com Vivek Kashyap IBM 15450 SW Koll Parkway Beaverton, OR 57006 USA Phone: +1-503-578-3422 Email: email@example.com Sukanta Ganguly Consultant Phone: Email: firstname.lastname@example.orgIntellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at email@example.com. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005).(2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society.