Transport Area Working Group                   S. Bailey    (Sandburst)
Internet-draft                                               Sandburst                                 J. Chase          (Duke)
Expires: January 2001 May 2002                              J. Pinkerton
                                                             Microsoft (Microsoft)
                                               A. Romanow       (Cisco)
                                               C. Sapuntzakis
                                                                 Cisco
                                                            M. Wakeley
                                                               Agilent   (Cisco)
                                               J. Wendt
                                                                    HP            (HP)
                                               J. Williams
                                                                Emulex     (Emulex)

                     TCP ULP Framing for TCP
                   draft-ietf-tsvwg-tcp-ulp-frame-00 Protocol (TUF)
                   draft-ietf-tsvwg-tcp-ulp-frame-01

Status of this Memo

     This document is an Internet-Draft and is in full conformance with
     all provisions of Section 10 of RFC2026.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF), its areas, and its working groups.  Note that
     other groups may also distribute working documents as Internet-
     Drafts.

     Internet-Drafts are draft documents valid for a maximum of six
     months and may be updated, replaced, or obsoleted by other
     documents at any time.  It is inappropriate to use Internet-Drafts
     as reference material or to cite them other than as "work in
     progress."

     The list of current Internet-Drafts can be accessed at
     http://www.ietf.org/ietf/1id-abstracts.txt

     The list of Internet-Draft Shadow Directories can be accessed at
     http://www.ietf.org/shadow.html.

Copyright Notice

     Copyright (C) The Internet Society (2001). All Rights Reserved.

Abstract

     The framing TCP ULP Framing (TUF) protocol accepts PDUs from defines a ULP (upper level protocol) shim layer protocol
     between an Upper Layer Protocol (ULP) and transports them over TCP.  TUF also depends on
     a specified TCP connection.  This is done in such a
     way that the PDUs can be recovered at segmentation convention between TUF endpoints.
     Together, the shim and segmentation conventions enable a TUF/TCP
     receiver even if
     preceding to recognize ULP data units within a TCP segments have not yet been received. segment
     independently of other TCP segments.  This is useful
     when the PDUs are self describing within capability simplifies
     the context design of enhanced network interfaces implementing direct data
     placement for ULPs using TCP.  Direct data placement is a protocol
     TCP connection.  In this case, the framing protocol allows incoming
     packets key step
     to be parsed (but not processed) making IP networking competitive with high-end interconnect
     solutions in the order received and
     their data to be placed directly in the ultimate destination memory
     instead of TCP reassembly buffers. centers and other high-performance application
     domains.

Table Of Contents

     1. Introduction     Definitions  . . . . . . . . . . . . . . . . . . . . . .   3
     2.     Overview . .   2
     2. Theory Of Operation . . . . . . . . . . . . . . . . . . . .   3
     3. ULP Support For Framing . .   4
     2.1.   Motivation . . . . . . . . . . . . . . . .   5
     4. Negotiating Use Of The Framing Protocol . . . . . . .   4
     2.2.   Approach . . .   6
     5. PDU Alignment Mode . . . . . . . . . . . . . . . . . . . . .   6
     5.1. Framing-aware TCP   5
     3.     Rational For TUF . . . . . . . . . . . . . . . . . . . .   8
     5.2. PDU Alignment Mode Exception Cases   6
     3.1.   Direct Data Placement  . . . . . . . . . . . .   9
     5.3. Validity Of Framing-aware TCP Segmentation . . . . .   7
     3.2.   Direct Data Placement with TCP . . .  10
     5.4. Receiving In PDU Alignment Mode . . . . . . . . . .   8
     3.2.1. The Simple Case: ULP-unaware Placement . . .  11
     6. Marker Mode . . . . . .   9
     3.2.2. The Complex Case: ULP-aware Placement  . . . . . . . . .   9
     3.2.3. The Problem of ULP-aware Placement with TCP  . . . . . .  10
     3.2.4. Finding ULPDUs In Out-of-order Segments  . . .  12
     7. Security Considerations . . . . .  11
     3.2.5. The TUF Solution . . . . . . . . . . . . .  12
     7.1. Security Protocol Interactions . . . . . . .  12
     3.2.6. TUF's ULP Assumptions  . . . . . . .  13
     7.2. Using IPSec With The Framing Protocol . . . . . . . . . .  13
     7.3. Using TLS With  12
     4.     The Framing Protocol . . . . . . . . . . .  13
     7.3.1. Using TLS In PDU Alignment Mode  . . . . . . . . . . . .  15
     7.3.2. Using TLS In Marker Mode .  13
     4.1.   The Framing Protocol Data Unit (FPDU)  . . . . . . . . .  13
     4.1.1. FPDU Format  . . . . . .  15
     7.4. Other Security Considerations . . . . . . . . . . . . . .  16
     8. IANA Considerations . .  13
     4.1.2. FPDU Size Selection  . . . . . . . . . . . . . . . . . .  16
     9. References  14
     4.2.   TUF-conforming TCP Sender Segmentation . . . . . . . . .  15
     4.3.   Negotiating TUF  . . . . . . . . . . . . . . . .  16
     Authors' Addresses . . . .  15
     4.4.   TUF Receiver ULPDU Containment Property Testing  . . . .  16
     5.     Protocol Characteristics . . . . . . . . . . . . . . . .  17
     A. Sockets Support For The Framing Protocol
     5.1.   Properties Of TUF-conforming TCP Senders . . . . . . . .  17
     5.2.   Exception Cases  . . . . . . . . . . . . . . . . . . . .  18
     5.2.1. Resegmenting Intermediaries  . . . . . . . . . . . . . .  18
     5.2.2. PMTU Reduction . . .  19
     A.1 Enabling The Framing Protocol . . . . . . . . . . . . . . .  20
     A.2 Sending Data Atomically . . .  19
     5.2.3. PMTU Increase  . . . . . . . . . . . . . . . . . . . . .  20
     A.3 Retrieving The Current
     5.2.4. Receive Window < EMSS  . . . . . . . . . . . . . . . .  21
     A.4 Disabling ULP PDU Packing .  21
     5.2.5. Size of ULPDU + 8 > EMSS . . . . . . . . . . . . . . . .  21
     A.5 Enabling Emergency Mode
     6.     Security Considerations  . . . . . . . . . . . . . . . .  22
     6.1.   Protocol-specific Security Considerations  . .  21
     A.6 Setting The Sending Marker Interval . . . . .  22
     6.2.   Using IPSec With TUF . . . . . . .  22
     A.7 Setting The Receiving Marker Interval . . . . . . . . . . .  22
     Full Copyright Statement
     6.3.   Using TLS With TUF . . . . . . . . . . . . . . . . . . .  22

1.  Introduction

     Many upper layer protocols (ULP)s, particularly those which perform
     bulk data transfer, permit the final location of transferred data
     (e.g. a ULP client buffer) to be known when the data is received.
     The information required to compute the final location of such data
     is contained in local protocol state and ULP protocol data unit
     (PDU) headers.  In this case, ULP data can be placed directly at
     its final destination by a network interface with knowledge of the
     ULP.  A direct placement network interface can offer extremely high
     performance since the host CPU does not copy the data at all, and
     the data only crosses system buses once.

     Both specific application ULPs, such as iSCSI, and generic hardware
     acceleration ULPs, such as an RDMA protocol, offer the potential
     for direct data placement.  The advantage of using a generic
     acceleration ULP for direct data placement is that the same direct
     placement network interface can be used to accelerate many
     different application protocols (e.g. iSCSI
     7.     IANA Considerations  . . . . . . . . . . . . . . . . . .  25
            References . . . . . . . . . . . . . . . . . . . . . . .  25
            Authors' Addresses . . . . . . . . . . . . . . . . . . .  26
     A.     Sample Sockets Support For TUF . . . . . . . . . . . . .  27
     A.1    Basic Principles . . . . . . . . . . . . . . . . . . . .  28
     A.2    Enabling TUF . . . . . . . . . . . . . . . . . . . . . .  28
     A.3    Sending Data . . . . . . . . . . . . . . . . . . . . . .  29
     A.4    Retrieving The Current EMSS or MULPDU  . . . . . . . . .  29
     A.5    Disabling ULPDU Packing  . . . . . . . . . . . . . . . .  29
     A.6    Disabling The Report of Oversized ULPDUs . . . . . . . .  30
            Full Copyright Statement . . . . . . . . . . . . . . . .  30

1.  Definitions

     The following terms and abbreviations are used in this document.

          data delivery - the delivery of received ULP payloads to the
          ULP application, i.e, notifying the application of data
          arrival by completing a receive operation or generating an
          event.

          data placement - the storage of received ULP payloads to host
          memory, pending delivery to the ULP application.

          direct data placement - the storage of received ULP payloads
          directly to application-specified buffers without intermediate
          buffering or copying.

          EMSS - the effective maximum segment size.  EMSS is the TCP
          maximum segment size (MSS) defined in RFC 793 [TCP] and
          exchanged during TCP connection establishment, adjusted by the
          current path maximum transfer unit (MTU) [PathMTU].

          FPDU - framing protocol data unit.  The protocol data unit
          defined by TUF.

          MULPDU - maximum upper layer protocol data unit size.  The
          size of the largest ULPDU that fits in an EMSS-sized FPDU.

          NIC - network interface controller.  The device that provides
          a host's access to a physical network link.

          PDU - protocol data unit.  A self-contained block of control
          and data defined by a particular protocol.

          RDMA - Remote Direct Memory Access protocol.  A data transfer
          protocol which uses memory access-style transfer mode(s) to
          provide generic direct data placement capabilities for
          arbitrary ULPs.

          TUF - TCP ULP Framing protocol.  The protocol defined in this
          document.

          ULP - upper layer protocol.  The client protocol using the
          services of the transport layer, or TUF.

          ULPDU - upper layer protocol data unit.

          ULPDU containment property - the property that a TCP segment
          contains exactly an integral number of ULPDUs.

2.  Overview

     This section summarizes the motivation for the TCP ULP Framing
     (TUF) protocol and explains its operation in brief.  Section 3
     (`Rational for TUF') develops the rationale for TUF in detail.
     Section 4 (`The Protocol') defines the protocol itself.  Section 5
     (`Protocol Characteristics') examines various properties of the
     protocol's operation.  Implementors may wish to refer directly to
     sections 4 and 5.

2.1.  Motivation

     The IP protocols are not usually used for high-performance high
     speed data transfers due to overhead in TCP processing. Instead, a
     number of special purpose protocols have been used. The domain of
     application for such high speed buffer transfer includes storage,
     video delivery and processing, and various applications of cluster
     computing, such as scalable database or application service.  For
     reasons discussed below, today, there is great industry interest in
     developing an IP standard for low overhead high bandwidth data
     transfer, which would decrease the costs of high speed
     interconnects and supplant special purpose protocols.

     The approach typically used for low overhead transfers is called
     direct data placement, in which the network interface places data
     directly in application buffers, avoiding the latency and memory
     bandwidth costs associated with copying.  Direct data placement can
     in principal be done with either of IP's reliable transports--SCTP
     or TCP.  This document considers what is needed to do direct data
     placement with TCP.

     In order to place data directly in application buffers, the network
     interface needs to use information in the Upper Layer Protocol Data
     Units (ULPDUs) contained in the TCP stream.  This can be
     accomplished routinely except when TCP segments arrive out of
     order.  If TCP segments arrive out of order, the location of the
     ULPDUs in the TCP segment cannot be found.  The TUF protocol
     addresses this problem of finding ULPDU headers in the TCP stream,
     even when TCP segments arrive out of order.

2.2.  Approach

     TUF is implemented as a shim layer between an ULP and TCP.  The
     end-to-end data flow is:

     0.   Use of TUF is negotiated end-to-end by the ULP.

     1.   The ULP delivers a data stream with ULPDUs delimited to TUF.

     2.   TUF inserts a header and delivers the shimmed ULPDUs to TCP.

     3.   The TUF-aware TCP sender preserves boundaries of shimmed
          ULPDUs (TUF FPDUs) as much as possible when delivering
          segments to the IP layer.

     4.   The receiving TCP delivers shimmed ULPDUs to the receiving TUF
          layer.

     5.   TUF removes the shim and delivers the ULPDUs to the ULP.

     In other words, the layering of TUF is:

                          ULP client
                               ^
                               |
                               | ULPDUs (in octet stream)
                               |
                               v
                              TUF
                               ^
                               |
                               | FPDUs (containing ULPDUs)
                               |
                               v
                      TUF-conforming TCP
                               ^
                               |
                               | TCP Segments (each containing an FPDU)
                               |
                               v
                             . . .

     Note that while the semantics of this protocol layering must be
     maintained, the receiving network interface may use the information
     in the framed ULPDUs to place the data in memory on the host.
     Whatever the case, the data is only delivered to the ULP when all
     preceding TCP data has arrived.

3.  Rational For TUF

     This document defines the TUF protocol as a shim layer between an
     Upper Layer Protocol (ULP) and TCP.  TUF also depends on a TCP
     segmentation convention between TUF/TCP endpoints specified in this
     document.  Taken together they provide the capability for a TUF/TCP
     receiver to recognize ULPDUs by processing each TCP segment
     independently, without requiring state from previous segments.

     The purpose of TUF is to enable practical designs for enhanced
     network interfaces (NICs) implementing direct data placement for
     TCP-based ULPs.  The purpose of direct data placement is to
     eliminate the need for a host to copy received data after it
     arrives in host memory.  This copying incurs CPU, memory and bus
     costs that are substantial and are not masked by advancing hardware
     technology.

     A general and practical solution to the receive copy problem has
     eluded the IP networking community for almost two decades.  There
     is a long history of research and experimental schemes to reduce or
     eliminate receiver copying overhead for IP networking in general,
     and for TCP/IP communication in particular.  While these systems
     have convincingly demonstrated the potential performance benefits
     of reducing copy costs, all such schemes suffer from one or more of
     the following limitations: they require a significant restructuring
     of operating system buffering and/or APIs; they are limited to
     specific modes of communication (e.g., bulk data transfer) or
     specific application ULPs; they do not scale on multiprocessor
     hosts; their benefits depend on specific properties of the network
     (e.g., large MTUs) or host buffer size and alignment.  Moreover,
     all such schemes require some degree of support from NICs to
     separate payloads from headers and/or ensure that their placement
     in host memory meets specific requirements (e.g., for page
     placement and alignment).

     Inherent copying costs for IP communication are one motivation to
     use alternative non-IP technologies for high-speed networking.  A
     number of specialized technologies have been developed for high
     speed data transfers in which network interfaces transfer data from
     application buffer to application buffer without software touching
     the data.  Some examples include the VAXCluster Interconnect in
     1983, Fibre Channel (FC) in 1994, and today InfiniBand (IB) and
     Virtual Interface Architecture (VIA).  These alternatives have
     eroded the popularity of IP technologies in application domains
     including network storage, video processing and delivery, and
     cluster computing for scientific applications and scalable
     database-related services.

     Until recently, several factors have limited interest in promoting
     IP networking as a solution in these application domains.  First,
     the competing network technologies offered significantly higher
     link speeds than the network hardware available for use with IP.
     Second, these application domains were a relatively small segment
     of the network market.  Recently, however, Ethernet networks have
     closed the bandwidth gap and even exceeded the bandwidth of
     alternatives such as FibreChannel, at much lower cost.  At the same
     time, an increasing number of applications are server-hosted in
     data centers to enable sharing and access from a growing number of
     IP-connected client devices and locations.  With the growth in
     importance and number of data centers, high-speed interconnection
     within the data center is now central to the everyday operation of
     Internet services.

     Thus, technology changes have created an opportunity and demand to
     extend the benefits of IP technologies to high-performance
     application domains, while simultaneously increasing the importance
     of those domains.  The ubiquity of IP offers economies of scale
     heavily favoring IP in these domains.  For example, reliance on
     specialized non-IP technologies for high-performance domains
     creates a need to support multiple protocols and redundant network
     infrastructure in data centers, and it compromises portability and
     interoperability of data center solutions.  Moreover, comprehensive
     support for network management and security is developing rapidly
     in the IP space.  Use of IP technologies would allow data centers
     to benefit from these enhancements.

3.1.  Direct Data Placement

     Direct data placement is a key step toward making IP networking
     competitive in data centers and other high-performance domains.
     Direct data placement refers to the ability of a NIC to place data
     directly from the network into designated application buffers,
     without intermediate copying.  Direct data placement is attractive
     relative to other solutions to the receive copy problem.  It is the
     only solution that can be implemented in a way that is compatible
     with existing operating systems, since the receiving NIC takes over
     most of the responsibility to avoid receive copying.  Also, direct
     data placement generalizes easily to a range of ULPs.  In
     particular, the establishment of an IETF standard for an IP
     transport-based direct data placement protocol, which would allow
     NICs to directly place data independent of the application ULP
     using it.

     The TUF protocol is necessary to permit easily deployable enhanced
     NICs supporting direct data placement.  Such NICs already exist and
     their usage is growing rapidly, but their development is impeded by
     the lack of standards.  Direct data placement is unnecessarily
     difficult and expensive to design and implement for existing TCP-
     based ULPs; the key objective of TUF is to define transport
     conventions to simplify the design of these NICs.  A related
     impediment is that in the absence of a general direct data
     placement protocol these products are limited to specific ULPs such
     as iSCSI.  TUF, and possibly additional, higher layer protocol
     definitions outside the scope of this document, would encourage the
     market by ensuring interoperability of product offerings from
     different vendors.

     This document defines a framing protocol (TUF) and TCP segmentation
     conventions that enable simple support of direct data placement for
     a class of TCP-based ULPs.  It does not propose a generic direct
     placement ULP, such as an RDMA protocol, or any facility for direct
     data placement, but only the foundations for building such a
     facility on TCP.  A key objective of TUF is to do this in a way
     that is compatible with existing standards and with the spirit of
     TCP's stream communication model.  TUF can simplify support for
     direct data placement for ULPs such as iSCSI, and it can serve as a
     basis for a future RDMA proposal.

     The key limitation of TUF as a solution to the receive copy problem
     is that it works only if the ULP standard and the sending and
     receiving implementations all support it.  Impact on RDMA).

     PDU shall mean the sender and
     ULPs is minimal, but ULPs must be adapted to allow use of TUF at
     the ULP/transport boundary.  The necessary modifications may be
     quite small.  Use of TUF is a negotiated option between the sender
     and receiver for each ULP PDU session, preserving interoperability
     among senders and receivers that do not support TUF.

3.2.  Direct Data Placement with TCP

     Direct data placement is widely used to accomplish high-performance
     data transfer in non-IP technologies such as block storage channels
     (SCSI, Fibre Channel, etc.), and other specialized high performance
     networks like InfiniBand.  This section considers how direct
     placement can be done with TCP.

     The Internet Protocol suite provides two transports that are prime
     candidates for use with direct data placement -- SCTP and TCP.  The
     framing features of the SCTP Stream Control Transmission Protocol
     [SCTP] make it more directly adaptable for direct data placement
     for future ULPs using SCTP.  However, the maturity and ubiquity of
     TCP make it desirable to define a flexible method for direct data
     placement for TCP-based ULPs as well.

     There has been a great deal of `moral confusion' concerning the
     interaction of direct data placement with TCP's ordering
     guarantees.  These ordering guarantees do not prohibit direct data
     placement, even if data is placed as it arrives out of order.

     TCP guarantees data delivery to the application ULP as an ordered,
     sequential stream [RFC793].  Data is delivered only when TCP has
     notified the application of its arrival and transferred ownership
     of the receive data buffer.  TCP does not specify how received data
     is stored prior to its delivery, and it does not preclude placement
     of data in application buffers out of order, as long as no data is
     delivered until all preceding data has also been delivered.  Out-
     of-order placement greatly simplifies direct data placement NICs
     because it streamlines data paths and eliminates the need for a TCP
     reassembly buffer on the remainder of NIC.

     An implementation performing direct data placement must still
     respect all TCP delivery semantics.  For example, if a checksum
     integrity check fails, the data must not be placed in ULP-supplied
     buffers, because, for example, the document unless
     otherwise indicated. TCP specifies that ports and the TCP sequence
     number are not trustworthy.

3.2.1.  The Simple Case: ULP-unaware Placement

     Direct data placement into a ULP is notified client-supplied buffer designated
     to hold the next data delivered to the ULP, regardless of the delivery
     contents of octets in the order in which they are presented to received data, is one of the sender.  Many ULPs
     rely on this sequencing guarantee.  While notification from TCP simplest possible
     forms of direct data placement.  This form of direct data placement
     is
     required already fully supported by existing TCP mechanisms.  New NIC
     products currently, or soon to be in-order, available, which claim to offer
     `full zero copy operation' typical provide only this does not prohibit arbitrary placement ULP-unaware
     form of TCP direct data received in any order.  Even if placement.

     While ULP-unaware direct data placement works well for a ULP is
     placed out-of-order, ULPs like
     FTP where the ULP may still only be notified entire contents of a TCP connection are known to be
     nothing but a single stream of such bulk client data, most widely used
     ULPs, e.g. HTTP [HTTP], BEEP [BEEP] and storage protocols,
     multiplex control and data, and possibly even interleave data in-order, in accordance with from
     different requests on the same TCP semantics.  In other words, connection.  The simple ULP-
     unaware direct data placement based upon ULP information is not at odds
     with TCP's stream-orientation, but rather is a natural application
     of TCP's philosophy that ULP PDU framing be performed at the layer
     above TCP.  RFC 879 also points out in its discussion inadequate to avoid data copies
     for these ULPs.

3.2.2.  The Complex Case: ULP-aware Placement

     An explicit goal of layering
     and modularity that this type of behavior proposal is completely in harmony
     with layered protocol design [RFC0879].

     Packet delay, loss to support out-of-order direct
     data placement for ULPs that provide additional transport-like
     features such as control and reordering are expected, common occurrences
     in IP networks.  Traditionally, data in multiplexing, layered above TCP segments is placed in
     an intermediate reassembly buffer to restore the sending order
     which may have been lost as a result of segment delay, loss
     (e.g., iSCSI or
     reordering.  While it is possible for a generic direct data placement network
     interface to implement a complete reassembly buffer, protocol such as
     RDMA).  In many ULPs, such as storage protocols, control
     information contained in the cost of
     doing so is prohibitive.  Such a reassembly buffer would need to
     have a size equal to ULP uniquely identifies the sum
     destination application buffer of the maximum window sizes each particular piece of all
     active connections.  On data.

     For example, suppose a client requests a read operation using a fast
     network link (e.g. > 1 Gb/s), storage ULP, specifying the
     window size for each connection can be very large, which would
     require a huge, very high speed reassembly destination buffer on for the network
     interface.

     A way to find PDUs when previous PDU headers are
     requested data.  The requesting ULP includes control information in delayed, lost
     or reordered segments will permit data
     the request (e.g., in these subsequent PDUs the ULPDU header) uniquely identifying that
     buffer, and the responder includes that information in the read
     response.  For some protocols, the identifier is a unique request
     ID, allowing the client ULP to
     be placed immediately by identify the buffer indirectly
     through a direct placement network interface.
     This will reduce table of pending requests.  If the storage protocol uses
     RDMA, the response may specify the buffer requirements for directly by means of a direct placement
     region identifier.

     A network interface.  Without such a mechanism, interface that understands the relevant ULP control
     information can use it to place the incoming data from
     subsequent PDUs must all be buffered (e.g., read
     response payload) directly in the adapter until all
     previous TCP segments are received.  Initial discussion of correct buffer.  In this
     issue, and how it relates specifically to iSCSI can be found case,
     data placement is guided by ULPDU headers embedded in an
     early iSCSI design team memo [Satran].

     This document specifies a protocol with two modes the TCP data
     stream.  The NIC accesses these headers as hints for efficiently
     finding PDUs in placement of
     the presence ULP payloads--a form of lost, delayed or reordered TCP
     segments.

2.  Theory Of Operation

     One very efficient way to guarantee that subsequent PDUs can always
     be found when a previous PDU header has been lost is to ensure integrated layer processing for each
     TCP segment begins as it arrives.  This is compatible with a PDU TCP's ordering
     properties if completion of ULP header processing and contains an integral number delivery of
     PDUs.  In this case,
     the payload data to the application are strictly in each TCP segment may be placed
     independently order.

3.2.3.  The Problem of all other segments.  No reassembly buffer is
     required at all.  Guaranteeing a ULP-aware Placement with TCP segment begins

     The problem with performing direct data placement as a PDU
     requires a modification function of
     ULP control information in TCP is that it may be difficult to TCP's sending behavior.  This document
     defines
     locate the behavior of ULP control information (ULPDU headers) within a TCP with a modified sender behavior,
     called a `framing-aware TCP'.  A framing-aware
     segment.

     If all TCP allows a segments are received in sequence order, ULP control
     information can be unambiguously located by the rules that permit
     any ULP implementation to ensure that do so.  For example, each TCP segment begins with ULPDU may
     contain a PDU.
     A framing-aware TCP is fully compliant with all RFCs governing TCP
     and fully interoperable with existing, compliant, non-framing-aware
     TCP implementations.  When length field that implicitly specifies the framing protocol can use a framing-
     aware TCP, it operates in `PDU alignment mode'.  The framing
     protocol in PDU alignment mode uses a combination location of a framing-
     aware TCP and an encapsulation
     the beginning of PDUs to permit error free PDU
     location when the subsequent ULPDU.

     If TCP segments are lost.

     Another way not received in sequence order, without taking
     additional measures, it may not be possible to unambiguously locate PDUs in the presence of lost TCP segments
     ULP control information needed for direct data placement.  For
     example, if ULPDU length information is
     to insert markers at in a known period TCP segment that is
     delayed or lost in transmission, assuming the TCP octet stream.  Each
     marker points to ULPDU length is the beginning
     only means of locating the next PDU.  If beginning of the marker
     frequency subsequent ULPDU, it is high relative
     impossible to packet loss rate (e.g. once per locate ULP control information for ULPDUs in
     subsequent TCP
     segment), the receiver can, with very high likelihood, learn the
     location of the next PDU from a marker even when a previous PDU
     header has been lost.  The receiver must still buffer the octets
     between segments until the lost or delayed TCP segment is
     received.  ULP control information, and the subsequent PDU, but this is
     likely to data whose placement
     depends on it may even be a much smaller buffer than in different TCP segments.  If the maximum ULP
     control information is in a TCP window
     size.  By limiting segment that is delayed or lost, it
     is impossible to directly place the maximum PDU size, data until the receiver buffering can
     be reasonably bounded.  This document defines ULP control
     information is received.

3.2.4.  Finding ULPDUs In Out-of-order Segments

     Early attempts at ULP-aware direct data placement in TCP took the
     approach of only directly placing data for TCP segments received
     in-order.  Otherwise, data was copied through a periodic marker
     mechanism which can be used to bound receiver reassembly buffers.

     Two framing protocol modes are defined because buffer
     as in a traditional implementation.  Unfortunately packet loss, and
     attendant out-of-order reception is a frequent, continuous
     characteristic of both wide-area, and switched local area networks
     of almost any size, as TCP adjusts to varying congestion
     conditions.  Under these conditions, a large portion of the substantial
     tradeoff between the modes.  Both modes can bound data
     transferred ends up being copied, rather than being directly
     placed.

     Another solution to this problem is to build a reassembly buffer
     on a direct placement network interface, but
     into the modes apply network interface.  Data received out-of-order can be held
     in
     disjoint circumstances.

     Marker mode has the following advantage:

     1.   Implementable without TCP sender modification

     The PDU alignment mode has the following advantages:

     1.   No network interface reassembly buffering required at buffer until all

     2.   Placement information preceding data
     is always at received, and then direct placement can be performed on the start of a TCP segment,
          substantially simplifying hardware processing

     PDU alignment mode
     reassembled data.  Within certain implementation assumptions, this
     is more powerful, reasonable approach, but, unfortunately there are a number of
     issues including very large memory requirements, limited
     scalability, and is preferable when
     available.  Marker mode still requires some high-speed increased latency, that make the reassembly
     memory, whose
     approach undesirable.

     The size of reassembly buffer needed in the network interface is a linear
     direct function of the number bandwidth * delay product of all active TCP
     connections.  Furthermore, marker mode only offers a probabilistic
     bound  Reasonable assumptions on the reassembly buffer size per active TCP connection.  In
     cases where many TCP segments with PDU headers are lost, the buffer
     size required for direct placement could approach that of bandwidth *
     delay product can imply a
     complete reassembly buffer.

     It is expected that ultimately PDU alignment mode will dominate
     because large amount of compelling cost and performance scalability advantages.
     However, until framing-aware TCPs are ubiquitous, marker mode
     offers an alternative for use with an unmodified TCP
     implementation.  To make transition from marker mode to PDU
     alignment mode easy, reassembly memory.
     Furthermore, this large reassembly memory must run at high
     speed---more than two times the sockets API extension defined link speed, to maintain full link
     bandwidth.

     Finally, performing reassembly in Appendix
     A supports both modes relatively transparently.  A ULP which
     implements the behavior required for PDU alignment mode can use
     marker mode without modification.

     Framing protocol receivers MAY implement either PDU alignment mode,
     or marker mode, or both.  Framing protocol senders, MUST implement
     marker mode, and MUST implement PDU alignment mode if network interface requires
     that the
     underlying TCP is framing-aware.

3.  ULP Support For Framing

     A ULP using bandwidth from the framing protocol will submit each complete PDU network interface to host memory be not
     just equal, but substantially greater than the maximum bandwidth of
     the network link, to ensure that the framing module in a single sending operation.  This behavior reassembly buffer is
     already common practice for most ULP implementations.

     When the framing protocol drained
     when reassembly is complete.  System bus and interconnect bandwidth
     are particularly scarce and expensive resources in PDU alignment mode, each PDU
     submitted most systems.

     What is limited needed to permit ULP-aware direct data placement without
     reassembly buffering is a way to ensure that the smaller of 2^16-8 (65528) ULP control
     information and the size
     that will fit entirely data associated with it is highly likely to be
     contained completely within a single TCP segment.  The framing protocol
     in PDU alignment mode MUST fail any attempt segment, and a way for a
     receiver to submit validate this containment property on TCP segments it
     receives.  If the receiver can determine that a PDU ULPDU starts at the
     beginning of a TCP segment, the receiver can perform ULP-aware
     direct placement for that is
     larger than will fit with an 8-byte framing header ULPDU, and subsequent ULPDUs contained in a
     that TCP segment.  The TCP maximum segment size (MSS) property that a ULPDU is defined in RFC 793 [TCP] as
     the segment size exchanged on completely
     contained within a TCP connection establishment.  In
     addition, there is the segment size presently used by TCP which is
     less than or equal to called the exchanged MSS, adjusted by `ULPDU containment
     property'.

3.2.5.  The TUF Solution

     The TUF protocol defines a shim layer above TCP and below the current
     path MTU [PathMTU].  This document calls ULP
     that allows the MSS presently in use receiver to validate the `effective maximum ULPDU containment property
     for each TCP segment size' (EMSS).  The EMSS is received, independently of
     primary concern to any other TCP
     segment.  The TUF protocol also defines a segmentation behavior for
     the TCP sender that ensures the ULPDU containment property holds as
     often as possible while still respecting the framing protocol in PDU alignment mode. requirements
     for TCP senders.

     The TUF-specified TCP EMSS can shrink segmentation behavior ensures that the ULPDU
     containment property is maintained as long as the receiver window
     size is at least equal to 8 octets [PathMTU] which leaves no room
     for a PDU in PDU alignment mode. If the EMSS goes below 512 octets, effective MSS (EMSS), the ULP MAY instruct path MTU
     (PMTU) does not change, and the framing protocol to enter TCP stream is not resegmented by an "emergency
     mode."
     intermediary.  In this mode, conditions where the framing module MUST accept PDUs up to 512
     octets and MAY fragment a PDU across TCP segments.

     The EMSS may change during the course of the connection.  The
     framing module in PDU alignment mode MUST notify the ULP sender of
     changes in receiver window size is
     smaller than EMSS, or the EMSS.  The framing module in PDU alignment mode MUST
     provide PMTU changes, the current value of segmentation behavior
     further ensures that once the path EMSS to relevant condition is restored, the ULP on request.

     When
     ULPDU containment property will be satisfied again.

     For the framing high-performance applications that this protocol is in marker mode, each PDU submitted is
     limited to 2^16-8 minus targets,
     small receiver window sizes, and PMTU changes are rare transients.
     Thus, the size of all interspersed markers.  The
     framing specified protocol ensures that ULP control information
     and its associated data are virtually always together in marker mode MUST fail any attempt to submit a
     PDU larger than this limit.  The framing module MAY impose a
     smaller, implementation specific size limit on PDUs.  In order to
     effectively bound the receiver's reassembly buffer size, the single
     TCP segment.

3.2.6.  TUF's ULP
     SHOULD submit PDUs limited in size by some appropriate function Assumptions

     A key assumption of TUF is that ULPs running on TUF can adjust
     ULPDU sizes to fit completely within an EMSS-sized TCP segment.
     Clearly, if a ULPDU does not fit within an EMSS-sized TCP segment,
     the receiver's reassembly buffer resources, ULPDU containment property can not be satisfied.  Most storage
     protocols (e.g. iSCSI), and other performance-targeted protocols
     (e.g. RDMA protocols) support this capability.  ULPs that can not
     adjust ULPDU sizes to fit within an EMSS-sized TCP segment, but no specific limit
     is imposed by
     still want the framing protocol.

4.  Negotiating Use Of The Framing Protocol

     Negotiating use performance advantages of the framing protocol is the responsibility direct data placement, can
     be mapped on top of an intermediate protocol (e.g. an RDMA
     protocol) that does support this data `chunking'.

     TUF does not change the ULP.  The use stream delivery semantics of TCP to the framing protocol MAY be negotiated
     separately for each direction on
     ULP, through the TUF implementation.  It merely inserts a particular connection.  The
     negotiation procedure MUST ensure shim
     header that when receive framing is
     enabled, the remote peer will not transmit can be used by direct placement network interfaces to
     verify the first TCP segment
     with framed data until it ULPDU containment property.  The shim header is certain that inserted
     by the local peer has
     actually enabled receive framing.

     If a receiver requests PDU alignment mode, sending TUF implementation and removed by the sender supports
     PDU alignment mode, then receiving TUF
     implementation, leaving a stream to be delivered to the sender MUST enable PDU alignment mode.
     This ensures that PDU alignment mode, with its favorable hardware
     characteristics, is used when possible. ULP.

4.  The specific negotiation mechanism for enabling Protocol

     This section defines the framing TUF protocol and choosing the framing mode is outside itself.  The first two
     sections are the scope core of this
     document.  However, note that framing protocol behavior is
     requested by the receiver and offered by protocol defining:

     o    the sender.  Negotiation
     will probably include exchange of:

     1. shim layer PDUs, called FPDUs,

     o    a TCP-conforming segmentation behavior which ensures the receiver's desired mode(s)

     2. ULPDU
          containment property holds under most conditions.

     The remaining sections cover other aspects of the sender's framing key if PDU alignment mode is selected

     2.   ULP packing behavior if PDU alignment mode is selected

     3. protocol which
     are primarily implications of the receiver's desired marker period if marker mode is
          selected

     4. core protocol:

     o    what ULP-specified negotiations to enable TUF must accomplish,

     o    how receivers can process received TCP segments to establish
          whether the receiver's desired maximum PDU size if marker mode is
          selected

5.  PDU Alignment Mode ULPDU containment property holds.

4.1.  The framing protocol in PDU alignment mode Framing Protocol Data Unit (FPDU)

     TUF sends groups of one or more complete ULP PDUs preceded by a framing header.  This framing
     header and set of ULP PDUs is called a `framing PDU'.  The framing
     protocol in PDU alignment mode is supported by a framing-aware TCP
     whose behavior is described ULPDUs in `Framing-Aware TCP', below. a framing
     protocol data unit (FPDU).

4.1.1.  FPDU Format

     The format of a framing PDU is as follows: an FPDU is:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |          Length               |             Key               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                              Key                              |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                                                               |
     |                                                               |
     ~                                                               ~
     ~                            ULPDUs                             ~
     |                                                               |
     |                                                               |
     |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |            ULPDUs             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Length: 16 bits (unsigned integer)
          This is the length in octets of the set of framed ULPDUs.  It
          does not include the length of the FPDU header itself.

     Key: 48 bits (unsigned integer)

          This is used by the receiver to validate the ULPDU containment
          property.  It is selected at random by the sender, and
          initially signaled to the receiver in a ULP-specified way,
          before the receiver attempts to test the ULPDU containment
          property.  All FPDUs sent on the same connection in the same
          direction must use the same key value.  A good quality random
          number generator MUST be used to generate the initial key.
          RFC 1750 discusses relevant characteristics and provides
          references for good quality random number generation
          [RFC1750].

     The length of an FPDU is 8 + L octets, where L is the length of the
     set of framed ULPDUs.  The 16-bit length field is sufficient to
     permit a TCP segment with an FPDU to completely fill a maximum-size
     IPv4 or IPv6 datagram.

4.1.2.  FPDU Size Selection

     Each FPDU SHOULD contain as many contiguous, complete ULPDUs as
     will fit within the current EMSS, unless ULPDU packing is disabled.
     If ULPDU packing is disabled each FPDU SHALL contain a single
     ULPDU.  ULPDU packing mode may be negotiated, or specified a priori
     by a ULP.  Disabling ULPDU packing is analogous to disabling the
     Nagle algorithm in TCP.

     TUF SHALL present the size of the largest ULPDU size fitting in an
     EMSS-sized FPDU (MULPDU) to the ULP.  MULPDU is EMSS - the FPDU
     header size (8 octets).  ULPs SHOULD submit as large ULPDUs as
     possible to TUF, up to MULPDU, subject to limits imposed by
     specific ULP properties.  The ULP MAY also chose to pack several
     ULPDUs into an EMSS-sized unit before submitting them as one ULPDU
     to TUF.  Depending upon the ULP, ULP packing may improve data
     transfer efficiency, and is unlikely to have any detrimental
     effect.

     A TUF implementation probing for PMTU increase SHOULD present an
     increased MULPDU value to the ULP PDUs                            ~
     |                                                               |
     |                                                               |
     |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     | until a large enough FPDU to
     perform the probe results.

     Under exceptional circumstances, the EMSS can become too small to
     accommodate even a single ULPDU.  For example, a ULP may define
     fixed-sized PDUs            |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     The "Length" field that are incompressible, or variable size PDUs
     with some absolute minimum size, such as the size of a data PDU
     containing a minimum amount of data.  It is 16 bits and contains possible for the length in EMSS
     to shrink to as small as 8 octets [PathMTU].  If the EMSS is too
     small to accommodate an incompressible ULPDU, the FPDU MUST contain
     only that ULPDU.  ULPs using TUF SHOULD NOT define ULPDUs with a
     minimum size greater than 128 octets.

4.2.  TUF-conforming TCP Sender Segmentation

     TCP senders are allowed substantial freedom in the choice of how to
     segment an outgoing TCP stream.  Within the set confines of framed ULP PDUs, excluding the framing header.

     The "Key" field is 48 bits
     receiver-advertised receive window, and is selected at random by the sender,
     and signalled sender computed
     congestion window, any segmentation is permitted.  Virtually all
     TCP implementations do attempt to segment outgoing TCP streams into
     EMSS-sized segments where possible because it improves performance.

     TUF-conforming TCP sender behavior ensures that the receiver in a ULP-specified way.  All framing
     PDUs sent on ULPDU
     containment property holds most of the same connection time.  To do this, a TUF-
     conforming TCP sender MUST respect a single additional rule in the same direction must use the
     same key value.
     performing segmentation:

          A good quality random number generator TUF-conforming TCP sender MUST be
     used to generate segment the initial key.  RFC 1750 discusses relevant
     characteristics and provides references for good quality random
     number generation [RFC1750].

     The length outgoing TCP
          stream such that the first octet of every FPDU is sent at the framing PDU in octets will be 8 + L, where L
          beginning of a TCP segment

4.3.  Negotiating TUF

     Negotiating the use of TUF is the length responsibility of the set ULP.  The
     use of framed ULP PDUs.

     Whether more than one ULP PDU may TUF MAY be packed into negotiated separately for each direction on a single framing
     PDU
     connection.  The negotiation procedure MUST ensure that when TUF is a controllable option of
     enabled or disabled, the framing module in PDU alignment
     mode.  Some receivers may choose to expect exactly one ULP PDU per remote peer will not transmit its first
     TCP segment when framing in the new mode until it is behaving nominally.  The sender MUST
     NOT pack more than one ULP PDU into a framing PDU if this behavior certain that the local peer
     has actually enabled or disabled TUF.

     TUF operation is desired characteristically requested by the receiver.  ULP packing behavior may be negotiated
     or specified priori receiver and
     offered by the ULP.

5.1.  Framing-aware TCP

     A framing-aware TCP SHALL send one complete framing PDU per TCP
     segment whenever possible.  Cases when it may not sender.  Before enabling TUF, the relevant
     parameters:

     1.   the sender's 48-bit key

     2.   ULPDU packing mode

     MUST be possible established at each peer.

     A natural way to
     send enable the use of TUF is a complete framing PDU ULP-defined negotiation
     exchange of the TUF parameters culminating in enabling TUF, if
     requested, for each TCP segment are described in
     `PDU Alignment Mode Exception Cases', below. transfer direction.  A framing-aware TCP MUST NOT send any TCP segment containing octets
     from more than one sending operation.  In other words, three-way handshake
     protocol can be used to ensure that the boundary
     between data of consecutive sending operations MUST occur between
     TCP segments.  By following point at which TUF is
     enabled is unambiguous and each end has time to perform local state
     changes.  A connection on which TUF is enabled is likely to be the
     same connection on which the negotiation occurs, but this rule, is not
     required.  A new connection could also use TUF from its initial
     establishment, if the sender guarantees TUF parameters and modes are known through
     some out-of-band mechanism.

     Use of TUF could be disabled during a connection using a similar
     ULP-defined three-way handshake.

     Other alternatives to parameter exchange include stipulating some
     parameters a priori.  For example, a ULP could specify that TUF
     with ULPDU packing enabled is always used in both directions.  In
     this case, only the event an exception causes PDU alignment 48-bit keys need to be lost
     temporarily, it will be regained as soon as possible.

     The use of oversize TCP segments sent by means of IP fragmentation exchanged before TUF is discouraged due to the limited size of the IP header
     Identification field and
     enabled.  Or, a ULP could determine TUF characteristics on the potential for undetected errors due to
     wrapping
     basis of the Identification value.  Framing-aware TCP
     implementations SHOULD resegment at the TCP layer according port number.

4.4.  TUF Receiver ULPDU Containment Property Testing

     A TUF receiver that wishes to use ULP control information to
     perform direct data placement must first verify the
     rule given in ULPDU
     containment property.  To do this, the previous paragraph when necessary to meet
     requirements of receiver MUST establish that
     the current maximum TCP segment size for a path.  In contains exactly one FPDU.  Abstractly, this document, EMSS means can be
     done by assuming the current TCP maximum segment size used
     for sending segments on a connection, which is initially negotiated
     during the connection handshake, payload begins with an FPDU, and subsequently adjusted by path
     maximum transfer unit (PMTU) discovery behavior [PathMTU].

     A framing-aware TCP must notify
     verifying the framing module following properties of changes in
     the EMSS. that putative FPDU:

     o    The framing module must be able to retrieve the EMSS
     from the framing-aware TCP.

     If the framing-aware TCP chooses to probe for path MTU increase
     using received TCP segment larger than payload length equals the path MTU, FPDU length
          plus the framing-aware TCP
     MUST report an appropriate EMSS increase. length of the FPDU header (8 octets).

     o    The candidate path MTU
     will only be probed 48-bit key equals the value signaled to the receiver when
          TUF was enabled for the framing protocol submits a framing PDU
     larger than connection.

     If these conditions are true, the current EMSS.  Immediately following TUF receiver MAY assume that the probing
     segment,
     ULPDU containment property holds, and use ULP control information
     to directly place data in the framing-aware TCP MUST reduce EMSS contained ULPDUs.

     TUF DOES NOT provide any information that a TUF receiver can use to its previous
     value until
     locate ULP control information beyond the candidate path MTU is confirmed.

     Probing for path MTU increase is optional [PathMTU], and ULPDU containment
     property.  In particular, a framing-
     aware TUF receiver MUST NOT scan TCP might elect not segments
     in an attempt to locate FPDUs that do so unless the EMSS becomes
     `inconveniently' small.  By not probing for path MTU increase when begin at the current EMSS provides adequate performance, beginning of
     a TCP segment.  However, even if the framing
     protocol will ULPDU containment property
     does not send the potentially unaligned PDUs that would hold, a TUF receiver may still be
     used to probe path MTU.

     Although framing-aware TCP is defined specifically able to support the
     framing protocol in reliably locate
     and use ULP alignment mode, it may be used by other
     clients, assuming framing validation is provided by some means. control information.  For example, as discussed below in `Security Considerations', a
     framing-aware TLS could use if a framing-aware received TCP directly without
     adding framing PDU headers, because TLS validation can serve
     segment contains the
     same purpose, and actually provides stronger framing validations
     guarantees than a framing PDU header.

5.2.  PDU Alignment Mode Exception Cases

     Although next unreceived data in the framing-aware TCP sender should place exactly one
     framing PDU stream, the
     location of ULPDUs in each TCP that segment there are exceptions when this unambiguous.  The behavior
     of a TUF receiver acting on ULP control information located with
     properties other than the ULPDU containment property is not possible.  These exceptions include
     specified here.

5.  Protocol Characteristics

     This section discusses some characteristics and behavior which are
     implications of the following.

     1. TUF protocol.

5.1.  Properties Of TUF-conforming TCP Senders

     The connection is in emergency mode and EMSS general practice of TCP senders to send as much data as
     possible within a TCP segment (up to EMSS) implies that an FPDU
     whose size is less than 512
          octets.

     2.   The EMSS has been reduced.  This or equal to EMSS, and whose first octet
     begins a TCP segment will result in be sent entirely within a window
          during which single TCP
     segment.  This ensures the ULP is not yet aware ULPDU containment property for that TCP
     segment.

     A TUF-conforming TCP sender still obeys all requirements of TCP.
     While the reduced EMSS.
          Since some framing PDUs may already segmentation of a TUF-conforming TCP sender will have been sent and
          possibly lost prior to being received,
     distinctive characteristics when viewed from the network wire, the
     same framing PDUs
          must be resent, if necessary, but in smaller segmentation behavior could also result from a stock TCP segments
          which conform to the new EMSS.

     3.
     sender.

     The remote end one property of a TUF-conforming TCP sender which arguably
     departs from traditional expectations is advertising that a window smaller than the EMSS.
          If both ends manage their window TUF-conforming TCP
     sender may not produce TCP segments which are as required close in RFC-1122
          [RFC1122], and size to
     EMSS as a reasonable amount of receive buffering is
          available, this case should not occur, but the sender, for
          robustness, must tolerate this.

     4.   The sender is probing an advertised window of zero.

     5. stock TCP sender.  The sender is probing need to determine if ensure the path MTU can be
          increased.

     In addition, there is another case ULPDU
     containment property may result in which the receiver will
     receive framing PDUs TCP segments which are not aligned with TCP segments.

     6.   There as
     full as if the property did not need to hold.  While this is a middle-box
     abstractly true, in the connection practice, several characteristics combine to
     minimize this effect.  Specifically:

     o    Packing ULPDUs into FPDUs gives behavior similar to that of
          stock TCP segmentation, albeit with coarser granularity.

     o    ULPs which benefit from data-dependent direct data placement
          (candidates for TUF) usually transfer large amounts of data in
          bulk.  This means that most ULPDUs are data-carrying, and will
          be EMSS-sized.  Even when control is resegmenting interleaved with data,
          the combination of a small number of control ULPDUs with a
          data ULPDU can be packed to fill an EMSS-sized segment.

     Therefore, a TUF-conforming TCP sender seems likely to behave
     similarly to a stock TCP data stream.

     If the framing protocol in PDU alignment mode must sender under most circumstances.  However,
     applications that both send an
     unaligned framing PDU, it SHALL take one of the following actions.

     1.   Send and receive data over the framing PDU same TCP
     connection, where there might be dependencies between incoming and
     outgoing data, are often subject to excessive delays attributable
     to TCP's Nagle algorithm and/or delayed-ACK algorithm [NagleDAck].
     These algorithms generally perform best when TCP always sends full-
     EMSS segments.  Because TUF can generate sub-EMSS segments as a single by-
     product of aligning FPDU boundaries with TCP segment using IP
          fragmentation.  While this behavior is discouraged, it is not
          prohibited by boundaries,
     TUF might be especially vulnerable to the framing protocol, or any other applicable
          RFCs.

     2.   Send known problems with the framing PDU as several TCP segments,
     Nagle and/or delayed-ACK algorithms.

     Further work, including implementation experience with each
          segment guaranteed not to appear TUF, as a well-formed, complete
          framing PDU on its own, at well
     as existing and future proposals for improvements to the time Nagle
     and/or delayed-ACK algorithms, might be necessary to optimize TUF
     performance while fully preserving the segment congestion-avoidance
     features of TCP.  This work is sent.  That
          is, currently outside the sender SHALL ensure that one scope of the following this
     document.

5.2.  Exception Cases

     The complete operational specification of TUF is true
          for every segment with a partial framing PDU:

          A.   octets 0-1 do not equal the segment length minus 8

          B.   octets 2-8 do not match the framing key value

          C. contained in the total segment length is less than
     rules for forming FPDUs, and sending those FPDUs in TCP segments.
     However, the framing PDU
               header operation of 8 octets

     These mechanisms ensure that the receiver TUF will not falsely
     misinterpret any piece of a framing PDU sent in several segments as be subject to a complete, valid framing PDU.  However if the TCP data stream variety of
     transient or exceptional conditions.  The behavior of TUF under
     those conditions is
     subjected discussed below to resegmenting by a middle-box, the sender may no longer
     control segmentation illustrate specifically how
     TUF addresses them.

5.2.1.  Resegmenting Intermediaries

     Resegmenting TCP-layer intermediaries (middleboxes) are one of received data.  In this case the framing
     protocol must rely on probability the
     most formidable obstacles to ensure that segments maintaining the ULPDU containment
     property.  In the presence of such an intermediary, the
     resegmented data stream will
     segmentation chosen by the sender may not appear as valid, complete framing
     PDUs, if they are not.

     In be the case where segmentation at
     the receiver detects a continuous stream of TCP
     segments which do receiver.  While such intermediaries may or may not contain complete framing PDUs, be common
     in particular networks, in many cases the ULP SHOULD
     disable use presence or absence of
     such resegmenting behavior is beyond the framing protocol, control or switch to marker mode if
     the ULP provides a means even knowledge
     of doing this, and the end points so
     choose.  Such a continuous stream of improperly framed TCP segments
     implies the presence of a resegmenting middle-box.  Such a
     detection process SHOULD NOT mistake a temporary sequence of
     improperly framed TCP segments resulting from an EMSS change with using TUF.  Therefore, TUF must detect such
     resegmentation by design.

     A primary reason for the presence of a resegmenting middle-box

5.3.  Validity Of Framing-aware TCP Segmentation

     A framing-aware TCP normally sends exactly one framing PDU per TCP
     segment.  This may therefore result in more segments being sent
     than would occur random key in a traditional TCP.  However, the framing module FPDU
     header is allowed to pack multiple ULP PDUs into a single framing PDU if
     ULP packing is enabled, which will give behavior approaching that
     of a traditional TCP.  Even with ULP packing disabled, detect such resegmentation.  An alternative to the behavior
     of a framing-aware TCP effectively corresponds
     random key which has been proposed, is to that of a
     traditional TCP sender with use ULP-specific
     validation criteria to determine the Nagle algorithm disabled (i.e.
     TCP_NODELAY), ULPDU containment property.
     For example, some ULP PDUs include relatively strong data integrity
     checks such as CRCs, and this is considered acceptable behavior.

     Framing-aware TCPs still respect congestion other ULP control windows, which
     are maintained as a octet count information can often be
     validated against various ULP-specific criteria.

     While such ULP-specific validation criteria may involve checking
     many more bits than the combination of the FPDU's 16-bit length and
     48-bit key, ULP-specific validation criteria may not as a segment count.

     On retransmission, actually offer
     a framing-aware TCP respects strong guarantee of the original stream
     segmentation.  This is allowed by RFC1122 [RFC1122], section
     4.2.2.15.

5.4.  Receiving In PDU Alignment Mode

     Because each framing PDU contains sufficient information to
     determine its length, ULPDU containment property.  For certain
     data streams, the beginning probability of a false-positive indication of the next framing PDU can be
     determined.  Therefore each successive PDU
     ULPDU containment property can be recovered.

     Conventional extremely high.

     Assume that the intermediary resegments to a granularity of no
     finer than G octets (e.g. 4).  Also assume that the TCP implementations will pass received data to stream
     contains predominantly application data.  If the ULP
     in order, so framing is easily recovered by the ULP.

     Special receive implementations a storage
     protocol, simply transferring a file containing a continuous,
     repeated stream of well-formed ULPDUs which exploit PDU alignment mode,
     typically found are some multiple of G
     in direct placement network interfaces, may allow size increases the ULP to do direct data placement on TCP segments received out probability of
     order.  The receiving end can safely assume that a framing PDU is
     exactly contained within TCP segment payload if false-positive indication of
     the following
     conditions ULPDU containment property to approximately:

             1 / (sizeof(repeated ULPDU)/G)

     If the well-formed ULPDUs are met.

     1.   Standard TCP processing indicates that this is relatively small (e.g. 32 octets
     where G=4 octets), the probability of a valid, in-
          window segment.

     2.   The payload false-positive indication
     of the ULPDU containment property is approximately 1/8, for EACH
     TCP segment, parsed as segment which does not actually begin with a framing PDU, has ULPDU.  Clearly,
     in this case, it would take only a
          length field which equals the very small number of TCP segment length minus 8, and
          a key field
     segments which matches do not begin with an actual ULPDU before the expected key for `fake'
     ULPDU in the framing
          protocol connection. application data is interpreted as an actual ULPDU.
     The framing protocol passes the contained ULP PDUs to consequences of such a ULP parser.
     The ULP parser performs direct placement false-positive interpretation could be
     dire, for example executing a destructive operation request.

     The 48-bit random key in the FPDU results in a low probability of a
     false-positive indication of the ULPDU containment property because
     it is effectively secret with respect to the PDUs.  The ULP
     parser MUST NOT execute application data
     stream.

     Note that although this analysis may appear to be security-minded,
     prompting the ULP protocol (i.e. none image of a sighted third-party adversary that can
     `sniff' the ULP
     protocol state variables change), until all preceding octets in the
     TCP stream have also been received.

6.  Marker Mode 48-bit key, it is actually considering a safety, rather
     than a security property.  The framing protocol in marker mode inserts framing markers security properties of TUF are
     discussed in Section 6 (`Security Considerations') below.

     Even though TUF can detect the
     TCP octet stream at presence of a period agreed upon by the framing protocol
     sender and receiver.  Each framing marker points to resegmenting
     intermediary, such an intermediary will almost certainly
     substantially reduce the next PDU in chance of the TCP octet stream.  Marker insertion in ULPDU containment property
     being satisfied.  A TUF implementation which detects a very low
     incidence of the TCP octet stream ULPDU containment property for a sustained
     interval (>> RTT) may assume that a resegmenting intermediary is
     not synchronized in any way with
     operation and SHOULD discontinue the ULP.  The ULP may use PDUs of
     any size up to 2^16-8-(4 * # of markers inserted) (determined by
     marker interval).  Markers will be inserted in ULP control information
     found using the resulting octet
     stream, possibly interrupting PDUs, as necessary to maintain ULPDU containment property.  In such cases, the
     interval.  Although ULP
     MAY elect to disable the placement use of each marker TUF altogether, or simply just stop
     exploiting the ULPDU containment property.

5.2.2.  PMTU Reduction

     When a PMTU reduction is not detected by a function
     of TUF-compliant TCP, the ULP PDU boundaries, TUF-
     compliant TCP sender may send FPDUs already committed to the contents TCP
     layer in one of each marker are.

     The format two ways:

     o    send unsegmented FPDUs in TCP segments of a framing marker is as follows:

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |        Next PDU Offset        |        Next PDU Offset        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     The "Next PDU Offset" contains the offset old EMSS size,
          and rely on IP fragmentation to deliver the next PDU, segments,
     o    segment FPDUs to fit in
     octets, from the end of TCP segments which respect the marker.

     The "Next PDU Offset" occurs twice new
          EMSS size.

     Stock TCPs face a similar choice on PMTU change, and both
     alternatives are used in practice.

     In the marker to guarantee case that
     when a marker is split across TUF-compliant TCP segments, chooses to segment FPDUs, it
     SHOULD segment them in such a complete copy of Next
     PDU Offset occurs way that, in at least one of the two TCP segments.

     The framing protocol receiver must remove (or otherwise ignore) absence of
     resegmentation by an intermediary, the
     periodic markers in segments are guaranteed not
     to give a false-positive indication of the received TCP octet stream ULPDU containment
     property.  There are various ways to reconstruct ensure this.  For example, no
     matter how the PDUs from FPDU is segmented, the sender.

     The first marker SHALL be sent in segment is guaranteed
     not to give a false-positive indication of the ULPDU containment
     property---the 48-bit key will match, but the length will not.  In
     the worst possible case, each subsequent TCP octet stream preceding
     any framed PDUs.  This first marker will, necessarily, have segment could be sent
     with fewer than 8 octets of data, also guaranteed not to give a Next
     PDU Pointer
     false-positive indication of 0.  The first marker corresponds the ULPDU containment property.  More
     efficient approaches are possible, but PMTU reduction is a rare
     event, and reacting to it is only a transient condition.
     Eventually a new MULPDU will be presented to the point ULP, and FPDUs
     that fit in the new EMSS will result.  During the transient
     condition, performance will suffer temporarily no matter how FPDUs
     are segmented.

     No matter what segmentation is chosen by a TUF-compliant TCP octet stream sender
     when segmenting an FPDU, if the framing protocol is enabled.

7.  Security Considerations

7.1.  Security Protocol Interactions

     The ULP framing protocol may be layered on top segments pass through a
     resegmenting intermediary, the correctness of IPSec, or TLS.  A
     direct placement network interface which supports connections
     secured with IPSec or TLS must directly implement security protocol
     processing as well as framing and direct placement support.

7.2.  Using IPSec With the ULPDU containment
     property remains strictly a matter of probability.

5.2.3.  PMTU Increase

     As described in `FPDU Size Selection' above, a TUF-compliant TCP
     probing for PMTU increase will present an increased MULPDU value to
     the ULP.  This should eventually lead to an FPDU large enough to
     actually perform the PMTU increase probe.  The Framing Protocol

     Since IPSec MULPDU value should
     not be further adjusted until the probe is designed actually performed.
     This behavior is similar to secure arbitrary IP packet streams,
     including streams where packets are lost, when a stock TCP would like to perform
     a PMTU increase, but less data is available than would fill the framing protocol
     could run cleanly
     desired segment.

     Also, note that depending on top the ULP, the actual distribution of IPSec without any change.

     Using IPSec end-to-end
     FPDU sizes may have a granularity coarser than a single octet.  An
     FPDU with the framing protocol in PDU alignment
     mode permits an optimization to the framing protocol.  Because
     IPSec validation criteria guarantee particular, desired TCP segment size may never be
     generated.  Therefore when probing for PMTU increase, a TUF-
     compliant TCP must be satisfied with an FPDU that IP packets received are
     equivalent produces a TCP
     segment size that is `close' to the IP packets sent, it is desired size.

     Finally, note that in cases where PMTU grows and shrinks relatively
     frequently, better performance may result from not possible probing for an
     intermediary to resegment PMTU
     increase at all, or probing very rarely.  This is because the
     performance disruption resulting from PMTU decrease can be
     substantial, and in many cases, implementations of TUF will be in
     hardware, so performance may less sensitive to differences in PMTU.

5.2.4.  Receive Window < EMSS

     A TUF-compliant TCP stream.  If IP fragmentation
     (rather than resegmenting) sender that is used presented with a receive window
     smaller than EMSS may be required to send committed data when segment FPDUs.  The TCP window
     probe is a limiting case of this condition where the
     EMSS changes, advertised
     receive window is 0, and the framing PDU validation header amount of data typically sent in
     response is not needed. a single octet.

     In this case, a ULP may run directly on top of a framing-aware TCP.

7.3.  Using TLS With The Framing Protocol

     Using TLS with TUF-compliant TCP sender will segment in accordance
     to the framing protocol is more complicated than using
     IPSec.  The combination requirements of TLS TCP, and the framing protocol must still
     provide a modest bound on reassembly buffer size rule defined in `TUF-conforming
     TCP Sender Segmentation' above.  In addition, as when resegmenting
     in response to be useful.

     TLS is PMTU decrease, a TUF-compliant TCP sender SHOULD
     segment in such a record-oriented protocol. TLS records are PDUs just like
     those used by ULPs that permit direct placement.  As with other
     ULPs, the only way that, in the absence of a resegmenting
     intermediary, segments are guaranteed not to avoid give a complete reassembly buffer false-positive
     indication of the ULPDU containment property.  In situations where
     the receive window is smaller than EMSS, data transfer performance
     is likely to be
     able to find TLS PDUs in the presence limited independently of lost any segmentation behavior
     by the TCP segments.
     Therefore, sender.  Furthermore, ULP implementations that choose to permit direct placement of ULPs secured with TLS, TLS
     should also
     use TUF will almost certainly be treated as a protocol which uses framing support.

     Using the framing protocol with TLS requires modification of a TLS
     implementation for the combination designed to perform effectively.
     Essentially, maintain a TLS implementation must become receiver
     window larger than EMSS, so a client small receiver window should occur
     extremely infrequently.

5.2.5.  Size of ULPDU + 8 > EMSS

     In cases where EMSS shrinks below the
     framing protocol.

     TLS provides minimum size of a similar interface ULPDU that
     a ULP wants to send, TUF will create FPDUs that are larger than
     EMSS, and a TUF-compliant TCP for sending protocol data.
     Protocol data submitted to sender will face the TLS same
     alternatives as during PMTU reduction:

     o    send interface may be coalesced
     with other protocol data unsegmented FPDUs and rely on IP fragmentation to deliver
          the segments

     o    segment FPDUs to fit in a single TLS PDU, or it may be
     segmented arbitrarily across more than one TLS PDU.  For TCP segments which respect the EMSS
          size

     A ULP which is presented with an MULPDU value that is too small to
     accommodate PDUs necessary operation SHOULD simply attempt to use
     ULPDUs which are as small as possible

     If the
     framing protocol in EMSS shrinks to properly support direct placement with TLS, a framing-aware TLS MUST provide pathologically small size, then a framing-aware interface to the
     ULP similar to TUF
     implementation SHOULD discontinue the one described in Appendix A.

     This layering looks like:

                             Framing use of ULP client
                                      |
                                      V
                         TLS-capable framing module
                                      |
                                      V
                              Framing-aware TLS
                                      |
                                      V
                               Framing module
                                      |
                                      V
                        TCP (possibly framing-aware)
                                      |
                                      V
                                    . . .

     Although some framing control
     information may be exposed in found using the clear when
     running TLS on ULPDU containment property.  In such
     cases, the framing protocol, this information does not add
     to what is already available ULP MAY elect to an attacker.  Framing only conveys disable the location use of TLS PDUs, TUF altogether, or
     simply just stop exploiting the ULPDU containment property.

     A path MTU which are already available results in an EMSS < 128 + 8 octets is an
     extremely unlikely occurrence and when it does occur, poor data
     transfer performance is a likely result, independent of TCP sender
     segmentation behavior.

6.  Security Considerations

     This section discusses both protocol-specific considerations and
     the clear.

     Unfortunately, ciphers defined for use implications of using TUF with TLS do not offer existing security mechanisms.

6.1.  Protocol-specific Security Considerations

     A third-party that can inject spoofed packets into the
     same independence network
     which can be delivered to a TUF receiver could launch a variety of TLS PDUs
     attacks that IPSec provides for IP datagrams. exploit TUF-specific behavior.  For one thing, TLS supports the use of stream ciphers, example a blind
     third-party adversary could inject random packets which IPSec
     does not.  Stream ciphers typically have dependencies reaching far
     back appear in
     the data stream for deciphering at the current point.
     Therefore it is probably valid TCP window and do not appropriate to negotiate the use of a
     stream cipher when securing the framing protocol.

     Block ciphers defined for use with TLS have similar properties to
     those defined for use begin with IPSec.  Specifically, they all operate
     in Cipher Block Chaining (CBC) mode.  However, while IPSec provides
     a CBC initialization vector for each IP datagram, TLS defines only valid FPDU headers.  A
     barrage of such packets might cause a single CBC initialization vector for use in TUF receiver to conclude that
     a resegmenting intermediary is present and disable the first block.  All
     subsequent blocks use the cipher-text of their predecessor.  To
     decipher the current TLS PDU, the final cipher-text block from the
     previous TLS PDU must be available.  Typically, block ciphers
     defined for use with TLS have an 8-octet block size. TUF
     and direct data placement.  This implies
     that for would substantially degrade
     performance.  However, it would probably also have more dire
     consequences than performance, such as causing the ULP direct placement to be possible with TLS, interpret
     the bogus data from as valid.  Furthermore, such a
     preceding TCP segment may be needed, where it is not when using third-party could
     also degrade performance just as effectively in a TUF-independent
     way by injecting spoofed ICMP packets which result in reduction of
     the
     framing protocol without TLS.  Note that if path MTU to an inefficiently small size.

     Fundamentally, the preceding vulnerabilities of TUF to active third-party
     interference are no more acute than to TCP
     segment without TUF.  In both
     cases, a communication security mechanism such as IPSec is missing, all cipher blocks within the current TCP
     segment may still be processed except the first one (assuming the
     bounds of the TLS PDU is known).

7.3.1. only
     way to completely prevent such attacks.

6.2.  Using TLS In PDU Alignment Mode

     To IPSec With TUF

     Since IPSec is designed to secure arbitrary IP packet streams,
     including streams where packets are lost, TUF can run the framing protocol running cleanly on TLS in PDU alignment mode,
     an integral number
     top of TLS PDUs IPSec without any change.  IPSec packets may be sent decrypted in each TCP segment
     the
     same way ULP PDUs order they are sent in received, and a TUF receiver may test and
     exploit the absence of TLS.  A framing-aware
     TLS would use ULPDU containment property just as if the framing-aware TCP.  In this case, IP datagram
     were unsecured.

6.3.  Using TLS With TUF

     Using TLS [TLS] with TUF, particularly trying to exploit the role ULPDU
     containment property to locate ULP control information, is not a
     straightforward process.  TUF can be directly layered on top of the
     framing PDU header in detecting unexpected modification
     TLS, but many of TCP
     segmentation is subsumed by the strong integrity checks performed
     on advantages of TUF are lost.  This document
     does not define a way of using TLS PDUs.  There with TUF that could offer better
     performance than stock reassembly buffer-based implementations.
     That task is no need left to encapsulate TLS PDUs in a framing
     PDU.  In fact, different document, if there is sufficient
     motivation to address the vulnerability problems.  This section does outlines
     some of the framing key known complications of trying to active
     attack is eliminated by do better than stock
     reassembly buffer-based implementations using TLS validation algorithms instead.

     Use of with TUF.

     TLS is a non-null record-oriented protocol.  TLS compression algorithm may interact badly records are PDUs with a framing-aware TLS implementation.  A TLS compression algorithm is
     allowed to increase content length by up
     similar structure to 1024, which may result ULPDUs defined in application ULPs.  As with
     other ULPs, the compressed TLS PDU no longer fitting within EMSS.
     Therefore, only TLS compression algorithms which are known not way to
     increase content length, or increase content length by avoid a small,
     manageable amount, should complete reassembly buffer is
     to be selected.

     The need able to receive the previous TCP segment before completing find TLS
     processing PDUs in the presence of current lost TCP segment means segments.
     The ULPDU containment property could be used to do this, which
     suggests that using the framing
     protocol in PDU alignment mode with TLS will require some high-
     speed receive packet buffer memory.  This defeats one itself should be layered on top of TUF.  In this
     case, the
     primary advantages FPDU header will travel in the clear, but this will
     probably not present serious vulnerabilities other than denial of PDU alignment mode.  Therefore, while it
     service attacks comparable to what is already possible to use without TUF.

     Once the TLS records are located and processed it still remains to secure
     locate the ULPDUs.  The simplest way to do this would be to have
     the TLS implementation be TUF-compliant, and ensure the ULPDU
     containment property within each TLS record.  In this case, the framing
     protocol in PDU alignment
     mode, IPSec layering would be look like:

                         ULP client
                              ^
                              |
                              | ULPDUs (in octet stream)
                              |
                              v
                      TUF-conforming TLS
                              ^
                              |
                              | TLS records (containing ULPDUs)
                              |
                              v
                             TUF
                              ^
                              |
                              | FPDUs (each containing a more appropriate choice TLS record)
                              |
                              v
                     TUF-conforming TCP
                              ^
                              |
                              | TCP Segments (each containing an FPDU)
                              |
                              v
                            . . .

     An obvious complications of using TLS with TUF is that ciphers
     defined for securing PDU
     alignment mode connections because it does use with TLS do not require any
     reassembly buffer memory.

7.3.2.  Using offer independence across TLS In Marker Mode

     To use
     records.  The most common cipher used with TLS on is RC4, which is a framing protocol connection in marker mode, the TCP
     stream must actually contain two, independent sets cipher.  Efficient decryption of periodic
     markers.  Clear-text markers in the TLS PDU an RC4 stream will permit TLS
     PDUs depends upon
     the entire preceding data stream.  In other words, it is simply not
     feasible to be found decrypt TLS records encrypted with RC4 in any order
     other than the presence of lost TCP segments.  Once a
     portion of the original, clear-text TCP stream is recovered by order.  This clearly defeats the purpose
     of TUF.

     TLS
     processing, markers is also defined to work with block ciphers such as 3DES in
     Cipher Block Chaining (CBC) mode.  In this case, the original octet stream are used to find
     ULP PDUs and perform direct placement.

7.4.  Other Security Considerations
     The modification dependency of
     the sender's TCP segmentation algorithm decryption operation on data in PDU
     alignment mode does not open any new attacks, since: 1) the
     segmentation algorithm previous TLS records is not based on input from less
     severe.  To decrypt the network, 2) current TLS record only requires ciphertext
     from the segmentation algorithm may pack small ULP PDUs into previous TLS record.  While this does not allow complete
     independence of processing TLS records, a single lost or delayed TCP
     segment so it containing a TLS record only prevents decrypting the
     immediately subsequent TLS record, not all TLS records after it.

     TLS compression presents another complication to using TLS with
     TUF.  TLS compression algorithms are allowed to increase the
     content length by up to 1024 octets.  If the content length does
     increase, the TLS record may not open packet flooding attacks.

     If an attacker can send fit within an in-window EMSS-sized TCP segment that is accepted,
     on an unsecured framing protocol connection
     segment, even if the attacker can
     probably force uncompressed ULPDU does.  If the risk of
     exceeding an EMSS-sized TCP receiver in segment is small, it may be acceptable
     to a framing protocol exception
     path, degrading service. However, such an attacker can also place
     arbitrary data into the stream, so merely forcing occasionally send FPDUs containing TLS records that span several
     TCP segments, or use IP fragmentation.  Some TLS compression
     algorithms may never increase the receiver on
     to an exception path is not a compelling attack.

8. content length, or only increase
     it by some small, manageable amount.

7.  IANA Considerations

     If framing is enabled a priori for a ULP by connecting to a well-
     known port, this well-known port would be registered for the framed
     ULP with IANA.

9.

8.  References

     [ALF]
          D. D. Clark and D. L. Tennenhouse, "Architectural
          considerations for a new generation of protocols," in SIGCOMM
          Symposium on Communications Architectures and Protocols ,
          (Philadelphia, Pennsylvania), pp. 200--208, IEEE, Sept. 1990.
          Computer Communications Review, Vol. 20(4), Sept. 1990.

     [SOCKS]
          Leech,

     [BEEP]
          Rose, M., "The Blocks Extensible Exchange Protocol Core", RFC
          3080, March 2001.

     [HTTP]
          Fielding, R. and others, "SOCKS "Hypertext Transfer Protocol Version 5," --
          HTTP/1.1.", RFC 1928,
          April 1996

     [RFC0879]
          Postel, 2616, June 1999.
          http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-
          initwin-00.txt.

     [NagleDAck]
          Minshall G., Mogul, J., "TCP Maximum Segment Size And Related Topics", RFC
          879, November 1983

     [RFC1112]
          Braden, R., ed., "Requirements for Saito, Y., Verghese, B., "Application
          performance pitfalls and TCP's Nagle algorithm", Workshop on
          Internet Hosts --
          Communications Layers", RFC 1122, October 1989 Server Performance, May 1999.

     [PathMTU]
          Mogul, J., and Deering, S., "Path MTU Discovery", RFC 1191,
          November 1990 1990.

     [RFC1750]
          Eastlake, D., Crocker, S., Schiller., J., "Randomness
          Recommendations for Security.", RFC 1750, December 1994 1994.

     [RFC2581]
          Allman, M. M., and others, "TCP Congestion Control," RFC 2581,
          April 1999 1999.

     [SCTP]
          Stewart, R.R. and others, "Stream Control Transmission
          Protocol," RFC2960, October 2000.

     [Stevens]
          Stevens, W. Richard, "Unix Network Programming Volume 1,"
          Prentice Hall, 1998, ISBN 0-13-490012-X 0-13-490012-X.

     [TCP]
          Postel, J., "Transmission Control Protocol - DARPA Internet
           Program Protocol Specification", RFC 793, September 1981 1981.

     [TLS]
          Dierks, T. and others, "The TLS Protocol, Version 1.0", RFC
          2246

     [Satran]
          Satran, J., "iSCSI - fragments, packets synchronization and
          RDMA", http://www.haifa.il.ibm.com/satran/ips/iSCSI-RDMA-
          memo.txt, July 2000.
          2246, January 1999.

Authors' Addresses

     Stephen Bailey
     Sandburst Corporation
     600 Federal Street
     Andover, MA  01810
     USA

     Phone: +1 978 689 1614
     Email: steph@sandburst.com

     Jeff Chase
     Department of Computer Science
     Duke University
     Durham, NC 27708-0129
     USA

     Phone: +1 919 660 6559
     Email: chase@cs.duke.edu

     Jim Pinkerton
     Microsoft, Inc.
     1 Microsoft Way
     Redmond, WA 98052
     USA

     EMail: jpink@microsoft.com

     Constantine Sapuntzakis
     Allyn Romanow
     Cisco Systems
     170 W Tasman Drive
     San Jose, CA 95134
     USA

     Phone: +1 408 525 5497
     EMail: csapuntz@cisco.com

     Matt Wakeley
     Agilent Technologies
     1101 Creekside Ridge 8836
     Email: allyn@cisco.com

     Constantine Sapuntzakis
     Cisco Systems
     170 W Tasman Drive
     Suite 100, M/S RH21
     Roseville,
     San Jose, CA 95661 95134
     USA

     Phone: +1 916 788 5670 408 525 5497
     EMail: matt_wakeley@agilent.com csapuntz@cisco.com

     Jim Wendt
     Hewlett Packard Corporation
     8000 Foothills Boulevard MS 5668
     Roseville, CA 95747-5668
     USA

     Phone: +1 916 785 5198
     EMail: jim_wendt@hp.com

     Jim Williams
     Emulex Corporation
     580 Main Street
     Bolton, MA 01740
     US
     USA

     Phone: +1 978 779 7224
     EMail: jim.williams@emulex.com

Appendix A. Sample Sockets Support For TUF

     The Framing Protocol sockets support for TUF described below is only a sketch.  It
     is provided as an aid to understanding TUF.  Implementing this
     interface is not a requirement for a TUF implementation.

     Other software interfaces are possible.  The described interface
     draws from the sockets interface for UDP.  The described interface
     might be natural for applications already designed to support both
     TCP and UCP, or that do network input and output in complete PDU
     units.  For applications that perform octet-at-a-time style input
     and output, an alternative interface that draws from the tradition
     of the TCP URG pointer interface (e.g. using a MSG_OOB flag to
     send()) is equally possible.  An implementation may even offer
     several different interfaces to TUF.

     That said, the sockets support sketched below might well provide
     the basis for a complete, standard interface to be described
     outside this draft.

A.1 Basic Principles

     The sockets support for the framing module TUF takes the form of a set of socket
     options which that may be set or requested to enable the appropriate
     behavior.

     A socket may be in one of three two TUF-related modes in the send
     direction:

     1.   Framing-aware   TUF-compliant TCP sender mode.  No data (FPDU headers) is
          added to the TCP octet
          stream (neither framing PDUs nor markers), stream, but each data buffer presented
          in a sending operation is to be sent atomically as
          a single according to the rules of
          TCP segment. and TUF-compliant TCP senders.  This mode provides direct
          access to a
          framing-aware TUF-compliant TCP sender for purposes such as
          implementing a
          framing-aware TLS. TUF.

     2.   Framing protocol PDU alignment   TUF sender mode.  A framing PDU  An FPDU header is added to data presented by
          an integral number of sending operations, and the resulting framing PDU FPDU is sent
          according to the rules of PDU alignment mode.

     3.   Framing protocol marker sender mode. Markers are inserted at
          fixed intervals which point
          passed to the octet past the current PDU
          submitted by a sending operation. TUF-compliant TCP sender for transmission

     A socket may be in one of two modes TUF-related mode in the receive direction:

     1.   Framing protocol PDU alignment   TUF receiver mode.  Framing PDUs  FPDUs are expected in each TCP segment.

     2.   Framing protocol marker receiver mode.  Markers are expected
          at a fixed interval in the TCP stream.

     Received TCP segments are processed as defined above.

     If a socket receiving operation is used to retrieve received data
     (as opposed to direct placement), framing PDU the data being directly placed), FPDU headers or markers are
     removed before the data is returned.

A.1

A.2 Enabling The Framing Protocol TUF
          /* Pick one a sending mode and one receiving mode */
          if (sendMode == ATOMIC)
            mode = TCP_FRAMING_SEND_ATOMIC
          else if (sendMode == ALIGN) TUF_TCP)
            mode = TCP_FRAMING_SEND_ALIGN; TUF_SEND_TCP
          else /* sendMode == MARKERS */
            mode = TCP_FRAMING_SEND_MARKERS;

          if (recvMode == ALIGN) TUF_SEND;

          mode |= TCP_FRAMING_RECV_ALIGN;
          else /* recvMode == MARKERS */
            mode |= TCP_FRAMING_RECV_MARKERS; TUF_RECEIVE;

          setsockopt (s, SOL_TCP, TCP_FRAMING_MODE, TUF_MODE, &mode, sizeof(mode));

     A framing module that does not support a requested mode MUST fail
     the setsockopt call.  Framing may be enabled on a socket before or
     after it is connected, subject to the requirements of Section 2.

A.2

A.3 Sending Data Atomically

     The standard socket sending operations, including send(), sendto(),
     sendmsg(), writev(), and others are used to send framed data units
     (ULP PDU)s with the framing protocol. ULPDUs in TUF.
     The EMSGSIZE error should be returned if the buffer passed to the
     sending operation would result in an FPDU that does not
     satisfied the size requirements defined fit in the `ULP Support For
     Framing' section above. an
     EMSS-sized TCP segment, unless oversized ULPDU errors are disabled,
     as described below.

     When the path EMSS increases, the TCP sending operation MAY return
     EMSGSIZE once to inform the client of client of the change.

A.4 Retrieving The Current EMSS or MULPDU

          getsockopt (s, SOL_TCP, TUF_MULPDU, &emss, sizeof(emss));

     If the socket is in TUF_SEND_TCP mode, this call returns the TCP
     EMSS.  If the socket is in TUF_SEND mode, the change.

A.3 Retrieving The Current EMSS

          getsockopt (s, SOL_TCP, TCP_SEND_EMSS, &emss, sizeof(emss));

     This call returns the
     maximum segment size ULPDU that can be submitted in a sending operation without fragmentation.  The number returned
     depends upon the current socket sending mode.  If the socket is in
     framing protocol PDU alignment mode, the returned EMSS is
     appropriately adjusted by the size
     requiring fragmentation of the framing header. associated FPDU.

     The number should not count any octets that go towards TCP options.  A
     framing protocol implementation which does not support PDU
     alignment mode, because the underlying TCP sender is not framing-
     aware, is not required to implement this getsockopt call.

A.4

A.5 Disabling ULP PDU ULPDU Packing

          flag = 0;
          setsockopt (s, SOL_TCP, TCP_FRAMING_PACK_PDUS, TUF_PACK_PDUS, &flag, sizeof(flag));

     This call disables the framing protocol in PDU alignment mode TUF from packing more than one ULP PDU ULPDU into a framing PDU. an
     FPDU.  By default, ULP PDU packing is enabled.

A.5 Enabling Emergency Mode

A.6 Disabling The Report of Oversized ULPDUs

          flag = 1; 0;
          setsockopt (s, SOL_TCP, TCP_FRAMING_EMERGENCY, TUF_REPORT_OVERSIZED, &flag,
                      sizeof(flag));

     This call enables emergency mode for PDU alignment mode. disables sending operations from returning EMSGSIZE in
     response to oversized ULPDUs.  It may be called at any time on a
     socket, whether connected or not, and
     whether the current EMSS is smaller than 512 octets or not.  By
     default emergency mode  It is disabled.

A.6 Setting The Sending Marker Interval

          ivl = 2048;
          setsockopt (s, SOL_TCP, TCP_FRAMING_SEND_INTERVAL, &ivl,
                      sizeof(ivl));

     This call sets the period at which markers will used to continue ULP
     operation when MULPDU is already known to be introduced too small to
     the permit
     some ULPDUs to be sent TCP octet stream.  The sending marker interval may with out segmentation.  Oversized ULPDU
     reporting can be set
     at any time, but it only has effect when sending markers is enabled
     for the socket.

A.7 Setting The Receiving Marker Interval

          ivl = 2048;
          setsockopt (s, SOL_TCP, TCP_FRAMING_RECV_INTERVAL, &ivl
                      sizeof(ivl));

     This call sets the period at which markers are expected in the
     received TCP octet stream.  The receiving marker interval may be
     set at any time, but it only has effect when receiving markers again if PMTU is
     enabled for the socket. discovered to have
     increased.

Full Copyright Statement

     Copyright (C) The Internet Society (2001). All Rights Reserved.

     This document and translations of it may be copied and furnished to
     others, and derivative works that comment on or otherwise explain
     it or assist in its implementation may be prepared, copied,
     published and distributed, in whole or in part, without restriction
     of any kind, provided that the above copyright notice and this
     paragraph are included on all such copies and derivative works.
     However, this document itself may not be modified in any way, such
     as by removing the copyright notice or references to the Internet
     Society or other Internet organizations, except as needed for the
     purpose of developing Internet standards in which case the
     procedures for copyrights defined in the Internet Standards process
     must be followed, or as required to translate it into languages
     other than English.

     The limited permissions granted above are perpetual and will not be
     revoked by the Internet Society or its successors or assigns.

     This document and the information contained herein is provided on
     an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
     ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
     IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
     THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
     WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.