[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits] [IPR]

Versions: 00 01 02 03 04 05 06 07 08 09 10 11 12 RFC 3821

IPS Working Group     M. Rajagopal, R. Bhagwat, LightSand Communications
INTERNET-DRAFT                         E. Rodriguez, Lucent Technologies
<draft-ietf-ips-fcovertcpip-02.txt>            V. Chau, Gadzoox Networks
(Expires October, 2001)                                 J. Nelson, Vixel
                                       S. Wilson, Brocade Communications
                                                    M. O'Donnell, McDATA
                                                      C. Carlson, QLogic
                                   S. Rupanagunta, Aarohi Communications
                                                       D. Fraser, Compaq
                                               M. Merhar, Pirus Networks
                                                      D. Peterson, Cisco
                                            V. Rangan, Rhapsody Networks
                                                   L. Lamers, SAN Valley

                    Fibre Channel Over TCP/IP (FCIP)

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC 2026 [1].

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as Reference
   material or to cite them other than as ``work in progress''.

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

1. Abstract

   Fibre Channel (FC) is a dominant technology used in Storage Area
   Networks (SAN). The purpose of this draft (FC over TCP/IP, FCIP) is
   to describe mechanisms that allow islands of FC SANs to be
   interconnected over IP-based networks to form a single, unified FC
   SAN fabric.  FC over TCP/IP relies on IP-based network services to
   provide the connectivity between the SAN islands over LANs, MANs, or
   WANs.  The FC over TCP/IP specification relies upon TCP for

Rajagopal, et al.                                               [Page 1]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

   congestion control and management and upon both TCP and FC for data
   error and data loss recovery.  FC over TCP/IP treats all classes of
   FC frames the same -- as datagrams.

2. Conventions used in this document

   The key words "MUST", "MUST NOT",  "REQUIRED",  "SHALL",  "SHALL
   in this document are to be interpreted as described in RFC 2119 [2].

3. Motivation and Objectives

   Fibre Channel (FC) is a gigabit speed networking technology primarily
   used for Storage Area Networking (SAN).  FC is standardized under
   American National Standard for Information Systems of the National
   Committee for Information Technology Standards (ANSI-NCITS) and has
   specified a number of documents describing its protocols, operations,
   and services [13].

   The motivation behind connecting remote sites include disk or tape
   backup and live mirroring, or simply distance extension between two
   or more FC Switch clusters (SAN islands) or two or more FC devices.

   The first fundamental assumption made in this specification is that
   the Fibre Channel traffic is carried over the IP network in such a
   manner that the Fibre Channel fabric and all Fibre Channel devices on
   the fabric are unaware of the fact. This means that the FC datagrams
   MUST be delivered in such time as to comply with existing Fibre
   Channel specifications.  The FC traffic may span LANs, MANs and WANs,
   so long as this fundamental assumption is adhered to.

   The second fundamental assumption made in this specification is that
   all Fibre Channel frames, regardless of whether the frames contain
   errors or not, are carried over the IP network. Thus, an FC frame
   which contains a bad CRC will be encapsulated and delivered to the
   receiving endpoint.

   The third assumption made in this specification is that the IP
   network is engineered in such a way that any errors introduced in the
   FCIP encapsulation layer occur at a very low rate. Thus the FCIP
   design has been optimized with this assumption in mind. While
   tunneling of Fibre Channel traffic over other IP networks not so
   engineered is not precluded, the above environment is an important
   one, and the FCIP design is optimized for such traffic, while not
   over-burdening other configured IP networks. Any error introduced in
   the FCIP encapsulation layer will result in the frame being dropped
   at the receiving end of the IP network. This will prevent the frame
   from being propagated to the FC network.

Rajagopal, et al.                                               [Page 2]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

   The objectives of this document are to:

          1) specify the encapsulation and mapping of FC frames using
             the FC encapsulation method specified in <TBD>.
          2) apply the mechanism described in (1) to an FC-switched
             backbone network using an IP-based network as a backbone,
             or more generally, between any two FC devices.
          3) address any FC concerns arising from tunneling FC traffic
             over an IP-based network, including security, data
             integrity (loss), congestion, and performance.  This will
             be accomplished, where appropriate, by utilizing the
             existing IETF-specified suite of protocols.
          4) be compatible with existing FC specifications.
             While new work may be undertaken in T11 [13] to optimize
             and enhance the bridging of FC networks/fabrics, this
             specification will not require adherence to such future

4. FCIP Protocol

   4.1 FCIP Device

      In this specification, the term FCIP device generally refers to
      any device that encapsulates FC frames into TCP byte streams and
      reassembles TCP byte streams to regenerate FC frames.

      Note: In an actual implementation, the FCIP device may be a
            stand-alone box or integrated with an FC device such
            as an FC backbone switch (BBW) or integrated with any
            TCP/IP device such as an IP switch or an IP router.

      The FCIP device is a transparent translation point.  The IP
      network is not aware of the FC payload that it is carrying.
      Likewise, the FC fabric and the FC end nodes are unaware of the
      IP-based transport.

   4.2 Protocol

      The FCIP protocol specifies the TCP/IP encapsulation, mapping and
      routing of FC frames and applies these mechanisms to an FC network
      utilizing IP for its backbone, or more generally, between any two
      FC devices. The FCIP protocol is summarized below:

        1. All FCIP protocol devices are peers and communicate using
           TCP/IP. Each FCIP device behaves like a TCP endpoint from the
           perspective of the IP-based network.  That is, these devices
           do not perform IP routing or IP switching but simply forward
           FC frames.

Rajagopal, et al.                                               [Page 3]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

        2. There is no requirement for an FCIP device to establish a
           login with a peer before communication begins.  However, FCIP
           devices MAY authenticate the IP packet before accepting it
           using the IPSec protocols. An FCIP device receiver simply
           listens to the appropriate destination TCP port number (to be
           assigned) to commence communication with other FCIP devices.

        3. Each FCIP device MAY be statically or dynamically configured
           with a list of IP addresses corresponding to all the
           participating FCIP devices.  Dynamic discovery of
           participating FCIP devices MAY be performed using Internet
           protocols such as LDAP, DHCP or other discovery protocols.
           (Discovery work is in progress).

        4. Discovery of FC addresses (accessible via the FCIP device) is
           provided by techniques and protocols within the FC
           architecture. These techniques and protocols are described in
           Fibre Channel ANSI standards ([3], [7], [15]). FCIP devices
           do not participate in the discovery of FC addresses although
           the FC fabric elements of which they are a part MAY
           participate. The establishment of relationships between FC
           addresses and TCP port numbers is outside the scope of this
           document. FCIP devices MAY discover FC domains reachable
           through other FCIP devices by exchanging FC routing
           information with each other. The exact method used to
           exchange FC routing information between FCIP devices is
           beyond the scope of this document. FCIP devices MAY also
           discover FC domains reachable through the fabric region they
           are attached to by exchanging FSPF routing information with
           the FC switches they are connected to.

        5. The exact path (route) taken by an FC over TCP/IP
           encapsulated packet follows the normal procedures of routing
           any IP packet.  From the perspective of the FCIP devices this
           communication is between only two FCIP devices for any given

        6. An FCIP device MAY send FC encapsulated TCP/IP packets to
           more than one FCIP device.  However, these encapsulated
           packets are treated as separate instances and are not
           correlated in any way by the FCIP protocol devices. The
           source FCIP device routes its packets based on the 3-byte FC
           destination Address Identifier (D_ID) contained in each
           encapsulated FC frame.

        7. The IPSec architecture MAY be used to provide secure
           communications for FCIP protocol across the IP-based network.
           Other security protocols are not precluded.

Rajagopal, et al.                                               [Page 4]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

        8. Any re-ordering of data due to IP MTU fragmentation, TCP MSS
           fragmentation, or IP packet re-ordering will be recovered in
           accordance with a normal TCP reliable delivery behavior. The
           FCIP device will be aware only of the TCP-delivered byte
           stream and not cognizant of the TCP recovery action taken to
           deliver the byte stream.

        9. FCIP relies on both TCP error recovery mechanism and normal
           FC recovery mechanisms to detect and recover from data loss
           and corruption within the IP portion of the overall FC and IP

        10. Fibre Channel provides support for several classes of
        service with
            differing loss, priority, and capacity characteristics.  An
            FCIP device MAY choose to map these classes to available
            DiffServ services of the IP network.

        11. FCIP uses the common encapsulation method as specified by

   4.3 FCIP's Interaction with FC and TCP

      The FCIP device always delivers entire FC frames to the FC ports
      connected to it. The FC ports MUST remain unaware of the existence
      of the IP network that provides, through the FCIP devices, the
      connection for these FC ports.

      The FCIP device SHALL treat all classes of FC frames the same - as
      datagrams to be inserted into the TCP byte stream.  Since FC
      Primitives and Primitive Sequences are not exchanged between FCIP
      devices, there may be times when an FC frame is lost within the IP
      network. When this event occurs it is the responsibility of the
      communicating FC devices to detect and correct the errors.  The
      FCIP devices MAY choose not to generate Fibre Channel's F_BSY or
      F_RJT frames or otherwise participate in FC frame recovery.

      Each FCIP data frame is built by adding an FCIP header to one FC
      frame delivered to the FCIP endpoint for transport. The FCIP data
      frames are handed in their entirety to TCP; TCP is responsible for
      delivering the same series of FCIP data frames to the receiving
      side in the same order as they are transmitted by the sending FCIP
      device.  The FCIP device MUST find the FCIP headers and deliver
      the FC frames wrapped inside the FCIP data frames to the correct
      FC ports connected to the FCIP device.

      Note that the order of the FC frames sent by the FCIP device may

Rajagopal, et al.                                               [Page 5]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

      not be the same as the order sent by the source FC device. This is
      due to the fact that FC frames may be re-ordered in the Fibre
      Channel network/fabric before reaching the ingress FCIP device.

      The relationship between FCIP and other protocols is illustrated
      in the following diagram:

     FC switch port           FCIP Device

       +--------+     +------------------------+
       | FC-SW  |     |          FC-IP         |
       +--------+     +--------+---------------+
       |  FC-2  |     |  FC-2  |      |  TCP   |
       +--------+     +---------      +--------+
       |  FC-1  |     |  FC-1  |      |   IP   |
       +--------+     +--------+      +--------+
       |  FC-0  |     |  FC-0  |      |  LINK  |
       +--------+     +--------+      +--------+
            |              |          |   PHY  |
            |              |          +--------+
            |              |               |
            |              |               |
            ----------------               -------> to the other
                                                    FC-IP Devices

               Fig. 2 Protocol Stack Diagram

5. FCIP Encapsulation

   5.1 FC Frame Format (INFORMATIVE)

      All FC frames have a standard format much like LAN's 802.x
      protocols. However, the exact size of each frame varies depending
      on the size of the variable fields.  The size of the variable
      field ranges from 0 to 2112-bytes as shown in the FC Frame Format
      in Fig. 3 resulting in the minimum size FC Frame of 36 bytes and
      the maximum size FC frame of 2148 bytes. Valid Fibre Channel frame
      lengths are always a multiple of four bytes.

          | SOF  |Frame   |Optional   |  Frame      | CRC  |  EOF |
          | (4B) |Header  |Header     | Payload     | (4B) | (4B) |
          |      |(24B)   |<----------------------->|      |      |
          |      |        | Data Field = (0-2112B)  |      |      |

Rajagopal, et al.                                               [Page 6]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

                            Fig. 3 FC Frame Format

      SOF and EOF Delimiters:

      On an FC link, Start-of-Frame (SOF) and End-Of-Frame (EOF) are
      called Ordered Sets and are sent as special words constructed from
      the 8B/10B comma character (K28.5) followed by three additional
      8B/10B data characters making them uniquely identifiable in the
      data stream.

      On an FC link the SOF delimiter serves to identify the beginning
      of a frame and prepares the receiver for frame reception.  The SOF
      contains information about the frame's Class of Service, position
      within a sequence, and in some cases, connection status.

      The EOF delimiter identifies the end of the frame and the final
      frame of a sequence.  In addition, it serves to force the running
      disparity to negative.  The EOF is used to end the connection in
      connection-oriented classes of service.

      It is therefore important to preserve the information conveyed by
      the delimiters across the IP-based network, so that the receiving
      FCIP device can correctly reconstruct the FC frame in its original
      SOF and EOF format before forwarding it to its ultimate FC
      destination on the FC link.

      When an FC frame is encapsulated and sent over a byte-oriented
      interface, the SOF and EOF delimiters are represented as sequences
      of four consecutive bytes, which carry the equivalent Class of
      Service and frame termination information as the FC ordered sets.
      This form of encoding can not provide unambiguous identification
      of frame beginning and end, however, and must rely on other
      mechanisms provided by the encapsulation protocol.

      Frame Header:

      The FC Frame Header is transparent to the FCIP device. The FC
      Frame Header is 24 bytes long and has several fields that are
      associated with the identification and control of the payload.
      Current FC Standards allow up to 3 Optional Header fields [4],

         - Network_Header (16-bytes)
         - Association_Header (32-bytes)
         - Device_Header (up to 64-bytes).

      Frame Payload:

Rajagopal, et al.                                               [Page 7]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

      The FC Frame Payload is transparent to the FCIP device.  An FC
      application level payload is called an Information Unit at the
      FC-4 Level.  This is mapped into the Frame Payload of the FC
      Frame. A large Information Unit is segmented using a structure
      consisting of FC Sequences.  Typically, a Sequence consists of
      more than one FC frames. FCIP does not maintain any state
      information regarding the relationship of frames within a FC


      The FC CRC is 4 bytes long and uses the same 32-bit polynomial
      used in FDDI and is specified in ANSI X3.139 Fiber Distributed
      Data Interface. This CRC value is calculated over the entire FC
      header and the FC payload; it does not include the SOF and EOF

      Note: When FC frames are encapsulated into FCIP frames, the FC
            CRC is untouched by the FCIP device.

   5.3 TCP Connection Management

      In order to realize a Virtual ISL between two FC end-points, an
      FCIP Device establishes TCP connections with its peer FCIP Device.
      In order to achieve better TCP aggregate throughput properties in
      the face of packet losses, a pair of peer FCIP devices MAY use
      multiple TCP connections between them, and use appropriate
      policies for mapping FC frames to these connections. It may also
      be useful to assign a pool of connections for transmission of
      priority and control messages (e.g., Class F messages) on
      connections so they do not encounter "head of line" blocking
      behind Class 2 or Class 3 traffic. The use of multiple connections
      and policies for distributing frames on these connections are
      described in Section 5.5.

      FCIP Devices SHALL listen for new TCP connection requests on the
      well- known port <TBD>. Any FCIP device establishing a TCP
      connection SHALL direct it to this well known port number. Also,
      an FCIP Device MAY use an existing connection, previously
      established by its peer.

      An FCIP device MAY also accept and establish TCP connections to a
      different TCP port number, as configured by the network

Rajagopal, et al.                                               [Page 8]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

      A Virtual ISL and the two FCIP Device endpoints that are involved
      are operational only after the first TCP connection is
      established. The sequence of operations performed in order to
      establish a Virtual ISL is as follows.

        1. The FCIP device initializes its local resources to enable
           it to listen for TCP connection requests.
        2. The FCIP device discovers the FCIP device endpoints that
           it can establish a virtual ISL. The result of the
           discovery SHALL be, at the minimum, the IP address and
           the TCP port of the peer endpoint. The discovery process
           may rely on administrative configuration or on services
           such as SLP or iSNS (TBD). (Needs to have its own section
        3. FCIP device endpoint initiates a TCP connection to the
           peer endpoint. It also sets up operational parameters
           for both TCP and IP layers for optimal performance, as
           described in section 5.3.1.
        4. The FCIP device endpoint SHALL exchange security context
           and authenticate itself to the peer endpoint. The use of
           security context is explained in section TBD. After
           connection establishment, FCIP devices use the FCIP
           frame encapsulation as defined in [common encapsulation
        5. At this point the FCIP device endpoint SHALL exchange
           Fibre Channel port initialization frames (Switch ISL)
           to enable and identify port operation. Port state
           machine and initialization are described in Fibre
           Channel Methodologies for Interconnect (FC-SW 2)
        6. An FCIP device operates in E-port or B-Port mode. When
           operating in E-Port mode, normal FC-SW2 FSPF messages
           are exchanged and the switch port becomes operational.
        7. For computing the link cost of the ISL, the following
           formula SHALL be used: [TBD].

      In certain deployments, a single FCIP device endpoint MAY
      establish virtual ISLs with multiple FCIP device endpoints. In
      this situation, the FCIP device endpoint SHALL manage TCP
      operational parameters independently for each ISL. Also, the FCIP
      Device Endpoint SHALL perform the E_Port or B_Port initialization
      independently, for each connection.

      An FCIP Device Endpoint uses normal TCP based flow control
      mechanisms for managing its internal resources and match that with
      the advertised TCP Receiver Window Size. Thus, an FCIP Device
      endpoint is NOT REQUIRED to advertise or manage Fibre Channel
      BB_Credits or process any R_RDY frames.

Rajagopal, et al.                                               [Page 9]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

      An FCIP Device Endpoint SHALL implement established TCP mechanisms
      as defined in RFC 2581 [20] for congestion control on its

      5.3.1 TCP Connection Parameters

         In order to provide efficient management of FCIP Device
         resources as well as link resources, certain TCP connection
         parameters are recommended.

         It is recommended that FCIP devices use the TCP mechanisms for
         Long Fat Networks (LFNs) (i.e. an IP networks with large
         (bandwidth*delay) product), as defined in RFC 1072 [22].
 TCP Selective Acknowledgement Option

            The Selective Acknowledgement option RFC 2883 [21] allows
            receiving end to acknowledge multiple lost packets in a
            single ACK, enabling faster recovery. An FCIP device MAY
            negotiate use of TCP SACK and use it for faster recovery
            from lost packets and holes in TCP sequence number space.
 TCP Window Scale option

            This option allows TCP window sizes larger than 16-bit
            limits to be advertised by the receiver. It is necessary to
            allow data in long fat networks to fill the available pipe.
            This also implies buffering on the TCP sender that matches
            the (bandwidth*RTT) product of the TCP connection. A TCP
            endpoint SHALL use locally available mechanisms to set a
            window size that matches the available local buffer
            resources and the desired throughput.
 IP DSCP Option

            The recommended IP DSCP field setting is 101110
            corresponding to the EF service. (Need better wording to fit
            current Diffserv specifications.)
 Protection against sequence number wrap

            It is recommended that TCP endpoints implement protection
            against sequence number wrap. It is quite possible that
            within a single connection, TCP sequence numbers wrap within
            a timeout window.
 TCP No Delay Option

Rajagopal, et al.                                              [Page 10]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

            TCP endpoints SHALL disable the Nagle TCP No Delay option.
            This option is designed for usage in a telnet environment.

   5.4 TCP Connection Error Recovery

      5.4.1 Determining loss of connectivity

         FCIP Device endpoints SHALL implement detection and recovery
         from lost TCP connections. In idle mode, a TCP connection "keep
         alive" option of TCP is normally used to keep a connection
         alive. However, this timeout is fairly large and may prevent
         early detection of loss of connectivity. In order to facilitate
         faster detection of loss of connectivity, FCIP devices exchange
         FCIP specific Extended Link Service command messages. These
         FCIP ELS messages use the same encapsulation mechanism as
         described in TBD.

         <Description of FCIP ELS messages>

         Upon detecting a loss of connectivity, an FCIP Device SHALL
         establish a new connection, or SHALL use an existing TCP
         connection to the same FCIP Device endpoint. An FCIP Device
         SHALL NOT retransmit an FCIP frame on the new connection. This
         is to ensure exactly-once delivery semantics to the Fibre
         Channel endpoint.

      5.4.2 TCP Synchronization Errors

         If the FCIP Framing and Encapsulation layer determines that it
         has lost synchronization or has received FCIP header CRC error,
         it SHALL drop the particular frame and attempt to synchronize
         at the earliest possible subsequent frame. For frames with FCIP
         header errors, the FCIP Device SHALL drop the frame and update
         appropriate error counters.

         FCIP device endpoints assume that if the TCP layer determines
         that there are TCP checksum errors, the TCP layer invokes
         appropriate TCP retransmission and error recovery procedures.
         So the FCIP layer gets an ordered delivery of FCIP frames.
         Hence, these errors are transparent to FCIP layer.

         If the FCIP Device layer is delivered frames which are delayed
         by more than R_A_TOV in the IP network, the FCIP Device layer
         SHALL drop the frame. This SHALL continue until a FCIP
         Encapsulated frame whose life in the IP network is smaller than
         R_A_TOV. Note that unlike a physical Fibre Channel link, the
         FCIP Device endpoints may involve IP routing dynamics that
         result in reliable, ordered delivery at the TCP layer, with the

Rajagopal, et al.                                              [Page 11]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

         result that some FC Fabric operating constraints may be
         violated.  The FCIP device is responsible for detecting
         violations of these FC Fabric constraints and discarding
         affected frames.

   5.5 Multiple Connection Management

      A pair of FCIP device endpoints MAY establish a certain number of
      TCP connections between them.  Since a Virtual ISL potentially
      maps a fairly large number of FC flows (where a flow is a pair of
      Fibre Channel S_ID, D_ID addresses), it may not be practical to
      establish a separate TCP connection for each Fibre Channel flow.
      In order to address this, an implementation MAY choose to manage a
      pool of TCP connections for a single Virtual ISL and map Fibre
      Channel flows to TCP connections of that ISL. However, while
      assigning Fibre Channel flows to TCP connections, an
      implementation SHALL follow the following rules:

        1. Once a  channel flow is assigned to a TCP connection
           within the virtual ISL, it SHALL send all Fibre Channel
           frames of that flow on that connection.
        2. When an FCIP endpoint processes any response traffic from
           a particular target, the Endpoint SHALL send the response
           on the same connection on which the request was sent.
        3. Any class 2 ACK frames SHALL be sent on the same
           connection in which the original frame was sent.

      These rules are in place to honor any in-order delivery guarantees
      that may have been made between the two end points of the Fibre
      Channel flow.

5.6 Multi Virtual ISL Management

   It is quite likely that a single switch may provide multiple Virtual
   ISLs, all providing alternate connectivity paths between two
   switches. In this situation, a switch SHALL select any of the
   available ISLs for mapping a FCIP flow. In doing so, a switch MUST
   follow a flow allegiance model, where a pair of Fibre Channel [S_ID,
   D_ID] end points are always mapped to the same Virtual ISL.
   Furthermore, switches SHALL implement a connection allegiance policy,
   which ensures that the responses to particular [S_ID, D_ID] pair is
   always sent back on the same Virtual ISL.

6. FCIP Network and Device model

   6.1 General FCIP Model

   Fibre Channel defines interconnections between FC Fabric Elements

Rajagopal, et al.                                              [Page 12]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

   [3]. Pairs of FCIP devices (described in this draft) connected by a
   TCP/IP network may replace the Fibre Channel connection between two
   FC Fabric Elements in ways specified by the Fibre Channel documents
   that reference this draft [13]. The concept of the FC/FCIP fabric and
   interconnections is shown in figure 5 below.

      All Fibre Channel Model

       Fibre Channel Fabric                         /
      /                                                        | | |                            | | |    /
      /   +---------+   Fibre Channel  +---------+          --| Fabric  |      defined     | Fabric  |-- /
      / --| Element |------------------| Element |--        --|    A    |     connection   |    B    |-- /
      /   +---------+                  +---------+             | | |                            | | |    /
      /                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      Fibre Channel + FCIP Model

       Fibre Channel Fabric                         /
      /                                                        | | |                            | | |    /
      /   +-----------+              +-----------+          --| Fabric    |     FCIP     |   Fabric  |-- /
      / --| Element FD|--------------|FD Element |--        --|    A      |  connection  |      B    |-- /
      /   +-----------+              +-----------+             | | |                            | | |    /
      /                                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      note: FD = FCIP Device

          Fig. 5 General FCIP Model with Respect to FC Fabric

      It is expected that the FCIP device will be integrated with the FC
      Fabric Element for which it supplies one end of an FCIP
      connection. There is no interface specified between the FCIP
      device and the FC Fabric Element.

      The following requirements are placed on the replacement of a
      Fibre Channel connection with an FCIP connection:

        1. The connection of both Fibre Channel components MUST have an
           analogous connection defined in the ANSI-NCITS Fibre Channel
           Standards [13].

        2. The same type of Fibre Channel components MUST be located at

Rajagopal, et al.                                              [Page 13]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

           ends of the connection served by FCIP devices.

        3. The requirements in this standard apply equally to any pair
        of FCIP
           devices and the FC Fabric Elements of which they are a part
           to tunnel FC frames across an IP network.

      Any FC routing protocol exchanges may still occur transparently to
      the pair of FCIP devices providing the connection. It should be
      noted that Fibre Channel Primitive Sequences and Primitives are
      not exchanged between pairs of FCIP devices.

   6.2 Model Details for Border Switches

+-----+                                         +-----+
| SW1 |                                         | SW3 |
+-----+                                         +-----+
   |       +-----+                   +-----+       |
   |       |     |                   |     |       |
+-----+E  B| BBW |   //////    | BBW |E  E+-----+
| SW2 |----|  1  |---           ---|  2  |----| SW4 |
+-----+    |     |   /           /   |     |    +-----+
           +-----+                 +-----+
                     /           /
                                                     /    IP     /
           +-----+     Network     +-----+
           |     |   /           /   |     |
+-----+E  E| BBW |                 | BBW |B  E+-----+
| SW6 |----|  3  |---/           /---|  4  |----| SW7 |
+-----+    |     |   //////    |     |    +-----+
   |       +-----+                   +-----+       |
   |                                               |
+-----+                                         +-----+
| SW5 |                                         | SW8 |
+-----+                                         +-----+

                    Fig. 6 FCIP Border Switch Model

      The Fibre Channel Border Switch model for an FCIP device is an
      extension of the Backbone WAN (BBW) device currently defined for
      ATM (BBW_ATM) and SONET (BBW_SONET) in FC-BB.  The planned
      extensions to the current BBW model include:

        1) Defining the Fibre Channel operational details of

Rajagopal, et al.                                              [Page 14]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

         a BBW_TCP/IP device 2) Providing for three Fibre Channel
         configurations transported by BBW_TCP/IP devices:
         a) [B_Port]..BBW_TCP/IP<---->BBW_TCP/IP..[B_Port]
         b) [E_Port]..BBW_TCP/IP<---->BBW_TCP/IP..[E_Port]
         c) [B_Port]..BBW_TCP/IP<---->BBW_TCP/IP..[E_Port]
         d) [E_Port]..BBW TCP/IP<---->BBW TCP/IP..[B_Port]

      Note: For an FCIP device, d) is the mirror case of c) and is
      architecturally identical to c) but operational conditions within
      the FC fabric may be different depending on whether case c) or its
      mirror d) is in effect.

      3) Defining Virtual inter-switch links (ISL's) between
         the E_ports on BBW_TCP/IP devices.

      Because substantial Fibre Channel fluency is required for the
      BBW_TCP/IP definition, it is anticipated that FCIP will contain
      only a high level overview of the model, with the details
      appearing in FC-BB-2 (an existing T11 project created specifically
      to extend the Backbone concept to TCP/IP).

7. Security Considerations

   Using a wide-area, general purpose network such as an IP internet in
   a position normally occupied by physical cabling introduces some
   security problems not normally encountered in Fibre Channel storage
   networks. Normal FC media are typically protected physically from
   outside access; IP internets typically invite outside access.

   The general effect is that the security of the entire Fibre Channel
   internetwork is only as good as the security of the entire IP
   internet through which it tunnels.  The following broad classes of
   attacks are possible:

   1. Unauthorized Fibre Channel controllers can gain access to
      resources through normal Fibre Channel processes.

   2. Unauthorized agents can monitor and manipulate Fibre Channel
      traffic flowing over physical media used by the IP internet and
      under control of the agent.

   To a large extent, these security risks are typical of the risks
   facing any other application using an IP internet.  They are
   mentioned here only because Fibre Channel storage networks are not
   normally suspicious of the media.  Fibre Channel storage network
   administrators will need to be aware of these additional security

Rajagopal, et al.                                              [Page 15]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001


   Security protocols and procedures used in other IP applications may
   be used for FCIP.

   For Virtual Private Networks, both authentication and encryption are
   generally desired, because it is important both to (1) assure that
   unauthorized users do not penetrate the virtual private network and
   (2) assure that eavesdroppers on the network cannot read messages
   sent over the network.

   Note: Use of the IPSec protocol suite is optional. Security work is
   in progress.

8. Data Integrity Considerations

   The material in this section is subject to change pending work in
   progress in the data integrity area.

   8.1 Loss of FCIP synchronization

   The use of the FCIP length with either or both of the EOF byte-code
   immediately preceding the FCIP header and the SOF byte-code
   immediately following the FCIP header provides enough verification
   that the FCIP devices communicating over a particular TCP connection
   are synchronized with each other.

   If a communicating pair of FCIP devices loses synchronization (the
   receiving FCIP device cannot find the next FCIP header) due to data
   loss, network congestion, or other error conditions in the TCP byte
   stream, the receiving FCIP device SHALL reset the TCP connection (set
   the RST bit). <The treatment of FC data and the aborted TCP
   connection need additional work to complete.>

   8.2 Loss

      Recovery from data loss due to IP datagram loss is provided via
      the TCP reliable delivery mechanism.   Note: Due to varying TCP
      timeouts, competing FC and TCP recovery schemes is a possibility.
      This issue is addressed in section 8.4.

   8.3 Corruption

      Data corruption is detected at two different levels: TCP checksum
      and Fibre Channel CRC.  Data corruption detected at the TCP level
      SHALL be recovered via TCP reliable data recovery mechanisms.
      Data corruption detected at the Fibre Channel level SHALL be

Rajagopal, et al.                                              [Page 16]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

      handled within the Fibre Channel end nodes. Also, each recovery
      technique is performed independent of the other.

      FCIP devices that perform a CRC integrity check on encapsulated FC
      frames SHALL modify the EOF sequence from EOF-normal to EOF-
      normal- invalid for each frame found to have a CRC error.

   8.4 Timeouts

      Fibre Channel has two important timeouts to consider in FCIP.
      These are: ED_TOV, and R_A_TOV.

      ED_TOV determines the life of an individual Fibre Channel frame in
      any particular fabric element.  The effects of ED_TOV on the
      fabric as a whole are typically cumulative since each fabric
      element contains it's own ED_TOV timers for any frame received.

      R_A_TOV determines the life of an individual Fibre Channel frame
      in the fabric as a whole.   For a fabric, R_A_TOV implies that no
      particular frame will remain in (and thus be emitted from) the
      fabric after the timer expires.

      TCP has a TCP acknowledgement timeout.  This is a variable
      timeout.  <TBD: Need to elaborate on TCP timeouts and define how
      Fibre Channel timeouts map to TCP timeouts.>

   8.5 Recovery Mechanisms

      When an FCIP data  frame is transported over an IP network, there
      is a possibility of the frame's getting dropped.  This can happen
      if there is congestion along the path within the IP network or if
      there are no empty buffers available on one of the incoming ports,
      due to bit errors, etc. When this happens, the TCP acknowledgement
      will not be received by the source, and normal TCP retry
      mechanisms will be activated.

      An issue may arise during these recovery mechanisms, since TCP
      timeout is variable, and may exceed Fibre Channel FC
      ED_TOV/R_A_TOV timeouts.

      <Placeholder: Need to add details on how to handle this here.>

9. Performance Considerations

   The FCIP protocol does not crack the FC Frame (except for attaching
   the correct byte-encoded SOF and EOF) nor does it do any FC payload
   processing.  This allows any FC traffic to be tunneled across at high
   throughput rates.

Rajagopal, et al.                                              [Page 17]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

   If fragmentation at the data link and IP layers is avoided by the use
   of path MTU discovery, throughput performance is enhanced.

   The Flow Control Protocol (discussed in the next section) provides
   the ability to stream gigabit FC data when using a large window size.

   9.1 QoS Support

      The Differentiated Services Architecture (diffserv) provides a
      "Class of Service" to a flow aggregate [6], [17]. At so-called
      diffserv boundaries, IP packets are classified and marked. Within
      the diffserv domain, resources – bandwidth and buffers – are
      allocated for each classification. Packets with the same
      classification use the resources allocated for the classification.
      IP packets with the same destination and class marking exit a
      diffserv capable router in the same order they arrived. Packets
      with the same destination but different class markings exit
      according to priorities assigned to the different class markings.
      The Diffserv has renamed the Ipv4 TOS field as Differentiated
      Services Code Point (DSCP). The DSCP indicates the particular
      behavior a packet is to receive at each router. How a packet gets
      marked is based on a policy administered and configured into the
      network. [18] and [19] provide various encodings of the DSCP field
      to achieve a specific behavior from the routers. There may be
      several ways to administer the policies and the policy definition
      is up to the network provider. That is one network provider may
      choose to mark all packets going from one source IP address to a
      specific destination as "high priority", while another might mark
      just a specific traffic type (e,g., HTTP) as "high priority". Thus
      packets carry the desired class information and each diffserv-
      capable router treats the packet according to the information in
      its DSCP field. This is referred to as Per Hop Behavior (PHB).
      Currently, the IETF standards define essentially 3 types of
      services: Expedited Forwarding (EF) [18], Assured Forwarding (AF)
      [19], or Default Forwarding (DF) [6], [17]– that corresponds to
      its DSCP.

      [17] specifies the AF service AF PHB provides a way to prioritize
      best- effort traffic. Currently, 4 AF classes and 3 drop
      precedence levels are specified providing 12 different levels of
      forwarding assurances. The DSCP value specifies a drop-order in
      the event that a packet experiences congestion at a subsequent
      diffserv router.

      [18] specifies the DSCP code point equal to 101110  EF service
      which is also sometimes refereed to as "Premium" service. When
      supported, this class behavior has the lowest levels of latency,
      packet loss, and delay variation. This service behavior most

Rajagopal, et al.                                              [Page 18]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

      closely matches the Fibre Channel characteristics. This is
      therefore the recommended DSCP setting in the IP DSCP field.

      What resources are not used for EF and AF are left for the DF
      services which is really a best-effort service.

      Note that if a packet is being forwarded over an underlying
      network without diffserv support, then the packet would simply
      receive best- effort service regardless of its DSCP field setting.

   9.2 TCP considerations

      In order to achieve better TCP aggregate throughput in the face of
      packet losses, a pair of peer FCIP devices may use multiple TCP
      connections between them, and use appropriate policies for mapping
      FC frames to these connections. The reason for this is the TCP's
      slow-start algorithm, which reduces TCP's window whenever it
      detects congestion in the network. If, on the other hand, the
      traffic is distributed across multiple connections, all the
      connections will not be affected at the same time, resulting in a
      better aggregate throughput.

      The use of multiple connections and policies for distributing
      frames on these connections are described in section TBD. Note
      that even though multiple connections provide better aggregate
      throughput (when packet losses occur on IP networks), it is not a
      requirement. A pair of FCIP devices may use single TCP connection
      to tunnel the FC traffic.

      It is recommended that FCIP devices use the TCP mechanisms for
      Long Fat Networks (LFNs) when they are used in IP networks with
      large (bandwidth*delay) product. These mechanisms include TCP
      window scale option, Selective Acknowledgement, among others.

10. Flow Control and Congestion Management

   FCIP protocol specifies encapsulating FC frames over IP networks,
   using TCP connections. The FCIP device is connected to both FC fabric
   and IP network and it needs to follow the flow control mechanisms on
   both the networks, which work independent of each other.

   This section provides guidelines as to how the FCIP device can map
   from one flow control mechanism to another, while encapsulating FC
   traffic over TCP connections and vice versa.

   There are two scenarios when the flow control management at FCIP
   device becomes crucial:

Rajagopal, et al.                                              [Page 19]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

   1. When there is mismatch between the line speeds of FC and IP

   Even though it is recommended that both FC and IP networks be of
   comparable speeds, it is possible that FC traffic is carried over an
   IP network of different line speed and bit error rates.

   2. When one of the networks (FC or IP) is congested.

   Even when both FC and IP networks are of comparable speeds, during
   the course of operation, one of the networks could be congested due
   to transient conditions.

   The FCIP device needs to use the available flow control mechanisms in
   TCP and FC protocols to handle these situations. The FCIP protocol
   does not specify any particular mechanism to handle the flow control,
   but leaves this to implementation's choice.

   10.1 Flow control on FC network

      When the Fibre channel traffic is encapsulated over TCP
      connection(s), FCIP device needs to ensure that TCP connections
      can handle the frame arrival rate from Fibre channel network. This
      MAY require FCIP device to use Buffer-to-buffer flow control on
      its Fibre channel Port(s), to control the frame arrival rate.
      Alternatively, the FCIP device MAY choose to send F_BSY frame to
      the originator of FC frame, for FC Class-1 (Connect Request) and
      Class-2 frames.

   10.2 Flow control on IP network

      When the FCIP device needs to forward frames from TCP
      connection(s) to Fibre channel ports, it needs to follow the
      Buffer-to-buffer credit mechanism on its FC port(s). If there is
      no available credit on the FC port(s), FCIP device MAY require to
      control the packet arrival rate from the IP network, by using TCP
      windowing techniques. This MAY involve advertising zero-window on
      TCP connections occasionally, so that the TCP connection is flow
      controlled while the FC network is congested.

11. References:

     [1] Bradner, S., "The Internet Standards Process -- Revision 3",
         BCP 9, RFC 2026, October 1996.

     [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement
         Levels", BCP 14, RFC 2119, March 1997

Rajagopal, et al.                                              [Page 20]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

     [3] NCITS 321-200x (ANSI) T11/Project 1305-D/Rev 4.9 "Fibre Channel
         Switch-Fabric-2", (FC-SW-2) November 14, 2000 (www.t11.org)

     [4] Fibre Channel Physical and Signaling Interface-3 (FC-PH-3), Rev.
         9.4, ANSI X3.303-1998

     [5] The Fibre Channel Consultant: A Comprehensive Introduction,
         "Robert W. Kembel", Northwest Learning Associates, 1998

     [6] Nichols, K., Blake, S., Baker, F.  and D. Black, " Definition
         of the Differentiated Services Field (DS Field) in the IPv4 and
         Ipv6 Headers", RFC 2474, December 1998.

     [7] NCITS T11/Project 1238-D/Rev4.7 "Fibre Channel Backbone", (FC-BB)
         June 8, 2000 (www.t11.org)

     [8] Kent, S. and Atkinson, R., "Security Architecture for the
         Internet Protocol", RFC 2401, Nov 1998

     [9] Kent, S. and Atkinson, R., "IP Authentication Header",
         RFC 2402, Nov 1998

     [10] Kent, S. and Atkinson, R., "IP Encapsulating Security
          Payload (ESP)", RFC 2406, Nov 1998

     [11] Maughan, D. et all, "Internet Security Association and Key
          Management Protocol (ISAKMP)", RFC 2408, Nov 1998

     [12] http://www.isi.edu/in-notes/iana/assignments/protocol-numbers

     [13] http://www.t11.org

     [14] Fibre Channel Physical and Signaling Interface (FC-PH), Rev 4.3,
          ANSI X3.230-1994.

     [15]  Fibre Channel  NCITS  321-200x (ANSI) T11/Project  1356-D/Rev4.3 "
           Fibre Channel - Generic  Services 3", June 2000 (www.t11.org)).

     [16] ISI, "Transmission Control Protocol", RFC 793, Sep 1981

     [17] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
          Weiss, W., "An Architecture for Differentiated Services",
          RFC 2475, Dec 1998

     [18] Jacobson, V., Nichols, K., Poduri, K., "An Expedited
          Forwarding PHB Group", RFC 2598, June 1999

     [19] Heinanen, J., Baker, F., Weiss, W.,  Wroclawski, J., "An

Rajagopal, et al.                                              [Page 21]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

          Assured Forwarding PHB", RFC 2597, June 1999

     [20] Allman, et al., "TCP Congestion Control", RFC 2581, April 1999

     [21] Floyd, et al, "SACK Extension", RFC 2883, July 2000

     [22] Jacobson & Braden, "TCP Extensions for Long-Delay Paths",
          RFC1072, October 1988

12. Acknowledgments

13. Authors' Addresses

     Murali Rajagopal
     LightSand Communications, Inc.
     24411 Ridge Route Dr.
     Suite 135
     Laguna Hills, CA 92653
     Phone: 949-837-1733 x101
     Email: muralir@lightsand.com

     Raj Bhagwat
     LightSand Communications, Inc.
     24411 Ridge Route Dr.
     Suite 135
     Laguna Hills, CA 92653
     Phone: 949-837-1733 x104
     Email: rajb@lightsand.com

     Elizabeth G. Rodriguez
     Lucent Technologies
     1202 Richardson Drive, Suite 210
     Richardson, TX 75080
     Phone: +1 972 231 0672
     Fax: +1 972 671 5476
     Email: egrodriguez@lucent.com

     Vi Chau
     Gadzoox Networks, Inc.
     16241 Laguna Canyon Road, Suite 100
     Irvine, CA 92618
     Phone: +1 949 789 4639
     Fax: +1 949 453 1271
     Email: vchau@gadzoox.com

     Gaby Hecht

Rajagopal, et al.                                              [Page 22]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

     Gadzoox Network, Inc.
     16241 Laguna Canyon Road, Suite 100
     Irvine, CA 92618
     Phone: +1 949 789 4642
     Email: ghecht@Gadzoox.com

     Ken Hirata
     Vixel Corporation
     15245 Alton Parkway, Suite 100
     Irvine, CA 92618
     Phone: +1 949 788 6368
     Email: ken.hirata@vixel.com

     Jim Nelson
     Vixel Corporation
     15245 Alton Parkway, Suite 100
     Irvine, CA 92618
     Phone: +1 949 450 6159
     Fax: +1 949 753 9500
     Email: Jim.Nelson@vixel.com

     Steve Wilson
     Brocade Communications Systems, Inc.
     1745 Technology Drive
     San Jose, CA. 95110
     Phone: 408-487-8128
     Fax: 408-487-8101
     email: swilson@brocade.com

     Bob Snively
     Brocade Communications Systems, Inc.
     1745 Technology Drive
     San Jose, CA 95110
     Phone: 408 487 8135
     Email: rsnively@brocade.com

     Ralph Weber
     ENDL Texas, representing Brocade
     Suite 102#178
     18484 Preston Road
     Dallas, TX 75252
     Phone: +1 214 912 1373
     Email: roweber@acm.org

     Michael E. O'Donnell
     McDATA Corporation
     310 Interlocken Parkway
     Broomfield, Co. 80021

Rajagopal, et al.                                              [Page 23]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

     Phone: +1 303 460 4142
     Fax: +1 303 465 4996
     Email: modonnell@mcdata.com

     Anil Rijhsinghani
     McDATA Corporation
     5 Brickyard lane
     Westboro, MA 01581
     Phone: +1 508 870 6593
     Email: anil.rijhsinghani@mcdata.com

     Craig W. Carlson
     QLogic Corporation
     6321 Bury Drive
     Eden Prairie, MN 55346
     Phone: +1 952 932 4064
     Email: craig.carlson@qlogic.com

     Sriram Rupanagunta
     Aarohi Communications
     3200 Montelena Drive
     San Jose, CA 95135
     Phone: 408-966-8309
     Email: sriramr@aarohi-inc.com

     Milan J. Merhar
     Pirus Networks
     Acton, MA 01720
     Phone: +1 978 206 9124
     Email: Milan@pirus.com

     Venkat Rangan
     Rhapsody Networks Inc.
     3450 W. Warren Ave
     Fremont, CA 94538
     Phone: +1 510 743 3018
     Fax: +1 510 687 0136
     Email: venkat@rhapsodynetworks.com

     Donald R. Fraser
     Compaq Computer Corporation
     301 Rockrimmon Blvd
     Colorado Springs, CO 80919
     Phone: 719-548-3272
     Email: don.fraser@compaq.com

ANNEX A: Relationship between FCIP and IP over

Rajagopal, et al.                                              [Page 24]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

              FC (IPFC)

   IPFC (RFC 2625) describes the encapsulation of IP packets in FC
   frames. It is intended to facilitate IP communication over an FC

   FCIP describes the encapsulation of FC frames in TCP segments which
   in turn are encapsulated inside IP packets for transporting over an
   IP network.  It gives no consideration to the type of FC frame that
   is being encapsulated.  Therefore, the FC frame may actually contain
   an IP packet as described in the IP over FC specification (RFC 2625).
   In such a case, the data packet would have:

          Data Link Header
          IP Header
          TCP Header
          FCIP Header
          FC Header
          IP Header

   Note:   The two IP headers would not be identical to each other.  One
           would have information pertaining to the final destination
           while the other would have information pertaining to the FCIP

   The two documents focus on different objectives.  As mentioned above,
   implementation of FCIP will lead to IP encapsulation within IP. While
   perhaps inefficient, this should not lead to issues with IP
   communication.  One caveat: if a Fibre Channel device is
   encapsulating IP packets in an FC frame (e.g. an IPFC device), and
   that device is communicating with a device running IP over a non-FC
   medium, a second IPFC device will need to act as a gateway between
   the two networks. This scenario is not specifically addressed by

   There is nothing in either of the specifications to prevent a single
   device from implementing both FCIP and IP-over-FC (IPFC), but this is
   implementation specific, and is beyond the scope of this document.

Full Copyright Statement

Copyright (C) The Internet Society (1999).  All Rights Reserved.

This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it or
assist in its implementation may be prepared, copied, published and
distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are included

Rajagopal, et al.                                              [Page 25]

Internet-Draft         Fibre Channel over TCP/IP             April, 2001

on all such copies and derivative works.  However, this document itself
may not be modified in any way, such as by removing the copyright notice
or references to the Internet Society or other Internet organizations,
except as needed for the purpose of developing Internet standards in
which case the procedures for copyrights defined in the Internet
Standards process must be followed, or as required to translate it into
languages other than English.

The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS


Funding for the RFC Editor function is currently provided by the
Internet Society.

[draft-ietf-ips-fcovertcpip-02.txt] [This INTERNET DRAFT expires in
October, 2001]

Rajagopal, et al.                                              [Page 26]

Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/