[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits] [IPR]

Versions: 00 01 02 03 04 05 06 07 08 09 10 11 12 RFC 3821

IPS Working Group     M. Rajagopal, R. Bhagwat, LightSand Communications
INTERNET-DRAFT                         E. Rodriguez, Lucent Technologies
<draft-ietf-ips-fcovertcpip-01.txt>            V. Chau, Gadzoox Networks
(Expires May 29, 2001)                                  S. Berman, Vixel
                                       S. Wilson, Brocade Communications
                                                    M. O'Donnell, McDATA
                                                      C. Carlson, QLogic

                    Fibre Channel Over TCP/IP (FCIP)

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC 2026 [1].

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as Reference
   material or to cite them other than as ``work in progress''.

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

1. Abstract

   Fibre Channel (FC) is a dominant technology used in Storage Area
   Networks (SAN). The purpose of this draft is to specify a standard
   way of encapsulating FC frames over TCP/IP and to describe mechanisms
   that allow islands of FC SANs to be interconnected  over IP-based
   networks. FC over TCP/IP relies on IP-based network services to
   provide the connectivity between the SAN islands over LANs, MANs, or
   WANs.  The FC over TCP/IP specification relies upon TCP for
   congestion control and management and upon both TCP and FC for data
   error and data loss recovery.  FC over TCP/IP treats all classes of
   FC frames the same -- as datagrams.

2. Conventions used in this document

Rajagopal, et al.                                               [Page 1]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

   The key words "MUST", "MUST NOT",  "REQUIRED",  "SHALL",  "SHALL
   in this document are to be interpreted as described in RFC 2119 [2].

3. Motivation and Objectives

   Fibre Channel (FC) is a gigabit speed networking technology primarily
   used for Storage Area Networking (SAN).  FC is standardized under
   American National Standard for Information Systems of the National
   Committee for Information Technology Standards (ANSI-NCITS) and has
   specified a number of documents describing its protocols, operations,
   and services [13].

   The motivation behind connecting remote sites include disk or tape
   backup and live mirroring, or simply distance extension between two
   FC devices or FC Switch clusters (SAN islands).

   A fundamental assumption made in this specification is that the Fibre
   Channel traffic is carried over the IP network in such a manner that
   the Fibre Channel fabric and all Fibre Channel devices on the fabric
   are unaware of the fact. This means that the FC datagrams must be
   delivered in such time as to comply with existing Fibre Channel
   specifications.  The FC traffic may span LANs, MANs and WANs, so long
   as this fundamental assumption is adhered to.

   Fibre Channel operates at Gigabit speeds.  This specification is
   written such that Fibre Channel traffic may be transported over an IP
   backbone that has been engineered to have equivalent or better bit-
   error-rate (BER) and line speed as the Fibre Channel environments
   being bridged.  While tunneling of Fibre Channel traffic over other
   IP networks not so engineered is not precluded, the above environment
   is an important one, and this specification has been written so that
   to optimize for such traffic, while not overburdening other
   configured IP networks.

   The main objectives of this document are to:

          1) specify the TCP encapsulation and mapping of FC frames
          2) specify a mechanism that delimits each FC frame within the
             TCP byte stream.
          3) apply the mechanism described in (1) to an FC-switched
             backbone network using an IP-based network as a backbone,
             or more generally, between any two FC devices.
          4) address any FC concerns arising from tunneling FC traffic
             over an IP-based network, including security, data
             integrity (loss), congestion, and performance.  This will
             be accomplished, where appropriate, by utilizing the
             existing IETF-specified suite of protocols.

Rajagopal, et al.                                               [Page 2]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

          5) be backwards compatible with existing FC specifications.
             While new work may be undertaken in T11 [13] to optimize
             and enhance the bridging of FC networks/fabrics, this
             specification will not require adherence to such future

4. FCIP Protocol

   4.1 FCIP Device

      In this specification, the term FCIP device generally refers to
      any device that encapsulates FC frames into TCP segments and
      reassembles TCP segments to regenerate FC frames.

      Note: In an actual implementation, the FCIP device may be a
            stand-alone box or integrated with an FC device such
            as an FC backbone switch (BBW) or integrated with any
            TCP/IP device such as an IP switch or an IP router.

      The FCIP device is a transparent translation point.  The IP
      network is not aware of the FC payload that it is carrying.
      Likewise, the FC fabric and the FC end nodes are unaware of the
      IP-based transport.

   4.2 Protocol

      The FCIP protocol specifies the TCP/IP encapsulation, mapping and
      routing of FC frames and applies these mechanisms to an FC network
      utilizing IP for its backbone, or more generally, between any two
      FC devices. The FCIP protocol is summarized below:

        1. All FCIP protocol devices are peers and communicate using
           TCP/IP. Each FCIP device behaves like a TCP end-node from the
           perspective of the IP-based network.  That is, these devices
           do not perform IP routing or IP switching but simply forward
           FC frames.

        2. There is no requirement for an FCIP device to establish a
           login with a peer before communication begins.  However, FCIP
           devices may authenticate the IP packet before accepting it
           using the IPSec protocols.  Each IP datagram is treated
           independently and an FCIP device receiver simply listens to
           the appropriate destination TCP port number (to be assigned)
           contained in the TCP header.

        3. Each FCIP device may be statically or dynamically configured
           with a list of IP addresses corresponding to all the
           participating FCIP devices.  Dynamic discovery of

Rajagopal, et al.                                               [Page 3]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

           participating FCIP devices may be performed using Internet
           protocols such as LDAP, DHCP or other discovery protocols.
           It is outside the scope of this specification to describe any
           static or dynamic scheme for participating FCIP device IP
           address discovery.

        4. Discovery of FC addresses (accessible via the FCIP device) is
           provided by techniques and protocols within the FC
           architecture. These techniques and protocols are described in
           Fibre Channel ANSI standards ([3], [7], [15]).  The FCIP
           device does not participate in the discovery of FC addresses.
           Routing in the IP plane and the FC plane are largely

        5. The exact path (route) taken by an FC over TCP/IP
           encapsulated packet follows the normal procedures of routing
           any IP packet.  From the perspective of the FCIP devices this
           communication is between only two FCIP devices for any given

        6. An FCIP device may send FC encapsulated TCP/IP packets to
           more than one FCIP device.  However, these encapsulated
           packets are treated as separate instances and are not
           correlated in any way by the FCIP protocol devices. The
           source FCIP device routes its packets based on the 3-byte FC
           destination Address Identifier (D_ID) contained in each
           encapsulated FC frame.

        7. The IPSec architecture may be used to provide secure
           communications for FCIP protocol across the IP-based network.
           Other security protocols are not precluded.

        8. Any re-ordering of data due to IP MTU fragmentation, TCP MSS
           fragmentation, or IP packet re-ordering will be recovered in
           accordance with a normal TCP reliable delivery behavior.

        9. FCIP relies on both TCP error recovery mechanism and normal
           FC recovery mechanisms to detect and recover from data loss.

        10. It is recommended that the IP packets carrying TCP
            encapsulated FCIP data frames set the DSCP bits
            corresponding to the Expedited Forwarding (EF) service which
            closely matches the characteristics of the Fibre Channel

        11. FCIP defines a data framing structure that provides the
            encapsulation of FC frames in the TCP data stream and a way
            to synchronize the FCIP devices that communicate with one

Rajagopal, et al.                                               [Page 4]

Internet-Draft         Fibre Channel over TCP/IP           November 2000


   4.3 FCIP Header Format

      Fig. 1 shows the format of the header for the FCIP adaptation

     |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1                     |
     |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 |
     |  VER. |HDR Len|          Reserved         | FCIP Frame Length  |

                       Fig. 1 FCIP Header Format

      FCIP Version Number:

      This 4-bit field shall be 0x1 for this version of the FCIP

      FCIP Header Length:

      This 4-bit field contains the number of 32-bit words in the FCIP
      header. The current version of the FCIP protocol dictates that
      this field shall have a value of 0x1.

      FCIP Frame Length:

      This 10-bit field contains the length of the entire FCIP frame
      including the FCIP header. This length is based on a unit of 32-
      bit word. All FC frames are 32-bit-word-aligned and the FCIP
      header shall always be defined to be word-aligned; therefore a
      32-bit word length is acceptable.

      Reserved bits:

      These bits do not carry any information; they should be set to 0.

      A Receiving FCIP device should use the combination of the FCIP
      length value found in the FCIP header and the SOF and EOF byte-
      codes to validate its synchronization with the sender.

   4.4 Loss of FCIP synchronization

      The use of the FCIP length with either or both of the EOF byte-
      code immediately preceding the FCIP header and the SOF byte-code
      immediately following the FCIP header provides enough verification

Rajagopal, et al.                                               [Page 5]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      that the FCIP devices communicating over a particular TCP
      connection are synchronized with each other.

      If a communicating pair of FCIP devices loses synchronization (the
      receiving FCIP device cannot find the next FCIP header) due to
      data loss, network congestion, or other error conditions in the
      TCP byte stream, the receiving FCIP device shall reset the TCP
      connection (set the RST bit). <The treatment of FC data and the
      aborted TCP connection need additional work to complete.>

   4.5 FCIP's Interaction with FC and TCP

      The FCIP device always delivers entire FC frames to the FC ports
      connected to it. The FC ports must remain unaware of the existence
      of the IP network that provides, through the FCIP devices, the
      connection for these FC ports.

      The FCIP device shall treat all classes of FC frames the same - as
      datagrams.  Since FC Primitives and Primitive Sequences are not
      exchanged between FCIP devices, there may be times when an FC
      frame is lost within the IP network. When this event occurs it is
      the responsibility of the communicating FC devices to detect and
      correct the errors.  The FCIP devices may choose not to generate
      Fibre Channel's F_BSY or F_RJT frames or otherwise participate in
      FC frame recovery. FCIP may not be suitable for the transport of
      FC Class 1 traffic since these frames are treated the same way as
      any Class 2 or 3 frames by the FCIP devices.

      FCIP data frames are built from an FCIP header and the FC frames
      that FCIP has to transport. The FCIP data frames are handed in
      their entirety to TCP; TCP is responsible for delivering the same
      series of FCIP data frames to the receiving side in the same order
      as they are transmitted by the sending FCIP device.  The FCIP
      device must find the FCIP headers and deliver the FC frames
      wrapped inside the FCIP data frames to the correct FC ports
      connected to the FCIP device.

      Note that the order of the FC frames sent by the FCIP device may
      not be the same as the order sent by the source FC device. This is
      due to the fact that FC frames may be re-ordered in the fibre
      channel network/fabric before reaching the ingress FCIP device.

      The relationship between FCIP and other protocols is illustrated
      in the following diagram:

Rajagopal, et al.                                               [Page 6]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

        FC End node           FCIP Device

       +--------+     +------------------------+
       |  ULP   |     |         FC-IP          |
       +--------+     +------------------------+
       |  FC-2  |     |  FC-2  |      |  TCP   |
       +--------+     +--------+      +--------+
       |  FC-1  |     |  FC-1  |      |   IP   |
       +--------+     +--------+      +--------+
       |  FC-0  |     |  FC-0  |      |   PHY  |
       +--------+     +--------+      +--------+
            |              |               |
            |              |               |
            ----------------               -------> to the other
                                                    FC-IP Devices

               Fig. 2 Protocol Stack Diagram

5. FCIP Encapsulation

   5.1 FC Frame Format

      All FC frames have a standard format much like LAN's 802.x
      protocols. However, the exact size of each frame varies depending
      on the size of the variable fields.  The size of the variable
      field ranges from 0 to 2112-bytes as shown in the FC Frame Format
      in Fig. 3 resulting in the minimum size FC Frame of 36 bytes and
      the maximum size FC frame of 2148 bytes.

          | SOF  |Frame   |Optional   |  Frame      | CRC  |  EOF |
          | (4B) |Header  |Header     | Payload     | (4B) | (4B) |
          |      |(24B)   |<----------------------->|      |      |
          |      |        | Data Field = (0-2112B)  |      |      |

                            Fig. 3 FC Frame Format

      SOF and EOF Delimiters:

      On an FC link, Start-of-Frame (SOF) and End-Of-Frame (EOF) are
      called Ordered Sets and are sent as special words constructed from
      the 10-bit comma character (K28.5) followed by three additional
      10-bit data characters.  On a non-Fibre Channel link, the SOF and
      EOF delimiters are both byte-encoded and 4 bytes long.

      On an FC link the SOF delimiter serves to identify the beginning
      of a frame and prepares the receiver for frame reception.  The SOF

Rajagopal, et al.                                               [Page 7]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      contains information about the frame's Class of Service, position
      within a sequence, and in some cases, connection status.

      The EOF delimiter identifies the end of the frame and the final
      frame of a sequence.  In addition, it serves to force the running
      disparity to negative.  The EOF is used to end the connection in
      connection- oriented classes of service.

      It is therefore important to preserve the information conveyed by
      the delimiters across the IP-based network, so that the receiving
      FCIP device can correctly reconstruct the FC frame in its original
      SOF and EOF format before forwarding it to its ultimate FC
      destination on the FC link.

      When an FC frame is encapsulated and sent over a byte-oriented
      interface, the byte-encoded frame format is used. This format uses
      32- bit Ordered Set Code Words that consists of an Ordered Set
      Byte Code and 3 reserved bytes. The Start of Frame (SOF) and End
      of Frame (EOF) ordered set byte codes are defined in Annex A.  The
      three reserved bytes in the Ordered Set Code Word do not
      necessarily carry any meaningful information and should be set to

      Frame Header:

      The Frame Header is 24 bytes long and has several fields that are
      associated with the identification and control of the payload.
      Current FC Standards allow up to 3 Optional Header fields [4],

         - Network_Header (16-bytes)
         - Association_Header (32-bytes)
         - Device_Header (up to 64-bytes).

      Frame Payload:

      The FC Frame Payload is transparent to the FCIP device.  An FC
      application level payload is called an Information Unit at the
      FC-4 Level.  This is mapped into the Frame Payload of the FC
      Frame. A large Information Unit is segmented using a structure
      consisting of FC Sequences.  Typically, a Sequence consists of
      more than one FC frames. FCIP does not maintain any state
      information regarding the relationship of frames within a FC


      The CRC is 4 bytes long and uses the same 32-bit polynomial used

Rajagopal, et al.                                               [Page 8]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      in FDDI and is specified in ANSI X3.139 Fiber Distributed Data
      Interface. This CRC value is calculated over the entire FC header
      and the FC payload; it does not include the SOF and EOF

      Note: When FC frames are encapsulated into FCIP frames, the CRC is

   5.2 FC Frame Encapsulation

      Fig. 4 shows the mapping of the FC frame into an FCIP data frame
      which is then placed inside a TCP byte stream.  The FC-to-FCIP
      mapping (and the reverse mapping) is one-to-one since the maximum
      size of the encapsulated FC Frame, including the FC header fields,
      does not exceed 2148 bytes which fits entirely within the range of
      the FCIP length field.

                        |      Data Link Header      |
                        |          IP Header         |
                        |         TCP Header         |
                        |         FCIP Header        |
                        |           FC SOF           |
                        |          FC Header         |
                        |          FC Payload        |
                        |           FC CRC           |
                        |           FC EOF           |

                            Fig. 4 FCIP Data Frame

      TCP has a Maximum Segment Size (MSS) parameter which may be less
      than the maximum FCIP data frame size so it is possible that an
      FCIP data frame is divided by TCP into multiple TCP segments.
      FCIP always gives TCP contiguous FC frames.  TCP is free to
      segment its byte stream based on its MSS and operating conditions.
      On the receiving side, TCP is responsible for delivering an
      ordered byte-stream to the layer above it; FCIP is then

Rajagopal, et al.                                               [Page 9]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      responsible for assembling the segments into contiguous FC frames
      and delivering them to the ultimate destination FC devices.

      It is recommended that each FCIP data frame fits entirely within
      the TCP MSS to avoid fragmentation.  The maximum FCIP data frame
      size (2152 bytes) exceeds the MTU of some networks' physical
      layers (e.g. Ethernet MTU = 1518 bytes).  FCIP devices should
      handle the fragmentation and must handle the re-assembly of FCIP
      data frames.  An FCIP device may use Path MTU Discovery (RFC 1191)
      or an equivalent mechanism to adjust FCIP maximum data frame size
      to avoid fragmentation; this means that the maximum FC frame size
      may be set to match the path MTU of the IP- based network to which
      the FC network is connected.

   5.3 Values in relevant header fields

      IP Header Fields Setting:

      DSCP (6 bits): The recommended DSCP field setting is 101110
      corresponding to the EF service.

      Protocol (8 bits): This 8-bit field defines the higher level
      protocol that uses the service of the IP layer.  In this case,
      this is set to the TCP Protocol Value 6 as defined in [12].

      Source IP Address (32 bits):  This is the IP address of the
      ingress FCIP device that is transmitting the FCIP data frames.

      Destination IP Address (32 bits): This is the IP address of the
      egress FCIP device that is receiving the FCIP data frames.

      TCP Header fields setting:

      Source Port: The port number selected by the sending FCIP device.

      Destination Port: The well-known port number assigned for FC
      encapsulation.  This number is <to be assigned>

      Sequence Number: The sequence number of the first byte of the
      current TCP segment payload.

      Acknowledge Number: If the ACK bit is set, this field contains the
      sequence number of the next byte that the receiving TCP expects to

      Data Offset: The number of 32-bit words in the TCP header.  The
      use of TCP options are not precluded for FCIP devices.

Rajagopal, et al.                                              [Page 10]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      Control (flag) bits: TCP may set these bits according to the
      desired effects and to the state of the TCP connection in
      compliance to the TCP specifications.

      Window: TCP shall operate this field according to the state of the
      TCP connection in compliance to the TCP specifications.

      Checksum: The FCIP device shall calculate the TCP checksum over
      the entire FCIP data frame or the portion of the FCIP data frame
      encapsulated inside the current TCP segment.  The checksum shall
      also include the TCP header and the pseudo IP header as specified
      in [16].

      Urgent Pointer: The use of this field is not precluded for FCIP

      Options and Padding: FCIP devices are allowed to use TCP options
      and padding as specified by the TCP specifications [16].

   5.4 Fibre Channel Bit and Byte Ordering

      Fibre Channel's FC-1 Level defines the method used to encode data
      prior to transmission and subsequently decode the data upon
      reception.  The method encodes 8-bit bytes into 10-bit
      transmission characters to improve the transmission
      characteristics of the serial data stream. In Fibre Channel data
      fields are aligned on word boundaries.  A word in FC is defined as
      4 bytes or 32 bits.  The resulting transmission word after the 8-
      bit to 10-bit encoding consists of 40 bits.

      Data words or Ordered Sets (special FC-2 Level control words) from
      the FC-2 Level map to the FC-1 Level with no change in order and
      the bytes in the word are transmitted in the Most Significant Byte
      first to Least Significant Byte order.  The transmission order of
      bits within each byte is the Least Significant Bit to the Most
      Significant Bit.

6. FCIP Network

   6.1 FC Backbone Switches

      Fibre Channel Standards [3] describe the operation and interaction
      of FC Switches.  Two distinct levels of switch interconnections
      are specified. Autonomous Regions (AR) are defined to allow
      clusters of FC Switches to be connected across a backbone network
      called an FSPF-backbone.  An AR is administratively defined with
      each AR encompassing one or more FC Domains.  The FSPF-backbone
      network is formed from one or more Backbone Switches (BSW) that

Rajagopal, et al.                                              [Page 11]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      run the FSPF-backbone routing protocol.  The FSPF- backbone
      routing protocol is based on OSPF and the FSPF backbone may
      consist of an arbitrary mesh network.  A BSW may communicate with
      multiple neighbors.  As specified in [3], native FC frames
      traverse the  FSPF backbone between BSW neighbors.  The FSPF-
      backbone Routing Protocol messages are exchanged between BSWs on
      the FSPF backbone.

      An example network consisting of 4 ARs and an FSPF backbone
      consisting of 3 links is given in Fig. 5.  There is no restriction
      in adding other links to this network as needed. The connection
      between BSWs below may in fact form a fully connected mesh.

       _______                                           _______
      |       |                                         |       |
      | AR #1 |_____                               _____| AR #4 |
      |_______|     |                             |     |_______|
                 ___|___                       ___|___
                | BSW 1 |---------------------| BSW 4 |
                |_______|                     |_______|
                 ___|___                       _______
                | BSW 2 |---------------------| BSW 3 |
                |_______|                     |_______|
       ___ ___      |                             |      _______
      |       |     |                             |     |       |
      | AR #2 |-----                               -----| AR #3 |
      |_______|                                         |_______|


      BSW 1 knows it is connected to BSWs 2 and 4;
      BSW 2 knows it is connected to BSWs 1 and 3;
      BSW 4 knows it is connected to BSWs 1.

      Fig. 5 Example Network showing FSPF Backbone Switching Architecture

      An FCIP device provides a single logical interface to the FSPF-
      backbone protocol connecting multiple BSW neighbors on the IP
      network.  From the FSPF-backbone routing point of view, the
      connection to each neighbor on the IP network is treated as a
      separate logical FC link.

      In FCIP, the native FC frames are first encapsulated in TCP
      segments which then traverse the IP-based network.  The IP network
      provides a new transport path for each emulated FSPF-backbone

Rajagopal, et al.                                              [Page 12]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      The IP network itself may consist of any number of hops between
      two FCIP devices.  Also, the route taken by the IP packet between
      any two FCIP devices is dictated by normal IP routing.

      A functional and logical diagram of an IP-based FSPF-backbone for
      the example network given in Fig. 4 is shown in Fig. 6.  In this
      figure, each BSW is logically connected to other BSWs.

     _______                                              _______
    |       |                                            |       |
    | AR #1 |---                                         | AR #4 |
    |_______|   |         ______    ________    ______   |_______|
              __|_ __    |      |  |        |  |      |   ___|___
             | BSW 1 |---| FCIP |--|   IP   |--| FCIP |--| BSW 4 |
             |_______|   |______|  | Network|  |______|  |_______|
                                   |        |
                         ______      |   |    ______
              ______    |      |     |   |   |      |    _______
             | BSW 2|---| FCIP |-----|   |---| FCIP |---| BSW 3 |
             |______|   |______|             |______|   |_______|
     ________   |                                        ___|___
    |        |  |                                       |       |
    |  AR #2 |__|                                       | AR #3 |
    |________|                                          |_______|

   Fig. 6 Example Network showing an IP-based FC Backbone Switching

      The IP-based network has transformed the FSPF-backbone into a
      fully connected network.  From the perspective of each BSW all
      remote BSWs therefore appear to be neighbors.  The FSPF-backbone
      routing protocol computation would make the IP based network
      topology appear as a fully connected mesh.

      The FSPF-backbone routing protocol exchanges between BSWs occur
      transparently to the FCIP devices.  Encapsulated FC frames are
      routed on the IP network according to the normal IP routing
      procedures.  In this mode, the FSPF-backbone routing protocol lays
      over the IP network and has no knowledge of the underlying IP
      protocol and IP routing or the underlying technology that carries
      the IP datagram.  This concept is shown in Fig. 7.

Rajagopal, et al.                                              [Page 13]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

     ________                                             _______
    |  AR #1 |                                           | AR #2 |
    |        |--                                         |       |
    |________|  |        ______    ________    _____     |_______|
              __|___    |      |  |        |  |     |   ____|__
             | BSW 1|---| FCIP |--|   IP   |--|FCIP |--| BSW 2 |
             |      |   |      |  | Network|  |     |  |       |
             |______|   |______|  |________|  |_____|  |_______|
                                  IP Routing
                           FSPF-backbone Routing Plane

         Note: IP Network routing may consist of multiple paths

        Fig. 7 FC packet routing over IP based backbone network

   6.2 FC Device

      The protocol encapsulation and mapping of the FC frame described
      in earlier sections applies equally to any pair of FC devices
      (e.g. switch-to-switch or host-to-storage subsystem) wishing to
      tunnel FC frames across an IP-based network.  Any FC routing
      protocol exchanges may still occur transparently to the FCIP
      devices.  It should be noted that Fibre Channel Primitive
      Sequences and Primitives are not exchanged between FCIP devices.

7. Security Considerations

   Using a wide-area, general purpose network such as an IP internet in
   a position normally occupied by physical cabling introduces some
   security problems not normally encountered in Fibre Channel storage
   networks. Normal media are typically protected physically from
   outside access; IP internets typically invite outside access.

   The general effect is that the security of the entire Fibre Channel
   internetwork is only as good as the security of the entire IP
   internet through which it tunnels.  The following broad classes of
   attacks are possible:

   1. Unauthorized Fibre Channel controllers can gain access to
      resources through normal Fibre Channel processes.

   2. Unauthorized agents can monitor and manipulate Fibre Channel
      traffic flowing over physical media used by the IP internet and
      under control of the agent.

Rajagopal, et al.                                              [Page 14]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

   To a large extent, these security risks are typical of the risks
   facing any other application using an IP internet.  They are
   mentioned here only because Fibre Channel storage networks are not
   normally suspicious of the media.  Fibre Channel storage network
   administrators will need to be aware of these additional security

   Security protocols and procedures used in other IP applications may
   be used for FCIP.

   For Virtual Private Networks, both authentication and encryption are
   generally desired, because it is important both to (1) assure that
   unauthorized users do not penetrate the virtual private network and
   (2) assure that eavesdroppers on the network cannot read messages
   sent over the network.

   Note: Use of the IPSec protocol suite is optional.

8. Data Integrity Considerations

   8.1 Loss

      Recovery from data loss due to IP datagram loss is provided via
      the TCP reliable delivery mechanism.   Note: Due to varying TCP
      timeouts, competing FC and TCP recovery schemes is a possibility.
      This issue is addressed in section 8.4.

   8.2 Corruption

      Data corruption is detected at two different levels: TCP checksum
      and Fibre Channel CRC.  Data corruption detected at the TCP level
      shall be recovered via TCP reliable data recovery mechanisms.
      Data corruption detected at the Fibre Channel level shall be
      handled within the Fibre Channel end nodes.

   8.3 Timeouts

      Fibre Channel has two important timeouts to consider in FCIP.
      These are: ED_TOV, and R_A_TOV.

      ED_TOV determines the life of an individual Fibre Channel frame in
      any particular fabric element.  The effects of ED_TOV on the
      fabric as a whole are typically cumulative since each fabric
      element contains it's own ED_TOV timers for any frame received.

      R_A_TOV determines the life of an individual Fibre Channel frame
      in the fabric as a whole.   For a fabric, R_A_TOV implies that no
      particular frame will remain in (and thus be emitted from) the

Rajagopal, et al.                                              [Page 15]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      fabric after the timer expires.

      TCP has a TCP acknowledgement timeout.  This is a variable
      timeout.  <TBD: Need to elaborate on TCP timeouts and define how
      Fibre Channel timeouts map to TCP timeouts.>

   8.4 Recovery Mechanisms

      When an FCIP data  frame is transported over an IP network, there
      is a possibility of the frame's getting dropped.  This can happen
      if there is congestion along the path within the IP network or if
      there are no empty buffers available on one of the incoming ports,
      due to bit errors, etc. When this happens, the TCP acknowledgement
      will not be received by the source, and normal TCP retry
      mechanisms will be activated.

      An issue may arise during these recovery mechanisms, since TCP
      timeout is variable, and may exceed Fibre Channel FC
      ED_TOV/R_A_TOV timeouts.

      <Placeholder: Need to add details on how to handle this here.>

9. Performance Considerations

   The FCIP protocol does not crack the FC Frame (except for attaching
   the correct byte-encoded SOF and EOF) nor does it do any FC payload
   processing.  This allows any FC traffic to be tunneled across at high
   throughput rates.

   In the case where there is no data link fragmentation, each FC frame
   has a one-to-one mapping to a TCP segment;  the TCP segment has a
   one-to-one mapping to an IP datagram; and the IP datagram has a one-
   to-one mapping to a data link frame. This has the tendency to further
   improve the throughput performance.

   Note:  Class 1 FC traffic expects dedicated bandwidth. This
          specification does not address this requirement.

   The Flow Control Protocol (discussed in the next section) provides
   the ability to stream gigabit FC data when using a large window size.

   9.1 QoS Support

      The Differentiated Services Architecture (diffserv) provides a
      "Class of Service" to a flow aggregate [6], [17]. At so-called
      diffserv boundaries, IP packets are classified and marked. Within
      the diffserv domain, resources û bandwidth and buffers û are
      allocated for each classification. Packets with the same

Rajagopal, et al.                                              [Page 16]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      classification use the resources allocated for the classification.
      IP packets with the same destination and class marking exit a
      diffserv capable router in the same order they arrived. IP with
      the same destination but different class markings exit according
      to priorities assigned to the different class markings. The
      Diffserv has renamed the Ipv4 TOS field as Differentiated Services
      Code Point (DSCP). The DSCP indicates the particular behavior a
      packet is to receive at each router. How a packet gets marked is
      based on a policy administered and configured into the network.
      [18] and [19] provide various encodings of the DSCP field to
      achieve a specific behavior from the routers. There may be several
      ways to administer the policies and the policy definition is up to
      the network provider. That is one network provider may choose to
      mark all packets going from one source IP address to a specific
      destination as "high priority", while another might mark just a
      specific traffic type (e,g., HTTP) as "high priority". Thus
      packets carry the desired class information and each diffserv-
      capable router treats the packet according to the information in
      its DSCP field. This is referred to as Per Hop Behavior (PHB).
      Currently, the IETF standards define essentially 3 types of
      services: Expedited Forwarding (EF) [18], Assured Forwarding (AF)
      [19], or Default Forwarding (DF) [6], [17]û that corresponds to
      its DSCP.

      [17] specifies the AF service AF PHB provides a way to prioritize
      best- effort traffic. Currently, 4 AF classes and 3 drop
      precedence levels are specified providing 12 different levels of
      forwarding assurances. The DSCP value specifies a drop-order in
      the event that a packet experiences congestion at a subsequent
      diffserv router.

      [18] specifies the DSCP code point equal to 101110  EF service
      which is also sometimes refereed to as "Premium" service. When
      supported, this class behavior has the lowest levels of latency,
      packet loss, and delay variation. This service behavior most
      closely matches the Fibre Channel characteristics. This is
      therefore the recommended  DSCP setting in the IP DSCP field.

      What resources are not used for EF and AF are left for the DF
      services which is really a best-effort service.

      Note that if a packet is being forwarded over an underlying
      network without diffserv support, then the packet would simply
      receive best- effort service regardless of its DSCP field setting.

10. Flow Control and Congestion Management

Rajagopal, et al.                                              [Page 17]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

      <Text to be added>

11. References:

     [1] Bradner, S., "The Internet Standards Process -- Revision 3",
         BCP 9, RFC 2026, October 1996.

     [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement
         Levels", BCP 14, RFC 2119, March 1997

     [3] NCITS 321-200x (ANSI) T11/Project 1305-D/Rev 4.9 "Fibre Channel
         Switch-Fabric-2", (FC-SW-2) November 14, 2000 (www.t11.org)

     [4] Fibre Channel Physical and Signaling Interface-3 (FC-PH-3), Rev.
         9.4, ANSI X3.303-1998

     [5] The Fibre Channel Consultant: A Comprehensive Introduction,
         "Robert W. Kembel", Northwest Learning Associates, 1998

     [6] Nichols, K., Blake, S., Baker, F.  and D. Black, " Definition
         of the Differentiated Services Field (DS Field) in the IPv4 and
         Ipv6 Headers", RFC 2474, December 1998.

     [7] NCITS T11/Project 1238-D/Rev4.7 "Fibre Channel Backbone", (FC-BB)
         June 8, 2000 (www.t11.org)

     [8] Kent, S. and Atkinson, R., "Security Architecture for the
         Internet Protocol", RFC 2401, Nov 1998

     [9] Kent, S. and Atkinson, R., "IP Authentication Header",
         RFC 2402, Nov 1998

     [10] Kent, S. and Atkinson, R., "IP Encapsulating Security
          Payload (ESP)", RFC 2406, Nov 1998

     [11] Maughan, D. et all, "Internet Security Association and Key
          Management Protocol (ISAKMP)", RFC 2408, Nov 1998

     [12] http://www.isi.edu/in-notes/iana/assignments/protocol-numbers

     [13] http://www.t11.org

     [14] Fibre Channel Physical and Signaling Interface (FC-PH), Rev 4.3,
          ANSI X3.230-1994.

     [15] Fibre  Channel  NCITS  321-200x (ANSI) T11/Project  1356-D/Rev4.3 "
          Fibre Channel - Generic  Services 3", June 2000 (www.t11.org)).

Rajagopal, et al.                                              [Page 18]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

     [16] ISI, "Transmission Control Protocol", RFC 793, Sep 1981

     [17] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
          Weiss, W., "An Architecture for Differentiated Services",
          RFC 2475, Dec 1998

     [18] Jacobson, V., Nichols, K., Poduri, K., "An Expedited
          Forwarding PHB Group", RFC 2598, June 1999

     [19] Heinanen, J., Baker, F., Weiss, W.,  Wroclawski, J., "An
          Assured Forwarding PHB", RFC 2598, June 1999

12. Acknowledgments

13. Authors' Addresses

     Murali Rajagopal
     LightSand Communications, Inc.
     375 Los Coches St.
     Milpitas, CA 95035

     Phone: +1 408 404 3164
     Email: muralir@lightsand.com

     Raj Bhagwat
     LightSand Communications, Inc.
     375 Los Coches St.
     Milpitas, CA 95035

     Phone: +1 408 404 3194
     Email: rajb@lightsand.com

     Elizabeth G. Rodriguez
     Lucent Technologies
     1202 Richardson Drive, Suite 210
     Richardson, TX 75080

     Phone: +1 972 231 0672
     Fax: +1 972 671 5476
     Email: egrodriguez@lucent.com

     Vi Chau
     Gadzoox Networks, Inc.
     16241 Laguna Canyon Road, Suite 100

Rajagopal, et al.                                              [Page 19]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

     Irvine, CA 92618

     Phone: +1 949 789 4639
     Fax: +1 949 453 1271
     Email: vchau@gadzoox.com

     Stuart Berman
     Vixel Corporation
     15245 Alton Parkway Suite 100
     Irvine, CA 92618

     Phone: +1 949 450 6100
     Fax: +1 949 753 9500
     Email: sberman@vixel.com

     Steve Wilson
     Brocade Communications Systems, Inc.
     1745 Technology Drive
     San Jose, CA. 95110
     Phone: 408-487-8128
     Fax: 408-487-8101
     email: swilson@brocade.com

     Michael E. O'Donnell
     McDATA Corporation
     310 Interlocken Parkway
     Broomfield, Co. 80021

     Phone: +1 303 460 4142
     Fax: +1 303 465 4996
     Email: modonnell@mcdata.com

     Craig W. Carlson
     QLogic Corporation
     6321 Bury Drive
     Eden Prairie, MN 55346

     Phone: +1 952 932 4064
     Email: craig.carlson@qlogic.com

ANNEX A: Fibre Channel EOF and SOF Byte Encodings

Rajagopal, et al.                                              [Page 20]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

   A.1 Ordered Sets

      On a Fibre Channel link, Ordered Sets (OS) are sent as special
      out-of- band words constructed of the 10-bit comma character
      (K28.5) followed by 3 additional 10-bit data characters.  The
      Ordered Sets defined by Fibre Channel include the Frame
      Delimiters, Start of Frame (SOF) and End of Frame (EOF), other
      Primitive Signals (IDLE, R_RDY, ARB, OPN, CLS, MRK), and Primitive
      Sequences (OLS,NOS, LR, LRR, LIP, LPB, LPE).

      When Fibre Chanel frames are encapsulated in a byte-oriented
      transport data packet, the Byte-encoded frame format is used. The
      Byte-encoded frame format uses 32-bit Ordered Set (OS) Code Words
      to represent valid Fibre channel frame delimiters. This format
      uses a single-byte OS Code to represent each FC Ordered Set.

      FCIP makes use of the OS Codes defined in Annex A of [7] for the
      frame delimiters. SOF and EOF codes defined in the figures (see
      below) in this Annex are inserted into the FC frame.

      Primitive Signals and Primitive Sequences are stripped at the FCIP
      devices and not sent between FCIP devices over the IP network.

      The frame delimiters are identified by their position. An
      encapsulated Byte-encoded frame must use the corresponding 32-bit
      OS Code Word as the first and last words in the encapsulated PDU.

      Fibre Channel frame delimiters shall be encoded in the format
      shown in table below.

   |Wrd|    <31:24>     |    <23:16>     |    <15:08>     |   <07:00>   |
   |0  |  OS Code       |                  Reserved                     |

                    Table 1. Frame Delimiter Format

   A.2 Encoded FC Frame Delimiters

      The SOF OS-codes are a single byte encoding of the SOF Ordered
      Set. The first word in an encapsulated Byte-encoded FC frame shall
      map the SOF Ordered Set to the corresponding 32-bit OS Code Word.

      The EOF OS-codes are a single byte encoding of the EOF Ordered
      Set. The last word in an encapsulated Byte-encoded FC frame shall
      map the the EOF Ordered Set to the corresponding 32-bit OS Code

Rajagopal, et al.                                              [Page 21]

Internet-Draft         Fibre Channel over TCP/IP           November 2000


                     |     OS-Code     | Delimiter Name |
                     |      (hex)      |                |
                     |      0x28       |     SOFf       |
                     |      0x3F       |     SOFc1      |
                     |      0x2F       |     SOFi1      |
                     |      0x37       |     SOFn1      |
                     |      0x3D       |     SOFc2      |
                     |      0x2D       |     SOFi2      |
                     |      0x35       |     SOFn2      |
                     |      0x3E       |     SOFc3      |
                     |      0x2E       |     SOFi3      |
                     |      0x36       |     SOFn3      |
                     |      0x39       |     SOFc4      |
                     |      0x29       |     SOFi4      |
                     |      0x31       |     SOFn4      |
                     |      0x38       |     SOFcf      |
                     |      0x30       |     SOFnf      |
                     |      0x41       |     EOFn       |
                     |      0x42       |     EOFt       |
                     |      0x46       |     EOFdt      |
                     |      0x44       |     EOFrt      |
                     |      0x49       |     EOFni      |
                     |      0x4E       |     EOFdti     |

Rajagopal, et al.                                              [Page 22]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

                     |      0x4F       |     EOFrti     |
                     |      0x50       |     EOFa       |

                        Table 2. Delimiter Byte-codes

ANNEX B: Relationship between FCIP and IP over FC (IPFC)

   IPFC (RFC 2625) describes the encapsulation of IP packets in FC
   frames. It is intended to facilitate IP communication over an FC

   FCIP describes the encapsulation of FC frames in TCP segments which
   in turn are encapsulated inside IP packets for transporting over an
   IP network.  It gives no consideration to the type of FC frame that
   is being encapsulated.  Therefore, the FC frame may actually contain
   an IP packet as described in the IP over FC specification (RFC 2625).
   In such a case, the data packet would have:

          Data Link Header
          IP Header
          TCP Header
          FCIP Header
          FC Header
          IP Header

   Note:   The two IP headers would not be identical to each other.  One
           would have information pertaining to the final destination
           while the other would have information pertaining to the FCIP

   The two documents focus on different objectives.  As mentioned above,
   implementation of FCIP will lead to IP encapsulation within IP. While
   perhaps inefficient, this should not lead to issues with IP
   communication.  One caveat: if a Fibre Channel device is
   encapsulating IP packets in an FC frame (e.g. an IPFC device), and
   that device is communicating with a device running IP over a non-FC
   medium, a second IPFC device will need to act as a gateway between
   the two networks. This scenario is not specifically addressed by

   There is nothing in either of the specifications to prevent a single
   device from implementing both FCIP and IP-over-FC (IPFC), but this is
   implementation specific, and is beyond the scope of this document.

Full Copyright Statement

Rajagopal, et al.                                              [Page 23]

Internet-Draft         Fibre Channel over TCP/IP           November 2000

Copyright (C) The Internet Society (1999).  All Rights Reserved.

This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it or
assist in its implementation may be prepared, copied, published and
distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are included
on all such copies and derivative works.  However, this document itself
may not be modified in any way, such as by removing the copyright notice
or references to the Internet Society or other Internet organizations,
except as needed for the purpose of developing Internet standards in
which case the procedures for copyrights defined in the Internet
Standards process must be followed, or as required to translate it into
languages other than English.

The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS


Funding for the RFC Editor function is currently provided by the
Internet Society.

[This INTERNET DRAFT expires on May 29, 2001]

Rajagopal, et al.                                              [Page 24]

Html markup produced by rfcmarkup 1.129b, available from https://tools.ietf.org/tools/rfcmarkup/