Internet Draft                                               L. Yang
        Expiration: June Dec 2003                                     Intel Corp.
        File: draft-ietf-forces-framework-04.txt draft-ietf-forces-framework-05.txt                    R. Dantu
        Working Group: ForCES                           Univ. of North Texas Dallas
                                                                 T. Anderson
                                                                 Intel Corp.
                                                               December 2002
                                                                    R. Gopal
                                                                       Nokia
                                                                  June 2003

            Forwarding and Control Element Separation (ForCES) Framework

                        draft-ietf-forces-framework-04.txt

                        draft-ietf-forces-framework-05.txt

     Status of this Memo

        This document is an Internet-Draft and is in full conformance with
        all provisions of Section 10 of RFC2026.  Internet-Drafts are
        working documents of the Internet Engineering Task Force (IETF), its
        areas, and its working groups.  Note that other groups may also
        distribute working documents as Internet-Drafts.

        Internet-Drafts are draft documents valid for a maximum of six
        months and may be updated, replaced, or obsoleted by other documents
        at any time.  It is inappropriate to use Internet-Drafts as
        reference material or to cite them other than as ``work in
        progress.''

        The list of current Internet-Drafts can be accessed at
        http://www.ietf.org/ietf/1id-abstracts.txt.

        The list of Internet-Draft Shadow Directories can be accessed at
        http://www.ietf.org/shadow.html.

     Copyright Notice

        Copyright (C) The Internet Society (2002).  All Rights Reserved.

     Abstract

        This document defines the architectural framework for the ForCES
        (Forwarding and Control Element Separation) network elements, and
        identifies the associated entities and the interaction among them.

     Table of Contents
        1. Definitions.....................................................2 Definitions.....................................................3
        2. Introduction to Forwarding and Control Element Separation
        (ForCES)...........................................................4
        (ForCES)...........................................................5
        3. Architecture....................................................7 Architecture....................................................8
           3.1. Control Elements and Fr Reference Point....................8 Point....................9
           3.2. Forwarding Elements and Fi reference point.................9 point................10
           3.3. CE Managers...............................................11 Managers...............................................13
           3.4. FE Managers...............................................11 Managers...............................................13
        4. Operational Phases.............................................12 Phases.............................................13
           4.1. Pre-association Phase.....................................12 Phase.....................................13
              4.1.1. Fl Reference Point...................................12 Point...................................13
              4.1.2. Ff Reference Point...................................13 Point...................................14
              4.1.3. Fc Reference Point...................................14 Point...................................15
           4.2. Post-association Phase and Fp reference point.............14 point.............15
              4.2.1. Proximity and Interconnect between CEs and FEs.......14 FEs.......15
              4.2.2. Association Establishment............................14 Establishment............................16
              4.2.3. Steady-state Communication...........................15 Communication...........................17
              4.2.4. Data Packets across Fp reference point...............16 point...............18
              4.2.5. Proxy FE.............................................17 FE.............................................19
           4.3. Association Re-establishment..............................17 Re-establishment..............................19
              4.3.1. CE graceful restart..................................19
              4.3.2. FE restart...........................................20
        5. Applicability to RFC1812.......................................18 RFC1812.......................................21
           5.1. General Router Requirements...............................19 Requirements...............................22
           5.2. Link Layer................................................20 Layer................................................23
           5.3. Internet Layer Protocols..................................20 Protocols..................................23
           5.4. Internet Layer Forwarding.................................21 Forwarding.................................24
           5.5. Transport Layer...........................................22 Layer...........................................24
           5.6. Application Layer -- Routing Protocols....................22 Protocols....................25
           5.7. Application Layer -- Network Management Protocol..........22 Protocol..........25
        6. Summary........................................................23 Summary........................................................25
        7. Normative References...........................................23 Security Considerations........................................26
           7.1. Analysis of Potential Threats Introduced by ForCES........26
              7.1.1. Join or Remove Flooding on CEs...................26
              7.1.2. Impersonation Attack.................................27
              7.1.3. Replay Attack........................................27
              7.1.4. Attack during Fail Over..............................27
              7.1.5. Data Integrity.......................................27
              7.1.6. Data Confidentiality.................................27
              7.1.7. Sharing security parameters..........................28
              7.1.8. Denial of Service Attack via External Interface......28
           7.2. Security Recommendations for ForCES.......................28
              7.2.1. Security Configuration...............................29
              7.2.2. Using TLS with ForCES................................29
              7.2.3. Using IPsec with ForCES..............................30
        8. Informative References.........................................23 Normative References...........................................31
        9. Security Considerations........................................24 Informative References.........................................32
        10. Acknowledgments...............................................24 Acknowledgement...............................................32
        11. Authors' Addresses............................................24 Addresses............................................33
        12. Intellectual Property Right...................................24 Right...................................33
        13. Full Copyright Statement......................................24 Statement......................................33

     Yang, et. al.      Expires Dec 2003                  [Page 2] 
     Conventions used in this document

        The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
        "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in
        this document are to be interpreted as described in [RFC-2119].

     1. Definitions

        A set of terminology associated with the ForCES requirements is
        defined in [3] and we only include the ones that are most relevant
        to this document here.

        Addressable Entity (AE) -                                - An entity that is directly addressable
        given some interconnect technology.  For example, on IP networks, it
        is a device to which we can communicate using an IP address; and on
        a switch fabric, it is a device to which we can communicate using a
        switch fabric port number.

        Physical Forwarding Element (PFE) - An AE that includes hardware
        used to provide per-packet processing and handling.  This hardware
        may consist of (but is not limited to) network processors, ASICs, or
        general processors, installed on line cards, daughter boards,
        mezzanine cards, or in stand-alone boxes.

        PFE Partition - A logical partition of a PFE consisting of some
        subset of each of the resources (e.g., ports, memory, forwarding
        table entries) available on the PFE.  This concept is analogous to
        that of the resources assigned to a virtual switching element as
        described in [8].

        Physical Control Element (PCE) - An AE that includes hardware used
        to provide control functionality.  This hardware typically includes
        a general-purpose processor.

        PCE Partition - A logical partition of a PCE consisting of some
        subset of each of the resources available on the PCE.

        Forwarding Element (FE) - A logical entity that implements the
        ForCES protocol.  FEs use the underlying hardware to provide per-
        packet processing and handling as directed by a CE via the ForCES
        protocol.

     Yang, et. al.      Expires June 2003                  [Page 2]  FEs may happen to be a single blade (or PFE), a partition
        of a PFE or multiple PFEs.

        Control Element (CE) - A logical entity that implements the ForCES
        protocol and uses it to instruct one or more FEs how to process
        packets.  CEs handle functionality such as the execution of control
        and signaling protocols.  CEs may consist of PCE partitions or whole
        PCEs.

        ForCES Network Element (NE) - An entity composed of one or more CEs
        and one or more FEs.  To entities outside a NE, the NE represents a
        single point of management.  Similarly, a NE usually hides its
        internal organization from external entities.

     Yang, et. al.      Expires Dec 2003                  [Page 3] 
        Pre-association Phase - The period of time during which a FE Manager
        (see below) and a CE Manager (see below) are determining which FE
        and CE should be part of the same network element.

        Post-association Phase - The period of time during which a FE does
        know which CE is to control it and vice versa, including the time
        during which the CE and FE are establishing communication with one
        another.

        ForCES Protocol - While there may be multiple protocols used within
        the overall ForCES architecture, the term "ForCES protocol" refers
        only to the ForCES post-association phase protocol (see below).

        ForCES Post-Association Phase Protocol - The protocol used for post-
        association phase communication between CEs and FEs.  This protocol
        does not apply to CE-to-CE communication, FE-to-FE communication, or
        to communication between FE and CE managers.  The ForCES protocol is
        a master-slave protocol in which FEs are slaves and CEs are masters.
        This protocol includes both the management of the communication
        channel (e.g., connection establishment, heartbeats) and the control
        messages themselves.  This protocol could be a single protocol or
        could consist of multiple protocols working together.

        FE Manager - A logical entity that operates in the pre-association
        phase and is responsible for determining to which CE(s) a FE should
        communicate.  This process is called CE discovery and may involve
        the FE manager learning the capabilities of available CEs.  A FE
        manager may use anything from a static configuration to a pre-
        association phase protocol (see below) to determine which CE(s) to
        use.
        use, however this is currently out of scope.  Being a logical
        entity, a FE manager might be physically combined with any of the
        other logical entities mentioned in this section.

        CE Manager - A logical entity that operates in the pre-association
        phase and is responsible for determining to which FE(s) a CE should
        communicate.  This process is called FE discovery and may involve
        the CE manager learning the capabilities of available FEs.  A CE
        manager may use anything from a static configuration to a pre-
        association phase protocol (see below) to determine which FE to use. use,
        however this is currently out of scope.   Being a logical entity, a
        CE manager might be physically combined with any of the other
        logical entities mentioned in this section.

     Yang, et. al.      Expires June 2003                  [Page 3]

        Pre-association Phase Protocol - A protocol between FE managers and
        CE managers that is used to determine which CEs or FEs to use.  A
        pre-association phase protocol may include a CE and/or FE capability
        discovery mechanism.  Note that this capability discovery process is
        wholly separate from (and does not replace) that used within the
        ForCES protocol.  However, the two capability discovery mechanisms
        may utilize the same FE model.

     Yang, et. al.      Expires Dec 2003                  [Page 4] 
        FE Model - A model that describes the logical processing functions
        of a FE.

        ForCES Protocol Element - A FE or CE.

     2. Introduction to Forwarding and Control Element Separation (ForCES)

        An IP network element (NE) appears to external entities as a
        monolithic piece

        FE Topology -- Representation of how the multiple FEs in a single NE
        are interconnected.  Sometimes it is called inter-FE topology, to be
        distinguished from intra-FE topology (or FE block topology) used by
        FE model.

        Inter-FE topology - see FE Topology.

     2. Introduction to Forwarding and Control Element Separation (ForCES)

        An IP network element (NE) appears to external entities as a
        monolithic piece of network equipment, e.g., a router, NAT,
        firewall, or load balancer.  Internally, however, an IP network
        element (NE) (such as a router) is composed of numerous logically
        separated entities that cooperate to provide a given functionality
        (such as routing).  Two types of network element components exist:
        control element (CE) in control plane and forwarding element (FE) in
        forwarding plane (or data plane).  Forwarding elements typically are
        ASIC, network-processor, or general-purpose processor-based devices
        that handle data path operations for each packet.  Control elements
        are typically based on general-purpose processors that provide
        control functionality like routing and signaling protocols.

        ForCES aims to define a framework and associated protocol(s) to
        standardize information exchange between the control and forwarding
        plane.  Having standard mechanisms allows CEs and FEs to become
        physically separated standard components.  This physical separation
        accrues several benefits to the ForCES architecture.  Separate
        components would allow component vendors to specialize in one
        component without having to become experts in all components.
        Standard protocol also allows the CEs and FEs from different
        component vendors to interoperate with each other and hence it
        becomes possible for system vendors to integrate together the CEs
        and FEs from different component suppliers.  This interoperability
        translates into a lot more design choices and flexibility to the
        system vendors.  Overall, ForCES will enable rapid innovation in
        both the control and forwarding planes while maintaining
        interoperability.  Scalability is also easily provided by this
        architecture in that additional forwarding or control capacity can
        be added to existing network elements without the need for forklift
        upgrades.

             -------------------------       -------------------------
             |  Control Blade A      |       |  Control Blade B      |
             |       (CE)            |       |          (CE)         |
             -------------------------       -------------------------
                     ^   |                           ^    |
                     |   |                           |    |
                     |   V                           |    V

     Yang, et. al.      Expires Dec 2003                  [Page 5] 
             ---------------------------------------------------------
             |               Switch Fabric Backplane                 |
             ---------------------------------------------------------
                    ^  |            ^  |                   ^  |
                    |  |            |  |                  |  |
                    |  V            |  V                   |  V
                ------------    ------------           ------------
                |Router    |    |Router    |           |Router    |
                |Blade #1  |    |Blade #2  |           |Blade #N  |
                |   (FE)   |    |   (FE)   |           |   (FE)   |
                ------------    ------------           ------------
                    ^  |            ^  |                   ^  |
                    |  |            |  |                  |  |
                    |  V            |  V                   |  V

             Figure 1. A router configuration example with separate blades.

        One example of such physical separation is at the blade level.
        Figure 1 shows such an example configuration of a router, with two
        control blades and multiple router (forwarding) blades, all
        interconnected into a switch fabric backplane.  In such chassis
        configuration, the control blades are the CEs while the router
        blades are FEs, and the switch fabric backplane provides the

     Yang, et. al.      Expires June 2003                  [Page 4]
        physical interconnect for all the blades.  Control blade A may be
        the primary CE while control blade B is the backup CE providing
        redundancy.  It is also possible to have a redundant switch fabric
        for high availability support.  Routers today with this kind of
        configuration use proprietary interface for messaging between CEs
        and FEs.  The goal of ForCES is to replace such proprietary
        interface with a standard protocol.  With a standard protocol like
        ForCES implemented on all blades, it becomes possible for control
        blades from vendor X and routing blades from vendor Y to work
        seamlessly together in one chassis.

             -------------------------       -------------------------
             |  Control Blade A      |       |  Control Blade B      |

             -------         -------
             |       (CE) CE1 |         |          (CE) CE2 |
             -------------------------       -------------------------
             -------         -------
                ^   |               ^
                |               |   |
                V               V
         ============================================ Ethernet
             ^       ^              ^
             |       |               |
             V                           |       V
             ---------------------------------------------------------               V
          -------  -------         --------
          |               Switch Fabric Backplane FE#1|  |
             --------------------------------------------------------- FE#2|         | FE#n |
          -------  -------         --------
            ^  |     ^  |            ^  |
            |  |     |  |                  |  |
                    |  V            |  V                   |  V
                ------------    ------------           ------------
                |Router    |    |Router    |           |Router    |
                |Blade #1  |    |Blade #2  |           |Blade #N  |
                |   (FE)   |    |   (FE)   |           |   (FE)   |
                ------------    ------------           ------------
                    ^  |            ^  |                   ^  |
                    |  |            |  |                      |  |
            |  V     |  V            |  V

             Figure 1. 2. A router configuration example with separate blades.

             -------         -------
             | CE1 |         | CE2 |
             -------         -------
                ^               ^
                |               |
                V               V
         ============================================ Ethernet
             ^       ^              ^
             |       |               |
             V       V               V
          -------  -------         --------
          | FE#1|  | FE#2|         | FE#n |
          -------  -------         --------
            ^  |     ^  |            ^  |
            |  |     |  |            |  |
            |  V     |  V            |  V boxes.

     Yang, et. al.      Expires June Dec 2003                  [Page 5] 
             Figure 2. A router configuration example with separate boxes. 6] 
        Another level of physical separation between the CEs and FEs can be
        at the box level.  In such configuration, all the CEs and FEs are
        physically separated boxes, interconnected with some kind of high
        speed LAN connection (like Gigabit Ethernet).  These separated CEs
        and FEs are only one hop away from each other within a local area
        network.  The CEs and FEs communicate to each other by running
        ForCES, and the collection of these CEs and FEs together become one
        routing unit to the external world. Figure 2 shows such an example.

        In both examples shown here, the same physical interconnect is used
        for both CE-to-FE and FE-to-FE communication.  However, that does
        not have to be the case.  One reason to use different interconnect
        is that CE-to-FE interconnect does not have to be as fast as the FE-
        to-FE interconnect, so the more expensive fast connections can be
        saved for FE-to-FE only.  The separate interconnects may also
        provide reliability and redundancy benefits for the NE.

             -------------------------------------------------
             |       |       |       |       |       |       |
             |OSPF   |RIP    |BGP    |RSVP   |LDP    |      |
             |       |       |       |       |       |       |
             -------------------------------------------------
             |               ForCES Interface                |
             -------------------------------------------------
                                     ^   ^
                             ForCES  |   |data
                             control |   |packets
                             messages|   |(e.g., routing packets)
                                     v   v
             -------------------------------------------------
             |               ForCES Interface                |
             -------------------------------------------------
             |       |       |       |       |       |       |
             |LPM Fwd|Meter  |Shaper |NAT    |Classi-|      |
             |       |       |       |       |fier   |       |
             -------------------------------------------------
             |               FE resources                    |
             -------------------------------------------------

                     Figure 3. Examples of CE and FE functions

        Some examples of control functions that can be implemented in the CE
        include routing protocols like RIP, OSPF and BGP, control and
        signaling protocols like RSVP (Resource Reservation Protocol), LDP
        (Label Distribution Protocol) for MPLS, etc.  Examples of forwarding
        functions in FE include LPM (longest prefix match) forwarder,
        classifiers, traffic shaper, meter, NAT, etc.  Figure 3 shows a
        diagram with examples in both CE and FE.  Any given NE may contain
        one or many of these CE and FE functions in it.  The diagram also
        shows that ForCES protocol is used to transport both the control
        messages for ForCES itself and the data packets that are

     Yang, et. al.      Expires June 2003                  [Page 6]
        originated/destined from/to the control functions in CE (e.g.,
        routing packets).  Section 4.2.4 provides more detail on this.

        A set of requirements for control and forwarding separation is
        identified in [3].  This document describes a

             -------------------------------------------------
             |       |       |       |       |       |       |
             |OSPF   |RIP    |BGP    |RSVP   |LDP    |      |
             |       |       |       |       |       |       |
             -------------------------------------------------
             |               ForCES architecture
        that satisfies the architectural requirements of that document and
        defines a framework for Interface                |
             -------------------------------------------------
                                     ^   ^
                             ForCES network elements and the associated  |   |data
                             control |   |packets
                             messages|   |(e.g., routing packets)
                                     v   v
             -------------------------------------------------
             |               ForCES Interface                |
             -------------------------------------------------
             |       |       |       |       |       |       |
             |LPM Fwd|Meter  |Shaper |NAT    |Classi-|      |
             |       |       |       |       |fier   |       |
             -------------------------------------------------
             |               FE resources                    |
             -------------------------------------------------

     Yang, et. al.      Expires Dec 2003                  [Page 7] 
                     Figure 3. Examples of CE and FE functions

        A set of requirements for control and forwarding separation is
        identified in [3].  This document describes a ForCES architecture
        that satisfies the architectural requirements of that document and
        defines a framework for ForCES network elements and the associated
        entities to facilitate protocol definition.  Whenever necessary,
        this document uses many examples to illustrate the issues and/or
        possible solutions in ForCES.  These examples are intended to be
        just examples, and should not be taken as the only or definite ways
        of doing certain things.  It is expected that separate document will
        be produced by the ForCES working group to specify the ForCES
        protocol(s).

     3. Architecture

        This section defines the ForCES architectural framework and the
        associated logical components.  This ForCES framework defines
        components of ForCES NEs including several ancillary components.
        These components may be connected in different kinds of topologies
        for flexible packet processing.

                                ---------------------------------------
                                | ForCES Network Element              |
         --------------   Fc    | --------------      --------------  |
         | CE Manager |---------+-|     CE 1   |------|    CE 2    |  |
         --------------         | |            |  Fr  |            |  |
               |                | --------------      --------------  |
               | Fl             |         |  |    Fp       /          |
               |                |       Fp|  |----------| /           |
               |                |         |             |/            |
               |                |         |             |             |
               |                |         |     Fp     /|----|        |
               |                |         |  /--------/      |        |
         --------------     Ff  | --------------      --------------  |
         | FE Manager |---------+-|     FE 1   |  Fi  |     FE 2   |  |
         --------------         | |            |------|            |  |
                                | --------------      --------------  |
                                |   |  |  |  |          |  |  |  |    |
                                ----+--+--+--+----------+--+--+--+-----
                                    |  |  |  |          |  |  |  |
                                    |  |  |  |          |  |  |  |
                                      Fi/f                   Fi/f

                     Figure 4. ForCES Architectural Diagram

        The diagram in Figure 4 shows the logical components of the ForCES
        architecture and their relationships.  There are two kinds of
        components inside a ForCES network element: control element (CE) and
        forwarding element (FE).  The framework allows multiple instances of
        CE and FE inside one NE.  Each FE contains one or more physical
        media interfaces for receiving and transmitting packets from/to the

     Yang, et. al.      Expires June Dec 2003                  [Page 7] 8] 
        external world.  The aggregation of these FE interfaces becomes the
        NEs external interfaces.  In addition to the external interfaces,
        there must also exist some kind of interconnect within the NE so
        that the CE and FE can communicate with each other, and one FE can
        forward packets to another FE.  The diagram also shows two entities
        outside of the ForCES NE: CE Manager and FE Manager.  These two
        entities provide configuration to the corresponding CE or FE in the
        pre-association phase (see Section 5.1).  There is no defined role
        for FE Manager and CE Manager in post-association phase, thus these
        logical components are not considered part of the ForCES NE.

        For convenience, the logical interactions between these components
        are labeled by reference points Fp, Fc, Ff, Fr, Fl, and Fi, as shown
        in Figure 4.  The FE external interfaces are labeled as Fi/f. More
        detail is provided in Section 4 and 5 for each of these reference
        points.  All these reference points are important in understanding
        the ForCES architecture, however, the ForCES protocol is only
        defined over one reference point -- Fp.

        The interface between two ForCES NEs is identical to the interface
        between two conventional routers and these two NEs exchange the
        protocol packets through the external interfaces at Fi/f.  ForCES
        NEs connect to existing routers transparently.

     3.1. Control Elements and Fr Reference Point

        It is not necessary to define any protocols across the Fr reference
        point to enable control and forwarding separation for simple
        configurations like single CE and multiple FEs.  However, this
        architecture permits multiple CEs to be present in a network
        element.  In cases where an implementation uses multiple CEs, the
        invariant that the CEs and FEs together appear as a single NE must
        be maintained.

        Multiple CEs may be used for redundancy, load sharing, distributed
        control, or other purposes.  Redundancy is the case where one or
        more CEs are prepared to take over should an active CE fail.  Load
        sharing is the case where two or more CEs are concurrently active
        and where any request that can be serviced by one of the CEs can also be
        serviced by any of the other CEs.  For both redundancy and load
        sharing, the CEs involved are equivalently capable.  The only
        difference between these two cases is in terms of how many active
        CEs there are.  Distributed control is the case where two or more
        CEs are concurrently active but where certain requests can only be
        serviced by certain CEs.

        When multiple CEs are employed in a ForCES NE, their internal
        organization is considered an implementation issue that is beyond
        the scope of ForCES. CEs are wholly responsible for coordinating
        amongst themselves via the Fr reference point to provide consistency
        and synchronization.  However, ForCES does not define the
        implementation or protocols used between CEs, nor does it define how
        to distribute functionality among CEs.  Nevertheless, ForCES will

     Yang, et. al.      Expires June Dec 2003                  [Page 8] 9] 
        support mechanisms for CE redundancy or fail over, and it is
        expected that vendors will provide redundancy or fail over solutions
        within this framework.

     3.2. Forwarding Elements and Fi reference point

        FEs perform

        FE is a logical entity that implements the ForCES protocol and uses
        the underlying hardware to provide per-packet processing and
        handling as directed by a CE.  It is possible to partition one
        physical FE into multiple logical FEs.  It is also possible for one
        FE to use multiple physical FEs.  The mapping between physical FE(s)
        and the logical FE(s) is beyond the scope of ForCES.  For example, a
        logical partition of a physical FE can be created by assigning some
        portion of each of the resources (e.g., ports, memory, forwarding
        table entries) available on the physical FE to each of the logical
        FEs.  Such concept of FE virtualization is analogous to a virtual
        switching element as described in [8].  FE virtualization should
        occur only in the pre-association phase and hence has no impact on
        ForCES.

        FEs perform all packet processing functions as directed by CEs.
        FEs have no initiative of their own.  Instead, FEs are slaves and
        only do as they are told.  FEs may communicate with one or more CEs
        concurrently across reference point Fp.  FEs have no notion of CE
        redundancy, load sharing, or distributed control.  Instead, FEs
        accept commands from any CE authorized to control them, and it is up
        to the CEs to coordinate among themselves to achieve redundancy,
        load sharing or distributed control.  The idea is to keep FEs as
        simple and dumb as possible so that FEs can focus its resource on
        the packet processing functions.

                     -------   Fr  -------
                     | CE1 | ------| CE2 |
                     -------       -------
                       |   \      /   |
                       |    \    /    |
                       |     \  /     |
                       |      \/Fp    |
                       |      /\      |
                       |     /  \     |
                       |    /    \    |
                     -------  Fi   -------
                     | FE1 |<----->| FE2 |
                     -------       -------

                         Figure 5. CE redundancy example.

        For example, in Figure 5, FE1 and FE2 can be configured to accept
        commands from both the primary CE (CE1) and the backup CE (CE2).
        Upon detection of CE1 failure, perhaps across the Fr or Fp reference

     Yang, et. al.      Expires Dec 2003                  [Page 10] 
        point, CE2 is configured to take over activities of CE1.  This is
        beyond the scope of ForCES and is not discussed further.

        Distributed control can be achieved in the similar fashion, without
        much intelligence on the part of FEs.  For example, FEs can be
        configured to detect RSVP and BGP protocol packets, and forward RSVP
        packets to one CE and BGP packets to another CE.  Hence, FEs may
        need to do packet filtering for forwarding packets to specific CEs.

        This architecture permits multiple FEs to be present in a NE. [3]
        dictates that the ForCES protocol must be able to scale to at least
        hundreds of FEs (see [3] Section 5, requirement #11).  Each of these
        FEs may potentially have a different set of packet processing
        functions, with different media interfaces.  FEs are responsible for

     Yang, et. al.      Expires June 2003                  [Page 9]
        basic maintenance of layer-2 connectivity with other FEs and with
        external entities.  Many layer-2 media include sophisticated control
        protocols.  The FORCES protocol (over the Fp reference point) will
        be able to carry messages for such protools protocols so that, in keeping
        with the "dumb FE model, the CE can provide appropriate
        intelligence and control over these media.

        When multiple FEs are present, ForCES requires that packets must be
        able to arrive at the NE by one FE and leave the NE via a different
        FE (See [3], Section 5, Requirement #3).  Packets that enter the NE
        via one FE and leave the NE via a different FE are transferred
        between FEs across the Fi reference point.  Fi reference point could
        be used by FEs to discovery their (inter-FE) topology, perhaps
        during pre-association phase.  The Fi reference point is a separate
        protocol from the Fp reference point and is not currently defined by
        the ForCES architecture.

                             -----------------
                             |      CE       |
                             -----------------
                              ^      ^      ^
                             /       |       \
                            /        v        \
                           /      -------      \
                          /    +->| FE3 |<-+    \
                         /     |  |     |  |     \
                        v      |  -------  |      v
                      -------  |           |  -------
                      | FE1 |<-+           +->| FE2 |
                      |     |<--------------->|     |
                      -------                 -------
                         ^  |                   ^  |
                         |  |                   |  |
                         |  v                   |  v

                     (a) Full mesh among FE1, FE2 and FE3.

     Yang, et. al.      Expires Dec 2003                  [Page 11] 
                             -----------
                             |   CE    |
                             -----------
                            ^ ^       ^ ^
                           /  |       |  \
                    /------   |       |   ------\
                    v         v       v          v
                -------   -------   -------   -------
                | FE1 |<->| FE2 |<->| FE3 |<->| FE4 |
                -------   -------   -------   -------
                  ^  |     ^  |       ^  |     ^  |
                  |  |     |  |       |  |     |  |
                  |  v     |  v       |  v     |  v

     Yang, et. al.      Expires June 2003                  [Page 10]

                     (b) Multiple FEs in a daisy chain

                                ^ |
                                | v
                             -----------
                             |   FE1   |<-----------------------|
                             -----------                        |
                               ^    ^                           |
                              /      \                          |
                       | ^   /        \   ^ |                   V
                       v |  v          v  | v                ----------
                     ---------        ---------              |        |
                     | FE2   |        |  FE3  |<------------>|   CE   |
                     ---------        ---------              |        |
                         ^  ^          ^                     ----------
                         |   \        /                        ^  ^
                         |    \      /                         |  |
                         |    v     v                          |  |
                         |   -----------                       |  |
                         |   |   FE4   |<----------------------|  |
                         |   -----------                          |
                         |      |  ^                              |
                         |      v  |                              |
                         |                                        |
                         |----------------------------------------|

                     (c) Multiple FEs connected by a ring

                     Figure 6. Some examples of FE topology.

        FEs could be connected in different kinds of topologies and packet
        processing may spread across several FEs in the topology.  Hence,
        logical packet flow may be different from physical FE topology.
        Figure 6 provides some topology examples.  When it is necessary to
        forward packets between FEs, the CE needs to understand the FE
        topology.  The FE topology can be queried from the FEs by CEs.

     Yang, et. al.      Expires Dec 2003                  [Page 12] 
     3.3. CE Managers

        CE managers are responsible for determining which FEs a CE should
        control.  It is legitimate for CE managers to be hard-coded with the
        knowledge of with which FEs its CEs should communicate.  A CE
        manager may also be physically embedded into a CE and be implemented
        as a simple keypad or other direct configuration mechanism on the
        CE.  Finally, CE managers may be physically and logically separate
        entities that configure the CE with FE information via such
        mechanisms as COPS-PR [6] or SNMP [4].

     3.4. FE Managers

     Yang, et. al.      Expires June 2003                  [Page 11]

        FE managers are responsible for determining to which CE any
        particular FE should initially communicate.  Like CE managers, no
        restrictions are placed on how a FE manager decides to which CE its
        FEs should communicate, nor are restrictions placed on how FE
        managers are implemented.  Each FE should have one and only one FE
        manager, while different FEs may have the same or different FE
        manager(s).  Each manager can choose to exist and operate
        independent of other manager.

     4. Operational Phases

        Both FEs and CEs require some configuration in place before they can
        start information exchange and function as a coherent network
        element.  Two operational phases are identified in this framework --
        pre-association and post-association.

     4.1.Pre-association Phase

        Pre-association phase is the period of time during which a an FE
        Manager and a CE Manager are determining which FE and CE should be
        part of the same network element.  The protocols used during this
        phase may include all or some of the message exchange over Fl, Ff
        and Fc reference points.  However, all these may be optional and
        none of this is within the scope of ForCES protocol.

     4.1.1. Fl Reference Point

             FE Manager      FE               CE Manager     CE
              |              |                 |             |
              |              |                 |             |
              |(security exchange)             |             |
             1|<------------------------------>|             |
              |              |                 |             |
              |(a list of CEs and their attributes)          |
             2|<-------------------------------|             |
              |              |                 |             |
              |(a list of FEs and their attributes)          |
             3|------------------------------->|             |
              |              |                 |             |
              |              |                 |             |
              |<----------------Fl------------>|             |

          Figure 7. An example of message exchange over Fl reference point

        CE managers and FE managers may communicate across the Fl reference
        point in the pre-association phase in order to determine which CEs
        and FEs should communicate with each other.  Communication across
        the Fl reference point is optional in this architecture.  No
        requirements are placed on this reference point.

        CE managers and FE managers may be operated by different entities.
        The operator of the CE manager may not want to divulge, except to
        specified FE managers, any characteristics of the CEs it manages.
        Similarly, the operator of the FE manager may not want to divulge FE

     Yang, et. al.      Expires June 2003                  [Page 12]
        characteristics, except to authorized entities.  As such, CE
        managers and FE managers may need to authenticate one another.
        Subsequent communication between CE managers and FE managers may

     Yang, et. al.      Expires Dec 2003                  [Page 13] 
        require other security functions such as privacy, non-repudiation,
        freshness, and integrity.

             FE Manager      FE               CE Manager     CE
              |              |                 |             |
              |              |                 |             |
              |(security exchange)             |             |
             1|<------------------------------>|             |
              |              |                 |             |
              |(a list of CEs and their attributes)          |
             2|<-------------------------------|             |
              |              |                 |             |
              |(a list of FEs and their attributes)          |
             3|------------------------------->|             |
              |              |                 |             |
              |              |                 |             |
              |<----------------Fl------------>|             |

          Figure 7. An example of message exchange over Fl reference point

        Once the necessary security functions have been performed, the CE
        and FE managers communicate to determine which CEs and FEs should
        communicate with each other.  At the very minimum, the CE and FE
        managers need to learn of the existence of available FEs and CEs
        respectively.  This discovery process may or may not entail one or
        both managers learning the capabilities of the discovered ForCES
        protocol elements.  Figure 7 shows an example of possible message
        exchange between CE manager and FE manager over Fl reference point.

     4.1.2. Ff Reference Point

        The Ff reference point is used to inform forwarding elements of the
        association decisions made by the FE Manager manager in pre-association
        phase.  Only authorized entities may instruct a FE               CE Manager     CE
              |              |                 |             |
              |              |                 |             |
              |(security exchange)             |(security exchange)
             1|<------------>|authentication  1|<----------->|authentication
              |              |                 |             |
              |(FE ID, attributes)             |(CE ID, attributes)
             2|<-------------|request         2|<------------|request
              |              |                 |             |
             3|------------->|response        3|------------>|response
              |(corresponding CE ID)           |(corresponding FE ID)
              |              |                 |             |
              |              |                 |             |
              |<-----Ff----->|                 |<-----Fc---->|

                          Figure 8. Examples of message exchange
                             over Ff and Fc reference points.

        The Ff reference point is used to inform forwarding elements of the
        association decisions made by the FE manager in pre-association
        phase.  Only authorized entities may instruct a FE with respect to
        which with respect to
        which CE should control it.  Therefore, privacy, integrity,
        freshness, and authentication are necessary between the FE manager
        and FEs when the FE manager is remote to the FE.  Once the
        appropriate security has been established, the FE manager instructs
        the FEs across this reference point to join a new NE or to
        disconnect from an existing NE. The FE Manager could also assign
        unique FE identifiers to the FEs using this reference point.  The FE
        identifiers are useful in post association phase to express FE
        topology.  Figure 8 shows example of message exchange over Ff
        reference point.

     Yang, et. al.      Expires June 2003                  [Page 13]

        Note that the FE manager function may be co-located with the FE
        (such as by manual keypad entry of the CE IP address), in which case
        this reference point is reduced to a built-in function.

             FE Manager      FE               CE Manager     CE
              |              |                 |             |

     Yang, et. al.      Expires Dec 2003                  [Page 14] 
              |              |                 |             |
              |(security exchange)             |(security exchange)
             1|<------------>|authentication  1|<----------->|authentication
              |              |                 |             |
              |(FE ID, attributes)             |(CE ID, attributes)
             2|<-------------|request         2|<------------|request
              |              |                 |             |
             3|------------->|response        3|------------>|response
              |(corresponding CE ID)           |(corresponding FE ID)
              |              |                 |             |
              |              |                 |             |
              |<-----Ff----->|                 |<-----Fc---->|

                          Figure 8. Examples of message exchange
                             over Ff and Fc reference points.

     4.1.3. Fc Reference Point

        The Fc reference point is used to inform control elements of the
        association decisions made by CE managers in pre-association phase.
        When the CE manager is remote, only authorized entities may instruct
        a CE to control certain FEs.  Privacy, integrity, freshness and
        authentication are also required across this reference point in such
        a configuration.  Once appropriate security has been established,
        the CE manager instructs CEs as to which FEs they should control and
        how they should control them.  Figure 7 shows example of message
        exchange over Fc reference point.

        As with the FE manager and FEs, configurations are possible where
        the CE manager and CE are co-located and no protocol is used for
        this function.

     4.2. Post-association Phase and Fp reference point

        Post-association phase is the period of time during which a FE and
        CE have been configured with information necessary to contact each
        other and includes both association establishment and steady-state
        communication.  The communication between CE and FE is performed
        across the Fp ("p" meaning protocol) reference point. ForCES
        protocol is exclusively used for all communication across the Fp
        reference point.

     4.2.1. Proximity and Interconnect between CEs and FEs

        The ForCES Working Group has made a conscious decision that the
        first version of ForCES will not be designed to support
        configurations where the CE and FE are located arbitrarily in the
        network.  In particular, ForCES is intended for "very close" CE/FE
        localities in IP networks, as defined by ForCES Applicability
        Statement ([7]).  Very Close localities consist of control and
        forwarding elements that either are components in the same physical
        box, or are separated at most by one local network hop.

     Yang, et. al.      Expires Dec 2003                  [Page 15] 
        CEs and FEs can be connected by a variety of interconnect
        technologies, including Ethernet connections, backplanes, ATM (cell)
        fabrics, etc.  ForCES should be able to support each of these
        interconnects (see [3] Section 5, requirement #1).  ForCES will make
        use of an existing RFC2914 ([2]) compliant L4 protocol with adequate
        reliability, security and congestion control (e.g. TCP, SCTP) for
        transport purposes.

     4.2.2. Association Establishment

                     FE                      CE
                     |                       |
                     |(Security exchange.)   |
                    1|<--------------------->|
                     |                       |
                     |(Let me join the NE please.)
                    2|---------------------->|
                     |                       |
                     |(What kind of FE are you? -- capability query)
                    3|<----------------------|
                     |                       |
                     |(Here is my FE functions/state: use model to describe)
                    4|---------------------->|
                     |                       |
                     |(How are you connected with other FEs?)
                    5|<----------------------|
                     |                       |
                     |(Here is the FE topology info)
                    6|---------------------->|
                     |                       |
                     |(Initial config for FE -- optional)
                    7|<----------------------|
                     |                       |
                     |(I am ready to go. Shall I?)
                    8|---------------------->|
                     |                       |
                     |(Go ahead!)            |
                    9|<----------------------|
                     |                       |

               Figure 9. Example of message exchange between CE and FE
                         over Fp to establish NE association

        As an example, figure 9 shows some of the message exchange that may
        happen before the association between the CE and FE is fully

     Yang, et. al.      Expires June 2003                  [Page 14]
        established.  Either the CE or FE can initiate the connection.
        Security handshake is necessary to authenticate the two
        communication endpoints to each other before any further message
        exchange can happen. The exact details of the security handshake
        depend on the security solution chosen by ForCES protocol.  It is
        most likely that either IPSec or TLS will be used.  Section 9

     Yang, et. al.      Expires Dec 2003                  [Page 16] 
        provides more details on the security considerations for ForCES.
        After the successful security handshake, the FE needs to inform the
        CE of its own capability and its topology in relation to other FEs.
        The capability of the FE is represented by the FE model, described
        in another a separate document.  The model would allow a FE to describe what
        kind of packet processing functions it contains, in what order the
        processing happens, what kinds of configurable parameters it allows,
        what statistics it collects and what events it might throw, etc.
        Once such information is available to the CE, the CE sends all necessary may choose to
        send some initial or default configuration to the FE so that the FE
        can start receiving and processing packets correctly.  For example, the CE might need to send a snapshot of the
        current forwarding table to  Such
        initialization may not be necessary if the FE so that already obtains the FE can start routing
        packets correctly.
        information from its own bootstrap process.  Once FE starts
        accepting packets for processing, we say the association of this FE
        with its CE is now established.  From then on, the CE and FE enter
        steady-state communication.

                     FE                      CE
                     |                       |
                     |(Hello, are you there?)|
                    1|<----------------------|
                     |                       |
                     |(Yes. let me join the NE please.)
                    2|---------------------->|
                     |                       |
                     |(What kind of FE are you? -- capability query)
                    3|<----------------------|
                     |                       |
                     |(Here is my FE functions/state: use model to describe)
                    4|---------------------->|
                     |                       |
                     |(How are you connected with others? -- topology query)
                    5|<----------------------|
                     |                       |
                     |(Here is the topology info)
                    6|---------------------->|
                     |                       |
                     |(Config for FE, e.g. forwarding table)
                    7|<----------------------|
                     |                       |
                     |(I am ready to go. Shall I?)
                    8|---------------------->|
                     |                       |
                     |(Go ahead!)            |
                    9|<----------------------|
                     |                       |

               Figure 9. Example of message exchange between CE and FE
                         over Fp to establish NE association

     4.2.3. Steady-state Communication

     Yang, et. al.      Expires June 2003                  [Page 15]

        Once an association is established between the CE and FE, the ForCES
        protocol is used by the CE and FE over Fp reference point to
        exchange information to facilitate packet processing.

                     FE                      CE
                     |                       |
                     |(Add these new routes.)|
                    1|<----------------------|
                     |                       |
                     |(Successful.)          |
                    2|---------------------->|
                     |                       |
                     |                       |
                     |(Query some stats.)    |
                    1|<----------------------|
                     |                       |
                     |(Reply with stats collected.)
                    2|---------------------->|
                     |                       |
                     |                       |
                     |(My port is down, with port #.)
                    1|---------------------->|
                     |                       |
                     |(Route to this port instead...)
                     |(Here is a new forwarding table)
                    2|<----------------------|
                     |                       |
                     |                       |

              Figure 10. Examples of message exchange between CE and FE
                      over Fp during steady-state communication

     Yang, et. al.      Expires Dec 2003                  [Page 17] 
        Based on the information acquired through CEs' control processing,
        CEs will frequently need to manipulate the packet-forwarding
        behaviors of their FE(s) by sending instructions to FEs.  For
        example, Figure 10 shows message exchange examples in which the CE
        sends new routes to the FE so that the FE can add them to its
        forwarding table.  The CE may query the FE for statistics collected
        by the FE and the FE may notify the CE of important events such as
        port failure.

     4.2.4. Data Packets across Fp reference point

        Control plane protocol packets (such as RIP, OSPF messages)
        addressed to any of NE's interfaces are typically redirected by the
        receiving FE to its CE, and CE may originate packets and have its FE
        deliver them to other NEs.  Therefore, ForCES protocol over Fp not
        only transports the ForCES protocol messages between CEs and FEs,
        but also encapsulates the data packets from control plane protocols.
        Moreover, one FE may be controlled by multiple CEs for distributed
        control.  In this configuration, the control protocols supported by
        the FORCES NEs may spread across multiple CEs.  For example, one CE
        may support routing protocols like OSPF and BGP, while a signaling
        and

     Yang, et. al.      Expires June 2003                  [Page 16] admission control protocol like RSVP is supported in another CE.
        FEs are configured to recognize and filter these protocol packets
        and forward them to the corresponding CE.

        Figure 11 shows one example of how the BGP packets originated by
        router A are passed to router B.  In this example, the ForCES
        protocol is used to transport the packets from the CE to the FE
        inside router A, and then from the FE to the CE inside router B.  In
        light of the fact that the ForCES protocol is responsible to
        transport both the control messages and the data packets between the
        CE and FE over Fp reference point, it is possible to use either a
        single protocol or multiple protocols to achieve that.

             ---------------------           ----------------------
             |                   |           |                    |
             |    +--------+     |           |     +--------+     |
             |    |CE(BGP) |     |           |     |CE(BGP) |     |
             |    +--------+     |           |     +--------+     |
             |        |          |           |          ^         |
             |        |Fp        |           |          |Fp       |
             |        v          |           |          |         |
             |    +--------+     |           |     +--------+     |
             |    |  FE    |     |           |     |   FE   |     |
             |    +--------+     |           |     +--------+     |
             |        |          |           |          ^         |
             | Router |          |           | Router   |         |
             | A      |          |           | B        |         |
             ---------+-----------           -----------+----------
                      v                                 ^
                      |                                 |
                      |                                 |
                      ------------------->---------------

            Figure 11. Example to show data packet flow between two NEs.

        Figure 11 shows one example of how the BGP packets originated by
        router A are passed to router B.  In this example, the ForCES
        protocol is used to transport the packets from the CE to the FE
        inside router A, and then from the FE to the CE inside router B.  In
        light of the fact that the ForCES protocol is responsible for
        transporting both the control messages and the data packets between

     Yang, et. al.      Expires Dec 2003                  [Page 18] 
        the CE and FE over Fp reference point, it is possible to use either
        a single protocol or multiple protocols to achieve this.

     4.2.5. Proxy FE

        In the case where a physical FE cannot implement (e.g., due to the
        lack of a general purpose CPU) the ForCES protocol directly, a proxy
        FE can be used in the middle of Fp reference point.  This allows the
        CE communicate to the physical FE via the proxy by using ForCES,
        while the proxy manipulates the physical FE using some intermediary
        form of communication (e.g., a non-ForCES protocol or DMA).  In such
        an implementation, the combination of the proxy and the physical FE
        becomes one logical FE entity.

     4.3. Association Re-establishment

        FEs and CEs may join and leave NEs dynamically (see [3] Section 5,
        requirements #12).  When a FE or CE leaves the NE, the association
        with the NE is broken.  If the leaving party rejoins a NE later, to
        re-establish the association, it may or may not need to re-enter the

     Yang, et. al.      Expires June 2003                  [Page 17]
        pre-association phase.  Loss of association can also happen
        unexpectedly due to loss of connection between the CE and the FE.
        Therefore, the framework allows the bi-directional transition
        between these two phases, but the ForCES protocol is only applicable
        for the post-association phase.  However, the protocol should
        provide mechanisms to support association re-establishment re-establishment. This
        includes the ability for CEs and FEs to determine when there is a
        loss of association between them, ability to restore association and
        efficient state (re)synchronization mechanisms (see [3] Section 5,
        requirement #7).

        The example in Figure 5 is used to illustrate what happens when the Note that security association is broken and later state must be
        also re-established again.  Section 3.2
        already explains what happens if CE1 fails and how CE2 can take
        over.  If no CE redundancy is provided, at to guarantee the association
        establishment phase FEs need to be told what to do in same level of security exists
        before and after the case association re-establishment.

     4.3.1. CE graceful restart

        The failure and restart of the CE
        failure.  FEs may be told to stop packet processing all together if
        its in a router can potentially cause
        much stress and disruption on the control plane throughout a
        network.  Because when a CE fails.  Or, FEs may be told has to continue forwarding packets restart for a limited time even in any reason, the face of CE failure.  No matter what
        router loses routing adjacencies or sessions with its routing
        neighbors.  Neighbors who detect the instruction is, it needs lost adjacency normally re-
        compute new routes and then send routing updates to be part of their own
        neighbors to communicate the configuration when lost adjacency.  Their neighbors do the association is established.
        same thing to propagate throughout the network.  In the same example in Figure 5, assuming CE1 is meantime,
        the working CE for restarting router cannot receive traffic from other routers
        because the moment, what would happen if one of neighbors have stopped using the FEs, say FE1, leaves routers previously
        advertised routes. When the
        NE temporarily?  FE1 may voluntarily decide restarting router restores adjacencies,
        neighbors must once again re-compute new routes and send out
        additional routing updates. The restarting router is unable to leave
        forward packets until it has re-established routing adjacencies with
        neighbors, received route updates through these adjacencies, and
        computed new routes.  Until convergence takes place throughout the
        association.  Or, FE1
        network, packets may stop functioning simply due to unexpected
        failure.  In former case, CE1 receives a "leave-association request"
        from FE1.  In be lost in transient black holes or forwarding
        loops.

     Yang, et. al.      Expires Dec 2003                  [Page 19] 
        A high availability mechanism known as the latter, CE1 detects "graceful restart" has
        been used by the failure of FE1 IP routing protocols (OSPF [10], BGP [11], BGP
        [11]) and MPLS label distribution protocol (LDP [9]) to help
        minimize the negative effects on routing throughout an entire
        network caused by some
        other mean.  In both cases, CE1 keeps a note of such event from FE1
        while continue commanding FE2.  When FE1 decides to rejoin again, or
        when it restarting router.  Route flap on neighboring
        routers is back up again from the failure, FE1 needs to re-discover
        its master (CE).  This avoided, and a restarting router can continue to forward
        packets that would otherwise be achieved by several means.  It may re-
        enter dropped.

        While the pre-association phase and get that information details differ from its FE
        manager.  It may retrieve protocol to protocol, the previous CE information from its
        cache, if it can validate general idea
        behind the information freshness.  Once it
        discovers its CE, it starts message exchange with CE to re-establish graceful restart mechanism remains the association just as outlined in Figure 9, with same.  With the possible
        exception that
        graceful restart, a restarting router can inform its neighbors when
        it might be able restarts.  The neighbors may detect the lost adjacency but do not
        recompute new routes or send routing updates to their neighbors.
        The neighbors also hold on to bypass the transport of routes received from the
        complete initialization information.  Suppose that FE1
        restarting router before restart and assume they are still has its
        routing table valid for
        a limited time.  By doing so, the restarting routers FEs can also
        continue to receive and other state information forward traffic from other neighbors for a
        limited time by using the last association,
        instead of sending routes they already have.  The restarting
        router then re-establishes routing adjacencies, downloads updated
        routes from all its neighbors, recomputes new routes and uses them
        to replace the information again from scratch, older routes it can
        choose to use more efficient mechanism was using. It then sends these
        updated routes to re-sync up the state with its CE.  For example, a checksum of neighbors and signals the state might give a quick
        indication completion of whether or not the state
        graceful restart process.

        Non-stop forwarding is in-sync with its CE.  By
        comparing its state with CE first, it sends information update only
        if it a requirement for graceful restart.  It is needed.

     5. Applicability to RFC1812

        [3] Section 5, requirement #9 dictates that "All proposed ForCES
        architecture must explain how that architecture supports all of the
        necessary so a router functions as defined in RFC1812."  RFC1812 discusses many
        important requirements for IPv4 routers from the link layer can continue to the
        application layer. forward packets while it is
        downloading routing information and recomputing new routes.  This section addresses
        ensures that packets will not be dropped.  As one can see, one of
        the relevant requirements
        in RFC1812 for implementing IPv4 routers based on ForCES

     Yang, et. al.      Expires June 2003                  [Page 18] 
        architecture and explains how ForCES satisfies these requirements benefits afforded by
        providing guidelines on how to separate the functionalities required
        into forwarding plane and control plane.

        In general, the forwarding plane carries out the bulk separation of the per-
        packet processing that CE and FE is required at line speed, while exactly the control
        plane carries most
        ability of non-stop forwarding in the computationally complex operations that
        are typical face of the control CE failure and signaling protocols.  However, it is
        impossible
        restart.  The support of dynamic changes to draw a rigid line CE/FE association in
        ForCES also makes it compatible with high availability mechanisms
        such as graceful restart.

        ForCES should be able to divide support CE graceful restart easily.  When
        the processing into CEs
        and FEs cleanly.  Nor should association is established the ForCES architecture limit first time, the
        innovative approaches in control and forwarding plane separation.
        As more and more processing power is available CE must inform
        the FEs what to do in the FEs, some case of CE failure.  If graceful restart
        is not supported, the control functions that traditionally are performed by CEs FEs may
        now be moved told to stop packet processing all
        together if its CE fails.  If graceful restart is supported, the FEs for better performance
        should be told to cache and scalability.  Such
        offloaded functions may include part of ICMP or TCP processing, or
        part of routing protocols.  Once off-loaded onto hold on to its FE state including the
        forwarding
        plane, tables across the restarts.  A timer must be included so
        that the timeout causes such CE functions, even though logically belonging cached state to expire eventually.
        Those timers should be settable by the
        control plane, now become part of the CE.

     4.3.2. FE functions.  Just like restart

        In the
        other logical functions performed by FEs, such off-loaded functions
        must be expressed as part same example in Figure 5, assuming CE1 is the working CE for
        the moment, what would happen if one of the FE model so that FEs, say FE1, leaves the CEs can
        NE temporarily?  FE1 may voluntarily decide
        how to best take advantage of these off-loaded functions when
        present on leave the FEs.

     5.1. General Router Requirements

        Routers have at least two or more logical interfaces.  When CEs and
        FEs are separated by ForCES within
        association.  Alternatively, FE1 may stop functioning simply due to
        unexpected failure.  In former case, CE1 receives a single NE, "leave-

     Yang, et. al.      Expires Dec 2003                  [Page 20] 
        association request" from FE1.  In the latter, CE1 detects the
        failure of FE1 by some additional
        interfaces are needed for intra-NE communications.  Figure 12 shows other means.  In both cases, CE1 must inform
        the routing protocols of such an example to illustrate that.  This NE contains one CE event, most likely prompting a
        reachability and two FEs.
        Each FE has four interfaces; two of them are used for receiving SPF (Shortest Path First) recalculation and
        transmitting packets
        associated downloading of new FIBs from CE1 to the external world, while the other two are
        for intra-NE connections.  CE has two logical interfaces #9 and #10,
        connected to interfaces #3 remaining
        FEs (only FE2 in this example).  Such recalculation and #6 FIB update
        will also be propagated from FE1 and FE2, respectively.
        Interface #4 and #5 the CE1 to its neighbors that are connected for FE1-FE2 communication.
        Therefore, this router NE provides four external interfaces (#1, 2,
        7 and 8).

                     ---------------------------------
                     |               router NE       |
                     |   -----------   -----------   |
                     |   |
        affected by the connectivity of FE1.

        When FE1   |   |   FE2   |   |
                     |   -----------   -----------   |
                     |   1| 2| 3| 4|   5| 6| 7| 8|   |
                     |    |  |  |  |    |  |  |  |   |
                     |    |  |  |  +----+  |  |  |   |
                     |    |  |  |          |  |  |   |
                     |    |  | 9|        10|  |  |   |
                     |    |  | -------------- |  |   |
                     |    |  | |    CE      | |  |   |
                     |    |  | -------------- |  |   |
                     |    |  |                |  |   |
                     -----+--+----------------+--+----

     Yang, et. al.      Expires June 2003                  [Page 19] 
                          |  |                |  |
                          |  |                |  |

                     Figure 12. A router NE example with four interfaces.

        IPv4 routers must implement IP decides to support rejoin again, or when it restarts again from the
        failure, FE1 needs to re-discover its packet forwarding
        function, which is driven master (CE).  This can be
        achieved by several means.  It may re-enter the pre-association
        phase and get that information from its FIB (Forwarding Information Base).
        This Internet layer forwarding (see RFC1812 [1] Section 5)
        functionality naturally belongs to FEs in FE manager.  It may retrieve
        the previous CE information from its cache, if it can validate the
        information freshness.  Once it discovers its CE, it starts message
        exchange with CE to re-establish the association just as outlined in
        Figure 9, with the possible exception that it might be able to
        bypass the transport of the complete initial configuration.  Suppose
        that FE1 still has its routing table and other state information
        from the last association, instead of sending all the information
        again from scratch, it may be able to use more efficient mechanism
        to re-sync up the state with its CE if such mechanism is supported
        by the ForCES protocol.  For example, CRC-32 of the state might give
        a quick indication of whether or not the state is in-sync with its
        CE.  By comparing its state with CE first, it sends information
        update only if it is needed.  ForCES protocol may choose to
        implement similar optimization mechanisms, but it may also choose
        not to, as this is not a requirement.

     5. Applicability to RFC1812

        [3] Section 5, requirement #9 dictates that "All proposed ForCES
        architecture must explain how that architecture supports all of the
        router functions as defined in RFC1812."  RFC1812 discusses many
        important requirements for IPv4 routers from the link layer to the
        application layer.  This section addresses the relevant requirements
        in RFC1812 for implementing IPv4 routers based on ForCES
        architecture and explains how ForCES satisfies these requirements by
        providing guidelines on how to separate the functionalities required
        into forwarding plane and control plane.

        In general, the forwarding plane carries out the bulk of the per-
        packet processing that is required at line speed, while the control
        plane carries most of the computationally complex operations that
        are typical of the control and signaling protocols.  However, it is
        impossible to draw a rigid line to divide the processing into CEs
        and FEs cleanly.  Nor should the ForCES architecture limit the
        innovative approaches in control and forwarding plane separation.
        As more and more processing power is available in the FEs, some of
        the control functions that traditionally are performed by CEs may
        now be moved to FEs for better performance and scalability.  Such
        offloaded functions may include part of ICMP or TCP processing, or

     Yang, et. al.      Expires Dec 2003                  [Page 21] 
        part of routing protocols.  Once off-loaded onto the forwarding
        plane, such CE functions, even though logically belonging to the
        control plane, now become part of the FE functions.  Just like the
        other logical functions performed by FEs, such off-loaded functions
        must be expressed as part of the FE model so that the CEs can decide
        how to best take advantage of these off-loaded functions when
        present on the FEs.

     5.1. General Router Requirements

        Routers have at least two or more logical interfaces.  When CEs and
        FEs are separated by ForCES within a single NE, some additional
        interfaces are needed for intra-NE communications.  Figure 12 shows
        an example to illustrate that.  This NE contains one CE and two FEs.
        Each FE has four interfaces; two of them are used for receiving and
        transmitting packets to the external world, while the other two are
        for intra-NE connections.  CE has two logical interfaces #9 and #10,
        connected to interfaces #3 and #6 from FE1 and FE2, respectively.
        Interface #4 and #5 are connected for FE1-FE2 communication.
        Therefore, this router NE provides four external interfaces (#1, 2,
        7 and 8).

                     ---------------------------------
                     |               router NE       |
                     |   -----------   -----------   |
                     |   |   FE1   |   |   FE2   |   |
                     |   -----------   -----------   |
                     |   1| 2| 3| 4|   5| 6| 7| 8|   |
                     |    |  |  |  |    |  |  |  |   |
                     |    |  |  |  +----+  |  |  |   |
                     |    |  |  |          |  |  |   |
                     |    |  | 9|        10|  |  |   |
                     |    |  | -------------- |  |   |
                     |    |  | |    CE      | |  |   |
                     |    |  | -------------- |  |   |
                     |    |  |                |  |   |
                     -----+--+----------------+--+----
                          |  |                |  |
                          |  |                |  |

                     Figure 12. A router NE example with four interfaces.

        IPv4 routers must implement IP to support its packet forwarding
        function, which is driven by its FIB (Forwarding Information Base).
        This Internet layer forwarding (see RFC1812 [1] Section 5)
        functionality naturally belongs to FEs in the ForCES architecture.

        A router may implement transport layer protocols (like TCP and UDP)
        that are required to support application layer protocols (see
        RFC1812 [1] Section 6).  One important class of application
        protocols is routing protocols (see RFC1812 [1] Section 7).  In
        ForCES architecture, routing protocols are naturally implemented by
        CEs.  Routing protocols require routers communicate with each other.

     Yang, et. al.      Expires Dec 2003                  [Page 22] 
        This communication between CEs in different routers is supported in
        ForCES by FEs' ability to redirect data packets addressed to routers
        (i.e., NEs) and CEs' ability to originate packets and have them
        delivered by their FEs.  This communication occurs across Fp
        reference point inside each router and between neighboring routers'
        external interfaces, as illustrated in Figure 11.

     5.2.Link Layer

        Since FEs own all the external interfaces for the router, FEs need
        to conform to the link layer requirements in RFC1812.  Arguably, ARP
        support may be implemented in either CEs or FEs.  As we will see
        later, a number of behaviors that RFC1812 mandates fall into this
        category -- they may be performed by the FE and may be performed by
        the CE.  A general guideline is needed to ensure interoperability
        between separated control and forwarding planes.  The guideline we
        offer here is that in general CEs are required to be capable of
        these kind of operations while FEs may or may not choose to
        implement them.  FE model should indicate its capabilities in this
        regard so that CEs can decide where these functions are implemented.

        Interface parameters, including MTU, IP address, etc., must be
        configurable by CEs via ForCES.  CEs must be able to determine
        whether a physical interface in an FE is available to send packets
        or not.  FEs must also inform CEs the status change of the
        interfaces (like link up/down) via ForCES.

     5.3.Internet Layer Protocols

        Both FEs and CEs must implement IP protocol and all mandatory
        extensions as RFC1812 specified.  CEs should implement IP options
        like source route and record route while FEs may choose to implement
        those as well.  Timestamp option should be implemented by FEs to
        insert the timestamp most accurately.  FE must interpret the IP
        options that it understands and preserve the rest unchanged for use
        by CEs.  Both FEs and CEs might choose to silently discard packets
        without sending ICMP errors, but such events should be logged and
        counted.  FEs may report statistics for such events to CEs via
        ForCES.

        When multiple FEs are involved to process packets, the appearance of
        single NE must be strictly maintained.  For example, Time-To-Live
        (TTL) must be decremented only once within a single NE.  For
        example, it can be always decremented by the last FE with egress
        function.

        FEs must receive and process normally any packets with a broadcast
        destination address or a multicast destination address that the
        router has asked to receive.  When IP multicast is supported in
        routers, IGMP is implemented in CEs.  CEs are also required of ICMP
        support, while it is optional for FEs to support ICMP.  Such an
        option can be communicated to CEs as part of the FE model.
        Therefore, FEs can always rely upon CEs to send out ICMP error

     Yang, et. al.      Expires Dec 2003                  [Page 23] 
        messages, but FEs also have the option to generate ICMP error
        messages themselves.

     5.4.Internet Layer Forwarding

        IP forwarding is implemented by FEs.  When the routing table is
        updated at CEs, ForCES is used to send the new route entries from
        CEs to FEs.  Each FE has its own forwarding table and uses this
        table to direct packets to the next hop interface.

        Upon receiving IP packets, FE verifies the IP header and processes
        most of the IP options.  Some options cannot be processed until the
        routing decision has been made.  Routing decision is made after
        examining the destination IP address.  If the destination address
        belongs to the router itself, the packets are filtered and either
        processed locally or forwarded to CE, depending upon the
        instructions set-up by CE.  Otherwise, FE determines the next hop IP
        address by looking up in its forwarding table.  FE also determines
        the network interface it uses to send the packets.  Sometimes FE may
        need to forward the packets to another FE before packets can be
        forwarded out to the next hop.  Right before packets are forwarded
        out to the next hop, FE decrements TTL by 1 and processes any IP
        options that cannot be processed before.  FE performs any IP
        fragmentation if necessary, determines link layer address (e.g., by
        ARP), and encapsulates the IP datagram (or each of the fragments
        thereof) in an appropriate link layer frame and queues it for output
        on the interface selected.

        Other options mentioned in RFC1812 for IP forwarding may also be
        implemented at FEs, for example, packet filtering.

        FEs typically forward packets destined locally to CEs.  FEs may also
        forward exceptional packets (packets that FEs do not know how to
        handle) to CEs.  CEs are required to handle packets forwarded by FEs
        for whatever different reasons.  It might be necessary for ForCES to
        attach some meta-data with the packets to indicate the reasons of
        forwarding from FEs to CEs.  Upon receiving packets with meta-data
        from FEs, CEs can decide to either process the packets themselves,
        or pass the packets to the upper layer protocols including routing
        and management protocols.  If CEs are to process the packets by
        themselves, CEs may choose to discard the packets, or modify and re-
        send the packets.  CEs may also originate new packets and deliver
        them to FEs for further forwarding.

        Any state change during router operation must also be handled
        correctly according to RFC1812.  For example, when an FE ceases
        forwarding, the entire NE may continue forwarding packets, but it
        needs to stop advertising routes that are affected by the failed FE.

     5.5.Transport Layer

        Transport layer is typically implemented at CEs to support higher
        layer application protocols like routing protocols.  In practice,

     Yang, et. al.      Expires Dec 2003                  [Page 24] 
        this means that most CEs implement both the Transmission Control
        Protocol (TCP) and the User Datagram Protocol (UDP).

        Both CEs and FEs need to implement ForCES protocol.  If some layer-4
        transport is used to support ForCES, then both CEs and FEs need to
        implement the L4 transport and ForCES protocols.

     5.6. Application Layer -- Routing Protocols

        Interior and exterior routing protocols are implemented on CEs.  The
        routing packets originated by CEs are forwarded to FEs for delivery.
        The results of such protocols (like forwarding table updates) are
        communicated to FEs via ForCES.

        For performance or scalability reasons, portions of the control
        plane functions that need faster response may be moved from the CEs
        and off-loaded onto the FEs.  For example in OSPF, the Hello
        protocol packets are generated and processed periodically.  When
        done at CEs, the inbound Hello packets have to traverse from the
        external interfaces at the FEs to the CEs via the internal CE-FE
        channel.  Similarly, the outbound Hello packets have to go from the
        CEs to the FEs and to the external interfaces.  Frequent Hello
        updates place heavy processing overhead on the CEs and can overwhelm
        the CE-FE channel as well.  Since typically there are far more FEs
        than CEs in a router, the off-loaded Hello packets are processed in
        a much more distributed and scalable fashion.  By expressing such
        off-loaded functions in the FE model, we can ensure
        interoperability.

     5.7. Application Layer -- Network Management Protocol

        RFC1812 also dictates "Routers MUST be manageable by SNMP."  (see
        [4] Section 8) In general, for post-association phase, most external
        management tasks (including SNMP) should be done through interaction
        with the ForCES architecture.

        A router may implement transport layer protocols (like TCP and UDP)
        that are required CE in order to support application layer protocols (see
        RFC1812 [1] Section 6).  One important class the appearance of application
        protocols a single
        functional device.  Therefore, it is routing protocols (see RFC1812 [1] Section 7).  In
        ForCES architecture, routing protocols are naturally recommended that SNMP
        management agent be implemented by CEs and the SNMP messages
        received by FEs be redirected to their CEs.  Routing protocols require routers communicate with each other.
        This communication between  AgentX framework
        defined in RFC2741 ([5]) may be applied here such that CEs act in different routers is supported
        the role of master agent to process SNMP protocol messages while FEs
        act in
        ForCES by FEs' ability the role of subagent to redirect data packets addressed provide access to routers
        (i.e., NEs) the MIB objects
        residing on FEs.  AgentX protocol messages between the master agent
        (CE) and CEs' ability to originate packets the subagent (FE) are encapsulated and have them
        delivered by their FEs. transported via
        ForCES, just like data packets from any other application layer
        protocols.

     6. Summary

        This communication occurs across Fp
        reference point inside each router document defines an architectural framework for ForCES.  It
        identifies the relevant components for a ForCES network element,
        including (one or more) FEs, (one or more) CEs, one optional FE
        manager, and between neighboring routers'
        external interfaces, as illustrated in Figure 11.

     5.2.Link Layer

        Since FEs own one optional CE manager.  It also identifies the
        interaction among these components and discusses all the major

     Yang, et. al.      Expires Dec 2003                  [Page 25] 
        reference points.  It is important to point out that, among all the external interfaces for
        reference points, only the router, Fp interface between CEs and FEs need
        to conform to is
        within the link layer requirements in RFC1812.  Arguably, ARP
        support scope of ForCES.  ForCES alone may not be implemented enough to
        support all desirable NE configurations.  However, we believe that
        ForCES over Fp interface is the most important element in either realizing
        physical separation and interoperability of CEs or and FEs, and hence
        the first interface that ought to be standardized.  Simple and
        useful configurations can still be implemented with only CE-FE
        interface being standardized, e.g., single CE with full-meshed FEs.  As we will see
        later,

     7. Security Considerations

        In general, the physical separation of two entities usually results
        in a number potentially insecure link between the two entities and hence
        much stricter security measurements are required.  For example, we
        pointed out in Section 4.1 that authentication becomes necessary
        between CE manager and FE manager, between CE and CE manager,
        between FE and FE manager in some configurations.  The physical
        separation of behaviors CE and FE also imposes serious security requirement
        for ForCES protocol over Fp interface.  This section first attempts
        to describe the security threats that RFC1812 mandates fall into this
        category -- they may be performed introduced by the FE
        physical separation of the FEs and may be performed by the CE.  A general guideline is needed to ensure interoperability
        between separated control CEs, and forwarding planes.  The guideline we
        offer here then it provides
        recommendation and guidelines for secure operation and management of
        ForCES protocol over Fp interface.

     7.1. Analysis of Potential Threats Introduced by ForCES

        This section provides the threat analysis for ForCES, with a focus
        on Fp interface. Each threat is that described in general CEs are details with the
        effects on the ForCES protocol entities or/and the NE as a whole,
        and the required functionalities that need to be capable in place to defend
        the threat.

     7.1.1. Join or Remove Flooding on CEs

        Threats:  A malicious node could send a stream of
        these kind false join NE or
        remove from NE requests on behalf of operations while FEs may non-existent or may not choose to
        implement them. unauthorized
        FE model should indicate its capabilities in this
        regard so that CEs can decide where these functions are implemented.

        Interface parameters, including MTU, IP address, etc., must be
        configurable by CEs via ForCES.  CEs must be able to determine
        whether legitimate CEs at a physical interface very rapid rate and thereby create
        unnecessary state in an FE is available to send packets
        or not.  FEs must also inform CEs the status change CEs.

        Effects: If by maintaining state for non-existent or unauthorized
        FEs, a CE may become unavailable for other processing and hence
        suffer from denial of service (DoS) attack similar to the
        interfaces (like link up/down) via ForCES.

     5.3.Internet Layer Protocols

        Both FEs and CEs must implement IP protocol and all mandatory
        extensions as RFC1812 specified. TCP SYN
        DoS. If multiple CEs should implement IP options
        like source route and record route while FEs are used, the unnecessary state information may choose to implement
        those as well.  Timestamp option should
        also be implemented by FEs conveyed to
        insert the timestamp most accurately.  FE must interpret multiple CEs via Fr interface (e.g., from the IP
        options that it understands and preserve
        active CE to the rest unchanged for use
        by CEs.  Both FEs stand-by CE) and hence subject multiple CEs might choose to silently discard packets
        without sending ICMP errors, but such events DoS
        attack.

        Requirement:  A CE that receives a join or remove request should be logged and
        not create any state information until it has authenticated the FE
        endpoint.

     Yang, et. al.      Expires June Dec 2003                  [Page 20] 
        counted.  FEs may report statistics for such events to CEs via
        ForCES.

        When multiple FEs are involved to process packets, the appearance of
        single NE must be strictly maintained.  For example, Time-To-Live
        (TTL) must be decremented only once within a single NE.  For
        example, it 26] 
     7.1.2. Impersonation Attack

        Threats: A malicious node can impersonate a CE or FE and send out
        false messages.

        Effects: The whole NE could be always decremented by the last compromised.

        Requirement:  The CE or FE with egress
        function.

        FEs must receive authenticate the message before
        accepting and process normally any packets with processing it.

     7.1.3. Replay Attack

        Threat: A malicious node could replay the entire message previously
        sent by a broadcast
        destination address FE or a multicast destination address that the
        router has asked to receive.  When IP multicast is supported in
        routers, IGMP is implemented in CEs.  CEs are also required of ICMP
        support, while it is optional for FEs CE entity to support ICMP.  Such an
        option can get around authentication.

        Effect: The NE could be communicated compromised.

        Requirement:  Replay protection mechanism needs to CEs as be part of the FE model.
        Therefore, FEs can always rely upon CEs
        security protocol to send out ICMP error
        messages, defend this attack.

     7.1.4. Attack during Fail Over

        Threat: A malicious node may exploit the CE fail-over mechanism to
        take over the control of NE. For example, suppose two CEs, say CE-A
        and CE-B, are controlling several FEs. CE-A is active and CE-B is
        stand-by. When CE-A fails, CE-B is taking over the active CE
        position.  The FEs already had a trusted relationship with CE-A, but
        the FEs also may not have the option same trusted relationship established with
        CE-B prior to generate ICMP error
        messages themselves.

     5.4.Internet Layer Forwarding

        IP forwarding the fail-over. A malicious node can take over as CE-B
        if such trusted relationship is implemented by FEs.  When not established during the fail-
        over.

        Effect: The NE may be compromised after such insecure fail-over.

        Requirement:  The level of trust relationship between the stand-by
        CE and the FEs must be as strong as the one between the routing table is
        updated at CEs, ForCES is used to send active CE
        and the new route entries from
        CEs to FEs.  Each FE has its own forwarding table FEs, and uses this
        table to direct packets such security association must be re-established
        during fail-over.

     7.1.5.  Data Integrity

        Threats: A malicious node may inject false messages to the next hop interface.

        Upon receiving IP packets, legitimate CE
        or FE.

        Effect: An FE verifies or CE receives the IP header fabricated packet and processes
        most of performs
        incorrect or catastrophic operation.

        Requirement: Protocol messages require integrity protection.

     7.1.6.  Data Confidentiality

     Yang, et. al.      Expires Dec 2003                  [Page 27] 
        Threat: When FE and CE are physically separated, a malicious node
        may eavesdrop the IP options. messages in transit. Some options can't be processed until of the
        routing decision has been made.  Routing decision is made after
        examining messages are
        critical to the destination IP address.  If functioning of the destination address
        belongs whole network, while others may
        contain confidential business data. Leaking of such information may
        result in compromise even beyond the immediate CE or FE.

        Effect: Sensitive information might be exposed between CE and FE.

        Requirement: Data confidentiality between FE and CE must be
        available for sensitive information.

     7.1.7. Sharing security parameters

        Threat: Consider a scenario where several FEs communicating to the router itself,
        same CE share the packets are filtered and either
        processed locally same authentication keys for the Fp interface. If
        any FE or forwarded the CE is compromised, all other entities are compromised.

        Effect: The whole NE is compromised.

        Requirement: To avoid this side effect, its better to configure
        different security parameters for each FE-CE communication over Fp
        interface.

     7.1.8. Denial of Service Attack via External Interface

        Threat: When an FE receives a packet that is destined for its CE, depending upon
        the
        instructions set-up by CE.  Otherwise, FE determines forwards the next hop IP
        address by looking up in its forwarding table.  FE also determines packet over the Fp interface. Malicious node can
        generate huge message storm like routing protocol packets etc.
        through the network external Fi/f interface it uses to send so that the packets.  Sometimes FE may
        need has to process
        and forward the all packets to another FE before CE through Fp interface.

        Effect: CE encounters resource exhaustion and bandwidth starvation
        on Fp interface due to an overwhelming number of packets from FEs.

        Requirement: Rate limiting mechanism needs to be in place at both FE
        and CE. Rate Limiter can be
        forwarded out to the next hop.  Right before packets configured at FE for each message type
        that are forwarded
        out being received through Fi/F interface.

     7.2. Security Recommendations for ForCES

        The requirements document [3] suggested that ForCES protocol should
        support reliability over Fp interface, but no particular transport
        protocol is yet specified for ForCES. This framework document does
        not intend to specify the next hop, FE decrements TTL by 1 particular transport either, and processes any IP
        options so we
        only provide recommendations and guidelines based on the existing
        standard security protocols that cannot can work with the common transport
        candidates suitable for ForCES.

        We highlight two existing security protocol solutions, namely IPsec
        (IP Security) [14] or TLS (Transport Layer Security) [13]. TLS works
        with reliable transports such as TCP or SCTP, while IPsec can be processed before.  FE performs
        used with any IP
        fragmentation transport (UDP, TCP, SCTP). Both TLS and IPsec can be

     Yang, et. al.      Expires Dec 2003                  [Page 28] 
        used potentially to satisfy all of the security requirements for
        ForCES protocol.

        It is important to realize that even if necessary, determines link layer address (e.g., by
        ARP), the NE is in a single-box,
        the DoS attacks can still be launched through Fi/f interfaces.
        Therefore, it is still important to have counter-measurement as
        stated in 1.1.9 for DoS while authentication, confidentiality and encapsulates
        integrity can be provided by the IP datagram (or physical security of the box.

     7.2.1. Security Configuration

        The NE administrator has the freedom to determine the exact security
        configuration that is needed for the specific deployment.  For
        example, ForCES may be deployed between CEs and FEs connected to
        each other inside a box over a backplane.  In such scenario,
        physical security of the fragments
        thereof) in an appropriate link layer frame box ensures that most of the attacks such
        as man-in-the-middle, snooping, and queues it for output impersonation are not possible,
        and hence ForCES architecture may rely on the interface selected.

        Other options mentioned in RFC1812 for IP forwarding may also be
        implemented at FEs, for example, packet filtering.

        FEs typically forward packets destined locally physical security of
        the box to CEs.  FEs defend against these attacks and protocol mechanisms may
        be turned off.  However, it is also
        forward exceptional packets (packets shown that FEs don't know how to
        handle) denial of service
        attack via external interface as described in Section 7.1.8 is still
        a potential threat even for such all-in-one-box deployment
        scenario and hence the rate limiting mechanism is still necessary.
        This is just one example to CEs.  CEs are required show that it is important to handle packets forwarded by FEs
        for whatever assess the
        security needs of the ForCES-enabled network elements under
        different reasons. deployment scenarios.  It might should be necessary possible for ForCES to
        attach some meta-data with the packets
        administrator to indicate configure the reasons level of
        forwarding from FEs to CEs.  Upon receiving packets security needed for the
        ForCES protocol.

     7.2.2. Using TLS with meta-data

     Yang, et. al.      Expires June 2003                  [Page 21] 
        from FEs, CEs ForCES

        TLS [13] can decide be used if a reliable transport such as TCP or SCTP is
        used for ForCES over Fp interface. The TLS handshake protocol is
        used during association establishment or re-establishment phase to either process
        negotiate a TLS session between the packets themselves,
        or pass CE and FE.  Once the session is
        in place, the TLS record protocol is used to secure ForCES
        communication messages between the CE and FE.

        A basic outline of how TLS can be used with ForCES is described
        below.  Steps 1) till 7) complete the security handshake as
        illustrated in Figure 9 while step 8) is for all the packets to further
        communication between the upper layer protocols CE and FE, including routing the rest of messages
        after the security handshake shown in Figure 9 and management protocols.  If CEs the steady-state
        communication shown in Figure 10.

        1) During Pre-association phase all FEs are to process configured with the packets by
        themselves, CEs may choose to discard
        (including both the packets, or modify active CE and re-
        send the packets.  CEs may also originate new packets standby CE).
        2) The FE establishes a TLS connection with the CE (master) and deliver
        them to FEs for further forwarding.

        Any state change during router operation must also be handled
        correctly according to RFC1812.  For example, when an
        negotiates a cipher suite.
        3) The FE ceases
        forwarding, (slave) gets the entire NE may continue forwarding packets, but it
        needs to stop advertising routes that are affected by CE certificate, validates the failed FE.

     5.5.Transport Layer

        Transport layer is typically implemented at CEs to support higher
        layer application protocols like routing protocols.  In practice,
        this means that most CEs implement both signature,
        checks the Transmission Control
        Protocol (TCP) and expiration date, checks if the User Datagram Protocol (UDP).

        Both CEs certificate has been
        revoked.

     Yang, et. al.      Expires Dec 2003                  [Page 29] 
        4) The CE (master) gets the FE certificate and FEs need to implement ForCES protocol. performs the same
        validation as the FE in step 4).
        5) If some layer-4
        transport is used to support ForCES, then both CEs and FEs need to
        implement any of the L4 transport check fails in step 5 or step 6, endpoint must
        generate an error message and ForCES protocols.  It abort.
        6) After successful mutual authentication, a TLS session is possible
        that all FEs inside an NE implements only one such protocol entity.

     5.6. Application Layer -- Routing Protocols

        Interior
        established between CE and exterior routing protocols are implemented on CEs. FE.
        7) The
        routing packets originated by CEs are forwarded FE sends a join NE message to FEs for delivery. the CE.
        8) The results of such protocols (like forwarding table updates) FE and CE use TLS session for further communication.

        Note that there are
        communicated different ways for the CE and FE to FEs via ForCES.

        For performance validate a
        received certificate. One way is to configure the FE Manager or scalability reasons, portions of CE
        Manager or other central component as CA, so that the control
        plane functions CE or FE can
        query this pre-configured CA to validate that need faster response may be moved from the CEs certificate has
        not been revoked.  Another way is to have the CE and off-loaded onto the FEs.  For example FE
        configured directly a list of valid certificates in OSPF, the Hello
        protocol packets are generated pre-
        association phase.

        In the case of fail-over, it is the responsibility of the active CE
        and processed periodically.  When
        done at CEs, the inbound Hello packets have standby CE to traverse from synchronize ForCES states including the
        external interfaces at TLS
        states to minimize the FEs state reestablishment during fail-over. Care
        must be taken to ensure that the CEs via standby CE is also authenticated in
        the internal CE-FE
        channel.  Similarly, same way as the outbound Hello packets have to go from active CE, either before or during the fail-
        over.

     7.2.3. Using IPsec with ForCES

        IPsec [14] can be used with any transport protocol, such as UDP,
        SCTP and TCP over Fp interface for ForCES.  We recommend using ESP
        in transport mode for ForCES because message confidentiality is
        required for ForCES and the communication between the CE and FE is
        point-to-point.

        IPsec can be used with both manual and automated SA and
        cryptographic key management.  But Ipsecs replay protection
        mechanisms are not available if manual key management is used.
        Hence, automatic key management is recommended if replay protection
        is deemed important.  Otherwise, manual key management might be
        sufficient for some deployment scenarios, esp. when the number of
        CEs to the FEs and to FEs is relatively small.  It is recommended that the external interfaces.  Frequent Hello
        updates place heavy processing overhead on keys be
        changed periodically even for manual key management.

        Unlike TLS, IPsec provides security services between the CEs CE and can overwhelm FE
        at IP level, and so the CE-FE channel security handshake as well.  Since typically there are far more FEs
        than CEs illustrated in Figure
        9 amounts to a router, no-op when manual key management is used.  The
        following outline the off-loaded Hello packets are processed steps taken for ForCES in
        a much more distributed and scalable fashion.  By expressing such
        off-loaded functions in the FE model, we can ensure
        interoperability.

     5.7. Application Layer -- Network Management Protocol

        RFC1812 also dictates "Routers MUST be manageable by SNMP."  (see
        [4] Section 8) In general, for post-association phase, most external
        management tasks (including SNMP) should be done through interaction a case.

        1) During Pre-association phase all FEs are configured with the CEs
        (including active CE in order and standby CE) and SA parameters manually.
        2) The FE sends a join NE message to support the appearance of a single
        functional device.  Therefore, it is recommended CE.  This message and all
        others that SNMP follow are afforded security service according to the
        manually configured IPsec SA parameters, but replay protection is
        not available.

     Yang, et. al.      Expires June Dec 2003                  [Page 22] 
        management agent 30] 
        It is up to the administrator to decide whether to share the same
        key across multiple FE-CE communication, but it is recommended that
        different keys be implemented by CEs used.  Similarly, it is recommended that different
        keys be used for inbound and the SNMP messages
        received by FEs outbound traffic.

        If automatic key management is needed, IKE [15] can be redirected to their CEs.  AgentX framework
        defined in RFC2741 ([5]) used for that
        purpose. Other automatic key distribution techniques such as
        Kerberos may be applied here such that CEs act used as well.  The key exchange process constitutes
        the security handshake as illustrated in Figure 9. The following
        shows the role of master agent steps involved in using IKE with IPsec for ForCES. Steps
        1) to process SNMP protocol messages while FEs
        act 6) constitute the security handshake in Figure 9.

        1) During Pre-association phase all FEs are configured with the role of subagent to provide access CEs
        (including active CE and standby CE), IPsec policy etc.
        2) The FE kicks off IKE process and tries to establish an IPsec SA
        with the MIB objects
        residing on FEs.  AgentX protocol messages between CE (master). The FE (Slave) gets the master agent
        (CE) and CE certificate as part
        of the subagent (FE) are encapsulated IKE negotiation. The FE validates signature, checks the
        expiration date, checks if the certificate has been revoked.
        3) The CE (master) gets the FE certificate and transported via
        ForCES, just like data packets from performs the same
        check as the FE in step 3).
        4) If any other application layer
        protocols.

     6. Summary

        This document defines of the check fails in step 3 or step 4, the endpoint must
        generate an architectural framework for ForCES.  It
        identifies error message and abort.
        5) After successful mutual authentication, IPsec session is
        established between the relevant components for CE and FE.
        6) The FE sends a ForCES network element,
        including (one or more) FEs, (one or more) CEs, one optional join NE message to CE. No SADB entry is created
        in FE yet.
        7) The FE
        manager, and one optional CE manager.  It also identifies use the
        interaction among these components IPsec session for further communication.

        FE Manager or CE Manager or other central component can be used as
        CA for validating CE and discusses all FE certificates during the major
        reference points.  It is important to point out that, among all IKE process.
        Alternatively, during the
        reference points, only pre-association phase, the interface between CEs CE and FEs is within FE can
        be configured directly with the scope required information such as
        certificates or passwords etc depending upon the type of ForCES.  ForCES alone may not be enough to support all
        desirablet NE configurations.  However, we believe
        authentication that ForCES administrator wants to configure.

        In the case of fail-over, it is the most important element in realizing physical separation and
        interoperability responsibility of CEs active CE and FEs,
        standby CE to synchronize ForCES states and hence IPsec states to minimize
        the first interface that
        ought state reestablishment during fail-over. Alternatively, the FE
        needs to be standardized.  Simple and useful configurations can
        still be implemented with only CE-FE interface being standardized,
        e.g., single CE establish different IPsec SA during the startup operation
        itself with full-meshed FEs and static configuration
        without each CEs. This will minimize the need for CE/FE managers.

     7. periodic state
        transfer across IPsec layer though Fr (CE-CE) Interface.

     8. Normative References

        [1] F. Baker, "Requirements Requirements for IP Version 4 Routers", RFC1812, June
        1995.

        [2] S. Floyd, "Congestion Congestion Control Principles", RFC2914, September
        2000.

     Yang, et. al.      Expires Dec 2003                  [Page 31] 
        [3] H. Khosravi, et. al., "Requirements Requirements for Separation of IP Control
        and Forwarding", work in progress, Oct 2002, May 2003, <draft-ietf-forces-
        requirements-07.txt>.

     8.
        requirements-09.txt>.

     9. Informative References

        [4] J. Case, et. al., "A A Simple Network Management Protocol (SNMP)",
        RFC1157, May 1990.

        [5] M. Daniele, et. al., "Agent Agent Extensibility (AgentX) Protocol
        Version 1", RFC2741, Jan 2000.

        [6] K. Chan, et. al., "COPS COPS Usage for Policy Provisioning (COPS-
        PR)", RFC3084, March 2001.

        [7] A. Crouch, et. al., "ForCES ForCES Applicability Statement", work in
        progress, June 2002, <draft-ietf-forces-applicability-00.txt>.

     Yang, et. al.      Expires June 2003                  [Page 23] 
     9. Security Considerations

        The security necessary across each reference point except Fp is
        discussed throughout the document.  In general,

        [8] T. Anderson, J. Buerkle, Requirements for the physical
        separation Dynamic
        Partitioning of two entities usually requires much stricter security
        measurement Switching Elements", RFC3532, May 2003.

        [9] M. Leelanivas, et. al., Graceful Restart Mechanism for Label
        Distribution Protocol, RFC 3478, Feb. 2003.

        [10] J. Moy, et. al., Graceful OSPF Restart, work in place.  For example, we pointed out progress,
        March 2003, <draft-ietf-ospf-hitless-restart-07.txt>.

        [11] S. R. Sangli, et. al., Graceful Restart Mechanism for BGP,
        work in Section 5.1
        that authentication becomes necessary between CE manager and FE
        manager, between CE and CE manager, between FE progress, Jan 203, < draft-ietf-idr-restart-06.txt>.

        [12] M. Shand and FE manager L. Ginsberg, Restart Signaling for IS-IS, work
        in
        some configuration.  The physical separation of CE progress, March 2003, <draft-ietf-isis-restart-03.txt>.

        [13] T. Dierks and FE also
        imposes serious security requirement C. Allen, The TLS Protocol, version 1.0,
        RFC2246, Jan. 1999.

        [14] S. Kent and R. Atkinson, Security Architecture for ForCES protocol.  The
        security requirements the
        Internet Protocol, RFC2401, Nov. 1998.

        [15] D. Harkins and D. Carrel, The Internet Key Exchange (IKE),
        RFC2409, Nov. 1998.

        [16] S. M. Bellovin, Guidelines for reference point Fp (i.e., ForCES protocol)
        are discussed in detail Mandating the Use of Ipsec,
        work in [3] Section 8. progress, Oct. 2002, <draft-bellovin-useipsec-00.txt>.

     10. Acknowledgments Acknowledgement

        Joel M. Halpern gave us many insightful comments and suggestions and
        pointed out several major issues.  T. Sridhar suggested that the
        AgentX protocol could be used with SNMP to manage the ForCES network
        elements.  Many of our colleagues and people in the ForCES mailing
        list also provided valuable feedback.

     Yang, et. al.      Expires Dec 2003                  [Page 32] 
     11. Authors' Addresses

        Lily L. Yang
        Intel Corp., MS JF3-206,
        2111 NE 25th Avenue
        Hillsboro, OR 97124 USA
        Phone: +1 503 264 8813
        Email: lily.l.yang@intel.com

        Ram Dantu
        Department of Computer Science,
        University of Texas Dallas
        2601 North Flyod Road
        Richardson Texas 75082 Texas,
        Denton, Texas, 76203
        Phone: +1 972 883 4653 940 565 2822
        Email: ram.dantu@utdallas.edu rdantu@unt.edu

        Todd A. Anderson
        Intel Corp.
        2111 NE 25th Avenue
        Hillsboro, OR 97124 USA
        Phone: +1 503 712 1760
        Email: todd.a.anderson@intel.com

        Ram Gopal
        Nokia Research Center
        5, Wayside Road,
        Burlington, MA 01803
        Phone: +1 781 993 3685
        Email: ram.gopal@nokia.com

     12. Intellectual Property Right

        The authors are not aware of any intellectual property right issues
        pertaining to this document.

     13. Full Copyright Statement

     Yang, et. al.      Expires June 2003                  [Page 24]

        Copyright (C) The Internet Society (2002). All Rights Reserved.

        This document and translations of it may be copied and furnished to
        others, and derivative works that comment on or otherwise explain it
        or assist in its implementation may be prepared, copied, published
        and distributed, in whole or in part, without restriction of any
        kind, provided that the above copyright notice and this paragraph
        are included on all such copies and derivative works.  However, this
        document itself may not be modified in any way, such as by removing
        the copyright notice or references to the Internet Society or other
        Internet organizations, except as needed for the purpose of
        developing Internet standards in which case the procedures for
        copyrights defined in the Internet Standards process must be
        followed, or as required to translate it into languages other than
        English.

     Yang, et. al.      Expires Dec 2003                  [Page 33] 
        The limited permissions granted above are perpetual and will not be
        revoked by the Internet Society or its successors or assigns.
        This document and the information contained herein is provided on an
        "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
        TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
        BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
        HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
        MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."

     Yang, et. al.      Expires June Dec 2003                  [Page 25] 34]