draft-ietf-forces-framework-05.txt   draft-ietf-forces-framework-06.txt 
Internet Draft L. Yang Internet Draft L. Yang
Expiration: Dec 2003 Intel Corp. Expiration: Dec 2003 Intel Corp.
File: draft-ietf-forces-framework-05.txt R. Dantu File: draft-ietf-forces-framework-06.txt R. Dantu
Working Group: ForCES Univ. of North Texas Working Group: ForCES Univ. of North Texas
T. Anderson T. Anderson
Intel Corp. Intel Corp.
R. Gopal R. Gopal
Nokia Nokia
June 2003 June 2003
Forwarding and Control Element Separation (ForCES) Framework Forwarding and Control Element Separation (ForCES) Framework
draft-ietf-forces-framework-05.txt draft-ietf-forces-framework-06.txt
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. Internet-Drafts are all provisions of Section 10 of RFC2026. Internet-Drafts are
working documents of the Internet Engineering Task Force (IETF), its working documents of the Internet Engineering Task Force (IETF),
areas, and its working groups. Note that other groups may also its areas, and its working groups. Note that other groups may also
distribute working documents as Internet-Drafts. distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other
at any time. It is inappropriate to use Internet-Drafts as documents at any time. It is inappropriate to use Internet-Drafts
reference material or to cite them other than as ``work in as reference material or to cite them other than as ``work in
progress.'' progress.''
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved. Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract Abstract
This document defines the architectural framework for the ForCES This document defines the architectural framework for the ForCES
(Forwarding and Control Element Separation) network elements, and (Forwarding and Control Element Separation) network elements, and
identifies the associated entities and the interaction among them. identifies the associated entities and the interaction among them.
Table of Contents Table of Contents
1. Definitions.....................................................3 1. Definitions...................................................3
2. Introduction to Forwarding and Control Element Separation 2. Introduction to Forwarding and Control Element Separation
(ForCES)...........................................................5 (ForCES).........................................................5
3. Architecture....................................................8 3. Architecture..................................................9
3.1. Control Elements and Fr Reference Point....................9 3.1. Control Elements and Fr Reference Point.................10
3.2. Forwarding Elements and Fi reference point................10 3.2. Forwarding Elements and Fi reference point..............11
3.3. CE Managers...............................................13 3.3. CE Managers.............................................14
3.4. FE Managers...............................................13 3.4. FE Managers.............................................14
4. Operational Phases.............................................13 4. Operational Phases...........................................14
4.1. Pre-association Phase.....................................13 4.1. Pre-association Phase...................................15
4.1.1. Fl Reference Point...................................13 4.1.1. Fl Reference Point.................................15
4.1.2. Ff Reference Point...................................14 4.1.2. Ff Reference Point.................................16
4.1.3. Fc Reference Point...................................15 4.1.3. Fc Reference Point.................................17
4.2. Post-association Phase and Fp reference point.............15 4.2. Post-association Phase and Fp reference point...........17
4.2.1. Proximity and Interconnect between CEs and FEs.......15 4.2.1. Proximity and Interconnect between CEs and FEs.....17
4.2.2. Association Establishment............................16 4.2.2. Association Establishment..........................18
4.2.3. Steady-state Communication...........................17 4.2.3. Steady-state Communication.........................19
4.2.4. Data Packets across Fp reference point...............18 4.2.4. Data Packets across Fp reference point.............20
4.2.5. Proxy FE.............................................19 4.2.5. Proxy FE...........................................21
4.3. Association Re-establishment..............................19 4.3. Association Re-establishment............................21
4.3.1. CE graceful restart..................................19 4.3.1. CE graceful restart................................21
4.3.2. FE restart...........................................20 4.3.2. FE restart.........................................23
5. Applicability to RFC1812.......................................21 5. Applicability to RFC1812.....................................24
5.1. General Router Requirements...............................22 5.1. General Router Requirements.............................24
5.2. Link Layer................................................23 5.2. Link Layer..............................................25
5.3. Internet Layer Protocols..................................23 5.3. Internet Layer Protocols................................26
5.4. Internet Layer Forwarding.................................24 5.4. Internet Layer Forwarding...............................26
5.5. Transport Layer...........................................24 5.5. Transport Layer.........................................27
5.6. Application Layer -- Routing Protocols....................25 5.6. Application Layer -- Routing Protocols..................28
5.7. Application Layer -- Network Management Protocol..........25 5.7. Application Layer -- Network Management Protocol........28
6. Summary........................................................25 6. Summary......................................................29
7. Security Considerations........................................26 7. Security Considerations......................................29
7.1. Analysis of Potential Threats Introduced by ForCES........26 7.1. Analysis of Potential Threats Introduced by ForCES......29
7.1.1. Join or Remove Flooding on CEs...................26 7.1.1. "Join" or "Remove" Message Flooding on CEs.........29
7.1.2. Impersonation Attack.................................27 7.1.2. Impersonation Attack...............................30
7.1.3. Replay Attack........................................27 7.1.3. Replay Attack......................................30
7.1.4. Attack during Fail Over..............................27 7.1.4. Attack during Fail Over............................30
7.1.5. Data Integrity.......................................27 7.1.5. Data Integrity.....................................31
7.1.6. Data Confidentiality.................................27 7.1.6. Data Confidentiality...............................31
7.1.7. Sharing security parameters..........................28 7.1.7. Sharing security parameters........................31
7.1.8. Denial of Service Attack via External Interface......28 7.1.8. Denial of Service Attack via External Interface....32
7.2. Security Recommendations for ForCES.......................28 7.2. Security Recommendations for ForCES.....................32
7.2.1. Security Configuration...............................29 7.2.1. Security Configuration.............................32
7.2.2. Using TLS with ForCES................................29 7.2.2. Using TLS with ForCES..............................33
7.2.3. Using IPsec with ForCES..............................30 7.2.3. Using IPsec with ForCES............................34
8. Normative References...........................................31 8. Normative References.........................................35
9. Informative References.........................................32 9. Informative References.......................................36
10. Acknowledgement...............................................32
11. Authors' Addresses............................................33 Yang, et al. Expires December 2003 [Page 2]
12. Intellectual Property Right...................................33 10. Acknowledgements............................................36
13. Full Copyright Statement......................................33 11. Authors' Addresses..........................................37
12. Intellectual Property Right.................................37
13. Full Copyright Statement....................................38
Yang, et. al. Expires Dec 2003 [Page 2]
Conventions used in this document Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in [RFC-2119]. this document are to be interpreted as described in [RFC-2119].
1. Definitions 1. Definitions
A set of terminology associated with the ForCES requirements is A set of terminology associated with the ForCES requirements is
defined in [3] and we only include the ones that are most relevant defined in [3] and we only include the definitions that are most
to this document here. relevant to this document here.
Addressable Entity (AE) - - An entity that is directly addressable Addressable Entity (AE) - An entity that is directly addressable
given some interconnect technology. For example, on IP networks, it given some interconnect technology. For example, on IP networks,
is a device to which we can communicate using an IP address; and on it is a device to which we can communicate using an IP address; on
a switch fabric, it is a device to which we can communicate using a a switch fabric, it is a device to which we can communicate using a
switch fabric port number. switch fabric port number.
Physical Forwarding Element (PFE) - An AE that includes hardware Physical Forwarding Element (PFE) - An AE that includes hardware
used to provide per-packet processing and handling. This hardware used to provide per-packet processing and handling. This hardware
may consist of (but is not limited to) network processors, ASICs, or may consist of (but is not limited to) network processors, ASICs
general processors, installed on line cards, daughter boards, (Application-Specific Integrated Circuits), or general processors,
mezzanine cards, or in stand-alone boxes. installed on line cards, daughter boards, mezzanine cards, or in
stand-alone boxes.
PFE Partition - A logical partition of a PFE consisting of some PFE Partition - A logical partition of a PFE consisting of some
subset of each of the resources (e.g., ports, memory, forwarding subset of each of the resources (e.g., ports, memory, forwarding
table entries) available on the PFE. This concept is analogous to table entries) available on the PFE. This concept is analogous to
that of the resources assigned to a virtual switching element as that of the resources assigned to a virtual switching element as
described in [8]. described in [8].
Physical Control Element (PCE) - An AE that includes hardware used Physical Control Element (PCE) - An AE that includes hardware used
to provide control functionality. This hardware typically includes to provide control functionality. This hardware typically includes
a general-purpose processor. a general-purpose processor.
PCE Partition - A logical partition of a PCE consisting of some PCE Partition - A logical partition of a PCE consisting of some
subset of each of the resources available on the PCE. subset of each of the resources available on the PCE.
Forwarding Element (FE) - A logical entity that implements the Forwarding Element (FE) - A logical entity that implements the
ForCES protocol. FEs use the underlying hardware to provide per- ForCES protocol. FEs use the underlying hardware to provide per-
packet processing and handling as directed by a CE via the ForCES packet processing and handling as directed by a CE via the ForCES
protocol. FEs may happen to be a single blade (or PFE), a partition
of a PFE or multiple PFEs. Yang, et al. Expires December 2003 [Page 3]
protocol. FEs may happen to be a single blade (or PFE), a
partition of a PFE or multiple PFEs.
Control Element (CE) - A logical entity that implements the ForCES Control Element (CE) - A logical entity that implements the ForCES
protocol and uses it to instruct one or more FEs how to process protocol and uses it to instruct one or more FEs how to process
packets. CEs handle functionality such as the execution of control packets. CEs handle functionality such as the execution of control
and signaling protocols. CEs may consist of PCE partitions or whole and signaling protocols. CEs may consist of PCE partitions or
PCEs. whole PCEs.
ForCES Network Element (NE) - An entity composed of one or more CEs ForCES Network Element (NE) - An entity composed of one or more CEs
and one or more FEs. To entities outside a NE, the NE represents a and one or more FEs. To entities outside an NE, the NE represents
single point of management. Similarly, a NE usually hides its a single point of management. Similarly, an NE usually hides its
internal organization from external entities. internal organization from external entities.
Yang, et. al. Expires Dec 2003 [Page 3] Pre-association Phase - The period of time during which an FE
Pre-association Phase - The period of time during which a FE Manager Manager (see below) and a CE Manager (see below) are determining
(see below) and a CE Manager (see below) are determining which FE which FE and CE should be part of the same network element.
and CE should be part of the same network element.
Post-association Phase - The period of time during which a FE does Post-association Phase - The period of time during which an FE does
know which CE is to control it and vice versa, including the time know which CE is to control it and vice versa, including the time
during which the CE and FE are establishing communication with one during which the CE and FE are establishing communication with one
another. another.
ForCES Protocol - While there may be multiple protocols used within ForCES Protocol - While there may be multiple protocols used within
the overall ForCES architecture, the term "ForCES protocol" refers the overall ForCES architecture, the term "ForCES protocol" refers
only to the ForCES post-association phase protocol (see below). only to the ForCES post-association phase protocol (see below).
ForCES Post-Association Phase Protocol - The protocol used for post- ForCES Post-Association Phase Protocol - The protocol used for
association phase communication between CEs and FEs. This protocol post-association phase communication between CEs and FEs. This
does not apply to CE-to-CE communication, FE-to-FE communication, or protocol does not apply to CE-to-CE communication, FE-to-FE
to communication between FE and CE managers. The ForCES protocol is communication, nor to communication between FE and CE managers.
a master-slave protocol in which FEs are slaves and CEs are masters. The ForCES protocol is a master-slave protocol in which FEs are
This protocol includes both the management of the communication slaves and CEs are masters. This protocol includes both the
channel (e.g., connection establishment, heartbeats) and the control management of the communication channel (e.g., connection
messages themselves. This protocol could be a single protocol or establishment, heartbeats) and the control messages themselves.
could consist of multiple protocols working together. This protocol could be a single protocol or could consist of
multiple protocols working together.
FE Manager - A logical entity that operates in the pre-association FE Manager - A logical entity that operates in the pre-association
phase and is responsible for determining to which CE(s) a FE should phase and is responsible for determining to which CE(s) an FE
communicate. This process is called CE discovery and may involve should communicate. This process is called CE discovery and may
the FE manager learning the capabilities of available CEs. A FE involve the FE manager learning the capabilities of available CEs.
manager may use anything from a static configuration to a pre- A FE manager may use anything from a static configuration to a pre-
association phase protocol (see below) to determine which CE(s) to association phase protocol (see below) to determine which CE(s) to
use, however this is currently out of scope. Being a logical use, however this is currently out of scope. Being a logical
entity, a FE manager might be physically combined with any of the entity, an FE manager might be physically combined with any of the
other logical entities mentioned in this section. other logical entities mentioned in this section.
Yang, et al. Expires December 2003 [Page 4]
CE Manager - A logical entity that operates in the pre-association CE Manager - A logical entity that operates in the pre-association
phase and is responsible for determining to which FE(s) a CE should phase and is responsible for determining to which FE(s) a CE should
communicate. This process is called FE discovery and may involve communicate. This process is called FE discovery and may involve
the CE manager learning the capabilities of available FEs. A CE the CE manager learning the capabilities of available FEs. A CE
manager may use anything from a static configuration to a pre- manager may use anything from a static configuration to a pre-
association phase protocol (see below) to determine which FE to use, association phase protocol (see below) to determine which FE to
however this is currently out of scope. Being a logical entity, a use, however this is currently out of scope. Being a logical
CE manager might be physically combined with any of the other entity, a CE manager might be physically combined with any of the
logical entities mentioned in this section. other logical entities mentioned in this section.
Pre-association Phase Protocol - A protocol between FE managers and Pre-association Phase Protocol - A protocol between FE managers and
CE managers that is used to determine which CEs or FEs to use. A CE managers that is used to determine which CEs or FEs to use. A
pre-association phase protocol may include a CE and/or FE capability pre-association phase protocol may include a CE and/or FE
discovery mechanism. Note that this capability discovery process is capability discovery mechanism. Note that this capability
wholly separate from (and does not replace) that used within the discovery process is wholly separate from (and does not replace)
ForCES protocol. However, the two capability discovery mechanisms that used within the ForCES protocol. However, the two capability
may utilize the same FE model. discovery mechanisms may utilize the same FE model.
Yang, et. al. Expires Dec 2003 [Page 4]
FE Model - A model that describes the logical processing functions FE Model - A model that describes the logical processing functions
of a FE. of an FE.
ForCES Protocol Element - A FE or CE. ForCES Protocol Element - A FE or CE.
FE Topology -- Representation of how the multiple FEs in a single NE FE Topology -- Representation of how the multiple FEs in a single
are interconnected. Sometimes it is called inter-FE topology, to be NE are interconnected. Sometimes it is called inter-FE topology,
distinguished from intra-FE topology (or FE block topology) used by to be distinguished from intra-FE topology (or FE block topology)
FE model. used by FE model.
Inter-FE topology - see FE Topology. Inter-FE topology - see FE Topology.
2. Introduction to Forwarding and Control Element Separation (ForCES) 2. Introduction to Forwarding and Control Element Separation (ForCES)
An IP network element (NE) appears to external entities as a An IP network element (NE) appears to external entities as a
monolithic piece of network equipment, e.g., a router, NAT, monolithic piece of network equipment, e.g., a router, NAT,
firewall, or load balancer. Internally, however, an IP network firewall, or load balancer. Internally, however, an IP network
element (NE) (such as a router) is composed of numerous logically element (NE) (such as a router) is composed of numerous logically
separated entities that cooperate to provide a given functionality separated entities that cooperate to provide a given functionality
(such as routing). Two types of network element components exist: (such as routing). Two types of network element components exist:
control element (CE) in control plane and forwarding element (FE) in control element (CE) in control plane and forwarding element (FE)
forwarding plane (or data plane). Forwarding elements typically are in forwarding plane (or data plane). Forwarding elements typically
ASIC, network-processor, or general-purpose processor-based devices are ASIC, network-processor, or general-purpose processor-based
that handle data path operations for each packet. Control elements devices that handle data path operations for each packet. Control
are typically based on general-purpose processors that provide elements are typically based on general-purpose processors that
control functionality like routing and signaling protocols. provide control functionality like routing and signaling protocols.
ForCES aims to define a framework and associated protocol(s) to ForCES aims to define a framework and associated protocol(s) to
standardize information exchange between the control and forwarding standardize information exchange between the control and forwarding
Yang, et al. Expires December 2003 [Page 5]
plane. Having standard mechanisms allows CEs and FEs to become plane. Having standard mechanisms allows CEs and FEs to become
physically separated standard components. This physical separation physically separated standard components. This physical separation
accrues several benefits to the ForCES architecture. Separate accrues several benefits to the ForCES architecture. Separate
components would allow component vendors to specialize in one components would allow component vendors to specialize in one
component without having to become experts in all components. component without having to become experts in all components.
Standard protocol also allows the CEs and FEs from different Standard protocol also allows the CEs and FEs from different
component vendors to interoperate with each other and hence it component vendors to interoperate with each other and hence it
becomes possible for system vendors to integrate together the CEs becomes possible for system vendors to integrate together the CEs
and FEs from different component suppliers. This interoperability and FEs from different component suppliers. This interoperability
translates into a lot more design choices and flexibility to the translates into a lot more design choices and flexibility for the
system vendors. Overall, ForCES will enable rapid innovation in system vendors. Overall, ForCES will enable rapid innovation in
both the control and forwarding planes while maintaining both the control and forwarding planes while maintaining
interoperability. Scalability is also easily provided by this interoperability. Scalability is also easily provided by this
architecture in that additional forwarding or control capacity can architecture in that additional forwarding or control capacity can
be added to existing network elements without the need for forklift be added to existing network elements without the need for forklift
upgrades. upgrades.
------------------------- ------------------------- ------------------------- -------------------------
| Control Blade A | | Control Blade B | | Control Blade A | | Control Blade B |
| (CE) | | (CE) | | (CE) | | (CE) |
------------------------- ------------------------- ------------------------- -------------------------
^ | ^ | ^ | ^ |
| | | | | | | |
| V | V | V | V
Yang, et. al. Expires Dec 2003 [Page 5]
--------------------------------------------------------- ---------------------------------------------------------
| Switch Fabric Backplane | | Switch Fabric Backplane |
--------------------------------------------------------- ---------------------------------------------------------
^ | ^ | ^ | ^ | ^ | ^ |
| | | | | | | | | | . . . | |
| V | V | V | V | V | V
------------ ------------ ------------ ------------ ------------ ------------
|Router | |Router | |Router | |Router | |Router | |Router |
|Blade #1 | |Blade #2 | |Blade #N | |Blade #1 | |Blade #2 | |Blade #N |
| (FE) | | (FE) | | (FE) | | (FE) | | (FE) | | (FE) |
------------ ------------ ------------ ------------ ------------ ------------
^ | ^ | ^ | ^ | ^ | ^ |
| | | | | | | | | | . . . | |
| V | V | V | V | V | V
Figure 1. A router configuration example with separate blades. Figure 1. A router configuration example with separate blades.
One example of such physical separation is at the blade level. One example of such physical separation is at the blade level.
Figure 1 shows such an example configuration of a router, with two Figure 1 shows such an example configuration of a router, with two
control blades and multiple router (forwarding) blades, all control blades and multiple router (forwarding) blades, all
interconnected into a switch fabric backplane. In such chassis interconnected into a switch fabric backplane. In such a chassis
configuration, the control blades are the CEs while the router configuration, the control blades are the CEs while the router
blades are FEs, and the switch fabric backplane provides the blades are FEs, and the switch fabric backplane provides the
Yang, et al. Expires December 2003 [Page 6]
physical interconnect for all the blades. Control blade A may be physical interconnect for all the blades. Control blade A may be
the primary CE while control blade B is the backup CE providing the primary CE while control blade B may be the backup CE providing
redundancy. It is also possible to have a redundant switch fabric redundancy. It is also possible to have a redundant switch fabric
for high availability support. Routers today with this kind of for high availability support. Routers today with this kind of
configuration use proprietary interface for messaging between CEs configuration use proprietary interfaces for messaging between CEs
and FEs. The goal of ForCES is to replace such proprietary and FEs. The goal of ForCES is to replace such proprietary
interface with a standard protocol. With a standard protocol like interfaces with a standard protocol. With a standard protocol like
ForCES implemented on all blades, it becomes possible for control ForCES implemented on all blades, it becomes possible for control
blades from vendor X and routing blades from vendor Y to work blades from vendor X and routing blades from vendor Y to work
seamlessly together in one chassis. seamlessly together in one chassis.
------- ------- ------- -------
| CE1 | | CE2 | | CE1 | | CE2 |
------- ------- ------- -------
^ ^ ^ ^
| | | |
V V V V
============================================ Ethernet ============================================ Ethernet
^ ^ ^ ^ ^ . . . ^
| | | | | |
V V V V V V
------- ------- -------- ------- ------- --------
| FE#1| | FE#2| | FE#n | | FE#1| | FE#2| | FE#n |
------- ------- -------- ------- ------- --------
^ | ^ | ^ | ^ | ^ | ^ |
| | | | | | | | | | | |
| V | V | V | V | V | V
Figure 2. A router configuration example with separate boxes. Figure 2. A router configuration example with separate boxes.
Yang, et. al. Expires Dec 2003 [Page 6]
Another level of physical separation between the CEs and FEs can be Another level of physical separation between the CEs and FEs can be
at the box level. In such configuration, all the CEs and FEs are at the box level. In such configuration, all the CEs and FEs are
physically separated boxes, interconnected with some kind of high physically separated boxes, interconnected with some kind of high
speed LAN connection (like Gigabit Ethernet). These separated CEs speed LAN connection (like Gigabit Ethernet). These separated CEs
and FEs are only one hop away from each other within a local area and FEs are only one hop away from each other within a local area
network. The CEs and FEs communicate to each other by running network. The CEs and FEs communicate to each other by running
ForCES, and the collection of these CEs and FEs together become one ForCES, and the collection of these CEs and FEs together become one
routing unit to the external world. Figure 2 shows such an example. routing unit to the external world. Figure 2 shows such an example.
In both examples shown here, the same physical interconnect is used In both examples shown here, the same physical interconnect is used
for both CE-to-FE and FE-to-FE communication. However, that does for both CE-to-FE and FE-to-FE communication. However, that does
not have to be the case. One reason to use different interconnect not have to be the case. One reason to use different interconnects
is that CE-to-FE interconnect does not have to be as fast as the FE- is that CE-to-FE interconnect does not have to be as fast as the
to-FE interconnect, so the more expensive fast connections can be FE-to-FE interconnect, so the more expensive fast connections can
saved for FE-to-FE only. The separate interconnects may also be saved for FE-to-FE. The separate interconnects may also provide
provide reliability and redundancy benefits for the NE. reliability and redundancy benefits for the NE.
Some examples of control functions that can be implemented in the CE Yang, et al. Expires December 2003 [Page 7]
include routing protocols like RIP, OSPF and BGP, control and Some examples of control functions that can be implemented in the
CE include routing protocols like RIP, OSPF and BGP, control and
signaling protocols like RSVP (Resource Reservation Protocol), LDP signaling protocols like RSVP (Resource Reservation Protocol), LDP
(Label Distribution Protocol) for MPLS, etc. Examples of forwarding (Label Distribution Protocol) for MPLS, etc. Examples of
functions in FE include LPM (longest prefix match) forwarder, forwarding functions in the FE include LPM (longest prefix match)
classifiers, traffic shaper, meter, NAT, etc. Figure 3 shows a forwarder, classifiers, traffic shaper, meter, NAT (Network Address
diagram with examples in both CE and FE. Any given NE may contain Translators), etc. Figure 3 provides example functions in both CE
one or many of these CE and FE functions in it. The diagram also and FE. Any given NE may contain one or many of these CE and FE
shows that ForCES protocol is used to transport both the control functions in it. The diagram also shows that ForCES protocol is
messages for ForCES itself and the data packets that are used to transport both the control messages for ForCES itself and
originated/destined from/to the control functions in CE (e.g., the data packets that are originated/destined from/to the control
routing packets). Section 4.2.4 provides more detail on this. functions in CE (e.g., routing packets). Section 4.2.4 provides
more detail on this.
------------------------------------------------- -------------------------------------------------
| | | | | | | | | | | | | |
|OSPF |RIP |BGP |RSVP |LDP | | |OSPF |RIP |BGP |RSVP |LDP |. . . |
| | | | | | | | | | | | | |
------------------------------------------------- -------------------------------------------------
| ForCES Interface | | ForCES Interface |
------------------------------------------------- -------------------------------------------------
^ ^ ^ ^
ForCES | |data ForCES | |data
control | |packets control | |packets
messages| |(e.g., routing packets) messages| |(e.g., routing packets)
v v v v
------------------------------------------------- -------------------------------------------------
| ForCES Interface | | ForCES Interface |
------------------------------------------------- -------------------------------------------------
| | | | | | | | | | | | | |
|LPM Fwd|Meter |Shaper |NAT |Classi-| | |LPM Fwd|Meter |Shaper |NAT |Classi-|. . . |
| | | | |fier | | | | | | |fier | |
------------------------------------------------- -------------------------------------------------
| FE resources | | FE resources |
------------------------------------------------- -------------------------------------------------
Yang, et. al. Expires Dec 2003 [Page 7]
Figure 3. Examples of CE and FE functions Figure 3. Examples of CE and FE functions
A set of requirements for control and forwarding separation is A set of requirements for control and forwarding separation is
identified in [3]. This document describes a ForCES architecture identified in [3]. This document describes a ForCES architecture
that satisfies the architectural requirements of that document and that satisfies the architectural requirements of that document and
defines a framework for ForCES network elements and the associated defines a framework for ForCES network elements and the associated
entities to facilitate protocol definition. Whenever necessary, entities to facilitate protocol definition. Whenever necessary,
this document uses many examples to illustrate the issues and/or this document uses many examples to illustrate the issues and/or
possible solutions in ForCES. These examples are intended to be possible solutions in ForCES. These examples are intended to be
Yang, et al. Expires December 2003 [Page 8]
just examples, and should not be taken as the only or definite ways just examples, and should not be taken as the only or definite ways
of doing certain things. It is expected that separate document will of doing certain things. It is expected that separate document
be produced by the ForCES working group to specify the ForCES will be produced by the ForCES working group to specify the ForCES
protocol(s). protocol(s).
3. Architecture 3. Architecture
This section defines the ForCES architectural framework and the This section defines the ForCES architectural framework and the
associated logical components. This ForCES framework defines associated logical components. This ForCES framework defines
components of ForCES NEs including several ancillary components. components of ForCES NEs including several ancillary components.
These components may be connected in different kinds of topologies These components may be connected in different kinds of topologies
for flexible packet processing. for flexible packet processing.
skipping to change at line 417 skipping to change at line 425
| | | | | | | | | | | | | | | | | | | |
----+--+--+--+----------+--+--+--+----- ----+--+--+--+----------+--+--+--+-----
| | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | |
Fi/f Fi/f Fi/f Fi/f
Figure 4. ForCES Architectural Diagram Figure 4. ForCES Architectural Diagram
The diagram in Figure 4 shows the logical components of the ForCES The diagram in Figure 4 shows the logical components of the ForCES
architecture and their relationships. There are two kinds of architecture and their relationships. There are two kinds of
components inside a ForCES network element: control element (CE) and components inside a ForCES network element: control element (CE)
forwarding element (FE). The framework allows multiple instances of and forwarding element (FE). The framework allows multiple
CE and FE inside one NE. Each FE contains one or more physical instances of CE and FE inside one NE. Each FE contains one or
media interfaces for receiving and transmitting packets from/to the more physical media interfaces for receiving and transmitting
packets from/to the external world. The aggregation of these FE
interfaces becomes the NEs external interfaces. In addition to
the external interfaces, there must also exist some kind of
interconnect within the NE so that the CE and FE can communicate
Yang, et. al. Expires Dec 2003 [Page 8] Yang, et al. Expires December 2003 [Page 9]
external world. The aggregation of these FE interfaces becomes the with each other, and one FE can forward packets to another FE.
NEs external interfaces. In addition to the external interfaces, The diagram also shows two entities outside of the ForCES NE: CE
there must also exist some kind of interconnect within the NE so Manager and FE Manager. These two entities provide configuration
that the CE and FE can communicate with each other, and one FE can to the corresponding CE or FE in the pre-association phase (see
forward packets to another FE. The diagram also shows two entities Section 5.1). There is no defined role for FE Manager and CE
outside of the ForCES NE: CE Manager and FE Manager. These two Manager in post-association phase, thus these logical components
entities provide configuration to the corresponding CE or FE in the are not considered part of the ForCES NE.
pre-association phase (see Section 5.1). There is no defined role
for FE Manager and CE Manager in post-association phase, thus these
logical components are not considered part of the ForCES NE.
For convenience, the logical interactions between these components For convenience, the logical interactions between these components
are labeled by reference points Fp, Fc, Ff, Fr, Fl, and Fi, as shown are labeled by reference points Fp, Fc, Ff, Fr, Fl, and Fi, as
in Figure 4. The FE external interfaces are labeled as Fi/f. More shown in Figure 4. The FE external interfaces are labeled as Fi/f.
detail is provided in Section 4 and 5 for each of these reference More detail is provided in Section 4 and 5 for each of these
points. All these reference points are important in understanding reference points. All these reference points are important in
the ForCES architecture, however, the ForCES protocol is only understanding the ForCES architecture, however, the ForCES protocol
defined over one reference point -- Fp. is only defined over one reference point -- Fp.
The interface between two ForCES NEs is identical to the interface The interface between two ForCES NEs is identical to the interface
between two conventional routers and these two NEs exchange the between two conventional routers and these two NEs exchange the
protocol packets through the external interfaces at Fi/f. ForCES protocol packets through the external interfaces at Fi/f. ForCES
NEs connect to existing routers transparently. NEs connect to existing routers transparently.
3.1. Control Elements and Fr Reference Point 3.1. Control Elements and Fr Reference Point
It is not necessary to define any protocols across the Fr reference It is not necessary to define any protocols across the Fr reference
point to enable control and forwarding separation for simple point to enable control and forwarding separation for simple
skipping to change at line 472 skipping to change at line 481
serviced by any of the other CEs. For both redundancy and load serviced by any of the other CEs. For both redundancy and load
sharing, the CEs involved are equivalently capable. The only sharing, the CEs involved are equivalently capable. The only
difference between these two cases is in terms of how many active difference between these two cases is in terms of how many active
CEs there are. Distributed control is the case where two or more CEs there are. Distributed control is the case where two or more
CEs are concurrently active but certain requests can only be CEs are concurrently active but certain requests can only be
serviced by certain CEs. serviced by certain CEs.
When multiple CEs are employed in a ForCES NE, their internal When multiple CEs are employed in a ForCES NE, their internal
organization is considered an implementation issue that is beyond organization is considered an implementation issue that is beyond
the scope of ForCES. CEs are wholly responsible for coordinating the scope of ForCES. CEs are wholly responsible for coordinating
amongst themselves via the Fr reference point to provide consistency amongst themselves via the Fr reference point to provide
and synchronization. However, ForCES does not define the
implementation or protocols used between CEs, nor does it define how
to distribute functionality among CEs. Nevertheless, ForCES will
Yang, et. al. Expires Dec 2003 [Page 9] Yang, et al. Expires December 2003 [Page 10]
support mechanisms for CE redundancy or fail over, and it is consistency and synchronization. However, ForCES does not define
expected that vendors will provide redundancy or fail over solutions the implementation or protocols used between CEs, nor does it
within this framework. define how to distribute functionality among CEs. Nevertheless,
ForCES will support mechanisms for CE redundancy or fail over, and
it is expected that vendors will provide redundancy or fail over
solutions within this framework.
3.2. Forwarding Elements and Fi reference point 3.2. Forwarding Elements and Fi reference point
FE is a logical entity that implements the ForCES protocol and uses An FE is a logical entity that implements the ForCES protocol and
the underlying hardware to provide per-packet processing and uses the underlying hardware to provide per-packet processing and
handling as directed by a CE. It is possible to partition one handling as directed by a CE. It is possible to partition one
physical FE into multiple logical FEs. It is also possible for one physical FE into multiple logical FEs. It is also possible for one
FE to use multiple physical FEs. The mapping between physical FE(s) FE to use multiple physical FEs. The mapping between physical
and the logical FE(s) is beyond the scope of ForCES. For example, a FE(s) and the logical FE(s) is beyond the scope of ForCES. For
logical partition of a physical FE can be created by assigning some example, a logical partition of a physical FE can be created by
portion of each of the resources (e.g., ports, memory, forwarding assigning some portion of each of the resources (e.g., ports,
table entries) available on the physical FE to each of the logical memory, forwarding table entries) available on the physical FE to
FEs. Such concept of FE virtualization is analogous to a virtual each of the logical FEs. Such concept of FE virtualization is
switching element as described in [8]. FE virtualization should analogous to a virtual switching element as described in [8]. FE
occur only in the pre-association phase and hence has no impact on virtualization should occur only in the pre-association phase and
ForCES. hence has no impact on ForCES.
FEs perform all packet processing functions as directed by CEs. FEs perform all packet processing functions as directed by CEs.
FEs have no initiative of their own. Instead, FEs are slaves and FEs have no initiative of their own. Instead, FEs are slaves and
only do as they are told. FEs may communicate with one or more CEs only do as they are told. FEs may communicate with one or more CEs
concurrently across reference point Fp. FEs have no notion of CE concurrently across reference point Fp. FEs have no notion of CE
redundancy, load sharing, or distributed control. Instead, FEs redundancy, load sharing, or distributed control. Instead, FEs
accept commands from any CE authorized to control them, and it is up accept commands from any CE authorized to control them, and it is
to the CEs to coordinate among themselves to achieve redundancy, up to the CEs to coordinate among themselves to achieve redundancy,
load sharing or distributed control. The idea is to keep FEs as load sharing or distributed control. The idea is to keep FEs as
simple and dumb as possible so that FEs can focus its resource on simple and dumb as possible so that FEs can focus their resource on
the packet processing functions. the packet processing functions.
For example, in Figure 5, FE1 and FE2 can be configured to accept
commands from both the primary CE (CE1) and the backup CE (CE2).
Upon detection of CE1 failure, perhaps across the Fr or Fp
reference point, CE2 is configured to take over activities of CE1.
This is beyond the scope of ForCES and is not discussed further.
Distributed control can be achieved in the similar fashion, without
much intelligence on the part of FEs. For example, FEs can be
configured to detect RSVP and BGP protocol packets, and forward
RSVP packets to one CE and BGP packets to another CE. Hence, FEs
may need to do packet filtering for forwarding packets to specific
CEs.
Yang, et al. Expires December 2003 [Page 11]
------- Fr ------- ------- Fr -------
| CE1 | ------| CE2 | | CE1 | ------| CE2 |
------- ------- ------- -------
| \ / | | \ / |
| \ / | | \ / |
| \ / | | \ / |
| \/Fp | | \/Fp |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
------- Fi ------- ------- Fi -------
| FE1 |<----->| FE2 | | FE1 |<----->| FE2 |
------- ------- ------- -------
Figure 5. CE redundancy example. Figure 5. CE redundancy example.
For example, in Figure 5, FE1 and FE2 can be configured to accept This architecture permits multiple FEs to be present in an NE. [3]
commands from both the primary CE (CE1) and the backup CE (CE2).
Upon detection of CE1 failure, perhaps across the Fr or Fp reference
Yang, et. al. Expires Dec 2003 [Page 10]
point, CE2 is configured to take over activities of CE1. This is
beyond the scope of ForCES and is not discussed further.
Distributed control can be achieved in the similar fashion, without
much intelligence on the part of FEs. For example, FEs can be
configured to detect RSVP and BGP protocol packets, and forward RSVP
packets to one CE and BGP packets to another CE. Hence, FEs may
need to do packet filtering for forwarding packets to specific CEs.
This architecture permits multiple FEs to be present in a NE. [3]
dictates that the ForCES protocol must be able to scale to at least dictates that the ForCES protocol must be able to scale to at least
hundreds of FEs (see [3] Section 5, requirement #11). Each of these hundreds of FEs (see [3] Section 5, requirement #11). Each of
FEs may potentially have a different set of packet processing these FEs may potentially have a different set of packet processing
functions, with different media interfaces. FEs are responsible for functions, with different media interfaces. FEs are responsible
basic maintenance of layer-2 connectivity with other FEs and with for basic maintenance of layer-2 connectivity with other FEs and
external entities. Many layer-2 media include sophisticated control with external entities. Many layer-2 media include sophisticated
protocols. The FORCES protocol (over the Fp reference point) will control protocols. The FORCES protocol (over the Fp reference
be able to carry messages for such protocols so that, in keeping point) will be able to carry messages for such protocols so that,
with the "dumb FE model, the CE can provide appropriate in keeping with the dumb FE model, the CE can provide appropriate
intelligence and control over these media. intelligence and control over these media.
When multiple FEs are present, ForCES requires that packets must be When multiple FEs are present, ForCES requires that packets must be
able to arrive at the NE by one FE and leave the NE via a different able to arrive at the NE by one FE and leave the NE via a different
FE (See [3], Section 5, Requirement #3). Packets that enter the NE FE (See [3], Section 5, Requirement #3). Packets that enter the NE
via one FE and leave the NE via a different FE are transferred via one FE and leave the NE via a different FE are transferred
between FEs across the Fi reference point. Fi reference point could between FEs across the Fi reference point. Fi reference point
be used by FEs to discovery their (inter-FE) topology, perhaps could be used by FEs to discovery their (inter-FE) topology,
during pre-association phase. The Fi reference point is a separate perhaps during pre-association phase. The Fi reference point is a
protocol from the Fp reference point and is not currently defined by separate protocol from the Fp reference point and is not currently
the ForCES architecture. defined by the ForCES architecture.
FEs could be connected in different kinds of topologies and packet
processing may spread across several FEs in the topology. Hence,
logical packet flow may be different from physical FE topology.
Figure 6 provides some topology examples. When it is necessary to
forward packets between FEs, the CE needs to understand the FE
topology. The FE topology can be queried from the FEs by CEs.
Yang, et al. Expires December 2003 [Page 12]
----------------- -----------------
| CE | | CE |
----------------- -----------------
^ ^ ^ ^ ^ ^
/ | \ / | \
/ v \ / v \
/ ------- \ / ------- \
/ +->| FE3 |<-+ \ / +->| FE3 |<-+ \
/ | | | | \ / | | | | \
v | ------- | v v | ------- | v
------- | | ------- ------- | | -------
| FE1 |<-+ +->| FE2 | | FE1 |<-+ +->| FE2 |
| |<--------------->| | | |<--------------->| |
------- ------- ------- -------
^ | ^ | ^ | ^ |
| | | | | | | |
| v | v | v | v
(a) Full mesh among FE1, FE2 and FE3. (a) Full mesh among FE1, FE2 and FE3.
Yang, et. al. Expires Dec 2003 [Page 11]
----------- -----------
| CE | | CE |
----------- -----------
^ ^ ^ ^ ^ ^ ^ ^
/ | | \ / | | \
/------ | | ------\ /------ | | ------\
v v v v v v v v
------- ------- ------- ------- ------- ------- ------- -------
| FE1 |<->| FE2 |<->| FE3 |<->| FE4 | | FE1 |<->| FE2 |<->| FE3 |<->| FE4 |
------- ------- ------- ------- ------- ------- ------- -------
skipping to change at line 605 skipping to change at line 621
(b) Multiple FEs in a daisy chain (b) Multiple FEs in a daisy chain
^ | ^ |
| v | v
----------- -----------
| FE1 |<-----------------------| | FE1 |<-----------------------|
----------- | ----------- |
^ ^ | ^ ^ |
/ \ | / \ |
Yang, et al. Expires December 2003 [Page 13]
| ^ / \ ^ | V | ^ / \ ^ | V
v | v v | v ---------- v | v v | v ----------
--------- --------- | | --------- --------- | |
| FE2 | | FE3 |<------------>| CE | | FE2 | | FE3 |<------------>| CE |
--------- --------- | | --------- --------- | |
^ ^ ^ ---------- ^ ^ ^ ----------
| \ / ^ ^ | \ / ^ ^
| \ / | | | \ / | |
| v v | | | v v | |
| ----------- | | | ----------- | |
skipping to change at line 626 skipping to change at line 644
| ----------- | | ----------- |
| | ^ | | | ^ |
| v | | | v | |
| | | |
|----------------------------------------| |----------------------------------------|
(c) Multiple FEs connected by a ring (c) Multiple FEs connected by a ring
Figure 6. Some examples of FE topology. Figure 6. Some examples of FE topology.
FEs could be connected in different kinds of topologies and packet
processing may spread across several FEs in the topology. Hence,
logical packet flow may be different from physical FE topology.
Figure 6 provides some topology examples. When it is necessary to
forward packets between FEs, the CE needs to understand the FE
topology. The FE topology can be queried from the FEs by CEs.
Yang, et. al. Expires Dec 2003 [Page 12]
3.3. CE Managers 3.3. CE Managers
CE managers are responsible for determining which FEs a CE should CE managers are responsible for determining which FEs a CE should
control. It is legitimate for CE managers to be hard-coded with the control. It is legitimate for CE managers to be hard-coded with
knowledge of with which FEs its CEs should communicate. A CE the knowledge of with which FEs its CEs should communicate with. A
manager may also be physically embedded into a CE and be implemented CE manager may also be physically embedded into a CE and be
as a simple keypad or other direct configuration mechanism on the implemented as a simple keypad or other direct configuration
CE. Finally, CE managers may be physically and logically separate mechanism on the CE. Finally, CE managers may be physically and
entities that configure the CE with FE information via such logically separate entities that configure the CE with FE
mechanisms as COPS-PR [6] or SNMP [4]. information via such mechanisms as COPS-PR [6] or SNMP [4].
3.4. FE Managers 3.4. FE Managers
FE managers are responsible for determining to which CE any FE managers are responsible for determining with which CE any
particular FE should initially communicate. Like CE managers, no particular FE should initially communicate. Like CE managers, no
restrictions are placed on how a FE manager decides to which CE its restrictions are placed on how an FE manager decides with which CE
FEs should communicate, nor are restrictions placed on how FE its FEs should communicate, nor are restrictions placed on how FE
managers are implemented. Each FE should have one and only one FE managers are implemented. Each FE should have one and only one FE
manager, while different FEs may have the same or different FE manager, while different FEs may have the same or different FE
manager(s). Each manager can choose to exist and operate manager(s). Each manager can choose to exist and operate
independent of other manager. independently of other manager.
4. Operational Phases 4. Operational Phases
Both FEs and CEs require some configuration in place before they can Both FEs and CEs require some configuration in place before they
start information exchange and function as a coherent network can start information exchange and function as a coherent network
element. Two operational phases are identified in this framework --
Yang, et al. Expires December 2003 [Page 14]
element. Two operational phases are identified in this framework:
pre-association and post-association. pre-association and post-association.
4.1.Pre-association Phase 4.1.Pre-association Phase
Pre-association phase is the period of time during which an FE Pre-association phase is the period of time during which an FE
Manager and a CE Manager are determining which FE and CE should be Manager and a CE Manager are determining which FE and CE should be
part of the same network element. The protocols used during this part of the same network element. The protocols used during this
phase may include all or some of the message exchange over Fl, Ff phase may include all or some of the message exchange over Fl, Ff
and Fc reference points. However, all these may be optional and and Fc reference points. However, all these may be optional and
none of this is within the scope of ForCES protocol. none of this is within the scope of ForCES protocol.
skipping to change at line 683 skipping to change at line 695
CE managers and FE managers may communicate across the Fl reference CE managers and FE managers may communicate across the Fl reference
point in the pre-association phase in order to determine which CEs point in the pre-association phase in order to determine which CEs
and FEs should communicate with each other. Communication across and FEs should communicate with each other. Communication across
the Fl reference point is optional in this architecture. No the Fl reference point is optional in this architecture. No
requirements are placed on this reference point. requirements are placed on this reference point.
CE managers and FE managers may be operated by different entities. CE managers and FE managers may be operated by different entities.
The operator of the CE manager may not want to divulge, except to The operator of the CE manager may not want to divulge, except to
specified FE managers, any characteristics of the CEs it manages. specified FE managers, any characteristics of the CEs it manages.
Similarly, the operator of the FE manager may not want to divulge FE Similarly, the operator of the FE manager may not want to divulge
characteristics, except to authorized entities. As such, CE FE characteristics, except to authorized entities. As such, CE
managers and FE managers may need to authenticate one another. managers and FE managers may need to authenticate one another.
Subsequent communication between CE managers and FE managers may Subsequent communication between CE managers and FE managers may
Yang, et. al. Expires Dec 2003 [Page 13]
require other security functions such as privacy, non-repudiation, require other security functions such as privacy, non-repudiation,
freshness, and integrity. freshness, and integrity.
FE Manager FE CE Manager CE FE Manager FE CE Manager CE
| | | | | | | |
| | | | | | | |
|(security exchange) | | |(security exchange) | |
1|<------------------------------>| | 1|<------------------------------>| |
| | | | | | | |
|(a list of CEs and their attributes) | |(a list of CEs and their attributes) |
2|<-------------------------------| | 2|<-------------------------------| |
| | | | | | | |
|(a list of FEs and their attributes) | |(a list of FEs and their attributes) |
3|------------------------------->| | 3|------------------------------->| |
| | | | | | | |
| | | | | | | |
|<----------------Fl------------>| | |<----------------Fl------------>| |
Figure 7. An example of message exchange over Fl reference point Figure 7. An example of message exchange over Fl reference point
Yang, et al. Expires December 2003 [Page 15]
Once the necessary security functions have been performed, the CE Once the necessary security functions have been performed, the CE
and FE managers communicate to determine which CEs and FEs should and FE managers communicate to determine which CEs and FEs should
communicate with each other. At the very minimum, the CE and FE communicate with each other. At the very minimum, the CE and FE
managers need to learn of the existence of available FEs and CEs managers need to learn of the existence of available FEs and CEs
respectively. This discovery process may or may not entail one or respectively. This discovery process may entail one or both
both managers learning the capabilities of the discovered ForCES managers learning the capabilities of the discovered ForCES
protocol elements. Figure 7 shows an example of possible message protocol elements. Figure 7 shows an example of possible message
exchange between CE manager and FE manager over Fl reference point. exchange between CE manager and FE manager over Fl reference point.
4.1.2. Ff Reference Point 4.1.2. Ff Reference Point
The Ff reference point is used to inform forwarding elements of the The Ff reference point is used to inform forwarding elements of the
association decisions made by the FE manager in pre-association association decisions made by the FE manager in pre-association
phase. Only authorized entities may instruct a FE with respect to phase. Only authorized entities may instruct an FE with respect to
which CE should control it. Therefore, privacy, integrity, which CE should control it. Therefore, privacy, integrity,
freshness, and authentication are necessary between the FE manager freshness, and authentication are necessary between the FE manager
and FEs when the FE manager is remote to the FE. Once the and FEs when the FE manager is remote to the FE. Once the
appropriate security has been established, the FE manager instructs appropriate security has been established, the FE manager instructs
the FEs across this reference point to join a new NE or to the FEs across this reference point to join a new NE or to
disconnect from an existing NE. The FE Manager could also assign disconnect from an existing NE. The FE Manager could also assign
unique FE identifiers to the FEs using this reference point. The FE unique FE identifiers to the FEs using this reference point. The
identifiers are useful in post association phase to express FE FE identifiers are useful in post association phase to express FE
topology. Figure 8 shows example of message exchange over Ff topology. Figure 8 shows example of message exchange over Ff
reference point. reference point.
Note that the FE manager function may be co-located with the FE
(such as by manual keypad entry of the CE IP address), in which case
this reference point is reduced to a built-in function.
FE Manager FE CE Manager CE FE Manager FE CE Manager CE
| | | | | | | |
Yang, et. al. Expires Dec 2003 [Page 14]
| | | | | | | |
|(security exchange) |(security exchange) |(security exchange) |(security exchange)
1|<------------>|authentication 1|<----------->|authentication 1|<------------>|authentication 1|<-----------
| | | | >|authentication | | |
|
|(FE ID, attributes) |(CE ID, attributes) |(FE ID, attributes) |(CE ID, attributes)
2|<-------------|request 2|<------------|request 2|<-------------|request 2|<------------|request
| | | | | | | |
3|------------->|response 3|------------>|response 3|------------->|response 3|------------>|response
|(corresponding CE ID) |(corresponding FE ID) |(corresponding CE ID) |(corresponding FE ID)
| | | | | | | |
| | | | | | | |
|<-----Ff----->| |<-----Fc---->| |<-----Ff----->| |<-----Fc---->|
Figure 8. Examples of message exchange Figure 8. Examples of message exchange
over Ff and Fc reference points. over Ff and Fc reference points.
Yang, et al. Expires December 2003 [Page 16]
Note that the FE manager function may be co-located with the FE
(such as by manual keypad entry of the CE IP address), in which
case this reference point is reduced to a built-in function.
4.1.3. Fc Reference Point 4.1.3. Fc Reference Point
The Fc reference point is used to inform control elements of the The Fc reference point is used to inform control elements of the
association decisions made by CE managers in pre-association phase. association decisions made by CE managers in pre-association phase.
When the CE manager is remote, only authorized entities may instruct When the CE manager is remote, only authorized entities may
a CE to control certain FEs. Privacy, integrity, freshness and instruct a CE to control certain FEs. Privacy, integrity,
authentication are also required across this reference point in such freshness and authentication are also required across this
a configuration. Once appropriate security has been established, reference point in such a configuration. Once appropriate security
the CE manager instructs CEs as to which FEs they should control and has been established, the CE manager instructs CEs as to which FEs
how they should control them. Figure 7 shows example of message they should control and how they should control them. Figure 8
exchange over Fc reference point. shows example of message exchange over Fc reference point.
As with the FE manager and FEs, configurations are possible where As with the FE manager and FEs, configurations are possible where
the CE manager and CE are co-located and no protocol is used for the CE manager and CE are co-located and no protocol is used for
this function. this function.
4.2. Post-association Phase and Fp reference point 4.2. Post-association Phase and Fp reference point
Post-association phase is the period of time during which a FE and Post-association phase is the period of time during which an FE and
CE have been configured with information necessary to contact each CE have been configured with information necessary to contact each
other and includes both association establishment and steady-state other and includes both association establishment and steady-state
communication. The communication between CE and FE is performed communication. The communication between CE and FE is performed
across the Fp ("p" meaning protocol) reference point. ForCES across the Fp ("p" meaning protocol) reference point. ForCES
protocol is exclusively used for all communication across the Fp protocol is exclusively used for all communication across the Fp
reference point. reference point.
4.2.1. Proximity and Interconnect between CEs and FEs 4.2.1. Proximity and Interconnect between CEs and FEs
The ForCES Working Group has made a conscious decision that the The ForCES Working Group has made a conscious decision that the
first version of ForCES will not be designed to support first version of ForCES will not be designed to support
configurations where the CE and FE are located arbitrarily in the configurations where the CE and FE are located arbitrarily in the
network. In particular, ForCES is intended for "very close" CE/FE network. In particular, ForCES is intended for "very close" CE/FE
localities in IP networks, as defined by ForCES Applicability localities in IP networks, as defined by ForCES Applicability
Statement ([7]). Very Close localities consist of control and Statement ([7]). Very Close localities consist of control and
forwarding elements that either are components in the same physical forwarding elements that either are components in the same physical
box, or are separated at most by one local network hop. box, or are separated at most by one local network hop.
Yang, et. al. Expires Dec 2003 [Page 15]
CEs and FEs can be connected by a variety of interconnect CEs and FEs can be connected by a variety of interconnect
technologies, including Ethernet connections, backplanes, ATM (cell) technologies, including Ethernet connections, backplanes, ATM
fabrics, etc. ForCES should be able to support each of these (cell) fabrics, etc. ForCES should be able to support each of
interconnects (see [3] Section 5, requirement #1). ForCES will make these interconnects (see [3] Section 5, requirement #1). ForCES
use of an existing RFC2914 ([2]) compliant L4 protocol with adequate will make use of an existing RFC2914 ([2]) compliant L4 protocol
reliability, security and congestion control (e.g. TCP, SCTP) for
transport purposes. Yang, et al. Expires December 2003 [Page 17]
with adequate reliability, security and congestion control (e.g.
TCP, SCTP) for transport purposes.
4.2.2. Association Establishment 4.2.2. Association Establishment
FE CE FE CE
| | | |
|(Security exchange.) | |(Security exchange.) |
1|<--------------------->| 1|<--------------------->|
| | | |
|(Let me join the NE please.) |(Let me join the NE please.)
2|---------------------->| 2|---------------------->|
| | | |
|(What kind of FE are you? -- capability query) |(What kind of FE are you? -- capability query)
3|<----------------------| 3|<----------------------|
| | | |
|(Here is my FE functions/state: use model to describe) |(Here is my FE functions/state: use model to
describe)
4|---------------------->| 4|---------------------->|
| | | |
|(How are you connected with other FEs?) |(How are you connected with other FEs?)
5|<----------------------| 5|<----------------------|
| | | |
|(Here is the FE topology info) |(Here is the FE topology info)
6|---------------------->| 6|---------------------->|
| | | |
|(Initial config for FE -- optional) |(Initial config for FE -- optional)
7|<----------------------| 7|<----------------------|
skipping to change at line 846 skipping to change at line 859
Figure 9. Example of message exchange between CE and FE Figure 9. Example of message exchange between CE and FE
over Fp to establish NE association over Fp to establish NE association
As an example, figure 9 shows some of the message exchange that may As an example, figure 9 shows some of the message exchange that may
happen before the association between the CE and FE is fully happen before the association between the CE and FE is fully
established. Either the CE or FE can initiate the connection. established. Either the CE or FE can initiate the connection.
Security handshake is necessary to authenticate the two Security handshake is necessary to authenticate the two
communication endpoints to each other before any further message communication endpoints to each other before any further message
exchange can happen. The exact details of the security handshake exchange can happen. The exact details of the security handshake
depend on the security solution chosen by ForCES protocol. It is depend on the security solution chosen by ForCES protocol. It is
most likely that either IPSec or TLS will be used. Section 9
Yang, et. al. Expires Dec 2003 [Page 16] Yang, et al. Expires December 2003 [Page 18]
most likely that either IPSec or TLS will be used. Section 9
provides more details on the security considerations for ForCES. provides more details on the security considerations for ForCES.
After the successful security handshake, the FE needs to inform the After the successful security handshake, the FE needs to inform the
CE of its own capability and its topology in relation to other FEs. CE of its own capability and its topology in relation to other FEs.
The capability of the FE is represented by the FE model, described The capability of the FE is represented by the FE model, described
in a separate document. The model would allow a FE to describe what in a separate document. The model would allow an FE to describe
kind of packet processing functions it contains, in what order the what kind of packet processing functions it contains, in what order
processing happens, what kinds of configurable parameters it allows, the processing happens, what kinds of configurable parameters it
what statistics it collects and what events it might throw, etc. allows, what statistics it collects and what events it might throw,
Once such information is available to the CE, the CE may choose to etc. Once such information is available to the CE, the CE may
send some initial or default configuration to the FE so that the FE choose to send some initial or default configuration to the FE so
can start receiving and processing packets correctly. Such that the FE can start receiving and processing packets correctly.
initialization may not be necessary if the FE already obtains the Such initialization may not be necessary if the FE already obtains
information from its own bootstrap process. Once FE starts the information from its own bootstrap process. Once FE starts
accepting packets for processing, we say the association of this FE accepting packets for processing, we say the association of this FE
with its CE is now established. From then on, the CE and FE enter with its CE is now established. From then on, the CE and FE enter
steady-state communication. steady-state communication.
4.2.3. Steady-state Communication 4.2.3. Steady-state Communication
Once an association is established between the CE and FE, the ForCES Once an association is established between the CE and FE, the
protocol is used by the CE and FE over Fp reference point to ForCES protocol is used by the CE and FE over Fp reference point to
exchange information to facilitate packet processing. exchange information to facilitate packet processing.
FE CE FE CE
| | | |
|(Add these new routes.)| |(Add these new routes.)|
1|<----------------------| 1|<----------------------|
| | | |
|(Successful.) | |(Successful.) |
2|---------------------->| 2|---------------------->|
| | | |
skipping to change at line 894 skipping to change at line 907
|(Reply with stats collected.) |(Reply with stats collected.)
2|---------------------->| 2|---------------------->|
| | | |
| | | |
|(My port is down, with port #.) |(My port is down, with port #.)
1|---------------------->| 1|---------------------->|
| | | |
|(Here is a new forwarding table) |(Here is a new forwarding table)
2|<----------------------| 2|<----------------------|
| | | |
| |
Yang, et al. Expires December 2003 [Page 19]
Figure 10. Examples of message exchange between CE and FE Figure 10. Examples of message exchange between CE and FE
over Fp during steady-state communication over Fp during steady-state communication
Yang, et. al. Expires Dec 2003 [Page 17]
Based on the information acquired through CEs' control processing, Based on the information acquired through CEs' control processing,
CEs will frequently need to manipulate the packet-forwarding CEs will frequently need to manipulate the packet-forwarding
behaviors of their FE(s) by sending instructions to FEs. For behaviors of their FE(s) by sending instructions to FEs. For
example, Figure 10 shows message exchange examples in which the CE example, Figure 10 shows message exchange examples in which the CE
sends new routes to the FE so that the FE can add them to its sends new routes to the FE so that the FE can add them to its
forwarding table. The CE may query the FE for statistics collected forwarding table. The CE may query the FE for statistics collected
by the FE and the FE may notify the CE of important events such as by the FE and the FE may notify the CE of important events such as
port failure. port failure.
4.2.4. Data Packets across Fp reference point 4.2.4. Data Packets across Fp reference point
Control plane protocol packets (such as RIP, OSPF messages)
addressed to any of NE's interfaces are typically redirected by the
receiving FE to its CE, and CE may originate packets and have its FE
deliver them to other NEs. Therefore, ForCES protocol over Fp not
only transports the ForCES protocol messages between CEs and FEs,
but also encapsulates the data packets from control plane protocols.
Moreover, one FE may be controlled by multiple CEs for distributed
control. In this configuration, the control protocols supported by
the FORCES NEs may spread across multiple CEs. For example, one CE
may support routing protocols like OSPF and BGP, while a signaling
and admission control protocol like RSVP is supported in another CE.
FEs are configured to recognize and filter these protocol packets
and forward them to the corresponding CE.
--------------------- ---------------------- --------------------- ----------------------
| | | | | | | |
| +--------+ | | +--------+ | | +--------+ | | +--------+ |
| |CE(BGP) | | | |CE(BGP) | | | |CE(BGP) | | | |CE(BGP) | |
| +--------+ | | +--------+ | | +--------+ | | +--------+ |
| | | | ^ | | | | | ^ |
| |Fp | | |Fp | | |Fp | | |Fp |
| v | | | | | v | | | |
| +--------+ | | +--------+ | | +--------+ | | +--------+ |
| | FE | | | | FE | | | | FE | | | | FE | |
skipping to change at line 947 skipping to change at line 945
| Router | | | Router | | | Router | | | Router | |
| A | | | B | | | A | | | B | |
---------+----------- -----------+---------- ---------+----------- -----------+----------
v ^ v ^
| | | |
| | | |
------------------->--------------- ------------------->---------------
Figure 11. Example to show data packet flow between two NEs. Figure 11. Example to show data packet flow between two NEs.
Control plane protocol packets (such as RIP, OSPF messages)
addressed to any of NE's interfaces are typically redirected by the
receiving FE to its CE, and CE may originate packets and have its
FE deliver them to other NEs. Therefore, ForCES protocol over Fp
not only transports the ForCES protocol messages between CEs and
FEs, but also encapsulates the data packets from control plane
protocols. Moreover, one FE may be controlled by multiple CEs for
distributed control. In this configuration, the control protocols
supported by the FORCES NEs may spread across multiple CEs. For
example, one CE may support routing protocols like OSPF and BGP,
while a signaling and admission control protocol like RSVP is
Yang, et al. Expires December 2003 [Page 20]
supported in another CE. FEs are configured to recognize and
filter these protocol packets and forward them to the corresponding
CE.
Figure 11 shows one example of how the BGP packets originated by Figure 11 shows one example of how the BGP packets originated by
router A are passed to router B. In this example, the ForCES router A are passed to router B. In this example, the ForCES
protocol is used to transport the packets from the CE to the FE protocol is used to transport the packets from the CE to the FE
inside router A, and then from the FE to the CE inside router B. In inside router A, and then from the FE to the CE inside router B.
light of the fact that the ForCES protocol is responsible for In light of the fact that the ForCES protocol is responsible for
transporting both the control messages and the data packets between transporting both the control messages and the data packets between
Yang, et. al. Expires Dec 2003 [Page 18]
the CE and FE over Fp reference point, it is possible to use either the CE and FE over Fp reference point, it is possible to use either
a single protocol or multiple protocols to achieve this. a single protocol or multiple protocols to achieve this.
4.2.5. Proxy FE 4.2.5. Proxy FE
In the case where a physical FE cannot implement (e.g., due to the In the case where a physical FE cannot implement (e.g., due to the
lack of a general purpose CPU) the ForCES protocol directly, a proxy lack of a general purpose CPU) the ForCES protocol directly, a
FE can be used in the middle of Fp reference point. This allows the proxy FE can be used in the middle of Fp reference point. This
CE communicate to the physical FE via the proxy by using ForCES, allows the CE communicate to the physical FE via the proxy by using
while the proxy manipulates the physical FE using some intermediary ForCES, while the proxy manipulates the physical FE using some
form of communication (e.g., a non-ForCES protocol or DMA). In such intermediary form of communication (e.g., a non-ForCES protocol or
an implementation, the combination of the proxy and the physical FE DMA). In such an implementation, the combination of the proxy and
becomes one logical FE entity. the physical FE becomes one logical FE entity.
4.3. Association Re-establishment 4.3. Association Re-establishment
FEs and CEs may join and leave NEs dynamically (see [3] Section 5, FEs and CEs may join and leave NEs dynamically (see [3] Section 5,
requirements #12). When a FE or CE leaves the NE, the association requirements #12). When an FE or CE leaves the NE, the association
with the NE is broken. If the leaving party rejoins a NE later, to with the NE is broken. If the leaving party rejoins an NE later,
re-establish the association, it may or may not need to re-enter the to re-establish the association, it may need to re-enter the pre-
pre-association phase. Loss of association can also happen association phase. Loss of association can also happen
unexpectedly due to loss of connection between the CE and the FE. unexpectedly due to loss of connection between the CE and the FE.
Therefore, the framework allows the bi-directional transition Therefore, the framework allows the bi-directional transition
between these two phases, but the ForCES protocol is only applicable between these two phases, but the ForCES protocol is only
for the post-association phase. However, the protocol should applicable for the post-association phase. However, the protocol
provide mechanisms to support association re-establishment. This should provide mechanisms to support association re-establishment.
includes the ability for CEs and FEs to determine when there is a This includes the ability for CEs and FEs to determine when there
loss of association between them, ability to restore association and is a loss of association between them, ability to restore
efficient state (re)synchronization mechanisms (see [3] Section 5, association and efficient state (re)synchronization mechanisms (see
requirement #7). Note that security association and state must be [3] Section 5, requirement #7). Note that security association and
also re-established to guarantee the same level of security exists state must be also re-established to guarantee the same level of
before and after the association re-establishment. security exists before and after the association re-establishment.
4.3.1. CE graceful restart 4.3.1. CE graceful restart
The failure and restart of the CE in a router can potentially cause The failure and restart of the CE in a router can potentially cause
much stress and disruption on the control plane throughout a much stress and disruption on the control plane throughout a
Yang, et al. Expires December 2003 [Page 21]
network. Because when a CE has to restart for any reason, the network. Because when a CE has to restart for any reason, the
router loses routing adjacencies or sessions with its routing router loses routing adjacencies or sessions with its routing
neighbors. Neighbors who detect the lost adjacency normally re- neighbors. Neighbors who detect the lost adjacency normally re-
compute new routes and then send routing updates to their own compute new routes and then send routing updates to their own
neighbors to communicate the lost adjacency. Their neighbors do the neighbors to communicate the lost adjacency. Their neighbors do
same thing to propagate throughout the network. In the meantime, the same thing to propagate throughout the network. In the
the restarting router cannot receive traffic from other routers meantime, the restarting router cannot receive traffic from other
because the neighbors have stopped using the routers previously routers because the neighbors have stopped using the routers
advertised routes. When the restarting router restores adjacencies, previously advertised routes. When the restarting router restores
neighbors must once again re-compute new routes and send out adjacencies, neighbors must once again re-compute new routes and
additional routing updates. The restarting router is unable to send out additional routing updates. The restarting router is
forward packets until it has re-established routing adjacencies with unable to forward packets until it has re-established routing
neighbors, received route updates through these adjacencies, and adjacencies with neighbors, received route updates through these
computed new routes. Until convergence takes place throughout the adjacencies, and computed new routes. Until convergence takes
network, packets may be lost in transient black holes or forwarding place throughout the network, packets may be lost in transient
loops. black holes or forwarding loops.
Yang, et. al. Expires Dec 2003 [Page 19]
A high availability mechanism known as the "graceful restart" has A high availability mechanism known as the "graceful restart" has
been used by the IP routing protocols (OSPF [10], BGP [11], BGP been used by the IP routing protocols (OSPF [10], BGP [11], BGP
[11]) and MPLS label distribution protocol (LDP [9]) to help [11]) and MPLS label distribution protocol (LDP [9]) to help
minimize the negative effects on routing throughout an entire minimize the negative effects on routing throughout an entire
network caused by a restarting router. Route flap on neighboring network caused by a restarting router. Route flap on neighboring
routers is avoided, and a restarting router can continue to forward routers is avoided, and a restarting router can continue to forward
packets that would otherwise be dropped. packets that would otherwise be dropped.
While the details differ from protocol to protocol, the general idea While the details differ from protocol to protocol, the general
behind the graceful restart mechanism remains the same. With the idea behind the graceful restart mechanism remains the same. With
graceful restart, a restarting router can inform its neighbors when the graceful restart, a restarting router can inform its neighbors
it restarts. The neighbors may detect the lost adjacency but do not when it restarts. The neighbors may detect the lost adjacency but
recompute new routes or send routing updates to their neighbors. do not recompute new routes or send routing updates to their
The neighbors also hold on to the routes received from the neighbors. The neighbors also hold on to the routes received from
restarting router before restart and assume they are still valid for the restarting router before restart and assume they are still
a limited time. By doing so, the restarting routers FEs can also valid for a limited time. By doing so, the restarting routers FEs
continue to receive and forward traffic from other neighbors for a can also continue to receive and forward traffic from other
limited time by using the routes they already have. The restarting neighbors for a limited time by using the routes they already have.
router then re-establishes routing adjacencies, downloads updated The restarting router then re-establishes routing adjacencies,
routes from all its neighbors, recomputes new routes and uses them downloads updated routes from all its neighbors, recomputes new
to replace the older routes it was using. It then sends these routes and uses them to replace the older routes it was using. It
updated routes to its neighbors and signals the completion of the then sends these updated routes to its neighbors and signals the
graceful restart process. completion of the graceful restart process.
Non-stop forwarding is a requirement for graceful restart. It is Non-stop forwarding is a requirement for graceful restart. It is
necessary so a router can continue to forward packets while it is necessary so a router can continue to forward packets while it is
downloading routing information and recomputing new routes. This downloading routing information and recomputing new routes. This
ensures that packets will not be dropped. As one can see, one of ensures that packets will not be dropped. As one can see, one of
the benefits afforded by the separation of CE and FE is exactly the the benefits afforded by the separation of CE and FE is exactly the
ability of non-stop forwarding in the face of the CE failure and ability of non-stop forwarding in the face of the CE failure and
Yang, et al. Expires December 2003 [Page 22]
restart. The support of dynamic changes to CE/FE association in restart. The support of dynamic changes to CE/FE association in
ForCES also makes it compatible with high availability mechanisms ForCES also makes it compatible with high availability mechanisms
such as graceful restart. such as graceful restart.
ForCES should be able to support CE graceful restart easily. When ForCES should be able to support CE graceful restart easily. When
the association is established the first time, the CE must inform the association is established the first time, the CE must inform
the FEs what to do in the case of CE failure. If graceful restart the FEs what to do in the case of CE failure. If graceful restart
is not supported, the FEs may be told to stop packet processing all is not supported, the FEs may be told to stop packet processing all
together if its CE fails. If graceful restart is supported, the FEs together if its CE fails. If graceful restart is supported, the
should be told to cache and hold on to its FE state including the FEs should be told to cache and hold on to its FE state including
forwarding tables across the restarts. A timer must be included so the forwarding tables across the restarts. A timer must be
that the timeout causes such cached state to expire eventually. included so that the timeout causes such cached state to expire
Those timers should be settable by the CE. eventually. Those timers should be settable by the CE.
4.3.2. FE restart 4.3.2. FE restart
In the same example in Figure 5, assuming CE1 is the working CE for In the same example in Figure 5, assuming CE1 is the working CE for
the moment, what would happen if one of the FEs, say FE1, leaves the the moment, what would happen if one of the FEs, say FE1, leaves
NE temporarily? FE1 may voluntarily decide to leave the the NE temporarily? FE1 may voluntarily decide to leave the
association. Alternatively, FE1 may stop functioning simply due to association. Alternatively, FE1 may stop functioning simply due to
unexpected failure. In former case, CE1 receives a "leave- unexpected failure. In former case, CE1 receives a "leave-
Yang, et. al. Expires Dec 2003 [Page 20]
association request" from FE1. In the latter, CE1 detects the association request" from FE1. In the latter, CE1 detects the
failure of FE1 by some other means. In both cases, CE1 must inform failure of FE1 by some other means. In both cases, CE1 must inform
the routing protocols of such an event, most likely prompting a the routing protocols of such an event, most likely prompting a
reachability and SPF (Shortest Path First) recalculation and reachability and SPF (Shortest Path First) recalculation and
associated downloading of new FIBs from CE1 to the other remaining associated downloading of new FIBs from CE1 to the other remaining
FEs (only FE2 in this example). Such recalculation and FIB update FEs (only FE2 in this example). Such recalculation and FIB update
will also be propagated from the CE1 to its neighbors that are will also be propagated from the CE1 to its neighbors that are
affected by the connectivity of FE1. affected by the connectivity of FE1.
When FE1 decides to rejoin again, or when it restarts again from the When FE1 decides to rejoin again, or when it restarts again from
failure, FE1 needs to re-discover its master (CE). This can be the failure, FE1 needs to re-discover its master (CE). This can be
achieved by several means. It may re-enter the pre-association achieved by several means. It may re-enter the pre-association
phase and get that information from its FE manager. It may retrieve phase and get that information from its FE manager. It may
the previous CE information from its cache, if it can validate the retrieve the previous CE information from its cache, if it can
information freshness. Once it discovers its CE, it starts message validate the information freshness. Once it discovers its CE, it
exchange with CE to re-establish the association just as outlined in starts message exchange with CE to re-establish the association
Figure 9, with the possible exception that it might be able to just as outlined in Figure 9, with the possible exception that it
bypass the transport of the complete initial configuration. Suppose might be able to bypass the transport of the complete initial
that FE1 still has its routing table and other state information configuration. Suppose that FE1 still has its routing table and
from the last association, instead of sending all the information other state information from the last association, instead of
again from scratch, it may be able to use more efficient mechanism sending all the information again from scratch, it may be able to
to re-sync up the state with its CE if such mechanism is supported use more efficient mechanism to re-sync up the state with its CE if
by the ForCES protocol. For example, CRC-32 of the state might give such mechanism is supported by the ForCES protocol. For example,
a quick indication of whether or not the state is in-sync with its CRC-32 of the state might give a quick indication of whether or not
CE. By comparing its state with CE first, it sends information the state is in-sync with its CE. By comparing its state with CE
update only if it is needed. ForCES protocol may choose to first, it sends information update only if it is needed. ForCES
implement similar optimization mechanisms, but it may also choose
not to, as this is not a requirement. Yang, et al. Expires December 2003 [Page 23]
protocol may choose to implement similar optimization mechanisms,
but it may also choose not to, as this is not a requirement.
5. Applicability to RFC1812 5. Applicability to RFC1812
[3] Section 5, requirement #9 dictates that "All proposed ForCES [3] Section 5, requirement #9 dictates "Any proposed ForCES
architecture must explain how that architecture supports all of the architecture must explain how that architecture supports all of the
router functions as defined in RFC1812." RFC1812 discusses many router functions as defined in RFC1812." RFC1812 discusses many
important requirements for IPv4 routers from the link layer to the important requirements for IPv4 routers from the link layer to the
application layer. This section addresses the relevant requirements application layer. This section addresses the relevant
in RFC1812 for implementing IPv4 routers based on ForCES requirements in RFC1812 for implementing IPv4 routers based on
architecture and explains how ForCES satisfies these requirements by ForCES architecture and explains how ForCES satisfies these
providing guidelines on how to separate the functionalities required requirements by providing guidelines on how to separate the
into forwarding plane and control plane. functionalities required into forwarding plane and control plane.
In general, the forwarding plane carries out the bulk of the per- In general, the forwarding plane carries out the bulk of the per-
packet processing that is required at line speed, while the control packet processing that is required at line speed, while the control
plane carries most of the computationally complex operations that plane carries most of the computationally complex operations that
are typical of the control and signaling protocols. However, it is are typical of the control and signaling protocols. However, it is
impossible to draw a rigid line to divide the processing into CEs impossible to draw a rigid line to divide the processing into CEs
and FEs cleanly. Nor should the ForCES architecture limit the and FEs cleanly. Nor should the ForCES architecture limit the
innovative approaches in control and forwarding plane separation. innovative approaches in control and forwarding plane separation.
As more and more processing power is available in the FEs, some of As more and more processing power is available in the FEs, some of
the control functions that traditionally are performed by CEs may the control functions that traditionally are performed by CEs may
now be moved to FEs for better performance and scalability. Such now be moved to FEs for better performance and scalability. Such
offloaded functions may include part of ICMP or TCP processing, or offloaded functions may include part of ICMP or TCP processing, or
Yang, et. al. Expires Dec 2003 [Page 21]
part of routing protocols. Once off-loaded onto the forwarding part of routing protocols. Once off-loaded onto the forwarding
plane, such CE functions, even though logically belonging to the plane, such CE functions, even though logically belonging to the
control plane, now become part of the FE functions. Just like the control plane, now become part of the FE functions. Just like the
other logical functions performed by FEs, such off-loaded functions other logical functions performed by FEs, such off-loaded functions
must be expressed as part of the FE model so that the CEs can decide must be expressed as part of the FE model so that the CEs can
how to best take advantage of these off-loaded functions when decide how to best take advantage of these off-loaded functions
present on the FEs. when present on the FEs.
5.1. General Router Requirements 5.1. General Router Requirements
Routers have at least two or more logical interfaces. When CEs and Routers have at least two or more logical interfaces. When CEs and
FEs are separated by ForCES within a single NE, some additional FEs are separated by ForCES within a single NE, some additional
interfaces are needed for intra-NE communications. Figure 12 shows interfaces are needed for intra-NE communications. Figure 12 shows
an example to illustrate that. This NE contains one CE and two FEs. an example to illustrate that. This NE contains one CE and two
Each FE has four interfaces; two of them are used for receiving and FEs. Each FE has four interfaces; two of them are used for
transmitting packets to the external world, while the other two are receiving and transmitting packets to the external world, while the
for intra-NE connections. CE has two logical interfaces #9 and #10, other two are for intra-NE connections. CE has two logical
connected to interfaces #3 and #6 from FE1 and FE2, respectively. interfaces #9 and #10, connected to interfaces #3 and #6 from FE1
Interface #4 and #5 are connected for FE1-FE2 communication. and FE2, respectively. Interface #4 and #5 are connected for FE1-
Therefore, this router NE provides four external interfaces (#1, 2,
7 and 8). Yang, et al. Expires December 2003 [Page 24]
FE2 communication. Therefore, this router NE provides four
external interfaces (#1, 2, 7 and 8).
--------------------------------- ---------------------------------
| router NE | | router NE |
| ----------- ----------- | | ----------- ----------- |
| | FE1 | | FE2 | | | | FE1 | | FE2 | |
| ----------- ----------- | | ----------- ----------- |
| 1| 2| 3| 4| 5| 6| 7| 8| | | 1| 2| 3| 4| 5| 6| 7| 8| |
| | | | | | | | | | | | | | | | | | | |
| | | | +----+ | | | | | | | | +----+ | | | |
| | | | | | | | | | | | | | | |
skipping to change at line 1169 skipping to change at line 1185
IPv4 routers must implement IP to support its packet forwarding IPv4 routers must implement IP to support its packet forwarding
function, which is driven by its FIB (Forwarding Information Base). function, which is driven by its FIB (Forwarding Information Base).
This Internet layer forwarding (see RFC1812 [1] Section 5) This Internet layer forwarding (see RFC1812 [1] Section 5)
functionality naturally belongs to FEs in the ForCES architecture. functionality naturally belongs to FEs in the ForCES architecture.
A router may implement transport layer protocols (like TCP and UDP) A router may implement transport layer protocols (like TCP and UDP)
that are required to support application layer protocols (see that are required to support application layer protocols (see
RFC1812 [1] Section 6). One important class of application RFC1812 [1] Section 6). One important class of application
protocols is routing protocols (see RFC1812 [1] Section 7). In protocols is routing protocols (see RFC1812 [1] Section 7). In
ForCES architecture, routing protocols are naturally implemented by ForCES architecture, routing protocols are naturally implemented by
CEs. Routing protocols require routers communicate with each other. CEs. Routing protocols require routers communicate with each
other. This communication between CEs in different routers is
Yang, et. al. Expires Dec 2003 [Page 22] supported in ForCES by FEs ability to redirect data packets
This communication between CEs in different routers is supported in addressed to routers (i.e., NEs) and CEs ability to originate
ForCES by FEs' ability to redirect data packets addressed to routers packets and have them delivered by their FEs. This communication
(i.e., NEs) and CEs' ability to originate packets and have them occurs across Fp reference point inside each router and between
delivered by their FEs. This communication occurs across Fp neighboring routers' external interfaces, as illustrated in Figure
reference point inside each router and between neighboring routers' 11.
external interfaces, as illustrated in Figure 11.
5.2.Link Layer 5.2.Link Layer
Since FEs own all the external interfaces for the router, FEs need Since FEs own all the external interfaces for the router, FEs need
to conform to the link layer requirements in RFC1812. Arguably, ARP to conform to the link layer requirements in RFC1812. Arguably,
support may be implemented in either CEs or FEs. As we will see ARP support may be implemented in either CEs or FEs. As we will
later, a number of behaviors that RFC1812 mandates fall into this
category -- they may be performed by the FE and may be performed by Yang, et al. Expires December 2003 [Page 25]
the CE. A general guideline is needed to ensure interoperability see later, a number of behaviors that RFC1812 mandates fall into
between separated control and forwarding planes. The guideline we this category -- they may be performed by the FE and may be
offer here is that in general CEs are required to be capable of performed by the CE. A general guideline is needed to ensure
these kind of operations while FEs may or may not choose to interoperability between separated control and forwarding planes.
implement them. FE model should indicate its capabilities in this The guideline we offer here is that CEs MUST be capable of these
regard so that CEs can decide where these functions are implemented. kind of operations while FEs MAY choose to implement them. FE
model should indicate its capabilities in this regard so that CEs
can decide where these functions are implemented.
Interface parameters, including MTU, IP address, etc., must be Interface parameters, including MTU, IP address, etc., must be
configurable by CEs via ForCES. CEs must be able to determine configurable by CEs via ForCES. CEs must be able to determine
whether a physical interface in an FE is available to send packets whether a physical interface in an FE is available to send packets
or not. FEs must also inform CEs the status change of the or not. FEs must also inform CEs the status change of the
interfaces (like link up/down) via ForCES. interfaces (like link up/down) via ForCES.
5.3.Internet Layer Protocols 5.3.Internet Layer Protocols
Both FEs and CEs must implement IP protocol and all mandatory Both FEs and CEs must implement IP protocol and all mandatory
extensions as RFC1812 specified. CEs should implement IP options extensions as RFC1812 specified. CEs should implement IP options
like source route and record route while FEs may choose to implement like source route and record route while FEs may choose to
those as well. Timestamp option should be implemented by FEs to implement those as well. The timestamp option should be
insert the timestamp most accurately. FE must interpret the IP implemented by FEs to insert the timestamp most accurately. The FE
options that it understands and preserve the rest unchanged for use must interpret the IP options that it understands and preserve the
by CEs. Both FEs and CEs might choose to silently discard packets rest unchanged for use by CEs. Both FEs and CEs might choose to
without sending ICMP errors, but such events should be logged and silently discard packets without sending ICMP errors, but such
counted. FEs may report statistics for such events to CEs via events should be logged and counted. FEs may report statistics for
ForCES. such events to CEs via ForCES.
When multiple FEs are involved to process packets, the appearance of When multiple FEs are involved to process packets, the appearance
single NE must be strictly maintained. For example, Time-To-Live of single NE must be strictly maintained. For example, Time-To-
(TTL) must be decremented only once within a single NE. For Live (TTL) must be decremented only once within a single NE. For
example, it can be always decremented by the last FE with egress example, it can be always decremented by the last FE with egress
function. function.
FEs must receive and process normally any packets with a broadcast FEs must receive and process normally any packets with a broadcast
destination address or a multicast destination address that the destination address or a multicast destination address that the
router has asked to receive. When IP multicast is supported in router has asked to receive. When IP multicast is supported in
routers, IGMP is implemented in CEs. CEs are also required of ICMP routers, IGMP is implemented in CEs. CEs are also required of ICMP
support, while it is optional for FEs to support ICMP. Such an support, while it is optional for FEs to support ICMP. Such an
option can be communicated to CEs as part of the FE model. option can be communicated to CEs as part of the FE model.
Therefore, FEs can always rely upon CEs to send out ICMP error Therefore, FEs can always rely upon CEs to send out ICMP error
Yang, et. al. Expires Dec 2003 [Page 23]
messages, but FEs also have the option to generate ICMP error messages, but FEs also have the option to generate ICMP error
messages themselves. messages themselves.
5.4.Internet Layer Forwarding 5.4.Internet Layer Forwarding
Yang, et al. Expires December 2003 [Page 26]
IP forwarding is implemented by FEs. When the routing table is IP forwarding is implemented by FEs. When the routing table is
updated at CEs, ForCES is used to send the new route entries from updated at CEs, ForCES is used to send the new route entries from
CEs to FEs. Each FE has its own forwarding table and uses this CEs to FEs. Each FE has its own forwarding table and uses this
table to direct packets to the next hop interface. table to direct packets to the next hop interface.
Upon receiving IP packets, FE verifies the IP header and processes Upon receiving IP packets, the FE verifies the IP header and
most of the IP options. Some options cannot be processed until the processes most of the IP options. Some options cannot be processed
routing decision has been made. Routing decision is made after until the routing decision has been made. The routing decision is
examining the destination IP address. If the destination address made after examining the destination IP address. If the
belongs to the router itself, the packets are filtered and either destination address belongs to the router itself, the packets are
processed locally or forwarded to CE, depending upon the filtered and either processed locally or forwarded to CE, depending
instructions set-up by CE. Otherwise, FE determines the next hop IP upon the instructions set-up by CE. Otherwise, the FE determines
address by looking up in its forwarding table. FE also determines the next hop IP address by looking up in its forwarding table. The
the network interface it uses to send the packets. Sometimes FE may FE also determines the network interface it uses to send the
need to forward the packets to another FE before packets can be packets. Sometimes an FE may need to forward the packets to
forwarded out to the next hop. Right before packets are forwarded another FE before packets can be forwarded out to the next hop.
out to the next hop, FE decrements TTL by 1 and processes any IP Right before packets are forwarded out to the next hop, the FE
options that cannot be processed before. FE performs any IP decrements TTL by 1 and processes any IP options that cannot be
fragmentation if necessary, determines link layer address (e.g., by processed before. The FE performs any IP fragmentation if
ARP), and encapsulates the IP datagram (or each of the fragments necessary, determines link layer address (e.g., by ARP), and
thereof) in an appropriate link layer frame and queues it for output encapsulates the IP datagram (or each of the fragments thereof) in
on the interface selected. an appropriate link layer frame and queues it for output on the
interface selected.
Other options mentioned in RFC1812 for IP forwarding may also be Other options mentioned in RFC1812 for IP forwarding may also be
implemented at FEs, for example, packet filtering. implemented at FEs, for example, packet filtering.
FEs typically forward packets destined locally to CEs. FEs may also FEs typically forward packets destined locally to CEs. FEs may
forward exceptional packets (packets that FEs do not know how to also forward exceptional packets (packets that FEs do not know how
handle) to CEs. CEs are required to handle packets forwarded by FEs to handle) to CEs. CEs are required to handle packets forwarded by
for whatever different reasons. It might be necessary for ForCES to FEs for whatever different reasons. It might be necessary for
attach some meta-data with the packets to indicate the reasons of ForCES to attach some meta-data with the packets to indicate the
forwarding from FEs to CEs. Upon receiving packets with meta-data reasons of forwarding from FEs to CEs. Upon receiving packets with
from FEs, CEs can decide to either process the packets themselves, meta-data from FEs, CEs can decide to either process the packets
or pass the packets to the upper layer protocols including routing themselves, or pass the packets to the upper layer protocols
and management protocols. If CEs are to process the packets by including routing and management protocols. If CEs are to process
themselves, CEs may choose to discard the packets, or modify and re- the packets by themselves, CEs may choose to discard the packets,
send the packets. CEs may also originate new packets and deliver or modify and re-send the packets. CEs may also originate new
them to FEs for further forwarding. packets and deliver them to FEs for further forwarding.
Any state change during router operation must also be handled Any state change during router operation must also be handled
correctly according to RFC1812. For example, when an FE ceases correctly according to RFC1812. For example, when an FE ceases
forwarding, the entire NE may continue forwarding packets, but it forwarding, the entire NE may continue forwarding packets, but it
needs to stop advertising routes that are affected by the failed FE. needs to stop advertising routes that are affected by the failed
FE.
5.5.Transport Layer 5.5.Transport Layer
Yang, et al. Expires December 2003 [Page 27]
Transport layer is typically implemented at CEs to support higher Transport layer is typically implemented at CEs to support higher
layer application protocols like routing protocols. In practice, layer application protocols like routing protocols. In practice,
Yang, et. al. Expires Dec 2003 [Page 24]
this means that most CEs implement both the Transmission Control this means that most CEs implement both the Transmission Control
Protocol (TCP) and the User Datagram Protocol (UDP). Protocol (TCP) and the User Datagram Protocol (UDP).
Both CEs and FEs need to implement ForCES protocol. If some layer-4 Both CEs and FEs need to implement ForCES protocol. If some layer-
transport is used to support ForCES, then both CEs and FEs need to 4 transport is used to support ForCES, then both CEs and FEs need
implement the L4 transport and ForCES protocols. to implement the L4 transport and ForCES protocols.
5.6. Application Layer -- Routing Protocols 5.6. Application Layer -- Routing Protocols
Interior and exterior routing protocols are implemented on CEs. The Interior and exterior routing protocols are implemented on CEs.
routing packets originated by CEs are forwarded to FEs for delivery. The routing packets originated by CEs are forwarded to FEs for
The results of such protocols (like forwarding table updates) are delivery. The results of such protocols (like forwarding table
communicated to FEs via ForCES. updates) are communicated to FEs via ForCES.
For performance or scalability reasons, portions of the control For performance or scalability reasons, portions of the control
plane functions that need faster response may be moved from the CEs plane functions that need faster response may be moved from the CEs
and off-loaded onto the FEs. For example in OSPF, the Hello and off-loaded onto the FEs. For example in OSPF, the Hello
protocol packets are generated and processed periodically. When protocol packets are generated and processed periodically. When
done at CEs, the inbound Hello packets have to traverse from the done at CEs, the inbound Hello packets have to traverse from the
external interfaces at the FEs to the CEs via the internal CE-FE external interfaces at the FEs to the CEs via the internal CE-FE
channel. Similarly, the outbound Hello packets have to go from the channel. Similarly, the outbound Hello packets have to go from the
CEs to the FEs and to the external interfaces. Frequent Hello CEs to the FEs and to the external interfaces. Frequent Hello
updates place heavy processing overhead on the CEs and can overwhelm updates place heavy processing overhead on the CEs and can
the CE-FE channel as well. Since typically there are far more FEs overwhelm the CE-FE channel as well. Since typically there are far
than CEs in a router, the off-loaded Hello packets are processed in more FEs than CEs in a router, the off-loaded Hello packets are
a much more distributed and scalable fashion. By expressing such processed in a much more distributed and scalable fashion. By
off-loaded functions in the FE model, we can ensure expressing such off-loaded functions in the FE model, we can ensure
interoperability. interoperability.
5.7. Application Layer -- Network Management Protocol 5.7. Application Layer -- Network Management Protocol
RFC1812 also dictates "Routers MUST be manageable by SNMP." (see RFC1812 also dictates "Routers MUST be manageable by SNMP." (see
[4] Section 8) In general, for post-association phase, most external [4] Section 8) In general, for post-association phase, most
management tasks (including SNMP) should be done through interaction external management tasks (including SNMP) should be done through
with the CE in order to support the appearance of a single interaction with the CE in order to support the appearance of a
functional device. Therefore, it is recommended that SNMP single functional device. Therefore, it is recommended that SNMP
management agent be implemented by CEs and the SNMP messages management agent be implemented by CEs and the SNMP messages
received by FEs be redirected to their CEs. AgentX framework received by FEs be redirected to their CEs. AgentX framework
defined in RFC2741 ([5]) may be applied here such that CEs act in defined in RFC2741 ([5]) may be applied here such that CEs act in
the role of master agent to process SNMP protocol messages while FEs the role of master agent to process SNMP protocol messages while
act in the role of subagent to provide access to the MIB objects FEs act in the role of subagent to provide access to the MIB
residing on FEs. AgentX protocol messages between the master agent objects residing on FEs. AgentX protocol messages between the
(CE) and the subagent (FE) are encapsulated and transported via master agent (CE) and the subagent (FE) are encapsulated and
ForCES, just like data packets from any other application layer
protocols. Yang, et al. Expires December 2003 [Page 28]
transported via ForCES, just like data packets from any other
application layer protocols.
6. Summary 6. Summary
This document defines an architectural framework for ForCES. It This document defines an architectural framework for ForCES. It
identifies the relevant components for a ForCES network element, identifies the relevant components for a ForCES network element,
including (one or more) FEs, (one or more) CEs, one optional FE including (one or more) FEs, (one or more) CEs, one optional FE
manager, and one optional CE manager. It also identifies the manager, and one optional CE manager. It also identifies the
interaction among these components and discusses all the major interaction among these components and discusses all the major
Yang, et. al. Expires Dec 2003 [Page 25]
reference points. It is important to point out that, among all the reference points. It is important to point out that, among all the
reference points, only the Fp interface between CEs and FEs is reference points, only the Fp interface between CEs and FEs is
within the scope of ForCES. ForCES alone may not be enough to within the scope of ForCES. ForCES alone may not be enough to
support all desirable NE configurations. However, we believe that support all desirable NE configurations. However, we believe that
ForCES over Fp interface is the most important element in realizing ForCES over Fp interface is the most important element in realizing
physical separation and interoperability of CEs and FEs, and hence physical separation and interoperability of CEs and FEs, and hence
the first interface that ought to be standardized. Simple and the first interface that ought to be standardized. Simple and
useful configurations can still be implemented with only CE-FE useful configurations can still be implemented with only CE-FE
interface being standardized, e.g., single CE with full-meshed FEs. interface being standardized, e.g., single CE with full-meshed FEs.
skipping to change at line 1359 skipping to change at line 1376
In general, the physical separation of two entities usually results In general, the physical separation of two entities usually results
in a potentially insecure link between the two entities and hence in a potentially insecure link between the two entities and hence
much stricter security measurements are required. For example, we much stricter security measurements are required. For example, we
pointed out in Section 4.1 that authentication becomes necessary pointed out in Section 4.1 that authentication becomes necessary
between CE manager and FE manager, between CE and CE manager, between CE manager and FE manager, between CE and CE manager,
between FE and FE manager in some configurations. The physical between FE and FE manager in some configurations. The physical
separation of CE and FE also imposes serious security requirement separation of CE and FE also imposes serious security requirement
for ForCES protocol over Fp interface. This section first attempts for ForCES protocol over Fp interface. This section first attempts
to describe the security threats that may be introduced by the to describe the security threats that may be introduced by the
physical separation of the FEs and the CEs, and then it provides physical separation of the FEs and the CEs, and then it provides
recommendation and guidelines for secure operation and management of recommendation and guidelines for secure operation and management
ForCES protocol over Fp interface. of ForCES protocol over Fp interface.
7.1. Analysis of Potential Threats Introduced by ForCES 7.1. Analysis of Potential Threats Introduced by ForCES
This section provides the threat analysis for ForCES, with a focus This section provides the threat analysis for ForCES, with a focus
on Fp interface. Each threat is described in details with the on Fp interface. Each threat is described in details with the
effects on the ForCES protocol entities or/and the NE as a whole, effects on the ForCES protocol entities or/and the NE as a whole,
and the required functionalities that need to be in place to defend and the required functionalities that need to be in place to defend
the threat. the threat.
7.1.1. Join or Remove Flooding on CEs 7.1.1. "Join" or "Remove" Message Flooding on CEs
Threats: A malicious node could send a stream of false join NE or Threats: A malicious node could send a stream of false "join NE"
remove from NE requests on behalf of non-existent or unauthorized or "remove from NE" requests on behalf of non-existent or
FE to legitimate CEs at a very rapid rate and thereby create
unnecessary state in the CEs. Yang, et al. Expires December 2003 [Page 29]
unauthorized FE to legitimate CEs at a very rapid rate and thereby
create unnecessary state in the CEs.
Effects: If by maintaining state for non-existent or unauthorized Effects: If by maintaining state for non-existent or unauthorized
FEs, a CE may become unavailable for other processing and hence FEs, a CE may become unavailable for other processing and hence
suffer from denial of service (DoS) attack similar to the TCP SYN suffer from denial of service (DoS) attack similar to the TCP SYN
DoS. If multiple CEs are used, the unnecessary state information may DoS. If multiple CEs are used, the unnecessary state information
also be conveyed to multiple CEs via Fr interface (e.g., from the may also be conveyed to multiple CEs via Fr interface (e.g., from
active CE to the stand-by CE) and hence subject multiple CEs to DoS the active CE to the stand-by CE) and hence subject multiple CEs to
attack. DoS attack.
Requirement: A CE that receives a join or remove request should Requirement: A CE that receives a "join" or "remove" request
not create any state information until it has authenticated the FE should not create any state information until it has authenticated
endpoint. the FE endpoint.
Yang, et. al. Expires Dec 2003 [Page 26]
7.1.2. Impersonation Attack 7.1.2. Impersonation Attack
Threats: A malicious node can impersonate a CE or FE and send out Threats: A malicious node can impersonate a CE or FE and send out
false messages. false messages.
Effects: The whole NE could be compromised. Effects: The whole NE could be compromised.
Requirement: The CE or FE must authenticate the message before Requirement: The CE or FE must authenticate the message before
accepting and processing it. accepting and processing it.
7.1.3. Replay Attack 7.1.3. Replay Attack
Threat: A malicious node could replay the entire message previously Threat: A malicious node could replay the entire message previously
sent by a FE or CE entity to get around authentication. sent by an FE or CE entity to get around authentication.
Effect: The NE could be compromised. Effect: The NE could be compromised.
Requirement: Replay protection mechanism needs to be part of the Requirement: Replay protection mechanism needs to be part of the
security protocol to defend this attack. security protocol to defend this attack.
7.1.4. Attack during Fail Over 7.1.4. Attack during Fail Over
Threat: A malicious node may exploit the CE fail-over mechanism to Threat: A malicious node may exploit the CE fail-over mechanism to
take over the control of NE. For example, suppose two CEs, say CE-A take over the control of NE. For example, suppose two CEs, say CE-A
and CE-B, are controlling several FEs. CE-A is active and CE-B is and CE-B, are controlling several FEs. CE-A is active and CE-B is
stand-by. When CE-A fails, CE-B is taking over the active CE stand-by. When CE-A fails, CE-B is taking over the active CE
position. The FEs already had a trusted relationship with CE-A, but position. The FEs already had a trusted relationship with CE-A,
the FEs may not have the same trusted relationship established with but the FEs may not have the same trusted relationship established
CE-B prior to the fail-over. A malicious node can take over as CE-B with CE-B prior to the fail-over. A malicious node can take over as
if such trusted relationship is not established during the fail-
over. Yang, et al. Expires December 2003 [Page 30]
CE-B if such trusted relationship is not established during the
fail-over.
Effect: The NE may be compromised after such insecure fail-over. Effect: The NE may be compromised after such insecure fail-over.
Requirement: The level of trust relationship between the stand-by Requirement: The level of trust relationship between the stand-by
CE and the FEs must be as strong as the one between the active CE CE and the FEs must be as strong as the one between the active CE
and the FEs, and such security association must be re-established and the FEs. The security association between the FEs and the
during fail-over. stand-by CE may be established prior to fail-over. If not already
in place, such security association must be re-established before
the stand-by CE takes over.
7.1.5. Data Integrity 7.1.5. Data Integrity
Threats: A malicious node may inject false messages to legitimate CE Threats: A malicious node may inject false messages to legitimate
or FE. CE or FE.
Effect: An FE or CE receives the fabricated packet and performs Effect: An FE or CE receives the fabricated packet and performs
incorrect or catastrophic operation. incorrect or catastrophic operation.
Requirement: Protocol messages require integrity protection. Requirement: Protocol messages require integrity protection.
7.1.6. Data Confidentiality 7.1.6. Data Confidentiality
Yang, et. al. Expires Dec 2003 [Page 27]
Threat: When FE and CE are physically separated, a malicious node Threat: When FE and CE are physically separated, a malicious node
may eavesdrop the messages in transit. Some of the messages are may eavesdrop the messages in transit. Some of the messages are
critical to the functioning of the whole network, while others may critical to the functioning of the whole network, while others may
contain confidential business data. Leaking of such information may contain confidential business data. Leaking of such information may
result in compromise even beyond the immediate CE or FE. result in compromise even beyond the immediate CE or FE.
Effect: Sensitive information might be exposed between CE and FE. Effect: Sensitive information might be exposed between CE and FE.
Requirement: Data confidentiality between FE and CE must be Requirement: Data confidentiality between FE and CE must be
available for sensitive information. available for sensitive information.
7.1.7. Sharing security parameters 7.1.7. Sharing security parameters
Threat: Consider a scenario where several FEs communicating to the Threat: Consider a scenario where several FEs communicating to the
same CE share the same authentication keys for the Fp interface. If same CE share the same authentication keys for the Fp interface. If
any FE or the CE is compromised, all other entities are compromised. any FE or the CE is compromised, all other entities are
compromised.
Effect: The whole NE is compromised. Effect: The whole NE is compromised.
Requirement: To avoid this side effect, its better to configure Requirement: To avoid this side effect, its better to configure
different security parameters for each FE-CE communication over Fp different security parameters for each FE-CE communication over Fp
interface. interface.
Yang, et al. Expires December 2003 [Page 31]
7.1.8. Denial of Service Attack via External Interface 7.1.8. Denial of Service Attack via External Interface
Threat: When an FE receives a packet that is destined for its CE, Threat: When an FE receives a packet that is destined for its CE,
the FE forwards the packet over the Fp interface. Malicious node can the FE forwards the packet over the Fp interface. Malicious node
generate huge message storm like routing protocol packets etc. can generate huge message storm like routing protocol packets etc.
through the external Fi/f interface so that the FE has to process through the external Fi/f interface so that the FE has to process
and forward all packets to CE through Fp interface. and forward all packets to CE through Fp interface.
Effect: CE encounters resource exhaustion and bandwidth starvation Effect: CE encounters resource exhaustion and bandwidth starvation
on Fp interface due to an overwhelming number of packets from FEs. on Fp interface due to an overwhelming number of packets from FEs.
Requirement: Rate limiting mechanism needs to be in place at both FE Requirement: Rate limiting mechanism needs to be in place at both
and CE. Rate Limiter can be configured at FE for each message type FE and CE. Rate Limiter can be configured at FE for each message
that are being received through Fi/F interface. type that are being received through Fi/F interface.
7.2. Security Recommendations for ForCES 7.2. Security Recommendations for ForCES
The requirements document [3] suggested that ForCES protocol should The requirements document [3] suggested that ForCES protocol should
support reliability over Fp interface, but no particular transport support reliability over Fp interface, but no particular transport
protocol is yet specified for ForCES. This framework document does protocol is yet specified for ForCES. This framework document does
not intend to specify the particular transport either, and so we not intend to specify the particular transport either, and so we
only provide recommendations and guidelines based on the existing only provide recommendations and guidelines based on the existing
standard security protocols that can work with the common transport standard security protocols that can work with the common transport
candidates suitable for ForCES. candidates suitable for ForCES.
We highlight two existing security protocol solutions, namely IPsec We highlight two existing security protocol solutions, namely IPsec
(IP Security) [14] or TLS (Transport Layer Security) [13]. TLS works (IP Security) [14] or TLS (Transport Layer Security) [13]. TLS
with reliable transports such as TCP or SCTP, while IPsec can be works with reliable transports such as TCP or SCTP, while IPsec can
used with any transport (UDP, TCP, SCTP). Both TLS and IPsec can be be used with any transport (UDP, TCP, SCTP). Both TLS and IPsec can
be used potentially to satisfy all of the security requirements for
Yang, et. al. Expires Dec 2003 [Page 28]
used potentially to satisfy all of the security requirements for
ForCES protocol. ForCES protocol.
It is important to realize that even if the NE is in a single-box, It is important to realize that even if the NE is in a single-box,
the DoS attacks can still be launched through Fi/f interfaces. the DoS attacks can still be launched through Fi/f interfaces.
Therefore, it is still important to have counter-measurement as Therefore, it is still important to have counter-measurement as
stated in 1.1.9 for DoS while authentication, confidentiality and stated in 1.1.9 for DoS while authentication, confidentiality and
integrity can be provided by the physical security of the box. integrity can be provided by the physical security of the box.
7.2.1. Security Configuration 7.2.1. Security Configuration
The NE administrator has the freedom to determine the exact security The NE administrator has the freedom to determine the exact
configuration that is needed for the specific deployment. For security configuration that is needed for the specific deployment.
example, ForCES may be deployed between CEs and FEs connected to For example, ForCES may be deployed between CEs and FEs connected
each other inside a box over a backplane. In such scenario, to each other inside a box over a backplane. In such scenario,
physical security of the box ensures that most of the attacks such physical security of the box ensures that most of the attacks such
as man-in-the-middle, snooping, and impersonation are not possible, as man-in-the-middle, snooping, and impersonation are not possible,
Yang, et al. Expires December 2003 [Page 32]
and hence ForCES architecture may rely on the physical security of and hence ForCES architecture may rely on the physical security of
the box to defend against these attacks and protocol mechanisms may the box to defend against these attacks and protocol mechanisms may
be turned off. However, it is also shown that denial of service be turned off. However, it is also shown that denial of service
attack via external interface as described in Section 7.1.8 is still attack via external interface as described in Section 7.1.8 is
a potential threat even for such all-in-one-box deployment still a potential threat even for such "all-in-one-box" deployment
scenario and hence the rate limiting mechanism is still necessary. scenario and hence the rate limiting mechanism is still necessary.
This is just one example to show that it is important to assess the This is just one example to show that it is important to assess the
security needs of the ForCES-enabled network elements under security needs of the ForCES-enabled network elements under
different deployment scenarios. It should be possible for the different deployment scenarios. It should be possible for the
administrator to configure the level of security needed for the administrator to configure the level of security needed for the
ForCES protocol. ForCES protocol.
7.2.2. Using TLS with ForCES 7.2.2. Using TLS with ForCES
TLS [13] can be used if a reliable transport such as TCP or SCTP is TLS [13] can be used if a reliable transport such as TCP or SCTP is
skipping to change at line 1548 skipping to change at line 1571
after the security handshake shown in Figure 9 and the steady-state after the security handshake shown in Figure 9 and the steady-state
communication shown in Figure 10. communication shown in Figure 10.
1) During Pre-association phase all FEs are configured with the CEs 1) During Pre-association phase all FEs are configured with the CEs
(including both the active CE and the standby CE). (including both the active CE and the standby CE).
2) The FE establishes a TLS connection with the CE (master) and 2) The FE establishes a TLS connection with the CE (master) and
negotiates a cipher suite. negotiates a cipher suite.
3) The FE (slave) gets the CE certificate, validates the signature, 3) The FE (slave) gets the CE certificate, validates the signature,
checks the expiration date, checks if the certificate has been checks the expiration date, checks if the certificate has been
revoked. revoked.
Yang, et. al. Expires Dec 2003 [Page 29]
4) The CE (master) gets the FE certificate and performs the same 4) The CE (master) gets the FE certificate and performs the same
validation as the FE in step 4). validation as the FE in step 4).
5) If any of the check fails in step 5 or step 6, endpoint must 5) If any of the check fails in step 5 or step 6, endpoint must
generate an error message and abort. generate an error message and abort.
6) After successful mutual authentication, a TLS session is 6) After successful mutual authentication, a TLS session is
established between CE and FE. established between CE and FE.
7) The FE sends a join NE message to the CE. 7) The FE sends a "join NE" message to the CE.
8) The FE and CE use TLS session for further communication. 8) The FE and CE use TLS session for further communication.
Note that there are different ways for the CE and FE to validate a Note that there are different ways for the CE and FE to validate a
received certificate. One way is to configure the FE Manager or CE received certificate. One way is to configure the FE Manager or CE
Manager or other central component as CA, so that the CE or FE can Manager or other central component as CA, so that the CE or FE can
Yang, et al. Expires December 2003 [Page 33]
query this pre-configured CA to validate that the certificate has query this pre-configured CA to validate that the certificate has
not been revoked. Another way is to have the CE and the FE not been revoked. Another way is to have the CE and the FE
configured directly a list of valid certificates in the pre- configured directly a list of valid certificates in the pre-
association phase. association phase.
In the case of fail-over, it is the responsibility of the active CE In the case of fail-over, it is the responsibility of the active CE
and the standby CE to synchronize ForCES states including the TLS and the standby CE to synchronize ForCES states including the TLS
states to minimize the state reestablishment during fail-over. Care states to minimize the state reestablishment during fail-over. Care
must be taken to ensure that the standby CE is also authenticated in must be taken to ensure that the standby CE is also authenticated
the same way as the active CE, either before or during the fail- in the same way as the active CE, either before or during the fail-
over. over.
7.2.3. Using IPsec with ForCES 7.2.3. Using IPsec with ForCES
IPsec [14] can be used with any transport protocol, such as UDP, IPsec [14] can be used with any transport protocol, such as UDP,
SCTP and TCP over Fp interface for ForCES. We recommend using ESP SCTP and TCP over Fp interface for ForCES. We recommend using ESP
in transport mode for ForCES because message confidentiality is in transport mode for ForCES because message confidentiality is
required for ForCES and the communication between the CE and FE is required for ForCES and the communication between the CE and FE is
point-to-point. point-to-point.
IPsec can be used with both manual and automated SA and IPsec can be used with both manual and automated SA and
cryptographic key management. But Ipsecs replay protection cryptographic key management. But Ipsecs replay protection
mechanisms are not available if manual key management is used. mechanisms are not available if manual key management is used.
Hence, automatic key management is recommended if replay protection Hence, automatic key management is recommended if replay protection
is deemed important. Otherwise, manual key management might be is deemed important. Otherwise, manual key management might be
sufficient for some deployment scenarios, esp. when the number of sufficient for some deployment scenarios, esp. when the number of
CEs and FEs is relatively small. It is recommended that the keys be CEs and FEs is relatively small. It is recommended that the keys
changed periodically even for manual key management. be changed periodically even for manual key management.
Unlike TLS, IPsec provides security services between the CE and FE Unlike TLS, IPsec provides security services between the CE and FE
at IP level, and so the security handshake as illustrated in Figure at IP level, and so the security handshake as illustrated in Figure
9 amounts to a no-op when manual key management is used. The 9 amounts to a "no-op" when manual key management is used. The
following outline the steps taken for ForCES in such a case. following outline the steps taken for ForCES in such a case.
1) During Pre-association phase all FEs are configured with the CEs 1) During Pre-association phase all FEs are configured with the CEs
(including active CE and standby CE) and SA parameters manually. (including active CE and standby CE) and SA parameters manually.
2) The FE sends a join NE message to the CE. This message and all 2) The FE sends a "join NE" message to the CE. This message and
others that follow are afforded security service according to the all others that follow are afforded security service according to
manually configured IPsec SA parameters, but replay protection is the manually configured IPsec SA parameters, but replay protection
not available. is not available.
Yang, et. al. Expires Dec 2003 [Page 30]
It is up to the administrator to decide whether to share the same It is up to the administrator to decide whether to share the same
key across multiple FE-CE communication, but it is recommended that key across multiple FE-CE communication, but it is recommended that
different keys be used. Similarly, it is recommended that different different keys be used. Similarly, it is recommended that
keys be used for inbound and outbound traffic. different keys be used for inbound and outbound traffic.
If automatic key management is needed, IKE [15] can be used for that Yang, et al. Expires December 2003 [Page 34]
purpose. Other automatic key distribution techniques such as If automatic key management is needed, IKE [15] can be used for
that purpose. Other automatic key distribution techniques such as
Kerberos may be used as well. The key exchange process constitutes Kerberos may be used as well. The key exchange process constitutes
the security handshake as illustrated in Figure 9. The following the security handshake as illustrated in Figure 9. The following
shows the steps involved in using IKE with IPsec for ForCES. Steps shows the steps involved in using IKE with IPsec for ForCES. Steps
1) to 6) constitute the security handshake in Figure 9. 1) to 6) constitute the security handshake in Figure 9.
1) During Pre-association phase all FEs are configured with the CEs 1) During Pre-association phase all FEs are configured with the CEs
(including active CE and standby CE), IPsec policy etc. (including active CE and standby CE), IPsec policy etc.
2) The FE kicks off IKE process and tries to establish an IPsec SA 2) The FE kicks off IKE process and tries to establish an IPsec SA
with the CE (master). The FE (Slave) gets the CE certificate as part with the CE (master). The FE (Slave) gets the CE certificate as
of the IKE negotiation. The FE validates signature, checks the part of the IKE negotiation. The FE validates signature, checks the
expiration date, checks if the certificate has been revoked. expiration date, checks if the certificate has been revoked.
3) The CE (master) gets the FE certificate and performs the same 3) The CE (master) gets the FE certificate and performs the same
check as the FE in step 3). check as the FE in step 3).
4) If any of the check fails in step 3 or step 4, the endpoint must 4) If any of the check fails in step 3 or step 4, the endpoint must
generate an error message and abort. generate an error message and abort.
5) After successful mutual authentication, IPsec session is 5) After successful mutual authentication, IPsec session is
established between the CE and FE. established between the CE and FE.
6) The FE sends a join NE message to CE. No SADB entry is created 6) The FE sends a "join NE" message to CE. No SADB entry is created
in FE yet. in FE yet.
7) The FE and CE use the IPsec session for further communication. 7) The FE and CE use the IPsec session for further communication.
FE Manager or CE Manager or other central component can be used as FE Manager or CE Manager or other central component can be used as
CA for validating CE and FE certificates during the IKE process. CA for validating CE and FE certificates during the IKE process.
Alternatively, during the pre-association phase, the CE and FE can Alternatively, during the pre-association phase, the CE and FE can
be configured directly with the required information such as be configured directly with the required information such as
certificates or passwords etc depending upon the type of certificates or passwords etc depending upon the type of
authentication that administrator wants to configure. authentication that administrator wants to configure.
In the case of fail-over, it is the responsibility of active CE and In the case of fail-over, it is the responsibility of active CE and
standby CE to synchronize ForCES states and IPsec states to minimize standby CE to synchronize ForCES states and IPsec states to
the state reestablishment during fail-over. Alternatively, the FE minimize the state reestablishment during fail-over. Alternatively,
needs to establish different IPsec SA during the startup operation the FE needs to establish different IPsec SA during the startup
itself with each CEs. This will minimize the periodic state operation itself with each CE. This will minimize the periodic
transfer across IPsec layer though Fr (CE-CE) Interface. state transfer across IPsec layer though Fr (CE-CE) Interface.
8. Normative References 8. Normative References
[1] F. Baker, Requirements for IP Version 4 Routers", RFC1812, June [1] Baker, F., "Requirements for IP Version 4 Routers", RFC 1812,
1995. June 1995.
[2] S. Floyd, Congestion Control Principles", RFC2914, September [2] Floyd, S., "Congestion Control Principles", RFC 2914, September
2000. 2000.
Yang, et. al. Expires Dec 2003 [Page 31] Yang, et al. Expires December 2003 [Page 35]
[3] H. Khosravi, et. al., Requirements for Separation of IP Control [3] Khosravi, H. et al., "Requirements for Separation of IP Control
and Forwarding", work in progress, May 2003, <draft-ietf-forces- and Forwarding", work in progress, May 2003, <draft-ietf-forces-
requirements-09.txt>. requirements-09.txt>.
9. Informative References 9. Informative References
[4] J. Case, et. al., A Simple Network Management Protocol (SNMP)", [4] Case, J., et al., "A Simple Network Management Protocol
RFC1157, May 1990. (SNMP)", RFC 1157, May 1990.
[5] M. Daniele, et. al., Agent Extensibility (AgentX) Protocol [5] Daniele, M. et al., "Agent Extensibility (AgentX) Protocol
Version 1", RFC2741, Jan 2000. Version 1", RFC 2741, January 2000.
[6] K. Chan, et. al., COPS Usage for Policy Provisioning (COPS- [6] Chan, K. et al., "COPS Usage for Policy Provisioning (COPS-
PR)", RFC3084, March 2001. PR)", RFC3084, March 2001.
[7] A. Crouch, et. al., ForCES Applicability Statement", work in [7] Crouch, A. et al., "ForCES Applicability Statement", work in
progress, June 2002, <draft-ietf-forces-applicability-00.txt>. progress, June 2002, <draft-ietf-forces-applicability-00.txt>.
[8] T. Anderson, J. Buerkle, Requirements for the Dynamic [8] Anderson, T. and J. Buerkle, "Requirements for the Dynamic
Partitioning of Switching Elements", RFC3532, May 2003. Partitioning of Switching Elements", RFC3532, May 2003.
[9] M. Leelanivas, et. al., Graceful Restart Mechanism for Label [9] Leelanivas, M. et al., "Graceful Restart Mechanism for Label
Distribution Protocol, RFC 3478, Feb. 2003. Distribution Protocol", RFC 3478, February 2003.
[10] J. Moy, et. al., Graceful OSPF Restart, work in progress, [10] Moy, J. et al., "Graceful OSPF Restart", work in progress,
March 2003, <draft-ietf-ospf-hitless-restart-07.txt>. March 2003, <draft-ietf-ospf-hitless-restart-07.txt>.
[11] S. R. Sangli, et. al., Graceful Restart Mechanism for BGP, [11] Sangli, S. et al., "Graceful Restart Mechanism for BGP", work
work in progress, Jan 203, < draft-ietf-idr-restart-06.txt>. in progress, January 2003, < draft-ietf-idr-restart-06.txt>.
[12] M. Shand and L. Ginsberg, Restart Signaling for IS-IS, work [12] Shand, M. and L. Ginsberg, "Restart Signaling for IS-IS", work
in progress, March 2003, <draft-ietf-isis-restart-03.txt>. in progress, March 2003, <draft-ietf-isis-restart-03.txt>.
[13] T. Dierks and C. Allen, The TLS Protocol, version 1.0, [13] Dierks, T. and C. Allen, "The TLS Protocol, version 1.0", RFC
RFC2246, Jan. 1999. 2246, January 1999.
[14] S. Kent and R. Atkinson, Security Architecture for the [14] Kent, S. and R. Atkinson, "Security Architecture for the
Internet Protocol, RFC2401, Nov. 1998. Internet Protocol", RFC 2401, November 1998.
[15] D. Harkins and D. Carrel, The Internet Key Exchange (IKE), [15] Harkins, D. and D. Carrel, "The Internet Key Exchange (IKE) ",
RFC2409, Nov. 1998. RFC 2409, November 1998.
[16] S. M. Bellovin, Guidelines for Mandating the Use of Ipsec, [16] Bellovin, S., "Guidelines for Mandating the Use of Ipsec",
work in progress, Oct. 2002, <draft-bellovin-useipsec-00.txt>. work in progress, October 2002, <draft-bellovin-useipsec-00.txt>.
10. Acknowledgement 10. Acknowledgements
Joel M. Halpern gave us many insightful comments and suggestions and Yang, et al. Expires December 2003 [Page 36]
pointed out several major issues. T. Sridhar suggested that the Joel M. Halpern gave us many insightful comments and suggestions
AgentX protocol could be used with SNMP to manage the ForCES network and pointed out several major issues. T. Sridhar suggested that
elements. Many of our colleagues and people in the ForCES mailing the AgentX protocol could be used with SNMP to manage the ForCES
list also provided valuable feedback. network elements. Many of our colleagues and people in the ForCES
mailing list also provided valuable feedback.
Yang, et. al. Expires Dec 2003 [Page 32]
11. Authors' Addresses 11. Authors' Addresses
Lily L. Yang Lily L. Yang
Intel Corp., MS JF3-206, Intel Corp., MS JF3-206,
2111 NE 25th Avenue 2111 NE 25th Avenue
Hillsboro, OR 97124 USA Hillsboro, OR 97124, USA
Phone: +1 503 264 8813 Phone: +1 503 264 8813
Email: lily.l.yang@intel.com Email: lily.l.yang@intel.com
Ram Dantu Ram Dantu
Department of Computer Science, Department of Computer Science,
University of North Texas, University of North Texas,
Denton, Texas, 76203 Denton, TX 76203, USA
Phone: +1 940 565 2822 Phone: +1 940 565 2822
Email: rdantu@unt.edu Email: rdantu@unt.edu
Todd A. Anderson Todd A. Anderson
Intel Corp. Intel Corp.
2111 NE 25th Avenue 2111 NE 25th Avenue
Hillsboro, OR 97124 USA Hillsboro, OR 97124, USA
Phone: +1 503 712 1760 Phone: +1 503 712 1760
Email: todd.a.anderson@intel.com Email: todd.a.anderson@intel.com
Ram Gopal Ram Gopal
Nokia Research Center Nokia Research Center
5, Wayside Road, 5, Wayside Road,
Burlington, MA 01803 Burlington, MA 01803, USA
Phone: +1 781 993 3685 Phone: +1 781 993 3685
Email: ram.gopal@nokia.com Email: ram.gopal@nokia.com
12. Intellectual Property Right 12. Intellectual Property Right
The authors are not aware of any intellectual property right issues The IETF takes no position regarding the validity or scope of any
pertaining to this document. intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on
the IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in RFC 2026. Copies
of
Yang, et al. Expires December 2003 [Page 37]
claims of rights made available for publication and any assurances
of licenses to be made available, or the result of an attempt made
to obtain a general license or permission for the use of such
proprietary rights by implementors or users of this specification
can be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF
Executive Director.
13. Full Copyright Statement 13. Full Copyright Statement
Copyright (C) The Internet Society (2002). All Rights Reserved. Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it others, and derivative works that comment on or otherwise explain
or assist in its implementation may be prepared, copied, published it or assist in its implementation may be prepared, copied,
and distributed, in whole or in part, without restriction of any published and distributed, in whole or in part, without restriction
kind, provided that the above copyright notice and this paragraph of any kind, provided that the above copyright notice and this
are included on all such copies and derivative works. However, this paragraph are included on all such copies and derivative works.
document itself may not be modified in any way, such as by removing However, this document itself may not be modified in any way, such
the copyright notice or references to the Internet Society or other as by removing the copyright notice or references to the Internet
Internet organizations, except as needed for the purpose of Society or other Internet organizations, except as needed for the
developing Internet standards in which case the procedures for purpose of developing Internet standards in which case the
copyrights defined in the Internet Standards process must be procedures for copyrights defined in the Internet Standards process
followed, or as required to translate it into languages other than must be followed, or as required to translate it into languages
English. other than English.
Yang, et. al. Expires Dec 2003 [Page 33]
The limited permissions granted above are perpetual and will not be The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns. revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an This document and the information contained herein is provided on
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE." WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."
Yang, et. al. Expires Dec 2003 [Page 34] Yang, et al. Expires December 2003 [Page 38]
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/