draft-ietf-forces-ceha-05.txt   draft-ietf-forces-ceha-06.txt 
Network Working Group K. Ogawa Network Working Group K. Ogawa
Internet-Draft NTT Corporation Internet-Draft NTT Corporation
Intended status: Standards Track W. M. Wang Intended status: Standards Track W. M. Wang
Expires: July 21, 2013 Zhejiang Gongshang University Expires: August 23, 2013 Zhejiang Gongshang University
E. Haleplidis E. Haleplidis
University of Patras University of Patras
J. Hadi Salim J. Hadi Salim
Mojatatu Networks Mojatatu Networks
January 17, 2013 February 19, 2013
ForCES Intra-NE High Availability ForCES Intra-NE High Availability
draft-ietf-forces-ceha-05 draft-ietf-forces-ceha-06
Abstract Abstract
This document discusses CE High Availability within a ForCES NE. This document discusses CE High Availability within a ForCES NE.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 21, 2013. This Internet-Draft will expire on August 23, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 13 skipping to change at page 2, line 13
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Document Scope . . . . . . . . . . . . . . . . . . . . . . 5 2.1. Document Scope . . . . . . . . . . . . . . . . . . . . . . 5
2.2. Quantifying Problem Scope . . . . . . . . . . . . . . . . 5 2.2. Quantifying Problem Scope . . . . . . . . . . . . . . . . 5
3. RFC5810 CE HA Framework . . . . . . . . . . . . . . . . . . . 6 3. RFC5810 CE HA Framework . . . . . . . . . . . . . . . . . . . 6
3.1. Current CE High Availability Support . . . . . . . . . . . 6 3.1. RFC 5810 CE High Availability Support . . . . . . . . . . 6
3.1.1. Cold Standby Interaction with ForCES Protocol . . . . 7 3.1.1. Cold Standby Interaction with ForCES Protocol . . . . 7
3.1.2. Responsibilities for HA . . . . . . . . . . . . . . . 9 3.1.2. Responsibilities for HA . . . . . . . . . . . . . . . 9
4. CE HA Hot Standby . . . . . . . . . . . . . . . . . . . . . . 10 4. CE HA Hot Standby . . . . . . . . . . . . . . . . . . . . . . 10
4.1. Changes to the FEPO model . . . . . . . . . . . . . . . . 10 4.1. Changes to the FEPO model . . . . . . . . . . . . . . . . 10
4.2. FEPO processing . . . . . . . . . . . . . . . . . . . . . 11 4.2. FEPO processing . . . . . . . . . . . . . . . . . . . . . 12
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16
6. Security Considerations . . . . . . . . . . . . . . . . . . . 16 6. Security Considerations . . . . . . . . . . . . . . . . . . . 16
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 17 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7.1. Normative References . . . . . . . . . . . . . . . . . . . 17 7.1. Normative References . . . . . . . . . . . . . . . . . . . 16
7.2. Informative References . . . . . . . . . . . . . . . . . . 17 7.2. Informative References . . . . . . . . . . . . . . . . . . 17
Appendix 1. Appendix I - New FEPO version . . . . . . . . . . . . 17 Appendix A. New FEPO version . . . . . . . . . . . . . . . . . . 17
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26
1. Definitions 1. Definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119. document are to be interpreted as described in RFC 2119.
The following definitions are taken from [RFC3654]and [RFC3746]: The following definitions are taken from [RFC3654]and [RFC3746]:
Logical Functional Block (LFB) -- A template that represents a fine- o Logical Functional Block (LFB) -- A template that represents a
grained, logically separate aspects of FE processing. fine-grained, logically separate aspects of FE processing.
ForCES Protocol -- The protocol used at the Fp reference point in the o ForCES Protocol -- The protocol used at the Fp reference point in
ForCES Framework in [RFC3746]. the ForCES Framework in [RFC3746].
ForCES Protocol Layer (ForCES PL) -- A layer in the ForCES o ForCES Protocol Layer (ForCES PL) -- A layer in the ForCES
architecture that embodies the ForCES protocol and the state transfer architecture that embodies the ForCES protocol and the state
mechanisms as defined in [RFC5810]. transfer mechanisms as defined in [RFC5810].
ForCES Protocol Transport Mapping Layer (ForCES TML) -- A layer in o ForCES Protocol Transport Mapping Layer (ForCES TML) -- A layer in
ForCES protocol architecture that specifically addresses the protocol ForCES protocol architecture that specifically addresses the
message transportation issues, such as how the protocol messages are protocol message transportation issues, such as how the protocol
mapped to different transport media (like SCTP, IP, TCP, UDP, ATM, messages are mapped to different transport media (like SCTP, IP,
Ethernet, etc), and how to achieve and implement reliability, TCP, UDP, ATM, Ethernet, etc), and how to achieve and implement
security, etc. reliability, security, etc.
2. Introduction 2. Introduction
Figure 1 illustrates a ForCES NE controlled by a set of redundant CEs Figure 1 illustrates a ForCES NE controlled by a set of redundant CEs
with CE1 being active and CE2 and CEn-1 being a backup. with CE1 being active and CE2 and CEN being a backup.
----------------------------------------- -----------------------------------------
| ForCES Network Element | | ForCES Network Element |
| +-----------+ | | +-----------+ |
| | CEn-1 | | | | CEN | |
| | (Backup) | | | | (Backup) | |
-------------- Fc | +------------+ +------------+ | | -------------- Fc | +------------+ +------------+ | |
| CE Manager |--------+-| CE1 |------| CE2 |-+ | | CE Manager |--------+-| CE1 |------| CE2 |-+ |
-------------- | | (Active) | Fr | (Backup) | | -------------- | | (Active) | Fr | (Backup) | |
| | +-------+--+-+ +---+---+----+ | | | +-------+--+-+ +---+---+----+ |
| Fl | | | Fp / | | | Fl | | | Fp / | |
| | | +---------+ / | | | | | +---------+ / | |
| | Fp| |/ |Fp | | | Fp| |/ |Fp |
| | | | | | | | | | | |
| | | Fp /+--+ | | | | | Fp /+--+ | |
skipping to change at page 4, line 52 skipping to change at page 4, line 52
Fi/f: FE external interface Fi/f: FE external interface
Figure 1: ForCES Architecture Figure 1: ForCES Architecture
The ForCES architecture allows FEs to be aware of multiple CEs but The ForCES architecture allows FEs to be aware of multiple CEs but
enforces that only one CE be the master controller. This is known in enforces that only one CE be the master controller. This is known in
the industry as 1+N redundancy. The master CE controls the FEs via the industry as 1+N redundancy. The master CE controls the FEs via
the ForCES protocol operating in the Fp interface. If the master CE the ForCES protocol operating in the Fp interface. If the master CE
becomes faulty, a backup CE takes over and NE operation continues. becomes faulty, a backup CE takes over and NE operation continues.
By definition, the current documented setup is known as cold-standby. By definition, the current documented setup is known as cold-standby.
The CE set is static and is passed to the FE by the FE Manager (FEM) The set of CEs controlling an FE is static and is passed to the FE by
via the Ff interface and to each CE by the CE Manager (CEM) in the Fc the FE Manager (FEM) via the Ff interface and to each CE by the CE
interface during the pre-association phase. Manager (CEM) in the Fc interface during the pre-association phase.
From an FE perspective, the knobs of control for a CE set are defined From an FE perspective, the knobs of control for a CE set are defined
by the FEPO LFB in [RFC5810], Appendix B. Section 3.1 of this by the FEPO LFB in [RFC5810], Appendix B. Section 3.1 of this
document details these knobs further. document details these knobs further.
2.1. Document Scope 2.1. Document Scope
It is assumed that the reader is aware of the ForCES architecture to It is assumed that the reader is aware of the ForCES architecture to
make sense of the changes made here. This document provides minimal make sense of the changes being described in this document. This
background to set the context of the discussion in Section 4. document provides background information to set the context of the
discussion in Section 4.
By current definition, the Fr interface is out of scope for the At the time this document is being written, the Fr interface is out
ForCES architecture. However, it is expected that organizations of scope for the ForCES architecture. However, it is expected that
implementing a set of CEs will need to have the CEs communicate to organizations implementing a set of CEs will need to have the CEs
each other via the Fr interface in order to achieve the communicate to each other via the Fr interface in order to achieve
synchronization necessary for controlling the FEs. the synchronization necessary for controlling the FEs.
The problem scope addressed by this document falls into 2 areas: The problem scope addressed by this document falls into 2 areas:
1. To describe with more clarity (than [RFC5810]) how current cold- 1. To describe with more clarity (than [RFC5810]) how current cold-
standby approach operates within the NE cluster. standby approach operates within the NE cluster.
2. To describe how to evolve the cold-standby setup to a hot-standby 2. To describe how to evolve the [RFC5810] cold-standby setup to a
redundancy setup so as to improve the failover time and NE hot-standby redundancy setup so as to improve the failover time
availability. and NE availability.
2.2. Quantifying Problem Scope 2.2. Quantifying Problem Scope
The NE recovery and availability is dependent on several time- The NE recovery and availability is dependent on several time-
sensitive metrics: sensitive metrics:
1. How fast the CE plane failure is detected the FE. 1. How fast the CE plane failure is detected by the FE.
2. How fast a backup CE becomes operational. 2. How fast a backup CE becomes operational.
3. How fast the FEs associate with the new master CE. 3. How fast the FEs associate with the new master CE.
4. How fast the FEs recover their state and become operational. 4. How fast the FEs recover their state and become operational.
The design goals of the current [RFC5810] choices to meet the above The design intent of the current [RFC5810] as well as this document
goals are driven by desire for simplicity. to meet the above goals are driven by desire for simplicity.
To quantify the above criteria with the current prescribed ForCES CE To quantify the above criteria with the current prescribed ForCES CE
setup in [RFC5810]: setup in [RFC5810]:
1. How fast the CE side detects a CE failure is left undefined. To 1. How fast the FE side detects a CE failure is left undefined. To
illustrate an extreme scenario, we could have a human operator illustrate an extreme scenario, we could have a human operator
acting as the monitoring entity to detect faulty CEs. How fast acting as the monitoring entity to detect faulty CEs. How fast
such detection happens could be in the range of seconds to days. such detection happens could be in the range of seconds to days.
A more active monitor on the Fr interface could improve this A more active monitor on the Fr interface could improve this
detection. detection.
2. How fast the backup CE becomes operational is also currently out 2. How fast the backup CE becomes operational is also currently out
of scope. In the current setup, a backup CE need not be of scope. In the current setup, a backup CE need not be
operational at all (for example, to save power) and therefore it operational at all (for example, to save power) and therefore it
is feasible for a monitoring entity to boot up a backup CE after is feasible for a monitoring entity to boot up a backup CE after
it detects the failure of the master CE. In this document it detects the failure of the master CE. In this document
Section 4 we suggest that at least one backup CE be online so as Section 4 we suggest that at least one backup CE be online so as
to improve this metric. to improve this metric.
3. How fast an FE associates with new master CE is also currently 3. How fast an FE associates with new master CE is also currently
undefined. The cost of an FE connecting and associating adds to undefined. The cost of an FE connecting and associating adds to
the recovery overhead. As mentioned above we suggest having at the recovery overhead. As mentioned above we suggest having at
least one backup CE online. In Section 4 we propose to zero out least one backup CE online. In Section 4 we propose to zero out
the connection and association cost on failover by having each FE the connection and association cost on failover by having each FE
associate with all online backup CEs after associating to the associate with all online backup CEs after associating to an
active CE. Note that if an FE pre-associates with backup CEs, active/master CE. Note that if an FE pre-associates with at
then the system will be technically operating in hot-standby least one backup CE, then the system will be technically
mode. operating in hot-standby mode.
4. And last: How fast an FE recovers its state depends on how much 4. And last: How fast an FE recovers its state depends on how much
NE state exists. By ForCES current definition, the new master CE NE state exists. By ForCES current definition, the new master CE
assumes zero state on the FE and starts from scratch to update assumes zero state on the FE and starts from scratch to update
the FE. So the larger the state, the longer the recovery. the FE. So the larger the state, the longer the recovery.
3. RFC5810 CE HA Framework 3. RFC5810 CE HA Framework
To achieve CE High Availabilty, FEs and CEs MUST inter-operate per To achieve CE High Availabilty, FEs and CEs MUST inter-operate per
[RFC5810] definition which is repeated for contextual reasons in [RFC5810] definition which is repeated for contextual reasons in
Section 3.1. It should be noted that in this default setup, which Section 3.1. It should be noted that in this default setup, which
MUST be implemented by CEs and FEs needing HA, the Fr plane is out of MUST be implemented by CEs and FEs needing HA, the Fr plane is out of
scope (and if available is proprietary to an implementation). scope (and if available is proprietary to an implementation).
3.1. Current CE High Availability Support 3.1. RFC 5810 CE High Availability Support
As mentioned earlier, although there can be multiple redundant CEs, As mentioned earlier, although there can be multiple redundant CEs,
only one CE actively controls FEs in a ForCES NE. In practice there only one CE actively controls FEs in a ForCES NE. In practice there
may be only one backup CE. At any moment in time only one master CE may be only one backup CE. At any moment in time, only one master CE
can control the FEs. In addition, the FE connects and associates to can control an FE. In addition, the FE connects and associates to
only the master CE. The FE and the CE PL are aware of the primary only the master CE. The FE and the CE are aware of the primary and
and one or more secondary CEs. This information (primary, secondary one or more secondary CEs. This information (primary, secondary CEs)
CEs) is configured on the FE and the CE PLs during pre-association by is configured on the FE and the CE during pre-association by the FEM
the FEM and the CEM respectively. and the CEM respectively.
Figure 2 below illustrates the Forces message sequences that the FE Figure 2 below illustrates the Forces message sequences that the FE
uses to recover the connection in current defined cold-standby uses to recover the connection in current defined cold-standby
scheme. scheme.
FE CE Primary CE Secondary FE CE Primary CE Secondary
| | | | | |
| Asso Estb,Caps exchg | | | Asso Estb,Caps exchg | |
1 |<--------------------->| | 1 |<--------------------->| |
| | | | | |
skipping to change at page 7, line 38 skipping to change at page 7, line 38
5 |<------------------------------------------>| 5 |<------------------------------------------>|
Figure 2: CE Failover for Cold Standby Figure 2: CE Failover for Cold Standby
3.1.1. Cold Standby Interaction with ForCES Protocol 3.1.1. Cold Standby Interaction with ForCES Protocol
High Availability parameterization in an FE is driven by configuring High Availability parameterization in an FE is driven by configuring
the FE Protocol Object (FEPO) LFB. the FE Protocol Object (FEPO) LFB.
The FEPO CEID component identifies the current master CE and the The FEPO CEID component identifies the current master CE and the
component table BackupCEs identifies the backup CEs. The FEPO FE component table BackupCEs identifies the configured backup CEs. The
Heartbeat Interval, CE Heartbeat Dead Interval, and CE Heartbeat FEPO FE Heartbeat Interval, CE Heartbeat Dead Interval, and CE
policy help in detecting connectivity problems between an FE and CE. Heartbeat policy help in detecting connectivity problems between an
The CE Failover policy defines how the FE should react on a detected FE and CE. The CE Failover policy defines how the FE should react on
failure. a detected failure. The FEObject FEState component [RFC5812] defines
the operational forwarding status and control. The CE can turn off
the FE's forwarding operations by setting the FEState to AdminDisable
and can turn it on by setting it to OperEnable. Note: [RFC5812]
section 5.1 has an erratta which describes the FEState as read-only
when it should be read-write.
Figure 3 illustrates the defined state machine that facilitates Figure 3 illustrates the defined state machine that facilitates
connection recovery. connection recovery.
The FE connects to the CE specified on FEPO CEID component. If it The FE connects to the CE specified on FEPO CEID component. If it
fails to connect to the defined CE, it moves it to the bottom of fails to connect to the defined CE, it moves it to the bottom of
table BackupCEs and sets its CEID component to be the first CE table BackupCEs and sets its CEID component to be the first CE
retrieved from table BackupCEs. The FE then attempts to associate retrieved from table BackupCEs. The FE then attempts to associate
with the CE designated as the new primary CE. The FE continues with the CE designated as the new primary CE. The FE continues
through this procedure until it successfully connects to one of the through this procedure until it successfully connects to one of the
CEs. CEs.
FE tries to associate FE tries to associate
+-->-----+ +-->-----+
| | | |
(CE issues Teardown || +---+--------v----+ (CE changes master || | |
Lost association) && | Pre-Association | CE issues Teardown || +---+--------v----+
CE failover policy = 0 | (Association | Lost association) && | Pre-Association |
+------------>-->-->| in +<----+ CE failover policy = 0 | (Association |
| | progress) | | +------------>-->-->| in +<----+
| CE Issues +--------+--------+ | | | progress) | |
| Association | | CEFTI | | | |
| Response V | timer | +--------+--------+ |
| +------------------+ | expires | CE Association | | CEFTI
| | ^ | Response V | timer
| V | | +------------------+ | expires
+-+-----------+ +------+-----+ | |FE issue CEPrimaryDown ^
| | | Not | | V |
| | (CE issues Teardown || | Associated | +-+-----------+ +------+-----+
| | Lost association) && | +->---+ | | (CE changes master || | Not |
| Associated | CE Failover Policy = 1 |(May | FE | | | CE issues Teardown || | Associated |
| | | Continue |try v | | Lost association) && | +->---+
| |-------->------->------>| Forwarding)|assn | | Associated | CE Failover Policy = 1 |(May | FE |
| | Start CEFTI timer | |-<---+ | | | Continue |try v
| | | | | |-------->------->------>| Forwarding)|assn |
+-------------+ +-------+-----+ | | Start CEFTI timer | |-<---+
^ | | | | |
| CE Issues V +-------------+ +-------+-----+
| Association | ^ |
| Setup | | Successful V
| (Cancel CEFTI Timer) | | Association |
+_________________________________________+ | Setup |
| (Cancel CEFTI Timer) |
+_________________________________________+
FE issue CEPrimaryDown event
Figure 3: FE State Machine considering HA Figure 3: FE State Machine considering HA
There are several events that trigger mastership changes: The master
CE may issue a mastership change (by changing the CEID value), or
teardown an existing association; and last, connectivity may be lost
between the CE and FE.
When communication fails between the FE and CE (which can be caused When communication fails between the FE and CE (which can be caused
by either the CE or link failure but not FE related), either the TML by either the CE or link failure but not FE related), either the TML
on the FE will trigger the FE PL regarding this failure or it will be on the FE will trigger the FE PL regarding this failure or it will be
detected using the HB messages between FEs and CEs. The detected using the HB messages between FEs and CEs. The
communication failure, regardless of how it is detected, MUST be communication failure, regardless of how it is detected, MUST be
considered as a loss of association between the CE and corresponding considered as a loss of association between the CE and corresponding
FE. FE.
If the FE's FEPO CE Failover Policy is configured to mode 0 (the If the FE's FEPO CE Failover Policy is configured to mode 0 (the
default), it will immediately transition to the pre-association default), it will immediately transition to the pre-association
phase. This means that if association is again established, all FE phase. This means that if association is later re-established with a
state will need to be re-established. CE, all FE state will need to be re-created.
If the FE's FEPO CE Failover Policy is configured to mode 1, it If the FE's FEPO CE Failover Policy is configured to mode 1, it
indicates that the FE is capable of HA restart recovery. In such a indicates that the FE will run in HA restart recovery. In such a
case, the FE transitions to the Not Associated state and the CEFTI case, the FE transitions to the Not Associated state and the CEFTI
timer[RFC 5810] is started. The FE MAY continue to forward packets timer [RFC5810] is started. The FE MAY continue to forward packets
during this state. It MAY also recycle through any configured backup during this state. The FE recycles through any configured backup CEs
CEs in a round-robin fashion. It first adds its primary CE to the in a round-robin fashion. It first adds its primary CE to the bottom
bottom of table BackupCEs and sets its CEID component to be the first of table BackupCEs and sets its CEID component to be the first
secondary retrieved from table BackupCEs. The FE then attempts to secondary retrieved from table BackupCEs. The FE then attempts to
associate with the CE designated as the new primary CE. If it fails associate with the CE designated as the new primary CE. If it fails
to re-associate with any CE and the CEFTI expires, the FE then to re-associate with any CE and the CEFTI expires, the FE then
transitions to the pre-association state. transitions to the pre-association state and FE will operationally
bring down its forwarding path (and set the [RFC5812] FEObject
FEState component to OperDisable).
If the FE, while in the not associated state, manages to reconnect to If the FE, while in the not associated state, manages to reconnect to
a new primary CE before CEFTI expires it transitions to the a new primary CE before CEFTI expires it transitions to the
Associated state. Once re-associated, the CE tries to synchronize Associated state. Once re-associated, the CE may try to synchronize
any state that the FE may have lost during the not associated state. any state that the FE may have lost during disconnection. How the CE
How the CE re-synchronizes such state is out of scope for the current re-synchronizes such state is out of scope for the current ForCES
ForCES architecture but would typically constitute the issuing of new architecture but would typically constitute the issuing of new
configs and queries. configs and queries.
An explicit message (a Config message setting Primary CE component in An explicit message (a Config message setting Primary CE component in
ForCES Protocol object) from the primary CE, can also be used to ForCES Protocol object) from the primary CE, can also be used to
change the Primary CE for an FE during normal protocol operation. In change the Primary CE for an FE during normal protocol operation. In
this case, the FE transitions to the Not Associated State and this case, the FE transitions to the Not Associated State and
attempts to Associate with the new CE. attempts to Associate with the new CE.
3.1.2. Responsibilities for HA 3.1.2. Responsibilities for HA
skipping to change at page 10, line 15 skipping to change at page 10, line 27
Section 3.1, are the PL's responsibility. Section 3.1, are the PL's responsibility.
To put the two together, if a path to a primary CE is down, the TML To put the two together, if a path to a primary CE is down, the TML
would take care of failing over to a backup path, if one is would take care of failing over to a backup path, if one is
available. If the CE is totally unreachable then the PL would be available. If the CE is totally unreachable then the PL would be
informed and it would take the appropriate actions described before. informed and it would take the appropriate actions described before.
4. CE HA Hot Standby 4. CE HA Hot Standby
In this section we describe small extensions to the existing scheme In this section we describe small extensions to the existing scheme
to enable hot standby HA. To achieve hot standby HA, we target to enable hot standby HA. To achieve hot standby HA, we target to
specific goals defined in Section 2.2, namely: improve the specific goals defined in Section 2.2, namely:
o How fast a backup CE becomes operational. o How fast a backup CE becomes operational.
o How fast the FEs associate with the new master CE. o How fast the FEs associate with the new master CE.
As described in Section 3.1, in the pre-association phase the FEM As described in Section 3.1, in the pre-association phase the FEM
configures the FE to make it aware of all the CEs in the NE. The FEM configures the FE to make it aware of all the CEs in the NE. The FEM
MUST configure the FE to make it aware which CE is the master and MAY MUST configure the FE to make it aware which CE is the master and MAY
specify any backup CE(s). specify any backup CE(s).
4.1. Changes to the FEPO model 4.1. Changes to the FEPO model
In order for the above to be achievable there is a need to make a few In order for the above to be achievable there is a need to make a few
changes in the FEPO model. Section 1 contains the xml definition of changes in the FEPO model. Appendix A contains the xml definition of
the new version 2 of the FEPO LFB. the new version 1.1 of the FEPO LFB.
Changes from the version 1 of FEPO are: Changes from the version 1 of FEPO are:
1. Added four new datatypes: 1. Added four new datatypes:
1. CEStatusType an unsigned char to specify status of a 1. CEStatusType an unsigned char to specify status of a
connection with a CE. Special values are 0 (Disconnected), 1 connection with a CE. Special values are:
(Connected), 2 (Associated), 3 (Lost_Connection) and 4
(Unreachable) + 0 (Disconnected) represents that no connection attempt has
been made with the CE yet
+ 1 (Connected) represents that the FE connection with the
CE at the TML has completed successfully
+ 2 (Associated) represents that the FE has successfully
associated with the CE
+ 3 (IsMaster) represents that the FE has associated with
the CE and is the master of the FE
+ 4 (LostConnection) represents that the FE was associated
with the CE at one point but lost the connection
+ 5 (Unreachable) represents the FE deems this CE
unreachable. i.e the FE has tried over a period to connect
to it but has failed.
2. HAModeValues an unsigned char to specify selected HA mode. 2. HAModeValues an unsigned char to specify selected HA mode.
Special values are 0 (No HA Mode), 1 (HA Mode - Cold Standby) Special values are:
and 2 (HA Mode - Hot Standby)
3. FEHACapab an unsigned char to specify HA capabilities of the + 0 (No HA Mode) The FE is not running in HA mode
FE. Special values are 0 (Graceful Restart), 1 (Cold
Standby) and 2 (Hot Standby)
4. AllCEType a struct of CE ID and CEStatusType to contain + 1 (HA Mode - Cold Standby) The FE is in HA mode cold
connection information for one CE. Used in the AllCEs array. Standby
2. Appended three new components: + and 2 (HA Mode - Hot Standby) The FE is in HA mode hot
Standby
1. AllCEs to hold status for all CEs. AllCEs is an Array of the 3. Statistics, a complex structure, representing the
AllCEType. communication statistics between the FE and CE. The
components are:
2. HAMode to specify current High Availability mode selected. + RecvPackets representing the packet count received from
An unsigned char with three special values 0 (No HA), 1 the CE
(Running Cold-Standby) and 2 (Running Hot-Standby)
3. AcceptBackupGets to provide the master CE to control whether + RecvBytes representing the byte count received from the CE
the FE will accept incoming queries from backup CEs.
3. Added two new capabilities.: + RecvErrPackets representing the erronous packets received
from the CE. This component logs badly formatted packets
as well as good packets sent to the FE by the CE to set
components whilst that CE is not the master. Erronous
packets are dropped(i.e not responded to).
1. HACapabilities, a table that defines which HA capabilities + RecvErrBytes representing the RecvErrPackets byte count
the FE supports. received from the CE
2. MaximumMultipleCEAssocations which defines the maximum + TxmitPackets representing the packet count transmitted to
associations with CEs this FE can have. the CE
4. Added one additional Event, the HAPrimaryCEDown event which + TxmitErrPackets representing the error packet count
reports last known CEID and tentative new master CEID. transmitted to the CE. Typically these would be failures
due to communication.
Since no component from the FEPO v1 has been changed FEPO v2 retains + TxmitBytes representing the byte count transmitted to the
backwards compatibility with CEs that know only version 1.0. These CE
CEs however cannot make use of the High Availability options that the
new FEPO provides. + TxmitErrBytes representing the byte count of errors from
transmit to the CE
4. AllCEType, a complex structure constituing the CE ID,
Statistics and CEStatusType to reflect connection information
for one CE. Used in the AllCEs component array.
2. Appended two new components:
1. Read-only AllCEs to hold status for all CEs. AllCEs is an
Array of the AllCEType.
2. Read-write HAMode of type HAModeValues to carry the HA mode
used by the FE.
3. Added one additional Event, PrimaryCEChanged, reporting the new
master CEID when there is a mastership change.
Since no component from the FEPO v1 has been changed FEPO v1.1
retains backwards compatibility with CEs that know only version 1.0.
These CEs however cannot make use of the High Availability options
that the new FEPO provides.
4.2. FEPO processing 4.2. FEPO processing
The FE's FEPO LFB version 2 AllCEs table contains all the CEIDs that The FE's FEPO LFB version 1.1 AllCEs table contains all the CEIDs
the FE may connect and associate with. The ordering of the CE IDs in that the FE may connect and associate with. The ordering of the CE
this table defines the priority order in which an FE will connect to IDs in this table defines the priority order in which an FE will
the CEs. In the pre-association phase, the first CE ID (lowest table connect to the CEs. This table is provisioned initially from the
index) in the AllCEs table MUST be the first CE ID that the FE will configuration plane (FEM). In the pre-association phase, the first
attempt to connect and associate with. If the FE fails to connect CE (lowest table index) in the AllCEs table MUST be the first CE that
and associate with the first CE ID, it will attempt to connect to the the FE will attempt to connect and associate with. If the FE fails
second CE ID and so forth, and cycles back to the beggining of the to connect and associate with the first listed CE, it will attempt to
list until there is a connection and an association. The FE MUST connect to the second CE and so forth, and cycles back to the
associate with at least one CE. Upon a successful association, the beggining of the list until there is a successful association. The
FEPO's CEID component identifies the current associated master CE. FE MUST associate with at least one CE. Upon a successful
association, the FEPO's CEID component identifies the current
associated master CE.
While it would be much simpler to have the FE not respond to any While it would be much simpler to have the FE not respond to any
messages from CE other than the master, it may be useful for the messages from a CE other than the master, in practise it has been
backup CEs to be able to query the FE. Query commands are sent found to be useful to respond to queries and hearbeats from backup
always on the high priority channel. In order to avoid missing CEs. For this reason, we allow backup CEs to issues queries to the
critical configuration or query commands from the master CE, all FE. Configuration messages (SET/DEL) from backup CEs MUST be dropped
query commands from backup CEs MUST be sent on the high priority by the FE and logged as received errors.
channel but with the least priority, the value of which is 4.
However since queries are high priority from heartbeats, if the
master CE waits for heartbeat responses and the backup CEs flood the
FE, the master CE may think that the FE is down. Therefore it is
prudent to add a control mechanism that will be able to control
whether the FE can respond to query messages from backup CEs. The
AcceptBackupGets component, a boolean, is designed for this occasion.
If the master CE sets it to true, the FE MUST accept and process
query commands from backup CEs. If the AcceptBackupGets is false,
the FE MUST drop query commands from backup CEs.
Asynchronous events that the master CE has subscribed to, as well as Asynchronous events that the master CE has subscribed to, as well as
heartbeats are sent to all associated-to CEs. Packet redirects heartbeats are sent to all associated-to CEs. Packet redirects
continue to be sent only to the master CE. The Heartbeat Interval, continue to be sent only to the master CE. The Heartbeat Interval,
the CEHB Policy and the FEHB Policy MUST be the same for all CEs. the CEHB Policy and the FEHB Policy are global for all CEs(and
changed only by the master CE).
Figure 4 illustrates the state machine that facilitates connection Figure 4 illustrates the state machine that facilitates connection
recovery with High Availability enabled. recovery with High Availability enabled.
FE tries to associate FE tries to associate
+-->-----+ +-->-----+
| | | |
^ v (CE changes master || | |
(CE issues Teardown || +----+--------+---+ CE issues Teardown || +---+--------v----+
Lost association) && | Pre-Association | Lost association) && | Pre-Association |
CE failover policy = 0 | (Association +<-------------------+ CE failover policy = 0 | (Association |
+------------>-->-->| in +<-----+ | +------------>-->-->| in +<----+
| | progress) | | | | | progress) | |
| CE Issues +--------+--------+ | | | | | |
| Association | | | | +--------+--------+ |
| Response V Not Found || CEFTI | | CE Association | | CEFTI
| +------------------+ timer expires | | Response V | timer
| | | | | +------------------+ | expires
| V ^ | | |FE issue CEPrimaryDown ^
+-+-----------+ +------+------+ | | |FE issue PrimaryCEChanged ^
| | (CE issues Teardown || | Not | | | V |
| | Lost association) && | Associated | | +-+-----------+ +------+-----+
| | (CE Failover Policy=1) | | CEFTI | | (CE changes master || | Not |
| Associated | | (May | timer | | CE issues Teardown || | Associated |
| | | Continue | expires | | Lost association) && | +->-----------+
| +---------->------->----->| Forwarding)| | | Associated | CE Failover Policy = 1 |(May |find first |
| | Start CEFTI Timer | | | | | | Continue |associated v
| | | Search for | | | |-------->------->------>| Forwarding)|CE or retry |
| | +--------->| next | | | | Start CEFTI timer | |associating |
| | | | associated | | | | | |-<-----------+
| | | | CE | | | | | |
| | | | (HAMode 2) | | +----+--------+ +-------+----+
+-------------+ | +-------------+ | | |
^ | V | ^ Found | associated CE
| | | | | or newly | associated CE
| | Found CE | | V
| CEHDI Expires Send Event of | | (Cancel CEFTI Timer) |
| | New CE ID. | +_________________________________________+
| | Start CEHDI Timer | FE issue CEPrimaryDown event
| | | | FE issue PrimaryCEChanged event
| | V |
| | +------+------+ |
| ^---------+ Confirm +-------^
| | State |
| Received +---->| |
| different | | Wait for CE |
| CE ID. ^ | to confirm |
| Resend Event | | new CE ID |
| Restart CEHDI Timer +----<| |
| +-----+-------+
| Received same CE ID |
| (Cancel CEFTI & CEHDI Timer) |
+_______________________________________+
Figure 4: FE State Machine considering HA Figure 4: FE State Machine considering HA
Once the FE has associated with a master CE it moves to the post- Once the FE has associated with a master CE it moves to the post-
association phase (Associated state). It MAY also instruct the FE to association phase (Associated state). It is assumed that the master
use a different master CE. It is assumed that the master CE will CE will communicate with other CEs within the NE for the purpose of
communicate with other CEs within the NE for the purpose of
synchronization via the CE-CE interface. The CE-CE interface is out synchronization via the CE-CE interface. The CE-CE interface is out
of scope for this document. of scope for this document. An election result amongst CEs may
result in desire to change mastership to a different associated CE;
at which point current assumed master CE will instruct the FE to use
a different master CE.
FE CE#1 CE#2 ... CE#N FE CE#1 CE#2 ... CE#N
| | | | | | | |
| Asso Estb,Caps exchg | | | | Asso Estb,Caps exchg | | |
1 |<-------------------->| | | 1 |<-------------------->| | |
| | | | | | | |
| state update | | | | state update | | |
2 |<-------------------->| | | 2 |<-------------------->| | |
| | | | | | | |
| Asso Estb,Caps exchg | | | Asso Estb,Caps exchg | |
3I|<--------------------------------->| | 3I|<--------------------------------->| |
... ... ... ... ... ... ... ...
| Asso Estb,Caps exchg | | Asso Estb,Caps exchg |
3N|<------------------------------------------>| 3N|<------------------------------------------>|
| | | | | | | |
4 |<-------------------->| | | 4 |<-------------------->| | |
. . . . . . . .
4x|<-------------------->| | | 4x|<-------------------->| | |
| FAILURE | | | FAILURE | |
| | | | | | | |
| Event Report (CE#2 is new master) | | | Event Report (LastCEID changed) | |
5 |---------------------------------->|------->| 5 |---------------------------------->|------->|
| Event Report (CE#2 is new master) | |
6 |---------------------------------->|------->|
| | | | | |
| Config (Set CEID to CEID of CE#2) | |
6 |<----------------------------------| |
7 |<--------------------------------->| | 7 |<--------------------------------->| |
. . . . . . . .
7x|<--------------------------------->| | 7x|<--------------------------------->| |
. . . . . . . .
Figure 5: CE Failover for Hot Standby Figure 5: CE Failover for Hot Standby
While in the post-association phase, if the CE Failover Policy is set While in the post-association phase, if the CE Failover Policy is set
to 1 and HAMode set to 2 (HotStandby) then the FE, after succesfully to 1 and HAMode set to 2 (HotStandby) then the FE, after succesfully
associating with the master CE, MUST attempt to connect and associate associating with the master CE, MUST attempt to connect and associate
with all the CEs that is aware of. Figure 5 steps #1 and #2 with all the CEs that it is aware of. Figure 5 steps #1 and #2
illustrates the FE associating with CE#1 as the master and then illustrates the FE associating with CE#1 as the master and then
proceeding to steps #3I to #3N the association with backup CE's CE#2 proceeding to steps #3I to #3N the association with backup CEs CE#2
to CE#N. If the FE fails to connect or associate with some CEs, the to CE#N. If the FE fails to connect or associate with some CEs, the
FE MAY flag them as unreachable to avoid continuous attempts to FE MAY flag them as unreachable to avoid continuous attempts to
connect. The FE MAY retry to reassociate with unreachable CEs when connect. The FE MAY retry to reassociate with unreachable CEs when
possible. possible.
When the master CE for any reason is considered to be down, then the When the master CE for any reason is considered to be down, then the
FE will try to find the first associated CE from the list of all CEs FE MUST try to find the first associated CE from the list of all CEs
in a round-robin fashion. in a round-robin fashion.
If the FE is unable to find an associated FE in its list of CEs, then If the FE is unable to find an associated FE in its list of CEs, then
it will attempt to connect and associate with the first from the list it MUST attempt to connect and associate with the first from the list
of all CEs and continue in a round-robin fashion until it connects of all CEs and continue in a round-robin fashion until it connects
and associates with a CE. and associates with a CE.
Once the FE selects the associated CE to use as the new master, the Once the FE selects an associated CE to use as the new master, the FE
FE then sends a High Availability Primary CE Changed Event issues a PrimaryCEDown Event Notification to all associated CEs to
Notification to all associated CEs to notifying them that the primary notify them that the last primary CE went down (and what its identity
CE is down as well as which CE the reporting FE considers to be the was); a second event PrimaryCEChanged identifying the new master CE
new master. is sent as well to identify which CE the reporting FE considers to be
the new master.
The new master CE MUST configure the CEID component of the FE within
the time limit defined in the CEHDI Failover Timeout as a
confirmation that the FE made the right choice.
FE CE#1 CE#2 ... CE#N
| | | |
| Asso Estb,Caps exchg | | |
1 |<-------------------->| | |
| | | |
| state update | | |
2 |<-------------------->| | |
| | | |
| Asso Estb,Caps exchg | |
3I|<--------------------------------->| |
| | | |
... ... ... ...
| Asso Estb,Caps exchg |
3N|<------------------------------------------>|
| | | |
4 |<-------------------->| | |
. . . .
4x|<-------------------->| | |
| FAILURE | |
| | | |
| Event Report (CE#2 is new master) | |
5 |---------------------------------->|------->|
| | | |
| CEHDI Failover Timeout | |
| | | |
| Event Report (CE#N is new master) | |
6 |---------------------------------->|------->|
| | | |
| Config (Set CEID to CEID of CE#N) |
7 |<-------------------------------------------|
8a|<------------------------------------------>|
. . . .
8x|<------------------------------------------>|
Figure 6: CE Failover for Hot Standby
If the FE does not get confirmation within the CEHDI Failover
Timeout, it picks the next CE on its list and advertises it as the
new master. Figure 6 illustrates in step #5 selecting CE#2 as its
new master. In step #6, the timeout occurs and it picks CE#N as its
new master. The FE receives confirmation that CE#N is the new master
in step #7.
If the CE the FE assumed to be the master discovers that it should
not be the new master CE, then it will configure the CEID with the ID
of the proper master CE. How the CE decides who the new master CE
is, is also out of scope of this document and is assumed to be done
via a CE-CE communication protocol. The FE must then associate with
then new CE.
If the CEFTI timer expires at either the not-associated or confirm
states without a new master CE confirmed, then the FE MUST revert to
the pre-association stage.
In most High Availability architectures there exists the possibility In most High Availability architectures there exists the possibility
of split-brain. However, since in our setup the FE will never accept of split-brain. However, since in our setup the FE will never accept
any configuration messages from any other than the master CE, we any configuration messages from any other than the master CE, we
consider the FE as fenced against data corruption from the other CEs consider the FE as fenced against data corruption from the other CEs
that consider themselves as the master. The split-brain issue that consider themselves as the master. The split-brain issue
becomes mostly a CE-CE communication problem which is considered to becomes mostly a CE-CE communication problem which is considered to
be out of scope. be out of scope.
By virtue of having multiple CE connections, the FE switchover to a By virtue of having multiple CE connections, the FE switchover to a
new master CE will be relatively much faster. The overall effect is new master CE will be relatively much faster. The overall effect is
improving the NE recovery time in case of communication failure or improving the NE recovery time in case of communication failure or
faults of the master CE. This satisfies the requirement we set to faults of the master CE. This satisfies the requirement we set to
achieve. achieve.
5. IANA Considerations 5. IANA Considerations
TBA XXX: This document updates an IANA registered FE Protocol object
Logical Functional Block (LFB). At minimal when it becomes RFC we
should update https://www.iana.org/assignments/forces/forces.xml
section on FEPO.
6. Security Considerations 6. Security Considerations
TBA Security consideration as defined in section 9 of [RFC5810] applies.
7. References 7. References
7.1. Normative References 7.1. Normative References
[RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang,
W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and
Control Element Separation (ForCES) Protocol Control Element Separation (ForCES) Protocol
Specification", RFC 5810, March 2010. Specification", RFC 5810, March 2010.
7.2. Informative References 7.2. Informative References
[RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation
skipping to change at page 17, line 24 skipping to change at page 17, line 18
of IP Control and Forwarding", RFC 3654, November 2003. of IP Control and Forwarding", RFC 3654, November 2003.
[RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal,
"Forwarding and Control Element Separation (ForCES) "Forwarding and Control Element Separation (ForCES)
Framework", RFC 3746, April 2004. Framework", RFC 3746, April 2004.
[RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control
Element Separation (ForCES) Forwarding Element Model", Element Separation (ForCES) Forwarding Element Model",
RFC 5812, March 2010. RFC 5812, March 2010.
1. Appendix I - New FEPO version Appendix A. New FEPO version
<LFBLibrary xmlns="urn:ietf:params:xml:ns:forces:lfbmodel:1.0" <LFBLibrary xmlns="urn:ietf:params:xml:ns:forces:lfbmodel:1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" provides="FEPO"> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="lfb-schema.xsd" provides="FEPO">
<!-- XXX -->
<dataTypeDefs> <dataTypeDefs>
<dataTypeDef> <dataTypeDef>
<name>CEHBPolicyValues</name>
<synopsis>
The possible values of CE heartbeat policy
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>CEHBPolicy0</name>
<synopsis>
The CE will send heartbeats to the FE
every CEHDI timeout if no other messages
have been sent since.
</synopsis>
</specialValue>
<specialValue value="1">
<name>CEHBPolicy1</name>
<synopsis>
The CE will not send heartbeats to the FE
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>FEHBPolicyValues</name> <name>FEHBPolicyValues</name>
<synopsis> <synopsis>
The possible values of FE heartbeat policy The possible values of FE heartbeat policy
</synopsis> </synopsis>
<atomic> <atomic>
<baseType>uchar</baseType> <baseType>uchar</baseType>
<specialValues> <specialValues>
<specialValue value="0"> <specialValue value="0">
<name>FEHBPolicy0</name> <name>FEHBPolicy0</name>
<synopsis> <synopsis>
The FE heartbeat policy 0 The FE will not generate any heartbeats to the CE
</synopsis> </synopsis>
</specialValue> </specialValue>
<specialValue value="1"> <specialValue value="1">
<name>FEHBPolicy1</name> <name>FEHBPolicy1</name>
<synopsis> <synopsis>
The FE heartbeat policy 1 The FE generates heartbeats to the CE every FEHI
</synopsis> if no other messages have been sent to the CE.
</synopsis>
</specialValue> </specialValue>
</specialValues> </specialValues>
</atomic> </atomic>
</dataTypeDef>
<dataTypeDef>
<name>HAModeValues</name>
<synopsis>
The possible values of HA modes
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>NoHA</name>
<synopsis>
The FE is not running in HA mode
</synopsis>
</specialValue>
<specialValue value="1">
<name>ColdStandby</name>
<synopsis>
The FE is running in HA mode cold Standby
</synopsis>
</specialValue>
<specialValue value="2">
<name>HotStandby</name>
<synopsis>
The FE is running in HA mode hot Standby
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef> </dataTypeDef>
<dataTypeDef> <dataTypeDef>
<name>FERestartPolicyValues</name> <name>FERestartPolicyValues</name>
<synopsis> <synopsis>
The possible values of FE restart policy The possible values of FE restart policy
</synopsis> </synopsis>
<atomic> <atomic>
<baseType>uchar</baseType> <baseType>uchar</baseType>
<specialValues> <specialValues>
<specialValue value="0"> <specialValue value="0">
<name>FERestartPolicy0</name> <name>FERestartPolicy0</name>
<synopsis> <synopsis>
The FE restart policy 0 The FE restart restats its state from scratch
</synopsis> </synopsis>
</specialValue> </specialValue>
</specialValues> </specialValues>
</atomic> </atomic>
</dataTypeDef> </dataTypeDef>
<dataTypeDef> <dataTypeDef>
<name>CEHBPolicyValues</name> <name>HAModeValues</name>
<synopsis>The possible values of CE heartbeat policy</synopsis> <synopsis>
<atomic> The possible values of HA modes
<baseType>uchar</baseType> </synopsis>
<specialValues> <atomic>
<specialValue value="0"> <baseType>uchar</baseType>
<name>CEHBPolicy0</name> <specialValues>
<synopsis>The CE heartbeat policy 0</synopsis> <specialValue value="0">
</specialValue> <name>NoHA</name>
<specialValue value="1"> <synopsis>
<name>CEHBPolicy1</name> The FE is not running in HA mode
<synopsis>The CE heartbeat policy 1</synopsis> </synopsis>
</specialValue> </specialValue>
</specialValues> <specialValue value="1">
</atomic> <name>ColdStandby</name>
<synopsis>
The FE is running in HA mode cold Standby
</synopsis>
</specialValue>
<specialValue value="2">
<name>HotStandby</name>
<synopsis>
The FE is running in HA mode hot Standby
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef> </dataTypeDef>
<dataTypeDef> <dataTypeDef>
<name>FEHACapab</name> <name>CEFailoverPolicyValues</name>
<synopsis> <synopsis>
The possible values of CE failover policy
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>CEFailoverPolicy0</name>
<synopsis>
The FE should stop functioning immediate and
transition to the FE OperDisable state
</synopsis>
</specialValue>
<specialValue value="1">
<name>CEFailoverPolicy1</name>
<synopsis>
The FE should continue forwarding even without an
associated CE for CEFTI. The FE goes to FE
OperDisable when the CEFTI expires and no
association. Requires graceful restart support.
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>FEHACapab</name>
<synopsis>
The supported HA features The supported HA features
</synopsis> </synopsis>
<atomic> <atomic>
<baseType>uchar</baseType> <baseType>uchar</baseType>
<specialValues> <specialValues>
<specialValue value="0"> <specialValue value="0">
<name>GracefullRestart</name> <name>GracefullRestart</name>
<synopsis> <synopsis>
The FE supports Graceful Restart The FE supports Graceful Restart
</synopsis> </synopsis>
</specialValue> </specialValue>
<specialValue value="1"> <specialValue value="1">
<name>HA</name> <name>HA</name>
<synopsis> <synopsis>
The FE supports cold-standby mode The FE supports HA
</synopsis> </synopsis>
</specialValue> </specialValue>
<specialValue value="2"> </specialValues>
<name>HOtStandBy</name> </atomic>
<synopsis>
The FE supports hot-standby mode
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef> </dataTypeDef>
<dataTypeDef> <dataTypeDef>
<name>CEStatusType</name> <name>CEStatusType</name>
<synopsis> <synopsis>Status values. Status for each CE</synopsis>
Status values. Status for each CE.
</synopsis>
<atomic> <atomic>
<baseType>uchar</baseType> <baseType>uchar</baseType>
<specialValues> <specialValues>
<specialValue value="0"> <specialValue value="0">
<name>Disconnected</name> <name>Disconnected</name>
<synopsis> <synopsis>No connection attempt with the CE yet
No connection attempt with the CE yet. </synopsis>
</synopsis>
</specialValue> </specialValue>
<specialValue value="1"> <specialValue value="1">
<name>Connected</name> <name>Connected</name>
<synopsis> <synopsis>The FE connection with the CE at the TML
The FE has connected with the CE. has been completed
</synopsis> </synopsis>
</specialValue> </specialValue>
<specialValue value="2"> <specialValue value="2">
<name>Associated</name> <name>Associated</name>
<synopsis> <synopsis>The FE has associated with the CE
The FE has associated with the CE. </synopsis>
</synopsis>
</specialValue> </specialValue>
<specialValue value="3"> <specialValue value="3">
<name>Lost_Connection</name> <name>IsMaster</name>
<synopsis> <synopsis>The CE is the master (and associated)
The FE was associated with the CE </synopsis>
but lost the connection.
</synopsis>
</specialValue> </specialValue>
<specialValue value="4"> <specialValue value="4">
<name>LostConnection</name>
<synopsis>The FE was associated with the CE but
lost the connection
</synopsis>
</specialValue>
<specialValue value="5">
<name>Unreachable</name> <name>Unreachable</name>
<synopsis> <synopsis>The CE is deemed as unreachable by the FE
The CE is deemed as unreachable by the FE. </synopsis>
</synopsis>
</specialValue> </specialValue>
</specialValues> </specialValues>
</atomic> </atomic>
</dataTypeDef> </dataTypeDef>
<dataTypeDef> <dataTypeDef>
<name>StatisticsType</name>
<synopsis>Statistics Definition</synopsis>
<struct>
<component componentID="1">
<name>RecvPackets</name>
<synopsis>Packets Received</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="2">
<name>RecvErrPackets</name>
<synopsis>Packets Received from CE with errors
</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="3">
<name>RecvBytes</name>
<synopsis>Bytes Received from CE</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="4">
<name>RecvErrBytes</name>
<synopsis>Bytes Received from CE in Error</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="5">
<name>TxmitPackets</name>
<synopsis>Packets Transmitted to CE</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="6">
<name>TxmitErrPackets</name>
<synopsis>
Packets Transmitted to CE that incurred
errors
</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="7">
<name>TxmitBytes</name>
<synopsis>Bytes Transmitted to CE</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="8">
<name>TxmitErrBytes</name>
<synopsis>Bytes Transmitted to CE incurring errors
</synopsis>
<typeRef>uint64</typeRef>
</component>
</struct>
</dataTypeDef>
<dataTypeDef>
<name>AllCEType</name> <name>AllCEType</name>
<synopsis> <synopsis>Table Type for AllCE component</synopsis>
Table Type for AllCE component.
</synopsis>
<struct> <struct>
<component componentID="1"> <component componentID="1">
<name>CEID</name> <name>CEID</name>
<synopsis>ID of the CE</synopsis> <synopsis>ID of the CE</synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="2"> <component componentID="2">
<name>Statistics</name>
<synopsis>Statistics per CE</synopsis>
<typeRef>StatisticsType</typeRef>
</component>
<component componentID="3">
<name>CEStatus</name> <name>CEStatus</name>
<synopsis>Status of the CE</synopsis> <synopsis>Status of the CE</synopsis>
<typeRef>CEStatusType</typeRef> <typeRef>CEStatusType</typeRef>
</component> </component>
</struct> </struct>
</dataTypeDef> </dataTypeDef>
</dataTypeDefs> </dataTypeDefs>
<LFBClassDefs> <LFBClassDefs>
<LFBClassDef LFBClassID="2"> <LFBClassDef LFBClassID="2">
<name>FEPO</name> <name>FEPO</name>
<synopsis> <synopsis>
The FE Protocol Object The FE Protocol Object, with new CEHA
</synopsis> </synopsis>
<version>2.0</version> <version>1.1</version>
<components> <components>
<component componentID="1" access="read-only"> <component componentID="1" access="read-only">
<name>CurrentRunningVersion</name> <name>CurrentRunningVersion</name>
<synopsis>Currently running ForCES version</synopsis> <synopsis>Currently running ForCES version</synopsis>
<typeRef>u8</typeRef> <typeRef>uchar</typeRef>
</component> </component>
<component componentID="2" access="read-only"> <component componentID="2" access="read-only">
<name>FEID</name> <name>FEID</name>
<synopsis>Unicast FEID</synopsis> <synopsis>Unicast FEID</synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="3" access="read-write"> <component componentID="3" access="read-write">
<name>MulticastFEIDs</name> <name>MulticastFEIDs</name>
<synopsis> <synopsis>
the table of all multicast IDs the table of all multicast IDs
</synopsis> </synopsis>
<array type="variable-size"> <array type="variable-size">
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</array> </array>
</component> </component>
<component componentID="4" access="read-write"> <component componentID="4" access="read-write">
<name>CEHBPolicy</name> <name>CEHBPolicy</name>
<synopsis> <synopsis>
The CE Heartbeat Policy The CE Heartbeat Policy
</synopsis> </synopsis>
<typeRef>CEHBPolicyValues</typeRef> <typeRef>CEHBPolicyValues</typeRef>
</component> </component>
<component componentID="5" access="read-write"> <component componentID="5" access="read-write">
<name>CEHDI</name> <name>CEHDI</name>
<synopsis> <synopsis>
The CE Heartbeat Dead Interval in millisecs The CE Heartbeat Dead Interval in millisecs
</synopsis>
</synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="6" access="read-write"> <component componentID="6" access="read-write">
<name>FEHBPolicy</name> <name>FEHBPolicy</name>
<synopsis> <synopsis>
The FE Heartbeat Policy The FE Heartbeat Policy
</synopsis> </synopsis>
<typeRef>FEHBPolicyValues</typeRef> <typeRef>FEHBPolicyValues</typeRef>
</component> </component>
<component componentID="7" access="read-write"> <component componentID="7" access="read-write">
<name>FEHI</name> <name>FEHI</name>
<synopsis> <synopsis>
The FE Heartbeat Interval in millisecs The FE Heartbeat Interval in millisecs
</synopsis> </synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="8" access="read-write"> <component componentID="8" access="read-write">
<name>CEID</name> <name>CEID</name>
<synopsis> <synopsis>
The Primary CE this FE is associated with The Primary CE this FE is associated with
</synopsis> </synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="9" access="read-write"> <component componentID="9" access="read-write">
<name>BackupCEs</name> <name>BackupCEs</name>
<synopsis> <synopsis>
The table of all backup CEs other than the primary The table of all backup CEs other than the
</synopsis> primary
<array type="variable-size"> </synopsis>
<typeRef>uint32</typeRef> <array type="variable-size">
</array> <typeRef>uint32</typeRef>
</component> </array>
</component>
<component componentID="10" access="read-write"> <component componentID="10" access="read-write">
<name>CEFailoverPolicy</name> <name>CEFailoverPolicy</name>
<synopsis> <synopsis>
The CE Failover Policy The CE Failover Policy
</synopsis> </synopsis>
<typeRef>CEFailoverPolicyValues</typeRef> <typeRef>CEFailoverPolicyValues</typeRef>
</component> </component>
<component componentID="11" access="read-write"> <component componentID="11" access="read-write">
<name>CEFTI</name> <name>CEFTI</name>
<synopsis> <synopsis>
The CE Failover Timeout Interval in millisecs The CE Failover Timeout Interval in millisecs
</synopsis> </synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="12" access="read-write"> <component componentID="12" access="read-write">
<name>FERestartPolicy</name> <name>FERestartPolicy</name>
<synopsis> <synopsis>
The FE Restart Policy The FE Restart Policy
</synopsis> </synopsis>
<typeRef>FERestartPolicyValues</typeRef> <typeRef>FERestartPolicyValues</typeRef>
</component> </component>
<component componentID="13" access="read-write"> <component componentID="13" access="read-write">
<name>LastCEID</name> <name>LastCEID</name>
<synopsis> <synopsis>
The Primary CE this FE was last associated with The Primary CE this FE was last associated
</synopsis> with
</synopsis>
<typeRef>uint32</typeRef> <typeRef>uint32</typeRef>
</component> </component>
<component componentID="14" access="read-only"> <component componentID="14" access="read-write">
<name>AllCEs</name> <name>HAMode</name>
<synopsis> <synopsis>
The table of all CEs. The HA mode used
</synopsis> </synopsis>
<typeRef>HAModeValues</typeRef>
</component>
<component componentID="15" access="read-only">
<name>AllCEs</name>
<synopsis>The table of all CEs</synopsis>
<array type="variable-size"> <array type="variable-size">
<typeRef>AllCEType</typeRef> <typeRef>AllCEType</typeRef>
</array> </array>
</component> </component>
<component componentID="15" access="read-write"> </components>
<name>HAMode</name>
<synopsis>
Mode selection for action in HA after loss of master CE
</synopsis>
<typeRef>HAModeValues</typeRef>
</component>
<component componentID="16" access="read-write">
<name>AcceptBackupGets</name>
<synopsis>If true, the FE will accept and respond to Queries
from BackupCEs.</synopsis>
<typeRef>Boolean</typeRef>
</component>
</components>
<capabilities> <capabilities>
<capability componentID="30"> <capability componentID="30">
<name>SupportableVersions</name> <name>SupportableVersions</name>
<synopsis> <synopsis>
the table of ForCES versions that FE supports the table of ForCES versions that FE supports
</synopsis> </synopsis>
<array type="variable-size"> <array type="variable-size">
<typeRef>uchar</typeRef> <typeRef>uchar</typeRef>
</array> </array>
</capability> </capability>
<capability componentID="31"> <capability componentID="31">
<name>HACapabilities</name> <name>HACapabilities</name>
<synopsis> <synopsis>
the table of HA capabilities the FE supports the table of HA capabilities the FE supports
</synopsis> </synopsis>
<array type="variable-size"> <array type="variable-size">
<typeRef>FEHACapab</typeRef> <typeRef>FEHACapab</typeRef>
</array> </array>
</capability> </capability>
<capability componentID="32">
<name>MaximumMultipleCEAssocations</name>
<synopsis>
The number of CEs this FE can associate with at the same
time
</synopsis>
<atomic>
<baseType>uint32</baseType>
</atomic>
</capability>
</capabilities> </capabilities>
<events baseID="61"> <events baseID="61">
<event eventID="1"> <event eventID="1">
<name>PrimaryCEDown</name> <name>PrimaryCEDown</name>
<synopsis> <synopsis>
The pimary CE has changed The primary CE has changed
</synopsis> </synopsis>
<eventTarget> <eventTarget>
<eventField>LastCEID</eventField> <eventField>LastCEID</eventField>
</eventTarget> </eventTarget>
<eventChanged/> <eventChanged/>
<eventReports> <eventReports>
<eventReport> <eventReport>
<eventField>LastCEID</eventField> <eventField>LastCEID</eventField>
</eventReport> </eventReport>
</eventReports> </eventReports>
</event> </event>
<event eventID="2"> <event eventID="2">
<name>HAPrimaryCEDown</name> <name>PrimaryCEChanged</name>
<synopsis>The primary CE has changed</synopsis> <synopsis>A New primary CE has been selected
</synopsis>
<eventTarget> <eventTarget>
<eventField>LastCEID</eventField> <eventField>CEID</eventField>
</eventTarget> </eventTarget>
<eventChanged/> <eventChanged/>
<eventReports> <eventReports>
<eventReport> <eventReport>
<eventField>CEID</eventField> <eventField>CEID</eventField>
<eventField>LastCEID</eventField>
</eventReport> </eventReport>
</eventReports> </eventReports>
</event> </event>
</events> </events>
</LFBClassDef> </LFBClassDef>
</LFBClassDefs> </LFBClassDefs>
</LFBLibrary> </LFBLibrary>
Authors' Addresses Authors' Addresses
Kentaro Ogawa Kentaro Ogawa
NTT Corporation NTT Corporation
3-9-11 Midori-cho 3-9-11 Midori-cho
skipping to change at page 25, line 28 skipping to change at page 27, line 4
Email: ogawa.kentaro@lab.ntt.co.jp Email: ogawa.kentaro@lab.ntt.co.jp
Weiming Wang Weiming Wang
Zhejiang Gongshang University Zhejiang Gongshang University
149 Jiaogong Road 149 Jiaogong Road
Hangzhou 310035 Hangzhou 310035
P.R.China P.R.China
Phone: +86-571-88057712 Phone: +86-571-88057712
Email: wmwang@mail.zjgsu.edu.cn Email: wmwang@mail.zjgsu.edu.cn
Evangelos Haleplidis Evangelos Haleplidis
University of Patras University of Patras
Patras Panepistimioupoli Patron
Patras 26504
Greece Greece
Email: ehalep@ece.upatras.gr Email: ehalep@ece.upatras.gr
Jamal Hadi Salim Jamal Hadi Salim
Mojatatu Networks Mojatatu Networks
Ottawa, Ontario Suite 400, 303 Moodie Dr.
Ottawa, Ontario K2H 9R4
Canada Canada
Email: hadi@mojatatu.com Email: hadi@mojatatu.com
 End of changes. 114 change blocks. 
464 lines changed or deleted 518 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/