draft-ietf-teas-native-ip-scenarios-12.txt   rfc8735.txt 
TEAS Working Group A. Wang Internet Engineering Task Force (IETF) A. Wang
Internet-Draft China Telecom Request for Comments: 8735 China Telecom
Intended status: Informational X. Huang Category: Informational X. Huang
Expires: May 1, 2020 C. Kou ISSN: 2070-1721 C. Kou
BUPT BUPT
Z. Li Z. Li
China Mobile China Mobile
P. Mi P. Mi
Huawei Technologies Huawei Technologies
October 29, 2019 February 2020
Scenarios and Simulation Results of PCE in Native IP Network Scenarios and Simulation Results of PCE in a Native IP Network
draft-ietf-teas-native-ip-scenarios-12
Abstract Abstract
Requirements for providing the End to End(E2E) performance assurance Requirements for providing the End-to-End (E2E) performance assurance
are emerging within the service provider networks. While there are are emerging within the service provider networks. While there are
various technology solutions, there is no single solution that can various technology solutions, there is no single solution that can
fulfill these requirements for a native IP network. In particular, fulfill these requirements for a native IP network. In particular,
there is a need for a universal (E2E) solution that can cover both there is a need for a universal E2E solution that can cover both
intra- and inter-domain scenarios. intra- and inter-domain scenarios.
One feasible E2E traffic engineering solution is the addition of One feasible E2E traffic-engineering solution is the addition of
central control in a native IP network. This document describes central control in a native IP network. This document describes
various complex scenarios and simulation results when applying the various complex scenarios and simulation results when applying the
Path Computation Element (PCE) in a native IP network. This Path Computation Element (PCE) in a native IP network. This
solution, referred to as Centralized Control Dynamic Routing (CCDR), solution, referred to as Centralized Control Dynamic Routing (CCDR),
integrates the advantage of using distributed protocols and the power integrates the advantage of using distributed protocols and the power
of a centralized control technology, providing traffic engineering of a centralized control technology, providing traffic engineering
for native IP networks in a manner that applies equally to intra- and for native IP networks in a manner that applies equally to intra- and
inter-domain scenarios. inter-domain scenarios.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering This document is a product of the Internet Engineering Task Force
Task Force (IETF). Note that other groups may also distribute (IETF). It represents the consensus of the IETF community. It has
working documents as Internet-Drafts. The list of current Internet- received public review and has been approved for publication by the
Drafts is at https://datatracker.ietf.org/drafts/current/. Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are candidates for any level of Internet
Standard; see Section 2 of RFC 7841.
Internet-Drafts are draft documents valid for a maximum of six months Information about the current status of this document, any errata,
and may be updated, replaced, or obsoleted by other documents at any and how to provide feedback on it may be obtained at
time. It is inappropriate to use Internet-Drafts as reference https://www.rfc-editor.org/info/rfc8735.
material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 1, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology
3. CCDR Scenarios . . . . . . . . . . . . . . . . . . . . . . . 4 3. CCDR Scenarios
3.1. QoS Assurance for Hybrid Cloud-based Application . . . . 4 3.1. QoS Assurance for Hybrid Cloud-Based Application
3.2. Link Utilization Maximization . . . . . . . . . . . . . . 5 3.2. Link Utilization Maximization
3.3. Traffic Engineering for Multi-Domain . . . . . . . . . . 6 3.3. Traffic Engineering for Multi-domain
3.4. Network Temporal Congestion Elimination . . . . . . . . . 7 3.4. Network Temporal Congestion Elimination
4. CCDR Simulation . . . . . . . . . . . . . . . . . . . . . . . 7 4. CCDR Simulation
4.1. Case Study for CCDR Algorithm . . . . . . . . . . . . . . 8 4.1. Case Study for CCDR Algorithm
4.2. Topology Simulation . . . . . . . . . . . . . . . . . . . 9 4.2. Topology Simulation
4.3. Traffic Matrix Simulation . . . . . . . . . . . . . . . . 10 4.3. Traffic Matrix Simulation
4.4. CCDR End-to-End Path Optimization . . . . . . . . . . . . 10 4.4. CCDR End-to-End Path Optimization
4.5. Network Temporal Congestion Elimination . . . . . . . . . 12 4.5. Network Temporal Congestion Elimination
5. CCDR Deployment Consideration . . . . . . . . . . . . . . . . 14 5. CCDR Deployment Consideration
6. Security Considerations . . . . . . . . . . . . . . . . . . . 14 6. Security Considerations
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 7. IANA Considerations
8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 15 8. References
9. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 15 8.1. Normative References
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 15 8.2. Informative References
10.1. Normative References . . . . . . . . . . . . . . . . . . 15 Acknowledgements
10.2. Informative References . . . . . . . . . . . . . . . . . 16 Contributors
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16 Authors' Addresses
1. Introduction 1. Introduction
A service provider network is composed of thousands of routers that A service provider network is composed of thousands of routers that
run distributed protocols to exchange the reachability information. run distributed protocols to exchange reachability information. The
The path for the destination network is mainly calculated, and path for the destination network is mainly calculated, and
controlled, by the distributed protocols. These distributed controlled, by the distributed protocols. These distributed
protocols are robust enough to support most applications, however, protocols are robust enough to support most applications; however,
they have some difficulties supporting the complexities needed for they have some difficulties supporting the complexities needed for
traffic engineering applications, e.g. E2E performance assurance, or traffic-engineering applications, e.g., E2E performance assurance, or
maximizing the link utilization within an IP network. maximizing the link utilization within an IP network.
Multiprotocol Label Switching (MPLS) using Traffic Engineering (TE) Multiprotocol Label Switching (MPLS) using Traffic-Engineering (TE)
technology (MPLS-TE)[RFC3209]is one solution for traffic engineering technology (MPLS-TE) [RFC3209] is one solution for TE networks, but
networks but it introduces an MPLS network and related technology it introduces an MPLS network along with related technology, which
which would be an overlay of the IP network. MPLS-TE technology is would be an overlay of the IP network. MPLS-TE technology is often
often used for Label Switched Path (LSP) protection and complex path used for Label Switched Path (LSP) protection and setting up complex
set-up within a domain. It has not been widely deployed for meeting paths within a domain. It has not been widely deployed for meeting
E2E (especially in inter-domain) dynamic performance assurance E2E (especially in inter-domain) dynamic performance assurance
requirements for an IP network. requirements for an IP network.
Segment Routing [RFC8402] is another solution that integrates some Segment Routing [RFC8402] is another solution that integrates some
advantages of using a distributed protocol and a centrally control advantages of using a distributed protocol and central control
technology, but it requires the underlying network, especially the technology, but it requires the underlying network, especially the
provider edge router, to do a label push and pop action in-depth, and provider edge router, to do an in-depth label push and pop action
adds complexity when coexisting with the Non-Segment Routing network. while adding complexity when coexisting with the non-segment routing
Additionally, it can only maneuver the E2E paths for MPLS and IPv6 network. Additionally, it can only maneuver the E2E paths for MPLS
traffic via different mechanisms. and IPv6 traffic via different mechanisms.
Deterministic Networking (DetNet)[RFC8578] is another possible Deterministic Networking (DetNet) [RFC8578] is another possible
solution. It is primarily focused on providing bounded latency for a solution. It is primarily focused on providing bounded latency for a
flow and introduces additional requirements on the domain edge flow and introduces additional requirements on the domain edge
router. The current DetNet scope is within one domain. The use router. The current DetNet scope is within one domain. The use
cases defined in this document do not require the additional cases defined in this document do not require the additional
complexity of deterministic properties and so differ from the DetNet complexity of deterministic properties and so differ from the DetNet
use cases. use cases.
This draft describes several scenarios for a native IP network where This document describes several scenarios for a native IP network
a Centralized Control Dynamic Routing (CCDR) framework can produce where a Centralized Control Dynamic Routing (CCDR) framework can
qualitative improvement in efficiency without requiring a change of produce qualitative improvement in efficiency without requiring a
the data-plane behavior on the router. Using knowledge of BGP(Border change to the data-plane behavior on the router. Using knowledge of
Gateway Protocol) session-specific prefixes advertised by a router, the Border Gateway Protocol (BGP) session-specific prefixes
the network topology and the near real time link utilization advertised by a router, the network topology and the near-real-time
information from network management systems, a central PCE is able to link-utilization information from network management systems, a
compute an optimal path and give the underlay routers the destination central PCE is able to compute an optimal path and give the
address to use to reach the BGP nexthop, such that the distributed underlying routers the destination address to use to reach the BGP
routing protocol will use the computed path via traditional recursive nexthop, such that the distributed routing protocol will use the
lookup procedure. Some results from simulations of path optimization computed path via traditional recursive lookup procedure. Some
are also presented, to concretely illustrate a variety of scenarios results from simulations of path optimization are also presented to
where CCDR shows significant improvement over traditional distributed concretely illustrate a variety of scenarios where CCDR shows
routing protocols. significant improvement over traditional distributed routing
protocols.
This draft is the base document of the following two drafts: the
universal solution draft, which is suitable for intra-domain and
inter-domain TE scenario, is described in
[I-D.ietf-teas-pce-native-ip]; the related protocol extension This document is the base document of the following two documents:
contents is described in [I-D.ietf-pce-pcep-extension-native-ip] the universal solution document, which is suitable for intra-domain
and inter-domain TE scenario, is described in [PCE-NATIVE-IP]; and
the related protocol extension contents is described in
[PCEP-NATIVE-IP-EXT].
2. Terminology 2. Terminology
This document uses the following terms defined in [RFC5440]: PCE. In this document, PCE is used as defined in [RFC5440]. The following
terms are used as described here:
The following terms are defined in this document:
o BRAS: Broadband Remote Access Server BRAS: Broadband Remote Access Server
o CD: Congestion Degree CD: Congestion Degree
o CR: Core Router CR: Core Router
o CCDR: Centralized Control Dynamic Routing CCDR: Centralized Control Dynamic Routing
o E2E: End to End E2E: End to End
o IDC: Internet Data Center IDC: Internet Data Center
o MAN: Metro Area Network MAN: Metro Area Network
o QoS: Quality of Service QoS: Quality of Service
o SR: Service Router SR: Service Router
o TE: Traffic Engineering TE: Traffic Engineering
o UID: Utilization Increment Degree UID: Utilization Increment Degree
o WAN: Wide Area Network WAN: Wide Area Network
3. CCDR Scenarios 3. CCDR Scenarios
The following sections describe various deployment scenarios where The following sections describe various deployment scenarios where
applying the CCDR framework is intuitively expected to produce applying the CCDR framework is intuitively expected to produce
improvements, based on the macro-scale properties of the framework improvements based on the macro-scale properties of the framework and
and the scenario. the scenario.
3.1. QoS Assurance for Hybrid Cloud-based Application 3.1. QoS Assurance for Hybrid Cloud-Based Application
With the emergence of cloud computing technologies, enterprises are With the emergence of cloud computing technologies, enterprises are
putting more and more services on a public oriented cloud putting more and more services on a public-oriented cloud environment
environment, but keeping core business within their private cloud. while keeping core business within their private cloud. The
The communication between the private and public cloud sites will communication between the private and public cloud sites spans the
span the Wide Area Network (WAN) network. The bandwidth requirements WAN. The bandwidth requirements between them are variable, and the
between them are variable and the background traffic between these background traffic between these two sites varies over time.
two sites varies over time. Enterprise applications require Enterprise applications require assurance of the E2E QoS performance
assurance of the E2E Quality of Service(QoS) performance on demand on demand for variable bandwidth services.
for variable bandwidth services.
CCDR, which integrates the merits of distributed protocols and the CCDR, which integrates the merits of distributed protocols and the
power of centralized control, is suitable for this scenario. The power of centralized control, is suitable for this scenario. The
possible solution framework is illustrated below: possible solution framework is illustrated below:
+------------------------+ +------------------------+
| Cloud Based Application| | Cloud-Based Application|
+------------------------+ +------------------------+
| |
+-----------+ +-----------+
| PCE | | PCE |
+-----------+ +-----------+
| |
| |
//--------------\\ //--------------\\
///// \\\\\ ///// \\\\\
Private Cloud Site || Distributed |Public Cloud Site Private Cloud Site || Distributed |Public Cloud Site
| Control Network | | Control Network |
\\\\\ ///// \\\\\ /////
\\--------------// \\--------------//
Figure 1: Hybrid Cloud Communication Scenario Figure 1: Hybrid Cloud Communication Scenario
As illustrated in Figure 1, the source and destination of the "Cloud As illustrated in Figure 1, the source and destination of the "Cloud-
Based Application" traffic are located at "Private Cloud Site" and Based Application" traffic are located at "Private Cloud Site" and
"Public Cloud Site" respectively. "Public Cloud Site", respectively.
By default, the traffic path between the private and public cloud By default, the traffic path between the private and public cloud
site is determined by the distributed control network. When site is determined by the distributed control network. When an
application requires the E2E QoS assurance, it can send these application requires E2E QoS assurance, it can send these
requirements to the PCE, and let the PCE compute one E2E path which requirements to the PCE and let the PCE compute one E2E path, which
is based on the underlying network topology and the real traffic is based on the underlying network topology and real traffic
information, to accommodate the application's QoS requirements. information, in order to accommodate the application's QoS
Section 4.4 of this document describes the simulation results for requirements. Section 4.4 of this document describes the simulation
this use case. results for this use case.
3.2. Link Utilization Maximization 3.2. Link Utilization Maximization
Network topology within a Metro Area Network (MAN) is generally in a Network topology within a Metro Area Network (MAN) is generally in a
star mode as illustrated in Figure 2, with different devices star mode as illustrated in Figure 2, with different devices
connected to different customer types. The traffic from these connected to different customer types. The traffic from these
customers is often in a tidal pattern, with the links between the customers is often in a tidal pattern with the links between the Core
Core Router(CR)/Broadband Remote Access Server(BRAS) and CR/Service Router (CR) / Broadband Remote Access Server (BRAS) and CR/Service
Router(SR) experiencing congestion in different periods, because the Router (SR) experiencing congestion in different periods due to
subscribers under BRAS often use the network at night, and the leased subscribers under BRAS often using the network at night and the
line users under SR often use the network during the daytime. The leased line users under SR often using the network during the
link between BRAS/SR and CR must satisfy the maximum traffic volume daytime. The link between BRAS/SR and CR must satisfy the maximum
between them, respectively, and this causes these links often to be traffic volume between them, respectively, which causes these links
under-utilized. to often be underutilized.
+--------+ +--------+
| CR | | CR |
+----|---+ +----|---+
| |
--------|--------|-------| |-------|--------|-------|
| | | | | | | |
+--|-+ +-|- +--|-+ +-|+ +--|-+ +-|+ +--|-+ +-|+
|BRAS| |SR| |BRAS| |SR| |BRAS| |SR| |BRAS| |SR|
+----+ +--+ +----+ +--+ +----+ +--+ +----+ +--+
Figure 2: Star-mode Network Topology within MAN Figure 2: Star-Mode Network Topology within MAN
If we consider connecting the BRAS/SR with a local link loop (which If we consider connecting the BRAS/SR with a local link loop (which
is usually lower cost), and control the overall MAN topology with the is usually lower cost) and control the overall MAN topology with the
CCDR framework, we can exploit the tidal phenomena between the BRAS/ CCDR framework, we can exploit the tidal phenomena between the BRAS/
CR and SR/CR links, maximizing the utilization of these central trunk CR and SR/CR links, maximizing the utilization of these central trunk
links (which are usually higher cost than the local loops). links (which are usually higher cost than the local loops).
+-------+ +-------+
----- PCE | ----- PCE |
| +-------+ | +-------+
+----|---+ +----|---+
| CR | | CR |
+----|---+ +----|---+
| |
--------|--------|-------| |-------|--------|-------|
| | | | | | | |
+--|-+ +-|- +--|-+ +-|+ +--|-+ +-|+ +--|-+ +-|+
|BRAS-----SR| |BRAS-----SR| |BRAS-----SR| |BRAS-----SR|
+----+ +--+ +----+ +--+ +----+ +--+ +----+ +--+
Figure 3: Link Utilization Maximization via CCDR Figure 3: Link Utilization Maximization via CCDR
3.3. Traffic Engineering for Multi-Domain 3.3. Traffic Engineering for Multi-domain
Service provider networks are often comprised of different domains, Service provider networks are often comprised of different domains,
interconnected with each other, forming a very complex topology as interconnected with each other, forming a very complex topology as
illustrated in Figure 4. Due to the traffic pattern to/from the MAN illustrated in Figure 4. Due to the traffic pattern to/from the MAN
and IDC, the utilization of the links between them are often and IDC, the utilization of the links between them are often
asymmetric. It is almost impossible to balance the utilization of asymmetric. It is almost impossible to balance the utilization of
these links via a distributed protocol, but this unbalance can be these links via a distributed protocol, but this unbalance can be
overcome utilizing the CCDR framework. overcome utilizing the CCDR framework.
+---+ +---+ +---+ +---+
|MAN|-----------------IDC| |MAN|----------------|IDC|
+-|-| | +-|-+ +---+ | +---+
| ---------| | | ---------- |
------|BackBone|------ |-----|Backbone|-----|
| ----|----| | | ----|----- |
| | | | | |
+-|-- | ----+ +---+ | +---+
|IDC|----------------|MAN| |IDC|----------------|MAN|
+---| |---+ +---+ +---+
Figure 4: Traffic Engineering for Complex Multi-Domain Topology Figure 4: Traffic Engineering for Complex Multi-domain Topology
A solution for this scenario requires the gathering of NetFlow A solution for this scenario requires the gathering of NetFlow
information, analysis of the source/destination AS, and determining information, analysis of the source/destination autonomous system
what is the main cause of the congested link(s). After this, the (AS), and determining what the main cause of the congested link(s)
operator can use the external Border Gateway Protocol(eBGP) sessions is. After this, the operator can use the external Border Gateway
to schedule the traffic among the different domains according to the Protocol (eBGP) sessions to schedule the traffic among the different
solution described in CCDR framework. domains according to the solution described in the CCDR framework.
3.4. Network Temporal Congestion Elimination 3.4. Network Temporal Congestion Elimination
In more general situations, there are often temporal congestion In more general situations, there is often temporal congestion within
within the service provider's network, for example due to daily or the service provider's network, for example, due to daily or weekly
weekly periodic bursts, or large events that are scheduled well in periodic bursts or large events that are scheduled well in advance.
advance. Such congestion phenomena often appear regularly, and if Such congestion phenomena often appear regularly, and if the service
the service provider has methods to mitigate it, it will certainly provider has methods to mitigate it, it will certainly improve their
improve their network operations capabilities and increase network operation capabilities and increase satisfaction for
satisfaction for their customers. CCDR is also suitable for such customers. CCDR is also suitable for such scenarios, as the
scenarios, as the controller can schedule traffic out of the controller can schedule traffic out of the congested links, lowering
congested links, lowering the utilization of them during these times. their utilization during these times. Section 4.5 describes the
Section 4.5 describes the simulation results of this scenario. simulation results of this scenario.
4. CCDR Simulation 4. CCDR Simulation
The following sections describe a specific case study to illustrate The following sections describe a specific case study to illustrate
the workings of the CCDR algorithm with concrete paths/metrics, as the workings of the CCDR algorithm with concrete paths/metrics, as
well as a procedure for generating topology and traffic matrices and well as a procedure for generating topology and traffic matrices and
the results from simulations applying CCDR for E2E QoS (assured path the results from simulations applying CCDR for E2E QoS (assured path
and congestion elimination) over the generated topologies and traffic and congestion elimination) over the generated topologies and traffic
matrices. In all cases examined, the CCDR algorithm produces matrices. In all cases examined, the CCDR algorithm produces
qualitatively significant improvement over the reference (OSPF) qualitatively significant improvement over the reference (OSPF)
skipping to change at page 8, line 4 skipping to change at line 334
well as a procedure for generating topology and traffic matrices and well as a procedure for generating topology and traffic matrices and
the results from simulations applying CCDR for E2E QoS (assured path the results from simulations applying CCDR for E2E QoS (assured path
and congestion elimination) over the generated topologies and traffic and congestion elimination) over the generated topologies and traffic
matrices. In all cases examined, the CCDR algorithm produces matrices. In all cases examined, the CCDR algorithm produces
qualitatively significant improvement over the reference (OSPF) qualitatively significant improvement over the reference (OSPF)
algorithm, suggesting that CCDR will have broad applicability. algorithm, suggesting that CCDR will have broad applicability.
The structure and scale of the simulated topology is similar to that The structure and scale of the simulated topology is similar to that
of the real networks. Multiple different traffic matrices were of the real networks. Multiple different traffic matrices were
generated to simulate different congestion conditions in the network. generated to simulate different congestion conditions in the network.
Only one of them is illustrated since the others produce similar Only one of them is illustrated since the others produce similar
results. results.
4.1. Case Study for CCDR Algorithm 4.1. Case Study for CCDR Algorithm
In this section we consider a specific network topology for case In this section, we consider a specific network topology for case
study, examining the path selected by OSPF and CCDR and evaluating study: examining the path selected by OSPF and CCDR and evaluating
how and why the paths differ. Figure 5 depicts the topology of the how and why the paths differ. Figure 5 depicts the topology of the
network in this case. There are 8 forwarding devices in the network. network in this case. There are eight forwarding devices in the
The original cost and utilization are marked on it, as shown in the network. The original cost and utilization are marked on it as shown
figure. For example, the original cost and utilization for the link in the figure. For example, the original cost and utilization for
(1,2) are 3 and 50% respectively. There are two flows: f1 and f2. the link (1, 2) are 3 and 50%, respectively. There are two flows: f1
Both of these two flows are from node 1 to node 8. For simplicity, and f2. Both of these two flows are from node 1 to node 8. For
it is assumed that the bandwidth of the link in the network is 10Mb/ simplicity, it is assumed that the bandwidth of the link in the
s. The flow rate of f1 is 1Mb/s, and the flow rate of f2 is 2Mb/s. network is 10 Mb/s. The flow rate of f1 is 1 Mb/s and the flow rate
The threshold of the link in congestion is 90%. of f2 is 2 Mb/s. The threshold of the link in congestion is 90%.
If OSPF protocol (ISIS is similar, because it also use the Dijstra's If the OSPF protocol, which adopts Dijkstra's algorithm (IS-IS is
algorithm) is applied in the network, which adopts Dijkstra's similar because it also uses Dijkstra's algorithm), is applied in the
algorithm, the two flows from node 1 to node 8 can only use the OSPF network, the two flows from node 1 to node 8 can only use the OSPF
path (p1: 1->2->3->8). It is because Dijkstra's algorithm mainly path (p1: 1->2->3->8). This is because Dijkstra's algorithm mainly
considers original cost of the link. Since CCDR considers cost and considers the original cost of the link. Since CCDR considers cost
utilization simultaneously, the same path as OSPF will not be and utilization simultaneously, the same path as OSPF will not be
selected due to the severe congestion of the link (2,3). In this selected due to the severe congestion of the link (2, 3). In this
case, f1 will select the path (p2: 1->5->6->7->8) since the new cost case, f1 will select the path (p2: 1->5->6->7->8) since the new cost
of this path is better than that of OSPF path. Moreover, the path p2 of this path is better than that of the OSPF path. Moreover, the
is also better than the path (p3: 1->2->4->7->8) for for flow f1. path p2 is also better than the path (p3: 1->2->4->7->8) for flow f1.
However, f2 will not select the same path since it will cause the new However, f2 will not select the same path since it will cause new
congestion in the link (6,7). As a result, f2 will select the path congestion in the link (6, 7). As a result, f2 will select the path
(p3: 1->2->4->7->8). (p3: 1->2->4->7->8).
+----+ f1 +-------> +-----+ ----> +-----+ +----+ f1 +-------> +-----+ ----> +-----+
|Edge|-----------+ |+--------| 3 |-------| 8 | |Edge|-----------+ |+--------| 3 |-------| 8 |
|Node|---------+ | ||+-----> +-----+ ----> +-----+ |Node|---------+ | ||+-----> +-----+ ----> +-----+
+----+ | | 4/95%||| 6/50% | +----+ | | 4/95%||| 6/50% |
| | ||| 5/60%| | | ||| 5/60%|
| v ||| | | v ||| |
+----+ +-----+ -----> +-----+ +-----+ +-----+ +----+ +-----+ -----> +-----+ +-----+ +-----+
|Edge|-------| 1 |--------| 2 |------| 4 |------| 7 | |Edge|-------| 1 |--------| 2 |------| 4 |------| 7 |
|Node|-----> +-----+ -----> +-----+7/60% +-----+5/45% +-----+ |Node|-----> +-----+ -----> +-----+7/60% +-----+5/45% +-----+
+----+ f2 | 3/50% | +----+ f2 | 3/50% |
| | | |
| 3/60% +-----+ 5/55%+-----+ 3/75% | | 3/60% +-----+ 5/55%+-----+ 3/75% |
+-----------| 5 |------| 6 |---------+ +-----------| 5 |------| 6 |---------+
+-----+ +-----+ +-----+ +-----+
(a) Dijkstra's Algorithm (OSPF/ISIS) (a) Dijkstra's Algorithm (OSPF/IS-IS)
+----+ f1 +-----+ ----> +-----+ +----+ f1 +-----+ ----> +-----+
|Edge|-----------+ +--------| 3 |-------| 8 | |Edge|-----------+ +--------| 3 |-------| 8 |
|Node|---------+ | | +-----+ ----> +-----+ |Node|---------+ | | +-----+ ----> +-----+
+----+ | | 4/95% | 6/50% ^|^ +----+ | | 4/95% | 6/50% ^|^
| | | 5/60%||| | | | 5/60%|||
| v | ||| | v | |||
+----+ +-----+ -----> +-----+ ---> +-----+ ---> +-----+ +----+ +-----+ -----> +-----+ ---> +-----+ ---> +-----+
|Edge|-------| 1 |--------| 2 |------| 4 |------| 7 | |Edge|-------| 1 |--------| 2 |------| 4 |------| 7 |
|Node|-----> +-----+ +-----+7/60% +-----+5/45% +-----+ |Node|-----> +-----+ +-----+7/60% +-----+5/45% +-----+
+----+ f2 || 3/50% |^ +----+ f2 || 3/50% |^
|| || || ||
|| 3/60% +-----+5/55% +-----+ 3/75% || || 3/60% +-----+5/55% +-----+ 3/75% ||
|+-----------| 5 |------| 6 |---------+| |+-----------| 5 |------| 6 |---------+|
+----------> +-----+ ---> +-----+ ---------+ +----------> +-----+ ---> +-----+ ---------+
(b) CCDR Algorithm (b) CCDR Algorithm
Figure 5: Case Study for CCDR's Algorithm Figure 5: Case Study for CCDR's Algorithm
4.2. Topology Simulation 4.2. Topology Simulation
Moving on from the specific case study, we now consider a class of Moving on from the specific case study, we now consider a class of
networks more representative of real deployments, with a fully-linked networks more representative of real deployments, with a fully linked
core network that serves to connect edge nodes, which themselves core network that serves to connect edge nodes, which themselves
connect to only a subset of the core. An example of such a topology connect to only a subset of the core. An example of such a topology
is shown in Figure 6, for the case of 4 core nodes and 5 edge nodes. is shown in Figure 6 for the case of 4 core nodes and 5 edge nodes.
The CCDR simulations presented in this work use topologies involving The CCDR simulations presented in this work use topologies involving
100 core nodes and 400 edge nodes. While the resulting graph does 100 core nodes and 400 edge nodes. While the resulting graph does
not fit on this page, this scale of network is similar to what is not fit on this page, this scale of network is similar to what is
deployed in production environments. deployed in production environments.
+----+ +----+
/|Edge|\ /|Edge|\
| +----+ | | +----+ |
| | | |
| | | |
+----+ +----+ +----+ +----+ +----+ +----+
|Edge|----|Core|-----|Core|---------+ |Edge|----|Core|-----|Core|---------+
+----+ +----+ +----+ | +----+ +----+ +----+ |
/ | \ / | | / | \ / | |
+----+ | \ / | | +----+ | \ / | |
|Edge| | X | | |Edge| | X | |
+----+ | / \ | | +----+ | / \ | |
\ | / \ | | \ | / \ | |
+----+ +----+ +----+ | +----+ +----+ +----+ |
|Edge|----|Core|-----|Core| | |Edge|----|Core|-----|Core| |
+----+ +----+ +----+ | +----+ +----+ +----+ |
| | | | | |
| +------\ +----+ | +------\ +----+
| ---|Edge| | ---|Edge|
+-----------------/ +----+ +-----------------/ +----+
Figure 6: Topology of Simulation Figure 6: Topology of Simulation
For the simulations, the number of links connecting one edge node to For the simulations, the number of links connecting one edge node to
the set of core nodes is randomly chosen between 2 to 30, and the the set of core nodes is randomly chosen between two and thirty, and
total number of links is more than 20000. Each link has a congestion the total number of links is more than 20,000. Each link has a
threshold, which can be arbitrarily set to (e.g.) 90% of the nominal congestion threshold, which can be arbitrarily set, for example, to
link capacity without affecting the simulation results. 90% of the nominal link capacity without affecting the simulation
results.
4.3. Traffic Matrix Simulation 4.3. Traffic Matrix Simulation
For each topology, a traffic matrix is generated based on the link For each topology, a traffic matrix is generated based on the link
capacity of topology. It can result in many kinds of situations, capacity of the topology. It can result in many kinds of situations
such as congestion, mild congestion and non-congestion. such as congestion, mild congestion, and non-congestion.
In the CCDR simulation, the dimension of the traffic matrix is In the CCDR simulation, the dimension of the traffic matrix is
500*500 (100 core nodes plus 400 edge nodes). About 20% of links are 500*500 (100 core nodes plus 400 edge nodes). About 20% of links are
overloaded when the Open Shortest Path First (OSPF) protocol is used overloaded when the Open Shortest Path First (OSPF) protocol is used
in the network. in the network.
4.4. CCDR End-to-End Path Optimization 4.4. CCDR End-to-End Path Optimization
The CCDR E2E path optimization is to find the best path which is the The CCDR E2E path optimization entails finding the best path, which
lowest in metric value and for each link of the path, the is the lowest in metric value, as well as having utilization far
utilizatioin is far below link's congestion threshold. Based on the below the congestion threshold for each link of the path. Based on
current state of the network, the PCE within CCDR framework combines the current state of the network, the PCE within CCDR framework
the shortest path algorithm with a penalty theory of classical combines the shortest path algorithm with a penalty theory of
optimization and graph theory. classical optimization and graph theory.
Given a background traffic matrix, which is unscheduled, when a set Given a background traffic matrix, which is unscheduled, when a set
of new flows comes into the network, the E2E path optimization finds of new flows comes into the network, the E2E path optimization finds
the optimal paths for them. The selected paths bring the least the optimal paths for them. The selected paths bring the least
congestion degree to the network. congestion degree to the network.
The link Utilization Increment Degree(UID), when the new flows are The link Utilization Increment Degree (UID), when the new flows are
added into the network, is shown in Figure 7. The first graph in added into the network, is shown in Figure 7. The first graph in
Figure 7 is the UID with OSPF and the second graph is the UID with Figure 7 is the UID with OSPF, and the second graph is the UID with
CCDR E2E path optimization. The average UID of the first graph is CCDR E2E path optimization. The average UID of the first graph is
more than 30%. After path optimization, the average UID is less than more than 30%. After path optimization, the average UID is less than
5%. The results show that the CCDR E2E path optimization has an eye- 5%. The results show that the CCDR E2E path optimization has an eye-
catching decrease in UID relative to the path chosen based on OSPF. catching decrease in UID relative to the path chosen based on OSPF.
While real-world results invariably differ from simulations (for While real-world results invariably differ from simulations (for
example, real-world topologies are likely to exhibit correlation in example, real-world topologies are likely to exhibit correlation in
the attachment patterns for edge nodes to the core, which are not the attachment patterns for edge nodes to the core, which are not
reflected in these results), the dramatic nature of the improvement reflected in these results), the dramatic nature of the improvement
in UID and the choice of simulated topology to resemble real-world in UID and the choice of simulated topology to resemble real-world
conditions suggests that real-world deployments will also experience conditions suggest that real-world deployments will also experience
significant improvement in UID results. significant improvement in UID results.
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| * * * *| | * * * *|
60| * * * * * *| 60| * * * * * *|
|* * ** * * * * * ** * * * * **| |* * ** * * * * * ** * * * * **|
|* * ** * * ** *** ** * * ** * * * ** * * *** **| |* * ** * * ** *** ** * * ** * * * ** * * *** **|
|* * * ** * ** ** *** *** ** **** ** *** **** ** *** **| |* * * ** * ** ** *** *** ** **** ** *** **** ** *** **|
40|* * * ***** ** *** *** *** ** **** ** *** ***** ****** **| 40|* * * ***** ** *** *** *** ** **** ** *** ***** ****** **|
UID(%)|* * ******* ** *** *** ******* **** ** *** ***** *********| UID(%)|* * ******* ** *** *** ******* **** ** *** ***** *********|
|*** ******* ** **** *********** *********** ***************| |*** ******* ** **** *********** *********** ***************|
|******************* *********** *********** ***************| |******************* *********** *********** ***************|
20|******************* ***************************************| 20|******************* ***************************************|
|******************* ***************************************| |******************* ***************************************|
|***********************************************************| |***********************************************************|
|***********************************************************| |***********************************************************|
0+-----------------------------------------------------------+ 0+-----------------------------------------------------------+
0 100 200 300 400 500 600 700 800 900 1000 0 100 200 300 400 500 600 700 800 900 1000
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| | | |
60| | 60| |
| | | |
| | | |
| | | |
40| | 40| |
UID(%)| | UID(%)| |
| | | |
| | | |
20| | 20| |
| *| | *|
| * *| | * *|
| * * * * * ** * *| | * * * * * ** * *|
0+-----------------------------------------------------------+ 0+-----------------------------------------------------------+
0 100 200 300 400 500 600 700 800 900 1000 0 100 200 300 400 500 600 700 800 900 1000
Flow Number Flow Number
Figure 7: Simulation Result with Congestion Elimination Figure 7: Simulation Results with Congestion Elimination
4.5. Network Temporal Congestion Elimination 4.5. Network Temporal Congestion Elimination
During the simulations, different degrees of network congestion were During the simulations, different degrees of network congestion were
considered. To examine the effect of CCDR on link congestion, we considered. To examine the effect of CCDR on link congestion, we
consider the Congestion Degree (CD) of a link, defined as the link consider the Congestion Degree (CD) of a link, defined as the link
utilization beyond its threshold. utilization beyond its threshold.
The CCDR congestion elimination performance is shown in Figure 8. The CCDR congestion elimination performance is shown in Figure 8.
The first graph is the CD distribution before the process of The first graph is the CD distribution before the process of
congestion elimination. The average CD of all congested links is congestion elimination. The average CD of all congested links is
about 20%. The second graph shown in Figure 8 is the CD distribution about 20%. The second graph shown in Figure 8 is the CD distribution
after using the congestion elimination process. It shows that only after using the congestion elimination process. It shows that only
12 links among the total of 20000 links exceed the threshold, and all twelve links among the total 20,000 exceed the threshold, and all the
the CD values are less than 3%. Thus, after scheduling of the traffic CD values are less than 3%. Thus, after scheduling the traffic away
away from the congested paths, the degree of network congestion is from the congested paths, the degree of network congestion is greatly
greatly eliminated and the network utilization is in balance. eliminated and the network utilization is in balance.
Before congestion elimination Before congestion elimination
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| * ** * ** ** *| | * ** * ** ** *|
20| * * **** * ** ** *| 20| * * **** * ** ** *|
|* * ** * ** ** **** * ***** *********| |* * ** * ** ** **** * ***** *********|
|* * * * * **** ****** * ** *** **********************| |* * * * * **** ****** * ** *** **********************|
15|* * * ** * ** **** ********* *****************************| 15|* * * ** * ** **** ********* *****************************|
|* * ****** ******* ********* *****************************| |* * ****** ******* ********* *****************************|
CD(%) |* ********* ******* ***************************************| CD(%) |* ********* ******* ***************************************|
10|* ********* ***********************************************| 10|* ********* ***********************************************|
|*********** ***********************************************| |*********** ***********************************************|
skipping to change at page 13, line 46 skipping to change at line 572
10| | 10| |
| | | |
| | | |
5 | | 5 | |
| | | |
| * ** * * * ** * ** * | | * ** * * * ** * ** * |
0 +-----------------------------------------------------------+ 0 +-----------------------------------------------------------+
0 0.5 1 1.5 2 0 0.5 1 1.5 2
Link Number(*10000) Link Number(*10000)
Figure 8: Simulation Result with Congestion Elimination Figure 8: Simulation Results with Congestion Elimination
It is clear that using an active path-computation mechanism that is It is clear that by using an active path-computation mechanism that
able to take into account observed link traffic/congestion, the is able to take into account observed link traffic/congestion, the
occurrence of congestion events can be greatly reduced. Only when a occurrence of congestion events can be greatly reduced. Only when a
preponderance of links in the network are near their congestion preponderance of links in the network are near their congestion
threshold will the central controller be unable to find a clear path, threshold will the central controller be unable to find a clear path
as opposed to when a static metric-based procedure is used, which as opposed to when a static metric-based procedure is used, which
will produce congested paths once a single bottleneck approaches its will produce congested paths once a single bottleneck approaches its
capacity. More detailed information about the algorithm can be found capacity. More detailed information about the algorithm can be found
in[PTCS] . in [PTCS].
5. CCDR Deployment Consideration 5. CCDR Deployment Consideration
The above CCDR scenarios and simulation results demonstrate that a The above CCDR scenarios and simulation results demonstrate that a
single general solution can be found that copes with multiple complex single general solution can be found that copes with multiple complex
situations. The specific situations considered are not known to have situations. The specific situations considered are not known to have
any special properties, so it is expected that the benefits any special properties, so it is expected that the benefits
demonstrated will have general applicability. Accordingly, the demonstrated will have general applicability. Accordingly, the
integrated use of a centralized controller for the more complex integrated use of a centralized controller for the more complex
optimal path computations in a native IP network should result in optimal path computations in a native IP network should result in
significant improvements without impacting the underlay network significant improvements without impacting the underlying network
infrastructure. infrastructure.
For intra-domain or inter-domain native IP TE scenarios, the For intra-domain or inter-domain native IP TE scenarios, the
deployment of a CCDR solution is similar, with the centralized deployment of a CCDR solution is similar with the centralized
controller being able to compute paths and no changes required to the controller being able to compute paths along with no changes being
underlying network infrastructure. This universal deployment required to the underlying network infrastructure. This universal
characteristic can facilitate a generic traffic engineering solution, deployment characteristic can facilitate a generic traffic-
where operators do not need to differentiate between intra-domain and engineering solution where operators do not need to differentiate
inter-domain TE cases. between intra-domain and inter-domain TE cases.
To deploy the CCDR solution, the PCE should collect the underlay To deploy the CCDR solution, the PCE should collect the underlying
network topology dynamically, for example via BGP-LS[RFC7752]. It network topology dynamically, for example, via Border Gateway
also needs to gather the network traffic information periodically Protocol - Link State (BGP-LS) [RFC7752]. It also needs to gather
from the network management platform. The simulation results show the network traffic information periodically from the network
that the PCE can compute the E2E optimal path within seconds, thus it management platform. The simulation results show that the PCE can
can cope with the change of underlay network on the scale of minutes. compute the E2E optimal path within seconds; thus, it can cope with a
More agile requirements would need to increase the sample rate of change to the underlying network in a matter of minutes. More agile
underlay network and decrease the detection and notification interval requirements would need to increase the sample rate of the underlying
of the underlay network. The methods to gather and decrease the network and decrease the detection and notification interval of the
latency of these information are out of the scope of this draft. underlying network. The methods of gathering this information as
well as decreasing its latency are out of the scope of this document.
6. Security Considerations 6. Security Considerations
This document considers mainly the integration of distributed This document considers mainly the integration of distributed
protocols and the central control capability of a PCE. While it protocols and the central control capability of a PCE. While it can
certainly can ease the management of network in various traffic certainly simplify the management of a network in various traffic-
engineering scenarios as described in this document, the centralized engineering scenarios as described in this document, the centralized
control also bring a new point that may be easily attacked. control also brings a new point that may be easily attacked.
Solutions for CCDR scenarios need to consider protection of the PCE Solutions for CCDR scenarios need to consider protection of the PCE
and communication with the underlay devices. and communication with the underlying devices.
[RFC5440] and [RFC8253] provide additional information. [RFC5440] and [RFC8253] provide additional information.
The control priority and interaction process should also be carefully The control priority and interaction process should also be carefully
designed for the combination of distributed protocol and central designed for the combination of the distributed protocol and central
control. Generally, the central control instruction should have control. Generally, the central control instructions should have
higher priority than the forwarding actions determined by the higher priority than the forwarding actions determined by the
distributed protocol. When the communication between PCE and the distributed protocol. When communication between PCE and the
underlay devices is not in function, the distributed protocol should underlying devices is disrupted, the distributed protocol should take
take over the control right of the underlay network. control of the underlying network. [PCE-NATIVE-IP] provides more
[I-D.ietf-teas-pce-native-ip] provides more considerations considerations corresponding to the solution.
corresponding to the solution.
7. IANA Considerations 7. IANA Considerations
This document does not require any IANA actions. This document has no IANA actions.
8. Contributors
Lu Huang contributed to the content of this draft.
9. Acknowledgement
The author would like to thank Deborah Brungard, Adrian Farrel,
Huaimo Chen, Vishnu Beeram and Lou Berger for their support and
comments on this draft.
Thanks Benjamin Kaduk for his careful review and valuable suggestions
to this draft. Also thanks Roman Danyliw, Alvaro Retana and Eric
Vyncke for their views and comments.
10. References 8. References
10.1. Normative References 8.1. Normative References
[RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation
Element (PCE) Communication Protocol (PCEP)", RFC 5440, Element (PCE) Communication Protocol (PCEP)", RFC 5440,
DOI 10.17487/RFC5440, March 2009, DOI 10.17487/RFC5440, March 2009,
<https://www.rfc-editor.org/info/rfc5440>. <https://www.rfc-editor.org/info/rfc5440>.
[RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and
S. Ray, "North-Bound Distribution of Link-State and S. Ray, "North-Bound Distribution of Link-State and
Traffic Engineering (TE) Information Using BGP", RFC 7752, Traffic Engineering (TE) Information Using BGP", RFC 7752,
DOI 10.17487/RFC7752, March 2016, DOI 10.17487/RFC7752, March 2016,
<https://www.rfc-editor.org/info/rfc7752>. <https://www.rfc-editor.org/info/rfc7752>.
[RFC8253] Lopez, D., Gonzalez de Dios, O., Wu, Q., and D. Dhody, [RFC8253] Lopez, D., Gonzalez de Dios, O., Wu, Q., and D. Dhody,
"PCEPS: Usage of TLS to Provide a Secure Transport for the "PCEPS: Usage of TLS to Provide a Secure Transport for the
Path Computation Element Communication Protocol (PCEP)", Path Computation Element Communication Protocol (PCEP)",
RFC 8253, DOI 10.17487/RFC8253, October 2017, RFC 8253, DOI 10.17487/RFC8253, October 2017,
<https://www.rfc-editor.org/info/rfc8253>. <https://www.rfc-editor.org/info/rfc8253>.
10.2. Informative References 8.2. Informative References
[I-D.ietf-pce-pcep-extension-native-ip] [PCE-NATIVE-IP]
Wang, A., Khasanov, B., Cheruathur, S., Zhu, C., and S. Wang, A., Zhao, Q., Khasanov, B., and H. Chen, "PCE in
Fang, "PCEP Extension for Native IP Network", draft-ietf- Native IP Network", Work in Progress, Internet-Draft,
pce-pcep-extension-native-ip-04 (work in progress), August draft-ietf-teas-pce-native-ip-05, 9 January 2020,
2019. <https://tools.ietf.org/html/draft-ietf-teas-pce-native-
ip-05>.
[I-D.ietf-teas-pce-native-ip] [PCEP-NATIVE-IP-EXT]
Wang, A., Zhao, Q., Khasanov, B., Chen, H., and R. Mallya, Wang, A., Khasanov, B., Fang, S., and C. Zhu, "PCEP
"PCE in Native IP Network", draft-ietf-teas-pce-native- Extension for Native IP Network", Work in Progress,
ip-04 (work in progress), August 2019. Internet-Draft, draft-ietf-pce-pcep-extension-native-ip-
05, 17 February 2020, <https://tools.ietf.org/html/draft-
ietf-pce-pcep-extension-native-ip-05>.
[PTCS] Zhang, P., Xie, K., Kou, C., Huang, X., Wang, A., and Q. [PTCS] Zhang, P., Xie, K., Kou, C., Huang, X., Wang, A., and Q.
Sun, "A Practical Traffic Control Scheme With Load Sun, "A Practical Traffic Control Scheme With Load
Balancing Based on PCE Architecture", IEEE Balancing Based on PCE Architecture",
Access 18526773, DOI 10.1109/ACCESS.2019.2902610, March DOI 10.1109/ACCESS.2019.2902610, IEEE Access 18526773,
2019, <http://ieeexplore.ieee.org/document/8657733>. March 2019,
<https://ieeexplore.ieee.org/document/8657733>.
[RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
<https://www.rfc-editor.org/info/rfc3209>. <https://www.rfc-editor.org/info/rfc3209>.
[RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L.,
Decraene, B., Litkowski, S., and R. Shakir, "Segment Decraene, B., Litkowski, S., and R. Shakir, "Segment
Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, Routing Architecture", RFC 8402, DOI 10.17487/RFC8402,
July 2018, <https://www.rfc-editor.org/info/rfc8402>. July 2018, <https://www.rfc-editor.org/info/rfc8402>.
[RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases",
RFC 8578, DOI 10.17487/RFC8578, May 2019, RFC 8578, DOI 10.17487/RFC8578, May 2019,
<https://www.rfc-editor.org/info/rfc8578>. <https://www.rfc-editor.org/info/rfc8578>.
Acknowledgements
The authors would like to thank Deborah Brungard, Adrian Farrel,
Huaimo Chen, Vishnu Beeram, and Lou Berger for their support and
comments on this document.
Thanks to Benjamin Kaduk for his careful review and valuable
suggestions on this document. Also, thanks to Roman Danyliw, Alvaro
Retana, and √Čric Vyncke for their reviews and comments.
Contributors
Lu Huang contributed to the content of this document.
Authors' Addresses Authors' Addresses
Aijun Wang Aijun Wang
China Telecom China Telecom
Beiqijia Town, Changping District Beiqijia Town, Changping District
Beijing, Beijing 102209 Beijing
Beijing, 102209
China China
Email: wangaj3@chinatelecom.cn Email: wangaj3@chinatelecom.cn
Xiaohong Huang Xiaohong Huang
Beijing University of Posts and Telecommunications Beijing University of Posts and Telecommunications
No.10 Xitucheng Road, Haidian District No.10 Xitucheng Road, Haidian District
Beijing Beijing
China China
Email: huangxh@bupt.edu.cn Email: huangxh@bupt.edu.cn
Caixia Kou Caixia Kou
Beijing University of Posts and Telecommunications Beijing University of Posts and Telecommunications
skipping to change at page 17, line 23 skipping to change at line 743
Beijing University of Posts and Telecommunications Beijing University of Posts and Telecommunications
No.10 Xitucheng Road, Haidian District No.10 Xitucheng Road, Haidian District
Beijing Beijing
China China
Email: koucx@lsec.cc.ac.cn Email: koucx@lsec.cc.ac.cn
Zhenqiang Li Zhenqiang Li
China Mobile China Mobile
32 Xuanwumen West Ave, Xicheng District 32 Xuanwumen West Ave, Xicheng District
Beijing 100053 Beijing
100053
China China
Email: li_zhenqiang@hotmail.com Email: li_zhenqiang@hotmail.com
Penghui Mi Penghui Mi
Huawei Technologies Huawei Technologies
Tower C of Bldg.2, Cloud Park, No.2013 of Xuegang Road Tower C of Bldg.2, Cloud Park, No.2013 of Xuegang Road
Shenzhen, Bantian,Longgang District 518129 Shenzhen
Bantian,Longgang District, 518129
China China
Email: mipenghui@huawei.com Email: mipenghui@huawei.com
 End of changes. 97 change blocks. 
371 lines changed or deleted 376 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/