draft-ietf-rtgwg-bgp-routing-large-dc-06.txt   draft-ietf-rtgwg-bgp-routing-large-dc-07.txt 
Routing Area Working Group P. Lapukhov Routing Area Working Group P. Lapukhov
Internet-Draft Facebook Internet-Draft Facebook
Intended status: Informational A. Premji Intended status: Informational A. Premji
Expires: February 20, 2016 Arista Networks Expires: February 29, 2016 Arista Networks
J. Mitchell, Ed. J. Mitchell, Ed.
August 19, 2015 August 28, 2015
Use of BGP for routing in large-scale data centers Use of BGP for routing in large-scale data centers
draft-ietf-rtgwg-bgp-routing-large-dc-06 draft-ietf-rtgwg-bgp-routing-large-dc-07
Abstract Abstract
Some network operators build and operate data centers that support Some network operators build and operate data centers that support
over one hundred thousand servers. In this document, such data over one hundred thousand servers. In this document, such data
centers are referred to as "large-scale" to differentiate them from centers are referred to as "large-scale" to differentiate them from
smaller infrastructures. Environments of this scale have a unique smaller infrastructures. Environments of this scale have a unique
set of network requirements with an emphasis on operational set of network requirements with an emphasis on operational
simplicity and network stability. This document summarizes simplicity and network stability. This document summarizes
operational experience in designing and operating large-scale data operational experience in designing and operating large-scale data
skipping to change at page 1, line 41 skipping to change at page 1, line 41
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 20, 2016. This Internet-Draft will expire on February 29, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 11, line 48 skipping to change at page 11, line 48
Layer 2 designs with active-active network paths while relying on STP Layer 2 designs with active-active network paths while relying on STP
as the backup for loop prevention. The major downsides of this as the backup for loop prevention. The major downsides of this
approach are the lack of ability to scale linearly past two in most approach are the lack of ability to scale linearly past two in most
implementations, lack of standards based implementations, and added implementations, lack of standards based implementations, and added
failure domain risk of keeping state between the devices. failure domain risk of keeping state between the devices.
It should be noted that building large, horizontally scalable, Layer It should be noted that building large, horizontally scalable, Layer
2 only networks without STP is possible recently through the 2 only networks without STP is possible recently through the
introduction of the TRILL protocol in [RFC6325]. TRILL resolves many introduction of the TRILL protocol in [RFC6325]. TRILL resolves many
of the issues STP has for large-scale DC design however due to the of the issues STP has for large-scale DC design however due to the
lack of maturity of the protocol, the limited number of limited number of implementations, and often the requirement for
implementations, and requirement for new equipment that supports it, specific equipment that supports it, this has limited its
this has limited its applicability and increased the cost of such applicability and increased the cost of such designs.
designs.
Finally, neither TRILL nor the M-LAG approach eliminate the Finally, neither the base TRILL specification nor the M-LAG approach
fundamental problem of the shared broadcast domain, that is so totally eliminate the problem of the shared broadcast domain, that is
detrimental to the operations of any Layer 2, Ethernet based so detrimental to the operations of any Layer 2, Ethernet based
solutions. solutions. Later TRILL extensions have been proposed to solve the
this problem statement primarily based on the approaches outlined in
[RFC7067], but this even further limits the number of available
interoperable implementations that can be used to build a fabric,
therefore TRILL based designs have issues meeting REQ2, REQ3, and
REQ4.
4.2. Hybrid L2/L3 Designs 4.2. Hybrid L2/L3 Designs
Operators have sought to limit the impact of data plane faults and Operators have sought to limit the impact of data plane faults and
build large-scale topologies through implementing routing protocols build large-scale topologies through implementing routing protocols
in either the Tier-1 or Tier-2 parts of the network and dividing the in either the Tier-1 or Tier-2 parts of the network and dividing the
Layer 2 domain into numerous, smaller domains. This design has Layer 2 domain into numerous, smaller domains. This design has
allowed data centers to scale up, but at the cost of complexity in allowed data centers to scale up, but at the cost of complexity in
the network managing multiple protocols. For the following reasons, the network managing multiple protocols. For the following reasons,
operators have retained Layer 2 in either the access (Tier-3) or both operators have retained Layer 2 in either the access (Tier-3) or both
skipping to change at page 13, line 37 skipping to change at page 13, line 42
service provider communities, it is not generally deployed as the service provider communities, it is not generally deployed as the
primary routing protocol within the data center for a number of primary routing protocol within the data center for a number of
reasons (some of which are interrelated): reasons (some of which are interrelated):
o BGP is perceived as a "WAN only protocol only" and not often o BGP is perceived as a "WAN only protocol only" and not often
considered for enterprise or data center applications. considered for enterprise or data center applications.
o BGP is believed to have a "much slower" routing convergence o BGP is believed to have a "much slower" routing convergence
compared to IGPs. compared to IGPs.
o BGP deployment within an Autonomous System typically assumes the o Large scale BGP deployments typically utilize an IGP for BGP next-
presence of an IGP for next-hop resolution. hop resolution as all nodes in the iBGP topology are not directly
connected.
o BGP is perceived to require significant configuration overhead and o BGP is perceived to require significant configuration overhead and
does not support neighbor auto-discovery. does not support neighbor auto-discovery.
This document discusses some of these perceptions, especially as This document discusses some of these perceptions, especially as
applicable to the proposed design, and highlights some of the applicable to the proposed design, and highlights some of the
advantages of using the protocol such as: advantages of using the protocol such as:
o BGP has less complexity in parts of its protocol design - internal o BGP has less complexity in parts of its protocol design - internal
data structures and state machine are simpler as compared to most data structures and state machine are simpler as compared to most
skipping to change at page 14, line 13 skipping to change at page 14, line 18
implementing adjacency formation, adjacency maintenance and/or implementing adjacency formation, adjacency maintenance and/or
flow-control, BGP simply relies on TCP as the underlying flow-control, BGP simply relies on TCP as the underlying
transport. This fulfills REQ2 and REQ3. transport. This fulfills REQ2 and REQ3.
o BGP information flooding overhead is less when compared to link- o BGP information flooding overhead is less when compared to link-
state IGPs. Since every BGP router calculates and propagates only state IGPs. Since every BGP router calculates and propagates only
the best-path selected, a network failure is masked as soon as the the best-path selected, a network failure is masked as soon as the
BGP speaker finds an alternate path, which exists when highly BGP speaker finds an alternate path, which exists when highly
symmetric topologies, such as Clos, are coupled with EBGP only symmetric topologies, such as Clos, are coupled with EBGP only
design. In contrast, the event propagation scope of a link-state design. In contrast, the event propagation scope of a link-state
IGP is an entire area, regardless of the failure type. This meets IGP is an entire area, regardless of the failure type. In this
REQ3 and REQ4. It is also worth mentioning that all widely way, BGP better meets REQ3 and REQ4. It is also worth mentioning
deployed link-state IGPs feature periodic refreshes of routing that all widely deployed link-state IGPs feature periodic
information, even if this rarely causes impact to modern router refreshes of routing information, even if this rarely causes
control planes, while BGP does not expire routing state. impact to modern router control planes, while BGP does not expire
routing state.
o BGP supports third-party (recursively resolved) next-hops. This o BGP supports third-party (recursively resolved) next-hops. This
allows for manipulating multipath to be non-ECMP based or allows for manipulating multipath to be non-ECMP based or
forwarding based on application-defined paths, through forwarding based on application-defined paths, through
establishment of a peering session with an application establishment of a peering session with an application
"controller" which can inject routing information into the system, "controller" which can inject routing information into the system,
satisfying REQ5. OSPF provides similar functionality using satisfying REQ5. OSPF provides similar functionality using
concepts such as "Forwarding Address", but with more difficulty in concepts such as "Forwarding Address", but with more difficulty in
implementation and far less control of information propagation implementation and far less control of information propagation
scope. scope.
skipping to change at page 18, line 33 skipping to change at page 18, line 33
attribute. This is typically done to avoid ASN number collisions attribute. This is typically done to avoid ASN number collisions
between different data centers and also to provide a uniform between different data centers and also to provide a uniform
AS_PATH length to the WAN for purposes of WAN ECMP to Anycast AS_PATH length to the WAN for purposes of WAN ECMP to Anycast
prefixes originated in the topology. An implementation specific prefixes originated in the topology. An implementation specific
BGP feature typically called "Remove Private AS" is commonly used BGP feature typically called "Remove Private AS" is commonly used
to accomplish this. Depending on implementation, the feature to accomplish this. Depending on implementation, the feature
should strip a contiguous sequence of Private Use ASNs found in should strip a contiguous sequence of Private Use ASNs found in
AS_PATH attribute prior to advertising the path to a neighbor. AS_PATH attribute prior to advertising the path to a neighbor.
This assumes that all ASNs used for intra data center numbering This assumes that all ASNs used for intra data center numbering
are from the Private Use ranges. The process for stripping the are from the Private Use ranges. The process for stripping the
Private Use ASNs is not currently standardized, but most Private Use ASNs is not currently standardized, see
implementations commonly follow the logic described in this [I-D.mitchell-grow-remove-private-as]. However most
vendor's document [REMOVE-PRIVATE-AS]. implementations at least follow the logic described in this
vendor's document [VENDOR-REMOVE-PRIVATE-AS], which is enough for
the design specified.
o Originate a default route to the data center devices. This is the o Originate a default route to the data center devices. This is the
only place where default route can be originated, as route only place where default route can be originated, as route
summarization is risky for the "scale-out" topology. summarization is risky for the "scale-out" topology.
Alternatively, Border Routers may simply relay the default route Alternatively, Border Routers may simply relay the default route
learned from WAN routers. Advertising the default route from learned from WAN routers. Advertising the default route from
Border Routers requires that all Border Routers be fully connected Border Routers requires that all Border Routers be fully connected
to the WAN Routers upstream, to provide resistance to a single- to the WAN Routers upstream, to provide resistance to a single-
link failure causing the black-holing of traffic. To prevent link failure causing the black-holing of traffic. To prevent
black-holing in the situation when all of the EBGP sessions to the black-holing in the situation when all of the EBGP sessions to the
skipping to change at page 21, line 9 skipping to change at page 21, line 9
devices. The Border Routers may need to have wider fan-out to be devices. The Border Routers may need to have wider fan-out to be
able to connect to multitude of Tier-1 devices if route summarization able to connect to multitude of Tier-1 devices if route summarization
at Border Router level is implemented as described in Section 5.2.5. at Border Router level is implemented as described in Section 5.2.5.
If a device's hardware does not support wider ECMP, logical link- If a device's hardware does not support wider ECMP, logical link-
grouping (link-aggregation at layer 2) could be used to provide grouping (link-aggregation at layer 2) could be used to provide
"hierarchical" ECMP (Layer 3 ECMP followed by Layer 2 ECMP) to "hierarchical" ECMP (Layer 3 ECMP followed by Layer 2 ECMP) to
compensate for fan-out limitations. Such approach, however, compensate for fan-out limitations. Such approach, however,
increases the risk of flow polarization, as less entropy will be increases the risk of flow polarization, as less entropy will be
available to the second stage of ECMP. available to the second stage of ECMP.
Most BGP implementations declare paths to be equal from ECMP Most BGP implementations declare paths to be equal from an ECMP
perspective if they match up to and including step (e) in perspective if they match up to and including step (e) in
Section 9.1.2.2 of [RFC4271]. In the proposed network design there Section 9.1.2.2 of [RFC4271]. In the proposed network design there
is no underlying IGP, so all IGP costs are assumed to be zero or is no underlying IGP, so all IGP costs are assumed to be zero or
otherwise the same value across all paths and policies may be applied otherwise the same value across all paths and policies may be applied
as necessary to equalize BGP attributes that vary in vendor defaults, as necessary to equalize BGP attributes that vary in vendor defaults,
such as MED and origin code. For historical reasons it is also such as MED and origin code. For historical reasons it is also
useful to not use 0 as the equalized MED value; this and some other useful to not use 0 as the equalized MED value; this and some other
useful BGP information is available in [RFC4277] . Routing loops are useful BGP information is available in [RFC4277] . Routing loops are
unlikely due to the BGP best-path selection process which prefers unlikely due to the BGP best-path selection process which prefers
shorter AS_PATH length, and longer paths through the Tier-1 devices shorter AS_PATH length, and longer paths through the Tier-1 devices
skipping to change at page 21, line 31 skipping to change at page 21, line 31
also not possible. also not possible.
6.2. BGP ECMP over Multiple ASNs 6.2. BGP ECMP over Multiple ASNs
For application load balancing purposes it is desirable to have the For application load balancing purposes it is desirable to have the
same prefix advertised from multiple Tier-3 devices. From the same prefix advertised from multiple Tier-3 devices. From the
perspective of other devices, such a prefix would have BGP paths with perspective of other devices, such a prefix would have BGP paths with
different AS_PATH attribute values, while having the same AS_PATH different AS_PATH attribute values, while having the same AS_PATH
attribute lengths. Therefore, BGP implementations must support load attribute lengths. Therefore, BGP implementations must support load
sharing over above-mentioned paths. This feature is sometimes known sharing over above-mentioned paths. This feature is sometimes known
as "multipath relax" and effectively allows for ECMP to be done as "multipath relax" or "multipath multiple-as" and effectively
across different neighboring ASNs if all other attributes are equal allows for ECMP to be done across different neighboring ASNs if all
as described in the previous section. other attributes are equal as already described in the previous
section.
6.3. Weighted ECMP 6.3. Weighted ECMP
It may be desirable for the network devices to implement "weighted" It may be desirable for the network devices to implement "weighted"
ECMP, to be able to send more traffic over some paths in ECMP fan- ECMP, to be able to send more traffic over some paths in ECMP fan-
out. This could be helpful to compensate for failures in the network out. This could be helpful to compensate for failures in the network
and send more traffic over paths that have more capacity. The and send more traffic over paths that have more capacity. The
prefixes that require weighted ECMP would have to be injected using prefixes that require weighted ECMP would have to be injected using
remote BGP speaker (central agent) over a multihop session as remote BGP speaker (central agent) over a multihop session as
described further in Section 8.1. If support in implementations is described further in Section 8.1. If support in implementations is
skipping to change at page 23, line 26 skipping to change at page 23, line 26
In the proposed design the impact of BGP Minimum Route Advertisement In the proposed design the impact of BGP Minimum Route Advertisement
Interval (MRAI) timer (See section 9.2.1.1 of [RFC4271]) should be Interval (MRAI) timer (See section 9.2.1.1 of [RFC4271]) should be
considered. Per the standard it is required for BGP implementations considered. Per the standard it is required for BGP implementations
to space out consecutive BGP UPDATE messages by at least MRAI to space out consecutive BGP UPDATE messages by at least MRAI
seconds, which is often a configurable value. The initial BGP UPDATE seconds, which is often a configurable value. The initial BGP UPDATE
messages after an event carrying withdrawn routes are commonly not messages after an event carrying withdrawn routes are commonly not
affected by this timer. The MRAI timer may present significant affected by this timer. The MRAI timer may present significant
convergence delays when a BGP speaker "waits" for the new path to be convergence delays when a BGP speaker "waits" for the new path to be
learned from its peers and has no local backup path information. learned from its peers and has no local backup path information.
In a Clos topology each EBGP speaker has either one path or N paths In a Clos topology each EBGP speaker typically has either one path
for the same prefix, where N is a significantly large number, e.g. (Tier-2 devices don't accept paths from other Tier-2 in the same
N=32 (the ECMP fan-out). Therefore, if a path fails there is either cluster due to same ASN) or N paths for the same prefix, where N is a
no backup path at all (e.g. from perspective of a Tier-2 switch significantly large number, e.g. N=32 (the ECMP fan-out to the next
losing link to a Tier-3 device), or the backup is readily available Tier). Therefore, if a link fails to another device from which a
in BGP Loc-RIB (e.g. from perspective of a Tier-2 device losing link path is received there is either no backup path at all (e.g. from
to a Tier-1 switch). In the former case, the BGP withdrawal perspective of a Tier-2 switch losing link to a Tier-3 device), or
announcement will propagate un-delayed and trigger re-convergence on the backup is readily available in BGP Loc-RIB (e.g. from perspective
affected devices. In the latter case, the best-path will be re- of a Tier-2 device losing link to a Tier-1 switch). In the former
evaluated and the local ECMP group corresponding to the new next-hop case, the BGP withdrawal announcement will propagate un-delayed and
set changed. If the BGP path was the best-path selected previously, trigger re-convergence on affected devices. In the latter case, the
an "implicit withdraw" will be sent via a BGP UPDATE message as best-path will be re-evaluated and the local ECMP group corresponding
described as Option b in Section 3.1 of [RFC4271] due to the BGP to the new next-hop set changed. If the BGP path was the best-path
AS_PATH attribute changing. selected previously, an "implicit withdraw" will be sent via a BGP
UPDATE message as described as Option b in Section 3.1 of [RFC4271]
due to the BGP AS_PATH attribute changing.
7.3. Impact of Clos Topology Fan-outs 7.3. Impact of Clos Topology Fan-outs
Clos topology has large fan-outs, which may impact the "Up->Down" Clos topology has large fan-outs, which may impact the "Up->Down"
convergence in some cases, as described in this section. In a convergence in some cases, as described in this section. In a
situation when a link between Tier-3 and Tier-2 device fails, the situation when a link between Tier-3 and Tier-2 device fails, the
Tier-2 device will send BGP UPDATE messages to all upstream Tier-1 Tier-2 device will send BGP UPDATE messages to all upstream Tier-1
devices, withdrawing the affected prefixes. The Tier-1 devices, in devices, withdrawing the affected prefixes. The Tier-1 devices, in
turn, will relay those messages to all downstream Tier-2 devices turn, will relay those messages to all downstream Tier-2 devices
(except for the originator). Tier-2 devices other than the one (except for the originator). Tier-2 devices other than the one
skipping to change at page 31, line 34 skipping to change at page 31, line 34
[RFC6774] Raszuk, R., Ed., Fernando, R., Patel, K., McPherson, D., [RFC6774] Raszuk, R., Ed., Fernando, R., Patel, K., McPherson, D.,
and K. Kumaki, "Distribution of Diverse BGP Paths", and K. Kumaki, "Distribution of Diverse BGP Paths",
RFC 6774, DOI 10.17487/RFC6774, November 2012, RFC 6774, DOI 10.17487/RFC6774, November 2012,
<http://www.rfc-editor.org/info/rfc6774>. <http://www.rfc-editor.org/info/rfc6774>.
[RFC6793] Vohra, Q. and E. Chen, "BGP Support for Four-Octet [RFC6793] Vohra, Q. and E. Chen, "BGP Support for Four-Octet
Autonomous System (AS) Number Space", RFC 6793, Autonomous System (AS) Number Space", RFC 6793,
DOI 10.17487/RFC6793, December 2012, DOI 10.17487/RFC6793, December 2012,
<http://www.rfc-editor.org/info/rfc6793>. <http://www.rfc-editor.org/info/rfc6793>.
[RFC7067] Dunbar, L., Eastlake 3rd, D., Perlman, R., and I.
Gashinsky, "Directory Assistance Problem and High-Level
Design Proposal", RFC 7067, DOI 10.17487/RFC7067, November
2013, <http://www.rfc-editor.org/info/rfc7067>.
[RFC7130] Bhatia, M., Ed., Chen, M., Ed., Boutros, S., Ed., [RFC7130] Bhatia, M., Ed., Chen, M., Ed., Boutros, S., Ed.,
Binderberger, M., Ed., and J. Haas, Ed., "Bidirectional Binderberger, M., Ed., and J. Haas, Ed., "Bidirectional
Forwarding Detection (BFD) on Link Aggregation Group (LAG) Forwarding Detection (BFD) on Link Aggregation Group (LAG)
Interfaces", RFC 7130, DOI 10.17487/RFC7130, February Interfaces", RFC 7130, DOI 10.17487/RFC7130, February
2014, <http://www.rfc-editor.org/info/rfc7130>. 2014, <http://www.rfc-editor.org/info/rfc7130>.
[RFC7196] Pelsser, C., Bush, R., Patel, K., Mohapatra, P., and O. [RFC7196] Pelsser, C., Bush, R., Patel, K., Mohapatra, P., and O.
Maennel, "Making Route Flap Damping Usable", RFC 7196, Maennel, "Making Route Flap Damping Usable", RFC 7196,
DOI 10.17487/RFC7196, May 2014, DOI 10.17487/RFC7196, May 2014,
<http://www.rfc-editor.org/info/rfc7196>. <http://www.rfc-editor.org/info/rfc7196>.
skipping to change at page 32, line 10 skipping to change at page 32, line 15
[I-D.ietf-idr-add-paths] [I-D.ietf-idr-add-paths]
Walton, D., Retana, A., Chen, E., and J. Scudder, Walton, D., Retana, A., Chen, E., and J. Scudder,
"Advertisement of Multiple Paths in BGP", draft-ietf-idr- "Advertisement of Multiple Paths in BGP", draft-ietf-idr-
add-paths-10 (work in progress), October 2014. add-paths-10 (work in progress), October 2014.
[I-D.ietf-idr-link-bandwidth] [I-D.ietf-idr-link-bandwidth]
Mohapatra, P. and R. Fernando, "BGP Link Bandwidth Mohapatra, P. and R. Fernando, "BGP Link Bandwidth
Extended Community", draft-ietf-idr-link-bandwidth-06 Extended Community", draft-ietf-idr-link-bandwidth-06
(work in progress), January 2013. (work in progress), January 2013.
[I-D.mitchell-grow-remove-private-as]
Mitchell, J., Rao, D., and R. Raszuk, "Private Autonomous
System (AS) Removal Requirements", draft-mitchell-grow-
remove-private-as-04 (work in progress), April 2015.
[CLOS1953] [CLOS1953]
Clos, C., "A Study of Non-Blocking Switching Networks: Clos, C., "A Study of Non-Blocking Switching Networks:
Bell System Technical Journal Vol. 32(2)", March 1953. Bell System Technical Journal Vol. 32(2)", March 1953.
[HADOOP] Apache, , "Apache HaDoop", August 2015, [HADOOP] Apache, , "Apache HaDoop", August 2015,
<https://hadoop.apache.org/>. <https://hadoop.apache.org/>.
[GREENBERG2009] [GREENBERG2009]
Greenberg, A., Hamilton, J., and D. Maltz, "The Cost of a Greenberg, A., Hamilton, J., and D. Maltz, "The Cost of a
Cloud: Research Problems in Data Center Networks", January Cloud: Research Problems in Data Center Networks", January
skipping to change at page 33, line 12 skipping to change at page 33, line 23
IEEE 802.3ad, , "IEEE Standard for Link aggregation for IEEE 802.3ad, , "IEEE Standard for Link aggregation for
parallel links", October 2000. parallel links", October 2000.
[ALLOWASIN] [ALLOWASIN]
Cisco Systems, , "Allowas-in Feature in BGP Configuration Cisco Systems, , "Allowas-in Feature in BGP Configuration
Example", February 2015, Example", February 2015,
<http://www.cisco.com/c/en/us/support/docs/ip/border- <http://www.cisco.com/c/en/us/support/docs/ip/border-
gateway-protocol-bgp/112236-allowas-in-bgp-config- gateway-protocol-bgp/112236-allowas-in-bgp-config-
example.html>. example.html>.
[REMOVE-PRIVATE-AS] [VENDOR-REMOVE-PRIVATE-AS]
Cisco Systems, , "Removing Private Autonomous System Cisco Systems, , "Removing Private Autonomous System
Numbers in BGP", August 2005, Numbers in BGP", August 2005,
<http://www.cisco.com/en/US/tech/tk365/ <http://www.cisco.com/en/US/tech/tk365/
technologies_tech_note09186a0080093f27.shtml>. technologies_tech_note09186a0080093f27.shtml>.
[CONDITIONALROUTE] [CONDITIONALROUTE]
Cisco Systems, , "Configuring and Verifying the BGP Cisco Systems, , "Configuring and Verifying the BGP
Conditional Advertisement Feature", August 2005, Conditional Advertisement Feature", August 2005,
<http://www.cisco.com/c/en/us/support/docs/ip/ <http://www.cisco.com/c/en/us/support/docs/ip/
border-gateway-protocol-bgp/16137-cond-adv.html>. border-gateway-protocol-bgp/16137-cond-adv.html>.
 End of changes. 15 change blocks. 
41 lines changed or deleted 62 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/