draft-ietf-nvo3-framework-04.txt   draft-ietf-nvo3-framework-05.txt 
Internet Engineering Task Force Marc Lasserre
Internet Draft Florin Balus
Intended status: Informational Alcatel-Lucent
Expires: July 2014
Thomas Morin
France Telecom Orange
Internet Engineering Task Force Marc Lasserre Nabil Bitar
Internet Draft Florin Balus Verizon
Intended status: Informational Alcatel-Lucent
Expires: May 2014
Thomas Morin
France Telecom Orange
Nabil Bitar
Verizon
Yakov Rekhter
Juniper
November 12, 2013
Framework for DC Network Virtualization
draft-ietf-nvo3-framework-04.txt
Abstract
This document provides a framework for Network Virtualization over
L3 (NVO3) and it defines a reference model along with logical
components required to design a solution.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 12, 2014.
Internet-Draft Framework for DC Network Virtualization November
2013
Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction..................................................3
1.1. General terminology......................................3
1.2. DC network architecture..................................6
2. Reference Models..............................................8
2.1. Generic Reference Model..................................8
2.2. NVE Reference Model.....................................11
2.3. NVE Service Types.......................................12
2.3.1. L2 NVE providing Ethernet LAN-like service.........12
2.3.2. L3 NVE providing IP/VRF-like service...............12
3. Functional components........................................12
3.1. Service Virtualization Components.......................12
3.1.1. Virtual Access Points (VAPs).......................12
3.1.2. Virtual Network Instance (VNI).....................13
3.1.3. Overlay Modules and VN Context.....................13
3.1.4. Tunnel Overlays and Encapsulation options..........14
3.1.5. Control Plane Components...........................14
3.1.5.1. Distributed vs Centralized Control Plane.........14
3.1.5.2. Auto-provisioning/Service discovery..............15
3.1.5.3. Address advertisement and tunnel mapping.........15
3.1.5.4. Overlay Tunneling................................16
3.2. Multi-homing............................................16
3.3. VM Mobility.............................................17
4. Key aspects of overlay networks..............................18
4.1. Pros & Cons.............................................18
4.2. Overlay issues to consider..............................20
4.2.1. Data plane vs Control plane driven.................20
4.2.2. Coordination between data plane and control plane..20
Internet-Draft Framework for DC Network Virtualization November
2013
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic...................................................20
4.2.4. Path MTU...........................................21
4.2.5. NVE location trade-offs............................22
4.2.6. Interaction between network overlays and underlays.23
5. Security Considerations......................................23
6. IANA Considerations..........................................24
7. References...................................................24
7.1. Normative References....................................24
7.2. Informative References..................................24
8. Acknowledgments..............................................24
1. Introduction
This document provides a framework for Data Center Network
Virtualization over Layer3 (L3) tunnels. This framework is intended
to aid in standardizing protocols and mechanisms to support large-
scale network virtualization for data centers.
[NVOPS] defines the rationale for using overlay networks in order to
build large multi-tenant data center networks. Compute, storage and
network virtualization are often used in these large data centers to
support a large number of communication domains and end systems.
This document provides reference models and functional components of
data center overlay networks as well as a discussion of technical
issues that have to be addressed.
1.1. General terminology
This document uses the following terminology:
NVO3 Network: An overlay network that provides an Layer2 (L2) or
Layer3 (L3) service to Tenant Systems over an L3 underlay network
using the architecture and protocols as defined by the NVO3 Working
Group.
Network Virtualization Edge (NVE). An NVE is the network entity that
sits at the edge of an underlay network and implements L2 and/or L3
network virtualization functions. The network-facing side of the NVE
uses the underlying L3 network to tunnel tenant frames to and from
other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a
Internet-Draft Framework for DC Network Virtualization November
2013
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
Virtual Network (VN): A VN is a logical abstraction of a physical
network that provides L2 or L3 network services to a set of Tenant
Systems. A VN is also known as a Closed User Group (CUG).
Virtual Network Instance (VNI): A specific instance of a VN from the
perspective of an NVE.
Virtual Network Context (VN Context) Identifier: Field in overlay
encapsulation header that identifies the specific VN the packet
belongs to. The egress NVE uses the VN Context identifier to deliver
the packet to the correct Tenant System. The VN Context identifier
can be a locally significant identifier or a globally unique
identifier.
Underlay or Underlying Network: The network that provides the
connectivity among NVEs and over which NVO3 packets are tunneled,
where an NVO3 packet carries an NVO3 overlay header followed by a
tenant packet. The Underlay Network does not need to be aware that
it is carrying NVO3 packets. Addresses on the Underlay Network
appear as "outer addresses" in encapsulated NVO3 packets. In
general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is IP.
Data Center (DC): A physical complex housing physical servers,
network switches and routers, network service appliances and
networked storage. The purpose of a Data Center is to provide
application, compute and/or storage services. One such service is
virtualized infrastructure data center services, also known as
Infrastructure as a Service.
Virtual Data Center (Virtual DC): A container for virtualized
compute, storage and network services. A Virtual DC is associated
with a single tenant, and can contain multiple VNs and Tenant
Systems connected to one or more of these VNs.
Virtual machine (VM): A software implementation of a physical
machine that runs programs as if they were executing on a physical,
non-virtualized machine. Applications (generally) do not know they
are running on a VM as opposed to running on a "bare metal" host or
server, though some systems provide a para-virtualization
Internet-Draft Framework for DC Network Virtualization November Yakov Rekhter
2013 Juniper
environment that allows an operating system or application to be January 20, 2014
aware of the presences of virtualization for optimization purposes.
Hypervisor: Software running on a server that allows multiple VMs to Framework for DC Network Virtualization
run on the same physical server. The hypervisor manages and provides draft-ietf-nvo3-framework-05.txt
shared compute/memory/storage and network connectivity to the VMs
that it hosts. Hypervisors often embed a Virtual Switch (see below).
Server: A physical end host machine that runs user applications. A Abstract
standalone (or "bare metal") server runs a conventional operating
system hosting a single-tenant application. A virtualized server
runs a hypervisor supporting one or more VMs.
Virtual Switch (vSwitch): A function within a Hypervisor (typically This document provides a framework for Network Virtualization
implemented in software) that provides similar forwarding services Overlays (NVO3) and it defines a reference model along with logical
to a physical Ethernet switch. A vSwitch forwards Ethernet frames components required to design a solution.
between VMs running on the same server, or between a VM and a
physical NIC card connecting the server to a physical Ethernet
switch or router. A vSwitch also enforces network isolation between
VMs that by policy are not permitted to communicate with each other
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
enabled on the host server.
Tenant: The customer using a virtual network and any associated Status of this Memo
resources (e.g., compute, storage and network). A tenant could be
an enterprise, or a department/organization within an enterprise.
Tenant System: A physical or virtual system that can play the role This Internet-Draft is submitted in full conformance with the
of a host, or a forwarding element such as a router, switch, provisions of BCP 78 and BCP 79.
firewall, etc. It belongs to a single tenant and connects to one or
more VNs of that tenant.
Tenant Separation: Tenant Separation refers to isolating traffic of Internet-Drafts are working documents of the Internet Engineering
different tenants such that traffic from one tenant is not visible Task Force (IETF). Note that other groups may also distribute
to or delivered to another tenant, except when allowed by policy. working documents as Internet-Drafts. The list of current Internet-
Tenant Separation also refers to address space separation, whereby Drafts is at http://datatracker.ietf.org/drafts/current/.
different tenants can use the same address space without conflict.
Virtual Access Points (VAPs): A logical connection point on the NVE Internet-Drafts are draft documents valid for a maximum of six
for connecting a Tenant System to a virtual network. Tenant Systems months and may be updated, replaced, or obsoleted by other documents
connect to VNIs at an NVE through VAPs. VAPs can be physical ports at any time. It is inappropriate to use Internet-Drafts as
or virtual ports identified through logical interface identifiers reference material or to cite them other than as "work in progress."
(e.g., VLAN ID, internal vSwitch Interface ID connected to a VM).
Internet-Draft Framework for DC Network Virtualization November This Internet-Draft will expire on July 20, 2014.
2013
End Device: A physical device that connects directly to the DC Copyright Notice
Underlay Network. This is in contrast to a Tenant System, which
connects to a corresponding tenant VN. An End Device is administered
by the DC operator rather than a tenant, and is part of the DC
infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
Network Virtualization Authority (NVA): Entity that provides Copyright (c) 2014 IETF Trust and the persons identified as the
reachability and forwarding information to NVEs. document authors. All rights reserved.
1.2. DC network architecture This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
A generic architecture for Data Centers is depicted in Figure 1: Table of Contents
Internet-Draft Framework for DC Network Virtualization November 1. Introduction..................................................3
2013 1.1. General terminology......................................3
1.2. DC network architecture..................................6
2. Reference Models..............................................8
2.1. Generic Reference Model..................................8
2.2. NVE Reference Model.....................................10
2.3. NVE Service Types.......................................11
2.3.1. L2 NVE providing Ethernet LAN-like service.........11
2.3.2. L3 NVE providing IP/VRF-like service...............11
3. Functional components........................................11
3.1. Service Virtualization Components.......................11
3.1.1. Virtual Access Points (VAPs).......................11
3.1.2. Virtual Network Instance (VNI).....................12
3.1.3. Overlay Modules and VN Context.....................12
3.1.4. Tunnel Overlays and Encapsulation options..........13
3.1.5. Control Plane Components...........................13
3.1.5.1. Distributed vs Centralized Control Plane.........13
3.1.5.2. Auto-provisioning/Service discovery..............14
3.1.5.3. Address advertisement and tunnel mapping.........14
3.1.5.4. Overlay Tunneling................................15
3.2. Multi-homing............................................15
3.3. VM Mobility.............................................16
4. Key aspects of overlay networks..............................17
4.1. Pros & Cons.............................................17
4.2. Overlay issues to consider..............................18
4.2.1. Data plane vs Control plane driven.................18
4.2.2. Coordination between data plane and control plane..19
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic...................................................19
4.2.4. Path MTU............................................20
4.2.5. NVE location trade-offs.............................21
4.2.6. Interaction between network overlays and underlays..21
5. Security Considerations.......................................22
6. IANA Considerations...........................................22
7. References....................................................23
7.1. Informative References...................................23
8. Acknowledgments...............................................23
,---------. 1. Introduction
,' `.
( IP/MPLS WAN )
`. ,'
`-+------+'
\ /
+--------+ +--------+
| DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| /
.--. .--.
( ' '.--.
.-.' Intra-DC '
( network )
( .'-'
'--'._.'. )\ \
/ / '--' \ \
/ / | | \ \
+--------+ +--------+ +--------+
| access | | access | | access |
| switch | | switch | | switch |
+--------+ +--------+ +--------+
/ \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------'
: End : : End : : End : : End :
: Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------'
Figure 1 : A Generic Architecture for Data Centers This document provides a framework for Data Center (DC) Network
Virtualization over Layer3 (L3) tunnels. This framework is intended
to aid in standardizing protocols and mechanisms to support large-
scale network virtualization for data centers.
An example of multi-tier DC network architecture is presented in [NVOPS] defines the rationale for using overlay networks in order to
Figure 1. It provides a view of physical components inside a DC. build large multi-tenant data center networks. Compute, storage and
network virtualization are often used in these large data centers to
support a large number of communication domains and end systems.
A DC network is usually composed of intra-DC networks and network This document provides reference models and functional components of
services, and inter-DC network and network connectivity services. data center overlay networks as well as a discussion of technical
issues that have to be addressed.
DC networking elements can act as strict L2 switches and/or provide 1.1. General terminology
IP routing capabilities, including network service virtualization.
In some DC architectures, some tier layers could provide L2 and/or This document uses the following terminology:
L3 services. In addition, some tier layers may be collapsed, and
Internet connectivity, inter-DC connectivity and VPN support may be
handled by a smaller number of nodes. Nevertheless, one can assume
Internet-Draft Framework for DC Network Virtualization November NVO3 Network: An overlay network that provides a Layer2 (L2) or
2013 Layer3 (L3) service to Tenant Systems over an L3 underlay network
using the architecture and protocols as defined by the NVO3 Working
Group.
that the network functional blocks in a DC fit in the architecture Network Virtualization Edge (NVE). An NVE is the network entity that
depicted in Figure 1. sits at the edge of an underlay network and implements L2 and/or L3
network virtualization functions. The network-facing side of the NVE
uses the underlying L3 network to tunnel tenant frames to and from
other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
The following components can be present in a DC: Virtual Network (VN): A VN is a logical abstraction of a physical
network that provides L2 or L3 network services to a set of Tenant
Systems. A VN is also known as a Closed User Group (CUG).
o Access switch: Hardware-based Ethernet switch aggregating all Virtual Network Instance (VNI): A specific instance of a VN from the
Ethernet links from the End Devices in a rack representing the perspective of an NVE.
entry point in the physical DC network for the hosts. It may
also provide routing functionality, virtual IP network
connectivity, or Layer2 tunneling over IP for instance. Access
switches are usually multi-homed to aggregation switches in the
Intra-DC network. A typical example of an access switch is a
Top of Rack (ToR) switch. Other deployment scenarios may use an
intermediate Blade Switch before the ToR, or an EoR (End of
Row) switch, to provide similar function as a ToR.
o Intra-DC Network: Network composed of high capacity core nodes Virtual Network Context (VN Context) Identifier: Field in overlay
(Ethernet switches/routers). Core nodes may provide virtual encapsulation header that identifies the specific VN the packet
Ethernet bridging and/or IP routing services. belongs to. The egress NVE uses the VN Context identifier to deliver
the packet to the correct Tenant System. The VN Context identifier
can be a locally significant identifier or a globally unique
identifier.
o DC Gateway (DC GW): Gateway to the outside world providing DC Underlay or Underlying Network: The network that provides the
Interconnect and connectivity to Internet and VPN customers. In connectivity among NVEs and over which NVO3 packets are tunneled,
the current DC network model, this may be simply a router where an NVO3 packet carries an NVO3 overlay header followed by a
connected to the Internet and/or an IP Virtual Private Network tenant packet. The Underlay Network does not need to be aware that
(VPN)/L2VPN PE. Some network implementations may dedicate DC it is carrying NVO3 packets. Addresses on the Underlay Network
GWs for different connectivity types (e.g., a DC GW for appear as "outer addresses" in encapsulated NVO3 packets. In
Internet, and another for VPN). general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is IP.
Note that End Devices may be single or multi-homed to access Data Center (DC): A physical complex housing physical servers,
switches. network switches and routers, network service appliances and
networked storage. The purpose of a Data Center is to provide
application, compute and/or storage services. One such service is
virtualized infrastructure data center services, also known as
Infrastructure as a Service.
2. Reference Models Virtual Data Center (Virtual DC): A container for virtualized
compute, storage and network services. A Virtual DC is associated
with a single tenant, and can contain multiple VNs and Tenant
Systems connected to one or more of these VNs.
2.1. Generic Reference Model Virtual machine (VM): A software implementation of a physical
machine that runs programs as if they were executing on a physical,
non-virtualized machine. Applications (generally) do not know they
are running on a VM as opposed to running on a "bare metal" host or
server, though some systems provide a para-virtualization
environment that allows an operating system or application to be
aware of the presence of virtualization for optimization purposes.
Figure 2 depicts a DC reference model for network virtualization Hypervisor: Software running on a server that allows multiple VMs to
using L3 (IP/MPLS) overlays where NVEs provide a logical run on the same physical server. The hypervisor manages and provides
interconnect between Tenant Systems that belong to a specific VN. shared compute/memory/storage and network connectivity to the VMs
that it hosts. Hypervisors often embed a Virtual Switch (see below).
Internet-Draft Framework for DC Network Virtualization November Server: A physical end host machine that runs user applications. A
2013 standalone (or "bare metal") server runs a conventional operating
system hosting a single-tenant application. A virtualized server
runs a hypervisor supporting one or more VMs.
+--------+ +--------+ Virtual Switch (vSwitch): A function within a Hypervisor (typically
| Tenant +--+ +----| Tenant | implemented in software) that provides similar forwarding services
| System | | (') | System | to a physical Ethernet switch. A vSwitch forwards Ethernet frames
+--------+ | ................. ( ) +--------+ between VMs running on the same server, or between a VM and a
| +---+ +---+ (_) physical NIC card connecting the server to a physical Ethernet
+--|NVE|---+ +---|NVE|-----+ switch or router. A vSwitch also enforces network isolation between
+---+ | | +---+ VMs that by policy are not permitted to communicate with each other
/ . +-----+ . (e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
/ . +--| NVA | . enabled on the host server.
/ . | +-----+ .
| . | .
| . | L3 Overlay +--+--++--------+
+--------+ | . | Network | NVE || Tenant |
| Tenant +--+ . | | || System |
| System | . \ +---+ +--+--++--------+
+--------+ .....|NVE|.........
+---+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
Figure 2 : Generic reference model for DC network virtualization Tenant: The customer using a virtual network and any associated
over a Layer3 (IP) infrastructure resources (e.g., compute, storage and network). A tenant could be
an enterprise, or a department/organization within an enterprise.
In order to get reachability information, NVEs may exchange Tenant System: A physical or virtual system that can play the role
information directly between themselves via a protocol. In this of a host, or a forwarding element such as a router, switch,
firewall, etc. It belongs to a single tenant and connects to one or
more VNs of that tenant.
Internet-Draft Framework for DC Network Virtualization November Tenant Separation: Tenant Separation refers to isolating traffic of
2013 different tenants such that traffic from one tenant is not visible
to or delivered to another tenant, except when allowed by policy.
Tenant Separation also refers to address space separation, whereby
different tenants can use the same address space without conflict.
case, a control plane module resides in every NVE. This is how Virtual Access Points (VAPs): A logical connection point on the NVE
routing control plane modules are implemented in routers for for connecting a Tenant System to a virtual network. Tenant Systems
instance. connect to VNIs at an NVE through VAPs. VAPs can be physical ports
or virtual ports identified through logical interface identifiers
(e.g., VLAN ID, internal vSwitch Interface ID connected to a VM).
It is also possible for NVEs to communicate with an external Network End Device: A physical device that connects directly to the DC
Virtualization Authority (NVA) to obtain reachability and forwarding Underlay Network. This is in contrast to a Tenant System, which
information. In this case, a protocol is used between NVEs and connects to a corresponding tenant VN. An End Device is administered
NVA(s) to exchange information. OpenFlow [OF] is one example of such by the DC operator rather than a tenant, and is part of the DC
a protocol. infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
It should be noted that NVAs may be organized in clusters for Network Virtualization Authority (NVA): Entity that provides
redundancy and scalability and can appear as one logically reachability and forwarding information to NVEs.
centralized controller. In this case, inter-NVA communication is
necessary to synchronize state among nodes within a cluster or share
information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information
exchanged across clusters.
A Tenant System can be attached to an NVE in several ways: 1.2. DC network architecture
- locally, by being co-located in the same End Device A generic architecture for Data Centers is depicted in Figure 1:
- remotely, via a point-to-point connection or a switched network ,---------.
,' `.
( IP/MPLS WAN )
`. ,'
`-+------+'
\ /
+--------+ +--------+
| DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| /
.--. .--.
( ' '.--.
.-.' Intra-DC '
( network )
( .'-'
'--'._.'. )\ \
/ / '--' \ \
/ / | | \ \
+--------+ +--------+ +--------+
| access | | access | | access |
| switch | | switch | | switch |
+--------+ +--------+ +--------+
/ \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------'
: End : : End : : End : : End :
: Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------'
When an NVE is co-located with a Tenant System, the state of the Figure 1 : A Generic Architecture for Data Centers
Tenant System can be provided without protocol assistance. For
instance, the operational status of a VM can be communicated via a
local API. When an NVE is remotely connected to a Tenant System, the
state of the Tenant System or NVE needs to be exchanged directly or
via a management entity, using a control plane protocol or API, or
directly via a dataplane protocol.
The functional components in Figure 2 do not necessarily map An example of multi-tier DC network architecture is presented in
directly to the physical components described in Figure 1. For Figure 1. It provides a view of physical components inside a DC.
example, an End Device can be a server blade with VMs and a virtual
switch. A VM can be a Tenant System and the NVE functions may be
performed by the host server. In this case, the Tenant System and
NVE function are co-located. Another example is the case where the
End Device is the Tenant System, and the NVE function can be
implemented by the connected ToR. In this case, the Tenant System
and NVE function are not co-located.
Underlay nodes utilize L3 technologies to interconnect NVE nodes. A DC network is usually composed of intra-DC networks and network
These nodes perform forwarding based on outer L3 header information, services, and inter-DC network and network connectivity services.
and generally do not maintain per tenant-service state albeit some
Internet-Draft Framework for DC Network Virtualization November DC networking elements can act as strict L2 switches and/or provide
2013 IP routing capabilities, including network service virtualization.
applications (e.g., multicast) may require control plane or In some DC architectures, some tier layers could provide L2 and/or
forwarding plane information that pertain to a tenant, group of L3 services. In addition, some tier layers may be collapsed, and
tenants, tenant service or a set of services that belong to one or Internet connectivity, inter-DC connectivity and VPN support may be
more tenants. Mechanisms to control the amount of state maintained handled by a smaller number of nodes. Nevertheless, one can assume
in the underlay may be needed. that the network functional blocks in a DC fit in the architecture
depicted in Figure 1.
2.2. NVE Reference Model The following components can be present in a DC:
Figure 3 depicts the NVE reference model. One or more VNIs can be - Access switch: Hardware-based Ethernet switch aggregating all
instantiated on an NVE. A Tenant System interfaces with a Ethernet links from the End Devices in a rack representing the
corresponding VNI via a VAP. An overlay module provides tunneling entry point in the physical DC network for the hosts. It may also
overlay functions (e.g., encapsulation and decapsulation of tenant provide routing functionality, virtual IP network connectivity, or
traffic, tenant identification and mapping, etc.). Layer2 tunneling over IP for instance. Access switches are usually
multi-homed to aggregation switches in the Intra-DC network. A
typical example of an access switch is a Top of Rack (ToR) switch.
Other deployment scenarios may use an intermediate Blade Switch
before the ToR, or an EoR (End of Row) switch, to provide similar
functions to a ToR.
+-------- L3 Network -------+ - Intra-DC Network: Network composed of high capacity core nodes
| | (Ethernet switches/routers). Core nodes may provide virtual
| Tunnel Overlay | Ethernet bridging and/or IP routing services.
+------------+---------+ +---------+------------+
| +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| |
| | | | | |
| +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+
| | | |
| | | |
Tenant Systems Tenant Systems
Figure 3 : Generic NVE reference model - DC Gateway (DC GW): Gateway to the outside world providing DC
Interconnect and connectivity to Internet and VPN customers. In
the current DC network model, this may be simply a router
connected to the Internet and/or an IP Virtual Private Network
(VPN)/L2VPN PE. Some network implementations may dedicate DC GWs
for different connectivity types (e.g., a DC GW for Internet, and
another for VPN).
Note that some NVE functions (e.g., data plane and control plane Note that End Devices may be single or multi-homed to access
functions) may reside in one device or may be implemented separately switches.
Internet-Draft Framework for DC Network Virtualization November 2. Reference Models
2013
in different devices. In addition, NVE functions can be implemented 2.1. Generic Reference Model
in a hierarchical fashion. For instance, an End Device can act as an
NVE Spoke, while an access switch can act as an NVE hub.
2.3. NVE Service Types Figure 2 depicts a DC reference model for network virtualization
overlay where NVEs provide a logical interconnect between Tenant
Systems that belong to a specific VN.
An NVE provides different types of virtualized network services to +--------+ +--------+
multiple tenants, i.e. an L2 service or an L3 service. Note that an | Tenant +--+ +----| Tenant |
NVE may be capable of providing both L2 and L3 services for a | System | | (') | System |
tenant. This section defines the service types and associated +--------+ | ................. ( ) +--------+
attributes. | +---+ +---+ (_)
+--|NVE|---+ +---|NVE|-----+
+---+ | | +---+
/ . +-----+ .
/ . +--| NVA |--+ .
/ . | +-----+ \ .
| . | \ .
| . | Overlay +--+--++--------+
+--------+ | . | Network | NVE || Tenant |
| Tenant +--+ . | | || System |
| System | . \ +---+ +--+--++--------+
+--------+ .....|NVE|.........
+---+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
2.3.1. L2 NVE providing Ethernet LAN-like service Figure 2 : Generic reference model for DC network virtualization
overlay
An L2 NVE implements Ethernet LAN emulation, an Ethernet based In order to obtain reachability information, NVEs may exchange
multipoint service similar to an IETF VPLS or EVPN service, where information directly between themselves via a control plane
the Tenant Systems appear to be interconnected by a LAN environment protocol. In this case, a control plane module resides in every NVE.
over an L3 overlay. As such, an L2 NVE provides per-tenant virtual
switching instance (L2 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant MAC frames across the underlay. Note that
the control plane for an L2 NVE could be implemented locally on the
NVE or in a separate control entity.
2.3.2. L3 NVE providing IP/VRF-like service It is also possible for NVEs to communicate with an external Network
Virtualization Authority (NVA) to obtain reachability and forwarding
information. In this case, a protocol is used between NVEs and
NVA(s) to exchange information. OpenFlow [OF] is one example of such
a protocol.
An L3 NVE provides Virtualized IP forwarding service, similar to It should be noted that NVAs may be organized in clusters for
IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service redundancy and scalability and can appear as one logically
definition perspective. That is, an L3 NVE provides per-tenant centralized controller. In this case, inter-NVA communication is
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling necessary to synchronize state among nodes within a cluster or share
encapsulation of tenant IP packets across the underlay. Note that information across clusters. The information exchanged between NVAs
routing could be performed locally on the NVE or in a separate of the same cluster could be different from the information
control entity. exchanged across clusters.
3. Functional components A Tenant System can be attached to an NVE in several ways:
This section decomposes the Network Virtualization architecture into - locally, by being co-located in the same End Device
functional components described in Figure 3 to make it easier to
discuss solution options for these components.
3.1. Service Virtualization Components - remotely, via a point-to-point connection or a switched network
3.1.1. Virtual Access Points (VAPs) When an NVE is co-located with a Tenant System, the state of the
Tenant System can be determined without protocol assistance. For
instance, the operational status of a VM can be communicated via a
local API. When an NVE is remotely connected to a Tenant System, the
state of the Tenant System or NVE needs to be exchanged directly or
via a management entity, using a control plane protocol or API, or
directly via a dataplane protocol.
Tenant Systems are connected to VNIs through Virtual Access Points The functional components in Figure 2 do not necessarily map
(VAPs). directly to the physical components described in Figure 1. For
example, an End Device can be a server blade with VMs and a virtual
switch. A VM can be a Tenant System and the NVE functions may be
performed by the host server. In this case, the Tenant System and
NVE function are co-located. Another example is the case where the
End Device is the Tenant System, and the NVE function can be
implemented by the connected ToR. In this case, the Tenant System
and NVE function are not co-located.
Internet-Draft Framework for DC Network Virtualization November Underlay nodes utilize L3 technologies to interconnect NVE nodes.
2013 These nodes perform forwarding based on outer L3 header information,
and generally do not maintain per tenant-service state albeit some
applications (e.g., multicast) may require control plane or
forwarding plane information that pertain to a tenant, group of
tenants, tenant service or a set of services that belong to one or
more tenants. Mechanisms to control the amount of state maintained
in the underlay may be needed.
VAPs can be physical ports or virtual ports identified through 2.2. NVE Reference Model
logical interface identifiers (e.g., VLAN ID, internal vSwitch
Interface ID connected to a VM).
3.1.2. Virtual Network Instance (VNI) Figure 3 depicts the NVE reference model. One or more VNIs can be
instantiated on an NVE. A Tenant System interfaces with a
corresponding VNI via a VAP. An overlay module provides tunneling
overlay functions (e.g., encapsulation and decapsulation of tenant
traffic, tenant identification and mapping, etc.).
A VNI is a specific VN instance on an NVE. Each VNI defines a +-------- L3 Network -------+
forwarding context that contains reachability information and | |
policies. | Tunnel Overlay |
+------------+---------+ +---------+------------+
| +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| |
| | | | | |
| +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+
| | | |
| | | |
Tenant Systems Tenant Systems
3.1.3. Overlay Modules and VN Context Figure 3 : Generic NVE reference model
Mechanisms for identifying each tenant service are required to allow Note that some NVE functions (e.g., data plane and control plane
the simultaneous overlay of multiple tenant services over the same functions) may reside in one device or may be implemented separately
underlay L3 network topology. In the data plane, each NVE, upon in different devices. In addition, NVE functions can be implemented
sending a tenant packet, must be able to encode the VN Context for in a hierarchical fashion. For instance, an End Device can act as an
the destination NVE in addition to the L3 tunneling information NVE Spoke, while an access switch can act as an NVE hub.
(e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the
tenant packet.
The Overlay module provides tunneling overlay functions: tunnel 2.3. NVE Service Types
initiation/termination as in the case of stateful tunnels (see
Section 3.1.4), and/or simply encapsulation/decapsulation of frames
from VAPs/L3 underlay.
In a multi-tenant context, tunneling aggregates frames from/to An NVE provides different types of virtualized network services to
different VNIs. Tenant identification and traffic demultiplexing are multiple tenants, i.e. an L2 service or an L3 service. Note that an
based on the VN Context identifier. NVE may be capable of providing both L2 and L3 services for a
tenant. This section defines the service types and associated
attributes.
The following approaches can be considered: 2.3.1. L2 NVE providing Ethernet LAN-like service
o One VN Context identifier per Tenant: A globally unique (on a An L2 NVE implements Ethernet LAN emulation, an Ethernet based
per-DC administrative domain) VN identifier is used to identify multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or
the corresponding VNI. Examples of such identifiers in existing EVPN [EVPN] service, where the Tenant Systems appear to be
technologies are IEEE VLAN IDs and ISID tags that identify interconnected by a LAN environment over an L3 overlay. As such, an
virtual L2 domains when using IEEE 802.1aq and IEEE 802.1ah, L2 NVE provides per-tenant virtual switching instance (L2 VNI), and
respectively. L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the
underlay. Note that the control plane for an L2 NVE could be
implemented locally on the NVE or in a separate control entity.
o One VN Context identifier per VNI: A per-VNI local value is 2.3.2. L3 NVE providing IP/VRF-like service
automatically generated by the egress NVE, or a control plane
associated with that NVE, and usually distributed by a control
Internet-Draft Framework for DC Network Virtualization November An L3 NVE provides Virtualized IP forwarding service, similar to
2013 IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service
definition perspective. That is, an L3 NVE provides per-tenant
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant IP packets across the underlay. Note that
routing could be performed locally on the NVE or in a separate
control entity.
plane protocol to all the related NVEs. An example of this 3. Functional components
approach is the use of per VRF MPLS labels in IP VPN [RFC4364].
o One VN Context identifier per VAP: A per-VAP local value is This section decomposes the Network Virtualization architecture into
assigned and usually distributed by a control plane protocol. functional components described in Figure 3 to make it easier to
An example of this approach is the use of per CE-PE MPLS labels discuss solution options for these components.
in IP VPN [RFC4364].
Note that when using one VN Context per VNI or per VAP, an 3.1. Service Virtualization Components
additional global identifier (e.g., a VN identifier or name) may be
used by the control plane to identify the Tenant context.
3.1.4. Tunnel Overlays and Encapsulation options 3.1.1. Virtual Access Points (VAPs)
Once the VN context identifier is added to the frame, an L3 Tunnel Tenant Systems are connected to VNIs through Virtual Access Points
encapsulation is used to transport the frame to the destination NVE. (VAPs).
Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS VAPs can be physical ports or virtual ports identified through
tunneling can be used. Tunneling could be stateless or stateful. logical interface identifiers (e.g., VLAN ID, internal vSwitch
Stateless tunneling simply entails the encapsulation of a tenant Interface ID connected to a VM).
packet with another header necessary for forwarding the packet
across the underlay (e.g., IP tunneling over an IP underlay).
Stateful tunneling on the other hand entails maintaining tunneling
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions should
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to carry the packet between an ingress NVE and an egress NVE across
the underlay. It should be noted that stateful tunneling, especially
when configuration is involved, does impose management overhead and
scale constraints. Thus, stateless tunneling is preferred when
feasible.
3.1.5. Control Plane Components 3.1.2. Virtual Network Instance (VNI)
3.1.5.1. Distributed vs Centralized Control Plane A VNI is a specific VN instance on an NVE. Each VNI defines a
forwarding context that contains reachability information and
policies.
A control/management plane entity can be centralized or distributed. 3.1.3. Overlay Modules and VN Context
Both approaches have been used extensively in the past. The routing
model of the Internet is a good example of a distributed approach.
Internet-Draft Framework for DC Network Virtualization November Mechanisms for identifying each tenant service are required to allow
2013 the simultaneous overlay of multiple tenant services over the same
underlay L3 network topology. In the data plane, each NVE, upon
sending a tenant packet, must be able to encode the VN Context for
the destination NVE in addition to the L3 tunneling information
(e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the
tenant packet.
Transport networks have usually used a centralized approach to The Overlay module provides tunneling overlay functions: tunnel
manage transport paths. initiation/termination as in the case of stateful tunnels (see
Section 3.1.4), and/or simply encapsulation/decapsulation of frames
from VAPs/L3 underlay.
It is also possible to combine the two approaches, i.e., using a In a multi-tenant context, tunneling aggregates frames from/to
hybrid model. A global view of network state can have many benefits different VNIs. Tenant identification and traffic demultiplexing are
but it does not preclude the use of distributed protocols within the based on the VN Context identifier.
network. Centralized models provide a facility to maintain global
state, and distribute that state to the network. When used in
combination with distributed protocols, greater network
efficiencies, improved reliability and robustness can be achieved.
Domain and/or deployment specific constraints define the balance
between centralized and distributed approaches.
3.1.5.2. Auto-provisioning/Service discovery The following approaches can be considered:
NVEs must be able to identify the appropriate VNI for each Tenant - One VN Context identifier per Tenant: A globally unique (on a per-
System. This is based on state information that is often provided by DC administrative domain) VN identifier is used to identify the
external entities. For example, in an environment where a VM is a corresponding VNI. Examples of such identifiers in existing
Tenant System, this information is provided by VM orchestration technologies are IEEE VLAN IDs and ISID tags that identify virtual
systems, since these are the only entities that have visibility of L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively.
which VM belongs to which tenant.
A mechanism for communicating this information to the NVE is - One VN Context identifier per VNI: A per-VNI local value is
required. VAPs have to be created and mapped to the appropriate VNI. automatically generated by the egress NVE, or a control plane
Depending upon the implementation, this control interface can be associated with that NVE, and usually distributed by a control
implemented using an auto-discovery protocol between Tenant Systems plane protocol to all the related NVEs. An example of this
and their local NVE or through management entities. In either case, approach is the use of per VRF MPLS labels in IP VPN [RFC4364].
appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant
isolation in the system.
NVEs may learn reachability information to VNIs on other NVEs via a - One VN Context identifier per VAP: A per-VAP local value is
control protocol exchanging such information among NVEs or via a assigned and usually distributed by a control plane protocol. An
management control entity. example of this approach is the use of per CE-PE MPLS labels in IP
VPN [RFC4364].
3.1.5.3. Address advertisement and tunnel mapping Note that when using one VN Context per VNI or per VAP, an
additional global identifier (e.g., a VN identifier or name) may be
used by the control plane to identify the Tenant context.
As traffic reaches an ingress NVE on a VAP, a lookup is performed to 3.1.4. Tunnel Overlays and Encapsulation options
determine which NVE or local VAP the packet needs to be sent to. If
the packet is to be sent to another NVE, the packet is encapsulated
with a tunnel header containing the destination information
(destination IP address or MPLS label) of the egress NVE.
Intermediate nodes (between the ingress and egress NVEs) switch or
route traffic based upon the tunnel destination information.
Internet-Draft Framework for DC Network Virtualization November Once the VN context identifier is added to the frame, an L3 Tunnel
2013 encapsulation is used to transport the frame to the destination NVE.
A key step in the above process consists of identifying the Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS
destination NVE the packet is to be tunneled to. NVEs are tunneling can be used. Tunneling could be stateless or stateful.
responsible for maintaining a set of forwarding or mapping tables Stateless tunneling simply entails the encapsulation of a tenant
that hold the bindings between destination VM and egress NVE packet with another header necessary for forwarding the packet
addresses. Several ways of populating these tables are possible: across the underlay (e.g., IP tunneling over an IP underlay).
control plane driven, management plane driven, or data plane driven. Stateful tunneling on the other hand entails maintaining tunneling
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions should
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to carry the packet between an ingress NVE and an egress NVE across
the underlay. It should be noted that stateful tunneling, especially
when configuration is involved, does impose management overhead and
scale constraints. Thus, stateless tunneling is preferred when
feasible.
When a control plane protocol is used to distribute address 3.1.5. Control Plane Components
reachability and tunneling information, the auto-
provisioning/Service discovery could be accomplished by the same
protocol. In this scenario, the auto-provisioning/Service discovery
could be combined with (be inferred from) the address advertisement
and associated tunnel mapping. Furthermore, a control plane protocol
that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling.
3.1.5.4. Overlay Tunneling 3.1.5.1. Distributed vs Centralized Control Plane
For overlay tunneling, and dependent upon the tunneling technology A control/management plane entity can be centralized or distributed.
used for encapsulating the Tenant System packets, it may be Both approaches have been used extensively in the past. The routing
sufficient to have one or more local NVE addresses assigned and used model of the Internet is a good example of a distributed approach.
in the source and destination fields of a tunneling encapsulating Transport networks have usually used a centralized approach to
header. Other information that is part of the manage transport paths.
tunneling encapsulation header may also need to be configured. In
certain cases, local NVE configuration may be sufficient while in
other cases, some tunneling related information may need to
be shared among NVEs. The information that needs to be shared will
be technology dependent. For instance, potential information could
include tunnel identity, encapsulation type, and/or tunnel
resources. In certain cases, such as when using IP multicast in the
underlay, tunnels may need to be established, interconnecting
NVEs. When tunneling information needs to be exchanged or shared
among NVEs, a control plane protocol may be required. For instance,
it may be necessary to provide active/standby status information
between NVEs, up/down status information, pruning/grafting
information for multicast tunnels, etc.
In addition, a control plane may be required to setup the tunnel It is also possible to combine the two approaches, i.e., using a
path for some tunneling technologies. This applies to both unicast hybrid model. A global view of network state can have many benefits
and multicast tunneling. but it does not preclude the use of distributed protocols within the
network. Centralized models provide a facility to maintain global
state, and distribute that state to the network. When used in
combination with distributed protocols, greater network
efficiencies, improved reliability and robustness can be achieved.
Domain and/or deployment specific constraints define the balance
between centralized and distributed approaches.
3.2. Multi-homing 3.1.5.2. Auto-provisioning/Service discovery
Multi-homing techniques can be used to increase the reliability of NVEs must be able to identify the appropriate VNI for each Tenant
an NVO3 network. It is also important to ensure that physical System. This is based on state information that is often provided by
external entities. For example, in an environment where a VM is a
Tenant System, this information is provided by VM orchestration
systems, since these are the only entities that have visibility of
which VM belongs to which tenant.
Internet-Draft Framework for DC Network Virtualization November A mechanism for communicating this information to the NVE is
2013 required. VAPs have to be created and mapped to the appropriate VNI.
Depending upon the implementation, this control interface can be
implemented using an auto-discovery protocol between Tenant Systems
and their local NVE or through management entities. In either case,
appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant
isolation in the system.
diversity in an NVO3 network is taken into account to avoid single NVEs may learn reachability information to VNIs on other NVEs via a
points of failure. control protocol that exchanges such information among NVEs, or via
a management control entity.
Multi-homing can be enabled in various nodes, from Tenant Systems 3.1.5.3. Address advertisement and tunnel mapping
into TORs, TORs into core switches/routers, and core nodes into DC
GWs.
The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP As traffic reaches an ingress NVE on a VAP, a lookup is performed to
routing as the means to re-route traffic upon failures techniques or determine which NVE or local VAP the packet needs to be sent to. If
on MPLS re-rerouting capabilities. the packet is to be sent to another NVE, the packet is encapsulated
with a tunnel header containing the destination information
(destination IP address or MPLS label) of the egress NVE.
Intermediate nodes (between the ingress and egress NVEs) switch or
route traffic based upon the tunnel destination information.
When a Tenant System is co-located with the NVE, the Tenant System A key step in the above process consists of identifying the
is effectively single homed to the NVE via a virtual port. When the destination NVE the packet is to be tunneled to. NVEs are
Tenant System and the NVE are separated, the Tenant System is responsible for maintaining a set of forwarding or mapping tables
connected to the NVE via a logical Layer2 (L2) construct such as a that hold the bindings between destination VM and egress NVE
VLAN and it can be multi-homed to various NVEs. An NVE may provide addresses. Several ways of populating these tables are possible:
an L2 service to the end system or an l3 service. An NVE may be control plane driven, management plane driven, or data plane driven.
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3
(L3). When an NVE provides an L2 service and is not co-located with
the end system, techniques such as Ethernet Link Aggregation Group
(LAG) or Spanning Tree Protocol (STP) can be used to switch traffic
between an end system and connected NVEs without creating
loops. Similarly, when the NVE provides L3 service, similar dual-
homing techniques can be used. When the NVE provides a L3 service to
the end system, it is possible that no dynamic routing protocol is
enabled between the end system and the NVE. The end system can be
multi-homed to multiple physically-separated L3 NVEs over multiple
interfaces. When one of the links connected to an NVE fails, the
other interfaces can be used to reach the end system.
External connectivity out of a DC can be handled by two or more DC When a control plane protocol is used to distribute address
gateways. Each gateway provides access to external networks such as reachability and tunneling information, the auto-
VPNs or the Internet. A gateway may be connected to two or more edge provisioning/Service discovery could be accomplished by the same
nodes in the external network for redundancy. When a connection to protocol. In this scenario, the auto-provisioning/Service discovery
an upstream node is lost, the alternative connection is used and the could be combined with (be inferred from) the address advertisement
failed route withdrawn. and associated tunnel mapping. Furthermore, a control plane protocol
that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling as
discussed in [RFC6820].
3.3. VM Mobility 3.1.5.4. Overlay Tunneling
In DC environments utilizing VM technologies, an important feature For overlay tunneling, and dependent upon the tunneling technology
is that VMs can move from one server to another server in the same used for encapsulating the Tenant System packets, it may be
or different L2 physical domains (within or across DCs) in a sufficient to have one or more local NVE addresses assigned and used
seamless manner. in the source and destination fields of a tunneling encapsulation
header. Other information that is part of the
tunneling encapsulation header may also need to be configured. In
certain cases, local NVE configuration may be sufficient while in
other cases, some tunneling related information may need to
be shared among NVEs. The information that needs to be shared will
be technology dependent. For instance, potential information could
include tunnel identity, encapsulation type, and/or tunnel
resources. In certain cases, such as when using IP multicast in the
underlay, tunnels which interconnect NVEs may need to be
established. When tunneling information needs to be exchanged or
shared among NVEs, a control plane protocol may be required. For
instance, it may be necessary to provide active/standby status
information between NVEs, up/down status information,
pruning/grafting information for multicast tunnels, etc.
Internet-Draft Framework for DC Network Virtualization November In addition, a control plane may be required to setup the tunnel
2013 path for some tunneling technologies. This applies to both unicast
and multicast tunneling.
A VM can be moved from one server to another in stopped or suspended 3.2. Multi-homing
state ("cold" VM mobility) or in running/active state ("hot" VM
mobility). With "hot" mobility, VM L2 and L3 addresses need to be
preserved. With "cold" mobility, it may be desired to preserve at
least VM L3 addresses.
Solutions to maintain connectivity while a VM is moved are necessary Multi-homing techniques can be used to increase the reliability of
in the case of "hot" mobility. This implies that connectivity among an NVO3 network. It is also important to ensure that physical
VMs is preserved. For instance, for L2 VNs, ARP caches are updated diversity in an NVO3 network is taken into account to avoid single
accordingly. points of failure.
Upon VM mobility, NVE policies that define connectivity among VMs Multi-homing can be enabled in various nodes, from Tenant Systems
must be maintained. into ToRs, ToRs into core switches/routers, and core nodes into DC
GWs.
During VM mobility, it is expected that the path to the VM's default The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP
gateway assures adequate performance to VM applications. routing as the means to re-route traffic upon failures techniques or
on MPLS re-rerouting capabilities.
4. Key aspects of overlay networks When a Tenant System is co-located with the NVE, the Tenant System
is effectively single homed to the NVE via a virtual port. When the
Tenant System and the NVE are separated, the Tenant System is
connected to the NVE via a logical Layer2 (L2) construct such as a
VLAN and it can be multi-homed to various NVEs. An NVE may provide
an L2 service to the end system or an l3 service. An NVE may be
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3
(L3). When an NVE provides an L2 service and is not co-located with
the end system, techniques such as Ethernet Link Aggregation Group
(LAG) or Spanning Tree Protocol (STP) can be used to switch traffic
between an end system and connected NVEs without creating
loops. Similarly, when the NVE provides L3 service, similar dual-
homing techniques can be used. When the NVE provides a L3 service to
the end system, it is possible that no dynamic routing protocol is
enabled between the end system and the NVE. The end system can be
multi-homed to multiple physically-separated L3 NVEs over multiple
interfaces. When one of the links connected to an NVE fails, the
other interfaces can be used to reach the end system.
The intent of this section is to highlight specific issues that External connectivity from a DC can be handled by two or more DC
proposed overlay solutions need to address. gateways. Each gateway provides access to external networks such as
VPNs or the Internet. A gateway may be connected to two or more edge
nodes in the external network for redundancy. When a connection to
an upstream node is lost, the alternative connection is used and the
failed route withdrawn.
4.1. Pros & Cons 3.3. VM Mobility
An overlay network is a layer of virtual network topology on top of In DC environments utilizing VM technologies, an important feature
the physical network. is that VMs can move from one server to another server in the same
or different L2 physical domains (within or across DCs) in a
seamless manner.
Overlay networks offer the following key advantages: A VM can be moved from one server to another in stopped or suspended
state ("cold" VM mobility) or in running/active state ("hot" VM
mobility). With "hot" mobility, VM L2 and L3 addresses need to be
preserved. With "cold" mobility, it may be desired to preserve at
least VM L3 addresses.
o Unicast tunneling state management and association of Tenant Solutions to maintain connectivity while a VM is moved are necessary
Systems reachability are handled at the edge of the network (at in the case of "hot" mobility. This implies that connectivity among
the NVE). Intermediate transport nodes are unaware of such VMs is preserved. For instance, for L2 VNs, ARP caches are updated
state. Note that when multicast is enabled in the underlay accordingly.
network to build multicast trees for tenant VNs, there would be
more state related to tenants in the underlay core network.
o Tunneling is used to aggregate traffic and hide tenant Upon VM mobility, NVE policies that define connectivity among VMs
addresses from the underlay network, and hence offer the must be maintained.
advantage of minimizing the amount of forwarding state required
within the underlay network
o Decoupling of the overlay addresses (MAC and IP) used by VMs During VM mobility, it is expected that the path to the VM's default
from the underlay network for tenant separation and separation gateway assures adequate performance to VM applications.
of the tenant address spaces from the underlay address space.
Internet-Draft Framework for DC Network Virtualization November 4. Key aspects of overlay networks
2013
o Support of a large number of virtual network identifiers The intent of this section is to highlight specific issues that
proposed overlay solutions need to address.
Overlay networks also create several challenges: 4.1. Pros & Cons
o Overlay networks have typically no control of underlay networks An overlay network is a layer of virtual network topology on top of
and lack underlay network information (e.g. underlay the physical network.
utilization):
o Overlay networks and/or their associated management Overlay networks offer the following key advantages:
entities typically probe the network to measure link or
path properties, such as available bandwidth or packet
loss rate. It is difficult to accurately evaluate network
properties. It might be preferable for the underlay
network to expose usage and performance information.
o - Unicast tunneling state management and association of Tenant
Miscommunication or lack of coordination between overlay Systems reachability are handled at the edge of the network (at
and underlay networks can lead to an inefficient usage of the NVE). Intermediate transport nodes are unaware of such
network resources. state. Note that when multicast is enabled in the underlay
network to build multicast trees for tenant VNs, there would be
more state related to tenants in the underlay core network.
o - Tunneling is used to aggregate traffic and hide tenant
When multiple overlays co-exist on top of a common underlay addresses from the underlay network, and hence offer the
network, the lack of coordination between overlays can advantage of minimizing the amount of forwarding state required
lead to performance issues and/or resource usage within the underlay network
inefficiencies.
o Traffic carried over an overlay may not traverse firewalls and - Decoupling of the overlay addresses (MAC and IP) used by VMs
NAT devices. from the underlay network for tenant separation and separation
of the tenant address spaces from the underlay address space.
o Multicast service scalability: Multicast support may be - Support of a large number of virtual network identifiers
required in the underlay network to address tenant flood
containment or efficient multicast handling. The underlay may
also be required to maintain multicast state on a per-tenant
basis, or even on a per-individual multicast flow of a given
tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
o Hash-based load balancing may not be optimal as the hash Overlay networks also create several challenges:
algorithm may not work well due to the limited number of
combinations of tunnel source and destination addresses. Other
NVO3 mechanisms may use additional entropy information than
source and destination addresses.
Internet-Draft Framework for DC Network Virtualization November - Overlay networks have typically no control of underlay networks
2013 and lack underlay network information (e.g. underlay
utilization):
4.2. Overlay issues to consider - Overlay networks and/or their associated management entities
typically probe the network to measure link or path
properties, such as available bandwidth or packet loss rate.
It is difficult to accurately evaluate network properties. It
might be preferable for the underlay network to expose usage
and performance information.
- Miscommunication or lack of coordination between overlay and
underlay networks can lead to an inefficient usage of network
resources.
- When multiple overlays co-exist on top of a common underlay
network, the lack of coordination between overlays can lead
to performance issues and/or resource usage inefficiencies.
4.2.1. Data plane vs Control plane driven - Traffic carried over an overlay may not traverse firewalls and
NAT devices.
In the case of an L2 NVE, it is possible to dynamically learn MAC - Multicast service scalability: Multicast support may be
addresses against VAPs. It is also possible that such addresses be required in the underlay network to address tenant flood
known and controlled via management or a control protocol for both containment or efficient multicast handling. The underlay may
L2 NVEs and L3 NVEs. Dynamic data plane learning implies that also be required to maintain multicast state on a per-tenant
flooding of unknown destinations be supported and hence implies that basis, or even on a per-individual multicast flow of a given
broadcast and/or multicast be supported or that ingress replication tenant. Ingress replication at the NVE eliminates that
be used as described in section 4.2.3. Multicasting in the underlay additional multicast state in the underlay core, but depending
network for dynamic learning may lead to significant scalability on the multicast traffic volume, it may cause inefficient use
limitations. Specific forwarding rules must be enforced to prevent of bandwidth.
loops from happening. This can be achieved using a spanning tree, a
shortest path tree, or a split-horizon mesh.
It should be noted that the amount of state to be distributed is - Hash-based load balancing may not be optimal as the hash
dependent upon network topology and the number of virtual machines. algorithm may not work well due to the limited number of
Different forms of caching can also be utilized to minimize state combinations of tunnel source and destination addresses. Other
distribution between the various elements. The control plane should NVO3 mechanisms may use additional entropy information than
not require an NVE to maintain the locations of all the Tenant source and destination addresses.
Systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane.
4.2.2. Coordination between data plane and control plane 4.2. Overlay issues to consider
For an L2 NVE, the NVE needs to be able to determine MAC addresses 4.2.1. Data plane vs Control plane driven
of the Tenant Systems connected via a VAP. This can be achieved via
dataplane learning or a control plane. For an L3 NVE, the NVE needs
to be able to determine IP addresses of the Tenant Systems connected
via a VAP.
In both cases, coordination with the NVE control protocol is needed In the case of an L2 NVE, it is possible to dynamically learn MAC
such that when the NVE determines that the set of addresses behind a addresses against VAPs. It is also possible that such addresses be
VAP has changed, it triggers the NVE control plane to distribute known and controlled via management or a control protocol for both
this information to its peers. L2 NVEs and L3 NVEs. Dynamic data plane learning implies that
flooding of unknown destinations be supported and hence implies that
broadcast and/or multicast be supported or that ingress replication
be used as described in section 4.2.3. Multicasting in the underlay
network for dynamic learning may lead to significant scalability
limitations. Specific forwarding rules must be enforced to prevent
loops from happening. This can be achieved using a spanning tree, a
shortest path tree, or a split-horizon mesh.
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic It should be noted that the amount of state to be distributed is
dependent upon network topology and the number of virtual machines.
Different forms of caching can also be utilized to minimize state
distribution between the various elements. The control plane should
not require an NVE to maintain the locations of all the Tenant
Systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane.
There are several options to support packet replication needed for 4.2.2. Coordination between data plane and control plane
broadcast, unknown unicast and multicast. Typical methods include:
Internet-Draft Framework for DC Network Virtualization November For an L2 NVE, the NVE needs to be able to determine MAC addresses
2013 of the Tenant Systems connected via a VAP. This can be achieved via
dataplane learning or a control plane. For an L3 NVE, the NVE needs
to be able to determine IP addresses of the Tenant Systems connected
via a VAP.
o Ingress replication In both cases, coordination with the NVE control protocol is needed
such that when the NVE determines that the set of addresses behind a
VAP has changed, it triggers the NVE control plane to distribute
this information to its peers.
o Use of underlay multicast trees 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic
There is a bandwidth vs state trade-off between the two approaches. There are several options to support packet replication needed for
Depending upon the degree of replication required (i.e. the number broadcast, unknown unicast and multicast. Typical methods include:
of hosts per group) and the amount of multicast state to maintain,
trading bandwidth for state should be considered.
When the number of hosts per group is large, the use of underlay - Ingress replication
multicast trees may be more appropriate. When the number of hosts is
small (e.g. 2-3) and/or the amount of multicast traffic is small,
ingress replication may not be an issue.
Depending upon the size of the data center network and hence the - Use of underlay multicast trees
number of (S,G) entries, but also the duration of multicast flows,
the use of underlay multicast trees can be a challenge.
When flows are well known, it is possible to pre-provision such There is a bandwidth vs state trade-off between the two approaches.
multicast trees. However, it is often difficult to predict Depending upon the degree of replication required (i.e. the number
application flows ahead of time, and hence programming of (S,G) of hosts per group) and the amount of multicast state to maintain,
entries for short-lived flows could be impractical. trading bandwidth for state should be considered.
A possible trade-off is to use in the underlay shared multicast When the number of hosts per group is large, the use of underlay
trees as opposed to dedicated multicast trees. multicast trees may be more appropriate. When the number of hosts is
small (e.g. 2-3) and/or the amount of multicast traffic is small,
ingress replication may not be an issue.
4.2.4. Path MTU Depending upon the size of the data center network and hence the
number of (S,G) entries, and also the duration of multicast flows,
the use of underlay multicast trees can be a challenge.
When using overlay tunneling, an outer header is added to the When flows are well known, it is possible to pre-provision such
original frame. This can cause the MTU of the path to the egress multicast trees. However, it is often difficult to predict
tunnel endpoint to be exceeded. application flows ahead of time, and hence programming of (S,G)
entries for short-lived flows could be impractical.
It is usually not desirable to rely on IP fragmentation for A possible trade-off is to use in the underlay shared multicast
performance reasons. Ideally, the interface MTU as seen by a Tenant trees as opposed to dedicated multicast trees.
System is adjusted such that no fragmentation is needed. TCP will
adjust its maximum segment size accordingly.
It is possible for the MTU to be configured manually or to be 4.2.4. Path MTU
discovered dynamically. Various Path MTU discovery techniques exist
in order to determine the proper MTU size to use:
o Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981] When using overlay tunneling, an outer header is added to the
original frame. This can cause the MTU of the path to the egress
tunnel endpoint to be exceeded.
o It is usually not desirable to rely on IP fragmentation for
Tenant Systems rely on ICMP messages to discover the MTU performance reasons. Ideally, the interface MTU as seen by a Tenant
of the end-to-end path to its destination. This method is System is adjusted such that no fragmentation is needed. TCP will
adjust its maximum segment size accordingly.
Internet-Draft Framework for DC Network Virtualization November It is possible for the MTU to be configured manually or to be
2013 discovered dynamically. Various Path MTU discovery techniques exist
in order to determine the proper MTU size to use:
not always possible, such as when traversing middle boxes - Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981]
(e.g. firewalls) which disable ICMP for security reasons
o Extended MTU Path Discovery techniques such as defined in - Tenant Systems rely on ICMP messages to discover the MTU of the
[RFC4821] end-to-end path to its destination. This method is not always
possible, such as when traversing middle boxes (e.g. firewalls)
which disable ICMP for security reasons
It is also possible to rely on the NVE to perform segmentation and - Extended MTU Path Discovery techniques such as defined in
reassembly operations without relying on the Tenant Systems to know [RFC4821]
about the end-to-end MTU. The assumption is that some hardware
assist is available on the NVE node to perform such SAR operations.
However, fragmentation by the NVE can lead to performance and
congestion issues due to TCP dynamics and might require new
congestion avoidance mechanisms from the underlay network [FLOYD].
Finally, the underlay network may be designed in such a way that the It is also possible to rely on the NVE to perform segmentation and
MTU can accommodate the extra tunneling and possibly additional NVO3 reassembly operations without relying on the Tenant Systems to know
header encapsulation overhead. about the end-to-end MTU. The assumption is that some hardware
assist is available on the NVE node to perform such SAR operations.
However, fragmentation by the NVE can lead to performance and
congestion issues due to TCP dynamics and might require new
congestion avoidance mechanisms from the underlay network [FLOYD].
4.2.5. NVE location trade-offs Finally, the underlay network may be designed in such a way that the
MTU can accommodate the extra tunneling and possibly additional NVO3
header encapsulation overhead.
In the case of DC traffic, traffic originated from a VM is native 4.2.5. NVE location trade-offs
Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements.
There are several criteria to consider when deciding where the NVE In the case of DC traffic, traffic originated from a VM is native
function should happen: Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements.
o Processing and memory requirements There are several criteria to consider when deciding where the NVE
function should happen:
o Datapath (e.g. lookups, filtering, - Processing and memory requirements
encapsulation/decapsulation)
o Control plane processing (e.g. routing, signaling, OAM) and - Datapath (e.g. lookups, filtering, encapsulation/decapsulation)
where specific control plane functions should be enabled
o FIB/RIB size - Control plane processing (e.g. routing, signaling, OAM) and
where specific control plane functions should be enabled
o Multicast support - FIB/RIB size
o Routing/signaling protocols - Multicast support
o Packet replication capability - Routing/signaling protocols
o Multicast FIB - Packet replication capability
Internet-Draft Framework for DC Network Virtualization November - Multicast FIB
2013
o Fragmentation support - Fragmentation support
o QoS support (e.g. marking, policing, queuing) - QoS support (e.g. marking, policing, queuing)
o Resiliency - Resiliency
4.2.6. Interaction between network overlays and underlays 4.2.6. Interaction between network overlays and underlays
When multiple overlays co-exist on top of a common underlay network, When multiple overlays co-exist on top of a common underlay network,
resources (e.g., bandwidth) should be provisioned to ensure that resources (e.g., bandwidth) should be provisioned to ensure that
traffic from overlays can be accommodated and QoS objectives can be traffic from overlays can be accommodated and QoS objectives can be
met. Overlays can have partially overlapping paths (nodes and met. Overlays can have partially overlapping paths (nodes and
links). links).
Each overlay is selfish by nature. It sends traffic so as to Each overlay is selfish by nature. It sends traffic so as to
optimize its own performance without considering the impact on other optimize its own performance without considering the impact on other
overlays, unless the underlay paths are traffic engineered on a per overlays, unless the underlay paths are traffic engineered on a per
overlay basis to avoid congestion of underlay resources. overlay basis to avoid congestion of underlay resources.
Better visibility between overlays and underlays, or generally Better visibility between overlays and underlays, or generally
coordination in placing overlay demand on an underlay network, may coordination in placing overlay demand on an underlay network, may
be achieved by providing mechanisms to exchange performance and be achieved by providing mechanisms to exchange performance and
liveliness information between the underlay and overlay(s) or the liveliness information between the underlay and overlay(s) or the
use of such information by a coordination system. Such information use of such information by a coordination system. Such information
may include: may include:
o Performance metrics (throughput, delay, loss, jitter) - Performance metrics (throughput, delay, loss, jitter)
o Cost metrics - Cost metrics
5. Security Considerations 5. Security Considerations
NVO3 solutions must at least consider and address the following: NVO3 solutions must at least consider and address the following:
. Secure and authenticated communication between an NVE and an - Secure and authenticated communication between an NVE and an NVE
NVE management system and/or control system. management system and/or control system.
. Isolation between tenant overlay networks. The use of per- - Isolation between tenant overlay networks. The use of per-tenant
tenant FIB tables (VNIs) on an NVE is essential. FIB tables (VNIs) on an NVE is essential.
. Security of any protocol used to carry overlay network - Security of any protocol used to carry overlay network
information. information.
. Preventing packets from reaching the wrong NVI, especially - Preventing packets from reaching the wrong NVI, especially during
during VM moves. VM moves.
Internet-Draft Framework for DC Network Virtualization November - It may desirable to restrict the types of information that can be
2013 exchanged between overlays and underlays (e.g. topology
information)
. It may desirable to restrict the types of information that can 6. IANA Considerations
be exchanged between overlays and underlays (e.g. topology
information)
6. IANA Considerations IANA does not need to take any action for this draft.
IANA does not need to take any action for this draft. 7. References
7. References 7.1. Informative References
7.1. Normative References [NVOPS] Narten, T. et al, "Problem Statement : Overlays for Network
Virtualization", draft-narten-nvo3-overlay-problem-
statement (work in progress)
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [OF] Open Networking Foundation, "OpenFlow Switch Specification
Requirement Levels", BCP 14, RFC 2119, March 1997. v1.4.0"
7.2. Informative References [FLOYD] Sally Floyd, Allyn Romanow, "Dynamics of TCP Traffic over
ATM Networks", IEEE JSAC, V. 13 N. 4, May 1995
[NVOPS] Narten, T. et al, "Problem Statement : Overlays for Network [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Virtualization", draft-narten-nvo3-overlay-problem- Networks (VPNs)", RFC 4364, February 2006.
statement (work in progress)
[OVCPREQ] Kreeger, L. et al, "Network Virtualization Overlay Control [RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS)
Protocol Requirements", draft-kreeger-nvo3-overlay-cp Using BGP for auto-discovery and Signaling", RFC4761,
(work in progress) January 2007
[FLOYD] Sally Floyd, Allyn Romanow, "Dynamics of TCP Traffic over [RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS)
ATM Networks", IEEE JSAC, V. 13 N. 4, May 1995 Using Label Distribution Protocol (LDP) Signaling",
RFC4762, January 2007
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private [EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft-
Networks (VPNs)", RFC 4364, February 2006. ietf-l2vpn-evpn (work in progress)
[RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990 [RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990
[RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981, [RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981,
August 1996 August 1996
[RFC4821] Mathis, M. et al, "Packetization Layer Path MTU [RFC4821] Mathis, M. et al, "Packetization Layer Path MTU
Discovery", RFC4821, March 2007 Discovery", RFC4821, March 2007
8. Acknowledgments [RFC6820] Narten, T. et al, "Address Resolution Problems in Large
Data Center Networks", RFC6820, January 2013
In addition to the authors the following people have contributed to 8. Acknowledgments
this document:
Internet-Draft Framework for DC Network Virtualization November In addition to the authors the following people have contributed to
2013 this document:
Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten, Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten,
Larry Kreeger. Larry Kreeger.
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
Authors' Addresses Authors' Addresses
Marc Lasserre Marc Lasserre
Alcatel-Lucent Alcatel-Lucent
Email: marc.lasserre@alcatel-lucent.com Email: marc.lasserre@alcatel-lucent.com
Florin Balus Florin Balus
Alcatel-Lucent Alcatel-Lucent
777 E. Middlefield Road 777 E. Middlefield Road
Mountain View, CA, USA 94043 Mountain View, CA, USA 94043
Email: florin.balus@alcatel-lucent.com Email: florin.balus@alcatel-lucent.com
Thomas Morin Thomas Morin
France Telecom Orange France Telecom Orange
Email: thomas.morin@orange.com Email: thomas.morin@orange.com
Nabil Bitar Nabil Bitar
Verizon Verizon
40 Sylvan Road 40 Sylvan Road
Waltham, MA 02145 Waltham, MA 02145
Email: nabil.bitar@verizon.com Email: nabil.bitar@verizon.com
Yakov Rekhter Yakov Rekhter
Juniper Juniper
Email: yakov@juniper.net Email: yakov@juniper.net
 End of changes. 218 change blocks. 
909 lines changed or deleted 823 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/