draft-ietf-nvo3-framework-05.txt   draft-ietf-nvo3-framework-06.txt 
Internet Engineering Task Force Marc Lasserre
Internet Draft Florin Balus
Intended status: Informational Alcatel-Lucent
Expires: July 2014
Thomas Morin
France Telecom Orange
Nabil Bitar Internet Engineering Task Force Marc Lasserre
Verizon Internet Draft Florin Balus
Intended status: Informational Alcatel-Lucent
Expires: Nov 2014
Thomas Morin
France Telecom Orange
Yakov Rekhter Nabil Bitar
Juniper Verizon
January 20, 2014 Yakov Rekhter
Juniper
Framework for DC Network Virtualization May 21, 2014
draft-ietf-nvo3-framework-05.txt
Abstract Framework for DC Network Virtualization
draft-ietf-nvo3-framework-06.txt
This document provides a framework for Network Virtualization Abstract
Overlays (NVO3) and it defines a reference model along with logical
components required to design a solution.
Status of this Memo This document provides a framework for Network Virtualization
Overlays (NVO3) and it defines a reference model along with logical
components required to design a solution.
This Internet-Draft is submitted in full conformance with the Status of this Memo
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This Internet-Draft is submitted in full conformance with the
Task Force (IETF). Note that other groups may also distribute provisions of BCP 78 and BCP 79.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are working documents of the Internet Engineering
months and may be updated, replaced, or obsoleted by other documents Task Force (IETF). Note that other groups may also distribute
at any time. It is inappropriate to use Internet-Drafts as working documents as Internet-Drafts. The list of current Internet-
reference material or to cite them other than as "work in progress." Drafts is at http://datatracker.ietf.org/drafts/current/.
This Internet-Draft will expire on July 20, 2014. Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
Copyright Notice This Internet-Draft will expire on Nov 20, 2014.
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright Notice
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Copyright (c) 2014 IETF Trust and the persons identified as the
Provisions Relating to IETF Documents document authors. All rights reserved.
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
1. Introduction..................................................3 Table of Contents
1.1. General terminology......................................3
1.2. DC network architecture..................................6
2. Reference Models..............................................8
2.1. Generic Reference Model..................................8
2.2. NVE Reference Model.....................................10
2.3. NVE Service Types.......................................11
2.3.1. L2 NVE providing Ethernet LAN-like service.........11
2.3.2. L3 NVE providing IP/VRF-like service...............11
3. Functional components........................................11
3.1. Service Virtualization Components.......................11
3.1.1. Virtual Access Points (VAPs).......................11
3.1.2. Virtual Network Instance (VNI).....................12
3.1.3. Overlay Modules and VN Context.....................12
3.1.4. Tunnel Overlays and Encapsulation options..........13
3.1.5. Control Plane Components...........................13
3.1.5.1. Distributed vs Centralized Control Plane.........13
3.1.5.2. Auto-provisioning/Service discovery..............14
3.1.5.3. Address advertisement and tunnel mapping.........14
3.1.5.4. Overlay Tunneling................................15
3.2. Multi-homing............................................15
3.3. VM Mobility.............................................16
4. Key aspects of overlay networks..............................17
4.1. Pros & Cons.............................................17
4.2. Overlay issues to consider..............................18
4.2.1. Data plane vs Control plane driven.................18
4.2.2. Coordination between data plane and control plane..19
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic...................................................19
4.2.4. Path MTU............................................20
4.2.5. NVE location trade-offs.............................21
4.2.6. Interaction between network overlays and underlays..21
5. Security Considerations.......................................22
6. IANA Considerations...........................................22
7. References....................................................23
7.1. Informative References...................................23
8. Acknowledgments...............................................23
1. Introduction 1. Introduction.................................................3
1.1. General terminology.....................................3
1.2. DC network architecture.................................6
2. Reference Models.............................................8
2.1. Generic Reference Model.................................8
2.2. NVE Reference Model....................................10
2.3. NVE Service Types......................................10
2.3.1. L2 NVE providing Ethernet LAN-like service........11
2.3.2. L3 NVE providing IP/VRF-like service..............11
2.4. Operational Management Considerations..................11
3. Functional components.......................................12
3.1. Service Virtualization Components......................12
3.1.1. Virtual Access Points (VAPs)......................12
3.1.2. Virtual Network Instance (VNI)....................12
3.1.3. Overlay Modules and VN Context....................12
3.1.4. Tunnel Overlays and Encapsulation options.........13
3.1.5. Control Plane Components..........................14
3.1.5.1. Distributed vs Centralized Control Plane........14
3.1.5.2. Auto-provisioning/Service discovery.............14
3.1.5.3. Address advertisement and tunnel mapping........15
3.1.5.4. Overlay Tunneling...............................15
3.2. Multi-homing...........................................16
3.3. VM Mobility............................................17
4. Key aspects of overlay networks.............................17
4.1. Pros & Cons............................................17
4.2. Overlay issues to consider.............................19
4.2.1. Data plane vs Control plane driven................19
4.2.2. Coordination between data plane and control plane.19
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic..................................................20
4.2.4. Path MTU..........................................20
4.2.5. NVE location trade-offs...........................21
4.2.6. Interaction between network overlays and underlays.22
5. Security Considerations......................................22
6. IANA Considerations.........................................23
7. References..................................................23
7.1. Informative References.................................23
8. Acknowledgments.............................................24
This document provides a framework for Data Center (DC) Network 1. Introduction
Virtualization over Layer3 (L3) tunnels. This framework is intended
to aid in standardizing protocols and mechanisms to support large-
scale network virtualization for data centers.
[NVOPS] defines the rationale for using overlay networks in order to This document provides a framework for Data Center (DC) Network
build large multi-tenant data center networks. Compute, storage and Virtualization over Layer3 (L3) tunnels. This framework is intended
network virtualization are often used in these large data centers to to aid in standardizing protocols and mechanisms to support large-
support a large number of communication domains and end systems. scale network virtualization for data centers.
This document provides reference models and functional components of [NVOPS] defines the rationale for using overlay networks in order to
data center overlay networks as well as a discussion of technical build large multi-tenant data center networks. Compute, storage and
issues that have to be addressed. network virtualization are often used in these large data centers to
support a large number of communication domains and end systems.
1.1. General terminology This document provides reference models and functional components of
data center overlay networks as well as a discussion of technical
issues that have to be addressed.
This document uses the following terminology: 1.1. General terminology
NVO3 Network: An overlay network that provides a Layer2 (L2) or This document uses the following terminology:
Layer3 (L3) service to Tenant Systems over an L3 underlay network
using the architecture and protocols as defined by the NVO3 Working
Group.
Network Virtualization Edge (NVE). An NVE is the network entity that NVO3 Network: An overlay network that provides a Layer2 (L2) or
sits at the edge of an underlay network and implements L2 and/or L3 Layer3 (L3) service to Tenant Systems over an L3 underlay network
network virtualization functions. The network-facing side of the NVE using the architecture and protocols as defined by the NVO3 Working
uses the underlying L3 network to tunnel tenant frames to and from Group.
other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
Virtual Network (VN): A VN is a logical abstraction of a physical Network Virtualization Edge (NVE). An NVE is the network entity that
network that provides L2 or L3 network services to a set of Tenant sits at the edge of an underlay network and implements L2 and/or L3
Systems. A VN is also known as a Closed User Group (CUG). network virtualization functions. The network-facing side of the NVE
uses the underlying L3 network to tunnel tenant frames to and from
other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
Virtual Network Instance (VNI): A specific instance of a VN from the Virtual Network (VN): A VN is a logical abstraction of a physical
perspective of an NVE. network that provides L2 or L3 network services to a set of Tenant
Systems. A VN is also known as a Closed User Group (CUG).
Virtual Network Context (VN Context) Identifier: Field in overlay Virtual Network Instance (VNI): A specific instance of a VN from the
encapsulation header that identifies the specific VN the packet perspective of an NVE.
belongs to. The egress NVE uses the VN Context identifier to deliver
the packet to the correct Tenant System. The VN Context identifier
can be a locally significant identifier or a globally unique
identifier.
Underlay or Underlying Network: The network that provides the Virtual Network Context (VN Context) Identifier: Field in overlay
connectivity among NVEs and over which NVO3 packets are tunneled, encapsulation header that identifies the specific VN the packet
where an NVO3 packet carries an NVO3 overlay header followed by a belongs to. The egress NVE uses the VN Context identifier to deliver
tenant packet. The Underlay Network does not need to be aware that the packet to the correct Tenant System. The VN Context identifier
it is carrying NVO3 packets. Addresses on the Underlay Network can be a locally significant identifier or a globally unique
appear as "outer addresses" in encapsulated NVO3 packets. In identifier.
general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is IP.
Data Center (DC): A physical complex housing physical servers, Underlay or Underlying Network: The network that provides the
network switches and routers, network service appliances and connectivity among NVEs and over which NVO3 packets are tunneled,
networked storage. The purpose of a Data Center is to provide where an NVO3 packet carries an NVO3 overlay header followed by a
application, compute and/or storage services. One such service is tenant packet. The Underlay Network does not need to be aware that
virtualized infrastructure data center services, also known as it is carrying NVO3 packets. Addresses on the Underlay Network
Infrastructure as a Service. appear as "outer addresses" in encapsulated NVO3 packets. In
general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is IP.
Virtual Data Center (Virtual DC): A container for virtualized Data Center (DC): A physical complex housing physical servers,
compute, storage and network services. A Virtual DC is associated network switches and routers, network service appliances and
with a single tenant, and can contain multiple VNs and Tenant networked storage. The purpose of a Data Center is to provide
Systems connected to one or more of these VNs. application, compute and/or storage services. One such service is
virtualized infrastructure data center services, also known as
Infrastructure as a Service.
Virtual machine (VM): A software implementation of a physical Virtual Data Center (Virtual DC): A container for virtualized
machine that runs programs as if they were executing on a physical, compute, storage and network services. A Virtual DC is associated
non-virtualized machine. Applications (generally) do not know they with a single tenant, and can contain multiple VNs and Tenant
are running on a VM as opposed to running on a "bare metal" host or Systems connected to one or more of these VNs.
server, though some systems provide a para-virtualization
environment that allows an operating system or application to be
aware of the presence of virtualization for optimization purposes.
Hypervisor: Software running on a server that allows multiple VMs to Virtual machine (VM): A software implementation of a physical
run on the same physical server. The hypervisor manages and provides machine that runs programs as if they were executing on a physical,
shared compute/memory/storage and network connectivity to the VMs non-virtualized machine. Applications (generally) do not know they
that it hosts. Hypervisors often embed a Virtual Switch (see below). are running on a VM as opposed to running on a "bare metal" host or
server, though some systems provide a para-virtualization
environment that allows an operating system or application to be
aware of the presence of virtualization for optimization purposes.
Server: A physical end host machine that runs user applications. A Hypervisor: Software running on a server that allows multiple VMs to
standalone (or "bare metal") server runs a conventional operating run on the same physical server. The hypervisor manages and provides
system hosting a single-tenant application. A virtualized server shared compute/memory/storage and network connectivity to the VMs
runs a hypervisor supporting one or more VMs. that it hosts. Hypervisors often embed a Virtual Switch (see below).
Virtual Switch (vSwitch): A function within a Hypervisor (typically Server: A physical end host machine that runs user applications. A
implemented in software) that provides similar forwarding services standalone (or "bare metal") server runs a conventional operating
to a physical Ethernet switch. A vSwitch forwards Ethernet frames system hosting a single-tenant application. A virtualized server
between VMs running on the same server, or between a VM and a runs a hypervisor supporting one or more VMs.
physical NIC card connecting the server to a physical Ethernet
switch or router. A vSwitch also enforces network isolation between
VMs that by policy are not permitted to communicate with each other
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
enabled on the host server.
Tenant: The customer using a virtual network and any associated Virtual Switch (vSwitch): A function within a Hypervisor (typically
resources (e.g., compute, storage and network). A tenant could be implemented in software) that provides similar forwarding services
an enterprise, or a department/organization within an enterprise. to a physical Ethernet switch. A vSwitch forwards Ethernet frames
between VMs running on the same server, or between a VM and a
physical NIC card connecting the server to a physical Ethernet
switch or router. A vSwitch also enforces network isolation between
VMs that by policy are not permitted to communicate with each other
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
enabled on the host server.
Tenant System: A physical or virtual system that can play the role Tenant: The customer using a virtual network and any associated
of a host, or a forwarding element such as a router, switch, resources (e.g., compute, storage and network). A tenant could be
firewall, etc. It belongs to a single tenant and connects to one or an enterprise, or a department/organization within an enterprise.
more VNs of that tenant.
Tenant Separation: Tenant Separation refers to isolating traffic of Tenant System: A physical or virtual system that can play the role
different tenants such that traffic from one tenant is not visible of a host, or a forwarding element such as a router, switch,
to or delivered to another tenant, except when allowed by policy. firewall, etc. It belongs to a single tenant and connects to one or
Tenant Separation also refers to address space separation, whereby more VNs of that tenant.
different tenants can use the same address space without conflict.
Virtual Access Points (VAPs): A logical connection point on the NVE Tenant Separation: Tenant Separation refers to isolating traffic of
for connecting a Tenant System to a virtual network. Tenant Systems different tenants such that traffic from one tenant is not visible
connect to VNIs at an NVE through VAPs. VAPs can be physical ports to or delivered to another tenant, except when allowed by policy.
or virtual ports identified through logical interface identifiers Tenant Separation also refers to address space separation, whereby
(e.g., VLAN ID, internal vSwitch Interface ID connected to a VM). different tenants can use the same address space without conflict.
End Device: A physical device that connects directly to the DC Virtual Access Points (VAPs): A logical connection point on the NVE
Underlay Network. This is in contrast to a Tenant System, which for connecting a Tenant System to a virtual network. Tenant Systems
connects to a corresponding tenant VN. An End Device is administered connect to VNIs at an NVE through VAPs. VAPs can be physical ports
by the DC operator rather than a tenant, and is part of the DC or virtual ports identified through logical interface identifiers
infrastructure. An End Device may implement NVO3 technology in (e.g., VLAN ID, internal vSwitch Interface ID connected to a VM).
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
Network Virtualization Authority (NVA): Entity that provides End Device: A physical device that connects directly to the DC
reachability and forwarding information to NVEs. Underlay Network. This is in contrast to a Tenant System, which
connects to a corresponding tenant VN. An End Device is administered
by the DC operator rather than a tenant, and is part of the DC
infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
1.2. DC network architecture Network Virtualization Authority (NVA): Entity that provides
reachability and forwarding information to NVEs.
A generic architecture for Data Centers is depicted in Figure 1: 1.2. DC network architecture
,---------. A generic architecture for Data Centers is depicted in Figure 1:
,' `.
( IP/MPLS WAN )
`. ,'
`-+------+'
\ /
+--------+ +--------+
| DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| /
.--. .--.
( ' '.--.
.-.' Intra-DC '
( network )
( .'-'
'--'._.'. )\ \
/ / '--' \ \
/ / | | \ \
+--------+ +--------+ +--------+
| access | | access | | access |
| switch | | switch | | switch |
+--------+ +--------+ +--------+
/ \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------'
: End : : End : : End : : End :
: Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------'
Figure 1 : A Generic Architecture for Data Centers ,---------.
,' `.
( IP/MPLS WAN )
`. ,'
`-+------+'
\ /
+--------+ +--------+
| DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| /
.--. .--.
( ' '.--.
.-.' Intra-DC '
( network )
( .'-'
'--'._.'. )\ \
/ / '--' \ \
/ / | | \ \
+--------+ +--------+ +--------+
| access | | access | | access |
| switch | | switch | | switch |
+--------+ +--------+ +--------+
/ \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------'
: End : : End : : End : : End :
: Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------'
An example of multi-tier DC network architecture is presented in Figure 1 : A Generic Architecture for Data Centers
Figure 1. It provides a view of physical components inside a DC.
A DC network is usually composed of intra-DC networks and network An example of multi-tier DC network architecture is presented in
services, and inter-DC network and network connectivity services. Figure 1. It provides a view of physical components inside a DC.
DC networking elements can act as strict L2 switches and/or provide A DC network is usually composed of intra-DC networks and network
IP routing capabilities, including network service virtualization. services, and inter-DC network and network connectivity services.
In some DC architectures, some tier layers could provide L2 and/or DC networking elements can act as strict L2 switches and/or provide
L3 services. In addition, some tier layers may be collapsed, and IP routing capabilities, including network service virtualization.
Internet connectivity, inter-DC connectivity and VPN support may be
handled by a smaller number of nodes. Nevertheless, one can assume
that the network functional blocks in a DC fit in the architecture
depicted in Figure 1.
The following components can be present in a DC: In some DC architectures, some tier layers could provide L2 and/or
L3 services. In addition, some tier layers may be collapsed, and
Internet connectivity, inter-DC connectivity and VPN support may be
handled by a smaller number of nodes. Nevertheless, one can assume
that the network functional blocks in a DC fit in the architecture
depicted in Figure 1.
- Access switch: Hardware-based Ethernet switch aggregating all The following components can be present in a DC:
Ethernet links from the End Devices in a rack representing the
entry point in the physical DC network for the hosts. It may also
provide routing functionality, virtual IP network connectivity, or
Layer2 tunneling over IP for instance. Access switches are usually
multi-homed to aggregation switches in the Intra-DC network. A
typical example of an access switch is a Top of Rack (ToR) switch.
Other deployment scenarios may use an intermediate Blade Switch
before the ToR, or an EoR (End of Row) switch, to provide similar
functions to a ToR.
- Intra-DC Network: Network composed of high capacity core nodes - Access switch: Hardware-based Ethernet switch aggregating all
(Ethernet switches/routers). Core nodes may provide virtual Ethernet links from the End Devices in a rack representing the
Ethernet bridging and/or IP routing services. entry point in the physical DC network for the hosts. It may also
provide routing functionality, virtual IP network connectivity, or
Layer2 tunneling over IP for instance. Access switches are usually
multi-homed to aggregation switches in the Intra-DC network. A
typical example of an access switch is a Top of Rack (ToR) switch.
Other deployment scenarios may use an intermediate Blade Switch
before the ToR, or an EoR (End of Row) switch, to provide similar
functions to a ToR.
- DC Gateway (DC GW): Gateway to the outside world providing DC - Intra-DC Network: Network composed of high capacity core nodes
Interconnect and connectivity to Internet and VPN customers. In (Ethernet switches/routers). Core nodes may provide virtual
the current DC network model, this may be simply a router Ethernet bridging and/or IP routing services.
connected to the Internet and/or an IP Virtual Private Network
(VPN)/L2VPN PE. Some network implementations may dedicate DC GWs
for different connectivity types (e.g., a DC GW for Internet, and
another for VPN).
Note that End Devices may be single or multi-homed to access - DC Gateway (DC GW): Gateway to the outside world providing DC
switches. Interconnect and connectivity to Internet and VPN customers. In
the current DC network model, this may be simply a router
connected to the Internet and/or an IP Virtual Private Network
(VPN)/L2VPN PE. Some network implementations may dedicate DC GWs
for different connectivity types (e.g., a DC GW for Internet, and
another for VPN).
2. Reference Models Note that End Devices may be single or multi-homed to access
switches.
2.1. Generic Reference Model 2. Reference Models
Figure 2 depicts a DC reference model for network virtualization 2.1. Generic Reference Model
overlay where NVEs provide a logical interconnect between Tenant
Systems that belong to a specific VN.
+--------+ +--------+ Figure 2 depicts a DC reference model for network virtualization
| Tenant +--+ +----| Tenant | overlay where NVEs provide a logical interconnect between Tenant
| System | | (') | System | Systems that belong to a specific VN.
+--------+ | ................. ( ) +--------+
| +---+ +---+ (_)
+--|NVE|---+ +---|NVE|-----+
+---+ | | +---+
/ . +-----+ .
/ . +--| NVA |--+ .
/ . | +-----+ \ .
| . | \ .
| . | Overlay +--+--++--------+
+--------+ | . | Network | NVE || Tenant |
| Tenant +--+ . | | || System |
| System | . \ +---+ +--+--++--------+
+--------+ .....|NVE|.........
+---+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
Figure 2 : Generic reference model for DC network virtualization +--------+ +--------+
overlay | Tenant +--+ +----| Tenant |
| System | | (') | System |
+--------+ | ................. ( ) +--------+
| +---+ +---+ (_)
+--|NVE|---+ +---|NVE|-----+
+---+ | | +---+
/ . +-----+ .
/ . +--| NVA |--+ .
/ . | +-----+ \ .
| . | \ .
| . | Overlay +--+--++--------+
+--------+ | . | Network | NVE || Tenant |
| Tenant +--+ . | | || System |
| System | . \ +---+ +--+--++--------+
+--------+ .....|NVE|.........
+---+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
In order to obtain reachability information, NVEs may exchange Figure 2 : Generic reference model for DC network virtualization
information directly between themselves via a control plane overlay
protocol. In this case, a control plane module resides in every NVE.
It is also possible for NVEs to communicate with an external Network In order to obtain reachability information, NVEs may exchange
Virtualization Authority (NVA) to obtain reachability and forwarding information directly between themselves via a control plane
information. In this case, a protocol is used between NVEs and protocol. In this case, a control plane module resides in every NVE.
NVA(s) to exchange information. OpenFlow [OF] is one example of such
a protocol.
It should be noted that NVAs may be organized in clusters for It is also possible for NVEs to communicate with an external Network
redundancy and scalability and can appear as one logically Virtualization Authority (NVA) to obtain reachability and forwarding
centralized controller. In this case, inter-NVA communication is information. In this case, a protocol is used between NVEs and
necessary to synchronize state among nodes within a cluster or share NVA(s) to exchange information.
information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information
exchanged across clusters.
A Tenant System can be attached to an NVE in several ways: It should be noted that NVAs may be organized in clusters for
redundancy and scalability and can appear as one logically
centralized controller. In this case, inter-NVA communication is
necessary to synchronize state among nodes within a cluster or share
information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information
exchanged across clusters.
- locally, by being co-located in the same End Device A Tenant System can be attached to an NVE in several ways:
- remotely, via a point-to-point connection or a switched network - locally, by being co-located in the same End Device
When an NVE is co-located with a Tenant System, the state of the - remotely, via a point-to-point connection or a switched network
Tenant System can be determined without protocol assistance. For
instance, the operational status of a VM can be communicated via a
local API. When an NVE is remotely connected to a Tenant System, the
state of the Tenant System or NVE needs to be exchanged directly or
via a management entity, using a control plane protocol or API, or
directly via a dataplane protocol.
The functional components in Figure 2 do not necessarily map When an NVE is co-located with a Tenant System, the state of the
directly to the physical components described in Figure 1. For Tenant System can be determined without protocol assistance. For
example, an End Device can be a server blade with VMs and a virtual instance, the operational status of a VM can be communicated via a
switch. A VM can be a Tenant System and the NVE functions may be local API. When an NVE is remotely connected to a Tenant System, the
performed by the host server. In this case, the Tenant System and state of the Tenant System or NVE needs to be exchanged directly or
NVE function are co-located. Another example is the case where the via a management entity, using a control plane protocol or API, or
End Device is the Tenant System, and the NVE function can be directly via a dataplane protocol.
implemented by the connected ToR. In this case, the Tenant System
and NVE function are not co-located.
Underlay nodes utilize L3 technologies to interconnect NVE nodes. The functional components in Figure 2 do not necessarily map
These nodes perform forwarding based on outer L3 header information, directly to the physical components described in Figure 1. For
and generally do not maintain per tenant-service state albeit some example, an End Device can be a server blade with VMs and a virtual
applications (e.g., multicast) may require control plane or switch. A VM can be a Tenant System and the NVE functions may be
forwarding plane information that pertain to a tenant, group of performed by the host server. In this case, the Tenant System and
tenants, tenant service or a set of services that belong to one or NVE function are co-located. Another example is the case where the
more tenants. Mechanisms to control the amount of state maintained End Device is the Tenant System, and the NVE function can be
in the underlay may be needed. implemented by the connected ToR. In this case, the Tenant System
and NVE function are not co-located.
2.2. NVE Reference Model Underlay nodes utilize L3 technologies to interconnect NVE nodes.
These nodes perform forwarding based on outer L3 header information,
and generally do not maintain per tenant-service state albeit some
applications (e.g., multicast) may require control plane or
forwarding plane information that pertain to a tenant, group of
tenants, tenant service or a set of services that belong to one or
more tenants. Mechanisms to control the amount of state maintained
in the underlay may be needed.
Figure 3 depicts the NVE reference model. One or more VNIs can be 2.2. NVE Reference Model
instantiated on an NVE. A Tenant System interfaces with a
corresponding VNI via a VAP. An overlay module provides tunneling
overlay functions (e.g., encapsulation and decapsulation of tenant
traffic, tenant identification and mapping, etc.).
+-------- L3 Network -------+ Figure 3 depicts the NVE reference model. One or more VNIs can be
| | instantiated on an NVE. A Tenant System interfaces with a
| Tunnel Overlay | corresponding VNI via a VAP. An overlay module provides tunneling
+------------+---------+ +---------+------------+ overlay functions (e.g., encapsulation and decapsulation of tenant
| +----------+-------+ | | +---------+--------+ | traffic, tenant identification and mapping, etc.).
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| |
| | | | | |
| +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+
| | | |
| | | |
Tenant Systems Tenant Systems
Figure 3 : Generic NVE reference model +-------- L3 Network -------+
| |
| Tunnel Overlay |
+------------+---------+ +---------+------------+
| +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| |
| | | | | |
| +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+
| | | |
| | | |
Tenant Systems Tenant Systems
Note that some NVE functions (e.g., data plane and control plane Figure 3 : Generic NVE reference model
functions) may reside in one device or may be implemented separately
in different devices. In addition, NVE functions can be implemented
in a hierarchical fashion. For instance, an End Device can act as an
NVE Spoke, while an access switch can act as an NVE hub.
2.3. NVE Service Types Note that some NVE functions (e.g., data plane and control plane
functions) may reside in one device or may be implemented separately
in different devices.
An NVE provides different types of virtualized network services to 2.3. NVE Service Types
multiple tenants, i.e. an L2 service or an L3 service. Note that an
NVE may be capable of providing both L2 and L3 services for a
tenant. This section defines the service types and associated
attributes.
2.3.1. L2 NVE providing Ethernet LAN-like service An NVE provides different types of virtualized network services to
multiple tenants, i.e. an L2 service or an L3 service. Note that an
NVE may be capable of providing both L2 and L3 services for a
tenant. This section defines the service types and associated
attributes.
An L2 NVE implements Ethernet LAN emulation, an Ethernet based 2.3.1. L2 NVE providing Ethernet LAN-like service
multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or
EVPN [EVPN] service, where the Tenant Systems appear to be
interconnected by a LAN environment over an L3 overlay. As such, an
L2 NVE provides per-tenant virtual switching instance (L2 VNI), and
L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the
underlay. Note that the control plane for an L2 NVE could be
implemented locally on the NVE or in a separate control entity.
2.3.2. L3 NVE providing IP/VRF-like service An L2 NVE implements Ethernet LAN emulation, an Ethernet based
multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or
EVPN [EVPN] service, where the Tenant Systems appear to be
interconnected by a LAN environment over an L3 overlay. As such, an
L2 NVE provides per-tenant virtual switching instance (L2 VNI), and
L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the
underlay. Note that the control plane for an L2 NVE could be
implemented locally on the NVE or in a separate control entity.
An L3 NVE provides Virtualized IP forwarding service, similar to 2.3.2. L3 NVE providing IP/VRF-like service
IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service
definition perspective. That is, an L3 NVE provides per-tenant
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant IP packets across the underlay. Note that
routing could be performed locally on the NVE or in a separate
control entity.
3. Functional components An L3 NVE provides Virtualized IP forwarding service, similar to
IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service
definition perspective. That is, an L3 NVE provides per-tenant
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant IP packets across the underlay. Note that
routing could be performed locally on the NVE or in a separate
control entity.
This section decomposes the Network Virtualization architecture into 2.4. Operational Management Considerations
functional components described in Figure 3 to make it easier to
discuss solution options for these components.
3.1. Service Virtualization Components NVO3 services are overlay services over an IP underlay.
3.1.1. Virtual Access Points (VAPs) As far as the IP underlay is concerned, existing IP OAM facilities
are used.
Tenant Systems are connected to VNIs through Virtual Access Points With regards to the NVO3 overlay, both L2 and L3 services can be
(VAPs). offered. it is expected that existing fault and performance OAM
facilities will be used. Sections 4.1. and 4.2.6. below provide
further discussion of additional fault and performance management
issues to consider.
VAPs can be physical ports or virtual ports identified through As far as configuration is concerned, the DC environment is driven
logical interface identifiers (e.g., VLAN ID, internal vSwitch by the need to bring new services up rapidly and is typically very
Interface ID connected to a VM). dynamic specifically in the context of virtualized services. It is
therefore critical to automate the configuration of NVO3 services.
3.1.2. Virtual Network Instance (VNI) 3. Functional components
A VNI is a specific VN instance on an NVE. Each VNI defines a This section decomposes the Network Virtualization architecture into
forwarding context that contains reachability information and functional components described in Figure 3 to make it easier to
policies. discuss solution options for these components.
3.1.3. Overlay Modules and VN Context 3.1. Service Virtualization Components
Mechanisms for identifying each tenant service are required to allow 3.1.1. Virtual Access Points (VAPs)
the simultaneous overlay of multiple tenant services over the same
underlay L3 network topology. In the data plane, each NVE, upon
sending a tenant packet, must be able to encode the VN Context for
the destination NVE in addition to the L3 tunneling information
(e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the
tenant packet.
The Overlay module provides tunneling overlay functions: tunnel Tenant Systems are connected to VNIs through Virtual Access Points
initiation/termination as in the case of stateful tunnels (see (VAPs).
Section 3.1.4), and/or simply encapsulation/decapsulation of frames
from VAPs/L3 underlay.
In a multi-tenant context, tunneling aggregates frames from/to VAPs can be physical ports or virtual ports identified through
different VNIs. Tenant identification and traffic demultiplexing are logical interface identifiers (e.g., VLAN ID, internal vSwitch
based on the VN Context identifier. Interface ID connected to a VM).
The following approaches can be considered: 3.1.2. Virtual Network Instance (VNI)
- One VN Context identifier per Tenant: A globally unique (on a per- A VNI is a specific VN instance on an NVE. Each VNI defines a
DC administrative domain) VN identifier is used to identify the forwarding context that contains reachability information and
corresponding VNI. Examples of such identifiers in existing policies.
technologies are IEEE VLAN IDs and ISID tags that identify virtual
L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively.
- One VN Context identifier per VNI: A per-VNI local value is 3.1.3. Overlay Modules and VN Context
automatically generated by the egress NVE, or a control plane
associated with that NVE, and usually distributed by a control
plane protocol to all the related NVEs. An example of this
approach is the use of per VRF MPLS labels in IP VPN [RFC4364].
- One VN Context identifier per VAP: A per-VAP local value is Mechanisms for identifying each tenant service are required to allow
assigned and usually distributed by a control plane protocol. An the simultaneous overlay of multiple tenant services over the same
example of this approach is the use of per CE-PE MPLS labels in IP underlay L3 network topology. In the data plane, each NVE, upon
VPN [RFC4364]. sending a tenant packet, must be able to encode the VN Context for
the destination NVE in addition to the L3 tunneling information
(e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the
tenant packet.
Note that when using one VN Context per VNI or per VAP, an The Overlay module provides tunneling overlay functions: tunnel
additional global identifier (e.g., a VN identifier or name) may be initiation/termination as in the case of stateful tunnels (see
used by the control plane to identify the Tenant context. Section 3.1.4), and/or simply encapsulation/decapsulation of frames
from VAPs/L3 underlay.
3.1.4. Tunnel Overlays and Encapsulation options In a multi-tenant context, tunneling aggregates frames from/to
different VNIs. Tenant identification and traffic demultiplexing are
based on the VN Context identifier.
Once the VN context identifier is added to the frame, an L3 Tunnel The following approaches can be considered:
encapsulation is used to transport the frame to the destination NVE.
Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS - VN Context identifier per Tenant: Globally unique (on a per-DC
tunneling can be used. Tunneling could be stateless or stateful. administrative domain) VN identifier used to identify the
Stateless tunneling simply entails the encapsulation of a tenant corresponding VNI. Examples of such identifiers in existing
packet with another header necessary for forwarding the packet technologies are IEEE VLAN IDs and ISID tags that identify virtual
across the underlay (e.g., IP tunneling over an IP underlay). L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively.
Stateful tunneling on the other hand entails maintaining tunneling Note that multiple VN identifiers can belong to a tenant.
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions should
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to carry the packet between an ingress NVE and an egress NVE across
the underlay. It should be noted that stateful tunneling, especially
when configuration is involved, does impose management overhead and
scale constraints. Thus, stateless tunneling is preferred when
feasible.
3.1.5. Control Plane Components - One VN Context identifier per VNI: Each VNI value is automatically
generated by the egress NVE, or a control plane associated with
that NVE, and usually distributed by a control plane protocol to
all the related NVEs. An example of this approach is the use of
per VRF MPLS labels in IP VPN [RFC4364]. The VNI value is
therefore locally significant to the egress NVE.
3.1.5.1. Distributed vs Centralized Control Plane - One VN Context identifier per VAP: A value locally significant to
an NVE is assigned and usually distributed by a control plane
protocol to identify a VAP. An example of this approach is the use
of per CE-PE MPLS labels in IP VPN [RFC4364].
A control/management plane entity can be centralized or distributed. Note that when using one VN Context per VNI or per VAP, an
Both approaches have been used extensively in the past. The routing additional global identifier (e.g., a VN identifier or name) may be
model of the Internet is a good example of a distributed approach. used by the control plane to identify the Tenant context.
Transport networks have usually used a centralized approach to
manage transport paths.
It is also possible to combine the two approaches, i.e., using a 3.1.4. Tunnel Overlays and Encapsulation options
hybrid model. A global view of network state can have many benefits
but it does not preclude the use of distributed protocols within the
network. Centralized models provide a facility to maintain global
state, and distribute that state to the network. When used in
combination with distributed protocols, greater network
efficiencies, improved reliability and robustness can be achieved.
Domain and/or deployment specific constraints define the balance
between centralized and distributed approaches.
3.1.5.2. Auto-provisioning/Service discovery Once the VN context identifier is added to the frame, an L3 Tunnel
encapsulation is used to transport the frame to the destination NVE.
NVEs must be able to identify the appropriate VNI for each Tenant Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS
System. This is based on state information that is often provided by tunneling can be used. Tunneling could be stateless or stateful.
external entities. For example, in an environment where a VM is a Stateless tunneling simply entails the encapsulation of a tenant
Tenant System, this information is provided by VM orchestration packet with another header necessary for forwarding the packet
systems, since these are the only entities that have visibility of across the underlay (e.g., IP tunneling over an IP underlay).
which VM belongs to which tenant. Stateful tunneling on the other hand entails maintaining tunneling
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions should
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to carry the packet between an ingress NVE and an egress NVE across
the underlay. It should be noted that stateful tunneling, especially
when configuration is involved, does impose management overhead and
scale constraints. When confidentiality is required, the use of
opportunistic encryption can be used as a stateless tunneling
solution.
A mechanism for communicating this information to the NVE is 3.1.5. Control Plane Components
required. VAPs have to be created and mapped to the appropriate VNI.
Depending upon the implementation, this control interface can be
implemented using an auto-discovery protocol between Tenant Systems
and their local NVE or through management entities. In either case,
appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant
isolation in the system.
NVEs may learn reachability information to VNIs on other NVEs via a 3.1.5.1. Distributed vs Centralized Control Plane
control protocol that exchanges such information among NVEs, or via
a management control entity.
3.1.5.3. Address advertisement and tunnel mapping A control/management plane entity can be centralized or distributed.
Both approaches have been used extensively in the past. The routing
model of the Internet is a good example of a distributed approach.
Transport networks have usually used a centralized approach to
manage transport paths.
As traffic reaches an ingress NVE on a VAP, a lookup is performed to It is also possible to combine the two approaches, i.e., using a
determine which NVE or local VAP the packet needs to be sent to. If hybrid model. A global view of network state can have many benefits
the packet is to be sent to another NVE, the packet is encapsulated but it does not preclude the use of distributed protocols within the
with a tunnel header containing the destination information network. Centralized models provide a facility to maintain global
(destination IP address or MPLS label) of the egress NVE. state, and distribute that state to the network. When used in
Intermediate nodes (between the ingress and egress NVEs) switch or combination with distributed protocols, greater network
route traffic based upon the tunnel destination information. efficiencies, improved reliability and robustness can be achieved.
Domain and/or deployment specific constraints define the balance
between centralized and distributed approaches.
A key step in the above process consists of identifying the 3.1.5.2. Auto-provisioning/Service discovery
destination NVE the packet is to be tunneled to. NVEs are
responsible for maintaining a set of forwarding or mapping tables
that hold the bindings between destination VM and egress NVE
addresses. Several ways of populating these tables are possible:
control plane driven, management plane driven, or data plane driven.
When a control plane protocol is used to distribute address NVEs must be able to identify the appropriate VNI for each Tenant
reachability and tunneling information, the auto- System. This is based on state information that is often provided by
provisioning/Service discovery could be accomplished by the same external entities. For example, in an environment where a VM is a
protocol. In this scenario, the auto-provisioning/Service discovery Tenant System, this information is provided by VM orchestration
could be combined with (be inferred from) the address advertisement systems, since these are the only entities that have visibility of
and associated tunnel mapping. Furthermore, a control plane protocol which VM belongs to which tenant.
that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling as
discussed in [RFC6820].
3.1.5.4. Overlay Tunneling A mechanism for communicating this information to the NVE is
required. VAPs have to be created and mapped to the appropriate VNI.
Depending upon the implementation, this control interface can be
implemented using an auto-discovery protocol between Tenant Systems
and their local NVE or through management entities. In either case,
appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant
isolation in the system.
For overlay tunneling, and dependent upon the tunneling technology NVEs may learn reachability information to VNIs on other NVEs via a
used for encapsulating the Tenant System packets, it may be control protocol that exchanges such information among NVEs, or via
sufficient to have one or more local NVE addresses assigned and used a management control entity.
in the source and destination fields of a tunneling encapsulation
header. Other information that is part of the
tunneling encapsulation header may also need to be configured. In
certain cases, local NVE configuration may be sufficient while in
other cases, some tunneling related information may need to
be shared among NVEs. The information that needs to be shared will
be technology dependent. For instance, potential information could
include tunnel identity, encapsulation type, and/or tunnel
resources. In certain cases, such as when using IP multicast in the
underlay, tunnels which interconnect NVEs may need to be
established. When tunneling information needs to be exchanged or
shared among NVEs, a control plane protocol may be required. For
instance, it may be necessary to provide active/standby status
information between NVEs, up/down status information,
pruning/grafting information for multicast tunnels, etc.
In addition, a control plane may be required to setup the tunnel 3.1.5.3. Address advertisement and tunnel mapping
path for some tunneling technologies. This applies to both unicast
and multicast tunneling.
3.2. Multi-homing As traffic reaches an ingress NVE on a VAP, a lookup is performed to
determine which NVE or local VAP the packet needs to be sent to. If
the packet is to be sent to another NVE, the packet is encapsulated
with a tunnel header containing the destination information
(destination IP address or MPLS label) of the egress NVE.
Intermediate nodes (between the ingress and egress NVEs) switch or
route traffic based upon the tunnel destination information.
Multi-homing techniques can be used to increase the reliability of A key step in the above process consists of identifying the
an NVO3 network. It is also important to ensure that physical destination NVE the packet is to be tunneled to. NVEs are
diversity in an NVO3 network is taken into account to avoid single responsible for maintaining a set of forwarding or mapping tables
points of failure. that hold the bindings between destination VM and egress NVE
addresses. Several ways of populating these tables are possible:
control plane driven, management plane driven, or data plane driven.
Multi-homing can be enabled in various nodes, from Tenant Systems When a control plane protocol is used to distribute address
into ToRs, ToRs into core switches/routers, and core nodes into DC reachability and tunneling information, the auto-
GWs. provisioning/Service discovery could be accomplished by the same
protocol. In this scenario, the auto-provisioning/Service discovery
could be combined with (be inferred from) the address advertisement
and associated tunnel mapping. Furthermore, a control plane protocol
that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling as
discussed in [RFC6820].
The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP 3.1.5.4. Overlay Tunneling
routing as the means to re-route traffic upon failures techniques or
on MPLS re-rerouting capabilities.
When a Tenant System is co-located with the NVE, the Tenant System For overlay tunneling, and dependent upon the tunneling technology
is effectively single homed to the NVE via a virtual port. When the used for encapsulating the Tenant System packets, it may be
Tenant System and the NVE are separated, the Tenant System is sufficient to have one or more local NVE addresses assigned and used
connected to the NVE via a logical Layer2 (L2) construct such as a in the source and destination fields of a tunneling encapsulation
VLAN and it can be multi-homed to various NVEs. An NVE may provide header. Other information that is part of the
an L2 service to the end system or an l3 service. An NVE may be tunneling encapsulation header may also need to be configured. In
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3 certain cases, local NVE configuration may be sufficient while in
(L3). When an NVE provides an L2 service and is not co-located with other cases, some tunneling related information may need to
the end system, techniques such as Ethernet Link Aggregation Group be shared among NVEs. The information that needs to be shared will
(LAG) or Spanning Tree Protocol (STP) can be used to switch traffic be technology dependent. For instance, potential information could
between an end system and connected NVEs without creating include tunnel identity, encapsulation type, and/or tunnel
loops. Similarly, when the NVE provides L3 service, similar dual- resources. In certain cases, such as when using IP multicast in the
homing techniques can be used. When the NVE provides a L3 service to underlay, tunnels which interconnect NVEs may need to be
the end system, it is possible that no dynamic routing protocol is established. When tunneling information needs to be exchanged or
enabled between the end system and the NVE. The end system can be shared among NVEs, a control plane protocol may be required. For
multi-homed to multiple physically-separated L3 NVEs over multiple instance, it may be necessary to provide active/standby status
interfaces. When one of the links connected to an NVE fails, the information between NVEs, up/down status information,
other interfaces can be used to reach the end system. pruning/grafting information for multicast tunnels, etc.
External connectivity from a DC can be handled by two or more DC In addition, a control plane may be required to setup the tunnel
gateways. Each gateway provides access to external networks such as path for some tunneling technologies. This applies to both unicast
VPNs or the Internet. A gateway may be connected to two or more edge and multicast tunneling.
nodes in the external network for redundancy. When a connection to
an upstream node is lost, the alternative connection is used and the
failed route withdrawn.
3.3. VM Mobility 3.2. Multi-homing
In DC environments utilizing VM technologies, an important feature Multi-homing techniques can be used to increase the reliability of
is that VMs can move from one server to another server in the same an NVO3 network. It is also important to ensure that physical
or different L2 physical domains (within or across DCs) in a diversity in an NVO3 network is taken into account to avoid single
seamless manner. points of failure.
A VM can be moved from one server to another in stopped or suspended Multi-homing can be enabled in various nodes, from Tenant Systems
state ("cold" VM mobility) or in running/active state ("hot" VM into ToRs, ToRs into core switches/routers, and core nodes into DC
mobility). With "hot" mobility, VM L2 and L3 addresses need to be GWs.
preserved. With "cold" mobility, it may be desired to preserve at
least VM L3 addresses.
Solutions to maintain connectivity while a VM is moved are necessary The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP
in the case of "hot" mobility. This implies that connectivity among routing as the means to re-route traffic upon failures techniques or
VMs is preserved. For instance, for L2 VNs, ARP caches are updated on MPLS re-rerouting capabilities.
accordingly.
Upon VM mobility, NVE policies that define connectivity among VMs When a Tenant System is co-located with the NVE, the Tenant System
must be maintained. is effectively single homed to the NVE via a virtual port. When the
Tenant System and the NVE are separated, the Tenant System is
connected to the NVE via a logical Layer2 (L2) construct such as a
VLAN and it can be multi-homed to various NVEs. An NVE may provide
an L2 service to the end system or an l3 service. An NVE may be
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3
(L3). When an NVE provides an L2 service and is not co-located with
the end system, techniques such as Ethernet Link Aggregation Group
(LAG) or Spanning Tree Protocol (STP) can be used to switch traffic
between an end system and connected NVEs without creating
loops. Similarly, when the NVE provides L3 service, similar dual-
homing techniques can be used. When the NVE provides a L3 service to
the end system, it is possible that no dynamic routing protocol is
enabled between the end system and the NVE. The end system can be
multi-homed to multiple physically-separated L3 NVEs over multiple
interfaces. When one of the links connected to an NVE fails, the
other interfaces can be used to reach the end system.
During VM mobility, it is expected that the path to the VM's default External connectivity from a DC can be handled by two or more DC
gateway assures adequate performance to VM applications. gateways. Each gateway provides access to external networks such as
VPNs or the Internet. A gateway may be connected to two or more edge
nodes in the external network for redundancy. When a connection to
an upstream node is lost, the alternative connection is used and the
failed route withdrawn.
4. Key aspects of overlay networks 3.3. VM Mobility
The intent of this section is to highlight specific issues that In DC environments utilizing VM technologies, an important feature
proposed overlay solutions need to address. is that VMs can move from one server to another server in the same
or different L2 physical domains (within or across DCs) in a
seamless manner.
4.1. Pros & Cons A VM can be moved from one server to another in stopped or suspended
state ("cold" VM mobility) or in running/active state ("hot" VM
mobility). With "hot" mobility, VM L2 and L3 addresses need to be
preserved. With "cold" mobility, it may be desired to preserve at
least VM L3 addresses.
An overlay network is a layer of virtual network topology on top of Solutions to maintain connectivity while a VM is moved are necessary
the physical network. in the case of "hot" mobility. This implies that connectivity among
VMs is preserved. For instance, for L2 VNs, ARP caches are updated
accordingly.
Overlay networks offer the following key advantages: Upon VM mobility, NVE policies that define connectivity among VMs
must be maintained.
- Unicast tunneling state management and association of Tenant During VM mobility, it is expected that the path to the VM's default
Systems reachability are handled at the edge of the network (at gateway assures adequate performance to VM applications.
the NVE). Intermediate transport nodes are unaware of such
state. Note that when multicast is enabled in the underlay
network to build multicast trees for tenant VNs, there would be
more state related to tenants in the underlay core network.
- Tunneling is used to aggregate traffic and hide tenant 4. Key aspects of overlay networks
addresses from the underlay network, and hence offer the
advantage of minimizing the amount of forwarding state required
within the underlay network
- Decoupling of the overlay addresses (MAC and IP) used by VMs The intent of this section is to highlight specific issues that
from the underlay network for tenant separation and separation proposed overlay solutions need to address.
of the tenant address spaces from the underlay address space.
- Support of a large number of virtual network identifiers 4.1. Pros & Cons
Overlay networks also create several challenges: An overlay network is a layer of virtual network topology on top of
the physical network.
- Overlay networks have typically no control of underlay networks Overlay networks offer the following key advantages:
and lack underlay network information (e.g. underlay
utilization):
- Overlay networks and/or their associated management entities - Unicast tunneling state management and association of Tenant
typically probe the network to measure link or path Systems reachability are handled at the edge of the network (at
properties, such as available bandwidth or packet loss rate. the NVE). Intermediate transport nodes are unaware of such
It is difficult to accurately evaluate network properties. It state. Note that when multicast is enabled in the underlay
might be preferable for the underlay network to expose usage network to build multicast trees for tenant VNs, there would be
and performance information. more state related to tenants in the underlay core network.
- Miscommunication or lack of coordination between overlay and
underlay networks can lead to an inefficient usage of network
resources.
- When multiple overlays co-exist on top of a common underlay
network, the lack of coordination between overlays can lead
to performance issues and/or resource usage inefficiencies.
- Traffic carried over an overlay may not traverse firewalls and - Tunneling is used to aggregate traffic and hide tenant
NAT devices. addresses from the underlay network, and hence offer the
advantage of minimizing the amount of forwarding state required
within the underlay network
- Multicast service scalability: Multicast support may be - Decoupling of the overlay addresses (MAC and IP) used by VMs
required in the underlay network to address tenant flood from the underlay network for tenant separation and separation
containment or efficient multicast handling. The underlay may of the tenant address spaces from the underlay address space.
also be required to maintain multicast state on a per-tenant
basis, or even on a per-individual multicast flow of a given
tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
- Hash-based load balancing may not be optimal as the hash - Support of a large number of virtual network identifiers
algorithm may not work well due to the limited number of
combinations of tunnel source and destination addresses. Other
NVO3 mechanisms may use additional entropy information than
source and destination addresses.
4.2. Overlay issues to consider Overlay networks also create several challenges:
4.2.1. Data plane vs Control plane driven - Overlay networks have typically no control of underlay networks
and lack underlay network information (e.g. underlay
utilization):
In the case of an L2 NVE, it is possible to dynamically learn MAC - Overlay networks and/or their associated management entities
addresses against VAPs. It is also possible that such addresses be typically probe the network to measure link or path
known and controlled via management or a control protocol for both properties, such as available bandwidth or packet loss rate.
L2 NVEs and L3 NVEs. Dynamic data plane learning implies that It is difficult to accurately evaluate network properties. It
flooding of unknown destinations be supported and hence implies that might be preferable for the underlay network to expose usage
broadcast and/or multicast be supported or that ingress replication and performance information.
be used as described in section 4.2.3. Multicasting in the underlay - Miscommunication or lack of coordination between overlay and
network for dynamic learning may lead to significant scalability underlay networks can lead to an inefficient usage of network
limitations. Specific forwarding rules must be enforced to prevent resources.
loops from happening. This can be achieved using a spanning tree, a - When multiple overlays co-exist on top of a common underlay
shortest path tree, or a split-horizon mesh. network, the lack of coordination between overlays can lead
to performance issues and/or resource usage inefficiencies.
It should be noted that the amount of state to be distributed is - Traffic carried over an overlay may not traverse firewalls and
dependent upon network topology and the number of virtual machines. NAT devices.
Different forms of caching can also be utilized to minimize state
distribution between the various elements. The control plane should
not require an NVE to maintain the locations of all the Tenant
Systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane.
4.2.2. Coordination between data plane and control plane - Multicast service scalability: Multicast support may be
required in the underlay network to address tenant flood
containment or efficient multicast handling. The underlay may
also be required to maintain multicast state on a per-tenant
basis, or even on a per-individual multicast flow of a given
tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
For an L2 NVE, the NVE needs to be able to determine MAC addresses - Hash-based load balancing may not be optimal as the hash
of the Tenant Systems connected via a VAP. This can be achieved via algorithm may not work well due to the limited number of
dataplane learning or a control plane. For an L3 NVE, the NVE needs combinations of tunnel source and destination addresses. Other
to be able to determine IP addresses of the Tenant Systems connected NVO3 mechanisms may use additional entropy information than
via a VAP. source and destination addresses.
In both cases, coordination with the NVE control protocol is needed 4.2. Overlay issues to consider
such that when the NVE determines that the set of addresses behind a
VAP has changed, it triggers the NVE control plane to distribute
this information to its peers.
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic 4.2.1. Data plane vs Control plane driven
There are several options to support packet replication needed for In the case of an L2 NVE, it is possible to dynamically learn MAC
broadcast, unknown unicast and multicast. Typical methods include: addresses against VAPs. It is also possible that such addresses be
known and controlled via management or a control protocol for both
L2 NVEs and L3 NVEs. Dynamic data plane learning implies that
flooding of unknown destinations be supported and hence implies that
broadcast and/or multicast be supported or that ingress replication
be used as described in section 4.2.3. Multicasting in the underlay
network for dynamic learning may lead to significant scalability
limitations. Specific forwarding rules must be enforced to prevent
loops from happening. This can be achieved using a spanning tree, a
shortest path tree, or a split-horizon mesh.
- Ingress replication It should be noted that the amount of state to be distributed is
dependent upon network topology and the number of virtual machines.
Different forms of caching can also be utilized to minimize state
distribution between the various elements. The control plane should
not require an NVE to maintain the locations of all the Tenant
Systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane.
- Use of underlay multicast trees 4.2.2. Coordination between data plane and control plane
There is a bandwidth vs state trade-off between the two approaches. For an L2 NVE, the NVE needs to be able to determine MAC addresses
Depending upon the degree of replication required (i.e. the number of the Tenant Systems connected via a VAP. This can be achieved via
of hosts per group) and the amount of multicast state to maintain, dataplane learning or a control plane. For an L3 NVE, the NVE needs
trading bandwidth for state should be considered. to be able to determine IP addresses of the Tenant Systems connected
via a VAP.
When the number of hosts per group is large, the use of underlay In both cases, coordination with the NVE control protocol is needed
multicast trees may be more appropriate. When the number of hosts is such that when the NVE determines that the set of addresses behind a
small (e.g. 2-3) and/or the amount of multicast traffic is small, VAP has changed, it triggers the NVE control plane to distribute
ingress replication may not be an issue. this information to its peers.
Depending upon the size of the data center network and hence the 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic
number of (S,G) entries, and also the duration of multicast flows,
the use of underlay multicast trees can be a challenge.
When flows are well known, it is possible to pre-provision such There are several options to support packet replication needed for
multicast trees. However, it is often difficult to predict broadcast, unknown unicast and multicast. Typical methods include:
application flows ahead of time, and hence programming of (S,G)
entries for short-lived flows could be impractical.
A possible trade-off is to use in the underlay shared multicast - Ingress replication
trees as opposed to dedicated multicast trees.
4.2.4. Path MTU - Use of underlay multicast trees
When using overlay tunneling, an outer header is added to the There is a bandwidth vs state trade-off between the two approaches.
original frame. This can cause the MTU of the path to the egress Depending upon the degree of replication required (i.e. the number
tunnel endpoint to be exceeded. of hosts per group) and the amount of multicast state to maintain,
trading bandwidth for state should be considered.
It is usually not desirable to rely on IP fragmentation for When the number of hosts per group is large, the use of underlay
performance reasons. Ideally, the interface MTU as seen by a Tenant multicast trees may be more appropriate. When the number of hosts is
System is adjusted such that no fragmentation is needed. TCP will small (e.g. 2-3) and/or the amount of multicast traffic is small,
adjust its maximum segment size accordingly. ingress replication may not be an issue.
It is possible for the MTU to be configured manually or to be Depending upon the size of the data center network and hence the
discovered dynamically. Various Path MTU discovery techniques exist number of (S,G) entries, and also the duration of multicast flows,
in order to determine the proper MTU size to use: the use of underlay multicast trees can be a challenge.
- Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981] When flows are well known, it is possible to pre-provision such
multicast trees. However, it is often difficult to predict
application flows ahead of time, and hence programming of (S,G)
entries for short-lived flows could be impractical.
- Tenant Systems rely on ICMP messages to discover the MTU of the A possible trade-off is to use in the underlay shared multicast
end-to-end path to its destination. This method is not always trees as opposed to dedicated multicast trees.
possible, such as when traversing middle boxes (e.g. firewalls)
which disable ICMP for security reasons
- Extended MTU Path Discovery techniques such as defined in 4.2.4. Path MTU
[RFC4821]
It is also possible to rely on the NVE to perform segmentation and When using overlay tunneling, an outer header is added to the
reassembly operations without relying on the Tenant Systems to know original frame. This can cause the MTU of the path to the egress
about the end-to-end MTU. The assumption is that some hardware tunnel endpoint to be exceeded.
assist is available on the NVE node to perform such SAR operations.
However, fragmentation by the NVE can lead to performance and
congestion issues due to TCP dynamics and might require new
congestion avoidance mechanisms from the underlay network [FLOYD].
Finally, the underlay network may be designed in such a way that the It is usually not desirable to rely on IP fragmentation for
MTU can accommodate the extra tunneling and possibly additional NVO3 performance reasons. Ideally, the interface MTU as seen by a Tenant
header encapsulation overhead. System is adjusted such that no fragmentation is needed. TCP will
adjust its maximum segment size accordingly.
4.2.5. NVE location trade-offs It is possible for the MTU to be configured manually or to be
discovered dynamically. Various Path MTU discovery techniques exist
in order to determine the proper MTU size to use:
In the case of DC traffic, traffic originated from a VM is native - Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981]
Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements.
There are several criteria to consider when deciding where the NVE - Tenant Systems rely on ICMP messages to discover the MTU of the
function should happen: end-to-end path to its destination. This method is not always
possible, such as when traversing middle boxes (e.g. firewalls)
which disable ICMP for security reasons
- Processing and memory requirements - Extended MTU Path Discovery techniques such as defined in
[RFC4821]
- Datapath (e.g. lookups, filtering, encapsulation/decapsulation) - Tenant Systems rely on detection of receipt and loss of probe
packets at receivers and communication of that receipt/loss
information to senders in order to discover the MTU of the end-
to-end path to its destination
- Control plane processing (e.g. routing, signaling, OAM) and It is also possible to rely on the NVE to perform segmentation and
where specific control plane functions should be enabled reassembly operations without relying on the Tenant Systems to know
about the end-to-end MTU. The assumption is that some hardware
assist is available on the NVE node to perform such SAR operations.
However, fragmentation by the NVE can lead to performance and
congestion issues due to TCP dynamics and might require new
congestion avoidance mechanisms from the underlay network [FLOYD].
- FIB/RIB size Finally, the underlay network may be designed in such a way that the
MTU can accommodate the extra tunneling and possibly additional NVO3
header encapsulation overhead.
- Multicast support 4.2.5. NVE location trade-offs
- Routing/signaling protocols In the case of DC traffic, traffic originated from a VM is native
Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements.
- Packet replication capability There are several criteria to consider when deciding where the NVE
function should happen:
- Multicast FIB - Processing and memory requirements
- Fragmentation support - Datapath (e.g. lookups, filtering, encapsulation/decapsulation)
- QoS support (e.g. marking, policing, queuing) - Control plane processing (e.g. routing, signaling, OAM) and
where specific control plane functions should be enabled
- Resiliency - FIB/RIB size
- Multicast support
4.2.6. Interaction between network overlays and underlays - Routing/signaling protocols
When multiple overlays co-exist on top of a common underlay network, - Packet replication capability
resources (e.g., bandwidth) should be provisioned to ensure that
traffic from overlays can be accommodated and QoS objectives can be
met. Overlays can have partially overlapping paths (nodes and
links).
Each overlay is selfish by nature. It sends traffic so as to - Multicast FIB
optimize its own performance without considering the impact on other
overlays, unless the underlay paths are traffic engineered on a per
overlay basis to avoid congestion of underlay resources.
Better visibility between overlays and underlays, or generally - Fragmentation support
coordination in placing overlay demand on an underlay network, may
be achieved by providing mechanisms to exchange performance and
liveliness information between the underlay and overlay(s) or the
use of such information by a coordination system. Such information
may include:
- Performance metrics (throughput, delay, loss, jitter) - QoS support (e.g. marking, policing, queuing)
- Cost metrics - Resiliency
5. Security Considerations 4.2.6. Interaction between network overlays and underlays
NVO3 solutions must at least consider and address the following: When multiple overlays co-exist on top of a common underlay network,
resources (e.g., bandwidth) should be provisioned to ensure that
traffic from overlays can be accommodated and QoS objectives can be
met. Overlays can have partially overlapping paths (nodes and
links).
- Secure and authenticated communication between an NVE and an NVE Each overlay is selfish by nature. It sends traffic so as to
management system and/or control system. optimize its own performance without considering the impact on other
overlays, unless the underlay paths are traffic engineered on a per
overlay basis to avoid congestion of underlay resources.
- Isolation between tenant overlay networks. The use of per-tenant Better visibility between overlays and underlays, or generally
FIB tables (VNIs) on an NVE is essential. coordination in placing overlay demand on an underlay network, may
be achieved by providing mechanisms to exchange performance and
liveliness information between the underlay and overlay(s) or the
use of such information by a coordination system. Such information
may include:
- Security of any protocol used to carry overlay network - Performance metrics (throughput, delay, loss, jitter)
information.
- Preventing packets from reaching the wrong NVI, especially during - Cost metrics
VM moves.
- It may desirable to restrict the types of information that can be 5. Security Considerations
exchanged between overlays and underlays (e.g. topology
information)
6. IANA Considerations Since NVEs and NVAs play a central role in NVO3, it is critical that
a secure access to NVEs and NVAs be ensured such that no
unauthorized access is possible.
IANA does not need to take any action for this draft. As discussed in section 3.1.5.2. , Tenant Systems identification is
based upon state that is often provided by management systems (e.g.
7. References a VM orchestration system in a virtualized environment). Secure
access to such management systems must also be ensured.
7.1. Informative References When an NVE receives data from a TS, the tenant identity needs to be
verified in order to guarantee that it is authorized to access the
corresponding VN. This can be achieved by identifying incoming
packets against specific VAPs in some cases. In other circumstances,
authentication may be necessary.
[NVOPS] Narten, T. et al, "Problem Statement : Overlays for Network Data integrity can be assured if authorized access to NVEs, NVAs,
Virtualization", draft-narten-nvo3-overlay-problem- and intermediate underlay nodes is ensured. Otherwise, encryption
statement (work in progress) must be used.
[OF] Open Networking Foundation, "OpenFlow Switch Specification NVO3 provides data confidentiality through data separation. The use
v1.4.0" of both VNIs and tunneling of tenant traffic by NVEs ensures that
NVO3 data is kept in a separate context and thus separated from
other tenant traffic. When NVO3 data traverses untrusted networks,
data encryption may be needed.
[FLOYD] Sally Floyd, Allyn Romanow, "Dynamics of TCP Traffic over Not only tenant data but also NVO3 control data must be secured
ATM Networks", IEEE JSAC, V. 13 N. 4, May 1995 (e.g. control traffic between NVAs and NVEs, between NVAs and
between NVEs).
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private It may also be desirable to restrict the types of information that
Networks (VPNs)", RFC 4364, February 2006. can be exchanged between overlays and underlays (e.g. topology
information).
[RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS) 6. IANA Considerations
Using BGP for auto-discovery and Signaling", RFC4761,
January 2007
[RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS) IANA does not need to take any action for this draft.
Using Label Distribution Protocol (LDP) Signaling",
RFC4762, January 2007
[EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft- 7. References
ietf-l2vpn-evpn (work in progress)
[RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990 7.1. Informative References
[RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981, [NVOPS] Narten, T. et al, "Problem Statement : Overlays for Network
August 1996 Virtualization", draft-narten-nvo3-overlay-problem-
statement (work in progress)
[RFC4821] Mathis, M. et al, "Packetization Layer Path MTU [OF] Open Networking Foundation, "OpenFlow Switch Specification
Discovery", RFC4821, March 2007 v1.4.0"
[RFC6820] Narten, T. et al, "Address Resolution Problems in Large [FLOYD] Sally Floyd, Allyn Romanow, "Dynamics of TCP Traffic over
Data Center Networks", RFC6820, January 2013 ATM Networks", IEEE JSAC, V. 13 N. 4, May 1995
8. Acknowledgments [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, February 2006.
In addition to the authors the following people have contributed to [RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS)
this document: Using BGP for auto-discovery and Signaling", RFC4761,
January 2007
Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten, [RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS)
Larry Kreeger. Using Label Distribution Protocol (LDP) Signaling",
RFC4762, January 2007
This document was prepared using 2-Word-v2.0.template.dot. [EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft-
ietf-l2vpn-evpn (work in progress)
Authors' Addresses [RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990
Marc Lasserre [RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981,
Alcatel-Lucent August 1996
Email: marc.lasserre@alcatel-lucent.com
Florin Balus [RFC4821] Mathis, M. et al, "Packetization Layer Path MTU
Alcatel-Lucent Discovery", RFC4821, March 2007
777 E. Middlefield Road
Mountain View, CA, USA 94043
Email: florin.balus@alcatel-lucent.com
Thomas Morin [RFC6820] Narten, T. et al, "Address Resolution Problems in Large
France Telecom Orange Data Center Networks", RFC6820, January 2013
Email: thomas.morin@orange.com
Nabil Bitar 8. Acknowledgments
Verizon
40 Sylvan Road
Waltham, MA 02145
Email: nabil.bitar@verizon.com
Yakov Rekhter In addition to the authors the following people have contributed to
Juniper this document:
Email: yakov@juniper.net
Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten,
Larry Kreeger, David Black.
This document was prepared using 2-Word-v2.0.template.dot.
Authors' Addresses
Marc Lasserre
Alcatel-Lucent
Email: marc.lasserre@alcatel-lucent.com
Florin Balus
Alcatel-Lucent
777 E. Middlefield Road
Mountain View, CA, USA 94043
Email: florin.balus@alcatel-lucent.com
Thomas Morin
France Telecom Orange
Email: thomas.morin@orange.com
Nabil Bitar
Verizon
40 Sylvan Road
Waltham, MA 02145
Email: nabil.bitar@verizon.com
Yakov Rekhter
Juniper
Email: yakov@juniper.net
 End of changes. 218 change blocks. 
823 lines changed or deleted 833 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/