draft-ietf-nvo3-framework-07.txt   draft-ietf-nvo3-framework-08.txt 
Internet Engineering Task Force Marc Lasserre
Internet Draft Florin Balus
Intended status: Informational Alcatel-Lucent
Expires: Jan 2015
Thomas Morin
France Telecom Orange
Internet Engineering Task Force Marc Lasserre Nabil Bitar
Internet Draft Florin Balus Verizon
Intended status: Informational Alcatel-Lucent
Expires: Dec 2014
Thomas Morin
France Telecom Orange
Nabil Bitar Yakov Rekhter
Verizon Juniper
Yakov Rekhter July 2, 2014
Juniper
June 5, 2014 Framework for DC Network Virtualization
draft-ietf-nvo3-framework-08.txt
Framework for DC Network Virtualization Abstract
draft-ietf-nvo3-framework-07.txt
Abstract This document provides a framework for Data Center (DC) Network
Virtualization Overlays (NVO3) and it defines a reference model
along with logical components required to design a solution.
This document provides a framework for Data Center (DC) Network Status of this Memo
Virtualization Overlays (NVO3) and it defines a reference model
along with logical components required to design a solution.
Status of this Memo This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
This Internet-Draft is submitted in full conformance with the Internet-Drafts are working documents of the Internet Engineering
provisions of BCP 78 and BCP 79. Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are draft documents valid for a maximum of six
Task Force (IETF). Note that other groups may also distribute months and may be updated, replaced, or obsoleted by other documents
working documents as Internet-Drafts. The list of current Internet- at any time. It is inappropriate to use Internet-Drafts as
Drafts is at http://datatracker.ietf.org/drafts/current/. reference material or to cite them other than as "work in progress."
Internet-Drafts are draft documents valid for a maximum of six This Internet-Draft will expire on Jan 2, 2015.
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on Dec 5, 2014. Copyright Notice
Copyright Notice Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
Copyright (c) 2014 IETF Trust and the persons identified as the This document is subject to BCP 78 and the IETF Trust's Legal
document authors. All rights reserved. Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
This document is subject to BCP 78 and the IETF Trust's Legal Table of Contents
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents 1. Introduction..................................................3
1.1. General terminology......................................3
1.2. DC network architecture..................................6
2. Reference Models..............................................8
2.1. Generic Reference Model..................................8
2.2. NVE Reference Model.....................................10
2.3. NVE Service Types.......................................10
2.3.1. L2 NVE providing Ethernet LAN-like service.........11
2.3.2. L3 NVE providing IP/VRF-like service...............11
2.4. Operational Management Considerations...................11
3. Functional components........................................12
3.1. Service Virtualization Components.......................12
3.1.1. Virtual Access Points (VAPs).......................12
3.1.2. Virtual Network Instance (VNI).....................12
3.1.3. Overlay Modules and VN Context.....................12
3.1.4. Tunnel Overlays and Encapsulation options..........13
3.1.5. Control Plane Components...........................14
3.1.5.1. Distributed vs Centralized Control Plane.........14
3.1.5.2. Auto-provisioning/Service discovery..............14
3.1.5.3. Address advertisement and tunnel mapping.........15
3.1.5.4. Overlay Tunneling................................15
3.2. Multi-homing............................................16
3.3. VM Mobility.............................................17
4. Key aspects of overlay networks..............................17
4.1. Pros & Cons.............................................17
4.2. Overlay issues to consider..............................19
4.2.1. Data plane vs Control plane driven.................19
4.2.2. Coordination between data plane and control plane..19
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic..................................................19
4.2.4. Path MTU..........................................20
4.2.5. NVE location trade-offs...........................21
4.2.6. Interaction between network overlays and underlays.22
5. Security Considerations.....................................22
6. IANA Considerations.........................................23
7. References..................................................23
7.1. Informative References.................................23
8. Acknowledgments.............................................24
1. Introduction.................................................3 1. Introduction
1.1. General terminology.....................................3
1.2. DC network architecture.................................6
2. Reference Models.............................................8
2.1. Generic Reference Model.................................8
2.2. NVE Reference Model....................................10
2.3. NVE Service Types......................................10
2.3.1. L2 NVE providing Ethernet LAN-like service........11
2.3.2. L3 NVE providing IP/VRF-like service..............11
2.4. Operational Management Considerations..................11
3. Functional components.......................................12
3.1. Service Virtualization Components......................12
3.1.1. Virtual Access Points (VAPs)......................12
3.1.2. Virtual Network Instance (VNI)....................12
3.1.3. Overlay Modules and VN Context....................12
3.1.4. Tunnel Overlays and Encapsulation options.........13
3.1.5. Control Plane Components..........................14
3.1.5.1. Distributed vs Centralized Control Plane........14
3.1.5.2. Auto-provisioning/Service discovery.............14
3.1.5.3. Address advertisement and tunnel mapping........15
3.1.5.4. Overlay Tunneling...............................15
3.2. Multi-homing...........................................16
3.3. VM Mobility............................................17
4. Key aspects of overlay networks.............................17
4.1. Pros & Cons............................................17
4.2. Overlay issues to consider.............................19
4.2.1. Data plane vs Control plane driven................19
4.2.2. Coordination between data plane and control plane.19
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic..................................................19
4.2.4. Path MTU..........................................20
4.2.5. NVE location trade-offs...........................21
4.2.6. Interaction between network overlays and underlays.22
5. Security Considerations.....................................22
6. IANA Considerations.........................................23
7. References..................................................23
7.1. Informative References.................................23
8. Acknowledgments.............................................24
1. Introduction This document provides a framework for Data Center (DC) Network
Virtualization over Layer3 (L3) tunnels. This framework is intended
to aid in standardizing protocols and mechanisms to support large-
scale network virtualization for data centers.
This document provides a framework for Data Center (DC) Network [NVOPS] defines the rationale for using overlay networks in order to
Virtualization over Layer3 (L3) tunnels. This framework is intended build large multi-tenant data center networks. Compute, storage and
to aid in standardizing protocols and mechanisms to support large- network virtualization are often used in these large data centers to
scale network virtualization for data centers. support a large number of communication domains and end systems.
[NVOPS] defines the rationale for using overlay networks in order to This document provides reference models and functional components of
build large multi-tenant data center networks. Compute, storage and data center overlay networks as well as a discussion of technical
network virtualization are often used in these large data centers to issues that have to be addressed.
support a large number of communication domains and end systems.
This document provides reference models and functional components of 1.1. General terminology
data center overlay networks as well as a discussion of technical
issues that have to be addressed.
1.1. General terminology This document uses the following terminology:
This document uses the following terminology: NVO3 Network: An overlay network that provides a Layer2 (L2) or
Layer3 (L3) service to Tenant Systems over an L3 underlay network
using the architecture and protocols as defined by the NVO3 Working
Group.
NVO3 Network: An overlay network that provides a Layer2 (L2) or Network Virtualization Edge (NVE). An NVE is the network entity that
Layer3 (L3) service to Tenant Systems over an L3 underlay network sits at the edge of an underlay network and implements L2 and/or L3
using the architecture and protocols as defined by the NVO3 Working network virtualization functions. The network-facing side of the NVE
Group. uses the underlying L3 network to tunnel tenant frames to and from
other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
Network Virtualization Edge (NVE). An NVE is the network entity that Virtual Network (VN): A VN is a logical abstraction of a physical
sits at the edge of an underlay network and implements L2 and/or L3 network that provides L2 or L3 network services to a set of Tenant
network virtualization functions. The network-facing side of the NVE Systems. A VN is also known as a Closed User Group (CUG).
uses the underlying L3 network to tunnel tenant frames to and from
other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
Virtual Network (VN): A VN is a logical abstraction of a physical Virtual Network Instance (VNI): A specific instance of a VN from the
network that provides L2 or L3 network services to a set of Tenant perspective of an NVE.
Systems. A VN is also known as a Closed User Group (CUG).
Virtual Network Instance (VNI): A specific instance of a VN from the Virtual Network Context (VN Context) Identifier: Field in overlay
perspective of an NVE. encapsulation header that identifies the specific VN the packet
belongs to. The egress NVE uses the VN Context identifier to deliver
the packet to the correct Tenant System. The VN Context identifier
can be a locally significant identifier or a globally unique
identifier.
Virtual Network Context (VN Context) Identifier: Field in overlay Underlay or Underlying Network: The network that provides the
encapsulation header that identifies the specific VN the packet connectivity among NVEs and over which NVO3 packets are tunneled,
belongs to. The egress NVE uses the VN Context identifier to deliver where an NVO3 packet carries an NVO3 overlay header followed by a
the packet to the correct Tenant System. The VN Context identifier tenant packet. The Underlay Network does not need to be aware that
can be a locally significant identifier or a globally unique it is carrying NVO3 packets. Addresses on the Underlay Network
identifier. appear as "outer addresses" in encapsulated NVO3 packets. In
general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is IP.
Underlay or Underlying Network: The network that provides the Data Center (DC): A physical complex housing physical servers,
connectivity among NVEs and over which NVO3 packets are tunneled, network switches and routers, network service appliances and
where an NVO3 packet carries an NVO3 overlay header followed by a networked storage. The purpose of a Data Center is to provide
tenant packet. The Underlay Network does not need to be aware that application, compute and/or storage services. One such service is
it is carrying NVO3 packets. Addresses on the Underlay Network virtualized infrastructure data center services, also known as
appear as "outer addresses" in encapsulated NVO3 packets. In Infrastructure as a Service.
general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is IP.
Data Center (DC): A physical complex housing physical servers, Virtual Data Center (Virtual DC): A container for virtualized
network switches and routers, network service appliances and compute, storage and network services. A Virtual DC is associated
networked storage. The purpose of a Data Center is to provide with a single tenant, and can contain multiple VNs and Tenant
application, compute and/or storage services. One such service is Systems connected to one or more of these VNs.
virtualized infrastructure data center services, also known as
Infrastructure as a Service.
Virtual Data Center (Virtual DC): A container for virtualized Virtual machine (VM): A software implementation of a physical
compute, storage and network services. A Virtual DC is associated machine that runs programs as if they were executing on a physical,
with a single tenant, and can contain multiple VNs and Tenant non-virtualized machine. Applications (generally) do not know they
Systems connected to one or more of these VNs. are running on a VM as opposed to running on a "bare metal" host or
server, though some systems provide a para-virtualization
environment that allows an operating system or application to be
aware of the presence of virtualization for optimization purposes.
Virtual machine (VM): A software implementation of a physical Hypervisor: Software running on a server that allows multiple VMs to
machine that runs programs as if they were executing on a physical, run on the same physical server. The hypervisor manages and provides
non-virtualized machine. Applications (generally) do not know they shared compute/memory/storage and network connectivity to the VMs
are running on a VM as opposed to running on a "bare metal" host or that it hosts. Hypervisors often embed a Virtual Switch (see below).
server, though some systems provide a para-virtualization
environment that allows an operating system or application to be
aware of the presence of virtualization for optimization purposes.
Hypervisor: Software running on a server that allows multiple VMs to Server: A physical end host machine that runs user applications. A
run on the same physical server. The hypervisor manages and provides standalone (or "bare metal") server runs a conventional operating
shared compute/memory/storage and network connectivity to the VMs system hosting a single-tenant application. A virtualized server
that it hosts. Hypervisors often embed a Virtual Switch (see below). runs a hypervisor supporting one or more VMs.
Server: A physical end host machine that runs user applications. A Virtual Switch (vSwitch): A function within a Hypervisor (typically
standalone (or "bare metal") server runs a conventional operating implemented in software) that provides similar forwarding services
system hosting a single-tenant application. A virtualized server to a physical Ethernet switch. A vSwitch forwards Ethernet frames
runs a hypervisor supporting one or more VMs. between VMs running on the same server, or between a VM and a
physical NIC card connecting the server to a physical Ethernet
switch or router. A vSwitch also enforces network isolation between
VMs that by policy are not permitted to communicate with each other
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
enabled on the host server.
Virtual Switch (vSwitch): A function within a Hypervisor (typically Tenant: The customer using a virtual network and any associated
implemented in software) that provides similar forwarding services resources (e.g., compute, storage and network). A tenant could be
to a physical Ethernet switch. A vSwitch forwards Ethernet frames an enterprise, or a department/organization within an enterprise.
between VMs running on the same server, or between a VM and a
physical NIC card connecting the server to a physical Ethernet
switch or router. A vSwitch also enforces network isolation between
VMs that by policy are not permitted to communicate with each other
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
enabled on the host server.
Tenant: The customer using a virtual network and any associated Tenant System: A physical or virtual system that can play the role
resources (e.g., compute, storage and network). A tenant could be of a host, or a forwarding element such as a router, switch,
an enterprise, or a department/organization within an enterprise. firewall, etc. It belongs to a single tenant and connects to one or
more VNs of that tenant.
Tenant System: A physical or virtual system that can play the role Tenant Separation: Tenant Separation refers to isolating traffic of
of a host, or a forwarding element such as a router, switch, different tenants such that traffic from one tenant is not visible
firewall, etc. It belongs to a single tenant and connects to one or to or delivered to another tenant, except when allowed by policy.
more VNs of that tenant. Tenant Separation also refers to address space separation, whereby
different tenants can use the same address space without conflict.
Tenant Separation: Tenant Separation refers to isolating traffic of Virtual Access Points (VAPs): A logical connection point on the NVE
different tenants such that traffic from one tenant is not visible for connecting a Tenant System to a virtual network. Tenant Systems
to or delivered to another tenant, except when allowed by policy. connect to VNIs at an NVE through VAPs. VAPs can be physical ports
Tenant Separation also refers to address space separation, whereby or virtual ports identified through logical interface identifiers
different tenants can use the same address space without conflict. (e.g., VLAN ID, internal vSwitch Interface ID connected to a VM).
Virtual Access Points (VAPs): A logical connection point on the NVE End Device: A physical device that connects directly to the DC
for connecting a Tenant System to a virtual network. Tenant Systems Underlay Network. This is in contrast to a Tenant System, which
connect to VNIs at an NVE through VAPs. VAPs can be physical ports connects to a corresponding tenant VN. An End Device is administered
or virtual ports identified through logical interface identifiers by the DC operator rather than a tenant, and is part of the DC
(e.g., VLAN ID, internal vSwitch Interface ID connected to a VM). infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
End Device: A physical device that connects directly to the DC Network Virtualization Authority (NVA): Entity that provides
Underlay Network. This is in contrast to a Tenant System, which reachability and forwarding information to NVEs.
connects to a corresponding tenant VN. An End Device is administered
by the DC operator rather than a tenant, and is part of the DC
infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
Network Virtualization Authority (NVA): Entity that provides 1.2. DC network architecture
reachability and forwarding information to NVEs.
1.2. DC network architecture A generic architecture for Data Centers is depicted in Figure 1:
A generic architecture for Data Centers is depicted in Figure 1: ,---------.
,' `.
( IP/MPLS WAN )
`. ,'
`-+------+'
\ /
+--------+ +--------+
| DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| /
.--. .--.
( ' '.--.
.-.' Intra-DC '
( network )
( .'-'
'--'._.'. )\ \
/ / '--' \ \
/ / | | \ \
+--------+ +--------+ +--------+
| access | | access | | access |
| switch | | switch | | switch |
+--------+ +--------+ +--------+
/ \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------'
: End : : End : : End : : End :
: Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------'
,---------. Figure 1 : A Generic Architecture for Data Centers
,' `.
( IP/MPLS WAN )
`. ,'
`-+------+'
\ /
+--------+ +--------+
| DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| /
.--. .--.
( ' '.--.
.-.' Intra-DC '
( network )
( .'-'
'--'._.'. )\ \
/ / '--' \ \
/ / | | \ \
+--------+ +--------+ +--------+
| access | | access | | access |
| switch | | switch | | switch |
+--------+ +--------+ +--------+
/ \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------'
: End : : End : : End : : End :
: Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------'
Figure 1 : A Generic Architecture for Data Centers An example of multi-tier DC network architecture is presented in
Figure 1. It provides a view of physical components inside a DC.
An example of multi-tier DC network architecture is presented in A DC network is usually composed of intra-DC networks and network
Figure 1. It provides a view of physical components inside a DC. services, and inter-DC network and network connectivity services.
A DC network is usually composed of intra-DC networks and network DC networking elements can act as strict L2 switches and/or provide
services, and inter-DC network and network connectivity services. IP routing capabilities, including network service virtualization.
DC networking elements can act as strict L2 switches and/or provide In some DC architectures, some tier layers could provide L2 and/or
IP routing capabilities, including network service virtualization. L3 services. In addition, some tier layers may be collapsed, and
Internet connectivity, inter-DC connectivity and VPN support may be
handled by a smaller number of nodes. Nevertheless, one can assume
that the network functional blocks in a DC fit in the architecture
depicted in Figure 1.
In some DC architectures, some tier layers could provide L2 and/or The following components can be present in a DC:
L3 services. In addition, some tier layers may be collapsed, and
Internet connectivity, inter-DC connectivity and VPN support may be
handled by a smaller number of nodes. Nevertheless, one can assume
that the network functional blocks in a DC fit in the architecture
depicted in Figure 1.
The following components can be present in a DC: - Access switch: Hardware-based Ethernet switch aggregating all
Ethernet links from the End Devices in a rack representing the
entry point in the physical DC network for the hosts. It may also
provide routing functionality, virtual IP network connectivity, or
Layer2 tunneling over IP for instance. Access switches are usually
multi-homed to aggregation switches in the Intra-DC network. A
typical example of an access switch is a Top of Rack (ToR) switch.
Other deployment scenarios may use an intermediate Blade Switch
before the ToR, or an EoR (End of Row) switch, to provide similar
functions to a ToR.
- Access switch: Hardware-based Ethernet switch aggregating all - Intra-DC Network: Network composed of high capacity core nodes
Ethernet links from the End Devices in a rack representing the (Ethernet switches/routers). Core nodes may provide virtual
entry point in the physical DC network for the hosts. It may also Ethernet bridging and/or IP routing services.
provide routing functionality, virtual IP network connectivity, or
Layer2 tunneling over IP for instance. Access switches are usually
multi-homed to aggregation switches in the Intra-DC network. A
typical example of an access switch is a Top of Rack (ToR) switch.
Other deployment scenarios may use an intermediate Blade Switch
before the ToR, or an EoR (End of Row) switch, to provide similar
functions to a ToR.
- Intra-DC Network: Network composed of high capacity core nodes - DC Gateway (DC GW): Gateway to the outside world providing DC
(Ethernet switches/routers). Core nodes may provide virtual Interconnect and connectivity to Internet and VPN customers. In
Ethernet bridging and/or IP routing services. the current DC network model, this may be simply a router
connected to the Internet and/or an IP Virtual Private Network
(VPN)/L2VPN PE. Some network implementations may dedicate DC GWs
for different connectivity types (e.g., a DC GW for Internet, and
another for VPN).
- DC Gateway (DC GW): Gateway to the outside world providing DC Note that End Devices may be single or multi-homed to access
Interconnect and connectivity to Internet and VPN customers. In switches.
the current DC network model, this may be simply a router
connected to the Internet and/or an IP Virtual Private Network
(VPN)/L2VPN PE. Some network implementations may dedicate DC GWs
for different connectivity types (e.g., a DC GW for Internet, and
another for VPN).
Note that End Devices may be single or multi-homed to access 2. Reference Models
switches.
2. Reference Models 2.1. Generic Reference Model
2.1. Generic Reference Model Figure 2 depicts a DC reference model for network virtualization
overlay where NVEs provide a logical interconnect between Tenant
Systems that belong to a specific VN.
Figure 2 depicts a DC reference model for network virtualization +--------+ +--------+
overlay where NVEs provide a logical interconnect between Tenant | Tenant +--+ +----| Tenant |
Systems that belong to a specific VN. | System | | (') | System |
+--------+ | ................. ( ) +--------+
| +---+ +---+ (_)
+--|NVE|---+ +---|NVE|-----+
+---+ | | +---+
/ . +-----+ .
/ . +--| NVA |--+ .
/ . | +-----+ \ .
| . | \ .
| . | Overlay +--+--++--------+
+--------+ | . | Network | NVE || Tenant |
| Tenant +--+ . | | || System |
| System | . \ +---+ +--+--++--------+
+--------+ .....|NVE|.........
+---+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
+--------+ +--------+ Figure 2 : Generic reference model for DC network virtualization
| Tenant +--+ +----| Tenant | overlay
| System | | (') | System |
+--------+ | ................. ( ) +--------+
| +---+ +---+ (_)
+--|NVE|---+ +---|NVE|-----+
+---+ | | +---+
/ . +-----+ .
/ . +--| NVA |--+ .
/ . | +-----+ \ .
| . | \ .
| . | Overlay +--+--++--------+
+--------+ | . | Network | NVE || Tenant |
| Tenant +--+ . | | || System |
| System | . \ +---+ +--+--++--------+
+--------+ .....|NVE|.........
+---+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
Figure 2 : Generic reference model for DC network virtualization In order to obtain reachability information, NVEs may exchange
overlay information directly between themselves via a control plane
protocol. In this case, a control plane module resides in every NVE.
In order to obtain reachability information, NVEs may exchange It is also possible for NVEs to communicate with an external Network
information directly between themselves via a control plane Virtualization Authority (NVA) to obtain reachability and forwarding
protocol. In this case, a control plane module resides in every NVE. information. In this case, a protocol is used between NVEs and
NVA(s) to exchange information.
It is also possible for NVEs to communicate with an external Network It should be noted that NVAs may be organized in clusters for
Virtualization Authority (NVA) to obtain reachability and forwarding redundancy and scalability and can appear as one logically
information. In this case, a protocol is used between NVEs and centralized controller. In this case, inter-NVA communication is
NVA(s) to exchange information. necessary to synchronize state among nodes within a cluster or share
information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information
exchanged across clusters.
It should be noted that NVAs may be organized in clusters for A Tenant System can be attached to an NVE in several ways:
redundancy and scalability and can appear as one logically
centralized controller. In this case, inter-NVA communication is
necessary to synchronize state among nodes within a cluster or share
information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information
exchanged across clusters.
A Tenant System can be attached to an NVE in several ways: - locally, by being co-located in the same End Device
- locally, by being co-located in the same End Device - remotely, via a point-to-point connection or a switched network
- remotely, via a point-to-point connection or a switched network When an NVE is co-located with a Tenant System, the state of the
Tenant System can be determined without protocol assistance. For
instance, the operational status of a VM can be communicated via a
local API. When an NVE is remotely connected to a Tenant System, the
state of the Tenant System or NVE needs to be exchanged directly or
via a management entity, using a control plane protocol or API, or
directly via a dataplane protocol.
When an NVE is co-located with a Tenant System, the state of the The functional components in Figure 2 do not necessarily map
Tenant System can be determined without protocol assistance. For directly to the physical components described in Figure 1. For
instance, the operational status of a VM can be communicated via a example, an End Device can be a server blade with VMs and a virtual
local API. When an NVE is remotely connected to a Tenant System, the switch. A VM can be a Tenant System and the NVE functions may be
state of the Tenant System or NVE needs to be exchanged directly or performed by the host server. In this case, the Tenant System and
via a management entity, using a control plane protocol or API, or NVE function are co-located. Another example is the case where the
directly via a dataplane protocol. End Device is the Tenant System, and the NVE function can be
implemented by the connected ToR. In this case, the Tenant System
and NVE function are not co-located.
The functional components in Figure 2 do not necessarily map Underlay nodes utilize L3 technologies to interconnect NVE nodes.
directly to the physical components described in Figure 1. For These nodes perform forwarding based on outer L3 header information,
example, an End Device can be a server blade with VMs and a virtual and generally do not maintain per tenant-service state albeit some
switch. A VM can be a Tenant System and the NVE functions may be applications (e.g., multicast) may require control plane or
performed by the host server. In this case, the Tenant System and forwarding plane information that pertain to a tenant, group of
NVE function are co-located. Another example is the case where the tenants, tenant service or a set of services that belong to one or
End Device is the Tenant System, and the NVE function can be more tenants. Mechanisms to control the amount of state maintained
implemented by the connected ToR. In this case, the Tenant System in the underlay may be needed.
and NVE function are not co-located.
Underlay nodes utilize L3 technologies to interconnect NVE nodes. 2.2. NVE Reference Model
These nodes perform forwarding based on outer L3 header information,
and generally do not maintain per tenant-service state albeit some
applications (e.g., multicast) may require control plane or
forwarding plane information that pertain to a tenant, group of
tenants, tenant service or a set of services that belong to one or
more tenants. Mechanisms to control the amount of state maintained
in the underlay may be needed.
2.2. NVE Reference Model Figure 3 depicts the NVE reference model. One or more VNIs can be
instantiated on an NVE. A Tenant System interfaces with a
corresponding VNI via a VAP. An overlay module provides tunneling
overlay functions (e.g., encapsulation and decapsulation of tenant
traffic, tenant identification and mapping, etc.).
Figure 3 depicts the NVE reference model. One or more VNIs can be +-------- L3 Network -------+
instantiated on an NVE. A Tenant System interfaces with a | |
corresponding VNI via a VAP. An overlay module provides tunneling | Tunnel Overlay |
overlay functions (e.g., encapsulation and decapsulation of tenant +------------+---------+ +---------+------------+
traffic, tenant identification and mapping, etc.). | +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| |
| | | | | |
| +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+
| | | |
| | | |
Tenant Systems Tenant Systems
+-------- L3 Network -------+ Figure 3 : Generic NVE reference model
| |
| Tunnel Overlay |
+------------+---------+ +---------+------------+
| +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| |
| | | | | |
| +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+
| | | |
| | | |
Tenant Systems Tenant Systems
Figure 3 : Generic NVE reference model Note that some NVE functions (e.g., data plane and control plane
functions) may reside in one device or may be implemented separately
in different devices.
Note that some NVE functions (e.g., data plane and control plane 2.3. NVE Service Types
functions) may reside in one device or may be implemented separately
in different devices.
2.3. NVE Service Types An NVE provides different types of virtualized network services to
multiple tenants, i.e. an L2 service or an L3 service. Note that an
NVE may be capable of providing both L2 and L3 services for a
tenant. This section defines the service types and associated
attributes.
An NVE provides different types of virtualized network services to 2.3.1. L2 NVE providing Ethernet LAN-like service
multiple tenants, i.e. an L2 service or an L3 service. Note that an
NVE may be capable of providing both L2 and L3 services for a
tenant. This section defines the service types and associated
attributes.
2.3.1. L2 NVE providing Ethernet LAN-like service An L2 NVE implements Ethernet LAN emulation, an Ethernet based
multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or
EVPN [EVPN] service, where the Tenant Systems appear to be
interconnected by a LAN environment over an L3 overlay. As such, an
L2 NVE provides per-tenant virtual switching instance (L2 VNI), and
L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the
underlay. Note that the control plane for an L2 NVE could be
implemented locally on the NVE or in a separate control entity.
An L2 NVE implements Ethernet LAN emulation, an Ethernet based 2.3.2. L3 NVE providing IP/VRF-like service
multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or
EVPN [EVPN] service, where the Tenant Systems appear to be
interconnected by a LAN environment over an L3 overlay. As such, an
L2 NVE provides per-tenant virtual switching instance (L2 VNI), and
L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the
underlay. Note that the control plane for an L2 NVE could be
implemented locally on the NVE or in a separate control entity.
2.3.2. L3 NVE providing IP/VRF-like service An L3 NVE provides Virtualized IP forwarding service, similar to
IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service
definition perspective. That is, an L3 NVE provides per-tenant
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant IP packets across the underlay. Note that
routing could be performed locally on the NVE or in a separate
control entity.
An L3 NVE provides Virtualized IP forwarding service, similar to 2.4. Operational Management Considerations
IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service
definition perspective. That is, an L3 NVE provides per-tenant
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant IP packets across the underlay. Note that
routing could be performed locally on the NVE or in a separate
control entity.
2.4. Operational Management Considerations NVO3 services are overlay services over an IP underlay.
NVO3 services are overlay services over an IP underlay. As far as the IP underlay is concerned, existing IP OAM facilities
are used.
As far as the IP underlay is concerned, existing IP OAM facilities With regards to the NVO3 overlay, both L2 and L3 services can be
are used. offered. it is expected that existing fault and performance OAM
facilities will be used. Sections 4.1. and 4.2.6. below provide
further discussion of additional fault and performance management
issues to consider.
With regards to the NVO3 overlay, both L2 and L3 services can be As far as configuration is concerned, the DC environment is driven
offered. it is expected that existing fault and performance OAM by the need to bring new services up rapidly and is typically very
facilities will be used. Sections 4.1. and 4.2.6. below provide dynamic specifically in the context of virtualized services. It is
further discussion of additional fault and performance management therefore critical to automate the configuration of NVO3 services.
issues to consider.
As far as configuration is concerned, the DC environment is driven 3. Functional components
by the need to bring new services up rapidly and is typically very
dynamic specifically in the context of virtualized services. It is
therefore critical to automate the configuration of NVO3 services.
3. Functional components This section decomposes the Network Virtualization architecture into
functional components described in Figure 3 to make it easier to
discuss solution options for these components.
This section decomposes the Network Virtualization architecture into 3.1. Service Virtualization Components
functional components described in Figure 3 to make it easier to
discuss solution options for these components.
3.1. Service Virtualization Components 3.1.1. Virtual Access Points (VAPs)
3.1.1. Virtual Access Points (VAPs) Tenant Systems are connected to VNIs through Virtual Access Points
(VAPs).
Tenant Systems are connected to VNIs through Virtual Access Points VAPs can be physical ports or virtual ports identified through
(VAPs). logical interface identifiers (e.g., VLAN ID, internal vSwitch
Interface ID connected to a VM).
VAPs can be physical ports or virtual ports identified through 3.1.2. Virtual Network Instance (VNI)
logical interface identifiers (e.g., VLAN ID, internal vSwitch
Interface ID connected to a VM).
3.1.2. Virtual Network Instance (VNI) A VNI is a specific VN instance on an NVE. Each VNI defines a
forwarding context that contains reachability information and
policies.
A VNI is a specific VN instance on an NVE. Each VNI defines a 3.1.3. Overlay Modules and VN Context
forwarding context that contains reachability information and
policies.
3.1.3. Overlay Modules and VN Context Mechanisms for identifying each tenant service are required to allow
the simultaneous overlay of multiple tenant services over the same
underlay L3 network topology. In the data plane, each NVE, upon
sending a tenant packet, must be able to encode the VN Context for
the destination NVE in addition to the L3 tunneling information
(e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the
tenant packet.
Mechanisms for identifying each tenant service are required to allow The Overlay module provides tunneling overlay functions: tunnel
the simultaneous overlay of multiple tenant services over the same initiation/termination as in the case of stateful tunnels (see
underlay L3 network topology. In the data plane, each NVE, upon Section 3.1.4), and/or simply encapsulation/decapsulation of frames
sending a tenant packet, must be able to encode the VN Context for from VAPs/L3 underlay.
the destination NVE in addition to the L3 tunneling information
(e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the
tenant packet.
The Overlay module provides tunneling overlay functions: tunnel In a multi-tenant context, tunneling aggregates frames from/to
initiation/termination as in the case of stateful tunnels (see different VNIs. Tenant identification and traffic demultiplexing are
Section 3.1.4), and/or simply encapsulation/decapsulation of frames based on the VN Context identifier.
from VAPs/L3 underlay.
In a multi-tenant context, tunneling aggregates frames from/to The following approaches can be considered:
different VNIs. Tenant identification and traffic demultiplexing are
based on the VN Context identifier.
The following approaches can be considered: - VN Context identifier per Tenant: Globally unique (on a per-DC
administrative domain) VN identifier used to identify the
corresponding VNI. Examples of such identifiers in existing
technologies are IEEE VLAN IDs and ISID tags that identify virtual
L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively.
Note that multiple VN identifiers can belong to a tenant.
- VN Context identifier per Tenant: Globally unique (on a per-DC - One VN Context identifier per VNI: Each VNI value is automatically
administrative domain) VN identifier used to identify the generated by the egress NVE, or a control plane associated with
corresponding VNI. Examples of such identifiers in existing that NVE, and usually distributed by a control plane protocol to
technologies are IEEE VLAN IDs and ISID tags that identify virtual all the related NVEs. An example of this approach is the use of
L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively. per VRF MPLS labels in IP VPN [RFC4364]. The VNI value is
Note that multiple VN identifiers can belong to a tenant. therefore locally significant to the egress NVE.
- One VN Context identifier per VNI: Each VNI value is automatically - One VN Context identifier per VAP: A value locally significant to
generated by the egress NVE, or a control plane associated with an NVE is assigned and usually distributed by a control plane
that NVE, and usually distributed by a control plane protocol to protocol to identify a VAP. An example of this approach is the use
all the related NVEs. An example of this approach is the use of of per CE-PE MPLS labels in IP VPN [RFC4364].
per VRF MPLS labels in IP VPN [RFC4364]. The VNI value is
therefore locally significant to the egress NVE.
- One VN Context identifier per VAP: A value locally significant to Note that when using one VN Context per VNI or per VAP, an
an NVE is assigned and usually distributed by a control plane additional global identifier (e.g., a VN identifier or name) may be
protocol to identify a VAP. An example of this approach is the use used by the control plane to identify the Tenant context.
of per CE-PE MPLS labels in IP VPN [RFC4364].
Note that when using one VN Context per VNI or per VAP, an 3.1.4. Tunnel Overlays and Encapsulation options
additional global identifier (e.g., a VN identifier or name) may be
used by the control plane to identify the Tenant context.
3.1.4. Tunnel Overlays and Encapsulation options Once the VN context identifier is added to the frame, an L3 Tunnel
encapsulation is used to transport the frame to the destination NVE.
Once the VN context identifier is added to the frame, an L3 Tunnel Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS
encapsulation is used to transport the frame to the destination NVE. tunneling can be used. Tunneling could be stateless or stateful.
Stateless tunneling simply entails the encapsulation of a tenant
packet with another header necessary for forwarding the packet
across the underlay (e.g., IP tunneling over an IP underlay).
Stateful tunneling on the other hand entails maintaining tunneling
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions should
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to carry the packet between an ingress NVE and an egress NVE across
the underlay. It should be noted that stateful tunneling, especially
when configuration is involved, does impose management overhead and
scale constraints. When confidentiality is required, the use of
opportunistic encryption can be used as a stateless tunneling
solution.
Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS 3.1.5. Control Plane Components
tunneling can be used. Tunneling could be stateless or stateful.
Stateless tunneling simply entails the encapsulation of a tenant
packet with another header necessary for forwarding the packet
across the underlay (e.g., IP tunneling over an IP underlay).
Stateful tunneling on the other hand entails maintaining tunneling
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions should
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to carry the packet between an ingress NVE and an egress NVE across
the underlay. It should be noted that stateful tunneling, especially
when configuration is involved, does impose management overhead and
scale constraints. When confidentiality is required, the use of
opportunistic encryption can be used as a stateless tunneling
solution.
3.1.5. Control Plane Components 3.1.5.1. Distributed vs Centralized Control Plane
3.1.5.1. Distributed vs Centralized Control Plane A control/management plane entity can be centralized or distributed.
Both approaches have been used extensively in the past. The routing
model of the Internet is a good example of a distributed approach.
Transport networks have usually used a centralized approach to
manage transport paths.
A control/management plane entity can be centralized or distributed. It is also possible to combine the two approaches, i.e., using a
Both approaches have been used extensively in the past. The routing hybrid model. A global view of network state can have many benefits
model of the Internet is a good example of a distributed approach. but it does not preclude the use of distributed protocols within the
Transport networks have usually used a centralized approach to network. Centralized models provide a facility to maintain global
manage transport paths. state, and distribute that state to the network. When used in
combination with distributed protocols, greater network
efficiencies, improved reliability and robustness can be achieved.
Domain and/or deployment specific constraints define the balance
between centralized and distributed approaches.
It is also possible to combine the two approaches, i.e., using a 3.1.5.2. Auto-provisioning/Service discovery
hybrid model. A global view of network state can have many benefits
but it does not preclude the use of distributed protocols within the
network. Centralized models provide a facility to maintain global
state, and distribute that state to the network. When used in
combination with distributed protocols, greater network
efficiencies, improved reliability and robustness can be achieved.
Domain and/or deployment specific constraints define the balance
between centralized and distributed approaches.
3.1.5.2. Auto-provisioning/Service discovery NVEs must be able to identify the appropriate VNI for each Tenant
System. This is based on state information that is often provided by
external entities. For example, in an environment where a VM is a
Tenant System, this information is provided by VM orchestration
systems, since these are the only entities that have visibility of
which VM belongs to which tenant.
NVEs must be able to identify the appropriate VNI for each Tenant A mechanism for communicating this information to the NVE is
System. This is based on state information that is often provided by required. VAPs have to be created and mapped to the appropriate VNI.
external entities. For example, in an environment where a VM is a Depending upon the implementation, this control interface can be
Tenant System, this information is provided by VM orchestration implemented using an auto-discovery protocol between Tenant Systems
systems, since these are the only entities that have visibility of and their local NVE or through management entities. In either case,
which VM belongs to which tenant. appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant
isolation in the system.
A mechanism for communicating this information to the NVE is NVEs may learn reachability information to VNIs on other NVEs via a
required. VAPs have to be created and mapped to the appropriate VNI. control protocol that exchanges such information among NVEs, or via
Depending upon the implementation, this control interface can be a management control entity.
implemented using an auto-discovery protocol between Tenant Systems
and their local NVE or through management entities. In either case,
appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant
isolation in the system.
NVEs may learn reachability information to VNIs on other NVEs via a 3.1.5.3. Address advertisement and tunnel mapping
control protocol that exchanges such information among NVEs, or via
a management control entity.
3.1.5.3. Address advertisement and tunnel mapping As traffic reaches an ingress NVE on a VAP, a lookup is performed to
determine which NVE or local VAP the packet needs to be sent to. If
the packet is to be sent to another NVE, the packet is encapsulated
with a tunnel header containing the destination information
(destination IP address or MPLS label) of the egress NVE.
Intermediate nodes (between the ingress and egress NVEs) switch or
route traffic based upon the tunnel destination information.
As traffic reaches an ingress NVE on a VAP, a lookup is performed to A key step in the above process consists of identifying the
determine which NVE or local VAP the packet needs to be sent to. If destination NVE the packet is to be tunneled to. NVEs are
the packet is to be sent to another NVE, the packet is encapsulated responsible for maintaining a set of forwarding or mapping tables
with a tunnel header containing the destination information that hold the bindings between destination VM and egress NVE
(destination IP address or MPLS label) of the egress NVE. addresses. Several ways of populating these tables are possible:
Intermediate nodes (between the ingress and egress NVEs) switch or control plane driven, management plane driven, or data plane driven.
route traffic based upon the tunnel destination information.
A key step in the above process consists of identifying the When a control plane protocol is used to distribute address
destination NVE the packet is to be tunneled to. NVEs are reachability and tunneling information, the auto-
responsible for maintaining a set of forwarding or mapping tables provisioning/Service discovery could be accomplished by the same
that hold the bindings between destination VM and egress NVE protocol. In this scenario, the auto-provisioning/Service discovery
addresses. Several ways of populating these tables are possible: could be combined with (be inferred from) the address advertisement
control plane driven, management plane driven, or data plane driven. and associated tunnel mapping. Furthermore, a control plane protocol
that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling as
discussed in [RFC6820].
When a control plane protocol is used to distribute address 3.1.5.4. Overlay Tunneling
reachability and tunneling information, the auto-
provisioning/Service discovery could be accomplished by the same
protocol. In this scenario, the auto-provisioning/Service discovery
could be combined with (be inferred from) the address advertisement
and associated tunnel mapping. Furthermore, a control plane protocol
that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling as
discussed in [RFC6820].
3.1.5.4. Overlay Tunneling For overlay tunneling, and dependent upon the tunneling technology
used for encapsulating the Tenant System packets, it may be
sufficient to have one or more local NVE addresses assigned and used
in the source and destination fields of a tunneling encapsulation
header. Other information that is part of the
tunneling encapsulation header may also need to be configured. In
certain cases, local NVE configuration may be sufficient while in
other cases, some tunneling related information may need to
be shared among NVEs. The information that needs to be shared will
be technology dependent. For instance, potential information could
include tunnel identity, encapsulation type, and/or tunnel
resources. In certain cases, such as when using IP multicast in the
underlay, tunnels which interconnect NVEs may need to be
established. When tunneling information needs to be exchanged or
shared among NVEs, a control plane protocol may be required. For
instance, it may be necessary to provide active/standby status
information between NVEs, up/down status information,
pruning/grafting information for multicast tunnels, etc.
For overlay tunneling, and dependent upon the tunneling technology In addition, a control plane may be required to setup the tunnel
used for encapsulating the Tenant System packets, it may be path for some tunneling technologies. This applies to both unicast
sufficient to have one or more local NVE addresses assigned and used and multicast tunneling.
in the source and destination fields of a tunneling encapsulation
header. Other information that is part of the
tunneling encapsulation header may also need to be configured. In
certain cases, local NVE configuration may be sufficient while in
other cases, some tunneling related information may need to
be shared among NVEs. The information that needs to be shared will
be technology dependent. For instance, potential information could
include tunnel identity, encapsulation type, and/or tunnel
resources. In certain cases, such as when using IP multicast in the
underlay, tunnels which interconnect NVEs may need to be
established. When tunneling information needs to be exchanged or
shared among NVEs, a control plane protocol may be required. For
instance, it may be necessary to provide active/standby status
information between NVEs, up/down status information,
pruning/grafting information for multicast tunnels, etc.
In addition, a control plane may be required to setup the tunnel 3.2. Multi-homing
path for some tunneling technologies. This applies to both unicast
and multicast tunneling.
3.2. Multi-homing Multi-homing techniques can be used to increase the reliability of
an NVO3 network. It is also important to ensure that physical
diversity in an NVO3 network is taken into account to avoid single
points of failure.
Multi-homing techniques can be used to increase the reliability of Multi-homing can be enabled in various nodes, from Tenant Systems
an NVO3 network. It is also important to ensure that physical into ToRs, ToRs into core switches/routers, and core nodes into DC
diversity in an NVO3 network is taken into account to avoid single GWs.
points of failure.
Multi-homing can be enabled in various nodes, from Tenant Systems The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP
into ToRs, ToRs into core switches/routers, and core nodes into DC routing as the means to re-route traffic upon failures techniques or
GWs. on MPLS re-rerouting capabilities.
The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP When a Tenant System is co-located with the NVE, the Tenant System
routing as the means to re-route traffic upon failures techniques or is effectively single homed to the NVE via a virtual port. When the
on MPLS re-rerouting capabilities. Tenant System and the NVE are separated, the Tenant System is
connected to the NVE via a logical Layer2 (L2) construct such as a
VLAN and it can be multi-homed to various NVEs. An NVE may provide
an L2 service to the end system or an l3 service. An NVE may be
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3
(L3). When an NVE provides an L2 service and is not co-located with
the end system, loop avoidance techniques must be used. Similarly,
when the NVE provides L3 service, similar dual-homing techniques can
be used. When the NVE provides a L3 service to the end system, it is
possible that no dynamic routing protocol is enabled between the end
system and the NVE. The end system can be multi-homed to
multiple physically-separated L3 NVEs over multiple interfaces. When
one of the links connected to an NVE fails, the other interfaces can
be used to reach the end system.
When a Tenant System is co-located with the NVE, the Tenant System External connectivity from a DC can be handled by two or more DC
is effectively single homed to the NVE via a virtual port. When the gateways. Each gateway provides access to external networks such as
Tenant System and the NVE are separated, the Tenant System is VPNs or the Internet. A gateway may be connected to two or more edge
connected to the NVE via a logical Layer2 (L2) construct such as a nodes in the external network for redundancy. When a connection to
VLAN and it can be multi-homed to various NVEs. An NVE may provide an upstream node is lost, the alternative connection is used and the
an L2 service to the end system or an l3 service. An NVE may be failed route withdrawn.
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3
(L3). When an NVE provides an L2 service and is not co-located with
the end system, loop avoidance techniques must be used. Similarly,
when the NVE provides L3 service, similar dual-homing techniques can
be used. When the NVE provides a L3 service to the end system, it is
possible that no dynamic routing protocol is enabled between the end
system and the NVE. The end system can be multi-homed to
multiple physically-separated L3 NVEs over multiple interfaces. When
one of the links connected to an NVE fails, the other interfaces can
be used to reach the end system.
External connectivity from a DC can be handled by two or more DC 3.3. VM Mobility
gateways. Each gateway provides access to external networks such as
VPNs or the Internet. A gateway may be connected to two or more edge
nodes in the external network for redundancy. When a connection to
an upstream node is lost, the alternative connection is used and the
failed route withdrawn.
3.3. VM Mobility In DC environments utilizing VM technologies, an important feature
is that VMs can move from one server to another server in the same
or different L2 physical domains (within or across DCs) in a
seamless manner.
In DC environments utilizing VM technologies, an important feature A VM can be moved from one server to another in stopped or suspended
is that VMs can move from one server to another server in the same state ("cold" VM mobility) or in running/active state ("hot" VM
or different L2 physical domains (within or across DCs) in a mobility). With "hot" mobility, VM L2 and L3 addresses need to be
seamless manner. preserved. With "cold" mobility, it may be desired to preserve at
least VM L3 addresses.
A VM can be moved from one server to another in stopped or suspended Solutions to maintain connectivity while a VM is moved are necessary
state ("cold" VM mobility) or in running/active state ("hot" VM in the case of "hot" mobility. This implies that connectivity among
mobility). With "hot" mobility, VM L2 and L3 addresses need to be VMs is preserved. For instance, for L2 VNs, ARP caches are updated
preserved. With "cold" mobility, it may be desired to preserve at accordingly.
least VM L3 addresses.
Solutions to maintain connectivity while a VM is moved are necessary Upon VM mobility, NVE policies that define connectivity among VMs
in the case of "hot" mobility. This implies that connectivity among must be maintained.
VMs is preserved. For instance, for L2 VNs, ARP caches are updated
accordingly.
Upon VM mobility, NVE policies that define connectivity among VMs During VM mobility, it is expected that the path to the VM's default
must be maintained. gateway assures adequate QoS to VM applications, i.e. QoS that
matches the expected service level agreement for these applications.
During VM mobility, it is expected that the path to the VM's default 4. Key aspects of overlay networks
gateway assures adequate QoS to VM applications, i.e. QoS that
matches the expected service level agreement for these applications.
4. Key aspects of overlay networks The intent of this section is to highlight specific issues that
proposed overlay solutions need to address.
The intent of this section is to highlight specific issues that 4.1. Pros & Cons
proposed overlay solutions need to address.
4.1. Pros & Cons An overlay network is a layer of virtual network topology on top of
the physical network.
An overlay network is a layer of virtual network topology on top of Overlay networks offer the following key advantages:
the physical network.
Overlay networks offer the following key advantages: - Unicast tunneling state management and association of Tenant
Systems reachability are handled at the edge of the network (at
the NVE). Intermediate transport nodes are unaware of such
state. Note that when multicast is enabled in the underlay
network to build multicast trees for tenant VNs, there would be
more state related to tenants in the underlay core network.
- Unicast tunneling state management and association of Tenant - Tunneling is used to aggregate traffic and hide tenant
Systems reachability are handled at the edge of the network (at addresses from the underlay network, and hence offer the
the NVE). Intermediate transport nodes are unaware of such advantage of minimizing the amount of forwarding state required
state. Note that when multicast is enabled in the underlay within the underlay network
network to build multicast trees for tenant VNs, there would be
more state related to tenants in the underlay core network.
- Tunneling is used to aggregate traffic and hide tenant - Decoupling of the overlay addresses (MAC and IP) used by VMs
addresses from the underlay network, and hence offer the from the underlay network for tenant separation and separation
advantage of minimizing the amount of forwarding state required of the tenant address spaces from the underlay address space.
within the underlay network
- Decoupling of the overlay addresses (MAC and IP) used by VMs - Support of a large number of virtual network identifiers
from the underlay network for tenant separation and separation
of the tenant address spaces from the underlay address space.
- Support of a large number of virtual network identifiers Overlay networks also create several challenges:
Overlay networks also create several challenges: - Overlay networks have typically no control of underlay networks
and lack underlay network information (e.g. underlay
utilization):
- Overlay networks have typically no control of underlay networks - Overlay networks and/or their associated management entities
and lack underlay network information (e.g. underlay typically probe the network to measure link or path
utilization): properties, such as available bandwidth or packet loss rate.
It is difficult to accurately evaluate network properties. It
might be preferable for the underlay network to expose usage
and performance information.
- Miscommunication or lack of coordination between overlay and
underlay networks can lead to an inefficient usage of network
resources.
- When multiple overlays co-exist on top of a common underlay
network, the lack of coordination between overlays can lead
to performance issues and/or resource usage inefficiencies.
- Overlay networks and/or their associated management entities - Traffic carried over an overlay might fail to traverse
typically probe the network to measure link or path firewalls and NAT devices.
properties, such as available bandwidth or packet loss rate.
It is difficult to accurately evaluate network properties. It
might be preferable for the underlay network to expose usage
and performance information.
- Miscommunication or lack of coordination between overlay and
underlay networks can lead to an inefficient usage of network
resources.
- When multiple overlays co-exist on top of a common underlay
network, the lack of coordination between overlays can lead
to performance issues and/or resource usage inefficiencies.
- Traffic carried over an overlay might fail to traverse - Multicast service scalability: Multicast support may be
firewalls and NAT devices. required in the underlay network to address tenant flood
containment or efficient multicast handling. The underlay may
also be required to maintain multicast state on a per-tenant
basis, or even on a per-individual multicast flow of a given
tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
- Multicast service scalability: Multicast support may be 4.2. Overlay issues to consider
required in the underlay network to address tenant flood
containment or efficient multicast handling. The underlay may
also be required to maintain multicast state on a per-tenant
basis, or even on a per-individual multicast flow of a given
tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
4.2. Overlay issues to consider 4.2.1. Data plane vs Control plane driven
4.2.1. Data plane vs Control plane driven In the case of an L2 NVE, it is possible to dynamically learn MAC
addresses against VAPs. It is also possible that such addresses be
known and controlled via management or a control protocol for both
L2 NVEs and L3 NVEs. Dynamic data plane learning implies that
flooding of unknown destinations be supported and hence implies that
broadcast and/or multicast be supported or that ingress replication
be used as described in section 4.2.3. Multicasting in the underlay
network for dynamic learning may lead to significant scalability
limitations. Specific forwarding rules must be enforced to prevent
loops from happening. This can be achieved using a spanning tree, a
shortest path tree, or a split-horizon mesh.
In the case of an L2 NVE, it is possible to dynamically learn MAC It should be noted that the amount of state to be distributed is
addresses against VAPs. It is also possible that such addresses be dependent upon network topology and the number of virtual machines.
known and controlled via management or a control protocol for both Different forms of caching can also be utilized to minimize state
L2 NVEs and L3 NVEs. Dynamic data plane learning implies that distribution between the various elements. The control plane should
flooding of unknown destinations be supported and hence implies that not require an NVE to maintain the locations of all the Tenant
broadcast and/or multicast be supported or that ingress replication Systems whose VNs are not present on the NVE. The use of a control
be used as described in section 4.2.3. Multicasting in the underlay plane does not imply that the data plane on NVEs has to maintain all
network for dynamic learning may lead to significant scalability the forwarding state in the control plane.
limitations. Specific forwarding rules must be enforced to prevent
loops from happening. This can be achieved using a spanning tree, a
shortest path tree, or a split-horizon mesh.
It should be noted that the amount of state to be distributed is 4.2.2. Coordination between data plane and control plane
dependent upon network topology and the number of virtual machines.
Different forms of caching can also be utilized to minimize state
distribution between the various elements. The control plane should
not require an NVE to maintain the locations of all the Tenant
Systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane.
4.2.2. Coordination between data plane and control plane For an L2 NVE, the NVE needs to be able to determine MAC addresses
of the Tenant Systems connected via a VAP. This can be achieved via
dataplane learning or a control plane. For an L3 NVE, the NVE needs
to be able to determine IP addresses of the Tenant Systems connected
via a VAP.
For an L2 NVE, the NVE needs to be able to determine MAC addresses In both cases, coordination with the NVE control protocol is needed
of the Tenant Systems connected via a VAP. This can be achieved via such that when the NVE determines that the set of addresses behind a
dataplane learning or a control plane. For an L3 NVE, the NVE needs VAP has changed, it triggers the NVE control plane to distribute
to be able to determine IP addresses of the Tenant Systems connected this information to its peers.
via a VAP.
In both cases, coordination with the NVE control protocol is needed 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic
such that when the NVE determines that the set of addresses behind a
VAP has changed, it triggers the NVE control plane to distribute
this information to its peers.
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic There are several options to support packet replication needed for
broadcast, unknown unicast and multicast. Typical methods include:
There are several options to support packet replication needed for - Ingress replication
broadcast, unknown unicast and multicast. Typical methods include: - Use of underlay multicast trees
- Ingress replication There is a bandwidth vs state trade-off between the two approaches.
- Use of underlay multicast trees Depending upon the degree of replication required (i.e. the number
of hosts per group) and the amount of multicast state to maintain,
trading bandwidth for state should be considered.
There is a bandwidth vs state trade-off between the two approaches. When the number of hosts per group is large, the use of underlay
Depending upon the degree of replication required (i.e. the number multicast trees may be more appropriate. When the number of hosts is
of hosts per group) and the amount of multicast state to maintain, small (e.g. 2-3) and/or the amount of multicast traffic is small,
trading bandwidth for state should be considered. ingress replication may not be an issue.
When the number of hosts per group is large, the use of underlay Depending upon the size of the data center network and hence the
multicast trees may be more appropriate. When the number of hosts is number of (S,G) entries, and also the duration of multicast flows,
small (e.g. 2-3) and/or the amount of multicast traffic is small, the use of underlay multicast trees can be a challenge.
ingress replication may not be an issue.
Depending upon the size of the data center network and hence the When flows are well known, it is possible to pre-provision such
number of (S,G) entries, and also the duration of multicast flows, multicast trees. However, it is often difficult to predict
the use of underlay multicast trees can be a challenge. application flows ahead of time, and hence programming of (S,G)
entries for short-lived flows could be impractical.
When flows are well known, it is possible to pre-provision such A possible trade-off is to use in the underlay shared multicast
multicast trees. However, it is often difficult to predict trees as opposed to dedicated multicast trees.
application flows ahead of time, and hence programming of (S,G)
entries for short-lived flows could be impractical.
A possible trade-off is to use in the underlay shared multicast 4.2.4. Path MTU
trees as opposed to dedicated multicast trees.
4.2.4. Path MTU When using overlay tunneling, an outer header is added to the
original frame. This can cause the MTU of the path to the egress
tunnel endpoint to be exceeded.
When using overlay tunneling, an outer header is added to the It is usually not desirable to rely on IP fragmentation for
original frame. This can cause the MTU of the path to the egress performance reasons. Ideally, the interface MTU as seen by a Tenant
tunnel endpoint to be exceeded. System is adjusted such that no fragmentation is needed.
It is usually not desirable to rely on IP fragmentation for It is possible for the MTU to be configured manually or to be
performance reasons. Ideally, the interface MTU as seen by a Tenant discovered dynamically. Various Path MTU discovery techniques exist
System is adjusted such that no fragmentation is needed. in order to determine the proper MTU size to use:
It is possible for the MTU to be configured manually or to be - Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981]
discovered dynamically. Various Path MTU discovery techniques exist
in order to determine the proper MTU size to use:
- Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981] - Tenant Systems rely on ICMP messages to discover the MTU of the
end-to-end path to its destination. This method is not always
possible, such as when traversing middle boxes (e.g. firewalls)
which disable ICMP for security reasons
- Tenant Systems rely on ICMP messages to discover the MTU of the - Extended MTU Path Discovery techniques such as defined in
end-to-end path to its destination. This method is not always [RFC4821]
possible, such as when traversing middle boxes (e.g. firewalls)
which disable ICMP for security reasons
- Extended MTU Path Discovery techniques such as defined in - Tenant Systems send probe packets of different sizes, and rely
[RFC4821] on confirmation of receipt or lack thereof from receivers to
allow a sender to discover the MTU of the end-to-end paths.
- Tenant Systems send probe packets of different sizes, and rely While it could also be possible to rely on the NVE to perform
on confirmation of receipt or lack thereof from receivers to segmentation and reassembly operations without relying on the Tenant
allow a sender to discover the MTU of the end-to-end paths. Systems to know about the end-to-end MTU, this would lead to
undesired performance and congestion issues as well as significantly
increase the complexity of hardware NVEs required for buffering and
reassembly logic.
While it could also be possible to rely on the NVE to perform Preferably, the underlay network should be designed in such a way
segmentation and reassembly operations without relying on the Tenant that the MTU can accommodate the extra tunneling and possibly
Systems to know about the end-to-end MTU, this would lead to additional NVO3 header encapsulation overhead.
undesired performance and congestion issues as well as significantly
increase the complexity of hardware NVEs required for buffering and
reassembly logic.
Preferably, the underlay network should be designed in such a way 4.2.5. NVE location trade-offs
that the MTU can accommodate the extra tunneling and possibly
additional NVO3 header encapsulation overhead.
4.2.5. NVE location trade-offs In the case of DC traffic, traffic originated from a VM is native
Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements.
In the case of DC traffic, traffic originated from a VM is native There are several criteria to consider when deciding where the NVE
Ethernet traffic. This traffic can be switched by a local virtual function should happen:
switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements.
There are several criteria to consider when deciding where the NVE - Processing and memory requirements
function should happen:
- Processing and memory requirements - Datapath (e.g. lookups, filtering, encapsulation/decapsulation)
- Datapath (e.g. lookups, filtering, encapsulation/decapsulation) - Control plane processing (e.g. routing, signaling, OAM) and
where specific control plane functions should be enabled
- Control plane processing (e.g. routing, signaling, OAM) and - FIB/RIB size
where specific control plane functions should be enabled
- FIB/RIB size - Multicast support
- Multicast support - Routing/signaling protocols
- Routing/signaling protocols - Packet replication capability
- Packet replication capability - Multicast FIB
- Multicast FIB - Fragmentation support
- QoS support (e.g. marking, policing, queuing)
- Fragmentation support - Resiliency
- QoS support (e.g. marking, policing, queuing)
- Resiliency 4.2.6. Interaction between network overlays and underlays
4.2.6. Interaction between network overlays and underlays When multiple overlays co-exist on top of a common underlay network,
resources (e.g., bandwidth) should be provisioned to ensure that
traffic from overlays can be accommodated and QoS objectives can be
met. Overlays can have partially overlapping paths (nodes and
links).
When multiple overlays co-exist on top of a common underlay network, Each overlay is selfish by nature. It sends traffic so as to
resources (e.g., bandwidth) should be provisioned to ensure that optimize its own performance without considering the impact on other
traffic from overlays can be accommodated and QoS objectives can be overlays, unless the underlay paths are traffic engineered on a per
met. Overlays can have partially overlapping paths (nodes and overlay basis to avoid congestion of underlay resources.
links).
Each overlay is selfish by nature. It sends traffic so as to Better visibility between overlays and underlays, or generally
optimize its own performance without considering the impact on other coordination in placing overlay demand on an underlay network, may
overlays, unless the underlay paths are traffic engineered on a per be achieved by providing mechanisms to exchange performance and
overlay basis to avoid congestion of underlay resources. liveliness information between the underlay and overlay(s) or the
use of such information by a coordination system. Such information
may include:
Better visibility between overlays and underlays, or generally - Performance metrics (throughput, delay, loss, jitter) such as
coordination in placing overlay demand on an underlay network, may defined in [RFC3148], [RFC2679], [RFC2680], and [RFC3393].
be achieved by providing mechanisms to exchange performance and
liveliness information between the underlay and overlay(s) or the
use of such information by a coordination system. Such information
may include:
- Performance metrics (throughput, delay, loss, jitter) - Cost metrics
- Cost metrics 5. Security Considerations
such as defined in [RFC2330]. There are three points-of-view when considering security for NVO3.
First, the service offered by a service provider via NVO3 technology
to a tenant must meet the mutually agreed security requirements.
Second, a network implementing NVO3 must be able to trust the
virtual network identity associated with packets received from a
tenant. Third, an NVO3 network must consider the security associated
with running as an overlay across the underlaying network.
5. Security Considerations To meet a tenant's security requirements, the NVO3 service must
deliver packets from the tenant to the indicated destination(s) in
the overlay network and external networks. The NVO3 service
provides data confidentiality through data separation. The use of
both VNIs and tunneling of tenant traffic by NVEs ensures that NVO3
data is kept in a separate context and thus separated from other
tenant traffic. The infrastructure supporting an NVO3 service (e.g.
Since NVEs and NVAs play a central role in NVO3, it is critical that management systems, NVEs, NVAs, and intermediate underlay networks)
a secure access to NVEs and NVAs be ensured such that no should be limited to authorized access so that data integrity can be
unauthorized access is possible. expected. If a tenant requires that its data be confidential, then
the tenant system may choose to encrypt its data before transmission
into the NVO3 service.
As discussed in section 3.1.5.2. , Tenant Systems identification is An NVO3 service must be able to verify the VNI received on a packet
based upon state that is often provided by management systems (e.g. from the tenant. To ensure this, not only tenant data but also NVO3
a VM orchestration system in a virtualized environment). Secure control data must be secured (e.g. control traffic between NVAs and
access to such management systems must also be ensured. NVEs, between NVAs and between NVEs). Since NVEs and NVAs play a
central role in NVO3, it is critical that a secure access to NVEs
and NVAs be ensured such that no unauthorized access is possible. As
discussed in section 3.1.5.2. , Tenant Systems identification is
based upon state that is often provided by management systems (e.g.
a VM orchestration system in a virtualized environment). Secure
access to such management systems must also be ensured. When an NVE
receives data from a Tenant System, the tenant identity needs to be
verified in order to guarantee that it is authorized to access the
corresponding VN. This can be achieved by identifying incoming
packets against specific VAPs in some cases. In other circumstances,
authentication may be necessary. Once this verification is done,
the packet is allowed into the NVO3 overlay and no integrity
protection is provided on the overlay packet encapsulation (e.g. the
VNI, destination VNE, etc.).
When an NVE receives data from a Tenant System, the tenant identity Since an NVO3 service can run across diverse underlay networks, when
needs to be verified in order to guarantee that it is authorized to the underlay network is not trusted to provide at least data
access the corresponding VN. This can be achieved by identifying integrity, data encryption is needed to assure correct packet
incoming packets against specific VAPs in some cases. In other delivery.
circumstances, authentication may be necessary.
Data integrity can be assured if authorized access to NVEs, NVAs, It is also desirable to restrict the types of information (e.g.
and intermediate underlay nodes is ensured. Otherwise, encryption topology information, such as discussed in Section 4.2.6) that can
must be used. be exchanged between an NVO3 service and underlaying networks based
upon their agreed security requirements.
NVO3 provides data confidentiality through data separation. The use 6. IANA Considerations
of both VNIs and tunneling of tenant traffic by NVEs ensures that
NVO3 data is kept in a separate context and thus separated from
other tenant traffic. When NVO3 data traverses untrusted networks,
data encryption may be needed.
Not only tenant data but also NVO3 control data must be secured IANA does not need to take any action for this draft.
(e.g. control traffic between NVAs and NVEs, between NVAs and
between NVEs).
It may also be desirable to restrict the types of information that 7. References
can be exchanged between overlays and underlays (e.g. topology
information).
6. IANA Considerations 7.1. Informative References
IANA does not need to take any action for this draft. [EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft-
ietf-l2vpn-evpn (work in progress)
7. References [NVOPS] Narten, T. et al, "Problem Statement : Overlays for
Network Virtualization", draft-ietf-nvo3-overlay-problem-
statement (work in progress)
7.1. Informative References [RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990
[NVOPS] Narten, T. et al, "Problem Statement : Overlays for [RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981,
Network Virtualization", draft-ietf-nvo3-overlay-problem- August 1996
statement (work in progress)
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private [RFC2679] Almes, G. et al, "A One-way Delay Metric for IPPM",
Networks (VPNs)", RFC 4364, February 2006. RFC2679, September 1999
[RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS) [RFC2680] Almes, G. et al, "A One-way Packet Loss Metric for IPPM",
Using BGP for auto-discovery and Signaling", RFC4761, RFC2680, September 1999
January 2007
[RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS) [RFC3148] Mathis, M. et al, "A Framework for Defining Empirical Bulk
Using Label Distribution Protocol (LDP) Signaling", Transfer Capacity Metrics", RFC3148, July 2001
RFC4762, January 2007
[EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft- [RFC3393] Demichelis, C. and Chimeto, P., "IP Packet Delay Variation
ietf-l2vpn-evpn (work in progress) Metric for IP Performance Metrics (IPPM)", RFC3393,
November 2002
[RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, February 2006.
[RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981, [RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS)
August 1996 Using BGP for auto-discovery and Signaling", RFC4761,
January 2007
[RFC4821] Mathis, M. et al, "Packetization Layer Path MTU [RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS)
Discovery", RFC4821, March 2007 Using Label Distribution Protocol (LDP) Signaling",
RFC4762, January 2007
[RFC6820] Narten, T. et al, "Address Resolution Problems in Large [RFC4821] Mathis, M. et al, "Packetization Layer Path MTU
Data Center Networks", RFC6820, January 2013 Discovery", RFC4821, March 2007
[RFC2330] Paxson, V. et al, "Framework for IP Performance Metrics", [RFC6820] Narten, T. et al, "Address Resolution Problems in Large
RFC2330, May 1998 Data Center Networks", RFC6820, January 2013
8. Acknowledgments 8. Acknowledgments
In addition to the authors the following people have contributed to In addition to the authors the following people have contributed to
this document: this document:
Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten, Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten,
Larry Kreeger, David Black. Larry Kreeger, David Black.
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
Authors' Addresses Authors' Addresses
Marc Lasserre Marc Lasserre
Alcatel-Lucent Alcatel-Lucent
Email: marc.lasserre@alcatel-lucent.com Email: marc.lasserre@alcatel-lucent.com
Florin Balus Florin Balus
Alcatel-Lucent Alcatel-Lucent
777 E. Middlefield Road 777 E. Middlefield Road
Mountain View, CA, USA 94043 Mountain View, CA, USA 94043
Email: florin.balus@alcatel-lucent.com Email: florin.balus@alcatel-lucent.com
Thomas Morin Thomas Morin
France Telecom Orange France Telecom Orange
Email: thomas.morin@orange.com Email: thomas.morin@orange.com
Nabil Bitar Nabil Bitar
Verizon Verizon
40 Sylvan Road 40 Sylvan Road
Waltham, MA 02145 Waltham, MA 02145
Email: nabil.bitar@verizon.com Email: nabil.bitar@verizon.com
Yakov Rekhter Yakov Rekhter
Juniper Juniper
Email: yakov@juniper.net Email: yakov@juniper.net
 End of changes. 223 change blocks. 
845 lines changed or deleted 871 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/