draft-ietf-nvo3-framework-09.txt   rfc7365.txt 
Internet Engineering Task Force Marc Lasserre
Internet Draft Florin Balus
Intended status: Informational Alcatel-Lucent
Expires: Jan 2015
Thomas Morin
France Telecom Orange
Nabil Bitar Internet Engineering Task Force (IETF) M. Lasserre
Verizon Request for Comments: 7365 F. Balus
Category: Informational Alcatel-Lucent
Yakov Rekhter ISSN: 2070-1721 T. Morin
Juniper Orange
N. Bitar
July 4, 2014 Verizon
Y. Rekhter
Juniper
October 2014
Framework for DC Network Virtualization Framework for Data Center (DC) Network Virtualization
draft-ietf-nvo3-framework-09.txt
Abstract Abstract
This document provides a framework for Data Center (DC) Network This document provides a framework for Data Center (DC) Network
Virtualization Overlays (NVO3) and it defines a reference model Virtualization over Layer 3 (NVO3) and defines a reference model
along with logical components required to design a solution. along with logical components required to design a solution.
Status of this Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This document is not an Internet Standards Track specification; it is
Task Force (IETF). Note that other groups may also distribute published for informational purposes.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six This document is a product of the Internet Engineering Task Force
months and may be updated, replaced, or obsoleted by other documents (IETF). It represents the consensus of the IETF community. It has
at any time. It is inappropriate to use Internet-Drafts as received public review and has been approved for publication by the
reference material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
This Internet-Draft will expire on Jan 4, 2015. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc7365.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with respect
respect to this document. Code Components extracted from this to this document. Code Components extracted from this document must
document must include Simplified BSD License text as described in include Simplified BSD License text as described in Section 4.e of
Section 4.e of the Trust Legal Provisions and are provided without the Trust Legal Provisions and are provided without warranty as
warranty as described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction..................................................3 1. Introduction ....................................................4
1.1. General terminology......................................3 1.1. General Terminology ........................................4
1.2. DC network architecture..................................6 1.2. DC Network Architecture ....................................7
2. Reference Models..............................................8 2. Reference Models ................................................8
2.1. Generic Reference Model..................................8 2.1. Generic Reference Model ....................................8
2.2. NVE Reference Model.....................................10 2.2. NVE Reference Model .......................................10
2.3. NVE Service Types.......................................10 2.3. NVE Service Types .........................................11
2.3.1. L2 NVE providing Ethernet LAN-like service.........11 2.3.1. L2 NVE Providing Ethernet LAN-Like Service .........11
2.3.2. L3 NVE providing IP/VRF-like service...............11 2.3.2. L3 NVE Providing IP/VRF-Like Service ...............11
2.4. Operational Management Considerations...................11 2.4. Operational Management Considerations .....................12
3. Functional components........................................12 3. Functional Components ..........................................12
3.1. Service Virtualization Components.......................12 3.1. Service Virtualization Components .........................12
3.1.1. Virtual Access Points (VAPs).......................12 3.1.1. Virtual Access Points (VAPs) .......................12
3.1.2. Virtual Network Instance (VNI).....................12 3.1.2. Virtual Network Instance (VNI) .....................12
3.1.3. Overlay Modules and VN Context.....................12 3.1.3. Overlay Modules and VN Context .....................14
3.1.4. Tunnel Overlays and Encapsulation options..........13 3.1.4. Tunnel Overlays and Encapsulation Options ..........14
3.1.5. Control Plane Components...........................14 3.1.5. Control-Plane Components ...........................14
3.1.5.1. Distributed vs Centralized Control Plane.........14 3.1.5.1. Distributed vs. Centralized
3.1.5.2. Auto-provisioning/Service discovery..............14 Control Plane .............................14
3.1.5.3. Address advertisement and tunnel mapping.........15 3.1.5.2. Auto-provisioning and Service Discovery ...15
3.1.5.4. Overlay Tunneling................................15 3.1.5.3. Address Advertisement and Tunnel Mapping ..15
3.2. Multi-homing............................................16 3.1.5.4. Overlay Tunneling .........................16
3.3. VM Mobility.............................................17 3.2. Multihoming ...............................................16
4. Key aspects of overlay networks..............................17 3.3. VM Mobility ...............................................17
4.1. Pros & Cons.............................................17 4. Key Aspects of Overlay Networks ................................17
4.2. Overlay issues to consider..............................19 4.1. Pros and Cons .............................................18
4.2.1. Data plane vs Control plane driven.................19 4.2. Overlay Issues to Consider ................................19
4.2.2. Coordination between data plane and control plane..19 4.2.1. Data Plane vs. Control Plane Driven ................19
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) 4.2.2. Coordination between Data Plane and Control Plane ..19
traffic...................................................19 4.2.3. Handling Broadcast, Unknown Unicast, and
4.2.4. Path MTU...........................................20 Multicast (BUM) Traffic ............................20
4.2.5. NVE location trade-offs............................21 4.2.4. Path MTU ...........................................20
4.2.6. Interaction between network overlays and underlays.22 4.2.5. NVE Location Trade-Offs ............................21
5. Security Considerations......................................22 4.2.6. Interaction between Network Overlays and
6. IANA Considerations..........................................23 Underlays ..........................................22
7. References...................................................23 5. Security Considerations ........................................22
7.1. Informative References..................................23 6. Informative References .........................................24
8. Acknowledgments..............................................25 Acknowledgments ...................................................26
Authors' Addresses ................................................26
1. Introduction 1. Introduction
This document provides a framework for Data Center (DC) Network This document provides a framework for Data Center (DC) Network
Virtualization over Layer3 (L3) tunnels. This framework is intended Virtualization over Layer 3 (NVO3) tunnels. This framework is
to aid in standardizing protocols and mechanisms to support large- intended to aid in standardizing protocols and mechanisms to support
scale network virtualization for data centers. large-scale network virtualization for data centers.
[NVOPS] defines the rationale for using overlay networks in order to [RFC7364] defines the rationale for using overlay networks in order
build large multi-tenant data center networks. Compute, storage and to build large multi-tenant data center networks. Compute, storage
network virtualization are often used in these large data centers to and network virtualization are often used in these large data centers
support a large number of communication domains and end systems. to support a large number of communication domains and end systems.
This document provides reference models and functional components of This document provides reference models and functional components of
data center overlay networks as well as a discussion of technical data center overlay networks as well as a discussion of technical
issues that have to be addressed. issues that have to be addressed.
1.1. General terminology 1.1. General Terminology
This document uses the following terminology: This document uses the following terminology:
NVO3 Network: An overlay network that provides a Layer2 (L2) or NVO3 Network: An overlay network that provides a Layer 2 (L2) or
Layer3 (L3) service to Tenant Systems over an L3 underlay network Layer 3 (L3) service to Tenant Systems over an L3 underlay network
using the architecture and protocols as defined by the NVO3 Working using the architecture and protocols as defined by the NVO3 Working
Group. Group.
Network Virtualization Edge (NVE). An NVE is the network entity that Network Virtualization Edge (NVE): An NVE is the network entity that
sits at the edge of an underlay network and implements L2 and/or L3 sits at the edge of an underlay network and implements L2 and/or L3
network virtualization functions. The network-facing side of the NVE network virtualization functions. The network-facing side of the NVE
uses the underlying L3 network to tunnel tenant frames to and from uses the underlying L3 network to tunnel tenant frames to and from
other NVEs. The tenant-facing side of the NVE sends and receives other NVEs. The tenant-facing side of the NVE sends and receives
Ethernet frames to and from individual Tenant Systems. An NVE could Ethernet frames to and from individual Tenant Systems. An NVE could
be implemented as part of a virtual switch within a hypervisor, a be implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split physical switch or router, or a Network Service Appliance, or it
across multiple devices. could be split across multiple devices.
Virtual Network (VN): A VN is a logical abstraction of a physical Virtual Network (VN): A VN is a logical abstraction of a physical
network that provides L2 or L3 network services to a set of Tenant network that provides L2 or L3 network services to a set of Tenant
Systems. A VN is also known as a Closed User Group (CUG). Systems. A VN is also known as a Closed User Group (CUG).
Virtual Network Instance (VNI): A specific instance of a VN from the Virtual Network Instance (VNI): A specific instance of a VN from the
perspective of an NVE. perspective of an NVE.
Virtual Network Context (VN Context) Identifier: Field in overlay Virtual Network Context (VN Context) Identifier: Field in an overlay
encapsulation header that identifies the specific VN the packet encapsulation header that identifies the specific VN the packet
belongs to. The egress NVE uses the VN Context identifier to deliver belongs to. The egress NVE uses the VN Context identifier to deliver
the packet to the correct Tenant System. The VN Context identifier the packet to the correct Tenant System. The VN Context identifier
can be a locally significant identifier or a globally unique can be a locally significant identifier or a globally unique
identifier. identifier.
Underlay or Underlying Network: The network that provides the Underlay or Underlying Network: The network that provides the
connectivity among NVEs and over which NVO3 packets are tunneled, connectivity among NVEs and that NVO3 packets are tunneled over,
where an NVO3 packet carries an NVO3 overlay header followed by a where an NVO3 packet carries an NVO3 overlay header followed by a
tenant packet. The Underlay Network does not need to be aware that tenant packet. The underlay network does not need to be aware that
it is carrying NVO3 packets. Addresses on the Underlay Network it is carrying NVO3 packets. Addresses on the underlay network
appear as "outer addresses" in encapsulated NVO3 packets. In appear as "outer addresses" in encapsulated NVO3 packets. In
general, the Underlay Network can use a completely different general, the underlay network can use a completely different protocol
protocol (and address family) from that of the overlay. In the case (and address family) from that of the overlay. In the case of NVO3,
of NVO3, the underlay network is IP. the underlay network is IP.
Data Center (DC): A physical complex housing physical servers, Data Center (DC): A physical complex housing physical servers,
network switches and routers, network service appliances and network switches and routers, network service appliances, and
networked storage. The purpose of a Data Center is to provide networked storage. The purpose of a data center is to provide
application, compute and/or storage services. One such service is application, compute, and/or storage services. One such service is
virtualized infrastructure data center services, also known as virtualized infrastructure data center services, also known as
Infrastructure as a Service. "Infrastructure as a Service".
Virtual Data Center (Virtual DC): A container for virtualized Virtual Data Center (Virtual DC): A container for virtualized
compute, storage and network services. A Virtual DC is associated compute, storage, and network services. A virtual DC is associated
with a single tenant, and can contain multiple VNs and Tenant with a single tenant and can contain multiple VNs and Tenant Systems
Systems connected to one or more of these VNs. connected to one or more of these VNs.
Virtual machine (VM): A software implementation of a physical Virtual Machine (VM): A software implementation of a physical machine
machine that runs programs as if they were executing on a physical, that runs programs as if they were executing on a physical, non-
non-virtualized machine. Applications (generally) do not know they virtualized machine. Applications (generally) do not know they are
are running on a VM as opposed to running on a "bare metal" host or running on a VM as opposed to running on a "bare metal" host or
server, though some systems provide a para-virtualization server, though some systems provide a para-virtualization environment
environment that allows an operating system or application to be that allows an operating system or application to be aware of the
aware of the presence of virtualization for optimization purposes. presence of virtualization for optimization purposes.
Hypervisor: Software running on a server that allows multiple VMs to Hypervisor: Software running on a server that allows multiple VMs to
run on the same physical server. The hypervisor manages and provides run on the same physical server. The hypervisor manages and provides
shared compute/memory/storage and network connectivity to the VMs shared computation, memory, and storage services and network
that it hosts. Hypervisors often embed a Virtual Switch (see below). connectivity to the VMs that it hosts. Hypervisors often embed a
virtual switch (see below).
Server: A physical end host machine that runs user applications. A Server: A physical end-host machine that runs user applications. A
standalone (or "bare metal") server runs a conventional operating standalone (or "bare metal") server runs a conventional operating
system hosting a single-tenant application. A virtualized server system hosting a single-tenant application. A virtualized server
runs a hypervisor supporting one or more VMs. runs a hypervisor supporting one or more VMs.
Virtual Switch (vSwitch): A function within a Hypervisor (typically Virtual Switch (vSwitch): A function within a hypervisor (typically
implemented in software) that provides similar forwarding services implemented in software) that provides similar forwarding services to
to a physical Ethernet switch. A vSwitch forwards Ethernet frames a physical Ethernet switch. A vSwitch forwards Ethernet frames
between VMs running on the same server, or between a VM and a between VMs running on the same server or between a VM and a physical
physical NIC card connecting the server to a physical Ethernet Network Interface Card (NIC) connecting the server to a physical
switch or router. A vSwitch also enforces network isolation between Ethernet switch or router. A vSwitch also enforces network isolation
VMs that by policy are not permitted to communicate with each other between VMs that by policy are not permitted to communicate with each
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is other (e.g., by honoring VLANs). A vSwitch may be bypassed when an
enabled on the host server. NVE is enabled on the host server.
Tenant: The customer using a virtual network and any associated Tenant: The customer using a virtual network and any associated
resources (e.g., compute, storage and network). A tenant could be resources (e.g., compute, storage, and network). A tenant could be
an enterprise, or a department/organization within an enterprise. an enterprise or a department/organization within an enterprise.
Tenant System: A physical or virtual system that can play the role Tenant System: A physical or virtual system that can play the role of
of a host, or a forwarding element such as a router, switch, a host or a forwarding element such as a router, switch, firewall,
firewall, etc. It belongs to a single tenant and connects to one or etc. It belongs to a single tenant and connects to one or more VNs
more VNs of that tenant. of that tenant.
Tenant Separation: Tenant Separation refers to isolating traffic of Tenant Separation: Refers to isolating traffic of different tenants
different tenants such that traffic from one tenant is not visible such that traffic from one tenant is not visible to or delivered to
to or delivered to another tenant, except when allowed by policy. another tenant, except when allowed by policy. Tenant separation
Tenant Separation also refers to address space separation, whereby also refers to address space separation, whereby different tenants
different tenants can use the same address space without conflict. can use the same address space without conflict.
Virtual Access Points (VAPs): A logical connection point on the NVE Virtual Access Points (VAPs): A logical connection point on the NVE
for connecting a Tenant System to a virtual network. Tenant Systems for connecting a Tenant System to a virtual network. Tenant Systems
connect to VNIs at an NVE through VAPs. VAPs can be physical ports connect to VNIs at an NVE through VAPs. VAPs can be physical ports
or virtual ports identified through logical interface identifiers or virtual ports identified through logical interface identifiers
(e.g., VLAN ID, internal vSwitch Interface ID connected to a VM). (e.g., VLAN ID or internal vSwitch Interface ID connected to a VM).
End Device: A physical device that connects directly to the DC End Device: A physical device that connects directly to the DC
Underlay Network. This is in contrast to a Tenant System, which underlay network. This is in contrast to a Tenant System, which
connects to a corresponding tenant VN. An End Device is administered connects to a corresponding tenant VN. An End Device is administered
by the DC operator rather than a tenant, and is part of the DC by the DC operator rather than a tenant and is part of the DC
infrastructure. An End Device may implement NVO3 technology in infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers, (e.g., server or server blade), storage systems (e.g., file servers
iSCSI storage systems), and network devices (e.g., firewall, load- and iSCSI storage systems), and network devices (e.g., firewall,
balancer, IPSec gateway). load-balancer, and IPsec gateway).
Network Virtualization Authority (NVA): Entity that provides Network Virtualization Authority (NVA): Entity that provides
reachability and forwarding information to NVEs. reachability and forwarding information to NVEs.
1.2. DC network architecture 1.2. DC Network Architecture
A generic architecture for Data Centers is depicted in Figure 1: A generic architecture for data centers is depicted in Figure 1:
,---------. ,---------.
,' `. ,' `.
( IP/MPLS WAN ) ( IP/MPLS WAN )
`. ,' `. ,'
`-+------+' `-+------+'
\ / \ /
+--------+ +--------+ +--------+ +--------+
| DC |+-+| DC | | DC |+-+| DC |
|gateway |+-+|gateway | |gateway |+-+|gateway |
skipping to change at page 6, line 44 skipping to change at page 7, line 39
| access | | access | | access | | access | | access | | access |
| switch | | switch | | switch | | switch | | switch | | switch |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
/ \ / \ / \ / \ / \ / \
__/_ \ / \ /_ _\__ __/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------' '--------' '--------' '--------' '--------'
: End : : End : : End : : End : : End : : End : : End : : End :
: Device : : Device : : Device : : Device : : Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------' '--------' '--------' '--------' '--------'
Figure 1 : A Generic Architecture for Data Centers Figure 1: A Generic Architecture for Data Centers
An example of multi-tier DC network architecture is presented in An example of multi-tier DC network architecture is presented in
Figure 1. It provides a view of physical components inside a DC. Figure 1. It provides a view of the physical components inside a DC.
A DC network is usually composed of intra-DC networks and network A DC network is usually composed of intra-DC networks and network
services, and inter-DC network and network connectivity services. services, and inter-DC network and network connectivity services.
DC networking elements can act as strict L2 switches and/or provide DC networking elements can act as strict L2 switches and/or provide
IP routing capabilities, including network service virtualization. IP routing capabilities, including network service virtualization.
In some DC architectures, some tier layers could provide L2 and/or In some DC architectures, some tier layers could provide L2 and/or L3
L3 services. In addition, some tier layers may be collapsed, and services. In addition, some tier layers may be collapsed, and
Internet connectivity, inter-DC connectivity and VPN support may be Internet connectivity, inter-DC connectivity, and VPN support may be
handled by a smaller number of nodes. Nevertheless, one can assume handled by a smaller number of nodes. Nevertheless, one can assume
that the network functional blocks in a DC fit in the architecture that the network functional blocks in a DC fit in the architecture
depicted in Figure 1. depicted in Figure 1.
The following components can be present in a DC: The following components can be present in a DC:
- Access switch: Hardware-based Ethernet switch aggregating all - Access switch: Hardware-based Ethernet switch aggregating all
Ethernet links from the End Devices in a rack representing the Ethernet links from the End Devices in a rack representing the
entry point in the physical DC network for the hosts. It may also entry point in the physical DC network for the hosts. It may also
provide routing functionality, virtual IP network connectivity, or provide routing functionality, virtual IP network connectivity, or
Layer2 tunneling over IP for instance. Access switches are usually Layer 2 tunneling over IP, for instance. Access switches are
multi-homed to aggregation switches in the Intra-DC network. A usually multihomed to aggregation switches in the Intra-DC
typical example of an access switch is a Top of Rack (ToR) switch. network. A typical example of an access switch is a Top-of-Rack
Other deployment scenarios may use an intermediate Blade Switch (ToR) switch. Other deployment scenarios may use an intermediate
before the ToR, or an EoR (End of Row) switch, to provide similar Blade Switch before the ToR, or an End-of-Row (EoR) switch, to
functions to a ToR. provide similar functions to a ToR.
- Intra-DC Network: Network composed of high capacity core nodes - Intra-DC Network: Network composed of high-capacity core nodes
(Ethernet switches/routers). Core nodes may provide virtual (Ethernet switches/routers). Core nodes may provide virtual
Ethernet bridging and/or IP routing services. Ethernet bridging and/or IP routing services.
- DC Gateway (DC GW): Gateway to the outside world providing DC - DC Gateway (DC GW): Gateway to the outside world providing DC
Interconnect and connectivity to Internet and VPN customers. In interconnect and connectivity to Internet and VPN customers. In
the current DC network model, this may be simply a router the current DC network model, this may be simply a router
connected to the Internet and/or an IP Virtual Private Network connected to the Internet and/or an IP VPN/L2VPN PE. Some network
(VPN)/L2VPN PE. Some network implementations may dedicate DC GWs implementations may dedicate DC GWs for different connectivity
for different connectivity types (e.g., a DC GW for Internet, and types (e.g., a DC GW for Internet and another for VPN).
another for VPN).
Note that End Devices may be single or multi-homed to access Note that End Devices may be single-homed or multihomed to access
switches. switches.
2. Reference Models 2. Reference Models
2.1. Generic Reference Model 2.1. Generic Reference Model
Figure 2 depicts a DC reference model for network virtualization Figure 2 depicts a DC reference model for network virtualization
overlay where NVEs provide a logical interconnect between Tenant overlays where NVEs provide a logical interconnect between Tenant
Systems that belong to a specific VN. Systems that belong to a specific VN.
+--------+ +--------+ +--------+ +--------+
| Tenant +--+ +----| Tenant | | Tenant +--+ +----| Tenant |
| System | | (') | System | | System | | (') | System |
+--------+ | ................. ( ) +--------+ +--------+ | ................. ( ) +--------+
| +---+ +---+ (_) | +---+ +---+ (_)
+--|NVE|---+ +---|NVE|-----+ +--|NVE|---+ +---|NVE|-----+
+---+ | | +---+ +---+ | | +---+
/ . +-----+ . / . +-----+ .
skipping to change at page 8, line 39 skipping to change at page 9, line 31
+---+ +---+
| |
| |
===================== =====================
| | | |
+--------+ +--------+ +--------+ +--------+
| Tenant | | Tenant | | Tenant | | Tenant |
| System | | System | | System | | System |
+--------+ +--------+ +--------+ +--------+
Figure 2 : Generic reference model for DC network virtualization Figure 2: Generic Reference Model for DC Network Virtualization
overlay Overlays
In order to obtain reachability information, NVEs may exchange In order to obtain reachability information, NVEs may exchange
information directly between themselves via a control plane information directly between themselves via a control-plane protocol.
protocol. In this case, a control plane module resides in every NVE. In this case, a control-plane module resides in every NVE.
It is also possible for NVEs to communicate with an external Network It is also possible for NVEs to communicate with an external Network
Virtualization Authority (NVA) to obtain reachability and forwarding Virtualization Authority (NVA) to obtain reachability and forwarding
information. In this case, a protocol is used between NVEs and information. In this case, a protocol is used between NVEs and
NVA(s) to exchange information. NVA(s) to exchange information.
It should be noted that NVAs may be organized in clusters for It should be noted that NVAs may be organized in clusters for
redundancy and scalability and can appear as one logically redundancy and scalability and can appear as one logically
centralized controller. In this case, inter-NVA communication is centralized controller. In this case, inter-NVA communication is
necessary to synchronize state among nodes within a cluster or share necessary to synchronize state among nodes within a cluster or share
information across clusters. The information exchanged between NVAs information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information of the same cluster could be different from the information exchanged
exchanged across clusters. across clusters.
A Tenant System can be attached to an NVE in several ways: A Tenant System can be attached to an NVE in several ways:
- locally, by being co-located in the same End Device - locally, by being co-located in the same End Device
- remotely, via a point-to-point connection or a switched network - remotely, via a point-to-point connection or a switched network
When an NVE is co-located with a Tenant System, the state of the When an NVE is co-located with a Tenant System, the state of the
Tenant System can be determined without protocol assistance. For Tenant System can be determined without protocol assistance. For
instance, the operational status of a VM can be communicated via a instance, the operational status of a VM can be communicated via a
local API. When an NVE is remotely connected to a Tenant System, the local API. When an NVE is remotely connected to a Tenant System, the
state of the Tenant System or NVE needs to be exchanged directly or state of the Tenant System or NVE needs to be exchanged directly or
via a management entity, using a control plane protocol or API, or via a management entity, using a control-plane protocol or API, or
directly via a dataplane protocol. directly via a data-plane protocol.
The functional components in Figure 2 do not necessarily map The functional components in Figure 2 do not necessarily map directly
directly to the physical components described in Figure 1. For to the physical components described in Figure 1. For example, an
example, an End Device can be a server blade with VMs and a virtual End Device can be a server blade with VMs and a virtual switch. A VM
switch. A VM can be a Tenant System and the NVE functions may be can be a Tenant System, and the NVE functions may be performed by the
performed by the host server. In this case, the Tenant System and host server. In this case, the Tenant System and NVE function are
NVE function are co-located. Another example is the case where the co-located. Another example is the case where the End Device is the
End Device is the Tenant System, and the NVE function can be Tenant System and the NVE function can be implemented by the
implemented by the connected ToR. In this case, the Tenant System connected ToR. In this case, the Tenant System and NVE function are
and NVE function are not co-located. not co-located.
Underlay nodes utilize L3 technologies to interconnect NVE nodes. Underlay nodes utilize L3 technologies to interconnect NVE nodes.
These nodes perform forwarding based on outer L3 header information, These nodes perform forwarding based on outer L3 header information,
and generally do not maintain per tenant-service state albeit some and generally do not maintain state for each tenant service, albeit
applications (e.g., multicast) may require control plane or some applications (e.g., multicast) may require control-plane or
forwarding plane information that pertain to a tenant, group of forwarding-plane information that pertains to a tenant, group of
tenants, tenant service or a set of services that belong to one or tenants, tenant service, or a set of services that belong to one or
more tenants. Mechanisms to control the amount of state maintained more tenants. Mechanisms to control the amount of state maintained
in the underlay may be needed. in the underlay may be needed.
2.2. NVE Reference Model 2.2. NVE Reference Model
Figure 3 depicts the NVE reference model. One or more VNIs can be Figure 3 depicts the NVE reference model. One or more VNIs can be
instantiated on an NVE. A Tenant System interfaces with a instantiated on an NVE. A Tenant System interfaces with a
corresponding VNI via a VAP. An overlay module provides tunneling corresponding VNI via a VAP. An overlay module provides tunneling
overlay functions (e.g., encapsulation and decapsulation of tenant overlay functions (e.g., encapsulation and decapsulation of tenant
traffic, tenant identification and mapping, etc.). traffic, tenant identification, and mapping, etc.).
+-------- L3 Network -------+ +-------- L3 Network -------+
| | | |
| Tunnel Overlay | | Tunnel Overlay |
+------------+---------+ +---------+------------+ +------------+---------+ +---------+------------+
| +----------+-------+ | | +---------+--------+ | | +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | | | | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ | | +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| | | |VN Context| | VN Context| |
| | | | | | | | | | | |
| +--------+-------+ | | +--------+-------+ | | +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| | | | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2 NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | | | | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+ +----+------------+----+ +----+-----------+-----+
| | | | | | | |
| | | | | | | |
Tenant Systems Tenant Systems Tenant Systems Tenant Systems
Figure 3 : Generic NVE reference model Figure 3: Generic NVE Reference Model
Note that some NVE functions (e.g., data plane and control plane Note that some NVE functions (e.g., data-plane and control-plane
functions) may reside in one device or may be implemented separately functions) may reside in one device or may be implemented separately
in different devices. in different devices.
2.3. NVE Service Types 2.3. NVE Service Types
An NVE provides different types of virtualized network services to An NVE provides different types of virtualized network services to
multiple tenants, i.e. an L2 service or an L3 service. Note that an multiple tenants, i.e., an L2 service or an L3 service. Note that an
NVE may be capable of providing both L2 and L3 services for a NVE may be capable of providing both L2 and L3 services for a tenant.
tenant. This section defines the service types and associated This section defines the service types and associated attributes.
attributes.
2.3.1. L2 NVE providing Ethernet LAN-like service 2.3.1. L2 NVE Providing Ethernet LAN-Like Service
An L2 NVE implements Ethernet LAN emulation, an Ethernet based An L2 NVE implements Ethernet LAN emulation, an Ethernet-based
multipoint service similar to an IETF VPLS [RFC4761][RFC4762] or multipoint service similar to an IETF Virtual Private LAN Service
EVPN [EVPN] service, where the Tenant Systems appear to be (VPLS) [RFC4761][RFC4762] or Ethernet VPN [EVPN] service, where the
interconnected by a LAN environment over an L3 overlay. As such, an Tenant Systems appear to be interconnected by a LAN environment over
L2 NVE provides per-tenant virtual switching instance (L2 VNI), and an L3 overlay. As such, an L2 NVE provides per-tenant virtual
L3 (IP/MPLS) tunneling encapsulation of tenant MAC frames across the switching instance (L2 VNI) and L3 (IP/MPLS) tunneling encapsulation
underlay. Note that the control plane for an L2 NVE could be of tenant Media Access Control (MAC) frames across the underlay.
implemented locally on the NVE or in a separate control entity. Note that the control plane for an L2 NVE could be implemented
locally on the NVE or in a separate control entity.
2.3.2. L3 NVE providing IP/VRF-like service 2.3.2. L3 NVE Providing IP/VRF-Like Service
An L3 NVE provides Virtualized IP forwarding service, similar to An L3 NVE provides virtualized IP forwarding service, similar to IETF
IETF IP VPN (e.g., BGP/MPLS IPVPN [RFC4364]) from a service IP VPN (e.g., BGP/MPLS IP VPN [RFC4364]) from a service definition
definition perspective. That is, an L3 NVE provides per-tenant perspective. That is, an L3 NVE provides per-tenant forwarding and
forwarding and routing instance (L3 VNI), and L3 (IP/MPLS) tunneling routing instance (L3 VNI) and L3 (IP/MPLS) tunneling encapsulation of
encapsulation of tenant IP packets across the underlay. Note that tenant IP packets across the underlay. Note that routing could be
routing could be performed locally on the NVE or in a separate performed locally on the NVE or in a separate control entity.
control entity.
2.4. Operational Management Considerations 2.4. Operational Management Considerations
NVO3 services are overlay services over an IP underlay. NVO3 services are overlay services over an IP underlay.
As far as the IP underlay is concerned, existing IP OAM facilities As far as the IP underlay is concerned, existing IP Operations,
are used. Administration, and Maintenance (OAM) facilities are used.
With regards to the NVO3 overlay, both L2 and L3 services can be With regard to the NVO3 overlay, both L2 and L3 services can be
offered. it is expected that existing fault and performance OAM offered. It is expected that existing fault and performance OAM
facilities will be used. Sections 4.1. and 4.2.6. below provide facilities will be used. Sections 4.1 and 4.2.6 provide further
further discussion of additional fault and performance management discussion of additional fault and performance management issues to
issues to consider. consider.
As far as configuration is concerned, the DC environment is driven As far as configuration is concerned, the DC environment is driven by
by the need to bring new services up rapidly and is typically very the need to bring new services up rapidly and is typically very
dynamic specifically in the context of virtualized services. It is dynamic, specifically in the context of virtualized services. It is
therefore critical to automate the configuration of NVO3 services. therefore critical to automate the configuration of NVO3 services.
3. Functional components 3. Functional Components
This section decomposes the Network Virtualization architecture into This section decomposes the network virtualization architecture into
functional components described in Figure 3 to make it easier to the functional components described in Figure 3 to make it easier to
discuss solution options for these components. discuss solution options for these components.
3.1. Service Virtualization Components 3.1. Service Virtualization Components
3.1.1. Virtual Access Points (VAPs) 3.1.1. Virtual Access Points (VAPs)
Tenant Systems are connected to VNIs through Virtual Access Points Tenant Systems are connected to VNIs through Virtual Access Points
(VAPs). (VAPs).
VAPs can be physical ports or virtual ports identified through VAPs can be physical ports or virtual ports identified through
logical interface identifiers (e.g., VLAN ID, internal vSwitch logical interface identifiers (e.g., VLAN ID and internal vSwitch
Interface ID connected to a VM). Interface ID connected to a VM).
3.1.2. Virtual Network Instance (VNI) 3.1.2. Virtual Network Instance (VNI)
A VNI is a specific VN instance on an NVE. Each VNI defines a A VNI is a specific VN instance on an NVE. Each VNI defines a
forwarding context that contains reachability information and forwarding context that contains reachability information and
policies. policies.
3.1.3. Overlay Modules and VN Context 3.1.3. Overlay Modules and VN Context
Mechanisms for identifying each tenant service are required to allow Mechanisms for identifying each tenant service are required to allow
the simultaneous overlay of multiple tenant services over the same the simultaneous overlay of multiple tenant services over the same
underlay L3 network topology. In the data plane, each NVE, upon underlay L3 network topology. In the data plane, each NVE, upon
sending a tenant packet, must be able to encode the VN Context for sending a tenant packet, must be able to encode the VN Context for
the destination NVE in addition to the L3 tunneling information the destination NVE in addition to the L3 tunneling information
(e.g., source IP address identifying the source NVE and the (e.g., source IP address identifying the source NVE and the
destination IP address identifying the destination NVE, or MPLS destination IP address identifying the destination NVE, or MPLS
label). This allows the destination NVE to identify the tenant label). This allows the destination NVE to identify the tenant
service instance and therefore appropriately process and forward the service instance and therefore appropriately process and forward the
tenant packet. tenant packet.
The Overlay module provides tunneling overlay functions: tunnel The overlay module provides tunneling overlay functions: tunnel
initiation/termination as in the case of stateful tunnels (see initiation/termination as in the case of stateful tunnels (see
Section 3.1.4), and/or simply encapsulation/decapsulation of frames Section 3.1.4) and/or encapsulation/decapsulation of frames from the
from VAPs/L3 underlay. VAPs/L3 underlay.
In a multi-tenant context, tunneling aggregates frames from/to In a multi-tenant context, tunneling aggregates frames from/to
different VNIs. Tenant identification and traffic demultiplexing are different VNIs. Tenant identification and traffic demultiplexing are
based on the VN Context identifier. based on the VN Context identifier.
The following approaches can be considered: The following approaches can be considered:
- VN Context identifier per Tenant: Globally unique (on a per-DC - VN Context identifier per Tenant: This is a globally unique (on a
administrative domain) VN identifier used to identify the per-DC administrative domain) VN identifier used to identify the
corresponding VNI. Examples of such identifiers in existing corresponding VNI. Examples of such identifiers in existing
technologies are IEEE VLAN IDs and ISID tags that identify virtual technologies are IEEE VLAN IDs and Service Instance IDs (I-SIDs)
L2 domains when using IEEE 802.1aq and IEEE 802.1ah, respectively. that identify virtual L2 domains when using IEEE 802.1Q and IEEE
Note that multiple VN identifiers can belong to a tenant. 802.1ah, respectively. Note that multiple VN identifiers can
belong to a tenant.
- One VN Context identifier per VNI: Each VNI value is automatically - One VN Context identifier per VNI: Each VNI value is automatically
generated by the egress NVE, or a control plane associated with generated by the egress NVE, or a control plane associated with
that NVE, and usually distributed by a control plane protocol to that NVE, and usually distributed by a control-plane protocol to
all the related NVEs. An example of this approach is the use of all the related NVEs. An example of this approach is the use of
per VRF MPLS labels in IP VPN [RFC4364]. The VNI value is per-VRF MPLS labels in IP VPN [RFC4364]. The VNI value is
therefore locally significant to the egress NVE. therefore locally significant to the egress NVE.
- One VN Context identifier per VAP: A value locally significant to - One VN Context identifier per VAP: A value locally significant to
an NVE is assigned and usually distributed by a control plane an NVE is assigned and usually distributed by a control-plane
protocol to identify a VAP. An example of this approach is the use protocol to identify a VAP. An example of this approach is the
of per CE-PE MPLS labels in IP VPN [RFC4364]. use of per-CE MPLS labels in IP VPN [RFC4364].
Note that when using one VN Context per VNI or per VAP, an Note that when using one VN Context per VNI or per VAP, an additional
additional global identifier (e.g., a VN identifier or name) may be global identifier (e.g., a VN identifier or name) may be used by the
used by the control plane to identify the Tenant context. control plane to identify the tenant context.
3.1.4. Tunnel Overlays and Encapsulation options 3.1.4. Tunnel Overlays and Encapsulation Options
Once the VN context identifier is added to the frame, an L3 Tunnel Once the VN Context identifier is added to the frame, an L3 tunnel
encapsulation is used to transport the frame to the destination NVE. encapsulation is used to transport the frame to the destination NVE.
Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS Different IP tunneling options (e.g., Generic Routing Encapsulation
tunneling can be used. Tunneling could be stateless or stateful. (GRE), the Layer 2 Tunneling Protocol (L2TP), and IPsec) and MPLS
tunneling can be used. Tunneling could be stateless or stateful.
Stateless tunneling simply entails the encapsulation of a tenant Stateless tunneling simply entails the encapsulation of a tenant
packet with another header necessary for forwarding the packet packet with another header necessary for forwarding the packet across
across the underlay (e.g., IP tunneling over an IP underlay). the underlay (e.g., IP tunneling over an IP underlay). Stateful
Stateful tunneling on the other hand entails maintaining tunneling tunneling, on the other hand, entails maintaining tunneling state at
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an the tunnel endpoints (i.e., NVEs). Tenant packets on an ingress NVE
ingress NVE can then be transmitted over such tunnels to a can then be transmitted over such tunnels to a destination (egress)
destination (egress) NVE by encapsulating the packets with a NVE by encapsulating the packets with a corresponding tunneling
corresponding tunneling header. The tunneling state at the endpoints header. The tunneling state at the endpoints may be configured or
may be configured or dynamically established. Solutions should dynamically established. Solutions should specify the tunneling
specify the tunneling technology used, whether it is stateful or technology used and whether it is stateful or stateless. In this
stateless. In this document, however, tunneling and tunneling document, however, tunneling and tunneling encapsulation are used
encapsulation are used interchangeably to simply mean the interchangeably to simply mean the encapsulation of a tenant packet
encapsulation of a tenant packet with a tunneling header necessary with a tunneling header necessary to carry the packet between an
to carry the packet between an ingress NVE and an egress NVE across ingress NVE and an egress NVE across the underlay. It should be
the underlay. It should be noted that stateful tunneling, especially noted that stateful tunneling, especially when configuration is
when configuration is involved, does impose management overhead and involved, does impose management overhead and scale constraints.
scale constraints. When confidentiality is required, the use of When confidentiality is required, the use of opportunistic security
opportunistic security [OPPSEC] can be used as a stateless tunneling [OPPSEC] can be used as a stateless tunneling solution.
solution.
3.1.5. Control Plane Components 3.1.5. Control-Plane Components
3.1.5.1. Distributed vs Centralized Control Plane 3.1.5.1. Distributed vs. Centralized Control Plane
A control/management plane entity can be centralized or distributed. Control- and management-plane entities can be centralized or
Both approaches have been used extensively in the past. The routing distributed. Both approaches have been used extensively in the past.
model of the Internet is a good example of a distributed approach. The routing model of the Internet is a good example of a distributed
Transport networks have usually used a centralized approach to approach. Transport networks have usually used a centralized
manage transport paths. approach to manage transport paths.
It is also possible to combine the two approaches, i.e., using a It is also possible to combine the two approaches, i.e., using a
hybrid model. A global view of network state can have many benefits hybrid model. A global view of network state can have many benefits,
but it does not preclude the use of distributed protocols within the but it does not preclude the use of distributed protocols within the
network. Centralized models provide a facility to maintain global network. Centralized models provide a facility to maintain global
state, and distribute that state to the network. When used in state and distribute that state to the network. When used in
combination with distributed protocols, greater network combination with distributed protocols, greater network efficiencies,
efficiencies, improved reliability and robustness can be achieved. improved reliability, and robustness can be achieved. Domain- and/or
Domain and/or deployment specific constraints define the balance deployment-specific constraints define the balance between
between centralized and distributed approaches. centralized and distributed approaches.
3.1.5.2. Auto-provisioning/Service discovery 3.1.5.2. Auto-provisioning and Service Discovery
NVEs must be able to identify the appropriate VNI for each Tenant NVEs must be able to identify the appropriate VNI for each Tenant
System. This is based on state information that is often provided by System. This is based on state information that is often provided by
external entities. For example, in an environment where a VM is a external entities. For example, in an environment where a VM is a
Tenant System, this information is provided by VM orchestration Tenant System, this information is provided by VM orchestration
systems, since these are the only entities that have visibility of systems, since these are the only entities that have visibility of
which VM belongs to which tenant. which VM belongs to which tenant.
A mechanism for communicating this information to the NVE is A mechanism for communicating this information to the NVE is
required. VAPs have to be created and mapped to the appropriate VNI. required. VAPs have to be created and mapped to the appropriate VNI.
Depending upon the implementation, this control interface can be Depending upon the implementation, this control interface can be
implemented using an auto-discovery protocol between Tenant Systems implemented using an auto-discovery protocol between Tenant Systems
and their local NVE or through management entities. In either case, and their local NVE or through management entities. In either case,
appropriate security and authentication mechanisms to verify that appropriate security and authentication mechanisms to verify that
Tenant System information is not spoofed or altered are required. Tenant System information is not spoofed or altered are required.
This is one critical aspect for providing integrity and tenant This is one critical aspect for providing integrity and tenant
isolation in the system. isolation in the system.
NVEs may learn reachability information to VNIs on other NVEs via a NVEs may learn reachability information for VNIs on other NVEs via a
control protocol that exchanges such information among NVEs, or via control protocol that exchanges such information among NVEs or via a
a management control entity. management-control entity.
3.1.5.3. Address advertisement and tunnel mapping 3.1.5.3. Address Advertisement and Tunnel Mapping
As traffic reaches an ingress NVE on a VAP, a lookup is performed to As traffic reaches an ingress NVE on a VAP, a lookup is performed to
determine which NVE or local VAP the packet needs to be sent to. If determine which NVE or local VAP the packet needs to be sent to. If
the packet is to be sent to another NVE, the packet is encapsulated the packet is to be sent to another NVE, the packet is encapsulated
with a tunnel header containing the destination information with a tunnel header containing the destination information
(destination IP address or MPLS label) of the egress NVE. (destination IP address or MPLS label) of the egress NVE.
Intermediate nodes (between the ingress and egress NVEs) switch or Intermediate nodes (between the ingress and egress NVEs) switch or
route traffic based upon the tunnel destination information. route traffic based upon the tunnel destination information.
A key step in the above process consists of identifying the A key step in the above process consists of identifying the
destination NVE the packet is to be tunneled to. NVEs are destination NVE the packet is to be tunneled to. NVEs are
responsible for maintaining a set of forwarding or mapping tables responsible for maintaining a set of forwarding or mapping tables
that hold the bindings between destination VM and egress NVE that hold the bindings between destination VM and egress NVE
addresses. Several ways of populating these tables are possible: addresses. Several ways of populating these tables are possible:
control plane driven, management plane driven, or data plane driven. control plane driven, management plane driven, or data plane driven.
When a control plane protocol is used to distribute address When a control-plane protocol is used to distribute address
reachability and tunneling information, the auto- reachability and tunneling information, the auto-provisioning and
provisioning/Service discovery could be accomplished by the same service discovery could be accomplished by the same protocol. In
protocol. In this scenario, the auto-provisioning/Service discovery this scenario, the auto-provisioning and service discovery could be
could be combined with (be inferred from) the address advertisement combined with (be inferred from) the address advertisement and
and associated tunnel mapping. Furthermore, a control plane protocol associated tunnel mapping. Furthermore, a control-plane protocol
that carries both MAC and IP addresses eliminates the need for ARP, that carries both MAC and IP addresses eliminates the need for the
and hence addresses one of the issues with explosive ARP handling as Address Resolution Protocol (ARP) and hence addresses one of the
discussed in [RFC6820]. issues with explosive ARP handling as discussed in [RFC6820].
3.1.5.4. Overlay Tunneling 3.1.5.4. Overlay Tunneling
For overlay tunneling, and dependent upon the tunneling technology For overlay tunneling, and dependent upon the tunneling technology
used for encapsulating the Tenant System packets, it may be used for encapsulating the Tenant System packets, it may be
sufficient to have one or more local NVE addresses assigned and used sufficient to have one or more local NVE addresses assigned and used
in the source and destination fields of a tunneling encapsulation in the source and destination fields of a tunneling encapsulation
header. Other information that is part of the header. Other information that is part of the tunneling
tunneling encapsulation header may also need to be configured. In encapsulation header may also need to be configured. In certain
certain cases, local NVE configuration may be sufficient while in cases, local NVE configuration may be sufficient while in other
other cases, some tunneling related information may need to cases, some tunneling-related information may need to be shared among
be shared among NVEs. The information that needs to be shared will NVEs. The information that needs to be shared will be technology
be technology dependent. For instance, potential information could dependent. For instance, potential information could include tunnel
include tunnel identity, encapsulation type, and/or tunnel identity, encapsulation type, and/or tunnel resources. In certain
resources. In certain cases, such as when using IP multicast in the cases, such as when using IP multicast in the underlay, tunnels that
underlay, tunnels which interconnect NVEs may need to be interconnect NVEs may need to be established. When tunneling
established. When tunneling information needs to be exchanged or information needs to be exchanged or shared among NVEs, a control-
shared among NVEs, a control plane protocol may be required. For plane protocol may be required. For instance, it may be necessary to
instance, it may be necessary to provide active/standby status provide active/standby status information between NVEs, up/down
information between NVEs, up/down status information, status information, pruning/grafting information for multicast
pruning/grafting information for multicast tunnels, etc. tunnels, etc.
In addition, a control plane may be required to setup the tunnel In addition, a control plane may be required to set up the tunnel
path for some tunneling technologies. This applies to both unicast path for some tunneling technologies. This applies to both unicast
and multicast tunneling. and multicast tunneling.
3.2. Multi-homing 3.2. Multihoming
Multi-homing techniques can be used to increase the reliability of Multihoming techniques can be used to increase the reliability of an
an NVO3 network. It is also important to ensure that physical NVO3 network. It is also important to ensure that the physical
diversity in an NVO3 network is taken into account to avoid single diversity in an NVO3 network is taken into account to avoid single
points of failure. points of failure.
Multi-homing can be enabled in various nodes, from Tenant Systems Multihoming can be enabled in various nodes, from Tenant Systems into
into ToRs, ToRs into core switches/routers, and core nodes into DC ToRs, ToRs into core switches/routers, and core nodes into DC GWs.
GWs.
The NVO3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP The NVO3 underlay nodes (i.e., from NVEs to DC GWs) rely on IP
routing as the means to re-route traffic upon failures techniques or routing techniques or MPLS re-rerouting capabilities as the means to
on MPLS re-rerouting capabilities. re-route traffic upon failures.
When a Tenant System is co-located with the NVE, the Tenant System When a Tenant System is co-located with the NVE, the Tenant System is
is effectively single homed to the NVE via a virtual port. When the effectively single-homed to the NVE via a virtual port. When the
Tenant System and the NVE are separated, the Tenant System is Tenant System and the NVE are separated, the Tenant System is
connected to the NVE via a logical Layer2 (L2) construct such as a connected to the NVE via a logical L2 construct such as a VLAN, and
VLAN and it can be multi-homed to various NVEs. An NVE may provide it can be multihomed to various NVEs. An NVE may provide an L2
an L2 service to the end system or an l3 service. An NVE may be service to the end system or an l3 service. An NVE may be multihomed
multi-homed to a next layer in the DC at Layer2 (L2) or Layer3 to a next layer in the DC at L2 or L3. When an NVE provides an L2
(L3). When an NVE provides an L2 service and is not co-located with service and is not co-located with the end system, loop-avoidance
the end system, loop avoidance techniques must be used. Similarly, techniques must be used. Similarly, when the NVE provides L3
when the NVE provides L3 service, similar dual-homing techniques can service, similar dual-homing techniques can be used. When the NVE
be used. When the NVE provides a L3 service to the end system, it is provides an L3 service to the end system, it is possible that no
possible that no dynamic routing protocol is enabled between the end dynamic routing protocol is enabled between the end system and the
system and the NVE. The end system can be multi-homed to NVE. The end system can be multihomed to multiple physically
multiple physically-separated L3 NVEs over multiple interfaces. When separated L3 NVEs over multiple interfaces. When one of the links
one of the links connected to an NVE fails, the other interfaces can connected to an NVE fails, the other interfaces can be used to reach
be used to reach the end system. the end system.
External connectivity from a DC can be handled by two or more DC External connectivity from a DC can be handled by two or more DC
gateways. Each gateway provides access to external networks such as gateways. Each gateway provides access to external networks such as
VPNs or the Internet. A gateway may be connected to two or more edge VPNs or the Internet. A gateway may be connected to two or more edge
nodes in the external network for redundancy. When a connection to nodes in the external network for redundancy. When a connection to
an upstream node is lost, the alternative connection is used and the an upstream node is lost, the alternative connection is used, and the
failed route withdrawn. failed route withdrawn.
3.3. VM Mobility 3.3. VM Mobility
In DC environments utilizing VM technologies, an important feature In DC environments utilizing VM technologies, an important feature is
is that VMs can move from one server to another server in the same that VMs can move from one server to another server in the same or
or different L2 physical domains (within or across DCs) in a different L2 physical domains (within or across DCs) in a seamless
seamless manner. manner.
A VM can be moved from one server to another in stopped or suspended A VM can be moved from one server to another in stopped or suspended
state ("cold" VM mobility) or in running/active state ("hot" VM state ("cold" VM mobility) or in running/active state ("hot" VM
mobility). With "hot" mobility, VM L2 and L3 addresses need to be mobility). With "hot" mobility, VM L2 and L3 addresses need to be
preserved. With "cold" mobility, it may be desired to preserve at preserved. With "cold" mobility, it may be desired to preserve at
least VM L3 addresses. least VM L3 addresses.
Solutions to maintain connectivity while a VM is moved are necessary Solutions to maintain connectivity while a VM is moved are necessary
in the case of "hot" mobility. This implies that connectivity among in the case of "hot" mobility. This implies that connectivity among
VMs is preserved. For instance, for L2 VNs, ARP caches are updated VMs is preserved. For instance, for L2 VNs, ARP caches are updated
accordingly. accordingly.
Upon VM mobility, NVE policies that define connectivity among VMs Upon VM mobility, NVE policies that define connectivity among VMs
must be maintained. must be maintained.
During VM mobility, it is expected that the path to the VM's default During VM mobility, it is expected that the path to the VM's default
gateway assures adequate QoS to VM applications, i.e. QoS that gateway assures adequate QoS to VM applications, i.e., QoS that
matches the expected service level agreement for these applications. matches the expected service-level agreement for these applications.
4. Key aspects of overlay networks 4. Key Aspects of Overlay Networks
The intent of this section is to highlight specific issues that The intent of this section is to highlight specific issues that
proposed overlay solutions need to address. proposed overlay solutions need to address.
4.1. Pros & Cons 4.1. Pros and Cons
An overlay network is a layer of virtual network topology on top of An overlay network is a layer of virtual network topology on top of
the physical network. the physical network.
Overlay networks offer the following key advantages: Overlay networks offer the following key advantages:
- Unicast tunneling state management and association of Tenant - Unicast tunneling state management and association of Tenant
Systems reachability are handled at the edge of the network (at Systems reachability are handled at the edge of the network (at
the NVE). Intermediate transport nodes are unaware of such the NVE). Intermediate transport nodes are unaware of such state.
state. Note that when multicast is enabled in the underlay Note that when multicast is enabled in the underlay network to
network to build multicast trees for tenant VNs, there would be build multicast trees for tenant VNs, there would be more state
more state related to tenants in the underlay core network. related to tenants in the underlay core network.
- Tunneling is used to aggregate traffic and hide tenant - Tunneling is used to aggregate traffic and hide tenant addresses
addresses from the underlay network, and hence offer the from the underlay network and hence offers the advantage of
advantage of minimizing the amount of forwarding state required minimizing the amount of forwarding state required within the
within the underlay network underlay network.
- Decoupling of the overlay addresses (MAC and IP) used by VMs - Decoupling of the overlay addresses (MAC and IP) used by VMs from
from the underlay network for tenant separation and separation the underlay network provides tenant separation and separation of
of the tenant address spaces from the underlay address space. the tenant address spaces from the underlay address space.
- Support of a large number of virtual network identifiers - Overlay networks support of a large number of virtual network
identifiers.
Overlay networks also create several challenges: Overlay networks also create several challenges:
- Overlay networks have typically no control of underlay networks - Overlay networks typically have no control of underlay networks
and lack underlay network information (e.g. underlay and lack underlay network information (e.g., underlay
utilization): utilization):
- Overlay networks and/or their associated management entities o Overlay networks and/or their associated management entities
typically probe the network to measure link or path typically probe the network to measure link or path properties,
properties, such as available bandwidth or packet loss rate. such as available bandwidth or packet loss rate. It is
It is difficult to accurately evaluate network properties. It difficult to accurately evaluate network properties. It might
might be preferable for the underlay network to expose usage be preferable for the underlay network to expose usage and
and performance information. performance information.
- Miscommunication or lack of coordination between overlay and
underlay networks can lead to an inefficient usage of network
resources.
- When multiple overlays co-exist on top of a common underlay
network, the lack of coordination between overlays can lead
to performance issues and/or resource usage inefficiencies.
- Traffic carried over an overlay might fail to traverse o Miscommunication or lack of coordination between overlay and
firewalls and NAT devices. underlay networks can lead to an inefficient usage of network
resources.
- Multicast service scalability: Multicast support may be o When multiple overlays co-exist on top of a common underlay
required in the underlay network to address tenant flood network, the lack of coordination between overlays can lead to
containment or efficient multicast handling. The underlay may performance issues and/or resource usage inefficiencies.
also be required to maintain multicast state on a per-tenant
basis, or even on a per-individual multicast flow of a given
tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
4.2. Overlay issues to consider - Traffic carried over an overlay might fail to traverse firewalls
and NAT devices.
4.2.1. Data plane vs Control plane driven - Multicast service scalability: Multicast support may be required
in the underlay network to address tenant flood containment or
efficient multicast handling. The underlay may also be required
to maintain multicast state on a per-tenant basis or even on a
per-individual multicast flow of a given tenant. Ingress
replication at the NVE eliminates that additional multicast state
in the underlay core, but depending on the multicast traffic
volume, it may cause inefficient use of bandwidth.
4.2. Overlay Issues to Consider
4.2.1. Data Plane vs. Control Plane Driven
In the case of an L2 NVE, it is possible to dynamically learn MAC In the case of an L2 NVE, it is possible to dynamically learn MAC
addresses against VAPs. It is also possible that such addresses be addresses against VAPs. It is also possible that such addresses be
known and controlled via management or a control protocol for both known and controlled via management or a control protocol for both L2
L2 NVEs and L3 NVEs. Dynamic data plane learning implies that NVEs and L3 NVEs. Dynamic data-plane learning implies that flooding
flooding of unknown destinations be supported and hence implies that of unknown destinations be supported and hence implies that broadcast
broadcast and/or multicast be supported or that ingress replication and/or multicast be supported or that ingress replication be used as
be used as described in section 4.2.3. Multicasting in the underlay described in Section 4.2.3. Multicasting in the underlay network for
network for dynamic learning may lead to significant scalability dynamic learning may lead to significant scalability limitations.
limitations. Specific forwarding rules must be enforced to prevent Specific forwarding rules must be enforced to prevent loops from
loops from happening. This can be achieved using a spanning tree, a happening. This can be achieved using a spanning tree, a shortest
shortest path tree, or a split-horizon mesh. path tree, or a split-horizon mesh.
It should be noted that the amount of state to be distributed is It should be noted that the amount of state to be distributed is
dependent upon network topology and the number of virtual machines. dependent upon network topology and the number of virtual machines.
Different forms of caching can also be utilized to minimize state Different forms of caching can also be utilized to minimize state
distribution between the various elements. The control plane should distribution between the various elements. The control plane should
not require an NVE to maintain the locations of all the Tenant not require an NVE to maintain the locations of all the Tenant
Systems whose VNs are not present on the NVE. The use of a control Systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane. the forwarding state in the control plane.
4.2.2. Coordination between data plane and control plane 4.2.2. Coordination between Data Plane and Control Plane
For an L2 NVE, the NVE needs to be able to determine MAC addresses For an L2 NVE, the NVE needs to be able to determine MAC addresses of
of the Tenant Systems connected via a VAP. This can be achieved via the Tenant Systems connected via a VAP. This can be achieved via
dataplane learning or a control plane. For an L3 NVE, the NVE needs data-plane learning or a control plane. For an L3 NVE, the NVE needs
to be able to determine IP addresses of the Tenant Systems connected to be able to determine the IP addresses of the Tenant Systems
via a VAP. connected via a VAP.
In both cases, coordination with the NVE control protocol is needed In both cases, coordination with the NVE control protocol is needed
such that when the NVE determines that the set of addresses behind a such that when the NVE determines that the set of addresses behind a
VAP has changed, it triggers the NVE control plane to distribute VAP has changed, it triggers the NVE control plane to distribute this
this information to its peers. information to its peers.
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic 4.2.3. Handling Broadcast, Unknown Unicast, and Multicast (BUM) Traffic
There are several options to support packet replication needed for There are several options to support packet replication needed for
broadcast, unknown unicast and multicast. Typical methods include: broadcast, unknown unicast, and multicast. Typical methods include:
- Ingress replication - Ingress replication
- Use of underlay multicast trees - Use of underlay multicast trees
There is a bandwidth vs state trade-off between the two approaches. There is a bandwidth vs. state trade-off between the two approaches.
Depending upon the degree of replication required (i.e. the number Depending upon the degree of replication required (i.e., the number
of hosts per group) and the amount of multicast state to maintain, of hosts per group) and the amount of multicast state to maintain,
trading bandwidth for state should be considered. trading bandwidth for state should be considered.
When the number of hosts per group is large, the use of underlay When the number of hosts per group is large, the use of underlay
multicast trees may be more appropriate. When the number of hosts is multicast trees may be more appropriate. When the number of hosts is
small (e.g. 2-3) and/or the amount of multicast traffic is small, small (e.g., 2-3) and/or the amount of multicast traffic is small,
ingress replication may not be an issue. ingress replication may not be an issue.
Depending upon the size of the data center network and hence the Depending upon the size of the data center network and hence the
number of (S,G) entries, and also the duration of multicast flows, number of (S,G) entries, and also the duration of multicast flows,
the use of underlay multicast trees can be a challenge. the use of underlay multicast trees can be a challenge.
When flows are well known, it is possible to pre-provision such When flows are well known, it is possible to pre-provision such
multicast trees. However, it is often difficult to predict multicast trees. However, it is often difficult to predict
application flows ahead of time, and hence programming of (S,G) application flows ahead of time; hence, programming of (S,G) entries
entries for short-lived flows could be impractical. for short-lived flows could be impractical.
A possible trade-off is to use in the underlay shared multicast A possible trade-off is to use in the underlay shared multicast trees
trees as opposed to dedicated multicast trees. as opposed to dedicated multicast trees.
4.2.4. Path MTU 4.2.4. Path MTU
When using overlay tunneling, an outer header is added to the When using overlay tunneling, an outer header is added to the
original frame. This can cause the MTU of the path to the egress original frame. This can cause the MTU of the path to the egress
tunnel endpoint to be exceeded. tunnel endpoint to be exceeded.
It is usually not desirable to rely on IP fragmentation for It is usually not desirable to rely on IP fragmentation for
performance reasons. Ideally, the interface MTU as seen by a Tenant performance reasons. Ideally, the interface MTU as seen by a Tenant
System is adjusted such that no fragmentation is needed. System is adjusted such that no fragmentation is needed.
It is possible for the MTU to be configured manually or to be It is possible for the MTU to be configured manually or to be
discovered dynamically. Various Path MTU discovery techniques exist discovered dynamically. Various Path MTU discovery techniques exist
in order to determine the proper MTU size to use: in order to determine the proper MTU size to use:
- Classical ICMP-based MTU Path Discovery [RFC1191] [RFC1981] - Classical ICMP-based Path MTU Discovery [RFC1191] [RFC1981]
- Tenant Systems rely on ICMP messages to discover the MTU of the Tenant Systems rely on ICMP messages to discover the MTU of the
end-to-end path to its destination. This method is not always end-to-end path to its destination. This method is not always
possible, such as when traversing middle boxes (e.g. firewalls) possible, such as when traversing middleboxes (e.g., firewalls)
which disable ICMP for security reasons that disable ICMP for security reasons.
- Extended MTU Path Discovery techniques such as defined in - Extended Path MTU Discovery techniques such as those defined in
[RFC4821] [RFC4821]
- Tenant Systems send probe packets of different sizes, and rely Tenant Systems send probe packets of different sizes and rely on
on confirmation of receipt or lack thereof from receivers to confirmation of receipt or lack thereof from receivers to allow a
allow a sender to discover the MTU of the end-to-end paths. sender to discover the MTU of the end-to-end paths.
While it could also be possible to rely on the NVE to perform While it could also be possible to rely on the NVE to perform
segmentation and reassembly operations without relying on the Tenant segmentation and reassembly operations without relying on the Tenant
Systems to know about the end-to-end MTU, this would lead to Systems to know about the end-to-end MTU, this would lead to
undesired performance and congestion issues as well as significantly undesired performance and congestion issues as well as significantly
increase the complexity of hardware NVEs required for buffering and increase the complexity of hardware NVEs required for buffering and
reassembly logic. reassembly logic.
Preferably, the underlay network should be designed in such a way Preferably, the underlay network should be designed in such a way
that the MTU can accommodate the extra tunneling and possibly that the MTU can accommodate the extra tunneling and possibly
additional NVO3 header encapsulation overhead. additional NVO3 header encapsulation overhead.
4.2.5. NVE location trade-offs 4.2.5. NVE Location Trade-Offs
In the case of DC traffic, traffic originated from a VM is native In the case of DC traffic, traffic originated from a VM is native
Ethernet traffic. This traffic can be switched by a local virtual Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can switch or ToR switch and then by a DC gateway. The NVE function can
be embedded within any of these elements. be embedded within any of these elements.
There are several criteria to consider when deciding where the NVE There are several criteria to consider when deciding where the NVE
function should happen: function should happen:
- Processing and memory requirements - Processing and memory requirements
- Datapath (e.g. lookups, filtering, encapsulation/decapsulation) o Datapath (e.g., lookups, filtering, and
encapsulation/decapsulation)
- Control plane processing (e.g. routing, signaling, OAM) and o Control-plane processing (e.g., routing, signaling, and OAM)
where specific control plane functions should be enabled and where specific control-plane functions should be enabled
- FIB/RIB size - FIB/RIB size
- Multicast support
- Multicast support o Routing/signaling protocols
- Routing/signaling protocols o Packet replication capability
- Packet replication capability o Multicast FIB
- Multicast FIB - Fragmentation support
- Fragmentation support - QoS support (e.g., marking, policing, and queuing)
- QoS support (e.g. marking, policing, queuing)
- Resiliency - Resiliency
4.2.6. Interaction between network overlays and underlays 4.2.6. Interaction between Network Overlays and Underlays
When multiple overlays co-exist on top of a common underlay network, When multiple overlays co-exist on top of a common underlay network,
resources (e.g., bandwidth) should be provisioned to ensure that resources (e.g., bandwidth) should be provisioned to ensure that
traffic from overlays can be accommodated and QoS objectives can be traffic from overlays can be accommodated and QoS objectives can be
met. Overlays can have partially overlapping paths (nodes and met. Overlays can have partially overlapping paths (nodes and
links). links).
Each overlay is selfish by nature. It sends traffic so as to Each overlay is selfish by nature. It sends traffic so as to
optimize its own performance without considering the impact on other optimize its own performance without considering the impact on other
overlays, unless the underlay paths are traffic engineered on a per overlays, unless the underlay paths are traffic engineered on a per-
overlay basis to avoid congestion of underlay resources. overlay basis to avoid congestion of underlay resources.
Better visibility between overlays and underlays, or generally Better visibility between overlays and underlays, or general
coordination in placing overlay demand on an underlay network, may coordination in placing overlay demands on an underlay network, may
be achieved by providing mechanisms to exchange performance and be achieved by providing mechanisms to exchange performance and
liveliness information between the underlay and overlay(s) or the liveliness information between the underlay and overlay(s) or by the
use of such information by a coordination system. Such information use of such information by a coordination system. Such information
may include: may include:
- Performance metrics (throughput, delay, loss, jitter) such as - Performance metrics (throughput, delay, loss, jitter) such as
defined in [RFC3148], [RFC2679], [RFC2680], and [RFC3393]. defined in [RFC3148], [RFC2679], [RFC2680], and [RFC3393].
- Cost metrics - Cost metrics
5. Security Considerations 5. Security Considerations
There are three points-of-view when considering security for NVO3. There are three points of view when considering security for NVO3.
First, the service offered by a service provider via NVO3 technology First, the service offered by a service provider via NVO3 technology
to a tenant must meet the mutually agreed security requirements. to a tenant must meet the mutually agreed security requirements.
Second, a network implementing NVO3 must be able to trust the Second, a network implementing NVO3 must be able to trust the virtual
virtual network identity associated with packets received from a network identity associated with packets received from a tenant.
tenant. Third, an NVO3 network must consider the security associated Third, an NVO3 network must consider the security associated with
with running as an overlay across the underlaying network. running as an overlay across the underlay network.
To meet a tenant's security requirements, the NVO3 service must To meet a tenant's security requirements, the NVO3 service must
deliver packets from the tenant to the indicated destination(s) in deliver packets from the tenant to the indicated destination(s) in
the overlay network and external networks. The NVO3 service the overlay network and external networks. The NVO3 service provides
provides data confidentiality through data separation. The use of data confidentiality through data separation. The use of both VNIs
both VNIs and tunneling of tenant traffic by NVEs ensures that NVO3 and tunneling of tenant traffic by NVEs ensures that NVO3 data is
data is kept in a separate context and thus separated from other kept in a separate context and thus separated from other tenant
tenant traffic. The infrastructure supporting an NVO3 service (e.g. traffic. The infrastructure supporting an NVO3 service (e.g.,
management systems, NVEs, NVAs, and intermediate underlay networks) management systems, NVEs, NVAs, and intermediate underlay networks)
should be limited to authorized access so that data integrity can be should be limited to authorized access so that data integrity can be
expected. If a tenant requires that its data be confidential, then expected. If a tenant requires that its data be confidential, then
the tenant system may choose to encrypt its data before transmission the Tenant System may choose to encrypt its data before transmission
into the NVO3 service. into the NVO3 service.
An NVO3 service must be able to verify the VNI received on a packet An NVO3 service must be able to verify the VNI received on a packet
from the tenant. To ensure this, not only tenant data but also NVO3 from the tenant. To ensure this, not only tenant data but also NVO3
control data must be secured (e.g. control traffic between NVAs and control data must be secured (e.g., control traffic between NVAs and
NVEs, between NVAs and between NVEs). Since NVEs and NVAs play a NVEs, between NVAs, and between NVEs). Since NVEs and NVAs play a
central role in NVO3, it is critical that a secure access to NVEs central role in NVO3, it is critical that secure access to NVEs and
and NVAs be ensured such that no unauthorized access is possible. As NVAs be ensured such that no unauthorized access is possible. As
discussed in section 3.1.5.2. , Tenant Systems identification is discussed in Section 3.1.5.2, identification of Tenant Systems is
based upon state that is often provided by management systems (e.g. based upon state that is often provided by management systems (e.g.,
a VM orchestration system in a virtualized environment). Secure a VM orchestration system in a virtualized environment). Secure
access to such management systems must also be ensured. When an NVE access to such management systems must also be ensured. When an NVE
receives data from a Tenant System, the tenant identity needs to be receives data from a Tenant System, the tenant identity needs to be
verified in order to guarantee that it is authorized to access the verified in order to guarantee that it is authorized to access the
corresponding VN. This can be achieved by identifying incoming corresponding VN. This can be achieved by identifying incoming
packets against specific VAPs in some cases. In other circumstances, packets against specific VAPs in some cases. In other circumstances,
authentication may be necessary. Once this verification is done, authentication may be necessary. Once this verification is done, the
the packet is allowed into the NVO3 overlay and no integrity packet is allowed into the NVO3 overlay, and no integrity protection
protection is provided on the overlay packet encapsulation (e.g. the is provided on the overlay packet encapsulation (e.g., the VNI,
VNI, destination VNE, etc.). destination NVE, etc.).
Since an NVO3 service can run across diverse underlay networks, when Since an NVO3 service can run across diverse underlay networks, when
the underlay network is not trusted to provide at least data the underlay network is not trusted to provide at least data
integrity, data encryption is needed to assure correct packet integrity, data encryption is needed to assure correct packet
delivery. delivery.
It is also desirable to restrict the types of information (e.g. It is also desirable to restrict the types of information (e.g.,
topology information, such as discussed in Section 4.2.6) that can topology information as discussed in Section 4.2.6) that can be
be exchanged between an NVO3 service and underlaying networks based exchanged between an NVO3 service and underlay networks based upon
upon their agreed security requirements. their agreed security requirements.
6. IANA Considerations
IANA does not need to take any action for this draft.
7. References
7.1. Informative References
[EVPN] Sajassi, A. et al, "BGP MPLS Based Ethernet VPN", draft-
ietf-l2vpn-evpn (work in progress)
[NVOPS] Narten, T. et al, "Problem Statement : Overlays for 6. Informative References
Network Virtualization", draft-ietf-nvo3-overlay-problem-
statement (work in progress)
[OPPSEC] Dukhovni, V. "Opportunistic Security: some protection most [EVPN] Sajassi, A., Aggarwal, R., Bitar, N., Isaac, A., and J.
of the time", draft-dukhovni-opportunistic-security (work Uttaro, "BGP MPLS Based Ethernet VPN", Work in Progress,
in progress) draft-ietf-l2vpn-evpn-10, October 2014.
[RFC1191] Mogul, J. "Path MTU Discovery", RFC1191, November 1990 [OPPSEC] Dukhovni, V. "Opportunistic Security: Some Protection Most
of the Time", Work in Progress, draft-dukhovni-
opportunistic-security-04, August 2014.
[RFC1981] McCann, J. et al, "Path MTU Discovery for IPv6", RFC1981, [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
August 1996 November 1990, <http://www.rfc-editor.org/info/rfc1191>.
[RFC2679] Almes, G. et al, "A One-way Delay Metric for IPPM", [RFC1981] McCann, J., Deering, S., and J. Mogul, "Path MTU Discovery
RFC2679, September 1999 for IP version 6", RFC 1981, August 1996,
<http://www.rfc-editor.org/info/rfc1981>.
[RFC2680] Almes, G. et al, "A One-way Packet Loss Metric for IPPM", [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
RFC2680, September 1999 Delay Metric for IPPM", RFC 2679, September 1999,
<http://www.rfc-editor.org/info/rfc2679>.
[RFC3148] Mathis, M. et al, "A Framework for Defining Empirical Bulk [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Transfer Capacity Metrics", RFC3148, July 2001 Packet Loss Metric for IPPM", RFC 2680, September 1999,
<http://www.rfc-editor.org/info/rfc2680>.
[RFC3393] Demichelis, C. and Chimeto, P., "IP Packet Delay Variation [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining
Metric for IP Performance Metrics (IPPM)", RFC3393, Empirical Bulk Transfer Capacity Metrics", RFC 3148, July
November 2002 2001, <http://www.rfc-editor.org/info/rfc3148>.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
Networks (VPNs)", RFC 4364, February 2006. Metric for IP Performance Metrics (IPPM)", RFC 3393,
November 2002, <http://www.rfc-editor.org/info/rfc3393>.
[RFC4761] Kompella, K. et al, "Virtual Private LAN Service (VPLS) [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Using BGP for auto-discovery and Signaling", RFC4761, Networks (VPNs)", RFC 4364, February 2006,
January 2007 <http://www.rfc-editor.org/info/rfc4364>.
[RFC4762] Lasserre, M. et al, "Virtual Private LAN Service (VPLS) [RFC4761] Kompella, K., Ed., and Y. Rekhter, Ed., "Virtual Private
Using Label Distribution Protocol (LDP) Signaling", LAN Service (VPLS) Using BGP for Auto-Discovery and
RFC4762, January 2007 Signaling", RFC 4761, January 2007,
<http://www.rfc-editor.org/info/rfc4761>.
[RFC4821] Mathis, M. et al, "Packetization Layer Path MTU [RFC4762] Lasserre, M., Ed., and V. Kompella, Ed., "Virtual Private
Discovery", RFC4821, March 2007 LAN Service (VPLS) Using Label Distribution Protocol (LDP)
Signaling", RFC 4762, January 2007,
<http://www.rfc-editor.org/info/rfc4762>.
[RFC6820] Narten, T. et al, "Address Resolution Problems in Large [RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU
Data Center Networks", RFC6820, January 2013 Discovery", RFC 4821, March 2007,
<http://www.rfc-editor.org/info/rfc4821>.
8. Acknowledgments [RFC6820] Narten, T., Karir, M., and I. Foo, "Address Resolution
Problems in Large Data Center Networks", RFC 6820, January
2013, <http://www.rfc-editor.org/info/rfc6820>.
In addition to the authors the following people have contributed to [RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
this document: Kreeger, L., and M. Napierala, "Problem Statement:
Overlays for Network Virtualization", RFC 7364, October
2014, <http://www.rfc-editor.org/info/rfc7364>.
Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten, Acknowledgments
Larry Kreeger, David Black.
This document was prepared using 2-Word-v2.0.template.dot. In addition to the authors, the following people contributed to this
document: Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas
Narten, Larry Kreeger, and David Black.
Authors' Addresses Authors' Addresses
Marc Lasserre Marc Lasserre
Alcatel-Lucent Alcatel-Lucent
Email: marc.lasserre@alcatel-lucent.com EMail: marc.lasserre@alcatel-lucent.com
Florin Balus Florin Balus
Alcatel-Lucent Alcatel-Lucent
777 E. Middlefield Road 777 E. Middlefield Road
Mountain View, CA, USA 94043 Mountain View, CA 94043
Email: florin.balus@alcatel-lucent.com United States
EMail: florin.balus@alcatel-lucent.com
Thomas Morin Thomas Morin
France Telecom Orange Orange
Email: thomas.morin@orange.com EMail: thomas.morin@orange.com
Nabil Bitar Nabil Bitar
Verizon Verizon
40 Sylvan Road 50 Sylvan Road
Waltham, MA 02145 Waltham, MA 02145
Email: nabil.bitar@verizon.com United States
EMail: nabil.n.bitar@verizon.com
Yakov Rekhter Yakov Rekhter
Juniper Juniper
Email: yakov@juniper.net EMail: yakov@juniper.net
 End of changes. 217 change blocks. 
592 lines changed or deleted 594 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/