draft-ietf-nvo3-framework-02.txt   draft-ietf-nvo3-framework-03.txt 
Internet Engineering Task Force Marc Lasserre Internet Engineering Task Force Marc Lasserre
Internet Draft Florin Balus Internet Draft Florin Balus
Intended status: Informational Alcatel-Lucent Intended status: Informational Alcatel-Lucent
Expires: August 2013 Expires: January 2014
Thomas Morin Thomas Morin
France Telecom Orange France Telecom Orange
Nabil Bitar Nabil Bitar
Verizon Verizon
Yakov Rekhter Yakov Rekhter
Juniper Juniper
February 4, 2013 July 4, 2013
Framework for DC Network Virtualization Framework for DC Network Virtualization
draft-ietf-nvo3-framework-02.txt draft-ietf-nvo3-framework-03.txt
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 4, 2013. This Internet-Draft will expire on January 4, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 25 skipping to change at page 2, line 25
large scale virtual data centers. This draft provides a framework large scale virtual data centers. This draft provides a framework
for Network Virtualization over L3 (NVO3) and is intended to help for Network Virtualization over L3 (NVO3) and is intended to help
plan a set of work items in order to provide a complete solution plan a set of work items in order to provide a complete solution
set. It defines a logical view of the main components with the set. It defines a logical view of the main components with the
intention of streamlining the terminology and focusing the solution intention of streamlining the terminology and focusing the solution
set. set.
Table of Contents Table of Contents
1. Introduction.................................................3 1. Introduction.................................................3
1.1. Conventions used in this document.......................4 1.1. Conventions used in this document.......................3
1.2. General terminology.....................................4 1.2. General terminology.....................................4
1.3. DC network architecture.................................6 1.3. DC network architecture.................................6
1.4. Tenant networking view..................................7
2. Reference Models.............................................8 2. Reference Models.............................................8
2.1. Generic Reference Model.................................8 2.1. Generic Reference Model.................................8
2.2. NVE Reference Model....................................10 2.2. NVE Reference Model....................................11
2.3. NVE Service Types......................................11 2.3. NVE Service Types......................................12
2.3.1. L2 NVE providing Ethernet LAN-like service........12 2.3.1. L2 NVE providing Ethernet LAN-like service........12
2.3.2. L3 NVE providing IP/VRF-like service..............12 2.3.2. L3 NVE providing IP/VRF-like service..............12
3. Functional components.......................................12 3. Functional components.......................................12
3.1. Service Virtualization Components......................12 3.1. Service Virtualization Components......................12
3.1.1. Virtual Access Points (VAPs)......................12 3.1.1. Virtual Access Points (VAPs)......................12
3.1.2. Virtual Network Instance (VNI)....................12 3.1.2. Virtual Network Instance (VNI)....................13
3.1.3. Overlay Modules and VN Context....................13 3.1.3. Overlay Modules and VN Context....................13
3.1.4. Tunnel Overlays and Encapsulation options.........14 3.1.4. Tunnel Overlays and Encapsulation options.........14
3.1.5. Control Plane Components..........................14 3.1.5. Control Plane Components..........................14
3.1.5.1. Distributed vs Centralized Control Plane........14 3.1.5.1. Distributed vs Centralized Control Plane........14
3.1.5.2. Auto-provisioning/Service discovery.............15 3.1.5.2. Auto-provisioning/Service discovery.............15
3.1.5.3. Address advertisement and tunnel mapping........15 3.1.5.3. Address advertisement and tunnel mapping........15
3.1.5.4. Overlay Tunneling...............................16 3.1.5.4. Overlay Tunneling...............................16
3.2. Multi-homing...........................................16 3.2. Multi-homing...........................................16
3.3. VM Mobility............................................17 3.3. VM Mobility............................................17
3.4. Service Overlay Topologies.............................18
4. Key aspects of overlay networks.............................18 4. Key aspects of overlay networks.............................18
4.1. Pros & Cons............................................18 4.1. Pros & Cons............................................18
4.2. Overlay issues to consider.............................20 4.2. Overlay issues to consider.............................19
4.2.1. Data plane vs Control plane driven................20 4.2.1. Data plane vs Control plane driven................19
4.2.2. Coordination between data plane and control plane.20 4.2.2. Coordination between data plane and control plane.20
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM)
traffic..................................................20 traffic..................................................20
4.2.4. Path MTU..........................................21 4.2.4. Path MTU..........................................21
4.2.5. NVE location trade-offs...........................22 4.2.5. NVE location trade-offs...........................22
4.2.6. Interaction between network overlays and underlays.23 4.2.6. Interaction between network overlays and underlays.22
5. Security Considerations.....................................23 5. Security Considerations.....................................23
6. IANA Considerations.........................................24 6. IANA Considerations.........................................23
7. References..................................................24 7. References..................................................23
7.1. Normative References...................................24 7.1. Normative References...................................23
7.2. Informative References.................................24 7.2. Informative References.................................24
8. Acknowledgments.............................................24 8. Acknowledgments.............................................24
1. Introduction 1. Introduction
This document provides a framework for Data Center Network This document provides a framework for Data Center Network
Virtualization over L3 tunnels. This framework is intended to aid in Virtualization over Layer3 (L3) tunnels. This framework is intended
standardizing protocols and mechanisms to support large scale to aid in standardizing protocols and mechanisms to support large-
network virtualization for data centers. scale network virtualization for data centers.
Several IETF drafts relate to the use of overlay networks for data [NVOPS] defines the rationale for using overlay networks in order to
centers. [NVOPS] defines the rationale for using overlay networks in build large multi-tenant data center networks. Compute, storage and
order to build large multi-tenant data center networks. Compute, network virtualization are often used in these large data centers to
storage and network virtualization are often used in these large support a large number of communication domains and end systems.
data centers to support a large number of communication domains and
end systems. [OVCPREQ] describes the requirements for a control
plane protocol required by overlay border nodes to exchange overlay
mappings.
This document provides reference models and functional components of This document provides reference models and functional components of
data center overlay networks as well as a discussion of technical data center overlay networks as well as a discussion of technical
issues that have to be addressed in the design of standards and issues that have to be addressed.
mechanisms for large-scale data centers.
1.1. Conventions used in this document 1.1. Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119 [RFC2119]. document are to be interpreted as described in RFC-2119 [RFC2119].
In this document, these words will appear with that interpretation In this document, these words will appear with that interpretation
only when in ALL CAPS. Lower case uses of these words are not to be only when in ALL CAPS. Lower case uses of these words are not to be
interpreted as carrying RFC-2119 significance. interpreted as carrying RFC-2119 significance.
1.2. General terminology 1.2. General terminology
This document uses the following terminology: This document uses the following terminology:
NVE: Network Virtualization Edge. It is a network entity that sits NVO3 Network: An overlay network that provides an Layer2 (L2) or
on the edge of the NVO3 network. It implements network Layer3 (L3) service to Tenant Systems over an L3 underlay network,
virtualization functions that allow for L2 and/or L3 tenant using the architecture and protocols as defined by the NVO3 Working
separation and for hiding tenant addressing information (MAC and IP Group.
addresses). An NVE could be implemented as part of a virtual switch
within a hypervisor, a physical switch or router, or a network
service appliance.
VN: Virtual Network. This is a virtual L2 or L3 domain that belongs Network Virtualization Edge (NVE). An NVE is the network entity that
to a tenant. sits at the edge of an underlay network and implements L2 and/or L3
network virtualization functions. The network-facing side of the NVE
uses the underlying L3 network to tunnel frames to and from other
NVEs. The tenant-facing side of the NVE sends and receives Ethernet
frames to and from individual Tenant Systems. An NVE could be
implemented as part of a virtual switch within a hypervisor, a
physical switch or router, a Network Service Appliance, or be split
across multiple devices.
VNI: Virtual Network Instance. This is one instance of a virtual Virtual Network (VN): A VN is a logical abstraction of a physical
overlay network. It refers to the state maintained for a given VN on network that provides L2 or L3 network services to a set of Tenant
a given NVE. Two Virtual Networks are isolated from one another and Systems. A VN is also known as a Closed User Group (CUG).
may use overlapping addresses.
Virtual Network Context or VN Context: Field that is part of the Virtual Network Instance (VNI): A specific instance of a VN.
overlay encapsulation header which allows the encapsulated frame to
be delivered to the appropriate virtual network endpoint by the
egress NVE. The egress NVE uses this field to determine the
appropriate virtual network context in which to process the packet.
This field MAY be an explicit, unique (to the administrative domain)
virtual network identifier (VNID) or MAY express the necessary
context information in other ways (e.g., a locally significant
identifier).
VNID: Virtual Network Identifier. In the case where the VN context Virtual Network Context (VN Context) Identifier: Field in overlay
identifier has global significance, this is the ID value that is encapsulation header that identifies the specific VN the packet
carried in each data packet in the overlay encapsulation that belongs to. The egress NVE uses the VN Context identifier to deliver
identifies the Virtual Network the packet belongs to. the packet to the correct Tenant System. The VN Context identifier
can be a locally significant identifier or a globally unique
identifier.
Underlay or Underlying Network: This is the network that provides Underlay or Underlying Network: The network that provides the
the connectivity between NVEs. The Underlying Network can be connectivity among NVEs and over which NVO3 packets are tunneled,
completely unaware of the overlay packets. Addresses within the where an NVO3 packet carries an NVO3 overlay header followed by a
Underlying Network are also referred to as "outer addresses" because tenant packet. The Underlay Network does not need to be aware that
they exist in the outer encapsulation. The Underlying Network can it is carrying NVO3 packets. Addresses on the Underlay Network
use a completely different protocol (and address family) from that appear as "outer addresses" in encapsulated NVO3 packets. In
of the overlay. general, the Underlay Network can use a completely different
protocol (and address family) from that of the overlay. In the case
of NVO3, the underlay network is typically IP.
Data Center (DC): A physical complex housing physical servers, Data Center (DC): A physical complex housing physical servers,
network switches and routers, network service sppliances and network switches and routers, network service appliances and
networked storage. The purpose of a Data Center is to provide networked storage. The purpose of a Data Center is to provide
application, compute and/or storage services. One such service is application, compute and/or storage services. One such service is
virtualized infrastructure data center services, also known as virtualized infrastructure data center services, also known as
Infrastructure as a Service. Infrastructure as a Service.
Virtual Data Center or Virtual DC: A container for virtualized Virtual Data Center (Virtual DC): A container for virtualized
compute, storage and network services. Managed by a single tenant, a compute, storage and network services. A Virtual DC is associated
Virtual DC can contain multiple VNs and multiple Tenant Systems that with a single tenant, and can contain multiple VNs and Tenant
are connected to one or more of these VNs. Systems connected to one or more of these VNs.
VM: Virtual Machine. Several Virtual Machines can share the Virtual machine (VM): A software implementation of a physical
resources of a single physical computer server using the services of machine that runs programs as if they were executing on a physical,
a Hypervisor (see below definition). non-virtualized machine. Applications (generally) do not know they
are running on a VM as opposed to running on a "bare metal" host or
server, though some systems provide a para-virtualization
environment that allows an operating system or application to be
aware of the presences of virtualization for optimization purposes.
Hypervisor: Server virtualization software running on a physical Hypervisor: Software running on a server that allows multiple VMs to
compute server that hosts Virtual Machines. The hypervisor provides run on the same physical server. The hypervisor manages and provides
shared compute/memory/storage and network connectivity to the VMs shared compute/memory/storage and network connectivity to the VMs
that it hosts. Hypervisors often embed a Virtual Switch (see below). that it hosts. Hypervisors often embed a Virtual Switch (see below).
Virtual Switch: A function within a Hypervisor (typically Server: A physical end host machine that runs user applications. A
implemented in software) that provides similar services to a standalone (or "bare metal") server runs a conventional operating
physical Ethernet switch. It switches Ethernet frames between VMs system hosting a single-tenant application. A virtualized server
virtual NICs within the same physical server, or between a VM and a runs a hypervisor supporting one or more VMs.
Virtual Switch (vSwitch): A function within a Hypervisor (typically
implemented in software) that provides similar forwarding services
to a physical Ethernet switch. A vSwitch forwards Ethernet frames
between VMs running on the same server, or between a VM and a
physical NIC card connecting the server to a physical Ethernet physical NIC card connecting the server to a physical Ethernet
switch or router. It also enforces network isolation between VMs switch or router. A vSwitch also enforces network isolation between
that should not communicate with each other. VMs that by policy are not permitted to communicate with each other
(e.g., by honoring VLANs). A vSwitch may be bypassed when an NVE is
enabled on the host server.
Tenant: In a DC, a tenant refers to a customer that could be an Tenant: The customer using a virtual network and any associated
organization within an enterprise, or an enterprise with a set of DC resources (e.g., compute, storage and network). A tenant could be
compute, storage and network resources associated with it. an enterprise, or a department/organization within an enterprise.
Tenant System: A physical or virtual system that can play the role Tenant System: A physical or virtual system that can play the role
of a host, or a forwarding element such as a router, switch, of a host, or a forwarding element such as a router, switch,
firewall, etc. It belongs to a single tenant and connects to one or firewall, etc. It belongs to a single tenant and connects to one or
more VNs of that tenant. more VNs of that tenant.
End device: A physical system to which networking service is Tenant Separation: Tenant Separation refers to isolating traffic of
provided. Examples include hosts (e.g. server or server blade), different tenants such that traffic from one tenant is not visible
storage systems (e.g., file servers, iSCSI storage systems), and to or delivered to another tenant, except when allowed by policy.
network devices (e.g., firewall, load-balancer, IPSec gateway). An Tenant Separation also refers to address space separation, whereby
end device may include internal networking functionality that different tenants can use the same address space without conflict.
interconnects the device's components (e.g. virtual switches that
interconnect VMs running on the same server). NVE functionality may
be implemented as part of that internal networking.
ELAN: MEF ELAN, multipoint to multipoint Ethernet service Virtual Access Points (VAPs): Tenant Systems are connected to VNIs
through VAPs. VAPs can be physical ports or virtual ports identified
through logical interface identifiers (e.g., VLAN ID, internal
vSwitch Interface ID connected to a VM).
EVPN: Ethernet VPN as defined in [EVPN] End Device: A physical device that connects directly to the DC
Underlay Network. This is in contrast to a tenant system, which
connects to a corresponding tenant VN. An End Device is administered
by the DC operator rather than a tenant, and is part of the DC
infrastructure. An End Device may implement NVO3 technology in
support of NVO3 functions. Examples of an End Device include hosts
(e.g., server or server blade), storage systems (e.g., file servers,
iSCSI storage systems), and network devices (e.g., firewall, load-
balancer, IPSec gateway).
Network Virtualization Authority (NVA): Entity that provides
reachability and forwarding information to NVEs. An NVA is also
known as a controller.
1.3. DC network architecture 1.3. DC network architecture
A generic architecture for Data Centers is depicted in Figure 1: A generic architecture for Data Centers is depicted in Figure 1:
,---------. ,---------.
,' `. ,' `.
( IP/MPLS WAN ) ( IP/MPLS WAN )
`. ,' `. ,'
`-+------+' `-+------+'
+--+--+ +-+---+ \ /
|DC GW|+-+|DC GW| +--------+ +--------+
+-+---+ +-----+ | DC |+-+| DC |
|gateway |+-+|gateway |
+--------+ +--------+
| / | /
.--. .--. .--. .--.
( ' '.--. ( ' '.--.
.-.' Intra-DC ' .-.' Intra-DC '
( network ) ( network )
( .'-' ( .'-'
'--'._.'. )\ \ '--'._.'. )\ \
/ / '--' \ \ / / '--' \ \
/ / | | \ \ / / | | \ \
+---+--+ +-`.+--+ +--+----+ +--------+ +--------+ +--------+
| ToR | | ToR | | ToR | | access | | access | | access |
+-+--`.+ +-+-`.-+ +-+--+--+ | switch | | switch | | switch |
/ \ / \ / \ +--------+ +--------+ +--------+
__/_ \ / \ /_ _\__ / \ / \ / \
__/_ \ / \ /_ _\__
'--------' '--------' '--------' '--------' '--------' '--------' '--------' '--------'
: End : : End : : End : : End : : End : : End : : End : : End :
: Device : : Device : : Device : : Device : : Device : : Device : : Device : : Device :
'--------' '--------' '--------' '--------' '--------' '--------' '--------' '--------'
Figure 1 : A Generic Architecture for Data Centers Figure 1 : A Generic Architecture for Data Centers
An example of multi-tier DC network architecture is presented in An example of multi-tier DC network architecture is presented in
this figure. It provides a view of physical components inside a DC. Figure 1. It provides a view of physical components inside a DC.
A cloud network is composed of intra-Data Center (DC) networks and A DC network is usually composed of intra-DC networks and network
network services, and inter-DC network and network connectivity services, and inter-DC network and network connectivity services.
services. Depending upon the scale, DC distribution, operations Depending upon the scale, DC distribution, operations model, Capital
model, Capex and Opex aspects, DC networking elements can act as expenditure (Capex) and Operational expenditure (Opex) aspects, DC
strict L2 switches and/or provide IP routing capabilities, including networking elements can act as strict L2 switches and/or provide IP
service virtualization. routing capabilities, including network service virtualization.
In some DC architectures, it is possible that some tier layers In some DC architectures, some tier layers could provide L2 and/or
provide L2 and/or L3 services, are collapsed, and that Internet L3 services. In addition, some tier layers may be collapsed, and
connectivity, inter-DC connectivity and VPN support are handled by a Internet connectivity, inter-DC connectivity and VPN support may be
smaller number of nodes. Nevertheless, one can assume that the handled by a smaller number of nodes. Nevertheless, one can assume
functional blocks fit in the architecture above. that the network functional blocks in a DC fit in the architecture
depicted in Figure 1.
The following components can be present in a DC: The following components can be present in a DC:
o Top of Rack (ToR): Hardware-based Ethernet switch aggregating o Access switch: Hardware-based Ethernet switch aggregating all
all Ethernet links from the End Devices in a rack representing Ethernet links from the End Devices in a rack representing the
the entry point in the physical DC network for the hosts. ToRs entry point in the physical DC network for the hosts. It may
may also provide routing functionality, virtual IP network also provide routing functionality, virtual IP network
connectivity, or Layer2 tunneling over IP for instance. ToRs connectivity, or Layer2 tunneling over IP for instance. Access
are usually multi-homed to switches in the Intra-DC network. swicthes are usually multi-homed to aggregation switches in the
Other deployment scenarios may use an intermediate Blade Switch Intra-DC network. A typical example of an access switch is a
before the ToR or an EoR (End of Row) switch to provide similar Top of Rack (ToR) switch. Other deployment scenarios may use an
function as a ToR. intermediate Blade Switch before the ToR, or an EoR (End of
Row) switch, to provide similar function as a ToR.
o Intra-DC Network: High capacity network composed of core
switches aggregating multiple ToRs. Core switches are usually
Ethernet switches but can also support routing capabilities.
o DC GW: Gateway to the outside world providing DC Interconnect
and connectivity to Internet and VPN customers. In the current
DC network model, this may be simply a Router connected to the
Internet and/or an IPVPN/L2VPN PE. Some network implementations
may dedicate DC GWs for different connectivity types (e.g., a
DC GW for Internet, and another for VPN).
Note that End Devices may be single or multi-homed to ToRs.
1.4. Tenant networking view
The DC network architecture is used to provide L2 and/or L3 service
connectivity to each tenant. An example is depicted in Figure 2:
+----- L3 Infrastructure ----+
| |
,--+--. ,--+--.
.....( Rtr1 )...... ( Rtr2 )
| `-----' | `-----'
| Tenant1 |LAN12 Tenant1|
|LAN11 ....|........ |LAN13
.............. | | ..............
| | | | | |
,-. ,-. ,-. ,-. ,-. ,-.
(VM )....(VM ) (VM )... (VM ) (VM )....(VM )
`-' `-' `-' `-' `-' `-'
Figure 2 : Logical Service connectivity for a single tenant o Intra-DC Network: Network composed of high capacity core nodes
(Ethernet switches/routers). Core nodes may provide virtual
Ethernet bridging and/or IP routing services.
In this example, one or more L3 contexts and one or more LANs (e.g., o DC Gateway (DC GW): Gateway to the outside world providing DC
one per application type) are assigned for DC tenant1. Interconnect and connectivity to Internet and VPN customers. In
the current DC network model, this may be simply a router
connected to the Internet and/or an IP Virtual Private Network
(VPN)/L2VPN PE. Some network implementations may dedicate DC
GWs for different connectivity types (e.g., a DC GW for
Internet, and another for VPN).
For a multi-tenant DC, a virtualized version of this type of service Note that End Devices may be single or multi-homed to access
connectivity needs to be provided for each tenant by the Network switches.
Virtualization solution.
2. Reference Models 2. Reference Models
2.1. Generic Reference Model 2.1. Generic Reference Model
The following diagram shows a DC reference model for network Figure 2 depicts a DC reference model for network virtualization
virtualization using L3 (IP/MPLS) overlays where NVEs provide a using L3 (IP/MPLS) overlays where NVEs provide a logical
logical interconnect between Tenant Systems that belong to a interconnect between Tenant Systems that belong to a specific VN.
specific tenant network.
+--------+ +--------+ +--------+ +--------+
| Tenant +--+ +----| Tenant | | Tenant +--+ +----| Tenant |
| System | | (') | System | | System | | (') | System |
+--------+ | ................... ( ) +--------+ +--------+ | ................. ( ) +--------+
| +-+--+ +--+-+ (_) | +---+ +---+ (_)
| | NV | | NV | | +--|NVE|---+ +---|NVE|-----+
+--|Edge| |Edge|---+ +---+ | | +---+
+-+--+ +--+-+ / . +-----+ .
/ . . / . +--| NVA | .
/ . L3 Overlay +--+-++--------+ / . | +-----+ .
+--------+ / . Network | NV || Tenant | | . | .
| Tenant +--+ . |Edge|| System | | . | L3 Overlay +--+--++--------+
| System | . +----+ +--+-++--------+ +--------+ | . | Network | NVE || Tenant |
+--------+ .....| NV |........ | Tenant +--+ . | | || System |
|Edge| | System | . \ +---+ +--+--++--------+
+----+ +--------+ .....|NVE|.........
+---+
| |
| |
===================== =====================
| | | |
+--------+ +--------+ +--------+ +--------+
| Tenant | | Tenant | | Tenant | | Tenant |
| System | | System | | System | | System |
+--------+ +--------+ +--------+ +--------+
Figure 3 : Generic reference model for DC network virtualization Figure 2 : Generic reference model for DC network virtualization
over a Layer3 infrastructure over a Layer3 (IP) infrastructure
A Tenant System can be attached to a Network Virtualization Edge In order to get reachability information, NVEs may exchange
(NVE) node in several ways: information directly between themselves via a protocol. In this
case, a control plane module resides in every NVE. This is how
routing control plane modules are implemented in routers for
instance.
- locally, by being co-located in the same device It is also possible for NVEs to communicate with an external Network
Virtualization Authority (NVA) to obtain reachability and forwarding
information. In this case, a protocol is used between NVEs and
NVA(s) to exchange information. OpenFlow [OF] is one example of such
a protocol.
- remotely, via a point-to-point connection or a switched network It should be noted that NVAs may be organized in clusters for
(e.g., Ethernet) redundancy and scalability and can appear as one logically
centralized controller. In this case, inter-NVA communication is
necessary to synchronize state among nodes within a cluster or share
information across clusters. The information exchanged between NVAs
of the same cluster could be different from the information
exchanged across clusters.
When an NVE is local, the state of Tenant Systems can be provided A Tenant System can be attached to an NVE in several ways:
without protocol assistance. For instance, the operational status of
a VM can be communicated via a local API. When an NVE is remote, the
state of Tenant Systems needs to be exchanged via a data or control
plane protocol, or via a management entity.
The functional components in Figure 3 do not necessarily map - locally, by being co-located in the same End Device
directly with the physical components described in Figure 1.
For example, an End Device can be a server blade with VMs and - remotely, via a point-to-point connection or a switched network
virtual switch, i.e. the VM is the Tenant System and the NVE
functions may be performed by the virtual switch and/or the
hypervisor. In this case, the Tenant System and NVE function are co-
located.
Another example is the case where an End Device can be a traditional When an NVE is co-located with a Tenant System, the state of the
physical server (no VMs, no virtual switch), i.e. the server is the Tenant System can be provided without protocol assistance. For
Tenant System and the NVE function may be performed by the ToR. instance, the operational status of a VM can be communicated via a
Other End Devices in this category are physical network appliances local API. When an NVE is remotely connected to a tenant system, the
or storage systems. state of the Tenant System or NVE needs to be exchanged directly or
via a management entity, using a control plane protocol or API, or
directly via a dataplane protocol.
The functional components in Figure 2 do not necessarily map
directly to the physical components described in Figure 1. For
example, an End Device can be a server blade with VMs and a virtual
switch. A VM can be a Tenant System and the NVE functions may be
performed by the host server. In this case, the Tenant System and
NVE function are co-located.
Another example is the case where the End Device is the tenant
System, and the NVE function can be implemented by the connected
ToR.
The NVE implements network virtualization functions that allow for The NVE implements network virtualization functions that allow for
L2 and/or L3 tenant separation and for hiding tenant addressing L2 and/or L3 tenant separation.
information (MAC and IP addresses), tenant-related control plane
activity and service contexts from the underlay nodes.
Underlay nodes utilize L3 techniques to interconnect NVE nodes in Underlay nodes utilize L3 technologies to interconnect NVE nodes.
support of the overlay network. These devices perform forwarding These nodes perform forwarding based on outer L3 header information,
based on outer L3 tunnel header, and generally do not maintain per and generally do not maintain per tenant-service state albeit some
tenant-service state albeit some applications (e.g., multicast) may applications (e.g., multicast) may require control plane or
require control plane or forwarding plane information that pertain forwarding plane information that pertain to a tenant, group of
to a tenant, group of tenants, tenant service or a set of services tenants, tenant service or a set of services that belong to one or
that belong to one or more tenants. When such tenant or tenant- more tenants. When such tenant or tenant-service related information
service related information is maintained in the underlay, overlay is maintained in the underlay, mechanisms to control that
virtualization provides knobs to control the magnitude of that information should be provided.
information.
2.2. NVE Reference Model 2.2. NVE Reference Model
One or more VNIs can be instantiated on an NVE. Tenant Systems Figure 3 depicts the NVE reference model. One or more VNIs can be
interface with a corresponding VNI via a Virtual Access Point (VAP). instantiated on an NVE. A Tenant System interfaces with a
An overlay module that provides tunneling overlay functions (e.g., corresponding VNI via a VAP. An overlay module provides tunneling
encapsulation and decapsulation of tenant traffic from/to the tenant overlay functions (e.g., encapsulation and decapsulation of tenant
forwarding instance, tenant identification and mapping, etc), as traffic, tenant identification and mapping, etc.).
described in figure 4:
+------- L3 Network ------+ +-------- L3 Network -------+
| | | |
| Tunnel Overlay | | Tunnel Overlay |
+------------+---------+ +---------+------------+ +------------+---------+ +---------+------------+
| +----------+-------+ | | +---------+--------+ | | +----------+-------+ | | +---------+--------+ |
| | Overlay Module | | | | Overlay Module | | | | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ | | +---------+--------+ | | +---------+--------+ |
| |VN context| | VN context| | | |VN context| | VN context| |
| | | | | | | | | | | |
| +--------+-------+ | | +--------+-------+ | | +--------+-------+ | | +--------+-------+ |
| | |VNI| . |VNI| | | | |VNI| . |VNI| | | | |VNI| . |VNI| | | | |VNI| . |VNI| |
NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2 NVE1 | +-+------------+-+ | | +-+-----------+--+ | NVE2
| | VAPs | | | | VAPs | | | | VAPs | | | | VAPs | |
+----+------------+----+ +----+-----------+-----+ +----+------------+----+ +----+-----------+-----+
| | | | | | | |
-------+------------+-----------------+-----------+------- | | | |
| | Tenant | |
| | Service IF | |
Tenant Systems Tenant Systems Tenant Systems Tenant Systems
Figure 4 : Generic reference model for NV Edge Figure 3 : Generic NVE reference model
Note that some NVE functions (e.g., data plane and control plane Note that some NVE functions (e.g., data plane and control plane
functions) may reside in one device or may be implemented separately functions) may reside in one device or may be implemented separately
in different devices. For example, the NVE functionality could in different devices. In addition, NVE functions can be implemented
reside solely on the End Devices, or be distributed between the End in a hierarchical fashion. For instance, an End Device can act as an
Devices and the ToRs. In the latter case we say that the End Device NVE Spoke, while an access switch can act as an NVE hub.
NVE component acts as the NVE Spoke, and ToRs act as NVE hubs.
Tenant Systems will interface with VNIs maintained on the NVE
spokes, and VNIs maintained on the NVE spokes will interface with
VNIs maintained on the NVE hubs.
2.3. NVE Service Types 2.3. NVE Service Types
NVE components may be used to provide different types of virtualized NVE components may be used to provide different types of virtualized
network services. This section defines the service types and network services. This section defines the service types and
associated attributes. Note that an NVE may be capable of providing associated attributes. Note that an NVE may be capable of providing
both L2 and L3 services. both L2 and L3 services.
2.3.1. L2 NVE providing Ethernet LAN-like service 2.3.1. L2 NVE providing Ethernet LAN-like service
L2 NVE implements Ethernet LAN emulation (ELAN), an Ethernet based L2 NVE implements Ethernet LAN emulation, an Ethernet based
multipoint service where the Tenant Systems appear to be multipoint service similar to an IETF VPLS or EVPN service, where
interconnected by a LAN environment over a set of L3 tunnels. It the Tenant Systems appear to be interconnected by a LAN environment
provides per tenant virtual switching instance with MAC addressing over an L3 overlay. As such, an L2 NVE provides per-tenant virtual
isolation and L3 (IP/MPLS) tunnel encapsulation across the underlay. switching instance (L2 VNI), and L3 (IP/MPLS) tunneling
encapsulation of tenant MAC frames across the underlay. Note that
the control plane for an L2 NVE could be implemented locally on the
NVE or in a separate control entity.
2.3.2. L3 NVE providing IP/VRF-like service 2.3.2. L3 NVE providing IP/VRF-like service
Virtualized IP routing and forwarding is similar from a service L3 NVE provides Virtualized IP forwarding service, similar from a
definition perspective with IETF IP VPN (e.g., BGP/MPLS IPVPN service definition perspective to IETF IP VPN (e.g., BGP/MPLS IPVPN
[RFC4364] and IPsec VPNs). It provides per tenant routing instance [RFC4364]). That is, an L3 NVE provides per-tenant forwarding and
with addressing isolation and L3 (IP/MPLS) tunnel encapsulation routing instance (L3 VNI), and L3 (IP/MPLS) tunneling encapsulation
across the underlay. of tenant IP packets across the underlay. Note that routing could be
performed locally on the NVE or in a separate control entity.
3. Functional components 3. Functional components
This section decomposes the Network Virtualization architecture into This section decomposes the Network Virtualization architecture into
functional components described in Figure 4 to make it easier to functional components described in Figure 3 to make it easier to
discuss solution options for these components. discuss solution options for these components.
3.1. Service Virtualization Components 3.1. Service Virtualization Components
3.1.1. Virtual Access Points (VAPs) 3.1.1. Virtual Access Points (VAPs)
Tenant Systems are connected to the VNI Instance through Virtual Tenant Systems are connected to VNIs through Virtual Access Points
Access Points (VAPs). (VAPs).
The VAPs can be physical ports or virtual ports identified through VAPs can be physical ports or virtual ports identified through
logical interface identifiers (e.g., VLAN ID, internal vSwitch logical interface identifiers (e.g., VLAN ID, internal vSwitch
Interface ID coonected to a VM). Interface ID connected to a VM).
3.1.2. Virtual Network Instance (VNI) 3.1.2. Virtual Network Instance (VNI)
The VNI represents a set of configuration attributes defining access A VNI is a specific VN instance on a NVE. Each VNI defines a
and tunnel policies and (L2 and/or L3) forwarding functions. forwarding context that contains reachability information and
policies.
Per tenant FIB tables and control plane protocol instances are used
to maintain separate private contexts between tenants. Hence tenants
are free to use their own addressing schemes without concerns about
address overlapping with other tenants.
3.1.3. Overlay Modules and VN Context 3.1.3. Overlay Modules and VN Context
Mechanisms for identifying each tenant service are required to allow Mechanisms for identifying each tenant service are required to allow
the simultaneous overlay of multiple tenant services over the same the simultaneous overlay of multiple tenant services over the same
underlay L3 network topology. In the data plane, each NVE, upon underlay L3 network topology. In the data plane, each NVE, upon
sending a tenant packet, must be able to encode the VN Context for sending a tenant packet, must be able to encode the VN Context for
the destination NVE in addition to the L3 tunnel information (e.g., the destination NVE in addition to the L3 tunneling information
source IP address identifying the source NVE and the destination IP (e.g., source IP address identifying the source NVE and the
address identifying the destination NVE, or MPLS label). This allows destination IP address identifying the destination NVE, or MPLS
the destination NVE to identify the tenant service instance and label). This allows the destination NVE to identify the tenant
therefore appropriately process and forward the tenant packet. service instance and therefore appropriately process and forward the
tenant packet.
The Overlay module provides tunneling overlay functions: tunnel The Overlay module provides tunneling overlay functions: tunnel
initiation/termination, encapsulation/decapsulation of frames from initiation/termination as in the case of stateful tunnels (see
VAPs/L3 Backbone and may provide for transit forwarding of IP Section 3.1.4), and/or simply encapsulation/decapsulation of frames
traffic (e.g., transparent tunnel forwarding). from VAPs/L3 underlay.
In a multi-tenant context, the tunnel aggregates frames from/to In a multi-tenant context, tunneling aggregates frames from/to
different VNIs. Tenant identification and traffic demultiplexing are different VNIs. Tenant identification and traffic demultiplexing are
based on the VN Context identifier (e.g., VNID). based on the VN Context identifier.
The following approaches can been considered: The following approaches can be considered:
o One VN Context per Tenant: A globally unique (on a per-DC o One VN Context identifier per Tenant: A globally unique (on a
administrative domain) VNID is used to identify the related per-DC administrative domain) VN identifier is used to identify
Tenant instances. An example of this approach is the use of the corresponding VNI. Examples of such identifiers in existing
IEEE VLAN or ISID tags to provide virtual L2 domains. technologies are IEEE VLAN IDs and ISID tags that identify
virtual L2 domains when using IEEE 802.1aq and IEEE 802.1ah,
respectively.
o One VN Context per VNI: A per-tenant local value is o One VN Context identifier per VNI: A per-VNI local value is
automatically generated by the egress NVE and usually automatically generated by the egress NVE, or a control plane
distributed by a control plane protocol to all the related associated with that NVE, and usually distributed by a control
NVEs. An example of this approach is the use of per VRF MPLS plane protocol to all the related NVEs. An example of this
labels in IP VPN [RFC4364]. approach is the use of per VRF MPLS labels in IP VPN [RFC4364].
o One VN Context per VAP: A per-VAP local value is assigned and o One VN Context identifier per VAP: A per-VAP local value is
usually distributed by a control plane protocol. An example of assigned and usually distributed by a control plane protocol.
this approach is the use of per CE-PE MPLS labels in IP VPN
[RFC4364]. An example of this approach is the use of per CE-PE MPLS labels
in IP VPN [RFC4364].
Note that when using one VN Context per VNI or per VAP, an Note that when using one VN Context per VNI or per VAP, an
additional global identifier may be used by the control plane to additional global identifier (e.g., a VN identifier or name) may be
identify the Tenant context. used by the control plane to identify the Tenant context.
3.1.4. Tunnel Overlays and Encapsulation options 3.1.4. Tunnel Overlays and Encapsulation options
Once the VN context identifier is added to the frame, a L3 Tunnel Once the VN context identifier is added to the frame, a L3 Tunnel
encapsulation is used to transport the frame to the destination NVE. encapsulation is used to transport the frame to the destination NVE.
The backbone devices do not usually keep any per service state, The underlay devices do not usually keep any per service state,
simply forwarding the frames based on the outer tunnel header. simply forwarding the frames based on the outer tunnel header.
Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS Different IP tunneling options (e.g., GRE, L2TP, IPSec) and MPLS
tunneling options (e.g., BGP VPN, VPLS) can be used. tunneling can be used. Tunneling could be stateless or stateful.
Stateless tunneling simply entails the encapsulation of a tenant
packet with another header necessary for forwarding the packet
across the underlay (e.g., IP tunneling over an IP underlay.
Stateful tunneling on the other hand entails maintaining tunneling
state at the tunnel endpoints (i.e., NVEs). Tenant packets on an
ingress NVE can then be transmitted over such tunnels to a
destination (egress) NVE by encapsulating the packets with a
corresponding tunneling header. The tunneling state at the endpoints
may be configured or dynamically established. Solutions SHOULD
specify the tunneling technology used, whether it is stateful or
stateless. In this document, however, tunneling and tunneling
encapsulation are used interchangeably to simply mean the
encapsulation of a tenant packet with a tunneling header necessary
to deliver the packet between an ingress NVE and an egress NVE
across the underlay. It should be noted that stateful tunneling,
especially when configuration is involved, does impose management
overhead and scale constraints. Thus, stateless tunneling is
preferred when feasible.
3.1.5. Control Plane Components 3.1.5. Control Plane Components
Control plane components may be used to provide the following
capabilities:
. Auto-provisioning/Service discovery
. Address advertisement and tunnel mapping
. Tunnel management
A control plane component can be an on-net control protocol
implemented on the NVE or a management control entity.
3.1.5.1. Distributed vs Centralized Control Plane 3.1.5.1. Distributed vs Centralized Control Plane
A control/management plane entity can be centralized or distributed. A control/management plane entity can be centralized or distributed.
Both approaches have been used extensively in the past. The routing Both approaches have been used extensively in the past. The routing
model of the Internet is a good example of a distributed approach. model of the Internet is a good example of a distributed approach.
Transport networks have usually used a centralized approach to Transport networks have usually used a centralized approach to
manage transport paths. manage transport paths.
It is also possible to combine the two approaches i.e. using a It is also possible to combine the two approaches, i.e., using a
hybrid model. A global view of network state can have many benefits hybrid model. A global view of network state can have many benefits
but it does not preclude the use of distributed protocols within the but it does not preclude the use of distributed protocols within the
network. Centralized controllers provide a facility to maintain network. Centralized models provide a facility to maintain global
global state, and distribute that state to the network which in state, and distribute that state to the network. When used in
combination with distributed protocols can aid in achieving greater combination with distributed protocols, greater network
network efficiencies, and improve reliability and robustness. Domain efficiencies, improved reliability and robustness can be achieved.
and/or deployment specific constraints define the balance between Domain and/or deployment specific constraints define the balance
centralized and distributed approaches. between centralized and distributed approaches.
On one hand, a control plane module can reside in every NVE. This is
how routing control plane modules are implemented in routers. At the
same time, an external controller can manage a group of NVEs via an
agent in each NVE. This is how an SDN controller could communicate
with the nodes it controls, via OpenFlow [OF] for instance.
In the case where a logically centralized control plane is
preferred, the controller will need to be distributed to more than
one node for redundancy and scalability in order to manage a large
number of NVEs. Hence, inter-controller communication is necessary
to synchronize state among controllers. It should be noted that
controllers may be organized in clusters. The information exchanged
between controllers of the same cluster could be different from the
information exchanged across clusters.
3.1.5.2. Auto-provisioning/Service discovery 3.1.5.2. Auto-provisioning/Service discovery
NVEs must be able to identify the appropriate VNI for each Tenant NVEs must be able to identify the appropriate VNI for each Tenant
System. This is based on state information that is often provided by System. This is based on state information that is often provided by
external entities. For example, in an environment where a VM is a external entities. For example, in an environment where a VM is a
Tenant System, this information is provided by compute management Tenant System, this information is provided by VM orchestration
systems, since these are the only entities that have visibility of systems, since these are the only entities that have visibility of
which VM belongs to which tenant. which VM belongs to which tenant.
A mechanism for communicating this information between Tenant A mechanism for communicating this information to the NVE is
Systems and the corresponding NVE is required. As a result the VAPs required. VAPs have to be created and mapped to the appropriate VNI.
are created and mapped to the appropriate VNI. Depending upon the Depending upon the implementation, this control interface can be
implementation, this control interface can be implemented using an implemented using an auto-discovery protocol between Tenant Systems
auto-discovery protocol between Tenant Systems and their local NVE and their local NVE or through management entities. In either case,
or through management entities. In either case, appropriate security appropriate security and authentication mechanisms to verify that
and authentication mechanisms to verify that Tenant System Tenant System information is not spoofed or altered are required.
information is not spoofed or altered are required. This is one This is one critical aspect for providing integrity and tenant
critical aspect for providing integrity and tenant isolation in the isolation in the system.
system.
A control plane protocol can also be used to advertize supported VNs NVEs may learn reachability information to VNIs on other NVEs via a
to other NVEs. Alternatively, management control entities can also control protocol exchanging such information among NVEs or via a
be used to perform these functions. management control entity.
3.1.5.3. Address advertisement and tunnel mapping 3.1.5.3. Address advertisement and tunnel mapping
As traffic reaches an ingress NVE, a lookup is performed to As traffic reaches an ingress NVE on a VAP, a lookup is performed to
determine which tunnel the packet needs to be sent to. It is then determine which NVE or local VAP the packet needs to be sent to. If
encapsulated with a tunnel header containing the destination the packet is to be sent to another NVE, the packet is encapsulated
information (destination IP address or MPLS label) of the egress with a tunnel header containing the destination information
overlay node. Intermediate nodes (between the ingress and egress (destination IP address or MPLS label) of the egress NVE.
NVEs) switch or route traffic based upon the outer destination Intermediate nodes (between the ingress and egress NVEs) switch or
information. route traffic based upon the tunnel destination information.
One key step in this process consists of mapping a final destination A key step in the above process consists of identifying the
information to the proper tunnel. NVEs are responsible for destination NVE the packet is to be tunneled to. NVEs are
maintaining such mappings in their forwarding tables. Several ways responsible for maintaining a set of forwarding or mapping tables
of populating these tables are possible: control plane driven, that hold the bindings between destination VM and egress NVE
management plane driven, or data plane driven. addresses. Several ways of populating these tables are possible:
control plane driven, management plane driven, or data plane driven.
When a control plane protocol is used to distribute address When a control plane protocol is used to distribute address
advertisement and tunneling information, the auto- reachability and tunneling information, the auto-
provisioning/Service discovery could be accomplished by the same provisioning/Service discovery could be accomplished by the same
protocol. In this scenario, the auto-provisioning/Service discovery protocol. In this scenario, the auto-provisioning/Service discovery
could be combined with (be inferred from) the address advertisement could be combined with (be inferred from) the address advertisement
and associated tunnel mapping. Furthermore, a control plane protocol and associated tunnel mapping. Furthermore, a control plane protocol
that carries both MAC and IP addresses eliminates the need for ARP, that carries both MAC and IP addresses eliminates the need for ARP,
and hence addresses one of the issues with explosive ARP handling. and hence addresses one of the issues with explosive ARP handling.
3.1.5.4. Overlay Tunneling 3.1.5.4. Overlay Tunneling
For overlay tunneling, and dependent upon the tunneling technology For overlay tunneling, and dependent upon the tunneling technology
skipping to change at page 17, line 10 skipping to change at page 17, line 10
Multi-homing techniques can be used to increase the reliability of Multi-homing techniques can be used to increase the reliability of
an nvo3 network. It is also important to ensure that physical an nvo3 network. It is also important to ensure that physical
diversity in an nvo3 network is taken into account to avoid single diversity in an nvo3 network is taken into account to avoid single
points of failure. points of failure.
Multi-homing can be enabled in various nodes, from tenant systems Multi-homing can be enabled in various nodes, from tenant systems
into TORs, TORs into core switches/routers, and core nodes into DC into TORs, TORs into core switches/routers, and core nodes into DC
GWs. GWs.
The nvo3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP The nvo3 underlay nodes (i.e. from NVEs to DC GWs) rely on IP
routing as the means to re-route traffic upon failures and/or ECMP routing as the means to re-route traffic upon failures techniques or
techniques or on MPLS re-rerouting capabilities. on MPLS re-rerouting capabilities.
When a tenant system is co-located with the NVE on the same end- When a tenant system is co-located with the NVE, the Tenant System
system, the tenant system is single homed to the NVE via a vport is single homed to the NVE via a virtual port. When the Tenant
that is virtual NIC (vNIC). When the end system and the NVEs are System and the NVE are separated, the Tenant System is connected to
separated, the end system is connected to the NVE via a logical the NVE via a logical Layer2 (L2) construct such as a VLAN and it
Layer2 (L2) construct such as a VLAN. In this latter case, an end can be multi-homed to various NVEs. An NVE may provide an L2 service
device or vSwitch on that device could be multi-homed to various to the end system or an l3 service. An NVE may be multi-homed to a
NVEs. An NVE may provide an L2 service to the end system or a l3 next layer in the DC at Layer2 (L2) or Layer3 (L3). When an NVE
service. An NVE may be multi-homed to a next layer in the DC at provides an L2 service and is not co-located with the end
Layer2 (L2) or Layer3 (L3). When an NVE provides an L2 service and system, techniques such as Ethernet Link Aggregation Group (LAG) or
is not co-located with the end system, techniques such as Ethernet Spanning Tree Protocol (STP) can be used to switch traffic
Link Aggregation Group (LAG) or Spanning Tree Protocol (STP) can be between an end system and connected NVEs without creating
used to switch traffic between an end system and connected loops. Similarly, when the NVE provides L3 service, similar dual-
NVEs without creating loops. Similarly, when the NVE provides L3 homing techniques can be used. When the NVE provides a L3 service to
service, similar dual-homing techniques can be used. When the NVE the end system, it is possible that no dynamic routing protocol is
provides a L3 service to the end system, it is possible that no enabled between the end system and the NVE. The end system can be
dynamic routing protocol is enabled between the end system and the multi-homed to multiple physically-separated L3 NVEs over multiple
NVE. The end system can be multi-homed to multiple physically- interfaces. When one of the links connected to an NVE fails, the
separated L3 NVEs over multiple interfaces. When one of the other interfaces can be used to reach the end system.
links connected to an NVE fails, the other interfaces can be used to
reach the end system.
External connectivity out of an nvo3 domain can be handled by two or External connectivity out of a DC can be handled by two or more DC
more nvo3 gateways. Each gateway is connected to a different domain gateways. Each gateway provides access to external networks such as
(e.g. ISP), providing access to external networks such as VPNs or VPNs or the Internet. A gateway may be connected to two or more edge
the Internet. A gateway may be connected to two nodes. When a nodes in the external network for redundancy. When a connection to
connection to an upstream node is lost, the alternative connection an upstream node is lost, the alternative connection is used and the
is used and the failed route withdrawn. failed route withdrawn.
3.3. VM Mobility 3.3. VM Mobility
In DC environments utilizing VM technologies, an important feature In DC environments utilizing VM technologies, an important feature
is that VMs can move from one server to another server in the same is that VMs can move from one server to another server in the same
or different L2 physical domains (within or across DCs) in a or different L2 physical domains (within or across DCs) in a
seamless manner. seamless manner.
A VM can be moved from one server to another in stopped or suspended A VM can be moved from one server to another in stopped or suspended
state ("cold" VM mobility) or in running/active state ("hot" VM state ("cold" VM mobility) or in running/active state ("hot" VM
mobility). With "hot" mobility, VM L2 and L3 addresses need to be mobility). With "hot" mobility, VM L2 and L3 addresses need to be
preserved. With "cold" mobility, it may be desired to preserve VM L3 preserved. With "cold" mobility, it may be desired to preserve at
addresses. least VM L3 addresses.
Solutions to maintain connectivity while a VM is moved are necessary Solutions to maintain connectivity while a VM is moved are necessary
in the case of "hot" mobility. This implies that transport in the case of "hot" mobility. This implies that transport
connections among VMs are preserved and that ARP caches are updated connections among VMs are preserved. For instance, for L2 VNs, ARP
accordingly. caches are updated accordingly.
Upon VM mobility, NVE policies that define connectivity among VMs Upon VM mobility, NVE policies that define connectivity among VMs
must be maintained. must be maintained.
Optimal routing during VM mobility is also an important aspect to Optimal routing during VM mobility is also an important aspect to
address. It is expected that the VM's default gateway be as close as address. It is expected that the VM's default gateway be as close as
possible to the server hosting the VM and triangular routing be possible to the server hosting the VM.
avoided.
3.4. Service Overlay Topologies
A number of service topologies may be used to optimize the service
connectivity and to address NVE performance limitations.
The topology described in Figure 3 suggests the use of a tunnel mesh
between the NVEs where each tenant instance is one hop away from a
service processing perspective. Partial mesh topologies and an NVE
hierarchy may be used where certain NVEs may act as service transit
points.
4. Key aspects of overlay networks 4. Key aspects of overlay networks
The intent of this section is to highlight specific issues that The intent of this section is to highlight specific issues that
proposed overlay solutions need to address. proposed overlay solutions need to address.
4.1. Pros & Cons 4.1. Pros & Cons
An overlay network is a layer of virtual network topology on top of An overlay network is a layer of virtual network topology on top of
the physical network. the physical network.
Overlay networks offer the following key advantages: Overlay networks offer the following key advantages:
o Unicast tunneling state management and association with tenant o Unicast tunneling state management and association of Tenant
systems reachability are handled at the edge of the network. Systems reachability are handled at the edge of the network (at
Intermediate transport nodes are unaware of such state. Note the NVE). Intermediate transport nodes are unaware of such
that this is not the case when multicast is enabled in the core state. Note that when multicast is enabled in the underlay
network. network to build multicast trees for tenant VNs, there would be
more state related to tenants in the underlay core network.
o Tunneling is used to aggregate traffic and hide tenant o Tunneling is used to aggregate traffic and hide tenant
addresses from the underkay network, and hence offer the addresses from the underlay network, and hence offer the
advantage of minimizing the amount of forwarding state required advantage of minimizing the amount of forwarding state required
within the underlay network within the underlay network
o Decoupling of the overlay addresses (MAC and IP) used by VMs o Decoupling of the overlay addresses (MAC and IP) used by VMs
from the underlay network. This offers a clear separation from the underlay network for tenant separation and separation
between addresses used within the overlay and the underlay of the tenant address spaces and the underlay address space.
networks and it enables the use of overlapping addresses spaces
by Tenant Systems
o Support of a large number of virtual network identifiers o Support of a large number of virtual network identifiers
Overlay networks also create several challenges: Overlay networks also create several challenges:
o Overlay networks have no controls of underlay networks and lack o Overlay networks have no controls of underlay networks and lack
critical network information critical underlay network information
o Overlay networks and/or their associated management
o Overlays typically probe the network to measure link or entities typically probe the network to measure link or
path properties, such as available bandwidth or packet path properties, such as available bandwidth or packet
loss rate. It is difficult to accurately evaluate network loss rate. It is difficult to accurately evaluate network
properties. It might be preferable for the underlay properties. It might be preferable for the underlay
network to expose usage and performance information. network to expose usage and performance information.
o Miscommunication or lack of coordination between overlay and o Miscommunication or lack of coordination between overlay and
underlay networks can lead to an inefficient usage of network underlay networks can lead to an inefficient usage of network
resources. resources.
o When multiple overlays co-exist on top of a common underlay o When multiple overlays co-exist on top of a common underlay
network, the lack of coordination between overlays can lead to network, the lack of coordination between overlays can lead to
performance issues. performance issues and/or resource usage inefficiencies.
o Overlaid traffic may not traverse firewalls and NAT devices. o Traffic carried over an overlay may not traverse firewalls and
NAT devices.
o Multicast service scalability. Multicast support may be o Multicast service scalability: Multicast support may be
required in the underlay network to address for each tenant required in the underlay network to address tenant flood
flood containment or efficient multicast handling. The underlay containment or efficient multicast handling. The underlay may
may be also be required to maintain multicast state on a per- also be required to maintain multicast state on a per-tenant
tenant basis, or even on a per-individual multicast flow of a basis, or even on a per-individual multicast flow of a given
given tenant. tenant. Ingress replication at the NVE eliminates that
additional multicast state in the underlay core, but depending
on the multicast traffic volume, it may cause inefficient use
of bandwidth.
o Hash-based load balancing may not be optimal as the hash o Hash-based load balancing may not be optimal as the hash
algorithm may not work well due to the limited number of algorithm may not work well due to the limited number of
combinations of tunnel source and destination addresses. Other combinations of tunnel source and destination addresses. Other
NVO3 mechanisms may use additional entropy information than NVO3 mechanisms may use additional entropy information than
source and destination addresses. source and destination addresses.
4.2. Overlay issues to consider 4.2. Overlay issues to consider
4.2.1. Data plane vs Control plane driven 4.2.1. Data plane vs Control plane driven
In the case of an L2NVE, it is possible to dynamically learn MAC In the case of an L2 NVE, it is possible to dynamically learn MAC
addresses against VAPs. It is also possible that such addresses be addresses against VAPs. It is also possible that such addresses be
known and controlled via management or a control protocol for both known and controlled via management or a control protocol for both
L2NVEs and L3NVEs. L2 NVEs and L3 NVEs. Dynamic data plane learning implies that
flooding of unknown destinations be supported and hence implies that
Dynamic data plane learning implies that flooding of unknown broadcast and/or multicast be supported or that ingress replication
destinations be supported and hence implies that broadcast and/or be used as described in section 4.2.3. Multicasting in the underlay
multicast be supported or that ingress replication be used as network for dynamic learning may lead to significant scalability
described in section 4.2.3. Multicasting in the underlay network for limitations. Specific forwarding rules must be enforced to prevent
dynamic learning may lead to significant scalability limitations. loops from happening. This can be achieved using a spanning tree, a
Specific forwarding rules must be enforced to prevent loops from shortest path tree, or a split-horizon mesh.
happening. This can be achieved using a spanning tree, a shortest
path tree, or a split-horizon mesh.
It should be noted that the amount of state to be distributed is It should be noted that the amount of state to be distributed is
dependent upon network topology and the number of virtual machines. dependent upon network topology and the number of virtual machines.
Different forms of caching can also be utilized to minimize state Different forms of caching can also be utilized to minimize state
distribution between the various elements. The control plane should distribution between the various elements. The control plane should
not require an NVE to maintain the locations of all the tenant not require an NVE to maintain the locations of all the tenant
systems whose VNs are not present on the NVE. The use of a control systems whose VNs are not present on the NVE. The use of a control
plane does not imply that the data plane on NVEs has to maintain all plane does not imply that the data plane on NVEs has to maintain all
the forwarding state in the control plane. the forwarding state in the control plane.
4.2.2. Coordination between data plane and control plane 4.2.2. Coordination between data plane and control plane
For an L2 NVE, the NVE needs to be able to determine MAC addresses For an L2 NVE, the NVE needs to be able to determine MAC addresses
of the end systems connected via a VAP. This can be achieved via of the Tenant Systems connected via a VAP. This can be achieved via
dataplane learning or a control plane. For an L3 NVE, the NVE needs dataplane learning or a control plane. For an L3 NVE, the NVE needs
to be able to determine IP addresses of the end systems connected to be able to determine IP addresses of the Tenant Systems connected
via a VAP. via a VAP.
In both cases, coordination with the NVE control protocol is needed In both cases, coordination with the NVE control protocol is needed
such that when the NVE determines that the set of addresses behind a such that when the NVE determines that the set of addresses behind a
VAP has changed, it triggers the local NVE control plane to VAP has changed, it triggers the NVE control plane to distribute
distribute this information to its peers. this information to its peers.
4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic 4.2.3. Handling Broadcast, Unknown Unicast and Multicast (BUM) traffic
There are two techniques to support packet replication needed for There are several options to support packet replication needed for
broadcast, unknown unicast and multicast: broadcast, unknown unicast and multicast. Typical methods include:
o Ingress replication o Ingress replication
o Use of underlay multicast trees o Use of underlay multicast trees
There is a bandwidth vs state trade-off between the two approaches. There is a bandwidth vs state trade-off between the two approaches.
Depending upon the degree of replication required (i.e. the number Depending upon the degree of replication required (i.e. the number
of hosts per group) and the amount of multicast state to maintain, of hosts per group) and the amount of multicast state to maintain,
trading bandwidth for state should be considered. trading bandwidth for state should be considered.
When the number of hosts per group is large, the use of underlay When the number of hosts per group is large, the use of underlay
multicast trees may be more appropriate. When the number of hosts is multicast trees may be more appropriate. When the number of hosts is
small (e.g. 2-3), ingress replication may not be an issue. small (e.g. 2-3) and/or the amount of multicast traffic is small,
ingress replication may not be an issue.
Depending upon the size of the data center network and hence the Depending upon the size of the data center network and hence the
number of (S,G) entries, but also the duration of multicast flows, number of (S,G) entries, but also the duration of multicast flows,
the use of underlay multicast trees can be a challenge. the use of underlay multicast trees can be a challenge.
When flows are well known, it is possible to pre-provision such When flows are well known, it is possible to pre-provision such
multicast trees. However, it is often difficult to predict multicast trees. However, it is often difficult to predict
application flows ahead of time, and hence programming of (S,G) application flows ahead of time, and hence programming of (S,G)
entries for short-lived flows could be impractical. entries for short-lived flows could be impractical.
skipping to change at page 22, line 8 skipping to change at page 21, line 45
o o
Tenant Systems rely on ICMP messages to discover the MTU of Tenant Systems rely on ICMP messages to discover the MTU of
the end-to-end path to its destination. This method is not the end-to-end path to its destination. This method is not
always possible, such as when traversing middle boxes always possible, such as when traversing middle boxes
(e.g. firewalls) which disable ICMP for security reasons (e.g. firewalls) which disable ICMP for security reasons
o Extended MTU Path Discovery techniques such as defined in o Extended MTU Path Discovery techniques such as defined in
[RFC4821] [RFC4821]
It is also possible to rely on the overlay layer to perform It is also possible to rely on the NVE to perform segmentation and
segmentation and reassembly operations without relying on the Tenant reassembly operations without relying on the Tenant Systems to know
Systems to know about the end-to-end MTU. The assumption is that about the end-to-end MTU. The assumption is that some hardware
some hardware assist is available on the NVE node to perform such assist is available on the NVE node to perform such SAR operations.
SAR operations. However, fragmentation by the overlay layer can lead However, fragmentation by the NVE can lead to performance and
to performance and congestion issues due to TCP dynamics and might congestion issues due to TCP dynamics and might require new
require new congestion avoidance mechanisms from then underlay congestion avoidance mechanisms from the underlay network [FLOYD].
network [FLOYD].
Finally, the underlay network may be designed in such a way that the Finally, the underlay network may be designed in such a way that the
MTU can accommodate the extra tunneling and possibly additional nvo3 MTU can accommodate the extra tunneling and possibly additional nvo3
header encapsulation overhead. header encapsulation overhead.
4.2.5. NVE location trade-offs 4.2.5. NVE location trade-offs
In the case of DC traffic, traffic originated from a VM is native In the case of DC traffic, traffic originated from a VM is native
Ethernet traffic. This traffic can be switched by a local virtual Ethernet traffic. This traffic can be switched by a local virtual
switch or ToR switch and then by a DC gateway. The NVE function can switch or ToR switch and then by a DC gateway. The NVE function can
skipping to change at page 23, line 37 skipping to change at page 23, line 26
o Performance metrics (throughput, delay, loss, jitter) o Performance metrics (throughput, delay, loss, jitter)
o Cost metrics o Cost metrics
5. Security Considerations 5. Security Considerations
Nvo3 solutions must at least consider and address the following: Nvo3 solutions must at least consider and address the following:
. Secure and authenticated communication between an NVE and an . Secure and authenticated communication between an NVE and an
NVE management system. NVE management system and/or control system.
. Isolation between tenant overlay networks. The use of per- . Isolation between tenant overlay networks. The use of per-
tenant FIB tables (VNIs) on an NVE is essential. tenant FIB tables (VNIs) on an NVE is essential.
. Security of any protocol used to carry overlay network . Security of any protocol used to carry overlay network
information. information.
. Avoiding packets from reaching the wrong NVI, especially during . Avoiding packets from reaching the wrong NVI, especially during
VM moves. VM moves.
 End of changes. 107 change blocks. 
435 lines changed or deleted 419 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/