draft-ietf-nvo3-geneve-08.txt   draft-ietf-nvo3-geneve-09.txt 
Network Working Group J. Gross, Ed. Network Working Group J. Gross, Ed.
Internet-Draft Internet-Draft
Intended status: Standards Track I. Ganga, Ed. Intended status: Standards Track I. Ganga, Ed.
Expires: April 10, 2019 Intel Expires: August 26, 2019 Intel
T. Sridhar, Ed. T. Sridhar, Ed.
VMware VMware
October 07, 2018 February 22, 2019
Geneve: Generic Network Virtualization Encapsulation Geneve: Generic Network Virtualization Encapsulation
draft-ietf-nvo3-geneve-08 draft-ietf-nvo3-geneve-09
Abstract Abstract
Network virtualization involves the cooperation of devices with a Network virtualization involves the cooperation of devices with a
wide variety of capabilities such as software and hardware tunnel wide variety of capabilities such as software and hardware tunnel
endpoints, transit fabrics, and centralized control clusters. As a endpoints, transit fabrics, and centralized control clusters. As a
result of their role in tying together different elements in the result of their role in tying together different elements in the
system, the requirements on tunnels are influenced by all of these system, the requirements on tunnels are influenced by all of these
components. Flexibility is therefore the most important aspect of a components. Flexibility is therefore the most important aspect of a
tunnel protocol if it is to keep pace with the evolution of the tunnel protocol if it is to keep pace with the evolution of the
skipping to change at page 1, line 41 skipping to change at page 1, line 41
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 10, 2019. This Internet-Draft will expire on August 26, 2019.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 2, line 30 skipping to change at page 2, line 30
2.2.1. Efficient Implementation . . . . . . . . . . . . . . 7 2.2.1. Efficient Implementation . . . . . . . . . . . . . . 7
2.3. Use of Standard IP Fabrics . . . . . . . . . . . . . . . 8 2.3. Use of Standard IP Fabrics . . . . . . . . . . . . . . . 8
3. Geneve Encapsulation Details . . . . . . . . . . . . . . . . 9 3. Geneve Encapsulation Details . . . . . . . . . . . . . . . . 9
3.1. Geneve Packet Format Over IPv4 . . . . . . . . . . . . . 9 3.1. Geneve Packet Format Over IPv4 . . . . . . . . . . . . . 9
3.2. Geneve Packet Format Over IPv6 . . . . . . . . . . . . . 10 3.2. Geneve Packet Format Over IPv6 . . . . . . . . . . . . . 10
3.3. UDP Header . . . . . . . . . . . . . . . . . . . . . . . 12 3.3. UDP Header . . . . . . . . . . . . . . . . . . . . . . . 12
3.4. Tunnel Header Fields . . . . . . . . . . . . . . . . . . 13 3.4. Tunnel Header Fields . . . . . . . . . . . . . . . . . . 13
3.5. Tunnel Options . . . . . . . . . . . . . . . . . . . . . 14 3.5. Tunnel Options . . . . . . . . . . . . . . . . . . . . . 14
3.5.1. Options Processing . . . . . . . . . . . . . . . . . 16 3.5.1. Options Processing . . . . . . . . . . . . . . . . . 16
4. Implementation and Deployment Considerations . . . . . . . . 17 4. Implementation and Deployment Considerations . . . . . . . . 17
4.1. Encapsulation of Geneve in IP . . . . . . . . . . . . . . 17 4.1. Applicability Statement . . . . . . . . . . . . . . . . . 17
4.1.1. IP Fragmentation . . . . . . . . . . . . . . . . . . 17 4.2. Congestion Control Functionality . . . . . . . . . . . . 18
4.1.2. DSCP and ECN . . . . . . . . . . . . . . . . . . . . 17 4.3. UDP Checksum . . . . . . . . . . . . . . . . . . . . . . 18
4.1.3. Broadcast and Multicast . . . . . . . . . . . . . . . 18 4.3.1. UDP Zero Checksum Handling with IPv6 . . . . . . . . 18
4.1.4. Unidirectional Tunnels . . . . . . . . . . . . . . . 18 4.4. Encapsulation of Geneve in IP . . . . . . . . . . . . . . 20
4.2. Constraints on Protocol Features . . . . . . . . . . . . 19 4.4.1. IP Fragmentation . . . . . . . . . . . . . . . . . . 20
4.2.1. Constraints on Options . . . . . . . . . . . . . . . 19 4.4.2. DSCP, ECN and TTL . . . . . . . . . . . . . . . . . . 21
4.3. NIC Offloads . . . . . . . . . . . . . . . . . . . . . . 19 4.4.3. Broadcast and Multicast . . . . . . . . . . . . . . . 22
4.4. Inner VLAN Handling . . . . . . . . . . . . . . . . . . . 20 4.4.4. Unidirectional Tunnels . . . . . . . . . . . . . . . 22
5. Interoperability Issues . . . . . . . . . . . . . . . . . . . 20 4.5. Constraints on Protocol Features . . . . . . . . . . . . 23
6. Security Considerations . . . . . . . . . . . . . . . . . . . 21 4.5.1. Constraints on Options . . . . . . . . . . . . . . . 23
6.1. Data Confidentiality . . . . . . . . . . . . . . . . . . 22 4.6. NIC Offloads . . . . . . . . . . . . . . . . . . . . . . 23
6.1.1. Inter-data center traffic . . . . . . . . . . . . . . 22 4.7. Inner VLAN Handling . . . . . . . . . . . . . . . . . . . 24
6.2. Data Integrity . . . . . . . . . . . . . . . . . . . . . 22 5. Interoperability Issues . . . . . . . . . . . . . . . . . . . 24
6.3. Authentication of NVE peers . . . . . . . . . . . . . . . 23 6. Security Considerations . . . . . . . . . . . . . . . . . . . 25
6.4. Multicast/Broadcast . . . . . . . . . . . . . . . . . . . 23 6.1. Data Confidentiality . . . . . . . . . . . . . . . . . . 26
6.5. Control plane communications . . . . . . . . . . . . . . 24 6.1.1. Inter-Data Center Traffic . . . . . . . . . . . . . . 26
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24 6.2. Data Integrity . . . . . . . . . . . . . . . . . . . . . 27
8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 25 6.3. Authentication of NVE peers . . . . . . . . . . . . . . . 27
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 6.4. Options Interpretation by Transit Devices . . . . . . . . 27
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 26 6.5. Multicast/Broadcast . . . . . . . . . . . . . . . . . . . 28
10.1. Normative References . . . . . . . . . . . . . . . . . . 26 6.6. Control Plane Communications . . . . . . . . . . . . . . 28
10.2. Informative References . . . . . . . . . . . . . . . . . 27
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 29 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28
8. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 29
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 30
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 31
10.1. Normative References . . . . . . . . . . . . . . . . . . 31
10.2. Informative References . . . . . . . . . . . . . . . . . 32
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34
1. Introduction 1. Introduction
Networking has long featured a variety of tunneling, tagging, and Networking has long featured a variety of tunneling, tagging, and
other encapsulation mechanisms. However, the advent of network other encapsulation mechanisms. However, the advent of network
virtualization has caused a surge of renewed interest and a virtualization has caused a surge of renewed interest and a
corresponding increase in the introduction of new protocols. The corresponding increase in the introduction of new protocols. The
large number of protocols in this space, ranging all the way from large number of protocols in this space, ranging all the way from
VLANs [IEEE.802.1Q_2014] and MPLS [RFC3031] through the more recent VLANs [IEEE.802.1Q_2014] and MPLS [RFC3031] through the more recent
VXLAN [RFC7348], NVGRE [RFC7637], often leads to questions about the VXLAN [RFC7348], NVGRE [RFC7637], often leads to questions about the
skipping to change at page 4, line 13 skipping to change at page 4, line 18
Furthermore, software and hardware components and controllers all Furthermore, software and hardware components and controllers all
have different advantages and rates of evolution - a fact that should have different advantages and rates of evolution - a fact that should
be viewed as a benefit, not a liability or limitation. This draft be viewed as a benefit, not a liability or limitation. This draft
describes Geneve, a protocol which seeks to avoid these problems by describes Geneve, a protocol which seeks to avoid these problems by
providing a framework for tunneling for network virtualization rather providing a framework for tunneling for network virtualization rather
than being prescriptive about the entire system. than being prescriptive about the entire system.
1.1. Requirements Language 1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
document are to be interpreted as described in [RFC2119]. "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
In this document, these words will appear with that interpretation capitals, as shown here.
only when in ALL CAPS. Lower case uses of these words are not to be
interpreted as carrying RFC-2119 significance.
1.2. Terminology 1.2. Terminology
The NVO3 framework [RFC7365] defines many of the concepts commonly The NVO3 framework [RFC7365] defines many of the concepts commonly
used in network virtualization. In addition, the following terms are used in network virtualization. In addition, the following terms are
specifically meaningful in this document: specifically meaningful in this document:
Checksum offload. An optimization implemented by many NICs which Checksum offload. An optimization implemented by many NICs which
enables computation and verification of upper layer protocol enables computation and verification of upper layer protocol
checksums in hardware on transmit and receive, respectively. This checksums in hardware on transmit and receive, respectively. This
skipping to change at page 4, line 40 skipping to change at page 4, line 43
computed by the protocol stack in software. computed by the protocol stack in software.
Clos network. A technique for composing network fabrics larger than Clos network. A technique for composing network fabrics larger than
a single switch while maintaining non-blocking bandwidth across a single switch while maintaining non-blocking bandwidth across
connection points. ECMP is used to divide traffic across the connection points. ECMP is used to divide traffic across the
multiple links and switches that constitute the fabric. Sometimes multiple links and switches that constitute the fabric. Sometimes
termed "leaf and spine" or "fat tree" topologies. termed "leaf and spine" or "fat tree" topologies.
ECMP. Equal Cost Multipath. A routing mechanism for selecting from ECMP. Equal Cost Multipath. A routing mechanism for selecting from
among multiple best next hop paths by hashing packet headers in order among multiple best next hop paths by hashing packet headers in order
to better utilize network bandwidth while avoiding reordering a to better utilize network bandwidth while avoiding reordering of
single stream. packets within a flow.
Geneve. Generic Network Virtualization Encapsulation. The tunnel Geneve. Generic Network Virtualization Encapsulation. The tunnel
protocol described in this draft. protocol described in this draft.
LRO. Large Receive Offload. The receive-side equivalent function of LRO. Large Receive Offload. The receive-side equivalent function of
LSO, in which multiple protocol segments (primarily TCP) are LSO, in which multiple protocol segments (primarily TCP) are
coalesced into larger data units. coalesced into larger data units.
NIC. Network Interface Card. A NIC could be part of a tunnel NIC. Network Interface Card. A NIC could be part of a tunnel
endpoint or transit device and can either process Geneve packets or endpoint or transit device and can either process Geneve packets or
aid in the processing of Geneve packets. aid in the processing of Geneve packets.
OAM. Operations, Administration, and Management. A suite of tools
used to monitor and troubleshoot network problems.
Transit device. A forwarding element along the path of the tunnel Transit device. A forwarding element along the path of the tunnel
making up part of the Underlay Network. A transit device MAY be making up part of the Underlay Network. A transit device MAY be
capable of understanding the Geneve packet format but does not capable of understanding the Geneve packet format but does not
originate or terminate Geneve packets. originate or terminate Geneve packets.
LSO. Large Segmentation Offload. A function provided by many LSO. Large Segmentation Offload. A function provided by many
commercial NICs that allows data units larger than the MTU to be commercial NICs that allows data units larger than the MTU to be
passed to the NIC to improve performance, the NIC being responsible passed to the NIC to improve performance, the NIC being responsible
for creating smaller segments of size less than or equal to the MTU for creating smaller segments of size less than or equal to the MTU
with correct protocol headers. When referring specifically to TCP/ with correct protocol headers. When referring specifically to TCP/
IP, this feature is often known as TSO (TCP Segmentation Offload). IP, this feature is often known as TSO (TCP Segmentation Offload).
Tunnel endpoint. A component performing encapsulation and Tunnel endpoint. A component performing encapsulation and
decapsulation of packets, such as Ethernet frames or IP datagrams, in decapsulation of packets, such as Ethernet frames or IP datagrams, in
Geneve headers. As the ultimate consumer of any tunnel metadata, Geneve headers. As the ultimate consumer of any tunnel metadata,
endpoints have the highest level of requirements for parsing and tunnel endpoints have the highest level of requirements for parsing
interpreting tunnel headers. Tunnel endpoints may consist of either and interpreting tunnel headers. Tunnel endpoints may consist of
software or hardware implementations or a combination of the two. either software or hardware implementations or a combination of the
Endpoints are frequently a component of an NVE but may also be found two. Tunnel endpoints are frequently a component of an NVE (Network
in middleboxes or other elements making up an NVO3 Network. Virtualization Edge) but may also be found in middleboxes or other
elements making up an NVO3 Network.
VM. Virtual Machine. VM. Virtual Machine.
2. Design Requirements 2. Design Requirements
Geneve is designed to support network virtualization use cases, where Geneve is designed to support network virtualization use cases, where
tunnels are typically established to act as a backplane between the tunnels are typically established to act as a backplane between the
virtual switches residing in hypervisors, physical switches, or virtual switches residing in hypervisors, physical switches, or
middleboxes or other appliances. An arbitrary IP network can be used middleboxes or other appliances. An arbitrary IP network can be used
as an underlay although Clos networks composed using ECMP links are a as an underlay although Clos networks composed using ECMP links are a
skipping to change at page 7, line 50 skipping to change at page 7, line 50
be reasonably handled across platforms along with a graceful be reasonably handled across platforms along with a graceful
mechanism to handle more advanced features in the appropriate mechanism to handle more advanced features in the appropriate
situations. situations.
The use of a variable length header and options in a protocol often The use of a variable length header and options in a protocol often
raises questions about whether it is truly efficiently implementable raises questions about whether it is truly efficiently implementable
in hardware. To answer this question in the context of Geneve, it is in hardware. To answer this question in the context of Geneve, it is
important to first divide "hardware" into two categories: tunnel important to first divide "hardware" into two categories: tunnel
endpoints and transit devices. endpoints and transit devices.
Endpoints must be able to parse the variable header, including any Tunnel endpoints must be able to parse the variable header, including
options, and take action. Since these devices are actively any options, and take action. Since these devices are actively
participating in the protocol, they are the most affected by Geneve. participating in the protocol, they are the most affected by Geneve.
However, as endpoints are the ultimate consumers of the data, However, as tunnel endpoints are the ultimate consumers of the data,
transmitters can tailor their output to the capabilities of the transmitters can tailor their output to the capabilities of the
recipient. As new functionality becomes sufficiently well defined to recipient. As new functionality becomes sufficiently well defined to
add to endpoints, supporting options can be designed using ordering add to tunnel endpoints, supporting options can be designed using
restrictions and other techniques to ease parsing. ordering restrictions and other techniques to ease parsing.
Transit devices MAY be able to interpret the options, however, as Options, if present in the packet, MUST be generated and terminated
non-terminating devices, transit devices do not originate or by tunnel endpoints. Transit devices MAY be able to interpret the
terminate the Geneve packet, hence MUST NOT insert or delete options, options, however, as non-terminating devices, transit devices do not
which is the responsibility of Geneve endpoints. The participation originate or terminate the Geneve packet, hence MUST NOT modify
of transit devices in interpreting options is OPTIONAL. Geneve headers and MUST NOT insert or delete options, which is the
responsibility of tunnel endpoints. The participation of transit
devices in interpreting options is OPTIONAL.
Further, either tunnel endpoints or transit devices MAY use offload Further, either tunnel endpoints or transit devices MAY use offload
capabilities of NICs such as checksum offload to improve the capabilities of NICs such as checksum offload to improve the
performance of Geneve packet processing. The presence of a Geneve performance of Geneve packet processing. The presence of a Geneve
variable length header SHOULD NOT prevent the tunnel endpoints and variable length header SHOULD NOT prevent the tunnel endpoints and
transit devices from using such offload capabilities. transit devices from using such offload capabilities.
2.3. Use of Standard IP Fabrics 2.3. Use of Standard IP Fabrics
IP has clearly cemented its place as the dominant transport mechanism IP has clearly cemented its place as the dominant transport mechanism
skipping to change at page 8, line 41 skipping to change at page 8, line 43
standard switching and routing. standard switching and routing.
In addition, nearly all underlay fabrics are designed to exploit In addition, nearly all underlay fabrics are designed to exploit
parallelism in traffic to spread load across multiple links without parallelism in traffic to spread load across multiple links without
introducing reordering in individual flows. These equal cost introducing reordering in individual flows. These equal cost
multipathing (ECMP) techniques typically involve parsing and hashing multipathing (ECMP) techniques typically involve parsing and hashing
the addresses and port numbers from the packet to select an outgoing the addresses and port numbers from the packet to select an outgoing
link. However, the use of tunnels often results in poor ECMP link. However, the use of tunnels often results in poor ECMP
performance without additional knowledge of the protocol as the performance without additional knowledge of the protocol as the
encapsulated traffic is hidden from the fabric by design and only encapsulated traffic is hidden from the fabric by design and only
endpoint addresses are available for hashing. tunnel endpoint addresses are available for hashing.
Since it is desirable for Geneve to perform well on these existing Since it is desirable for Geneve to perform well on these existing
fabrics, it is necessary for entropy from encapsulated packets to be fabrics, it is necessary for entropy from encapsulated packets to be
exposed in the tunnel header. The most common technique for this is exposed in the tunnel header. The most common technique for this is
to use the UDP source port, which is discussed further in to use the UDP source port, which is discussed further in
Section 3.3. Section 3.3.
3. Geneve Encapsulation Details 3. Geneve Encapsulation Details
The Geneve packet format consists of a compact tunnel header The Geneve packet format consists of a compact tunnel header
skipping to change at page 12, line 49 skipping to change at page 12, line 49
Dest port: IANA has assigned port 6081 as the fixed well-known Dest port: IANA has assigned port 6081 as the fixed well-known
destination port for Geneve. Although the well-known value should destination port for Geneve. Although the well-known value should
be used by default, it is RECOMMENDED that implementations make be used by default, it is RECOMMENDED that implementations make
this configurable. The chosen port is used for identification of this configurable. The chosen port is used for identification of
Geneve packets and MUST NOT be reversed for different ends of a Geneve packets and MUST NOT be reversed for different ends of a
connection as is done with TCP. connection as is done with TCP.
UDP length: The length of the UDP packet including the UDP header. UDP length: The length of the UDP packet including the UDP header.
UDP checksum: The checksum MAY be set to zero on transmit for UDP checksum: In order to protect the Geneve header, options and
packets encapsulated in both IPv4 and IPv6 [RFC6935]. When a payload from potential data corruption, UDP checksum SHOULD be
packet is received with a UDP checksum of zero it MUST be accepted generated as specified in [RFC0768] and [RFC1112] when Geneve is
and decapsulated. If the originating tunnel endpoint optionally encapsulated in IPv4. To protect the IP header, Geneve header,
encapsulates a packet with a non-zero checksum, it MUST be a options and payload from potential data corruption, the UDP
correctly computed UDP checksum. Upon receiving such a packet, checksum MUST be generated by default as specified in [RFC0768]
the egress endpoint MUST validate the checksum. If the checksum and [RFC2460] when Geneve is encapsulated in IPv6. Upon receiving
is not correct, the packet MUST be dropped, otherwise the packet such packets with non-zero UDP checksum, the receiving tunnel
MUST be accepted for decapsulation. It is RECOMMENDED that the endpoints MUST validate the checksum. If the checksum is not
UDP checksum be computed to protect the Geneve header and options correct, the packet MUST be dropped, otherwise the packet MUST be
in situations where the network reliability is not high and the accepted for decapsulation.
packet is not protected by another checksum or CRC.
Under certain conditions, the UDP checksum MAY be set to zero on
transmit for packets encapsulated in both IPv4 and IPv6 [RFC6935].
See Section 4.3 for additional requirements that apply for using
zero UDP checksum with IPv4 and IPv6. Disabling the use of UDP
checksums is an operational consideration that should take into
account the risks and effects of packet corruption.
3.4. Tunnel Header Fields 3.4. Tunnel Header Fields
Ver (2 bits): The current version number is 0. Packets received by Ver (2 bits): The current version number is 0. Packets received by
an endpoint with an unknown version MUST be dropped. Non- a tunnel endpoint with an unknown version MUST be dropped.
terminating devices processing Geneve packets with an unknown Transit devices interpreting Geneve packets with an unknown
version number MUST treat them as UDP packets with an unknown version number MUST treat them as UDP packets with an unknown
payload. payload.
Opt Len (6 bits): The length of the options fields, expressed in Opt Len (6 bits): The length of the options fields, expressed in
four byte multiples, not including the eight byte fixed tunnel four byte multiples, not including the eight byte fixed tunnel
header. This results in a minimum total Geneve header size of 8 header. This results in a minimum total Geneve header size of 8
bytes and a maximum of 260 bytes. The start of the payload bytes and a maximum of 260 bytes. The start of the payload
headers can be found using this offset from the end of the base headers can be found using this offset from the end of the base
Geneve header. Geneve header.
O (1 bit): OAM packet. This packet contains a control message O (1 bit): Control packet. This packet contains a control message.
instead of a data payload. Control messages are sent between Control messages are sent between tunnel endpoints. Tunnel
Geneve endpoints. Endpoints MUST NOT forward the payload and Endpoints MUST NOT forward the payload and transit devices MUST
transit devices MUST NOT attempt to interpret or process it. NOT attempt to interpret it. Since these are infrequent control
Since these are infrequent control messages, it is RECOMMENDED messages, it is RECOMMENDED that tunnel endpoints direct these
that endpoints direct these packets to a high priority control packets to a high priority control queue (for example, to direct
queue (for example, to direct the packet to a general purpose CPU the packet to a general purpose CPU from a forwarding ASIC or to
from a forwarding ASIC or to separate out control traffic on a separate out control traffic on a NIC). Transit devices MUST NOT
NIC). Transit devices MUST NOT alter forwarding behavior on the alter forwarding behavior on the basis of this bit, such as ECMP
basis of this bit, such as ECMP link selection. link selection.
C (1 bit): Critical options present. One or more options has the C (1 bit): Critical options present. One or more options has the
critical bit set (see Section 3.5). If this bit is set then critical bit set (see Section 3.5). If this bit is set then
tunnel endpoints MUST parse the options list to interpret any tunnel endpoints MUST parse the options list to interpret any
critical options. On endpoints where option parsing is not critical options. On tunnel endpoints where option parsing is not
supported the packet MUST be dropped on the basis of the 'C' bit supported the packet MUST be dropped on the basis of the 'C' bit
in the base header. If the bit is not set tunnel endpoints MAY in the base header. If the bit is not set tunnel endpoints MAY
strip all options using 'Opt Len' and forward the decapsulated strip all options using 'Opt Len' and forward the decapsulated
packet. Transit devices MUST NOT drop packets on the basis of packet. Transit devices MUST NOT drop packets on the basis of
this bit. this bit.
The critical bit allows hardware implementations the flexibility The critical bit allows hardware implementations the flexibility
to handle options processing in the hardware fastpath or in the to handle options processing in the hardware fastpath or in the
exception (slow) path without the need to process all the options. exception (slow) path without the need to process all the options.
For example, a critical option such as secure hash to provide For example, a critical option such as secure hash to provide
Geneve header integrity check must be processed by tunnel Geneve header integrity check must be processed by tunnel
endpoints and typically processed in the hardware fastpath. endpoints and typically processed in the hardware fastpath.
Rsvd. (6 bits): Reserved field which MUST be zero on transmission Rsvd. (6 bits): Reserved field, which MUST be zero on transmission
and ignored on receipt. and MUST be ignored on receipt.
Protocol Type (16 bits): The type of the protocol data unit Protocol Type (16 bits): The type of the protocol data unit
appearing after the Geneve header. This follows the EtherType appearing after the Geneve header. This follows the EtherType
[ETYPES] convention with Ethernet itself being represented by the [ETYPES] convention with Ethernet itself being represented by the
value 0x6558. value 0x6558.
Virtual Network Identifier (VNI) (24 bits): An identifier for a Virtual Network Identifier (VNI) (24 bits): An identifier for a
unique element of a virtual network. In many situations this may unique element of a virtual network. In many situations this may
represent an L2 segment, however, the control plane defines the represent an L2 segment, however, the control plane defines the
forwarding semantics of decapsulated packets. The VNI MAY be used forwarding semantics of decapsulated packets. The VNI MAY be used
skipping to change at page 15, line 22 skipping to change at page 15, line 27
implementation may use option types from a variety of sources. In implementation may use option types from a variety of sources. In
addition, IANA will be requested to reserve specific ranges for addition, IANA will be requested to reserve specific ranges for
standardized and experimental options. standardized and experimental options.
Type (8 bits): Type indicating the format of the data contained in Type (8 bits): Type indicating the format of the data contained in
this option. Options are primarily designed to encourage future this option. Options are primarily designed to encourage future
extensibility and innovation and so standardized forms of these extensibility and innovation and so standardized forms of these
options will be defined in a separate document. options will be defined in a separate document.
The high order bit of the option type indicates that this is a The high order bit of the option type indicates that this is a
critical option. If the receiving endpoint does not recognize critical option. If the receiving tunnel endpoint does not
this option and this bit is set then the packet MUST be dropped. recognize this option and this bit is set then the packet MUST be
If the critical bit is set in any option then the 'C' bit in the dropped. If the 'C' bit (critical bit) is set in any option then
Geneve base header MUST also be set. Transit devices MUST NOT the 'C' bit in the Geneve base header MUST also be set. Transit
drop packets on the basis of this bit. The following figure shows devices MUST NOT drop packets on the basis of this bit. The
the location of the 'C' bit in the 'Type' field: following figure shows the location of the 'C' bit in the 'Type'
field:
0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
|C| Type | |C| Type |
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
The requirement to drop a packet with an unknown critical option The requirement to drop a packet with an unknown option with the
applies to the entire tunnel endpoint system and not a particular 'C' bit set applies to the entire tunnel endpoint system and not a
component of the implementation. For example, in a system particular component of the implementation. For example, in a
comprised of a forwarding ASIC and a general purpose CPU, this system comprised of a forwarding ASIC and a general purpose CPU,
does not mean that the packet must be dropped in the ASIC. An this does not mean that the packet must be dropped in the ASIC.
implementation may send the packet to the CPU using a rate-limited An implementation may send the packet to the CPU using a rate-
control channel for slow-path exception handling. limited control channel for slow-path exception handling.
R (3 bits): Option control flags reserved for future use. MUST be R (3 bits): Option control flags reserved for future use. MUST be
zero on transmission and ignored on receipt. zero on transmission and ignored on receipt.
Length (5 bits): Length of the option, expressed in four byte Length (5 bits): Length of the option, expressed in four byte
multiples excluding the option header. The total length of each multiples excluding the option header. The total length of each
option may be between 4 and 128 bytes. A value of 0 in the Length option may be between 4 and 128 bytes. A value of 0 in the Length
field implies an option with only the option header without the field implies an option with only the option header without the
variable option data. Packets in which the total length of all variable option data. Packets in which the total length of all
options is not equal to the 'Opt Len' in the base header are options is not equal to the 'Opt Len' in the base header are
invalid and MUST be silently dropped if received by an endpoint. invalid and MUST be silently dropped if received by a tunnel
endpoint that processes the options.
Variable Option Data: Option data interpreted according to 'Type'. Variable Option Data: Option data interpreted according to 'Type'.
3.5.1. Options Processing 3.5.1. Options Processing
Geneve options are intended to be originated and processed by tunnel Geneve options are intended to be originated and processed by tunnel
endpoints. However, options MAY be interpreted by transit devices endpoints. However, options MAY be interpreted by transit devices
along the tunnel path. Transit devices not processing Geneve headers along the tunnel path. Transit devices not interpreting Geneve
SHOULD process Geneve packets as any other UDP packet and maintain headers (that may or may not include Options) SHOULD handle Geneve
consistent forwarding behavior. packets as any other UDP packet and maintain consistent forwarding
behavior.
In tunnel endpoints, the generation and interpretation of options is In tunnel endpoints, the generation and interpretation of options is
determined by the control plane, which is out of the scope of this determined by the control plane, which is out of the scope of this
document. However, to ensure interoperability between heterogeneous document. However, to ensure interoperability between heterogeneous
devices some requirements are imposed on options and the devices that devices some requirements are imposed on options and the devices that
process them: process them:
o Receiving endpoints MUST drop packets containing unknown options o Receiving tunnel endpoints MUST drop packets containing unknown
with the 'C' bit set in the option type. Conversely, transit options with the 'C' bit set in the option type. Conversely,
devices MUST NOT drop packets as a result of encountering unknown transit devices MUST NOT drop packets as a result of encountering
options, including those with the 'C' bit set. unknown options, including those with the 'C' bit set.
o Some options may be defined in such a way that the position in the o Some options may be defined in such a way that the position in the
option list is significant. Options or their ordering, MUST NOT option list is significant. Options MUST NOT be changed by
be changed by transit devices. transit devices.
o An option MUST NOT affect the parsing or interpretation of any o An option SHOULD NOT be dependent upon any other option in the
packet, i.e., options can be processed independent of one another.
An option MUST NOT affect the parsing or interpretation of any
other option. other option.
When designing a Geneve option, it is important to consider how the When designing a Geneve option, it is important to consider how the
option will evolve in the future. Once an option is defined it is option will evolve in the future. Once an option is defined it is
reasonable to expect that implementations may come to depend on a reasonable to expect that implementations may come to depend on a
specific behavior. As a result, the scope of any future changes must specific behavior. As a result, the scope of any future changes must
be carefully described upfront. be carefully described upfront.
Unexpectedly significant interoperability issues may result from Unexpectedly significant interoperability issues may result from
changing the length of an option that was defined to be a certain changing the length of an option that was defined to be a certain
skipping to change at page 17, line 5 skipping to change at page 17, line 16
over time or for different use cases. This property is part of the over time or for different use cases. This property is part of the
definition of the option and conveyed by the 'Type'. For fixed definition of the option and conveyed by the 'Type'. For fixed
length options, some implementations may choose to ignore the length length options, some implementations may choose to ignore the length
field in the option header and instead parse based on the well known field in the option header and instead parse based on the well known
length associated with the type. In this case, redefining the length length associated with the type. In this case, redefining the length
will impact not only parsing of the option in question but also any will impact not only parsing of the option in question but also any
options that follow. Therefore, options that are defined to be fixed options that follow. Therefore, options that are defined to be fixed
length in size MUST NOT be redefined to a different length. Instead, length in size MUST NOT be redefined to a different length. Instead,
a new 'Type' should be allocated. a new 'Type' should be allocated.
Options may be processed by NIC hardware utilizing offloads (e.g.
LSO and LRO) as described in Section 4.6. Careful consideration
should be given to how the offload capabilities outlined in
Section 4.6 impact an option's design.
4. Implementation and Deployment Considerations 4. Implementation and Deployment Considerations
4.1. Encapsulation of Geneve in IP 4.1. Applicability Statement
Geneve is a network virtualization overlay encapsulation protocol
designed to establish tunnels between NVEs over an existing IP
network. It is intended for use in public or private data center
environments, for deploying multi-tenant overlay networks over an
existing IP underlay network.
Geneve is an UDP based encapsulation protocol transported over
existing IPv4 and IPv6 networks. Hence, as an UDP based protocol,
Geneve needs to meet the UDP usage guidelines as specified in
[RFC8085]. The applicability of these guidelines are dependent on
the underlay IP network and the nature of Geneve payload protocol
(example TCP/IP, IP/Ethernet).
[RFC8085] outlines two applicability scenarios for UDP applications,
1) general Internet and 2) controlled environment. The controlled
environment means a single administrative domain or adjacent set of
cooperating domains. A network in a controlled environment can be
managed to operate under certain conditions whereas in general
Internet this cannot be done. Hence requirements for a tunnel
protocol operating under a controlled environment can be less
restrictive than the requirements of general internet.
Geneve is intended to be deployed in a data center network
environment operated by a single operator or adjacent set of
cooperating network operators that fits with the definition of
controlled environments in [RFC8085].
For the purpose of this document, a traffic-managed controlled
environment (TMCE) is defined as an IP network that is traffic-
engineered and/or otherwise managed (e.g., via use of traffic rate
limiters) to avoid congestion. The concept of TMCE is outlined in
[RFC8086]. Significant portions of text in Section 4.1 through
Section 4.3 are based on [RFC8086] as applicable to Geneve.
It is the responsibility of the operator to ensure that the
guidelines/requirements in this section are followed as applicable to
their Geneve deployment(s).
4.2. Congestion Control Functionality
Geneve does not natively provide congestion control functionality and
relies on the payload protocol traffic for congestion control. As
such Geneve MUST be used with congestion controlled traffic or within
a network that is traffic managed to avoid congestion (TMCE). An
operator of a traffic managed network (TMCE) may avoid congestion by
careful provisioning of their networks, rate-limiting of user data
traffic and traffic engineering according to path capacity.
4.3. UDP Checksum
In order to provide integrity of Geneve headers, options and payload,
for example to avoid mis-delivery of payload to different tenant
systems in case of data corruption, outer UDP checksum SHOULD be used
with Geneve when transported over IPv4. An operator MAY choose to
disable UDP checksum and use zero checksum if Geneve packet integrity
is provided by other data integrity mechanisms such as IPsec or
additional checksums or if one of the conditions in Section 4.3.1 a,
b, c are met.
By default, UDP checksum MUST be used when Geneve is transported over
IPv6. A tunnel endpoint MAY be configured for use with zero UDP
checksum if additional requirements in Section 4.3.1 are met.
4.3.1. UDP Zero Checksum Handling with IPv6
When Geneve is used over IPv6, UDP checksum is used to protect IPv6
headers, UDP headers and Geneve headers, options and payload from
potential data corruption. As such by default Geneve MUST use UDP
checksum when transported over IPv6. An operator MAY choose to
configure to operate with zero UDP checksum if operating in a traffic
managed controlled environment as stated in Section 4.1 if one of the
following conditions are met.
a. It is known that the packet corruption is exceptionally unlikely
(perhaps based on knowledge of equipment types in their underlay
network) and the operator is willing to take a risk of undetected
packet corruption
b. It is judged through observational measurements (perhaps through
historic or current traffic flows that use non zero checksum)
that the level of packet corruption is tolerably low and where
the operator is willing to take the risk of undetected
corruption.
c. Geneve payload is carrying applications that are tolerant of
misdelivered or corrupted packets (perhaps through higher layer
checksum validation and/or reliability through retransmission)
In addition Geneve tunnel implementations using Zero UDP checksum
MUST meet the following requirements:
1. Use of UDP checksum over IPv6 MUST be the default configuration
for all Geneve tunnels.
2. If Geneve is used with zero UDP checksum over IPv6 then such
tunnel endpoint implementation MUST meet all the requirements
specified in section 4 of [RFC6936] and requirements 1 as
specified in section 5 of [RFC6936].
3. The Geneve tunnel endpoint that decapsulates the tunnel SHOULD
check the source and destination IPv6 addresses are valid for the
Geneve tunnel that is configured to receive Zero UDP checksum and
discard other packets for which such check fails.
4. The Geneve tunnel endpoint that encapsulates the tunnel MAY use
different IPv6 source addresses for each Geneve tunnel that uses
Zero UDP checksum mode in order to strengthen the decapsulator's
check of the IPv6 source address (i.e the same IPv6 source
address is not to be used with more than one IPv6 destination
address, irrespective of whether that destination address is a
unicast or multicast address). When this is not possible, it is
RECOMMENDED to use each source address for as few Geneve tunnels
that use zero UDP checksum as is feasible.
5. Measures SHOULD be taken to prevent Geneve traffic over IPv6 with
zero UDP checksum from escaping into the general Internet.
Examples of such measures include employing packet filters at the
Gateways or edge of Geneve network and/or keeping logical or
physical separation of Geneve network from networks carrying
General Internet.
The above requirements do not change either the requirements
specified in [RFC2460] as modified by [RFC6935] or the requirements
specified in [RFC6936].
The requirement to check the source IPv6 address in addition to the
destination IPv6 address, plus the recommendation against reuse of
source IPv6 addresses among Geneve tunnels collectively provide some
mitigation for the absence of UDP checksum coverage of the IPv6
header. A traffic-managed controlled environment that satisfies at
least one of three conditions listed at the beginning of this section
provides additional assurance.
Editorial Note (The following paragraph to be removed by the RFC
Editor before publication)
It was discussed during TSVART early review if the level of
requirement for using different IPv6 source addresses for different
tunnel destinations would need to be "MAY" or "SHOULD". The
discussion concluded that it was appropriate to keep this as "MAY",
since it was considered not realistic for control planes having to
maintain a high level of state on a per tunnel destination basis. In
addition, the text above provides sufficient guidance to operators
and implementors on possible mitigations.
4.4. Encapsulation of Geneve in IP
As an IP-based tunnel protocol, Geneve shares many properties and As an IP-based tunnel protocol, Geneve shares many properties and
techniques with existing protocols. The application of some of these techniques with existing protocols. The application of some of these
are described in further detail, although in general most concepts are described in further detail, although in general most concepts
applicable to the IP layer or to IP tunnels generally also function applicable to the IP layer or to IP tunnels generally also function
in the context of Geneve. in the context of Geneve.
4.1.1. IP Fragmentation 4.4.1. IP Fragmentation
To prevent fragmentation and maximize performance, the best practice
when using Geneve is to ensure that the MTU of the physical network
is greater than or equal to the MTU of the encapsulated network plus
tunnel headers. Manual or upper layer (such as TCP MSS clamping)
configuration can be used to ensure that fragmentation never takes
place, however, in some situations this may not be feasible.
It is strongly RECOMMENDED that Path MTU Discovery ([RFC1191], It is strongly RECOMMENDED that Path MTU Discovery ([RFC1191],
[RFC1981]) be used by setting the DF bit in the IP header when Geneve [RFC8201]) be used by setting the DF bit in the IP header when Geneve
packets are transmitted over IPv4 (this is the default with IPv6). packets are transmitted over IPv4 (this is the default with IPv6).
The use of Path MTU Discovery on the transit network provides the The use of Path MTU Discovery on the transit network provides the
encapsulating endpoint with soft-state about the link that it may use encapsulating tunnel endpoint with soft-state about the link that it
to prevent or minimize fragmentation depending on its role in the may use to prevent or minimize fragmentation depending on its role in
virtualized network. For example, recommendations/guidance for the virtualized network. The NVE control plane MAY use configuration
handling fragmenation in similar overlay encapsulation services like mechanism or path discovery information to maintain the MTU size of
PWE3 are provided in section 5.3 of [RFC3985]. the tunnel link(s) associated with the tunnel endpoint, so if a
tenant system sends large packets that when encapsulated exceed the
MTU size of the tunnel link, the tunnel endpoint can discard such
packets and send exception messages to the tenant system(s). If the
tunnel endpoint is associated with a routing or forwarding function
and/or has the capability to send ICMP messages, the encapsulating
tunnel endpoint MAY send ICMP fragmentation needed [RFC0792] or
Packet Too Big [RFC4443] messages to the tenant system(s). For
example, recommendations/guidance for handling fragmentation in
similar overlay encapsulation services like PWE3 are provided in
section 5.3 of [RFC3985].
Note that some implementations may not be capable of supporting Note that some implementations may not be capable of supporting
fragmentation or other less common features of the IP header, such as fragmentation or other less common features of the IP header, such as
options and extension headers. options and extension headers. For example, some of the issues
associated with MTU size and fragmentation in IP tunneling and use of
ICMP messages is outlined in section 4.2 of
[I-D.ietf-intarea-tunnels].
4.1.2. DSCP and ECN Editorial Note (The following paragraph to be removed by the RFC
Editor before publication)
It was discussed during TSVART early review if the level of
requirement for maintaining tunnel MTU at the ingress has to be "MAY"
or "SHOULD". The discussion concluded that it was appropriate to
leave this as "MAY", considering the high level of state to be
maintained.
4.4.2. DSCP, ECN and TTL
When encapsulating IP (including over Ethernet) packets in Geneve, When encapsulating IP (including over Ethernet) packets in Geneve,
there are several considerations for propagating DSCP and ECN bits there are several considerations for propagating DSCP and ECN bits
from the inner header to the tunnel on transmission and the reverse from the inner header to the tunnel on transmission and the reverse
on reception. on reception.
[RFC2983] provides guidance for mapping DSCP between inner and outer [RFC2983] provides guidance for mapping DSCP between inner and outer
IP headers. Network virtualization is typically more closely aligned IP headers. Network virtualization is typically more closely aligned
with the Pipe model described, where the DSCP value on the tunnel with the Pipe model described, where the DSCP value on the tunnel
header is set based on a policy (which may be a fixed value, one header is set based on a policy (which may be a fixed value, one
skipping to change at page 18, line 13 skipping to change at page 21, line 50
and egress) may also apply, such as the ability to remark the inner and egress) may also apply, such as the ability to remark the inner
header on tunnel egress based on transit marking. However, the header on tunnel egress based on transit marking. However, the
Uniform model is not conceptually consistent with network Uniform model is not conceptually consistent with network
virtualization, which seeks to provide strong isolation between virtualization, which seeks to provide strong isolation between
encapsulated traffic and the physical network. encapsulated traffic and the physical network.
[RFC6040] describes the mechanism for exposing ECN capabilities on IP [RFC6040] describes the mechanism for exposing ECN capabilities on IP
tunnels and propagating congestion markers to the inner packets. tunnels and propagating congestion markers to the inner packets.
This behavior MUST be followed for IP packets encapsulated in Geneve. This behavior MUST be followed for IP packets encapsulated in Geneve.
4.1.3. Broadcast and Multicast Though Uniform or Pipe models could be used for TTL (or Hop Limit in
case of IPv6) handling when tunneling IP packets, Pipe model is more
aligned with network virtualization. [RFC2003] provides guidance on
handling TTL between inner IP header and outer IP tunnels; this model
is more aligned with the Pipe model and is recommended for use with
Geneve for network virtualization applications.
4.4.3. Broadcast and Multicast
Geneve tunnels may either be point-to-point unicast between two Geneve tunnels may either be point-to-point unicast between two
endpoints or may utilize broadcast or multicast addressing. It is tunnel endpoints or may utilize broadcast or multicast addressing.
not required that inner and outer addressing match in this respect. It is not required that inner and outer addressing match in this
For example, in physical networks that do not support multicast, respect. For example, in physical networks that do not support
encapsulated multicast traffic may be replicated into multiple multicast, encapsulated multicast traffic may be replicated into
unicast tunnels or forwarded by policy to a unicast location multiple unicast tunnels or forwarded by policy to a unicast location
(possibly to be replicated there). (possibly to be replicated there).
With physical networks that do support multicast it may be desirable With physical networks that do support multicast it may be desirable
to use this capability to take advantage of hardware replication for to use this capability to take advantage of hardware replication for
encapsulated packets. In this case, multicast addresses may be encapsulated packets. In this case, multicast addresses may be
allocated in the physical network corresponding to tenants, allocated in the physical network corresponding to tenants,
encapsulated multicast groups, or some other factor. The allocation encapsulated multicast groups, or some other factor. The allocation
of these groups is a component of the control plane and therefore of these groups is a component of the control plane and therefore
outside of the scope of this document. When physical multicast is in outside of the scope of this document. When physical multicast is in
use, the 'C' bit in the Geneve header may be used with groups of use, the 'C' bit in the Geneve header may be used with groups of
devices with heterogeneous capabilities as each device can interpret devices with heterogeneous capabilities as each device can interpret
only the options that are significant to it if they are not critical. only the options that are significant to it if they are not critical.
4.1.4. Unidirectional Tunnels In addition, [RFC8293] provides examples of various mechanisms that
can be used for multicast handling in network virtualization overlay
networks.
4.4.4. Unidirectional Tunnels
Generally speaking, a Geneve tunnel is a unidirectional concept. IP Generally speaking, a Geneve tunnel is a unidirectional concept. IP
is not a connection oriented protocol and it is possible for two is not a connection oriented protocol and it is possible for two
endpoints to communicate with each other using different paths or to tunnel endpoints to communicate with each other using different paths
have one side not transmit anything at all. As Geneve is an IP-based or to have one side not transmit anything at all. As Geneve is an
protocol, the tunnel layer inherits these same characteristics. IP-based protocol, the tunnel layer inherits these same
characteristics.
It is possible for a tunnel to encapsulate a protocol, such as TCP, It is possible for a tunnel to encapsulate a protocol, such as TCP,
which is connection oriented and maintains session state at that which is connection oriented and maintains session state at that
layer. In addition, implementations MAY model Geneve tunnels as layer. In addition, implementations MAY model Geneve tunnels as
connected, bidirectional links, such as to provide the abstraction of connected, bidirectional links, such as to provide the abstraction of
a virtual port. In both of these cases, bidirectionality of the a virtual port. In both of these cases, bidirectionality of the
tunnel is handled at a higher layer and does not affect the operation tunnel is handled at a higher layer and does not affect the operation
of Geneve itself. of Geneve itself.
4.2. Constraints on Protocol Features 4.5. Constraints on Protocol Features
Geneve is intended to be flexible to a wide range of current and Geneve is intended to be flexible to a wide range of current and
future applications. As a result, certain constraints may be placed future applications. As a result, certain constraints may be placed
on the use of metadata or other aspects of the protocol in order to on the use of metadata or other aspects of the protocol in order to
optimize for a particular use case. For example, some applications optimize for a particular use case. For example, some applications
may limit the types of options which are supported or enforce a may limit the types of options which are supported or enforce a
maximum number or length of options. Other applications may only maximum number or length of options. Other applications may only
handle certain encapsulated payload types, such as Ethernet or IP. handle certain encapsulated payload types, such as Ethernet or IP.
This could be either globally throughout the system or, for example, This could be either globally throughout the system or, for example,
restricted to certain classes of devices or network paths. restricted to certain classes of devices or network paths.
These constraints may be communicated to tunnel endpoints either These constraints may be communicated to tunnel endpoints either
explicitly through a control plane or implicitly by the nature of the explicitly through a control plane or implicitly by the nature of the
application. As Geneve is defined as a data plane protocol that is application. As Geneve is defined as a data plane protocol that is
control plane agnostic, the exact mechanism is not defined in this control plane agnostic, the exact mechanism is not defined in this
document. document.
4.2.1. Constraints on Options 4.5.1. Constraints on Options
While Geneve options are more flexible, a control plane may restrict While Geneve options are more flexible, a control plane may restrict
the number of option TLVs as well as the order and size of the TLVs, the number of option TLVs as well as the order and size of the TLVs,
between tunnel endpoints, to make it simpler for a data plane between tunnel endpoints, to make it simpler for a data plane
implementation in software or hardware to handle implementation in software or hardware to handle
[I-D.ietf-nvo3-encap]. For example, there may be some critical [I-D.ietf-nvo3-encap]. For example, there may be some critical
information such as a secure hash that must be processed in a certain information such as a secure hash that must be processed in a certain
order to provide lowest latency. order to provide lowest latency.
A control plane may negotiate a subset of option TLVs and certain TLV A control plane may negotiate a subset of option TLVs and certain TLV
ordering, as well may limit the total number of option TLVs present ordering, as well may limit the total number of option TLVs present
in the packet, for example, to accommodate hardware capable of in the packet, for example, to accommodate hardware capable of
processing fewer options [I-D.ietf-nvo3-encap]. Hence, a control processing fewer options [I-D.ietf-nvo3-encap]. Hence, a control
plane needs to have the ability to describe the supported TLVs subset plane needs to have the ability to describe the supported TLVs subset
and their order to the tunnel end points. In the absence of a and their order to the tunnel endpoints. In the absence of a control
control plane, alternative configuration mechanisms may be used for plane, alternative configuration mechanisms may be used for this
this purpose. The exact mechanism is not defined in this document. purpose. The exact mechanism is not defined in this document.
4.3. NIC Offloads 4.6. NIC Offloads
Modern NICs currently provide a variety of offloads to enable the Modern NICs currently provide a variety of offloads to enable the
efficient processing of packets. The implementation of many of these efficient processing of packets. The implementation of many of these
offloads requires only that the encapsulated packet be easily parsed offloads requires only that the encapsulated packet be easily parsed
(for example, checksum offload). However, optimizations such as LSO (for example, checksum offload). However, optimizations such as LSO
and LRO involve some processing of the options themselves since they and LRO involve some processing of the options themselves since they
must be replicated/merged across multiple packets. In these must be replicated/merged across multiple packets. In these
situations, it is desirable to not require changes to the offload situations, it is desirable to not require changes to the offload
logic to handle the introduction of new options. To enable this, logic to handle the introduction of new options. To enable this,
some constraints are placed on the definitions of options to allow some constraints are placed on the definitions of options to allow
skipping to change at page 20, line 20 skipping to change at page 24, line 20
override this rule and specify different behavior in supporting override this rule and specify different behavior in supporting
devices. Conversely, when performing LRO, a NIC MAY assume that a devices. Conversely, when performing LRO, a NIC MAY assume that a
binary comparison of the options (including unknown options) is binary comparison of the options (including unknown options) is
sufficient to ensure equality and MAY merge packets with equal sufficient to ensure equality and MAY merge packets with equal
Geneve headers. Geneve headers.
o Options MUST NOT be reordered during the course of offload o Options MUST NOT be reordered during the course of offload
processing, including when merging packets for the purpose of LRO. processing, including when merging packets for the purpose of LRO.
o NICs performing offloads MUST NOT drop packets with unknown o NICs performing offloads MUST NOT drop packets with unknown
options, including those marked as critical. options, including those marked as critical, unless explicitly
configured.
There is no requirement that a given implementation of Geneve employ There is no requirement that a given implementation of Geneve employ
the offloads listed as examples above. However, as these offloads the offloads listed as examples above. However, as these offloads
are currently widely deployed in commercially available NICs, the are currently widely deployed in commercially available NICs, the
rules described here are intended to enable efficient handling of rules described here are intended to enable efficient handling of
current and future options across a variety of devices. current and future options across a variety of devices.
4.4. Inner VLAN Handling 4.7. Inner VLAN Handling
Geneve is capable of encapsulating a wide range of protocols and Geneve is capable of encapsulating a wide range of protocols and
therefore a given implementation is likely to support only a small therefore a given implementation is likely to support only a small
subset of the possibilities. However, as Ethernet is expected to be subset of the possibilities. However, as Ethernet is expected to be
widely deployed, it is useful to describe the behavior of VLANs widely deployed, it is useful to describe the behavior of VLANs
inside encapsulated Ethernet frames. inside encapsulated Ethernet frames.
As with any protocol, support for inner VLAN headers is OPTIONAL. In As with any protocol, support for inner VLAN headers is OPTIONAL. In
many cases, the use of encapsulated VLANs may be disallowed due to many cases, the use of encapsulated VLANs may be disallowed due to
security or implementation considerations. However, in other cases security or implementation considerations. However, in other cases
trunking of VLAN frames across a Geneve tunnel can prove useful. As trunking of VLAN frames across a Geneve tunnel can prove useful. As
a result, the processing of inner VLAN tags upon ingress or egress a result, the processing of inner VLAN tags upon ingress or egress
from a tunnel endpoint is based upon the configuration of the from a tunnel endpoint is based upon the configuration of the tunnel
endpoint and/or control plane and not explicitly defined as part of endpoint and/or control plane and not explicitly defined as part of
the data format. the data format.
5. Interoperability Issues 5. Interoperability Issues
Viewed exclusively from the data plane, Geneve does not introduce any Viewed exclusively from the data plane, Geneve does not introduce any
interoperability issues as it appears to most devices as UDP packets. interoperability issues as it appears to most devices as UDP packets.
However, as there are already a number of tunnel protocols deployed However, as there are already a number of tunnel protocols deployed
in network virtualization environments, there is a practical question in network virtualization environments, there is a practical question
of transition and coexistence. of transition and coexistence.
Since Geneve is a superset of the functionality of the most common Since Geneve is a superset of the functionality of the most common
protocols used for network virtualization (VXLAN, NVGRE ) it should protocols used for network virtualization (VXLAN,NVGRE) it should be
be straightforward to port an existing control plane to run on top of straightforward to port an existing control plane to run on top of it
it with minimal effort. With both the old and new packet formats with minimal effort. With both the old and new packet formats
supporting the same set of capabilities, there is no need for a hard supporting the same set of capabilities, there is no need for a hard
transition - endpoints directly communicating with each other use any transition - tunnel endpoints directly communicating with each other
common protocol, which may be different even within a single overall use any common protocol, which may be different even within a single
system. As transit devices are primarily forwarding packets on the overall system. As transit devices are primarily forwarding packets
basis of the IP header, all protocols appear similar and these on the basis of the IP header, all protocols appear similar and these
devices do not introduce additional interoperability concerns. devices do not introduce additional interoperability concerns.
To assist with this transition, it is strongly suggested that To assist with this transition, it is strongly suggested that
implementations support simultaneous operation of both Geneve and implementations support simultaneous operation of both Geneve and
existing tunnel protocols as it is expected to be common for a single existing tunnel protocols as it is expected to be common for a single
node to communicate with a mixture of other nodes. Eventually, older node to communicate with a mixture of other nodes. Eventually, older
protocols may be phased out as they are no longer in use. protocols may be phased out as they are no longer in use.
6. Security Considerations 6. Security Considerations
As encapsulated within an UDP/IP packet, Geneve does not have any As encapsulated within an UDP/IP packet, Geneve does not have any
inherent security mechanisms. As a result, an attacker with access inherent security mechanisms. As a result, an attacker with access
to the underlay network transporting the IP packets has the ability to the underlay network transporting the IP packets has the ability
to snoop or inject packets. Legitimate but malicious tunnel to snoop or inject packets. Compromised tunnel endpoints may also
endpoints may also spoof identifiers in the tunnel header to gain spoof identifiers in the tunnel header to gain access to networks
access to networks owned by other tenants. owned by other tenants.
Within a particular security domain, such as a data center operated Within a particular security domain, such as a data center operated
by a single service provider, the most common and highest performing by a single service provider, the most common and highest performing
security mechanism is isolation of trusted components. Tunnel security mechanism is isolation of trusted components. Tunnel
traffic can be carried over a separate VLAN and filtered at any traffic can be carried over a separate VLAN and filtered at any
untrusted boundaries. In addition, tunnel endpoints should only be untrusted boundaries. In addition, tunnel endpoints should only be
operated in environments controlled by the service provider, such as operated in environments controlled by the service provider, such as
the hypervisor itself rather than within a customer VM. the hypervisor itself rather than within a customer VM.
When crossing an untrusted link, such as the public Internet, IPsec When crossing an untrusted link, such as the public Internet, IPsec
[RFC4301] may be used to provide authentication and/or encryption of [RFC4301] may be used to provide authentication and/or encryption of
the IP packets formed as part of Geneve encapsulation. the IP packets formed as part of Geneve encapsulation.
Geneve does not otherwise affect the security of the encapsulated Geneve does not otherwise affect the security of the encapsulated
packets. As per the guidelines of BCP72 [RFC3552], the following packets. As per the guidelines of BCP 72 [RFC3552], the following
sections describe potential security risks that may be applicable to sections describe potential security risks that may be applicable to
Geneve deployments and approaches to mitigate such risks. It is also Geneve deployments and approaches to mitigate such risks. It is also
noted that not all such risks are applicable to all Geneve deployment noted that not all such risks are applicable to all Geneve deployment
scenarios, i.e., only a subset may be applicable to certain scenarios, i.e., only a subset may be applicable to certain
deployments. So an operator has to make an assessment based on their deployments. So an operator has to make an assessment based on their
network environment and determine the risks that are applicable to network environment and determine the risks that are applicable to
their specific environment and use appropriate mitigation approaches their specific environment and use appropriate mitigation approaches
as applicable. as applicable.
6.1. Data Confidentiality 6.1. Data Confidentiality
Geneve is a network virtualization overlay encapsulation protocol Geneve is a network virtualization overlay encapsulation protocol
designed to establish tunnels between network virtualization end designed to establish tunnels between NVEs over an existing IP
points (NVE) over an existing IP network. It can be used to deploy network. It can be used to deploy multi-tenant overlay networks over
multi-tenant overlay networks over an existing IP underlay network in an existing IP underlay network in a public or private data center.
a public or private data center. The overlay service is typically The overlay service is typically provided by a service provider, for
provided by a service provider, for example a cloud services provider example a cloud services provider or a private data center operator,
or a private data center operator. Due to the nature of multi- this may or not may be the same provider as an underlay service
tenancy in such environments, a tenant system may expect data provider. Due to the nature of multi-tenancy in such environments, a
confidentiality to ensure its packet data is not tampered with tenant system may expect data confidentiality to ensure its packet
(active attack) in transit or a target of unauthorized monitoring data is not tampered with (active attack) in transit or a target of
(passive attack). A tenant may expect the overlay service provider unauthorized monitoring (passive attack). A tenant may expect the
to provide data confidentiality as part of the service or a tenant overlay service provider to provide data confidentiality as part of
may bring its own data confidentiality mechanisms like IPsec or TLS the service or a tenant may bring its own data confidentiality
to protect the data end to end between its tenant systems. mechanisms like IPsec or TLS to protect the data end to end between
its tenant systems.
If an operator determines data confidentiality is necessary in their If an operator determines data confidentiality is necessary in their
environment based on their risk analysis, for example as in multi- environment based on their risk analysis, for example as in multi-
tenant environments, then an encryption mechanism SHOULD be used to tenant environments, then an encryption mechanism SHOULD be used to
encrypt the tenant data end to end between the NVEs. The NVEs may encrypt the tenant data end to end between the NVEs. The NVEs may
use existing well established encryption mechanisms such as IPsec, use existing well established encryption mechanisms such as IPsec,
DTLs, etc., The operator may choose not to enable the encryption if, DTLS, etc.
for example, the packet data is already encrypted by the tenant
system.
6.1.1. Inter-data center traffic 6.1.1. Inter-Data Center Traffic
A tenant system in a customer premises (private data center) may want A tenant system in a customer premises (private data center) may want
to connect to tenant systems on their tenant overlay network in a to connect to tenant systems on their tenant overlay network in a
public cloud data center or a tenant may want to have its tenant public cloud data center or a tenant may want to have its tenant
systems located in multiple geographically separated data centers for systems located in multiple geographically separated data centers for
high availability. Geneve data traffic between tenant systems across high availability. Geneve data traffic between tenant systems across
such separated networks should be protected from threats when such separated networks should be protected from threats when
traversing public networks. Any Geneve overlay data leaving the data traversing public networks. Any Geneve overlay data leaving the data
center network beyond the operator's security domain, for example center network beyond the operator's security domain SHOULD be
over the public Internet, SHOULD be secured by encryption mechanisms secured by encryption mechanisms such as IPsec or other VPN
such as IPsec or other VPN mechanisms to protect the communications mechanisms to protect the communications between the NVEs when they
between the NVEs when they are geographically separated over are geographically separated over untrusted network links.
untrusted network links. Implementation of specific data protection Specification of data protection mechanisms employed between data
mechanisms employed between data centers is beyond the scope of this centers is beyond the scope of this document.
document.
6.2. Data Integrity 6.2. Data Integrity
Geneve encapsulation is used between NVEs to establish overlay Geneve encapsulation is used between NVEs to establish overlay
tunnels over an existing IP underlay network. In a multi-tenant data tunnels over an existing IP underlay network. In a multi-tenant data
center, a rogue or compromised tenant system may try to launch a center, a rogue or compromised tenant system may try to launch a
passive attack such as monitoring the traffic of other tenants, or an passive attack such as monitoring the traffic of other tenants, or an
active attack such as spoofing or trying to inject unauthorized active attack such as trying to inject unauthorized Geneve
Geneve encapsulated traffic into the network. To prevent such encapsulated traffic such as spoofing, replay, etc., into the
attacks, an NVE MUST not propagate Geneve packets beyond the NVE to network. To prevent such attacks, an NVE MUST NOT propagate Geneve
tenant systems and SHOULD employ packet filtering mechanisms so as packets beyond the NVE to tenant systems and SHOULD employ packet
not to forward unauthorized traffic between TSs in different tenant filtering mechanisms so as not to forward unauthorized traffic
networks. between TSs in different tenant networks.
A compromised network node or a transit device within a data center A compromised network node or a transit device within a data center
may launch an active attack trying to tamper with the Geneve packet may launch an active attack trying to tamper with the Geneve packet
data between NVEs. Malicious tampering of Geneve header fields may data between NVEs. Malicious tampering of Geneve header fields may
cause the packet from one tenant to be forwarded to a different cause the packet from one tenant to be forwarded to a different
tenant network. If an operator determines the possibility of such tenant network. If an operator determines the possibility of such
threat in their environment, the operator may choose to employ data threat in their environment, the operator may choose to employ data
integrity mechanisms between NVEs. In order to prevent such risks, a integrity mechanisms between NVEs. In order to prevent such risks, a
data integrity mechanism SHOULD be used in such environments to data integrity mechanism SHOULD be used in such environments to
protect the integrity of Geneve packets including packet headers, protect the integrity of Geneve packets including packet headers,
options and payload on communications between NVE pairs. A options and payload on communications between NVE pairs. A
cryptographic data protection mechanism such as IPsec may be used to cryptographic data protection mechanism such as IPsec may be used to
provide data integrity protection. A data center operator may choose provide data integrity protection. A data center operator may choose
to deploy any other data integrity mechanisms as applicable and to deploy any other data integrity mechanisms as applicable and
supported in their underlay networks. supported in their underlay networks.
Geneve supports Geneve Options, so an operator may choose to use a
Geneve option TLV to provide a cryptographic data protection
mechanism, to verify the data integrity of the Geneve header, Geneve
options or the entire Geneve packet including the payload.
Implementation of such a mechanism is beyond the scope of this
document.
6.3. Authentication of NVE peers 6.3. Authentication of NVE peers
A rogue network device or a compromised NVE in a data center A rogue network device or a compromised NVE in a data center
environment might be able to spoof Geneve packets as if it came from environment might be able to spoof Geneve packets as if it came from
a legitimate NVE. In order to mitigate such a risk, an operator a legitimate NVE. In order to mitigate such a risk, an operator
SHOULD use an Authentication mechanism, such as IPsec to ensure that SHOULD use an authentication mechanism, such as IPsec to ensure that
the Geneve packet originated from the intended NVE peer, in the Geneve packet originated from the intended NVE peer, in
environments where the operator determines spoofing or rogue devices environments where the operator determines spoofing or rogue devices
is a potential threat. Other simpler source checks such as ingress is a potential threat. Other simpler source checks such as ingress
filtering for VLAN/MAC/IP address, reverse path forwarding checks, filtering for VLAN/MAC/IP address, reverse path forwarding checks,
etc., may be used in certain trusted environments to ensure Geneve etc., may be used in certain trusted environments to ensure Geneve
packets originated from the intended NVE peer. packets originated from the intended NVE peer.
6.4. Multicast/Broadcast 6.4. Options Interpretation by Transit Devices
Options, if present in the packet, are generated and terminated by
tunnel endpoints. As indicated in Section 2.2.1, transit devices may
interpret the options. However, if the packet is protected by tunnel
endpoint to tunnel endpoint encryption, for example through IPsec,
transit devices will not have visibility into the Geneve header or
options in the packet. In cases where options are interpreted by
transit devices, the operator MUST ensure that transit devices are
trusted and not compromised. Implementation of a mechanism to ensure
this trust is beyond the scope of this document.
6.5. Multicast/Broadcast
In typical data center networks where IP multicasting is not In typical data center networks where IP multicasting is not
supported in the underlay network, multicasting may be supported supported in the underlay network, multicasting may be supported
using multiple unicast tunnels. The same security requirements as using multiple unicast tunnels. The same security requirements as
described in the above sections can be used to protect Geneve described in the above sections can be used to protect Geneve
communications between NVE peers. If IP multicasting is supported in communications between NVE peers. If IP multicasting is supported in
the underlay network and the operator chooses to use it for multicast the underlay network and the operator chooses to use it for multicast
traffic among Geneve endpoints, then the operator in such traffic among tunnel endpoints, then the operator in such
environments may use data protection mechanisms such as IPsec with environments may use data protection mechanisms such as IPsec with
Multicast extensions [RFC5374] to protect multicast traffic among Multicast extensions [RFC5374] to protect multicast traffic among
Geneve NVE groups. Geneve NVE groups.
6.5. Control plane communications 6.6. Control Plane Communications
A Network Virtualization Authority (NVA) as outlined in [RFC8014] may A Network Virtualization Authority (NVA) as outlined in [RFC8014] may
be used as a control plane for configuring and managing the Geneve be used as a control plane for configuring and managing the Geneve
NVEs. The data center operator is expected to use security NVEs. The data center operator is expected to use security
mechanisms to protect the communications between the NVA to NVEs and mechanisms to protect the communications between the NVA to NVEs and
use authentication mechanisms to detect any rogue or compromised NVEs use authentication mechanisms to detect any rogue or compromised NVEs
within their administrative domain. Data protection mechanisms for within their administrative domain. Data protection mechanisms for
control plane communication or authentication mechanisms between the control plane communication or authentication mechanisms between the
NVA and the NVEs is beyond the scope of this document. NVA and the NVEs is beyond the scope of this document.
7. IANA Considerations 7. IANA Considerations
IANA has allocated UDP port 6081 as the well-known destination port IANA has allocated UDP port 6081 as the well-known destination port
for Geneve. Upon publication, the registry should be updated to cite for Geneve. Upon publication, the registry should be updated to cite
this document. The original request was: this document. The original request was:
Service Name: geneve Service Name: geneve
Transport Protocol(s): UDP Transport Protocol(s): UDP
Assignee: Jesse Gross <jgross@vmware.com> Assignee: Jesse Gross <jesse@kernel.org>
Contact: Jesse Gross <jgross@vmware.com> Contact: Jesse Gross <jesse@kernel.org>
Description: Generic Network Virtualization Encapsulation (Geneve) Description: Generic Network Virtualization Encapsulation (Geneve)
Reference: This document Reference: This document
Port Number: 6081 Port Number: 6081
In addition, IANA is requested to create a "Geneve Option Class" In addition, IANA is requested to create a "Geneve Option Class"
registry to allocate Option Classes. This shall be a registry of registry to allocate Option Classes. This shall be a registry of
16-bit hexadecimal values along with descriptive strings. The 16-bit hexadecimal values along with descriptive strings. The
identifiers 0x0-0xFF are to be reserved for standardized options for identifiers 0x0-0xFF are to be reserved for standardized options for
allocation by IETF Review [RFC5226] and 0xFFF0-0xFFFF for allocation by IETF Review [RFC8126] and 0xFFF0-0xFFFF for
Experimental Use. Otherwise, identifiers are to be assigned to any Experimental Use. Otherwise, identifiers are to be assigned to any
organization with an interest in creating Geneve options on a First organization with an interest in creating Geneve options on a First
Come First Served basis. The registry is to be populated with the Come First Served basis. The registry is to be populated with the
following initial values: following initial values:
+----------------+--------------------------------------+ +----------------+--------------------------------------+
| Option Class | Description | | Option Class | Description |
+----------------+--------------------------------------+ +----------------+--------------------------------------+
| 0x0000..0x00FF | Unassigned - IETF Review | | 0x0000..0x00FF | Unassigned - IETF Review |
| 0x0100 | Linux | | 0x0100 | Linux |
| 0x0101 | Open vSwitch | | 0x0101 | Open vSwitch (OVS) |
| 0x0102 | Open Virtual Networking (OVN) | | 0x0102 | Open Virtual Networking (OVN) |
| 0x0103 | In-band Network Telemetry (INT) | | 0x0103 | In-band Network Telemetry (INT) |
| 0x0104 | VMware | | 0x0104 | VMware, Inc. |
| 0x0105 | Amazon | | 0x0105 | Amazon.com, Inc. |
| 0x0106 | Cisco | | 0x0106 | Cisco Systems, Inc. |
| 0x0107..0xFFEF | Unassigned - First Come First Served | | 0x0107 | Oracle Corporation |
| 0x0108..0xFFEF | Unassigned - First Come First Served |
| 0xFFF0..FFFF | Experimental | | 0xFFF0..FFFF | Experimental |
+----------------+--------------------------------------+ +----------------+--------------------------------------+
8. Contributors 8. Contributors
The following individuals were authors of an earlier version of this The following individuals were authors of an earlier version of this
document and made significant contributions: document and made significant contributions:
Pankaj Garg Pankaj Garg
Microsoft Corporation Microsoft Corporation
skipping to change at page 26, line 34 skipping to change at page 30, line 40
Menlo Park, CA 94025 Menlo Park, CA 94025
USA USA
Email: ahendel@fb.com Email: ahendel@fb.com
9. Acknowledgements 9. Acknowledgements
The authors wish to thank Martin Casado, Bruce Davie and Dave Thaler The authors wish to thank Martin Casado, Bruce Davie and Dave Thaler
for their input, feedback, and helpful suggestions. for their input, feedback, and helpful suggestions.
The authors would like to thank Magnus Nystrom for his reviews and
feedback during the SECDIR early review.
Thanks to Daniel Migault, Anoop Ghanwani, Greg Mirksy, and Tal
Mizrahi for their reviews, comments and feedback during the Working
Group Last Call process.
The authors would like to thank David Black for his detailed reviews
and valuable inputs during the TSVART early review.
Thanks to Sami Boutros for his inputs and helpful feedback.
The authors would like to thank Matthew Bocci, Sam Aldrin, Benson
Schliesser, Martin Vigoureux, and Alia Atlas for their guidance
throughout the process.
10. References 10. References
10.1. Normative References 10.1. Normative References
[RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, [RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768,
DOI 10.17487/RFC0768, August 1980, DOI 10.17487/RFC0768, August 1980,
<https://www.rfc-editor.org/info/rfc768>. <https://www.rfc-editor.org/info/rfc768>.
[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5,
RFC 792, DOI 10.17487/RFC0792, September 1981,
<https://www.rfc-editor.org/info/rfc792>.
[RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5,
RFC 1112, DOI 10.17487/RFC1112, August 1989,
<https://www.rfc-editor.org/info/rfc1112>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet
IANA Considerations Section in RFCs", RFC 5226, Control Message Protocol (ICMPv6) for the Internet
DOI 10.17487/RFC5226, May 2008, Protocol Version 6 (IPv6) Specification", STD 89,
<https://www.rfc-editor.org/info/rfc5226>. RFC 4443, DOI 10.17487/RFC4443, March 2006,
<https://www.rfc-editor.org/info/rfc4443>.
[RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and
UDP Checksums for Tunneled Packets", RFC 6935,
DOI 10.17487/RFC6935, April 2013,
<https://www.rfc-editor.org/info/rfc6935>.
[RFC6936] Fairhurst, G. and M. Westerlund, "Applicability Statement
for the Use of IPv6 UDP Datagrams with Zero Checksums",
RFC 6936, DOI 10.17487/RFC6936, April 2013,
<https://www.rfc-editor.org/info/rfc6936>.
[RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for
Writing an IANA Considerations Section in RFCs", BCP 26,
RFC 8126, DOI 10.17487/RFC8126, June 2017,
<https://www.rfc-editor.org/info/rfc8126>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
10.2. Informative References 10.2. Informative References
[ETYPES] The IEEE Registration Authority, "IEEE 802 Numbers", 2013, [ETYPES] The IEEE Registration Authority, "IEEE 802 Numbers", 2013,
<http://www.iana.org/assignments/ieee-802-numbers/ <http://www.iana.org/assignments/ieee-802-numbers/
ieee-802-numbers.xml>. ieee-802-numbers.xml>.
[I-D.ietf-intarea-tunnels]
Touch, J. and M. Townsley, "IP Tunnels in the Internet
Architecture", draft-ietf-intarea-tunnels-09 (work in
progress), July 2018.
[I-D.ietf-nvo3-dataplane-requirements] [I-D.ietf-nvo3-dataplane-requirements]
Bitar, N., Lasserre, M., Balus, F., Morin, T., Jin, L., Bitar, N., Lasserre, M., Balus, F., Morin, T., Jin, L.,
and B. Khasnabish, "NVO3 Data Plane Requirements", draft- and B. Khasnabish, "NVO3 Data Plane Requirements", draft-
ietf-nvo3-dataplane-requirements-03 (work in progress), ietf-nvo3-dataplane-requirements-03 (work in progress),
April 2014. April 2014.
[I-D.ietf-nvo3-encap] [I-D.ietf-nvo3-encap]
Boutros, S., Ganga, I., Garg, P., Manur, R., Mizrahi, T., Boutros, S., "NVO3 Encapsulation Considerations", draft-
Mozes, D., Nordmark, E., Smith, M., Aldrin, S., and I. ietf-nvo3-encap-02 (work in progress), September 2018.
Bagdonas, "NVO3 Encapsulation Considerations", draft-ietf-
nvo3-encap-01 (work in progress), October 2017.
[IEEE.802.1Q_2014] [IEEE.802.1Q_2014]
IEEE, "IEEE Standard for Local and metropolitan area IEEE, "IEEE Standard for Local and metropolitan area
networks--Bridges and Bridged Networks", IEEE 802.1Q-2014, networks--Bridges and Bridged Networks", IEEE 802.1Q-2014,
DOI 10.1109/ieeestd.2014.6991462, December 2014, DOI 10.1109/ieeestd.2014.6991462, December 2014,
<http://ieeexplore.ieee.org/servlet/ <http://ieeexplore.ieee.org/servlet/
opac?punumber=6991460>. opac?punumber=6991460>.
[RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
DOI 10.17487/RFC1191, November 1990, DOI 10.17487/RFC1191, November 1990,
<https://www.rfc-editor.org/info/rfc1191>. <https://www.rfc-editor.org/info/rfc1191>.
[RFC1981] McCann, J., Deering, S., and J. Mogul, "Path MTU Discovery [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003,
for IP version 6", RFC 1981, DOI 10.17487/RFC1981, August DOI 10.17487/RFC2003, October 1996,
1996, <https://www.rfc-editor.org/info/rfc1981>. <https://www.rfc-editor.org/info/rfc2003>.
[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460,
December 1998, <https://www.rfc-editor.org/info/rfc2460>.
[RFC2983] Black, D., "Differentiated Services and Tunnels", [RFC2983] Black, D., "Differentiated Services and Tunnels",
RFC 2983, DOI 10.17487/RFC2983, October 2000, RFC 2983, DOI 10.17487/RFC2983, October 2000,
<https://www.rfc-editor.org/info/rfc2983>. <https://www.rfc-editor.org/info/rfc2983>.
[RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
Label Switching Architecture", RFC 3031, Label Switching Architecture", RFC 3031,
DOI 10.17487/RFC3031, January 2001, DOI 10.17487/RFC3031, January 2001,
<https://www.rfc-editor.org/info/rfc3031>. <https://www.rfc-editor.org/info/rfc3031>.
skipping to change at page 28, line 23 skipping to change at page 33, line 33
[RFC5374] Weis, B., Gross, G., and D. Ignjatic, "Multicast [RFC5374] Weis, B., Gross, G., and D. Ignjatic, "Multicast
Extensions to the Security Architecture for the Internet Extensions to the Security Architecture for the Internet
Protocol", RFC 5374, DOI 10.17487/RFC5374, November 2008, Protocol", RFC 5374, DOI 10.17487/RFC5374, November 2008,
<https://www.rfc-editor.org/info/rfc5374>. <https://www.rfc-editor.org/info/rfc5374>.
[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion
Notification", RFC 6040, DOI 10.17487/RFC6040, November Notification", RFC 6040, DOI 10.17487/RFC6040, November
2010, <https://www.rfc-editor.org/info/rfc6040>. 2010, <https://www.rfc-editor.org/info/rfc6040>.
[RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and
UDP Checksums for Tunneled Packets", RFC 6935,
DOI 10.17487/RFC6935, April 2013,
<https://www.rfc-editor.org/info/rfc6935>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3 Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>. <https://www.rfc-editor.org/info/rfc7348>.
[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network Rekhter, "Framework for Data Center (DC) Network
Virtualization", RFC 7365, DOI 10.17487/RFC7365, October Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
skipping to change at page 29, line 5 skipping to change at page 34, line 11
Virtualization Using Generic Routing Encapsulation", Virtualization Using Generic Routing Encapsulation",
RFC 7637, DOI 10.17487/RFC7637, September 2015, RFC 7637, DOI 10.17487/RFC7637, September 2015,
<https://www.rfc-editor.org/info/rfc7637>. <https://www.rfc-editor.org/info/rfc7637>.
[RFC8014] Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T. [RFC8014] Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T.
Narten, "An Architecture for Data-Center Network Narten, "An Architecture for Data-Center Network
Virtualization over Layer 3 (NVO3)", RFC 8014, Virtualization over Layer 3 (NVO3)", RFC 8014,
DOI 10.17487/RFC8014, December 2016, DOI 10.17487/RFC8014, December 2016,
<https://www.rfc-editor.org/info/rfc8014>. <https://www.rfc-editor.org/info/rfc8014>.
[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
March 2017, <https://www.rfc-editor.org/info/rfc8085>.
[RFC8086] Yong, L., Ed., Crabbe, E., Xu, X., and T. Herbert, "GRE-
in-UDP Encapsulation", RFC 8086, DOI 10.17487/RFC8086,
March 2017, <https://www.rfc-editor.org/info/rfc8086>.
[RFC8201] McCann, J., Deering, S., Mogul, J., and R. Hinden, Ed.,
"Path MTU Discovery for IP version 6", STD 87, RFC 8201,
DOI 10.17487/RFC8201, July 2017,
<https://www.rfc-editor.org/info/rfc8201>.
[RFC8293] Ghanwani, A., Dunbar, L., McBride, M., Bannai, V., and R.
Krishnan, "A Framework for Multicast in Network
Virtualization over Layer 3", RFC 8293,
DOI 10.17487/RFC8293, January 2018,
<https://www.rfc-editor.org/info/rfc8293>.
[VL2] Greenberg, A., et al., "VL2: A Scalable and Flexible Data [VL2] Greenberg, A., et al., "VL2: A Scalable and Flexible Data
Center Network", ACM SIGCOMM Computer Communication Center Network", ACM SIGCOMM Computer Communication
Review, DOI 10.1145/1594977.1592576, 2009, Review, DOI 10.1145/1594977.1592576, 2009,
<http://www.sigcomm.org/sites/default/files/ccr/ <http://www.sigcomm.org/sites/default/files/ccr/
papers/2009/October/1594977-1592576.pdf>. papers/2009/October/1594977-1592576.pdf>.
Authors' Addresses Authors' Addresses
Jesse Gross (editor) Jesse Gross (editor)
 End of changes. 71 change blocks. 
219 lines changed or deleted 483 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/