< draft-ietf-nvo3-encap-03.txt   draft-ietf-nvo3-encap-04.txt >
INTERNET-DRAFT Sami Boutros(Ed.) NVO3 Workgroup S. Boutros, Ed.
Intended Status: Informational VMware Internet-Draft Ciena
Intended status: Standards Track January 23, 2020
Expires: January 2, 2020 July 1, 2019 Expires: July 26, 2020
NVO3 Encapsulation Considerations NVO3 Encapsulation Considerations
draft-ietf-nvo3-encap-03 draft-ietf-nvo3-encap-04
Abstract Abstract
As communicated by WG Chairs, the IETF NVO3 chairs and Routing Area As communicated by WG Chairs, the IETF NVO3 chairs and Routing Area
director have chartered a design team to take forward the director have chartered a design team to take forward the
encapsulation discussion and see if there is potential to design a encapsulation discussion and see if there is potential to design a
common encapsulation that addresses the various technical concerns. common encapsulation that addresses the various technical concerns.
There are implications of different encapsulations in real There are implications of different encapsulations in real
environments consisting of both software and hardware implementations environments consisting of both software and hardware implementations
and spanning multiple data centers. For example, OAM functions such and spanning multiple data centers. For example, OAM functions such
as path MTU discovery become challenging with multiple encapsulations as path MTU discovery become challenging with multiple encapsulations
along the data path. along the data path.
The design team recommend Geneve with few modifications as the common The design team recommend Geneve with few modifications as the common
encapsulation, more details are described in section 7. encapsulation, more details are described in section 7.
Status of this Memo Status of This Memo
This Internet-Draft is submitted to IETF in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF). Note that other groups may also distribute
other groups may also distribute working documents as working documents as Internet-Drafts. The list of current Internet-
Internet-Drafts. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at This Internet-Draft will expire on July 26, 2020.
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
Copyright and License Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 4 1. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 3
2. Design Team Goals . . . . . . . . . . . . . . . . . . . . . . . 4 2. Design Team Goals . . . . . . . . . . . . . . . . . . . . . . 3
3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . 4
5. Issues with current Encapsulations . . . . . . . . . . . . . . 5 5. Issues with current Encapsulations . . . . . . . . . . . . . 4
5.1 Geneve . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 5.1. Geneve . . . . . . . . . . . . . . . . . . . . . . . . . 4
5.2 GUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 5.2. GUE . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5.3 VXLAN-GPE . . . . . . . . . . . . . . . . . . . . . . . . . 5 5.3. VXLAN-GPE . . . . . . . . . . . . . . . . . . . . . . . . 5
6. Common Encapsulation Considerations . . . . . . . . . . . . . . 6 6. Common Encapsulation Considerations . . . . . . . . . . . . . 5
6.1 Current Encapsulations . . . . . . . . . . . . . . . . . . . 6 6.1. Current Encapsulations . . . . . . . . . . . . . . . . . 5
6.2 Useful Extensions Use cases . . . . . . . . . . . . . . . . 6 6.2. Useful Extensions Use cases . . . . . . . . . . . . . . . 5
6.2.1. Telemetry extensions. . . . . . . . . . . . . . . . . . 6 6.2.1. Telemetry extensions. . . . . . . . . . . . . . . . . 6
6.2.2. Security/Integrity extensions . . . . . . . . . . . . . 7 6.2.2. Security/Integrity extensions . . . . . . . . . . . . 6
6.2.3. Group Base Policy . . . . . . . . . . . . . . . . . . . 7 6.2.3. Group Base Policy . . . . . . . . . . . . . . . . . . 7
6.3 Hardware Considerations . . . . . . . . . . . . . . . . . . 7 6.3. Hardware Considerations . . . . . . . . . . . . . . . . . 7
6.4 Extension Size . . . . . . . . . . . . . . . . . . . . . . . 8 6.4. Extension Size . . . . . . . . . . . . . . . . . . . . . 7
6.5 Extension Ordering . . . . . . . . . . . . . . . . . . . . . 9 6.5. Extension Ordering . . . . . . . . . . . . . . . . . . . 8
6.6 TLV vs Bit Fields . . . . . . . . . . . . . . . . . . . . . 9 6.6. TLV vs Bit Fields . . . . . . . . . . . . . . . . . . . . 8
6.7 Control Plane Considerations . . . . . . . . . . . . . . . . 10 6.7. Control Plane Considerations . . . . . . . . . . . . . . 9
6.8 Split NVE . . . . . . . . . . . . . . . . . . . . . . . . . 11 6.8. Split NVE . . . . . . . . . . . . . . . . . . . . . . . . 10
6.9 Larger VNI Considerations . . . . . . . . . . . . . . . . . 11 6.9. Larger VNI Considerations . . . . . . . . . . . . . . . . 11
7. Design team recommendations . . . . . . . . . . . . . . . . . . 11 7. Design team recommendations . . . . . . . . . . . . . . . . . 11
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 14 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13
9. Security Considerations . . . . . . . . . . . . . . . . . . . . 14 9. Security Considerations . . . . . . . . . . . . . . . . . . . 14
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 14 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14
10.1 Normative References . . . . . . . . . . . . . . . . . . . 14 11. Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . 14
10.2 Informative References . . . . . . . . . . . . . . . . . . 15 11.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 14
11. Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . 15 11.2. Extensibility . . . . . . . . . . . . . . . . . . . . . 14
11.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 15 11.2.1. Native Extensibility Support . . . . . . . . . . . . 14
11.2. Extensibility . . . . . . . . . . . . . . . . . . . . . . 15 11.2.2. Extension Parsing . . . . . . . . . . . . . . . . . 14
11.2.1. Native Extensibility Support . . . . . . . . . . . . 15 11.2.3. Critical Extensions . . . . . . . . . . . . . . . . 15
11.2.2. Extension Parsing . . . . . . . . . . . . . . . . . . 15 11.2.4. Maximal Header Length . . . . . . . . . . . . . . . 15
11.2.3. Critical Extensions . . . . . . . . . . . . . . . . . 16 11.3. Encapsulation Header . . . . . . . . . . . . . . . . . . 15
11.2.4. Maximal Header Length . . . . . . . . . . . . . . . . 16 11.3.1. Virtual Network Identifier (VNI) . . . . . . . . . . 15
11.3. Encapsulation Header . . . . . . . . . . . . . . . . . . 16 11.3.2. Next Protocol . . . . . . . . . . . . . . . . . . . 15
11.3.1. Virtual Network Identifier (VNI) . . . . . . . . . . 16 11.3.3. Other Header Fields . . . . . . . . . . . . . . . . 16
11.3.2. Next Protocol . . . . . . . . . . . . . . . . . . . . 16
11.3.3. Other Header Fields . . . . . . . . . . . . . . . . . 17
11.4. Comparison Summary . . . . . . . . . . . . . . . . . . . 17
Authors' Addresses (In alphabetical order) . . . . . . . . . . . . 18
1. Problem Statement 11.4. Comparison Summary . . . . . . . . . . . . . . . . . . . 16
12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 17
13. References . . . . . . . . . . . . . . . . . . . . . . . . . 18
13.1. Normative References . . . . . . . . . . . . . . . . . . 18
13.2. Informative References . . . . . . . . . . . . . . . . . 18
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 19
1. Problem Statement
As communicated by WG Chairs, the NVO3 WG charter states that it may As communicated by WG Chairs, the NVO3 WG charter states that it may
produce requirements for network virtualization data planes based on produce requirements for network virtualization data planes based on
encapsulation of virtual network traffic over an IP-based underlay encapsulation of virtual network traffic over an IP-based underlay
data plane. Such requirements should consider OAM and security. Based data plane. Such requirements should consider OAM and security.
on these requirements the WG will select, extend, and/or develop one Based on these requirements the WG will select, extend, and/or
or more data plane encapsulation format(s). develop one or more data plane encapsulation format(s).
This has led to drafts describing three encapsulations being adopted This has led to drafts describing three encapsulations being adopted
by the working group: by the working group:
- draft-ietf-nvo3-geneve-03 - [I-D.ietf-nvo3-geneve]
- draft-ietf-nvo3-gue-04 - [I-D.ietf-nvo3-gue]
- draft-ietf-nvo3-vxlan-gpe-02 - [I-D.ietf-nvo3-vxlan-gpe]
Discussion on the list and in face-to-face meetings has identified a Discussion on the list and in face-to-face meetings has identified a
number of technical problems with each of these encapsulations. number of technical problems with each of these encapsulations.
Furthermore, there was clear consensus at the IETF meeting in Berlin Furthermore, there was clear consensus at the IETF meeting in Berlin
that it is undesirable for the working group to progress more than that it is undesirable for the working group to progress more than
one data plane encapsulation. Although consensus could not be reached one data plane encapsulation. Although consensus could not be
on the list, the overall consensus was for a single encapsulation reached on the list, the overall consensus was for a single
(RFC2418, Section 3.3). Nonetheless there has been resistance to encapsulation [RFC2418],Section 3.3.
converging on a single encapsulation format.
2. Design Team Goals Nonetheless there has been resistance to converging on a single
encapsulation format.
2. Design Team Goals
As communicated by WG Chairs, the design team should take one of the As communicated by WG Chairs, the design team should take one of the
proposed encapsulations and enhance it to address the technical proposed encapsulations and enhance it to address the technical
concerns. The simple evolution of deployed networks as well as concerns. The simple evolution of deployed networks as well as
applicability to all locations in the NVO3 architecture are goals. applicability to all locations in the NVO3 architecture are goals.
The DT should specifically avoid a design that is burdensome on The DT should specifically avoid a design that is burdensome on
hardware implementations, but should allow future extensibility. The hardware implementations, but should allow future extensibility. The
chosen design should also operate well with ICMP and in ECMP chosen design should also operate well with ICMP and in ECMP
environments. If further extensibility is required, then it should be environments. If further extensibility is required, then it should
done in such a manner that it does not require the consent of an be done in such a manner that it does not require the consent of an
entity outside of the IETF. entity outside of the IETF.
3. Terminology 3. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in [RFC2119].
4. Abbreviations
4. Abbreviations
NVO3 Network Virtualization Overlays over Layer 3 NVO3 Network Virtualization Overlays over Layer 3
OAM Operations, Administration, and Maintenance OAM Operations, Administration, and Maintenance
TLV Type, Length, and Value TLV Type, Length, and Value
VNI Virtual Network Identifier VNI Virtual Network Identifier
NVE Network Virtualization Edge NVE Network Virtualization Edge
NVA Network Virtualization Authority NVA Network Virtualization Authority
NIC Network interface card NIC Network interface card
Transit device Underlay network devices between NVE(s). Transit device Underlay network devices between NVE(s).
5. Issues with current Encapsulations 5. Issues with current Encapsulations
As summarized by WG Chairs. As summarized by WG Chairs.
5.1 Geneve 5.1. Geneve
- Can't be implemented cost-effectively in all use cases because - Can't be implemented cost-effectively in all use cases because
variable length header and order of the TLVs makes is costly (in variable length header and order of the TLVs makes is costly (in
terms of number of gates) to implement in hardware terms of number of gates) to implement in hardware
- Header doesn't fit into largest commonly available parse buffer - Header doesn't fit into largest commonly available parse buffer
(256 bytes in NIC). Cannot justify doubling buffer size unless it is (256 bytes in NIC). Cannot justify doubling buffer size unless it is
mandatory for hardware to process additional option fields. mandatory for hardware to process additional option fields.
5.2 GUE 5.2. GUE
- There were a significant number of objections related to the - There were a significant number of objections related to the
complexity of implementation in hardware, similar to those noted for complexity of implementation in hardware, similar to those noted for
Geneve above. Geneve above.
5.3 VXLAN-GPE 5.3. VXLAN-GPE
- GPE is not day-1 backwards compatible with VXLAN. Although the - GPE is not day-1 backwards compatible with VXLAN. Although the
frame format is similar, it uses a different UDP port, so would frame format is similar, it uses a different UDP port, so would
require changes to existing implementations even if the rest of the require changes to existing implementations even if the rest of the
GPE frame is the same. GPE frame is the same.
- GPE is insufficiently extensible. Numerous extensions and options - GPE is insufficiently extensible. Numerous extensions and options
have been designed for GUE and Geneve. Note that these have not yet have been designed for GUE and Geneve. Note that these have not yet
been validated by the WG. been validated by the WG.
- Security e.g. of the VNI has not been addressed by GPE. Although a - Security e.g. of the VNI has not been addressed by GPE. Although a
shim header could be used for security and other extensions, this has shim header could be used for security and other extensions, this has
not been defined yet and its implications on offloading in NICs are not been defined yet and its implications on offloading in NICs are
not understood. not understood.
6. Common Encapsulation Considerations 6. Common Encapsulation Considerations
6.1 Current Encapsulations 6.1. Current Encapsulations
Appendix A includes a detailed comparison between the three proposed Appendix A includes a detailed comparison between the three proposed
encapsulations. The comparison indicates several common properties, encapsulations. The comparison indicates several common properties,
but also three major differences among the encapsulations: but also three major differences among the encapsulations:
- Extensibility: Geneve and GUE were defined with built-in - Extensibility: Geneve and GUE were defined with built-in
extensibility, while VXLAN-GPE is not inherently extensible. Note extensibility, while VXLAN-GPE is not inherently extensible. Note
that any of the three encapsulations can be extended using the that any of the three encapsulations can be extended using the
Network Service Header (NSH). Network Service Header (NSH).
- Extension method: Geneve is extensible using Type/Length/Value - Extension method: Geneve is extensible using Type/Length/Value
(TLV) fields, while GUE uses a small set of possible extensions, and (TLV) fields, while GUE uses a small set of possible extensions, and
a set of flags that indicate which extension is present. a set of flags that indicate which extension is present.
- Length field: Geneve and GUE include a Length field, indicating the - Length field: Geneve and GUE include a Length field, indicating the
length of the encapsulation header, while VXLAN-GPE does not include length of the encapsulation header, while VXLAN-GPE does not include
such a field. such a field.
6.2 Useful Extensions Use cases 6.2. Useful Extensions Use cases
Non vendor specific TLV MUST follow the standardization process. The Non vendor specific TLV MUST follow the standardization process. The
following use cases for extensions shows that there is a strong following use cases for extensions shows that there is a strong
requirement to support variable length extensions with possible requirement to support variable length extensions with possible
different subtypes. different subtypes.
6.2.1. Telemetry extensions. 6.2.1. Telemetry extensions.
In several scenarios it is beneficial to make information about the In several scenarios it is beneficial to make information about the
path a packet took through the network or through a network device as path a packet took through the network or through a network device as
well as associated telemetry information available to the operator. well as associated telemetry information available to the operator.
This includes not only tasks like debugging, troubleshooting, as well This includes not only tasks like debugging, troubleshooting, as well
as network planning and network optimization but also policy or as network planning and network optimization but also policy or
service level agreement compliance checks. service level agreement compliance checks.
Packet scheduling algorithms, especially for balancing traffic across Packet scheduling algorithms, especially for balancing traffic across
equal cost paths or links, often leverage information contained equal cost paths or links, often leverage information contained
within the packet, such as protocol number, IP-address or MAC- within the packet, such as protocol number, IP-address or MAC-
address. Probe packets would thus either need to be sent from the address. Probe packets would thus either need to be sent from the
exact same endpoints with the exact same parameters, or probe packets exact same endpoints with the exact same parameters, or probe packets
would need to be artificially constructed as "fake" packets and would need to be artificially constructed as "fake" packets and
inserted along the path. Both approaches are often not feasible from inserted along the path. Both approaches are often not feasible from
an operational perspective, be it that access to the end-system is an operational perspective, be it that access to the end-system is
not feasible, or that the diversity of parameters and associated not feasible, or that the diversity of parameters and associated
probe packets to be created is simply too large. An in-band telemetry probe packets to be created is simply too large. An in-band
mechanism in extensions is an alternative in those cases. telemetry mechanism in extensions is an alternative in those cases.
6.2.2. Security/Integrity extensions 6.2.2. Security/Integrity extensions
Since the currently proposed NVO3 encapsulations do not protect their Since the currently proposed NVO3 encapsulations do not protect their
headers a single bit corruption in the VNI field could deliver a headers a single bit corruption in the VNI field could deliver a
packet to the wrong tenant. Extensions are needed to use any packet to the wrong tenant. Extensions are needed to use any
sophisticated security. sophisticated security.
The possibility of VNI spoofing with an NVO3 protocol is exacerbated The possibility of VNI spoofing with an NVO3 protocol is exacerbated
by the use of UDP. Systems typically have no restrictions on by the use of UDP. Systems typically have no restrictions on
applications being able to send to any UDP port so an unprivileged applications being able to send to any UDP port so an unprivileged
application can trivially spoof for instance, VXLAN packets, application can trivially spoof for instance, VXLAN packets,
including using arbitrary VNIs. including using arbitrary VNIs.
One can envision HMAC-like support in some NVO3 extension to One can envision HMAC-like support in some NVO3 extension to
authenticate the header and the outer IP addresses, thereby authenticate the header and the outer IP addresses, thereby
preventing attackers from injecting packets with spoofed VNIs. preventing attackers from injecting packets with spoofed VNIs.
An other aspect of security is payload security. Essentially this is An other aspect of security is payload security. Essentially this is
to make packets that look like IP|UDP|NVO3 Encap|DTLS/IPSEC-ESP to make packets that look like IP|UDP|NVO3 Encap|DTLS/IPSEC-ESP
Extension|payload. This is nice since we still have the UDP header Extension|payload. This is nice since we still have the UDP header
for ECMP, the NVO3 header is in plain text so it can by read by for ECMP, the NVO3 header is in plain text so it can by read by
network elements, and different security or other payload transforms network elements, and different security or other payload transforms
can be supported on a single UDP port (we don't need a separate UDP can be supported on a single UDP port (we don't need a separate UDP
for DTLS/IPSEC). for DTLS/IPSEC).
6.2.3. Group Base Policy 6.2.3. Group Base Policy
Another use case would be to carry the Group Based Policy (GBP) Another use case would be to carry the Group Based Policy (GBP)
source group information within a NVO3 header extension in a similar source group information within a NVO3 header extension in a similar
manner as has been implemented for VXLAN [VXLAN-GBP]. This allows manner as has been implemented for VXLAN
various forms of policy such as access control and QoS to be applied [I-D.smith-vxlan-group-policy]. This allows various forms of policy
between abstract groups rather than coupled to specific endpoint such as access control and QoS to be applied between abstract groups
addresses. rather than coupled to specific endpoint addresses.
6.3 Hardware Considerations 6.3. Hardware Considerations
Hardware restrictions should be taken into consideration along with Hardware restrictions should be taken into consideration along with
future hardware enhancements that may provide more flexible metadata future hardware enhancements that may provide more flexible metadata
processing. However, the set of options that need to and will be processing. However, the set of options that need to and will be
implemented in hardware will be a subset of what is implemented in implemented in hardware will be a subset of what is implemented in
software, since software NVEs are likely to grow features, and hence software, since software NVEs are likely to grow features, and hence
option support, at a more rapid rate. option support, at a more rapid rate.
We note that it is hard to predict which options will be implemented We note that it is hard to predict which options will be implemented
in which piece of hardware and when. That depends on whether the in which piece of hardware and when. That depends on whether the
hardware will be in the form of a NIC providing increasing offload hardware will be in the form of a NIC providing increasing offload
capabilities to software NVEs, or a switch chip being used as an NVE capabilities to software NVEs, or a switch chip being used as an NVE
gateway towards non-NVO3 parts of the network, or even an transit gateway towards non-NVO3 parts of the network, or even an transit
devices that participates in the NVO3 dataplane e.g. for OAM devices that participates in the NVO3 dataplane e.g. for OAM
purposes. purposes.
A result of this is that it doesn't look useful to prescribe some A result of this is that it doesn't look useful to prescribe some
order of the option so that the ones that are likely to be order of the option so that the ones that are likely to be
implemented in hardware come first; we can't decide such an order implemented in hardware come first; we can't decide such an order
when we define the options, however a control plane can enforce such when we define the options, however a control plane can enforce such
order for some hardware implementations. order for some hardware implementations.
We do know that hardware needs to initially be able to efficiently We do know that hardware needs to initially be able to efficiently
skip over the NVO3 header to find the inner payload. That is needed skip over the NVO3 header to find the inner payload. That is needed
for both NICs doing e.g. TCP offload and transit devices and NVEs for both NICs doing e.g. TCP offload and transit devices and NVEs
applying policy/ACLs to the inner payload. applying policy/ACLs to the inner payload.
6.4 Extension Size 6.4. Extension Size
Extension header length has a significant impact to hardware and Extension header length has a significant impact to hardware and
software implementations. A total header length that is too small software implementations. A total header length that is too small
will unnecessarily constrained software flexibility. A total header will unnecessarily constrained software flexibility. A total header
length that is too large will place a nontrivial cost on hardware length that is too large will place a nontrivial cost on hardware
implementations. Thus, the design team recommends that there be a implementations. Thus, the design team recommends that there be a
minimum and maximum total extension header length selected. The minimum and maximum total extension header length selected. The
maximum total header length is determined by the bits allocated for maximum total header length is determined by the bits allocated for
the total extension header length field. The risk with this approach the total extension header length field. The risk with this approach
is that it may be difficult to extend the total header size in the is that it may be difficult to extend the total header size in the
future. The minimum total header length is determined by a future. The minimum total header length is determined by a
requirement in the specifications that all implementations must meet. requirement in the specifications that all implementations must meet.
The risk with this approach is that all implementations will only The risk with this approach is that all implementations will only
implement the minimum total header length which would then become the implement the minimum total header length which would then become the
de facto maximum total header length. The recommended minimum total de facto maximum total header length. The recommended minimum total
header length is 64 bytes. header length is 64 bytes.
Single Extension size should always be 4 bytes aligned. Single Extension size should always be 4 bytes aligned.
The maximum length of a single option should be large enough to meet The maximum length of a single option should be large enough to meet
the different extension use case requirements e.g. in-band telemetry the different extension use case requirements e.g. in-band telemetry
and future use. and future use.
6.5 Extension Ordering 6.5. Extension Ordering
In order to support hardware nodes at the tunnel endpoint or at the In order to support hardware nodes at the tunnel endpoint or at the
transit that can process one or few extensions TLVs in TCAM. A transit that can process one or few extensions TLVs in TCAM. A
control plane in such a deployment can signal a capability to ensure control plane in such a deployment can signal a capability to ensure
a specific TLV will always appear in a specific order for example the a specific TLV will always appear in a specific order for example the
first one in the packet. first one in the packet.
The order of the TLVs should be HW friendly for both the sender and The order of the TLVs should be HW friendly for both the sender and
the receiver and possibly the transit node too. the receiver and possibly the transit node too.
Transit nodes doesn't participate in control plane communication Transit nodes doesn't participate in control plane communication
between the end points and are not required to process the options between the end points and are not required to process the options
however, if they do, they need to process only a small subset of however, if they do, they need to process only a small subset of
options that will be consumed by tunnel endpoints. options that will be consumed by tunnel endpoints.
6.6 TLV vs Bit Fields 6.6. TLV vs Bit Fields
If there is a well-known initial set of options that are likely to be If there is a well-known initial set of options that are likely to be
implemented in software and in hardware, it can be efficient to use implemented in software and in hardware, it can be efficient to use
the bit-field approach as in GUE. However, as described in section the bit-field approach as in GUE. However, as described in section
6.3, if options are added over time and different subsets of options 6.3, if options are added over time and different subsets of options
are likely to be implemented in different pieces of hardware, then it are likely to be implemented in different pieces of hardware, then it
would be hard for the IETF to specify which options should get the would be hard for the IETF to specify which options should get the
early bit fields. TLVs are a lot more flexible, which avoids the need early bit fields. TLVs are a lot more flexible, which avoids the
to determine the relative importance different options. However, need to determine the relative importance different options.
general TLV of arbitrary order, size, and repetition of the same However, general TLV of arbitrary order, size, and repetition of the
order is difficult to implement in hardware. A middle ground is to same order is difficult to implement in hardware. A middle ground is
use TLV with restrictions on the size and alignment, observing that to use TLV with restrictions on the size and alignment, observing
individual TLVs can have a fixed length, and support in the control that individual TLVs can have a fixed length, and support in the
plane such that an NVE will only receive options that to needs and control plane such that an NVE will only receive options that to
implements. The control plane approach can potentially be used to needs and implements. The control plane approach can potentially be
control the order of the TLVs sent to a particular NVE. Note that used to control the order of the TLVs sent to a particular NVE. Note
transit devices are not likely to participate in the control plane that transit devices are not likely to participate in the control
hence to the extent that they need to participate in option plane hence to the extent that they need to participate in option
processing they need more effort, But transit devices would have processing they need more effort, But transit devices would have
issues with future GUE bits being defined for future options as well. issues with future GUE bits being defined for future options as well.
A benefit of TLVs from a HW perspective is that they are self A benefit of TLVs from a HW perspective is that they are self
describing i.e., all the information is in the TLV. In a Bit fields describing i.e., all the information is in the TLV. In a Bit fields
approach the hardware needs to look up the bit to determine the approach the hardware needs to look up the bit to determine the
length of the data associated with the bit through some separate length of the data associated with the bit through some separate
table, which would add hardware complexity. table, which would add hardware complexity.
There are use cases where multiple modules of software are running on There are use cases where multiple modules of software are running on
NVE. This can be modules such as a diagnostic module by one vendor NVE. This can be modules such as a diagnostic module by one vendor
that does packet sampling and another module from a different vendor that does packet sampling and another module from a different vendor
that does a firewall. Using a TLV format, it is easier to have that does a firewall. Using a TLV format, it is easier to have
different software modules process different TLVs, which could be different software modules process different TLVs, which could be
standard extensions or vendor specific extensions defined by the standard extensions or vendor specific extensions defined by the
different vendors, without conflicting with each other. This can help different vendors, without conflicting with each other. This can
with hardware modularity as well. There are some implementations with help with hardware modularity as well. There are some
options that allows different software like mac learning and security implementations with options that allows different software like mac
handle different options. learning and security handle different options.
6.7 Control Plane Considerations 6.7. Control Plane Considerations
Given that we want to allow large flexibility and extensibility for Given that we want to allow large flexibility and extensibility for
e.g. software NVEs, yet be able to support key extensions in less e.g. software NVEs, yet be able to support key extensions in less
flexible e.g. hardware NVEs, it is useful to consider the control flexible e.g. hardware NVEs, it is useful to consider the control
plane. By control plane in this context we mean both protocols such plane. By control plane in this context we mean both protocols such
as EVPN and others, and also deployment specific configuration. as EVPN and others, and also deployment specific configuration.
If each NVE can express in the control plane that they only care If each NVE can express in the control plane that they only care
about particular extensions (could be a single extension, or a few), about particular extensions (could be a single extension, or a few),
and the source NVEs only include requested extensions in the NVO3 and the source NVEs only include requested extensions in the NVO3
packets, then the target NVE can both use a simpler parser (e.g., a packets, then the target NVE can both use a simpler parser (e.g., a
TCAM might be usable to look for a single NVO3 extension) and the TCAM might be usable to look for a single NVO3 extension) and the
depth of the inner payload in the NVO3 packet will be minimized. depth of the inner payload in the NVO3 packet will be minimized.
Furthermore, if the target NVE cares about a few extensions and can Furthermore, if the target NVE cares about a few extensions and can
express in the control plane the desired order of those extensions in express in the control plane the desired order of those extensions in
the NVO3 packets, then it can provide useful functionality with the NVO3 packets, then it can provide useful functionality with
minimal hardware requirements. minimal hardware requirements.
Note that transit devices that are not aware of the NVO3 extensions Note that transit devices that are not aware of the NVO3 extensions
somewhat benefit from such an approach, since the inner payload is somewhat benefit from such an approach, since the inner payload is
less deep in the packet if no extraneous extensions are included in less deep in the packet if no extraneous extensions are included in
the packet. However, in general a transit device is not likely to the packet. However, in general a transit device is not likely to
participate in the NVO3 control plane. (However, configuration participate in the NVO3 control plane. (However, configuration
mechanisms can take into account limitations of the transit devices mechanisms can take into account limitations of the transit devices
used in particular deployments.) used in particular deployments.)
Note that in this approach different NVEs could desire different Note that in this approach different NVEs could desire different
(sets of) extensions, which means that the source NVE needs to be (sets of) extensions, which means that the source NVE needs to be
able to place different sets of extensions in different NVO3 packets, able to place different sets of extensions in different NVO3 packets,
and perhaps in different order. It also assumes that underlay and perhaps in different order. It also assumes that underlay
multicast or replication servers are not used together with NVO3 multicast or replication servers are not used together with NVO3
extensions. extensions.
There is a need to consider mandatory extensions versus optional There is a need to consider mandatory extensions versus optional
extensions. Mandatory extensions require the receiver to drop the extensions. Mandatory extensions require the receiver to drop the
packet if the extension is unknown. A control plane mechanism can packet if the extension is unknown. A control plane mechanism can
prevent the need for dropping unknown extensions, since they would prevent the need for dropping unknown extensions, since they would
not be included to targets that do not support them. not be included to targets that do not support them.
The control planes defined today need to add the ability to describe The control planes defined today need to add the ability to describe
the different encapsulations. Thus perhaps EVPN, and any other the different encapsulations. Thus perhaps EVPN, and any other
control plane protocol that the IETF defines, should have a way to control plane protocol that the IETF defines, should have a way to
enumerate the supported NVO3 extensions and their order. enumerate the supported NVO3 extensions and their order.
The WG should consider developing a separate draft on guidance for The WG should consider developing a separate draft on guidance for
option processing and control plane participation. This should option processing and control plane participation. This should
provide examples/guidance on range of usage models and deployments provide examples/guidance on range of usage models and deployments
scenarios for specific options and ordering that are relevant for scenarios for specific options and ordering that are relevant for
that specific deployment. This includes end points and middle boxes that specific deployment. This includes end points and middle boxes
using the options. So, having the control plane negotiate the using the options. So, having the control plane negotiate the
constraints is most appropriate and flexible way to address these constraints is most appropriate and flexible way to address these
requirements. requirements.
6.8 Split NVE 6.8. Split NVE
If the working group sees a need for having the hosts send and If the working group sees a need for having the hosts send and
receive options in a split NVE case, this is possible using any of receive options in a split NVE case, this is possible using any of
the existing extensible encapsulations (Geneve, GUE, GPE+NSH) by the existing extensible encapsulations (Geneve, GUE, GPE+NSH) by
defining a way to carry those over other transports. NSH can already defining a way to carry those over other transports. NSH can already
be used over different transports. be used over different transports.
If we need to do this with other encapsulations it can be done by If we need to do this with other encapsulations it can be done by
defining an Ether type for other encapsulations so that it can be defining an Ether type for other encapsulations so that it can be
carried over Ethernet and 802.1Q. carried over Ethernet and 802.1Q.
If we need to carry other encapsulations over MPLS, it would require If we need to carry other encapsulations over MPLS, it would require
an EVPN control plane to signal that other encapsulation header + an EVPN control plane to signal that other encapsulation header +
options will be present in front of the L2 packet. The VNI can be options will be present in front of the L2 packet. The VNI can be
ignored in the header, and the MPLS label will be the one used to ignored in the header, and the MPLS label will be the one used to
identify the EVPN L2 instance. identify the EVPN L2 instance.
6.9 Larger VNI Considerations 6.9. Larger VNI Considerations
We discussed whether we should make VNI 32-bits or larger. The We discussed whether we should make VNI 32-bits or larger. The
benefit of 24-bit VNI would be to avoid unnecessary changes with benefit of 24-bit VNI would be to avoid unnecessary changes with
existing proposals and implementations that are almost all, if not existing proposals and implementations that are almost all, if not
all, are using 24-bit VNI. If we need a larger VNI, an extension can all, are using 24-bit VNI. If we need a larger VNI, an extension can
be used to support that. be used to support that.
7. Design team recommendations 7. Design team recommendations
We concluded that Geneve is most suitable as a starting point for We concluded that Geneve is most suitable as a starting point for
proposed standard for network virtualization, for the following proposed standard for network virtualization, for the following
reasons: reasons:
1. We studied whether VNI should be in base header or in extensions 1. We studied whether VNI should be in base header or in extensions
and whether it should be 24-bit or 32-bit. The design team agreed and whether it should be 24-bit or 32-bit. The design team agreed
that VNI is critical information for network virtualization and MUST that VNI is critical information for network virtualization and MUST
be present in all packets. Design team also agreed that 24-bit VNI be present in all packets. Design team also agreed that 24-bit VNI
matches the existing widely used encapsulation format i.e. VxLAN and matches the existing widely used encapsulation format i.e. VxLAN and
NVGRE and hence more suitable to use going forward. NVGRE and hence more suitable to use going forward.
2. Geneve has the total options length that allow skipping over the 2. Geneve has the total options length that allow skipping over the
options for NIC offload operations, and will allow transit devices to options for NIC offload operations, and will allow transit devices to
view flow information in the inner payload. view flow information in the inner payload.
3. We considered the option of using NSH with VxLAN-GPE but given 3. We considered the option of using NSH with VxLAN-GPE but given
that NSH is targeted at service chaining and contains service that NSH is targeted at service chaining and contains service
chaining information, it is less suitable for the network chaining information, it is less suitable for the network
virtualization use case. The other downside for VxLAN-GPE was lack of virtualization use case. The other downside for VxLAN-GPE was lack
header length in VxLAN-GPE and hence makes skipping over the headers of header length in VxLAN-GPE and hence makes skipping over the
to process inner payload more difficult. Total Option Length is headers to process inner payload more difficult. Total Option Length
present in Geneve. It is not possible to skip any options in the is present in Geneve. It is not possible to skip any options in the
middle with VxLAN-GPE. In principle a split between a base header and middle with VxLAN-GPE. In principle a split between a base header
a header with options is interesting (whether that options header is and a header with options is interesting (whether that options header
NSH or some new header without ties to a service path). We explored is NSH or some new header without ties to a service path). We
whether it would make sense to either use NSH for this, or define a explored whether it would make sense to either use NSH for this, or
new NVO3 options header. However, we observed that this makes it define a new NVO3 options header. However, we observed that this
slightly harder to find the inner payload since the length field is makes it slightly harder to find the inner payload since the length
not in the NVO3 header itself. Thus one more field would have to be field is not in the NVO3 header itself. Thus one more field would
extracted to compute the start of the inner payload. Also, if the have to be extracted to compute the start of the inner payload.
experience with IPv6 extension headers is a guidance, there would be Also, if the experience with IPv6 extension headers is a guidance,
a risk that key pieces of hardware might not implement the options there would be a risk that key pieces of hardware might not implement
header, resulting in future calls to deprecate its use. Making the the options header, resulting in future calls to deprecate its use.
options part of the base NVO3 header has less of those issues. Even Making the options part of the base NVO3 header has less of those
though the implementation of any particular option can not be issues. Even though the implementation of any particular option can
predicted ahead of time, the option mechanism and ability to skip the not be predicted ahead of time, the option mechanism and ability to
options is likely to be broadly implemented. skip the options is likely to be broadly implemented.
4. We compared the TLV vs Bit-fields style extension and it was 4. We compared the TLV vs Bit-fields style extension and it was
deemed that parsing both TLV and bit-fields is expensive and while deemed that parsing both TLV and bit-fields is expensive and while
bit-fields may be simpler to parse, it is also more restrictive and bit-fields may be simpler to parse, it is also more restrictive and
requires guessing which extensions will be widely implemented so they requires guessing which extensions will be widely implemented so they
can get early bit assignments, given that half the bits are already can get early bit assignments, given that half the bits are already
assigned in GUE, a widely deployed extension may appear in a flag assigned in GUE, a widely deployed extension may appear in a flag
extension, and this will require extra processing, to dig the flag extension, and this will require extra processing, to dig the flag
from the flag extension and then look for the extension itself. As from the flag extension and then look for the extension itself. As
well Bit-fields are not flexible enough to address the requirements well Bit-fields are not flexible enough to address the requirements
from OAM, Telemetry and security extensions, for variable length from OAM, Telemetry and security extensions, for variable length
option and different subtypes of the same option. While TLV are more option and different subtypes of the same option. While TLV are more
flexible, a control plane can restrict the number of option TLVs as flexible, a control plane can restrict the number of option TLVs as
well the order and size of the TLVs to make it simpler for a well the order and size of the TLVs to make it simpler for a
dataplane implementation to handle. dataplane implementation to handle.
5. We briefly discussed multi-vendor NVE case, and the need to allow 5. We briefly discussed multi-vendor NVE case, and the need to allow
vendors to put their own extensions in the NVE header. This is vendors to put their own extensions in the NVE header. This is
possible with TLVs. possible with TLVs.
6. We also agreed that the C bit in Geneve is helpful to allow 6. We also agreed that the C bit in Geneve is helpful to allow
receiver NVE to easily decide whether to process options or not. For receiver NVE to easily decide whether to process options or not. For
example a UUID based packet trace and how an optional extension such example a UUID based packet trace and how an optional extension such
as that can be ignored by receiver NVE and thus make it easy for NVE as that can be ignored by receiver NVE and thus make it easy for NVE
to skip over the options. Thus the C-bit remains as defined in to skip over the options. Thus the C-bit remains as defined in
Geneve. Geneve.
7. There are already some extensions that are being discussed (see 7. There are already some extensions that are being discussed (see
section 6.2) of varying sizes, by using Geneve option it is possible section 6.2) of varying sizes, by using Geneve option it is possible
to get in band parameters like: switch id, ingress port, egress port, to get in band parameters like: switch id, ingress port, egress port,
internal delay, and queue in telemetry defined extension TLV from internal delay, and queue in telemetry defined extension TLV from
switches. It is also possible to add Security extension TLVs like switches. It is also possible to add Security extension TLVs like
HMAC and DTLS/IPSEC to authenticate the Geneve packet header and HMAC and DTLS/IPSEC to authenticate the Geneve packet header and
secure the Geneve packet payload by software or hardware tunnel secure the Geneve packet payload by software or hardware tunnel
endpoints. As well, a Group Based Policy extension TLV can be endpoints. As well, a Group Based Policy extension TLV can be
carried. carried.
8. There are implemented Geneve options today in production. There 8. There are implemented Geneve options today in production. There
are as well new HW supporting Geneve TLV parsing. In addition In-band are as well new HW supporting Geneve TLV parsing. In addition In-
Telemetry (INT) specification being developed by P4.org illustrates band Telemetry (INT) specification being developed by P4.org
the option of INT meta data carried over Geneve. OVN/OVS have also illustrates the option of INT meta data carried over Geneve. OVN/OVS
defined some option TLV(s) for Geneve. have also defined some option TLV(s) for Geneve.
9. The DT has addressed the usage models while considering the 9. The DT has addressed the usage models while considering the
requirements and implementations in general that includes software requirements and implementations in general that includes software
and hardware. and hardware.
There seems to be interest to standardize some well known secure There seems to be interest to standardize some well known secure
option TLVs to secure the header and payload to guarantee option TLVs to secure the header and payload to guarantee
encapsulation header integrity and tenant data privacy. The design encapsulation header integrity and tenant data privacy. The design
team recommends that the working group consider standardizing such team recommends that the working group consider standardizing such
option(s). option(s).
We recommend the following enhancements to Geneve to make it more We recommend the following enhancements to Geneve to make it more
suitable to hardware and yet provide the flexibility for software: suitable to hardware and yet provide the flexibility for software:
We would propose a text such as, while TLV are more flexible, a We would propose a text such as, while TLV are more flexible, a
control plane can restrict the number of option TLVs as well the control plane can restrict the number of option TLVs as well the
order and size of the TLVs to make it simpler for a data plane order and size of the TLVs to make it simpler for a data plane
implementation in software or hardware to handle. For example, there implementation in software or hardware to handle. For example, there
may be some critical information such as secure hash that must be may be some critical information such as secure hash that must be
processed in certain order at lowest latency. processed in certain order at lowest latency.
A control plane can negotiate a subset of option TLVs and certain TLV A control plane can negotiate a subset of option TLVs and certain TLV
ordering, as well can limit the total number of option TLVs present ordering, as well can limit the total number of option TLVs present
in the packet, for example, to allow hardware capable of processing in the packet, for example, to allow hardware capable of processing
fewer options. Hence, the control planes need to have the ability to fewer options. Hence, the control planes need to have the ability to
describe the supported TLVs subset and their order. describe the supported TLVs subset and their order.
The Geneve draft could specify that the subset and order of option The Geneve draft could specify that the subset and order of option
TLVs should be configurable for each remote NVE in the absence of a TLVs should be configurable for each remote NVE in the absence of a
protocol control plane. protocol control plane.
We recommend Geneve to follow fragmentation recommendations in We recommend Geneve to follow fragmentation recommendations in
overlay services like PWE3, and L2/L3 VPN recommendation to guarantee overlay services like PWE3, and L2/L3 VPN recommendation to guarantee
larger MTU for the tunnel overhead larger MTU for the tunnel overhead [RFC3985],Section 5.3.
https://tools.ietf.org/html/rfc3985#section-5.3
We request Geneve to provide a recommendation for critical bit We request Geneve to provide a recommendation for critical bit
processing - text could look like how critical bits can be used with processing - text could look like how critical bits can be used with
control plane specifying the critical options. control plane specifying the critical options.
Given that there is a telemetry option use case for a length of 256 Given that there is a telemetry option use case for a length of 256
bytes, we recommend Geneve to increase the Single TLV option length bytes, we recommend Geneve to increase the Single TLV option length
to 256. to 256.
We request Geneve to address Requirements for OAM considerations for We request Geneve to address Requirements for OAM considerations for
alternate marking and for performance measurements that need 2 bits alternate marking and for performance measurements that need 2 bits
in the header. And clarify the need of the current OAM bit in the in the header. And clarify the need of the current OAM bit in the
Geneve Header. Geneve Header.
We recommend the WG to work on security options for Geneve. We recommend the WG to work on security options for Geneve.
8. Acknowledgements 8. Acknowledgements
The authors would like to thank Tom Herbert for providing the The authors would like to thank Tom Herbert for providing the
motivation for the Security/Integrity extension, and for his valuable motivation for the Security/Integrity extension, and for his valuable
comments, and would like to thank T. Sridhar for his valuable comments, and would like to thank T. Sridhar for his valuable
comments and feedback. comments and feedback.
9. Security Considerations 9. Security Considerations
This document does not introduce any additional security constraints. This document does not introduce any additional security constraints.
10. References 10. IANA Considerations
10.1 Normative References
[KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
10.2 Informative References
[Geneve] Generic Network Virtualization Encapsulation [I-D.ietf-nvo3-
geneve]
[GUE] Generic UDP Encapsulation [I-D.ietf-nvo3-gue]
[NSH] Network Service Header [I-D.ietf-sfc-nsh]
[VXLAN-GPE] Virtual eXtensible Local Area Network - Generic Protocol
Extension [I-D.ietf-nvo3-vxlan-gpe]
[VXLAN-GBP] VXLAN Group Policy Option - [I-D.draft-smith-vxlan-group- This document has no actions for IANA.
policy-03]
11. Appendix A 11. Appendix A
11.1. Overview 11.1. Overview
This section presents a comparison of the three NVO3 encapsulation This section presents a comparison of the three NVO3 encapsulation
proposals, Geneve, GUE, and VXLAN-GPE. The three encapsulations use proposals, Geneve, GUE, and VXLAN-GPE. The three encapsulations use
an outer UDP/IP transport. Geneve and VXLAN-GPE use an 8-octet an outer UDP/IP transport. Geneve and VXLAN-GPE use an 8-octet
header, while GUE uses a 4-octet header. In addition to the base header, while GUE uses a 4-octet header. In addition to the base
header, optional extensions may be included in the encapsulation, as header, optional extensions may be included in the encapsulation, as
discussed in Section 3.2 below. discussed in Section 3.2 below.
11.2. Extensibility 11.2. Extensibility
11.2.1. Native Extensibility Support 11.2.1. Native Extensibility Support
The Geneve and GUE encapsulations both enable optional headers to be The Geneve and GUE encapsulations both enable optional headers to be
incorporated at the end of the base encapsulation header. incorporated at the end of the base encapsulation header.
VXLAN-GPE does not provide native support for header extensions. VXLAN-GPE does not provide native support for header extensions.
However, as discussed in [I-D.ietf-nvo3-vxlan-gpe], extensibility can However, as discussed in [I-D.ietf-nvo3-vxlan-gpe], extensibility can
be attained to some extent if the Network Service Header (NSH) [I- be attained to some extent if the Network Service Header (NSH)
D.ietf-sfc-nsh] is used immediately following the VXLAN-GPE header. [RFC8300] is used immediately following the VXLAN-GPE header. NSH
NSH supports either a fixed-size extension (MD Type 1), or a supports either a fixed-size extension (MD Type 1), or a variable-
variable-size TLV-based extension (MD Type 2). It should be noted size TLV-based extension (MD Type 2). It should be noted that NSH-
that NSH-over-VXLAN-GPE implies an additional overhead of the 8- over-VXLAN-GPE implies an additional overhead of the 8- octets NSH
octets NSH header, in addition to the VXLAN-GPE header. header, in addition to the VXLAN-GPE header.
11.2.2. Extension Parsing 11.2.2. Extension Parsing
The Geneve Variable Length Options are defined as The Geneve Variable Length Options are defined as Type/Length/
Type/Length/Value(TLV) extensions. Similarly, VXLAN-GPE, when using Value(TLV) extensions. Similarly, VXLAN-GPE, when using NSH, can
NSH, can include NSH TLV-based extensions. In contrast, GUE defines include NSH TLV-based extensions. In contrast, GUE defines a small
a small set of possible extension fields (proposed in [I-D.herbert- set of possible extension fields (proposed in
gue-extensions]), and a set of flags in the GUE header that indicate [I-D.herbert-gue-extensions], and a set of flags in the GUE header
for each extension type whether it is present or not. that indicate for each extension type whether it is present or not.
TLV-based extensions, as defined in Geneve, provide the flexibility TLV-based extensions, as defined in Geneve, provide the flexibility
for a large number of possible extension types. Similar behavior can for a large number of possible extension types. Similar behavior can
be supported in NSH-over-VXLAN-GPE when using MD Type 2. The flag- be supported in NSH-over-VXLAN-GPE when using MD Type 2. The flag-
based approach taken in GUE strives to simplify implementations by based approach taken in GUE strives to simplify implementations by
defining a small number of possible extensions, used in a fixed defining a small number of possible extensions, used in a fixed
order. order.
The Geneve and GUE headers both include a length field, defining the The Geneve and GUE headers both include a length field, defining the
total length of the encapsulation, including the optional extensions. total length of the encapsulation, including the optional extensions.
The length field simplifies the parsing of transit devices that skip The length field simplifies the parsing of transit devices that skip
the encapsulation header without parsing its extensions. the encapsulation header without parsing its extensions.
11.2.3. Critical Extensions 11.2.3. Critical Extensions
The Geneve encapsulation header includes the 'C' field, which The Geneve encapsulation header includes the 'C' field, which
indicates whether the current Geneve header includes critical indicates whether the current Geneve header includes critical
options, which must be parsed by the tunnel endpoint. If the endpoint options, which must be parsed by the tunnel endpoint. If the
is not able to process the critical option, the packet is discarded. endpoint is not able to process the critical option, the packet is
discarded.
11.2.4. Maximal Header Length 11.2.4. Maximal Header Length
The maximal header length in Geneve, including options, is 260 The maximal header length in Geneve, including options, is 260
octets. GUE defines the maximal header to be 128 octets. VXLAN-GPE octets. GUE defines the maximal header to be 128 octets. VXLAN-GPE
uses a fixed-length header of 8 octets, unless NSH-over-VXLAN-GPE is uses a fixed-length header of 8 octets, unless NSH-over-VXLAN-GPE is
used, yielding an encapsulation header of up to 264 octets. used, yielding an encapsulation header of up to 264 octets.
11.3. Encapsulation Header 11.3. Encapsulation Header
11.3.1. Virtual Network Identifier (VNI) 11.3.1. Virtual Network Identifier (VNI)
The Geneve and VXLAN-GPE headers both include a 24-bit VNI field. The Geneve and VXLAN-GPE headers both include a 24-bit VNI field.
GUE, on the other hand, enables the use of a 32-bit field called GUE, on the other hand, enables the use of a 32-bit field called
VNID; this field is not included in the GUE header, but was defined VNID; this field is not included in the GUE header, but was defined
as an optional extension in [I-D.herbert-gue-extensions]. as an optional extension in [I-D.herbert-gue-extensions].
The VXLAN-GPE header includes the 'I' bit, indicating that the VNI The VXLAN-GPE header includes the 'I' bit, indicating that the VNI
field is valid in the current header. A similar indicator is defined field is valid in the current header. A similar indicator is defined
as a flag in the GUE header [I-D.herbert-gue-extensions]. as a flag in the GUE header herbert-gue-extensions.
11.3.2. Next Protocol 11.3.2. Next Protocol
The three encapsulation headers include a field that specifies the The three encapsulation headers include a field that specifies the
type of the next protocol header, which resides after the NVO3 type of the next protocol header, which resides after the NVO3
encapsulation header. The Geneve header includes a 16-bit field that encapsulation header. The Geneve header includes a 16-bit field that
uses the IEEE Ethertype convention. GUE uses an 8-bit field, which uses the IEEE Ethertype convention. GUE uses an 8-bit field, which
uses the IANA Internet protocol numbering. The VXLAN-GPE header uses the IANA Internet protocol numbering. The VXLAN-GPE header
incorporates an 8-bit Next Protocol field, using a VXLAN-GPE-specific incorporates an 8-bit Next Protocol field, using a VXLAN-GPE-specific
registry, defined in [I-D.ietf-nvo3-vxlan-gpe]. registry, defined in [I-D.ietf-nvo3-vxlan-gpe].
skipping to change at page 17, line 36 skipping to change at page 16, line 32
field, which is currently defined to be zero. field, which is currently defined to be zero.
The Geneve and VXLAN-GPE headers include reserved fields; 14 bits in The Geneve and VXLAN-GPE headers include reserved fields; 14 bits in
the Geneve header, and 27 bits in the VXLAN-GPE header are reserved. the Geneve header, and 27 bits in the VXLAN-GPE header are reserved.
11.4. Comparison Summary 11.4. Comparison Summary
The following table summarizes the comparison between the three NVO3 The following table summarizes the comparison between the three NVO3
encapsulations. encapsulations.
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| | Geneve | GUE | VXLAN-GPE | | | Geneve | GUE | VXLAN-GPE |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Outer transport| UDP/IP | UDP/IP | UDP/IP | | Outer transport| UDP/IP | UDP/IP | UDP/IP |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Base header | 8 octets | 4 octets | 8 octets | | Base header | 8 octets | 4 octets | 8 octets |
| length | | | (16 octets | | length | | | (16 octets |
| | | | using NSH) | | | | | using NSH) |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Extensibility |Variable length |Extension fields| No native ext- | | Extensibility |Variable length |Extension fields| No native ext- |
| | options | | ensibility. | | | options | | ensibility. |
| | | | Extensible | | | | | Extensible |
| | | | using NSH. | | | | | using NSH. |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Extension | TLV-based | Flag-based | TLV-based | | Extension | TLV-based | Flag-based | TLV-based |
| parsing method | | |(using NSH with | | parsing method | | |(using NSH with |
| | | | MD Type 2) | | | | | MD Type 2) |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Extension | Variable | Fixed | Variable | | Extension | Variable | Fixed | Variable |
| order | | | (using NSH) | | order | | | (using NSH) |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Length field | + | + | - | | Length field | + | + | - |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Max Header | 260 octets | 128 octets | 8 octets | | Max Header | 260 octets | 128 octets | 8 octets |
| Length | | |(264 using NSH) | | Length | | |(264 using NSH) |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Critical exte- | + | - | - | | Critical exte- | + | - | - |
| nsion bit | | | | | nsion bit | | | |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| VNI field size | 24 bits | 32 bits | 24 bits | | VNI field size | 24 bits | 32 bits | 24 bits |
| | | (extension) | | | | | (extension) | |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Next protocol | 16 bits | 8 bits | 8 bits | | Next protocol | 16 bits | 8 bits | 8 bits |
| field | Ethertype | Internet prot- | New registry | | field | Ethertype | Internet prot- | New registry |
| | registry | ocol registry | | | | registry | ocol registry | |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Next protocol | - | - | + | | Next protocol | - | - | + |
| indicator | | | | | indicator | | | |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| OAM / control | OAM bit | Control bit | OAM bit | | OAM / control | OAM bit | Control bit | OAM bit |
| field | | | | | field | | | |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Version field | 2 bits | 2 bits | 2 bits | | Version field | 2 bits | 2 bits | 2 bits |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
| Reserved bits | 14 bits | - | 27 bits | | Reserved bits | 14 bits | - | 27 bits |
+----------------+----------------+----------------+----------------+ +----------------+----------------+----------------+----------------+
Figure 1: NVO3 Encapsulation Comparison Figure 1: NVO3 Encapsulation Comparison
Authors' Addresses (In alphabetical order) 12. Contributors
Sami Boutros the following co-authors have contributed to this document.
VMware
Email: boutross@vmware.com
Ilango Ganga Ilango Ganga Intel Email: ilango.s.ganga@intel.com
Intel
Email: ilango.s.ganga@intel.com
Pankaj Garg Pankaj Garg Microsoft Email: pankajg@microsoft.com
Microsoft
Email: pankajg@microsoft.com
Rajeev Manur Rajeev Manur Broadcom Email: rajeev.manur@broadcom.com
Broadcom
Email: rajeev.manur@broadcom.com
Tal Mizrahi Tal Mizrahi Marvell Email: talmi@marvell.com
Marvell
Email: talmi@marvell.com
David Mozes David Mozes Email: mosesster@gmail.com
Email: mosesster@gmail.com
Erik Nordmark Erik Nordmark Email: nordmark@sonic.net
Email: nordmark@sonic.net
Michael Smith Michael Smith Cisco Email: michsmit@cisco.com
Cisco Sam Aldrin Google Email: aldrin.ietf@gmail.com
Email: michsmit@cisco.com
Sam Aldrin Ignas Bagdonas Equinix Email: ibagdona.ietf@gmail.com
Google
Email: aldrin.ietf@gmail.com
Ignas Bagdonas 13. References
Equinix
Email: ibagdona.ietf@gmail.com 13.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
13.2. Informative References
[I-D.herbert-gue-extensions]
Herbert, T., Yong, L., and F. Templin, "Extensions for
Generic UDP Encapsulation", draft-herbert-gue-
extensions-01 (work in progress), October 2016.
[I-D.ietf-nvo3-geneve]
Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic
Network Virtualization Encapsulation", draft-ietf-
nvo3-geneve-14 (work in progress), September 2019.
[I-D.ietf-nvo3-gue]
Herbert, T., Yong, L., and O. Zia, "Generic UDP
Encapsulation", draft-ietf-nvo3-gue-05 (work in progress),
October 2016.
[I-D.ietf-nvo3-vxlan-gpe]
Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol
Extension for VXLAN", draft-ietf-nvo3-vxlan-gpe-09 (work
in progress), December 2019.
[I-D.smith-vxlan-group-policy]
Smith, M. and L. Kreeger, "VXLAN Group Policy Option",
draft-smith-vxlan-group-policy-05 (work in progress),
October 2018.
[RFC2418] Bradner, S., "IETF Working Group Guidelines and
Procedures", BCP 25, RFC 2418, DOI 10.17487/RFC2418,
September 1998, <https://www.rfc-editor.org/info/rfc2418>.
[RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
Edge-to-Edge (PWE3) Architecture", RFC 3985,
DOI 10.17487/RFC3985, March 2005,
<https://www.rfc-editor.org/info/rfc3985>.
[RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed.,
"Network Service Header (NSH)", RFC 8300,
DOI 10.17487/RFC8300, January 2018,
<https://www.rfc-editor.org/info/rfc8300>.
Author's Address
Sami Boutros (editor)
Ciena
USA
Email: sboutros@ciena.com
 End of changes. 130 change blocks. 
324 lines changed or deleted 295 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/