Internet-Draft EVPN C-MAC Reduction November 2020
Wang & Chen Expires 18 May 2021 [Page]
Intended Status:
Standards Track
Y. Wang
ZTE Corporation
R. Chen
ZTE Corporation

Reduction of EVPN C-MAC Overload


When there are too many customer-MACs (C-MACs), the RRs and/or ASBRs will be overloaded by the RT-2 routes for these MACs according to [RFC7432]. This issue can be simply solved by making the remote C-MAC entries learnt via data-plane MAC learning (like what PBB VPLS have done since [RFC7041]) rather than received from RT-2 routes. This simplified solution will works as well as PBB VPLS. But this simplified solution will lose many important features that based on the ESI concept. Because the ingress-ESI can't be learnt via data-plane MAC learning at the egress PE. So when the data packets is forwarded following these MAC entries, they can't benefit from the EAD/EVI routes as per RFC7432. So the All-Active Redundancy mode for ES can't be supported. This make the simplified solution can't work as well as PBB EVPN ([RFC7623]).

This document proposes some new extensions to [RFC7432] to achieve all-active mode ES redundancy on TPEs and reduce the C-MAC loads for RRs and ASBRs at the same time. The new solution will work even more better than PBB EVPN under the help of these extensions, especially when there is no deployment of MPLS dataplane.

Furthermore, it naturally brings the benefits of high scalability, faster network convergence, and reduced operational complexity, and we call it light-weighted EVPNs because of these advantages.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 18 May 2021.

Table of Contents

1. Introduction

In [RFC7432], the C-MACs is advertised via RT-2 route. This behavior is inheritted by [RFC8365] and [I-D.ietf-bess-srv6-services]. but in order to solve the C-MAC overload problem for RRs and ASBRs, we have to return to a PBB-like dataplane C-MAC learning procedures.

We discuss all the requirements for a light-weighted EVPN solution which pushes no C-MAC entries into the backbone network in Section 2. Note that some of these requirements is not supported well by PBB EVPN.

In this document, the light-weighted EVPN solutions are also called as EVPN-lite for short. A total of four EVPN-lite solutions are proposed in Section 3. These solutions are VXLAN over EVPN IP-VRF, light-weighted VXLAN EVPN, light-weighted MPLS EVPN, light-weighted SRv6 EVPN.

Note that the VXLAN-based EVPN-lite and MPLS-based EVPN-lite are both a evolution of VXLAN over IP-VRF. The former takes the MPLS factors off the VXLAN over IP-VRF solution, and becomes a pure VXLAN solution. The latter takes the VXLAN factors off the VXLAN over IP-VRF solution, and becomes a pure MPLS solution.

In order to compare these five solutions with [RFC7348] and [RFC7623] whose C-MAC entries are also not pushed into the backbone network, two terms are introduced in this document, because the comparisons need to be done in unified terminology. One term is "Global ESI Indicator (GEI)", which is called as B-MAC in PBB EVPN. The other term is "EVI's Global Dicreminator (EGD)", which is called as I-SID in PBB EVPN.

Note that the EVI here corresponds to the I-Component of [RFC7623], not the B-Component. In fact, there will be no typical B-components in some of the above seven solutions.

Note that the GEI and EGD in different EVPN-lite solutions are very different. The details will be described in Section 3.

On the basis of GEI concept, then we define two route-types for EVPN-lite: The first route type is GEI/ES route, which is called as RT-2 route in PBB EVPN. The second route type is GEI/EVI route, which is called as EAD/EVI roue in [RFC7432].

The details of these terms are described in Section 1.1.

1.1. Terminology

Most of the terminology used in this documents comes from [RFC7432] and [I-D.ietf-bess-srv6-services] except for the following:

  • Light-weighted EVPN: The EVPN solution with high scalability and reduced operational complexity.
  • EVPN-lite: The Light-weighted EVPN is also called EVPN-lite for short.
  • C-MAC: Customer MAC, it is the same as the C-MAC of PBB EVPN.
  • ISID: a broadcast domain identifier in PBB I-Component.
  • LDV: Local Discreminating Value. It is similar to the Local Discreminating Value of type 3 ESI.
  • GDV: Global Discreminating Value. An identifier with global uniqueness.
  • EGD: EVI-GDV, an EVI's Global Discreminator, it is a GDV for an EVI instance. A EGD is used to idenfify an EVPN Instance (EVI) in data plane. The EGD is a Global Discreminating Value (GDV) of that EVI, so it is also the abbreviation of EVI-GDV. e.g. The EGD of [RFC7348] is a global VNI.
  • ESI Indicator: A Global ID for an ESI. Note that different PE may assign different ESI-indicator for the same ESI, espacially when the ES redundancy mode is single-active. e.g. The ESI indicator of [RFC7623] is B-MAC.
  • GEI: Global ESI Indicator. It is the same as the "ESI Indicator" except for the emphasization to its global uniqueness. A GEI is used in data plane to identify an ESI, because it have global uniqueness across the service domain of a corresponding EVPN Instance (EVI). But an ESI may have a few GEIs, each for a TPE, espacially in the single-active mode of ES redundancy. And in E-Tree scenarios, an ESI may have two GEIs on the same PE, one for Root ACs, one for Leaf ACs. e.g. The GEIs for an ESI of [RFC8317] is two B-MACs, one for root ACs, one for Leaf ACs.
  • GEI/ES: The EVPN route which is used to advertise the relation between ESE and its GEI. Note that the GEI/ES route is advertised per ESI basis on a specified PE. In PBB EVPN, the GEI/ES route is the MAC Advertisement Route. Note that different solutions may have different GEI/ES routes. Note that a GEI/ES don't have to be an EAD/ES route.
  • EAD/EVI: An Ethernet A-D route per EVI.
  • GEI/EVI: The EVPN route which is used to advertise the relation between <ESI/GEI, EVI> and its EVPN label and MPLS nexthops. Note that in PBB EVPN, such route is not used. Note that different solutions may have different GEI/EVI routes. Note that a GEI/EVI don't have to be an EAD/EVI route.
  • ARG.EGD: The argument part of a SID of the End.ESI function is called as ARG.EGD, because the value of that argument will be a EGD.
  • RT-2: MAC/IP Advertise Route.
  • MAC Entry: An entry in the EVPN MAC table in data-plane.
  • ESI SID: An SRv6 SID whose function type is End.ESI. Note that when the ESI is all-active mode, the ESI SID is the same on all PEs of that ES, according to Section 3.5. In such case, the ESI SID can be called as ES anycast SID too.
  • ESI IP: An End.ESI SID with its Argument part being set to zero.
  • VXLAN EVPN: EVPN per [RFC8365].
  • EVPN VXLAN: A broadcast domain per [RFC7348], but use IMET routes of [RFC8365] to construct VXLAN tunnels. Note that an EVPN VXLAN will not use EAD/EVI routes or MAC/IP Advertisement Routes.
  • SPE - Stitching PE, the PEs to do label swapping operation for the EVPN labels. It is similar to the SPE of MS-PWs.
  • TPE - Target PE, the PEs to do EVPN forwarding for the overlay network.
  • PLR - A router at the point of local repair in the underlay network. In egress node protection, it is the penultimate hop router on an anycast tunnel.

2. Requirements

EVPN C-MAC Reduction should be provided together with the following requirements:

2.1. No C-MAC Awareness in the Backbone

In typical operation, an EVPN PE sends a BGP MAC Advertisement route per C-MAC address. In certain applications, this poses scalability challenges, as is the case in data center interconnect (DCI) scenarios where the number of virtual machines (VMs), and hence the number of C-MAC addresses, can be in the millions. This is called as C-MAC overload of DC Backbone. In such scenarios, it is required to reduce the number of BGP MAC Advertisement routes by relying on a 'EVPN-lite' scheme, as is provided by ESI and its equivalents (e.g. Pseudo B-MAC, ESI IP).

2.2. EVPN IRB Support

The PBB-VPLS/PBB-EVPN is not friendly to IRB usecase because of its complicated Protocol Stack, so it is used just in pure L2VPN usecase up to now in the industry.

The solution should provider efficient forwarding performance in EVPN IRB use cases.

2.3. Unified Encapsulation per Scenario

PBB EVPN, especially the MPLS encapsulation of its B-VPLS, is typically not used in DC Scenario. So we bring PBB and MPLS encapsulation to DC Backbone just due to the C-MAC overload problem. EVPN IRB is widely deplyed in DC scenarios, but PBB EVPN is not friendly for EVPN IRB use cases. So we have to use different solutions in EVPN IRB and C-MAC reduction use cases. We believe that if we choose VXLAN/Geneve data-plane, we will prefer to use the same data-plane in all use cases, e.g. EVPN IRB, C-MAC reduction. So it is necessary to make NVO3/MPLS/SRv6 EVPN to support Section 2.1 in order to provider a unified solution for data center and other secenarios.

2.4. ESI Features Remain Supported

Two redundancy modes are defined in [RFC7432]. They are All-Active mode and Single-Active mode.

In All-active mode, the C-MAC movement among the different adjacent PE nodes of the same ESI should not be considered as C-MAC mobility. In Single-Active mode, such movements can be considered as C-MAC mobility.

2.5. Flexible Multi-homing Remains Supported

Flexible multi-homing means that different ES instances can have different adjacent-PEs. We call all the adjacent-PEs of the same ES instances as that ES's location-set in this document. Flexible multi-homing means that different ES can have different location-set.

For example, ES1's location-set is {PE1}, ES2's location-set is {PE2, PE3}, ES3's location-set is {PE1, PE3}, and ES4's location-set is {PE2,PE4}.

2.6. C-MAC Address Learning and Confinement

In EVPN, all the PE nodes participating in the same EVPN instance are exposed to all the C-MAC addresses learned by any one of these PE nodes because a C-MAC learned by one of the PE nodes is advertised in BGP to other PE nodes in that EVPN instance. This is the case even if some of the PE nodes for that EVPN instance are not involved in forwarding traffic to, or from, these C-MAC addresses. Even if an implementation does not install hardware forwarding entries for C-MAC addresses that are not part of active traffic flows on that PE, the device memory is still consumed by keeping record of the C-MAC addresses in the routing information base (RIB) table. In network applications with millions of C-MAC addresses, this introduces a non- trivial waste of PE resources. As such, it is required to confine the scope of visibility of C-MAC addresses to only those PE nodes that are actively involved in forwarding traffic to, or from, these addresses.

2.7. No C-MAC Flushing for All-Active ESes

Just as in [RFC7432], it is required to avoid C-MAC address flushing upon link, port, or node failure for remote All-Active multihomed segments.

2.8. Independent C-MAC Flushing for Single-Active ESes

Just as in [RFC7432], upon singel-active ES1's link or port failure, the C-MACs of other single-active ESes from the same PE will not be flushed.

2.9. Independent Convergency per <ESI, EVI>

When the physical port of an All-Active ES works well, but a single Ethernet Tag ID (ETI) of that ES fails, The traffic to that ETI of that ES will be re-routed to other adjacent PE of the same ES, but the traffic to other ETIs of the same ES will not be affected.

Note that when AC (ES link) fails but PE node still works well, there should not be steady bypassing traffic either. The steady bypassing problem is discussed in [].

2.10. Route Aggregation and Default Route in Backbone

The routes per ESIs can be aggregated in Backbone network. Even the default route should be supported when the B-Component is an EVPN IP-VRF (e.g. in VXLAN over IP-VRF solutions).

2.11. Applicable to SRv6 BE use-cases

The leight-weighted SRv6 EVPN mechanisms should be applicable to SRv6 BE use-cases, not just the SRv6 TE use-cases.

2.12. ARP Suppression

The ARP suppression requires <IP,MAC> entries to be steadily held on all TPEs, So it conflicts with Section 2.6. But if the C-MAC confinement requirements is not so important in some scenarios, The ARP Suppression can be activated. This is an option.

2.13. ESI Indicator Aggregation

When two ESes are attached to the same group of PEs, they can share the same ESI indicator. But this will bring out some issues too. One of these issues is that they may be attached to different groups of PEs in the future. Another issue is that when only one of the ESes fails, the ESI indicator can't be withdrawn by that PE, so the steady bypass of that ES arises immediately after its failture on that PE. If these issues are not so important in some scenarios, The ESI Indicator Aggregation may be activated. This is an option.

Note that when ESI Indicator Aggregation is activated, the local-bias ES split-horizon procedures or its variations (like what [I-D.eastlake-bess-evpn-vxlan-bypass-vtep] does) should be used.

Note that ESI Indicator Aggregation works well with single-active ESIs (see Section 4.5), its steadby bypassing problem will arise with all-active ESIs only.

3. Solution Overview

3.1. Common Overview

We assign a Global Discreminator EGD1 to an EVI instance EVI1, the EGD1 is a number consists of N bits. We assign an ESI-indicator GEI1 to ESI1 on PE1, and we assign an ESI-indicator GEI2 to ESI1 on PE2. We call the relationship between ESI1 and its two ESI-indicators as ESI1_GEI1 and ESI1_GEI2 respectively. The EGD and GEIs MUST have global uniqueness in EVI1's service domain.

                   PE1           |          |
              +-------------+    |          |
              | ESI1_GEI1   |    |          |         PE3
             /|             |----|          |   +-------------+
            / |             |    | IP/MPLS  |   |             |
       LAG /  +-------------+    | Backbone |   |   ESI2_GEI3 |---CE2
   CE1=====                      |   with   |   |             |
           \  +-------------+    |   EVPN   |---|             |
            \ |             |    |   RRs    |   +-------------+
             \|             |----|   and    |
              | ESI1_GEI2   |    |   SPEs   |
              +-------------+    |          |
                   PE2           |          |
Figure 1: EVPN MAC Reduction Usecase

We use IMET routes to build a broadcast-list. The broadcast-list is used to forward BUM traffics. The data-plane MAC learning for BUM traffics produces the first batch of C-MAC entries. The subsequent C-MAC entries can be learnt from Unicast traffics and/or BUM traffics. It is clear that we don't use MAC/IP routes to advertise C-MAC entries as usual, that is for fear that the RRs and/or ASBRs are overloaded by these C-MACs.

3.2. VXLAN over IP-VRF overview

We can replace the data plane of PBB EVPN with VXLAN data plane, but keep its control plane and management plane unchanged from [RFC7623]. This is called Pseudo PBB EVPN, and its details are defined in [Revision-01_section3.2].

But it is a bit weird to use PBB EVPN management plane together with VXLAN over IP-VRF data-plane. So we can make a step forward and use VXLAN over IP-VRF management/control plane instead.

We configure ESI-IP and VTEP IP instead of B-MAC, VXLAN instance instead of I-VPLS, Backbone IP-VRF instead of B-VPLS, VNI instead of I-SID.

The ESI-IP and VTEP IP are advertised by RT-5 routes, not RT-2 routes. Although these IPs can be carried in the RT-2 route's IP address field too, it may be a bit weird. So we choose RT-5 route to do the advertisement.

Note that the GEI/ES route in VXLAN over IP-VRF will be RT-5 route, And the ESI may be not advertised together with its GEI.

3.3. VXLAN Solution Overview

Although the VXLAN encapsulation is used in Pseudo PBB-EVPN, the MPLS data plane is also needed, because that the B-Component of Pseudo PBB-EVPN is an IP-VRF. So we introduce a completely VXLAN encapsulation in this section, it is called as VXLAN solution.

In VXLAN solution, the GEI is composed of a VTEP IP and an ESI label. It is illustrated as the following:

|           32 bits                     |      16 bits      |
|           VTEP IP                     |     ESI Label     |

Figure 2: VXLAN GEI Format

The VTEP IP is PE1/PE2/PE3's IP address, the ESI label is a local discreminating value (LDV) that is used to identify a ESI. So the GEI has global uniqueness in the EVPN domain.

Note that the GEIs (on PE1 and PE2) of ESI1 don't have to be the same. But if we let them to be the same through configuration, it will work well too.

We use the following encapsulation in this solution:

| IPv4 Header (SIP=VTEP IP)              |
| UDP Header (Source Port ^= ESI label) |
| VXLAN Header (VNI=EGD)                 |
| Ethernet Header                        |
| Ethernet Payload                       |
| Ethernet FCS                           |
Figure 3: VXLAN Encapsulation for EVPN-lite

Note that only the ingress GEI will be encapsulated in data-plane, the VTEP IP of ingress GEI is encapsulated as source IP, the ESI label is encrypted into the source port.

Note that the cryptographic key for ESI label of the inner packet's intrinsic entropy. The intrinsic entropy is a 16 bits unsigned integer that is computed from nothing but the inner packet's fields. When the ESI label is encrypted into the source port, then the source port will contain both the context entropy and the intrinsic entropy of the inner packet. The source port is now called as the compositive entropy of the inner packet.

The GEI/ES route of VXLAN-based EVPN-lite is the RT-1 per ES route. The ESI-label attribute is used to carry the GEI1's ESI-label field. The OPE TLV or nexthop is used to carry the GEI1's VTEP IP field.

On receiving the RT-1 per ESI1 route R1 from PE1, PE3 will install a GEI mapping entry ME1 into the data-plane. On receiving a VXLAN packet VP1 of Figure 3 format, PE3 recomputes the intrinsic entropy of VP1 by the same algorithm, the PE3 decrypts GEI1's ESI label part from VP1's UDP source port using the intrinsic entropy. Then PE3 learn's VP1's inner S-MAC MAC1 whose destination ESI is ESI1 according to the GEI mapping entry ME1.

Note that the simplest encryption algorithm may be the bitwise XOR. And it is good enough for our use case.

On receiving a ethernet packet EP1 whose D-MAC is MAC1, PE3 will forwarded EP1 to PE1/PE2 following ESI1's RT-1 per EVI routes for EVI1.

On receiving the RT-1 per ESI route R2 from PE1, PE2 will install a GEI mapping entry ME2 into the data-plane. Then, on receiving a VXLAN packet VP2 of Figure 3 format, when VP2 is about to be forwarded to ESI1, PE2 will drop VP2 because that its ingress GEI is ESI1 (according to the GEI mapping entry ME2) too.

The conceptual comparisons between light-weighted VXLAN EVPN and (Pseudo-) PBB EVPN is illustrated in [Revision-01_section3.6].

3.4. MPLS Solution Overview

In MPLS EVPN control plane, we use a 24 bits unsigned number as the EGD of EVI1, and it has global uniqueness in EVI1's service domain. In data plane, we use QinQ tags to carry the EGD.

We use a Global Unique Label (GUL) to identify a ESI in EVI1's service domain. So the ESI-GUL is also its Global ESI Indicator. The ESI-GULs are avertised through RT-1 per ES routes, and they are considered to be an ESI-label by these routes. The label in RT-3 route's PMSI-Tunnel Attribute (PTA-Label) whose tunnel type is ingress replication is called as Ingress Replication Multicast Label (IRML) in this document.

We use the following encapsulation in MPLS-based EVPN-lite:

     Format #1                    Format #2
+-----------------------+     +----------------------------+
| PSN Labels            |     | PSN Labels                 |
+-----------------------+     +----------------------------+
| IRML (EVI1)           |     | Destination-ESI GUL (ESI1) |
+-----------------------+     +----------------------------+
| Source-ESI GUL (ESI1) |     | Source-ESI GUL (ESI2)      |
+-----------------------+     +----------------------------+
| Ethernet Header       |     | Ethernet Header (EVI1)     |
+-----------------------+     +----------------------------+
| Ethernet Payload      |     | Ethernet Payload           |
+-----------------------+     +----------------------------+
| Ethernet FCS          |     | Ethernet FCS               |
+-----------------------+     +----------------------------+
Figure 4: MPLS Encapsulation for EVPN-lite

Note that the GUL can be a single Label Stack Entry (LSE), in such case, it should be allocated in DCB label space. Given that the ESIs and vESIs may be too many to be allocated in DCB in certain scenarios, so the GUL should be allocated in a few context-specific label spaces, each identified by a Context Label Space ID (CLS-ID) per [I-D.ietf-bess-mvpn-evpn-aggregation-label] in such case. In such case, the ESI-GUL is the entirety of ESI-label and its Context Label Space ID (CLS-ID), so it means two LSEs in the Label Stack at that time.

Note that the ESI GULs are assigned by a center authority, which may be a DC controller or an administrator.

Note that the ESI-label (ESI-GUL) should be pushed onto the Label Stack whether the packet is BUM or not. The ESI-GUL can't identify the EVPN Instance EVI1, so we have to use the EGD in the inner ethernet header of "Format #2" to find EVI1 out.

Note that the GUL concept is very different with the "upstream-assigned label (UAL)" concept. Because that when a SPE receives a GUL from a remote PE, the GUL is considered as an outgoing-label to that remote PE, and although the GUL is also considered as a incoming-label of the current SPE, and the label operation for the GUL will be a "swap", to be precise, The SPE will swap it to itself and then push the MPLS Label Stack to that remote PE. When the same GUL is received from different remote PEs, MPLS ECMP or FRR procedures will be applied.

So when the GUL is two LSEs in the label stack, we can say that the Context-specific Label Space (CLS) of the ESI-label (inside the GUL) takes the role of B-MAC of PBB EVPN, and the CLS-ID label inside the GUL takes the role of the B-VPLS label of PBB EVPN. So no B-VPLS instances will be found here.

Note that the GEI/ES route of MPLS-based EVPN-lite is the RT-1 per ES route.

Note that the light-weighted MPLS EVPN solutions can be used whether or not the SR-MPLS LSPs are used in the underlay network.

The conceptual comparisons between light-weighted MPLS EVPN and (Pseudo-) PBB EVPN is illustrated in [Revision-01_section3.6].

3.5. SRv6 Solution Overview

We introduce a SRv6 function named End.ESI to carry the ESI-indicator in SRv6 dataplane. A SID with the End.ESI function is called as an "ESI SID" in this document. The GEI is the locator and fuction part of its corresponding ESI SID. The argument part of the ESI SID is the EGD for an EVI. The EGD works like the function part of an End.DT2U/DT2M SID. But the EGD has a global meaning like a global VNI or an PBB ISID but the function part for an End.DT2U/DT2M SID typically is only a local discreminator on the egress PE. The argument part of the ESI SID is called as ARG.EGD in this document, where the EGD is the abbreviation of EVI-GDV.

The SRv6 SID in IMET route is an End.DT2M SID with a zero argument length. The GEI1 and GEI2 are SRv6 SID of End.ESI function that is defined in the following figure. We use IGP protocols to advertise GEI1 and GEI2 to PE3 respectively in SRv6 underlay. So we don't use EAD/ES route or EAD/EVI route in SRv6 EVPN in this section. If ESI1 is single-active mode, GEI1 is different from GEI2, but if ESI1 is all-active mode, GEI1 is the same as GEI2.

Note that when PE1 node fails and the ESI is all active, the PLR node will do underlay anycast FRR switching for GEI1(=GEI2). This will bring out fast network convergency.

Note that when the PE-CE link of GEI1 fails, the IGP route of GEI1 will be withdrawn, So there will be no steady bypassing for that ES, but a temporary bypassing can be performed to further improve the convergency.

    |       ESI-Indicator(128-N bits)     |        N bits           |
    |    Block   |   Node     | ESI.LDV   |        ARG.EGD          |

Figure 5: End.ESI SID Format

Note that an ESI-indicator is composed of Locator and Function, an ESI IP is an 128 bits SID with a zero argument. The function part is a Local Discreminating Value (LDV) on that PE for the ESI. The argument part is a EVI-GDV (EGD) for the EVPN Instance. The argument part is also called ARG.EGD in this document.

Note that although the EGD can be carried in the VLAN-IDs of the inner ethernet packet (like MPLS EVPN-lite solutions), it will be better to make it be carried in the SRv6 SID.

The conceptual comparisons between light-weighted SRv6 EVPN and (Pseudo-) PBB EVPN is illustrated in [Revision-01_section3.6].

4. Dataplane-specific Procedures

4.1. Packet Walkthrough

[PE1 forward ARP Request to PE2/PE3]
  • When CE1 requests CE2's ARP, PE1 will receive the ARP Request BUM1 from a AC (say AC1) of ESI1. PE1 will forward the ARP Request following the broadcast-list of AC1's EVI instance(say EVI1). The broadcast-list is constructed by IMET routes from PE2/PE3.

    PE1 will forward the ARP Request to PE2/PE3. The ARP Request is encapsulated with GEI1 and EVI1_GDV1. The inner SMAC of the ARP request is M1 which is CE1's MAC address.

[PE2/PE3's Dataplane MAC Learning]
  • When PE2/PE3 receives the ARP Request packet BUM1, they do dataplane MAC learning independently. They will learn that M1 is behind GEI1.

    Note that when PE2 learns that M1 is behind GEI1, it will assume that M1 is behind the local AC whose ESI-indicator is GEI1 too. The local AC may have more higher priority than the remote one.

    After the dataplane MAC learning, the ARP request packet BUM1 is broadcasted to the local ACs, behind one of which is CE2.

[PE2 Discard ARP Request to CE1]
  • On receiving BUM1 from PE1, PE2 use the ingress GEI information in BUM1 to determine its ingress ESI ESI1, When ESI1 is all-active mode and PE2 is about to forward the ARP request to CE1, PE2 will find that the ESI for the outgoing AC is also ESI1, so PE2 discards it for ESI loop-free considerations.

    When ESI1 is single-active mode, the outgoing AC may be in blocking state, otherwise its corresponding sub-interface on CE1 will take charge of packet-drop behavior instead. So alghough the ESI for the outgoing AC is not the same as ESI1, no loop will arise in the Ethernet Segment.

[PE3 Forward ARP Replay to PE1/PE2]
  • When CE2 replies to CE1 for the ARP request, PE3 will forward the ARP reply U1 according to the MAC entry M1 learned previously as above.

    PE3 will forward the ARP reply U1 to PE1 or PE2 according to ESI1's RT-1 per EVI routes and RT-1 per ES routes:

    When ESI1 is all-active mode, GEI1 may be the same as GEI2, in such case, we call both of them GEI21 instead. The traffics to M1 will be load-balanced between PE1 and PE2. Because that GEI21 is advertised by both PE1 and PE2l.

[PE1 Forward ARP Replay to CE1]
  • Whe PE1 received the ARP reply packet U1 from PE3, PE1 first match the packet to the its EVI instance EVI1 by U1's EGD information. And PE1 will not discard it because the egress ESI is not the same as the ingress ESI which is determined by U1's GEI information.

4.2. VXLAN over IP-VRF

We don't use multicast IP address as the underlay destination IP address of the BUM packets. We use the VTEP IP address per each egress PE instead. But the IMET route is not advertised for the backbone IP-VRF instance, they are advertised for the VXLAN instance. We use the Originator Router's IP (ORIP) field of the IMET route as the underlay destination IP address instead of the nexthop. It means that ORIP should be advertised for the backbone IP-VRF instance, and the ORIP should be the VTEP IP address of corresponding PE.

There are no other changes from Pseudo PBB-EVPN in data-plane. So it is also a MPLS-based solution.

4.3. NVO3-specific EVPN-lite Procedures

In Section 3.3, We use VXLAN encapsulation as an example for NVO3 EVPN. But when GENEVE or MPLSoGRE encapsulation is used, the ESI-label will have its own space in packet headers, so we don't have to encapsulate ESI-label in UDP Source port.

Note that in step #4 the egress GEI is not encapsulated in U1. U1's underlay DIP will be determined by these RT-1 per EVI routes.

4.4. MPLS-specific EVPN-lite Procedures

According to [RFC7432], When the IMET route's PTA's tunnel type is ingress replication, the ESI-label is considered to be downstream-assigned too. Because that nothing of RT-1 per ES route will indicate whether the ESI-label is upstream-assigned or not.

Alghough ESI-GUL can be a single LSE or two LSEs in the Label Stack, we assume that it is a single LSE by default in this section, it is for simplification purpose.


In Step #1, "Format #1" of Figure 4 will be used.

Although the Ingress Replication Multicat Label (IRML) of "Format #1" can identify EVI1 by itself, we suppose that the ethernet header of it should also carry EGD as what [M4] does.

Note that there isn't a B-VPLS here, so the IRML identifies the EVI1 itself. The EVI1 here equals I-VPLS of PBB EVPN.

Note that when that ARP Request packet comes from a SHD (single-homed device), the ESI of its AC will be null. The Source-ESI GUL in "Format #1" will be replaced with a MPLS label identifying the ingress TPE. When we assume that the underlay network is a SR-MPLS network, that TPE-identifying label can be the node SID label of that ingress TPE. This method follows [], and the context of the TPE-identifying label is identified by the EVI1's IRML of "Format #1".

Note that the TPE-identifying label typically will do nothing to the all-active ESes, they are used just for the single-homed ESes. But when Section 2.13 is activated, and all ESIs share the same ESI indicator, an anycast TPE-identifying label in the DCB can be used as that ESI indicator.


In Step #2, "Format #1" of Figure 4 will be received. PE3 knows the packet is for EVI1 with the help of the IRML label. Then PE3 can learn the relation between the ingress-GEI (ingress-ESI GUL) and S-MAC of BUM1 directly, no GEI to ESI lookup needed.


In Step #3, PE2 can compare the ingress-GEI (ingress-ESI GUL) of BUM1 and the egress-GEI (ESI-GUL of outgoing AC) directly, no GEI to ESI lookup needed.


In Step #4, "Format #2" of Figure 4 will be used. The source-ESI GUL, from which the corresponding MAC entry M1 is previously learnt, will be encapsulated as the destination-ESI GUL directly. No GEI to ESI lookup needed only if we don't care the requirements of Section 2.9. Otherwise we should refer the corresponding RT-1 per EVI routes of ESI1 to forward the packet. These RT-1 per EVI routes are advertised for EVI1, so the Ethernet Tag ID (ETI) of these routes don't have to be the EGD.

Note that when ESI1 is single-active mode, ESI-GUL of ESI1 will be different on PE1 and PE2. But the MAC entry M1 will use the newest one only, the swithover between them is called as MAC-move.


In Step #5, Whe PE1 received the ARP reply packet from PE3, PE1 first match the packet to ESI1 by Destination-ESI GUL, then match the packet to the EVI instance EVI1 by the QinQ tags of Ethernet header.

Note that we suppose that the original tags from ingress AC will be processed following the Raw mode per [RFC4448]. Although the tagged mode can be used technically. Note that the original tags (if they are kept in the packet) will be the inner tags of the EGD.

Note that when RT-1 per EVI route are used, as specified in [M4]. There is no need to carry EGD in unicast data-packets too.

4.5. SRv6-specific EVPN-lite Procedures


In Step #1, PE1 will forward the ARP Request to PE2/PE3 with the following SRv6 BE encapsulation: It's underlay Source IP is the End.ESI SID on PE1 for ESI1; It's underlay Destination IP is the End.DT2M SID on PE2/PE3. The locator and function part of the End.ESI SID is GEI1. The Argument part of the End.ESI SID is 0.

Note that the underlay SIP will be the End.DT2U SID (because they don't need an ESI SID) for the single-homed ingress ACs. The multi-homed ingress ACs with single-active behavior may not be assigned with an dedicated ESI-indicator either. In such situations, the underlay SIP will be the End.DT2U SID too. Note that in such situations, the ESI indicator of all single-active ESIs for the same EVI are aggregated into the same IPv6 address.


In Step #3, PE2 can compare the ingress-GEI of BUM1 and the GEI of outgoing AC directly, no GEI to ESI lookup needed.


In Step #4, PE3 will forward the ARP reply to PE1 with the following SRv6 BE encapsulation: It's underlay Source IP is the End.ESI SID on PE3 for ESI2; It's underlay Destination IP is the End.ESI SID on PE1 for ESI1 according to the MAC entry M1. The ARG.EGD for the End.ESI SID in DIP is the EGD configured on PE3. Note that the EGD for the same EVI is configured with the same value on PE1/PE2/PE3.

When ESI1 is all-active mode, GEI1 will be the same as GEI2, so we call both of them GEI21 instead. The traffics to M1 will be load-balanced between PE1 and PE2 by the underlay network on PE3. Because GEI21 is advertised by both PE1 and PE2 in the underlay IGP protocol.

Note that if the DIP is the anycast node SID of PE1 and PE2, when the PE-CE link of ESI1 fails, the traffic will be steadily bypassed untill that link recovers again.


In Step #5, Whe PE1 received the SRv6 encapsulated ARP reply packet from PE3, PE1 first match the packet to the End.ESI SID of ESI1 by DIP, then match the packet to the EVI instance EVI1 by ARG.EGD.

4.5.1. End.ESI Function and Arg.EGD

The "Endpoint with decapsulation and ESI-specific L2 table forwarding" behavior (End.ESI for short) is a variant of the End.DX2 behavior.

Two of the applications of the End.ESI behavior are the EVPN VPLS [RFC7432] and the EVPN ETREE [RFC8317]use-cases.

Any SID instance of this behavior is associated with an ESI E. The behavior also takes an argument: "Arg.EGD". This argument provides a local mapping to an EVI V and its L2 tabel T. The outgoing AC-interface corresponding to <E,V> (ESI E and EVI V) is OIF.

The End.ESI SID MUST be the last segment in a SR Policy.

When N receives a packet whose IPv6 DA is S and S is a local End.ESI SID, the processing is identical to the End.DX2 behavior except for the Upper-layer header processing which is as follows:

   S01. If (Upper-Layer Header type == 143(Ethernet) ) {
   S02.    Remove the outer IPv6 Header with all its extension headers.
   S03.    Learn the exposed MAC Source Address in L2 Table T.
   S04.    Find out the OIF, and forward the Ethernet frame to the OIF.
   S05. } Else {
   S06.    Process as per Section 4.1.1
               of [I-D.ietf-spring-srv6-network-programming].
   S07. }

Note that the EVI V is determined by the End.ESI SID's ARG.EGD argument.

Note that the MAC learning should not be applied unless the EVI V is an E-LAN service.

Note that the OIF may be found out using the MAC-entries in L2 Table T, when the EVI V is an E-LAN service and the AC-aware service interface is used.

Note that we can use the ARG.EGD to find out whether the EVI V is an E-LAN service or not.

5. Other Considerations

5.1. ESI Indicator Advertisement Optimization

Although we can advertise End.ESI SID in underlay IGP protocols, But it is better to use the SRv6 SID Structure Sub-Sub-TLV to indicate the length of the ARG.EGD in the End.ESI SID at the same time. Otherwise we have to keep the consistence between the length of ARG.EGD and the length of local EGD by means of manual configurations.

So we can use EAD/ES route (or EAD/EVI route) to advertise Global ESI Indicator (GEI) (and EGD), these EAD routes is called as GEI/ES or GEI/EVI route in this document. When the GEI/EVI route is used to advertise GEI, the End.ESI SID is advertised in its SRv6 L2 Service TLV, not in its nexthop. The EGD may be carried in the ARG.EGD field of the End.ESI SID, or it can also be determined from its EVI-RTs.

Either GEI/EVI routes (or GEI/ES) routes will be advertised/imported for Global Routing Table (GRT), so their Route-Targets (RT) will be configured with GRT. Because there isn't a dedicated B-component like PBB VPLS and PBB EVPN. Note that the GEI/EVI routes can be installed as /128 routes and the ARG.EGD part can be set to the actual EGD of the corresponding EVI. In such case, when a C-MAC is learnt over an End.ESI SID (as IPv6 SA) in the data-plane, the ARG.EGD field of that SID should be set to the EVI's EGD when the C-MAC entry is installed.

Although GEIs is imported to GRT, they are awared only on PE nodes, the transit nodes in underlay network won't be aware of GEIs (they can aware the common prefix of these GEIs) in order to reduce the FIB consumption. We can use the argument length in the SRv6 SID Structure Sub-Sub-TLV to check whether the EGD is too big for the End.ESI SID, So we can avoid the destruction to the function part of the End.ESI and we can use flexible EGD length.

5.2. C-MAC Flush Notification Procedure

The withdraw of GEI Advertisement can be used as C-MAC flush notification like what have been done by [RFC8317] and [I-D.ietf-bess-pbb-evpn-isid-cmacflush].

Note that even if the GEI/EVI routes of Section 5.1 are not advertised, the withdraw of those GEI/EVI route can still be used as a C-MAC flush notification of their <ESI,EVI>.

5.3. E-Tree Support Considerations

E-tree Supprot extensions is similar to [RFC8317] section 5 except for the following notable differences: The leaf B-MACs are replaced by leaf GEIs, the root B-MACs are replaced by root GEIs. the PBB encapsulation is replaced by other encapsulations, the B-component is replaced by an IP-VRF or the underlay GRT. The B-MAC Advertisement Route is replaced by GEI/EVI route or ESI/IP Route.

5.4. EVPN IRB Support Considerations

The dataplane in this draft is no more complex with typical SRv6 EVPN. So it will work as efficient as we should expect in SRv6 EVPN IRB usecase.

5.5. Use End.ESI SID in MAC/IP Advertisement Routes

In [I-D.ietf-bess-srv6-services] the downstream assigned ESI label is encapsulated in the Arg.FE2 part of End.DT2M SID, And the ESI label present as Arg.FE2 only when the egress PE is adjacent with the ingress ESI. So it is difficult (if not impossible) to do data-plane C-MAC learning via End.DT2M SID and its unwarranted Arg.FE2 presence. Alghough upstream assigned ESI label (like NVO3-specific EVPN-lite solutions) may be used to learn ingress ESI-indicator on egress PE node, that will be difficult.

But the End.ESI SID can be used in MAC/IP advertisement route, even if C-MAC overload is not a real threat. By doing this, the data-plane can be unified among these usecases. The details for using End.ESI SID in MAC/IP Advertisement Route will be described in future versions.

5.6. Hierarchical VPLS in EVPN-lite

In hierachical topology (as illustrated in the following figure), the PEs are separated into two groups, the Target PEs (TPEs) and the Superstratum PEs (SPEs).

    ___TPE5___        SPE3       ___TPE4_____
   /AC5       \      /   \      /            \AC4
CE3            \    /     \    /              >=====CE2
   \___         \  /       \  /          ____/AC2
   /AC3          /                       \
CE1         ____/                         \
   \____TPE1                               \___CE6

Figure 6: EVPN-lite H-VPLS

The TPEs works like the IB-BEB-PE in PBB VPLS, the SPE works like the BCB-PE in PBB VPLS. The BCB-PEs in PBB VPLS do BUM replication based on the PBB header. There are no PBB hearder in EVPN-lite solutions, but the SPEs won't learn the C-MACs, which is the same as BCB-PEs in PBB VPLS. The forwarding behaviors of these EVPN-lite solutions are very different from each other:

  • The SPE in Pseudo PBB-EVPN do BUM replication based on the Multicast Group IP address.
  • The SPEs in VXLAN over IP-VRF needn't aware of the BUM packets, because the destination IP address of the BUM packets will be an ingress replication tunnel address to the egress TPE.
  • The SPEs in MPLS-based EVPN-lite don't have to aware of the BUM packets, because that, for IMET routes, they work like the ASBRs in inter-AS option B. In such case, the TPEs do ingress-replication for all other TPEs by themselves.

    The SPEs in MPLS-based EVPN-lite may terminate the IMET routes that were received from their TPEs. These IMET routes are imported into an corresponding BD, but may not be passed through other SPEs, so as not to cause duplicated BUM packets. In such case, take SPE1 for example, there are two split-horizon-groups, one group is TPE1/TPE3/TPE5, another split-horizon-group is SPE1/SPE2. The BUM packets are replicated between different split-horizon-groups. In such case, the TPEs do ingress-replication for its directly connected TPEs and SPEs, not for the indirectly connected TPEs and SPEs. But the unicast packet will not be forwarded by that BD on the SPEs. The unicast packets will be label-swapped in the context-specific label-space for the corresponding GULs.

    Note that the BCB-PE in PBB VPLS is typically supported in the industry, But it seems that the BCB-PE in PBB EVPN is typically not supported in the industry up to now. Because the BCB-PE function can be replaced in MPLS EVPN by a label-swapping operation which is like the inter-AS option B scenarios.

Note that the BUM packets here are defined based on the destination C-MAC addresses.

6. Comparison with Other Solutions

6.1. Questions

We compare EVPN-lite with other solutions in this section. These solutions are:

  • Anycast VTEP IP - [RFC7348] plus IMET routes of [RFC8365], VTEP group of [I-D.eastlake-bess-evpn-vxlan-bypass-vtep] and bypass tunnel of []. Data plane C-MAC learning is used and the RT-2 routes are eliminated. But RT-1 per ES route may still be used for local-bias ES split-horizon.

    Note that the bypass-tunnel is used for the communication between the single-homed CEs of the same VTEP group. The bypass-tunnels should not use anycast VTEP IP as their destination addresses.

  • SRv6 Anycast Node SID - The transplantation of Anycast-VTEP-IP solution in SRv6 data-plane, where the Anycast Node SID is the equivalent of the Anycast VTEP IP address. SRv6 Anycast Node SID is the ultimate Aggregation of ESI indicators.

We use the following questions for these solutions to do the comparison:

No C-MAC Awareness in the Backbone ?
EVPN IRB Support ?
Unified Encapsulation per Scenario ?
ESI Features Remain Supported ?
Flexible Multi-homing Remains Supported ?
C-MAC Address Learning and Confinement ?
No C-MAC Flushing for All-Active ESes ?
Independent C-MAC Flushing for Single-Active ESes ?
[<ESI,EVI> Converge]
Independent Convergency per <ESI, EVI> ?
[Route Aggregation]
Route Aggregation and Default Route in Backbone ?

6.2. Summary Comparisons

We place the detailed comparisons about the answers of these questions for each solution in separated sections, but we place the brief comparisons in the following table:

Note that the comparisons with PBB-EVPN and PBB-VPLS can be found in [Revision-01_section6.2].

7. Security Considerations

Security considerations will be added in future versions.

8. IANA Considerations


IANA is requested to allocate a new code points for the new SRv6 Endpoint Behaviors defined in this document.

| Type | Description | Reference     |
| TBD1 | END.ESI     | This Document |

Figure 7: END.ESI

8.2. Global Unique ESI-label in EAD per ES Route

When we use Global Unique ESI-label in EAD per ES route, especially in ingress-replication use case, It should be explicitly indicated in the EAD per ES route. The details will be added in future versions.

9. Acknowledgements

The authors would like to thank the following for their comments and review of this document:

Ye Shu.

10. Normative References

Zhang, Z., Rosen, E., Lin, W., Li, Z., and I. Wijnands, "MVPN/EVPN Tunnel Aggregation with Common Labels", Work in Progress, Internet-Draft, draft-ietf-bess-mvpn-evpn-aggregation-label-03, , <>.
Dawra, G., Filsfils, C., Talaulikar, K., Raszuk, R., Decraene, B., Zhuang, S., and J. Rabadan, "SRv6 BGP based Overlay services", Work in Progress, Internet-Draft, draft-ietf-bess-srv6-services-05, , <>.
Filsfils, C., Camarillo, P., Leddy, J., Voyer, D., Matsushima, S., and Z. Li, "SRv6 Network Programming", Work in Progress, Internet-Draft, draft-ietf-spring-srv6-network-programming-24, , <>.
Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, "Encapsulation Methods for Transport of Ethernet over MPLS Networks", RFC 4448, DOI 10.17487/RFC4448, , <>.
Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, , <>.
Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, , <>.
Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W. Henderickx, "Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623, , <>.
Sajassi, A., Ed., Salam, S., Drake, J., Uttaro, J., Boutros, S., and J. Rabadan, "Ethernet-Tree (E-Tree) Support in Ethernet VPN (EVPN) and Provider Backbone Bridging EVPN (PBB-EVPN)", RFC 8317, DOI 10.17487/RFC8317, , <>.
Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., Uttaro, J., and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, , <>.

11. Informative References

Eastlake, D., Li, Z., and S. Zhuang, "EVPN VXLAN Bypass VTEP", Work in Progress, Internet-Draft, draft-eastlake-bess-evpn-vxlan-bypass-vtep-06, , <>.
Rabadan, J., Sathappan, S., Nagaraj, K., Miyake, M., and T. Matsuda, "PBB-EVPN ISID-based CMAC-Flush", Work in Progress, Internet-Draft, draft-ietf-bess-pbb-evpn-isid-cmacflush-01, , <>.
Wang, Y., "'SR-MPLS signalling for CSL-based Context VC' in", , <>.
Wang, Y., "'Steady Bypassing Problems' in", , <>.
"Pseudo PBB EVPN", , <>.
"Comparisons of Relative Concepts", , <>.
"Comparisons with PBB EVPN and PBB VPLS", , <>.
Balus, F., Ed., Sajassi, A., Ed., and N. Bitar, Ed., "Extensions to the Virtual Private LAN Service (VPLS) Provider Edge (PE) Model for Provider Backbone Bridging", RFC 7041, DOI 10.17487/RFC7041, , <>.

Authors' Addresses

Yubao Wang
ZTE Corporation
No.68 of Zijinghua Road, Yuhuatai Distinct
Ran Chen
ZTE Corporation
No. 50 Software Ave, Yuhuatai Distinct