[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00

rtgwg                                                              Y. Li
Internet-Draft                                                L. Iannone
Intended status: Informational                       Huawei Technologies
Expires: May 4, 2021                                               J. He
                                            City University of Hong Kong
                                                                 L. Geng
                                                                  P. Liu
                                                            China Mobile
                                                                  Y. Cui
                                                     Tsinghua University
                                                        October 31, 2020

   Architecture of Dynamic-Anycast in Compute First Networking (CFN-


   Compute First Networking (CFN) Dynamic Anycast refers to in-network
   edge computing, where a single service offered by a provider has
   multiple instances attached to multiple edge sites.  In this
   scenario, flows are assigned and consistently forwarded to a specific
   instance through an anycast approach based on the network status as
   well as the status of the different instance.

   This document describes an architecture for the Dynamic Anycast
   (Dyncast) in Compute First Networking (CFN).  It provides an
   overview, a description of the various components, and a workflow
   example showing how to provide a balanced multi-edge based service in
   terms of both computing and networking resources through dynamic
   anycast in real time.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

Li, et al.                 Expires May 4, 2021                  [Page 1]

Internet-Draft          CFN-dyncast Architecture            October 2020

   This Internet-Draft will expire on May 4, 2021.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Definition of Terms . . . . . . . . . . . . . . . . . . . . .   3
   3.  CFN-Dyncast Architecture Overview . . . . . . . . . . . . . .   4
   4.  Architectural Components and Interactions . . . . . . . . . .   5
     4.1.  Service Identity and Bindings . . . . . . . . . . . . . .   5
     4.2.  Service Notification between Instances and CFN node . . .   7
     4.3.  CFN Dyncast Control Plane . . . . . . . . . . . . . . . .   9
     4.4.  Service Demand Dispatching  . . . . . . . . . . . . . . .   9
     4.5.  CFN Dispatcher  . . . . . . . . . . . . . . . . . . . . .  10
   5.  Summary of the key elements of CFN Dyncast Architecture . . .  12
   6.  Conclusion (and call for contributions) . . . . . . . . . . .  13
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  13
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  13
   9.  Informative References  . . . . . . . . . . . . . . . . . . .  14
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  14
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  14

1.  Introduction

   Dynamic anycast in Compute First Networking (CFN-Dyncast) use cases
   and problem statements document
   [I-D.geng-rtgwg-cfn-dyncast-ps-usecase] shows the usage scenarios
   that require an edge to be dynamically selected from multiple edge
   sites to serve an edge computing service demand based on computing
   resource available at the site and network status in real time.
   Multiple edges provide service equivalency and service dynamism in
   CFN.  The current network architecture in edge computing provides
   relatively static service dispatching, for example, to the closest
   edge, or to the server with the most computing resources without

Li, et al.                 Expires May 4, 2021                  [Page 2]

Internet-Draft          CFN-dyncast Architecture            October 2020

   considering the network status.  Dynamic Anycast takes the dynamic
   nature of computing load as well as the network status as metrics for
   deciding flow's service dispatch and at the same time maintains the
   flow affinity in a service life cycle.

   CFN-Dyncast architecture presents an anycast based service and access
   models.  The aim is to solve the problematic aspects of existing
   network layer edge computing service deployment, including the
   unawareness of computing resource information of service, static edge
   selection, isolated network and computing metrics and/or slow refresh
   of status.

   CFN-Dyncast assumes there are multiple equivalent edge instances
   implementing the same single service (think about the same service
   function instantiated on several edge nodes).  A single edge node has
   limited computing resources attached, and different edge nodes may
   have different resources available such as CPU or GPU.  Because
   multiple edge nodes are interconnected and can collaborate with each
   other, it is possible to balance the service load and network load in
   CFN.  Computing resource available to serve a request is usually
   considered the main metric to assign a service demand to an instance
   of the service.  However, the status of the network, in particular
   paths toward the instances, varies over time and may get congested,
   hence, becoming another key attribute to be considered.  CFN-Dyncast
   aims at providing a layer 3 protocol framework able to dispatch the
   service demand to the "best" edge node in terms of both computing
   resources and network status, in real time and no application and/or
   service specific dependencies.

   This document describes the a general architecture for the service
   notification, status update and service dispatch in CFN edge

2.  Definition of Terms

   CFN: Compute First Networking

   SID: Service ID, an anycast IP address representing a service and the
   clients use it to access that service.  SID is independent of which
   service instance serves the service demand.  Usually multiple service
   instances serve a single service.

   BID: Binding ID, an address to reach a service instance for a given
   SID.  It is usually a unicast IP.  A service can be provided by
   multiple service instances with different BID.

   CFN-Dyncast: as defined in [I-D.geng-rtgwg-cfn-dyncast-ps-usecase].

Li, et al.                 Expires May 4, 2021                  [Page 3]

Internet-Draft          CFN-dyncast Architecture            October 2020

3.  CFN-Dyncast Architecture Overview

   Service instances can be hosted on servers, virtual machines, access
   routers or gateway in edge data center.  The CFN node is the glue
   allowing CFN-Dyncast network to provide the capability to exchange
   the information about the computing resource information of service
   instances attached to it, but also to forward flows consistently
   toward such instances.

   Figure 1 shows the architecture of CFN-Dyncast.  CFN nodes are
   usually deployed at the edges of the operator infrastructure, where
   clients are connected.  As such, we can consider that clients are
   logically connected to CFN nodes.  A CFN node has the purpose to
   constantly direct flows coming from clients to an instance of the
   service the flow is supposed to go through.  Service instances are
   initiated at different edge sites, where a CFN node is also running.
   A single service can have a huge number of instances running on
   different CFN nodes.  A "Service ID" (SID) is used to uniquely
   identify a service, at the same time identifying the whole set of
   instances of that specific service, no matter where those instances
   are running.  There can be several instances of the service running
   on the the same CFN node (e.g., one instance per CPU core), there can
   also be on several different CFN nodes (e.g., one instance per PGW-U
   in a 5G network).  Each instance is associated to a "Binding ID"
   indicating where the instance is running.  Hence, there is a dynamic
   binding between an SID (the service) and a set of BIDs (the instances
   of the service) and such bindings are enriched with information
   concerning the network state and the available resources so that at
   each new service request (a new flow) CFN nodes can decide which
   instance is the most appropriate to handle the request.  This
   highlights the anycast part of CFN-Dyncast, since flow are routed
   toward one service end-point among a set of equivalent , i.e., one-

   When a clients sends a service demand, it will be delivered to the
   most appropriate instance of the service attached to a CFN node.  A
   service demand is normally the first packet of a data flow, not
   necessarily an explicit out of band service request.  Once the CFN
   node has decided which instance has to serve the flow, flow affinity
   must be guaranteed, meaning that all packets belonging to the same
   flow have to go through the same service instance.

Li, et al.                 Expires May 4, 2021                  [Page 4]

Internet-Draft          CFN-dyncast Architecture            October 2020

       edge site 1          edge site 2           edge site 3

       +------------+                         +------------+ |
     +-+----------+ |                       +------------+ |-+
     |  service   | |                       |  service   | |
     |  instance  |-+                       |  instance  |-+
     +------------+                         +------------+
           |                                        |
           |           +-----------------+          |
      +----------+     |                 |     +----------+
      |CFN node 1| ----|  Infrastructure |---- |CFN node 3|
      +----------+     |                 |     +----------+
           |           +-----------------+
           |                    |
           |                    |
      +----------+         +----------+
      |    CFN   |         |CFN node 2|
      |Dispatcher|         +----------+
      +----------+              |
           |                    |
           |                    |
         +-----+              +------+
       +------+|            +------+ |
       |client|+            |client|-+
       +------+             +------+

                    Figure 1: CFN-Dyncast Architecture

4.  Architectural Components and Interactions

   Figure 1 also shows that the local components of the architecture are
   service instance, CFN node, CFN dispatcher and client.  The following
   subsections provide an overview of how some of these architectural
   components interact.  The figures accompanying the examples do not
   show the interconnecting infrastructure to avoid making them too

4.1.  Service Identity and Bindings

   As previously stated, the CFN-Dyncast architecture uses Service ID
   (SID) and Binding ID (BID) in order to identify services and their

   Service ID (SID) is an anycast service identifier (which may or may
   not be a routable IP address).  It is used to access a specific
   service no matter which service instance eventually handles the

Li, et al.                 Expires May 4, 2021                  [Page 5]

Internet-Draft          CFN-dyncast Architecture            October 2020

   client's flow.  CFN nodes must be able to know SIDs (and their
   bindings) in advance and must be able to identify which flow needs
   which service.  This can be achieved in different ways, for example,
   use a special range or coding of anycast IP address as SID, or use

   Binding ID (BID) is a unicast IP address.  It is usually the
   interface IP address of a service instance.  Mapping and binding from
   a SID to a BID is dynamic and depends on the computing resousrces and
   network state at the time the service demand is made.  The CFN node
   must be able to guarantee flow affinity, i.e., steering the flow
   always toward the same instance.

   Figure 2 shows an abstract example of the use of SIDs and BIDs.
   There are three services, namely SID1, SID2, and SID3.  In
   particular, SID2 has two instances on different CFN nodes (CFN node 2
   and CFN node 3).  In this case the complete list of bindings (only in
   term of SID and BID, no network or resource state) are:

   o  SID1:BID21

   o  SID2:BID22,BID32

   o  SID3:BID33

Li, et al.                 Expires May 4, 2021                  [Page 6]

Internet-Draft          CFN-dyncast Architecture            October 2020

    SID: Service ID
    BID: Binding ID

                                                    +--------+  service
                                                 +--| BID21  | instance1
                                                 |  +--------+
                              +----------+       |
                       +------|CFN node 2|-------|     SID2
                       |      +----------+       |  +--------+  service
                       |                         +--| BID22  | instance2
                       |                            +--------+
     +------+   +----------+
     |client|---|CFN node 1|                           SID2
     +------+   +----------+                        +--------+  service
                       |                         +--| BID32  | instance3
                       |                         |  +--------+
                       |      +----------+       |
                       +------|CFN node 3|-------|     SID3
                              +----------+       |  +--------+  service
                                                 +--| BID33  | instance4

            Figure 2: CFN-Dyncast Architectural Concept Example

4.2.  Service Notification between Instances and CFN node

   CFN-Dyncast service side is responsible to notify its attaching CFN
   node about the mapping information of SID and BID when a new service
   is instantiated, terminated, or its metrics (e.g., load) change, as
   shown in Figure 3.

Li, et al.                 Expires May 4, 2021                  [Page 7]

Internet-Draft          CFN-dyncast Architecture            October 2020

    SID: Service ID
    BID: Binding ID                          service info
                                       (SID1, BID21, metrics)
                                       (SID2, BID22, metrics)
                                                    +--------+  service
                                                 +--| BID21  | instance1
                                                 |  +--------+
                              +----------+       |
                       +------|CFN node 2|-------|     SID2
                       |      +----------+       |  +--------+  service
                       |                         +--| BID22  | instance2
                       |                            +--------+
     +------+   +----------+
     |client|---|CFN node 1|                           SID2
     +------+   +----------+                        +--------+  service
                       |                         +--| BID32  | instance3
                       |                         |  +--------+
                       |      +----------+       |
                       +------|CFN node 3|-------|     SID3
                              +----------+       |  +--------+  service
                                                 +--| BID33  | instance4

                                         service info
                                    (SID2, BID32, metrics)
                                    (SID3, BID32, metrics)

                Figure 3: CFN-Dyncast Service Notification

   Computing resource information of service instances is key
   information in CFN-Dyncast.  Some of them are relatively static like
   CPU/GPU capacity, and some are very dynamic, for example, CPU/GPU
   utilization, number of sessions associated, number of queuing
   requests.  The service side has to notify and refresh this
   information to its attaching CFN node.  Various ways can be used, for
   instance via protocol or via an API of the management system.
   Conceptually, a CFN node keeps track of the SIDs and computing
   metrics of all service instances attached to it in real-time.

Li, et al.                 Expires May 4, 2021                  [Page 8]

Internet-Draft          CFN-dyncast Architecture            October 2020

4.3.  CFN Dyncast Control Plane

   CFN Dyncast needs a control plane allowing to share information about
   resources and costs.  Through the control plane, CFN nodes share and
   update among themselves the service information and the associated
   computing metrics for the service instances attached to it.  As a
   network node, CFN node also monitors the network state to other CFN
   nodes.  In this way, each CFN node is able to aggregate the
   information and create a complete vision of the resources avaible and
   the cost to reach them.  For instance, for the scenario in Figure 3,
   the different CFN nodes will learn that there exists two instances of
   SID2, each of which has a certain computational capacity expressed in
   the metrics.  Different mechanisms can be used in updating the
   status, for instance, BGP [RFC4760], IGP or controller based

   An important question CFN Dyncast raises is on the different ways to
   represent the computing metrics.  A single digitalized value
   calculated from weighted attributes like CPU/GPU consumption and/or
   number of sessions associated may be the easiest.  However, it may
   not accurately reflect the computing resources of interest.  Multi-
   dimensional variables may give finer information, however the
   structure and the algorithmic processing should be sufficiently
   general to accommodate different type of services (i.e., metrics).

   A second important issue is related to the system stability and
   signaling overhead.  As computing metrics may change very frequently,
   when and how frequent such information should be exchanged among CFN
   nodes should be determined.  A spectrum of approaches can be
   employed, interval based update, threshold update, policy based
   update, etc.

4.4.  Service Demand Dispatching

   Assuming that the set of metric are well defined and that the update
   rate is tailored so to have a stable system, the CFN Dyncast data
   plane has the task to dispatch flows to the "best" service instance.
   When a new flow comes to a CFN ingress, CFN ingress node selects the
   most appropriate CFN egress in terms of the network status and the
   computing resources of the attached service instances and guarantees
   flow affinity for the flow from now on.

   Flow affinity is one of the critical features that CFN-Dyncast should
   support.  The flow affinity means the packets from the same flow for
   a service should always be sent to the same CFN egress to be
   processed by the same service instance.

Li, et al.                 Expires May 4, 2021                  [Page 9]

Internet-Draft          CFN-dyncast Architecture            October 2020

   At the time that the most appropriate CFN egress and service instance
   is determined when a new flow comes, a flow binding table should save
   this flow binding information which may include flow identifier,
   selected CFN node, affinity timeout value, etc.  The subsequent
   packets of the flow are forwarded based on the table.  Figure 4 shows
   an example of what a flow binding table at CFN ingress node can look

    |       Flow Identifier                   |            |        |
    +------+--------+---------+--------+------+ CFN egress | timeout|
    |src_IP| dst_IP |src_port |dst_port|proto |            |        |
    | X    | SID2   |   -     |  8888  | tcp  | CFN node 2 |  xxx   |
    | Y    | SID2   |   -     |  8888  | tcp  | CFN node 3 |  xxx   |

                  Figure 4: Example of flow binding table

   A flow entry in the flow binding table can be identified using the
   classic 5-tuple value.  However, it is worth noting that different
   services may have different granularity of flow identification.  For
   instance, an RTP video streaming may use different port numbers for
   video and audio, and it may be identified as two flows if 5-tuple
   flow identifier is used.  However they certainly should be treated as
   the same flow.  Therefore 3-tuple based flow identifier is more
   suitable for this case.  Hence, it is desired to provide certain
   level of flexibility in identifying flows in order to apply flow

   Flow affinity attributes information can be configured per service in
   advance.  For each service, the information can include the flow
   identifier type, affinity timeout value, etc.  The flow identifier
   type can indicate what are the values, for instance, 5-tuple, 3-tuple
   or anything else that can be used as the flow identifier.  Because we
   deal with single services the matching rules have to be disjoint,
   meaning that two different services need not have non-overlapping
   matching flow set.

4.5.  CFN Dispatcher

   When a CFN node maintains the flow binding table, the memory consumed
   is determined by the number of flows that CFN ingress node handles.
   The ingress node can be an edge data center gateway, hence it may
   cover hundreds of thousands of users and each user may have tens of
   flows.  The memory space consumption on binding table at the CFN

Li, et al.                 Expires May 4, 2021                 [Page 10]

Internet-Draft          CFN-dyncast Architecture            October 2020

   ingress node can be a concern.  To alleviate it, a functional entity
   called CFN Dispatcher can help.

   CFN Dispatcher is deployed closer to the clients and it normally
   handles the flows for a limited number of clients.  In this case, the
   memory space required by the binding table will be much smaller.  CFN
   dispatcher is a client side located entity which directs traffic to
   an CFN egress node.  It is not a CFN node itself, that is to say, it
   does not participate in the status update about network and computing
   metrics among CFN nodes.  CFN dispatcher does not determine the best
   CFN egress to forward packets for a new flow by itself.  It has to
   learn such information from a CFN node and maintains it to ensure the
   flow affinity for the subsequent packets.  In this way, the CFN node
   simply selects the most appropriate egress for the new flows and
   informs CFN dispatcher in explicit or implicit way.  It is relieved
   from flow binding table maintenance.

   Figure 5 shows the interaction between an CFN Dispatcher and a CFN
   node.  After CFN node makes the service demand dispatch, it informs
   the CFN dispatcher about the selected CFN egress node for the flow.
   Then CFN dispatcher maintains the flow binding table to ensure the
   flow affinity.  Message exchange between the CFN dispatcher and its
   corresponding CFN node needs to be defined.  The CFN dispatcher can
   simply forward the first packet of a flow to the CFN node, who takes
   the decision of which instance to use and pushes this information in
   the flow binding table of the CFN dispatcher.  However, in case of
   failures, e.g., CFN egress not reachable anymore, further interaction
   is needed between the CFN dispacther and the CFN node.

Li, et al.                 Expires May 4, 2021                 [Page 11]

Internet-Draft          CFN-dyncast Architecture            October 2020

    SID: Service ID
    BID: Binding ID

                                                    +--| BID21  |
                  binding info                      |  +--------+
                 (flow1,egress2)     +----------+   |
                 (flow2,egress3)  +--|CFN node 2|---|     SID2
                      <-----      |  +----------+   |  +--------+
   +------+                       |                 +--| BID22  |
   |Client|-+                     |                    +--------+
   +------+  \                    |
              \                   |
       +--------------+     +----------+
       |CFN Dispatcher|-----|CFN Node 1|
       +--------------+     +----------+
              /                   |                       SID3
   +------+  /                    |                    +--------+
   |Client|-+                     |                 +--| BID32  |
   +------+                       |                 |  +--------+
                                  |  +----------+   |
                                  +--|CFN node 3|---|     SID3
                                     +----------+   |  +--------+
                                                    +--| BID33  |

           Figure 5: Service Demand Dispatch with CFN Dispatcher

5.  Summary of the key elements of CFN Dyncast Architecture

   o  CFN Control Plane:

      *  SID: CFN nodes have to made aware of existing services through
         the existence of the corresponding SID.  It can be achieved in
         different ways.  For example, use a special range or coding of
         anycast IP address as service IDs or use DNS.

      *  BID bindings: SID are bound to a set of BID representing the
         different instances of the service.  Associated to these BID
         there is as well a set of metrics describing the state of the
         instance.  These bindings have to be shared among the CFN nodes
         so that they are aware of the different instances and their
         computing resource status.

Li, et al.                 Expires May 4, 2021                 [Page 12]

Internet-Draft          CFN-dyncast Architecture            October 2020

      *  Network state: CFN nodes have to be able to share network
         status so to have an idea on the impact of the dispatching
         decision in terms of link congestion.

      *  Metric and network status updates need to be sufficiently
         sparse so to limit the signaling overhead and keep the system
         stable, but also sufficiently regular so to make the system
         reactive to sudden traffic fluctuations.

   o  CFN Data Plane:

      *  In case of a new flow: CFN ingress node selects the most
         appropriate CFN egress in terms of the network status and the
         computing resource of the service instance attached to the

      *  Flow affinity: CFN ingress nodes make sure the subsequent
         packets of an existing flow are always delivered to the same
         CFN egress node so that they can be served by the same service

6.  Conclusion (and call for contributions)

   This document introduces an architecture for CFN Dyncast, enabling
   the service demand request to be sent to an optimal edge to improve
   the overall system load balancing.  It can dynamically adapt to the
   computing resources consumption and network status change and avoid
   overloading single edges.  CFN-Dyncast is a network based
   architecture that supports a large number of edges and is independent
   of the applications or services hosted on the edge.

   This present document is a strawman for defining CFN-Dyncast

   More discussions on control plane and data plane approach are

7.  Security Considerations


8.  IANA Considerations

   No IANA action is required so far.

Li, et al.                 Expires May 4, 2021                 [Page 13]

Internet-Draft          CFN-dyncast Architecture            October 2020

9.  Informative References

   [RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
              "Multiprotocol Extensions for BGP-4", RFC 4760,
              DOI 10.17487/RFC4760, January 2007,

              Geng, L., Liu, P., and P. Willis, "Dynamic-Anycast in
              Compute First Networking (CFN-Dyncast) Use Cases and
              Problem Statement", draft-geng-rtgwg-cfn-dyncast-ps-
              usecase-00 (work in progress), October 2020.



Authors' Addresses

   Yizhou Li
   Huawei Technologies

   Email: liyizhou@huawei.com

   Luigi Iannone
   Huawei Technologies

   Email: Luigi.iannone@huawei.com

   Jianfei He
   City University of Hong Kong

   Email: jianfeihe2-c@my.cityu.edu.hk

   Liang Geng
   China Mobile

   Email: gengliang@chinamobile.com

   Peng Liu
   China Mobile

   Email: liupengyjy@chinamobile.com

Li, et al.                 Expires May 4, 2021                 [Page 14]

Internet-Draft          CFN-dyncast Architecture            October 2020

   Yong Cui
   Tsinghua University

   Email: cuiyong@tsinghua.edu.cn

Li, et al.                 Expires May 4, 2021                 [Page 15]

Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/