[Docs] [txt|pdf|xml|html] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01

ALTO WG                                                        D. Lachos
Internet-Draft                                             C. Rothenberg
Intended status: Informational                                   Unicamp
Expires: January 3, 2019                                    July 2, 2018


         ALTO-based Broker-assisted Multi-domain Orchestration
                draft-lachosrothenberg-alto-brokermdo-01

Abstract

   Evolving networking scenarios (e.g., 5G) demand new multiple
   administrative domain (aka multi-domain) orchestration models.  This
   document proposes the use of Application-Layer Traffic Optimization
   (ALTO) services to offer topology and resources addressing network
   service discovery and provisioning by multi-domain orchestrators.
   The ALTO services with the proposed protocol extension offer
   aggregated views on various types of resources contributing to a more
   simple and scalable solution for resource and service discovery in
   multi-domain, multi-technology environments.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 3, 2019.








Lachos & Rothenberg      Expires January 3, 2019                [Page 1]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Changes Since Version -00 . . . . . . . . . . . . . . . . . .   3
   3.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   4.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
   5.  Problem Statement and Challenges  . . . . . . . . . . . . . .   5
   6.  Proposed Approach . . . . . . . . . . . . . . . . . . . . . .   6
     6.1.  Inter-domain Resource (IdR) Component . . . . . . . . . .   7
     6.2.  Inter-domain Topology (IdT) Component . . . . . . . . . .   8
     6.3.  ALTO Server Functionalities . . . . . . . . . . . . . . .   8
     6.4.  Filtered Cost Map Extension . . . . . . . . . . . . . . .   8
       6.4.1.  Accept Input Parameters . . . . . . . . . . . . . . .   9
       6.4.2.  Response  . . . . . . . . . . . . . . . . . . . . . .   9
     6.5.  Examples of Message Exchange  . . . . . . . . . . . . . .  10
       6.5.1.  Property Map Service  . . . . . . . . . . . . . . . .  10
       6.5.2.  Filtered Cost Map Service . . . . . . . . . . . . . .  11
   7.  Discussion  . . . . . . . . . . . . . . . . . . . . . . . . .  14
     7.1.  Benefits  . . . . . . . . . . . . . . . . . . . . . . . .  14
     7.2.  Open Issues . . . . . . . . . . . . . . . . . . . . . . .  15
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  15
   9.  Security Considerations . . . . . . . . . . . . . . . . . . .  16
   10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  16
   11. References  . . . . . . . . . . . . . . . . . . . . . . . . .  16
     11.1.  Normative References . . . . . . . . . . . . . . . . . .  16
     11.2.  Informative References . . . . . . . . . . . . . . . . .  16
     11.3.  URIs . . . . . . . . . . . . . . . . . . . . . . . . . .  18
   Appendix A.  Proof of Concept Use Case Implementation . . . . . .  18
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  22







Lachos & Rothenberg      Expires January 3, 2019                [Page 2]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


1.  Introduction

   Envisioned 5G network architectures and related service models
   consider broader cooperation between stakeholders in order to provide
   flexible multi-operator multi-domain services.  These multi-provider
   orchestration operations will require the information exchange across
   Multi-domain Orchestrators (MdOs).  The key information to be
   exchanged between MdOs includes the abstract network topology,
   resource availability (e.g., CPUs, Memory, and Storage) and
   capability (e.g., supported network functions).

   This document presents a federation networking paradigm where a
   broker-plane works on top of the management and orchestration plane
   to assist and coordinate the creation of an End-to-End Network
   Service (E2ENS) spanning over multi-operator multi-domain networks.
   Our design resorts to the Application-Layer Traffic Optimization
   (ALTO) protocol  [RFC7285] to address the lack of abstractions to
   discover and adequately represent in confidentiality-preserving
   fashion the resource and topology information from different
   administrative domains.  Moreover, this draft introduces an extension
   to the ALTO base protocol for inter-domain connectivity information
   discovery.

2.  Changes Since Version -00

   o  Many minor style and grammar edits.

   o  Updated Problem Statement and Challenges section.

   o  Removed Property Map Extension section.  The current Property Map
      draft [DRAFT-PM] already supports property values encoded as
      JSONArray.

   o  Added section on benefits and open questions in our proposed
      architecture.

3.  Terminology

   We use the following definitions, as established in [ETSI-NFV-DEF]:

   Administrative Domain:  Collection of systems and networks operated
      by a single organization or administrative authority.

   Network Function (NF):  Functional block within a network
      infrastructure that has well-defined external interfaces and well-
      defined functional behaviour.





Lachos & Rothenberg      Expires January 3, 2019                [Page 3]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   Network Functions Virtualisation (NFV):  The principle of separating
      network functions from the hardware they run on by using virtual
      hardware abstraction.

   NF Forwarding Graph: (NFFG):  Graph of logical links connecting NF
      nodes for the purpose of describing traffic flow between these
      network functions.

   Network Service Orchestration (NSO):  Function responsible for
      network service lifecycle management.

   Resource Orchestration (RO):  Function responsible for global
      resource management governance.

   Our proof of concept implementation follows the architectural
   proposal of the 5GEx project [H2020.5GEX].  Some additional 5GEx
   terms commonly used in this document are defined below:

   Domain Orchestrator (DO):  Performs Resource Orchestration and/or
      Service Orchestration within the same administrative domain.

   Multi-domain Orchestrator (MdO):  Coordinates resource and/or service
      orchestration at multi-domain level, where multi-domain may refer
      to multiple DOs or multiple administrative domains.

   Resource Topology (RT):  Functional module that is responsible for
      keeping an updated global view of the underlying infrastructure
      topology exposed by DOs.

   Service Graph (SG):  A high-level data model for defining flexible
      network services (including traffic steering primitives).

   Service Access Point (SAP):  A named/tagged port supporting stitching
      (service to service, domain to domain, etc.)

4.  Scope

   Existing proposals for the network service orchestration are
   intrinsically conceived for single administrative domain scenarios.
   For example, in the standard service orchestration model described in
   ETSI NFV MANO framework [ETSI-NFV-MANO], one orchestrator is supposed
   to work within one administrative domain.  The analysis of
   orchestration and management of network Services over multiple
   administrative domains have begun to be addressed by ETSI
   in [ETSI-NFV-MANO-MDO].

   Envisioned 5G scenarios are expected to work not only with
   heterogeneous technologies but also across different network



Lachos & Rothenberg      Expires January 3, 2019                [Page 4]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   operators.  Many ongoing initiatives and projects related are
   addressing the multi-provider multi-domain orchestration challenges
   under different approaches.  For example, [H2020.5GEX] seeks to
   integrate multiple administrations and technologies through the
   collaboration between operations.  Other studies are envisioned to
   use a centralized approach, where each domain advertises its
   capabilities to a federation layer which will act as a
   broker [VITAL][T-NOVA].  The proposed architecture in [ICAF] allows
   the creation of cloud services from different administrative domains,
   however, it is not related to the provisioning of NFV-based cross-
   domain network services.

   All such proposals described above envision the potential
   introduction of new business model approaches, including federation
   models [PPP-5:2013] among administrative domains.  In this context,
   this document considers each network operator involved in the
   community advertises its abstracted capabilities (e.g., software/
   hardware resources, physical/virtual network functions, etc.) to a
   broker (i.e., 3rd party).  This latter, in its turn, provides or
   assists coordinate E2E network services spanning multi-domain
   networks.

5.  Problem Statement and Challenges

   The provision of a complete E2E network service requires chaining
   services provided by multiple network operator with multiple
   technologies.  In this multi-domain environment, the orchestration
   process will require an advertise mechanism through which single
   domains can describe their capabilities, resources, and VNFs in an
   interoperable manner.  Moreover, a discovery mechanism is also
   necessary so that source domains can obtain candidate domains (with
   the corresponding connectivity information) which can provide a part
   of the service and/or slide in an E2ENS requirement.

   In order to the advertising and discovery process works in a proper
   way, a number of challenges can be identified:

   Lack of Abstractions:  Multiple vendors with heterogeneous
        technologies need an information model to adequately represent
        in confidentiality-preserving fashion the resource and topology
        information.

   Scalability:  Involves the distribution of topology and resource
        information in a peer-to-peer fashion (MdO-to-MdO).  Multi-
        operator multi-domain environments where the information
        distribution is advertised in a peer-to-peer model scales
        linearly.  It means more MdO interconnections one has, the more
        it "costs" to distribute.



Lachos & Rothenberg      Expires January 3, 2019                [Page 5]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   Flexibility:  Considers that a distributed approach does not allow
        domains without physical infrastructure (e.g., without BGP or
        BGP-LS) to advertise resource capabilities and networking
        resources.  Such procedures consist in deploying and configuring
        physical peering points for these domains.

   Complexity:  Refers to the discovery mechanism to pre-select
        candidate domains, accounting for resources and capabilities,
        necessary for an E2E network service deployment.  An intrinsic
        complexity exists in the process of assembling, logically
        organizing, and enabling abstraction views of different
        resources and capabilities in multi-domain scenarios.

6.  Proposed Approach

   The primary design goal for ALTO-based Broker-assisted Multi-domain
   Orchestration is to discover resource and topology information from
   different administrative domains involved in the federation, while
   also safeguarding the privacy and autonomy of every domain.

   In the architectural proposal shown in Figure 1, a broker component
   is conceived to be working as coordinator of a set of MdOs, whose key
   components are: the Inter-domain Resource (IdR), the Inter-domain
   Topology (IdT) and the ALTO Server.



























Lachos & Rothenberg      Expires January 3, 2019                [Page 6]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


                              BROKER COMPONENT
              +--------------------------------------------+
              |                                            |
              |             +-----------------+            |
              |             |                 |            |
          XXXXXXXXXXXXXXXXXXX ALTO SERVER(s)  |            |
          X   |             |                 +            |
          X   |             +----------------+\            |
          X   |            /                   \           |
          X   |           /                     \          |
          X   |  +------+/+-------+     +---------------+  |
          X   |  | Inter-domain   |     | Inter-domain  |  |
          X   |  | Topology (IdT) |     | Resource (IdR)|  |
          X   |  +-^------^-------+     +---^-------^---+  |
          X   |    |      |                 |       |      |
          X   +--------------------------------------------+
          X        |      |                 |       |
          X        |      |                 |       |
       +--X--------+---+--------------------+       |
       |               |  |                         |
       |               |  +------------+------------+---+
       |     MdO1      |               |                |
       |               +<------------->+                +---+
       +---------------+               |      MdO2      |   |
                                       |                |   |
                                       +-+--------------+   |
                                         |                  |
   Legend:                               |      MdO(n)      |
   XXX ALTO Protocol                     +------------------+



       Figure 1: Broker-assisted Multi-operator Network Architecture

6.1.  Inter-domain Resource (IdR) Component

   It creates a hierarchical database that contains inter-domain
   resource information such as resource availability (i.e., CPU,
   memory, and storage), Virtual Network Functions (VNFs) and Physical
   Network Functions (PNFs) supported and Service Access Points (SAPs)
   to access those resources.  UNIFY [UNIFY.NFFG], TOSCA [TOSCA], ETSI-
   NFV [ETSI-NFV-MANO], among other data models can be used to create
   the interface between IdR and MdOs.








Lachos & Rothenberg      Expires January 3, 2019                [Page 7]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


6.2.  Inter-domain Topology (IdT) Component

   A hierarchical TED (Traffic Engineering Database) that contains
   inter-domain network topology information including additional key
   parameters (e.g., throughput and latency of links).  This information
   can be retrieved from each MdO through BGP-LS or REST interfaces.

6.3.  ALTO Server Functionalities

   The ALTO server component is the core of the broker layer.  Multiple
   logically centralized ALTO servers use the information collected from
   IdR and IdT modules to create and provide abstract maps with a
   simplified view, yet enough information about MdOs involved in the
   federation.  This information includes domain-level topology, storage
   resources, computation resources, networking resources and PNF/VNF
   capabilities.

   As an ALTO client, each MdO sends ALTO service queries to the ALTO
   server.  This server provides aggregated inter-domain information
   exposed as set ALTO base services defined in [RFC7285], e.g., Network
   Map, Cost Map and ALTO extension services, e.g., Property
   Map [DRAFT-PM], Multi-Cost Map [RFC8189], Path Vector [DRAFT-PV].

   For example, when a source MdO receives a customer service request,
   it checks whether or not it can deliver the full service.  If it is
   unable to do so, the MdO consumes from the ALTO Server the Property
   Map service to have a clear global view of the resource information
   offered by other MdOs.  This information allows discovering which
   candidate MdOs may be contacted to deliver the remaining requirements
   of a requested end-to-end service deployment.  The connectivity
   information among discovered MdOs can be retrieved by a Cost Map
   service, responding, for instance, a path vector with the AS-level
   topology distance between the source MdO and candidate MdOs.

6.4.  Filtered Cost Map Extension

   The ALTO server MUST provide connectivity information for every SG
   link in the SG path for an E2E requirement.  This information is the
   AS-level topology distance in the form of path vector, and it
   includes all possible ways for each (source node, destination node)
   pair in the SG link.

   In this section, we introduce a non-normative overview of the
   Filtered Cost Map defined in Section 6.1 of [DRAFT-PV] [1].

   The specifications for the "Media Types", "HTTP method",
   "Capabilities" and "Uses" (described in Section 6.1 of [DRAFT-PV]
   [2]) are unchanged.



Lachos & Rothenberg      Expires January 3, 2019                [Page 8]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


6.4.1.  Accept Input Parameters

   The ReqFilteredCostMap object in Section 6.1.2 of [DRAFT-PV] [3] is
   extended as follow:


     object {
       [NFFG sg;]
     } ReqFilteredCostMap;

     object {
       JSONString nfs<1..*>;
       JSONString saps<1..*>;
       NextHops sg_links<1..*>;
       REQs reqs<1..*>;
     } NFFG;

     object {
       JSONNumber id;
       JSONString src-node;
       JSONString dst-node;
     } NextHops;

     object {
       JSONString id;
       JSONString src-node;
       JSONString dst-node;
       JSONNumber sg-path<1..*>;
     } REQs;



   sg:  If present, the ALTO Server MUST allow the request input to
      include an SG with a formatted body as an NFFG object.  An NFFG
      object contains NFs, SAPs, SG links representing logical
      connections between NFs, SAPs or both and E2E requirements as a
      list of ids of SG links.

   It is worth noting that further versions of this draft will define a
   more elaborated NFFG object to support extended parameters such as
   monitoring parameters, resource requirements, etc.

6.4.2.  Response

   If the ALTO client includes the path vector cost mode in the "cost-
   type" or "multi-cost-types" field of the input parameter, the
   response for each SG link in each E2E requirement MUST be encoded as
   a JSONArray of JSONArrays of JSONStrings.  Anyone of the sub-arrays



Lachos & Rothenberg      Expires January 3, 2019                [Page 9]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   indicates a potential candidate path calculated as the per-domain
   topological distance corresponding to the amount of traversing
   domains.

   Moreover, as defined in Section 6.3.6 of [DRAFT-PV] [4], If an ALTO
   client sends a request of the media type "application/alto-
   costmapfilter+json" and accepts "multipart/related", the ALTO server
   MUST provide path vector information along with the associated
   Property Map information (e.g., entry points of the corresponding
   foreign domains), in the same body of the response.

   Section 6.5.2 gives an example of the Filtered Cost Map query and the
   corresponding responses.

6.5.  Examples of Message Exchange

   This section list a couple of examples of the Property Map and
   Filtered Cost Map queries and the corresponding responses.  These
   responses are based on the information in Table 1 and Table 2 of a
   use case implementation described in Appendix A.

6.5.1.  Property Map Service

   In this example, the ALTO client wants to retrieve the entire
   Property Map for PID entities with the "entry-point", "cpu", "mem",
   "storage", "port" and "nf" properties.

























Lachos & Rothenberg      Expires January 3, 2019               [Page 10]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


     GET /propmap/full/inet-ucmspn HTTP/1.1
     Host: alto.example.com
     Accept: application/alto-propmap+json,application/alto-error+json

     HTTP/1.1 200 OK
     Content-Length: ###
     Content-Type: application/alto-propmap+json
     {
       "property-map": {
           "pid:AS1": {
               "entry-point": [ "http://172.25.0.10:8888/escape" ],
               "cpu": [ "50.0" ],
               "mem": [ "60.0" ],
               "storage": [ "70.0" ],
               "port": [ "SAP1" ],
               "nf": [ "NF1", "NF3" ]
           },
           "pid:AS2": {
               "entry-point": [ "http://172.26.0.10:8888/escape" ],
               "cpu": [ "10.0" ],
               "mem": [ "20.0" ],
               "storage": [ "30.0" ],
               "nf": [ "NF2" ]
           },
           "pid:AS3": {
               "entry-point": [ "http://172.27.0.10:8888/escape" ],
               "cpu": [ "80.0" ],
               "mem": [ "90.0" ],
               "storage": [ "100.0" ],
               "port": [ "SAP2" ],
               "nf": [ "NF1", "NF3" ]
           }
       }
     }


6.5.2.  Filtered Cost Map Service

   The following example uses the Filtered Cost Map service to request
   the path vector for a given E2E requirement.  The SG request
   information in Table 2 is used to describe the service, and it is
   composed of three NFs (NF1, NF2, and NF3) and two SAPs (SAP1 and
   SAP2).  Links connecting the NFs and SAPs ("sg_links" tag) are also
   included, followed by an E2E requirement ("reqs" tag) with
   information about the order in which NFs are traversed from SAP1 to
   SAP2.





Lachos & Rothenberg      Expires January 3, 2019               [Page 11]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   Note that the request accepts "multipart/related" media type.  This
   means the ALTO server will include associated property information in
   the same response.


       POST /costmap/pv HTTP/1.1
       Host: alto.example.com
       Accept: multipart/related, application/alto-costmap+json,
             application/alto-propmap+json, application/alto-error+json
       Content-Length: [TBD]
       Content-Type: application/alto-costmapfilter+json

       {
         "cost-type": {
           "cost-mode": "array",
           "cost-metric": "ane-path"
         },
         "sg": {
           "nfs": [ "NF1", "NF2", "NF3" ],
           "saps": [ "SAP1", "SAP2" ],
           "sg_links":[
             {
               "id": 2,
               "src-node": "SAP1",
               "dst-node": "NF1",

             },
             {
               "id": 2,
               "src-node": "NF1",
               "dst-node": "NF2",
             },
             {
               "id": 3,
               "src-node": "NF2",
               "dst-node": "NF3",
             },
             {
               "id": 4,
               "src-node": "NF3",
               "dst-node": "SAP2",
             }
           ],
           "reqs": [
             {
               "id": 1,
               "src-node": "SAP1",
               "dst-node": "SAP2",



Lachos & Rothenberg      Expires January 3, 2019               [Page 12]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


               "sg-path": [ 1, 2, 3, 4 ]
             }
           ]
         }
       }


   The ALTO server returns connectivity information for the E2E
   requirement provided by the ALTO Client request of the above example.
   Also, the response includes Property Map information for each element
   in the path vector.  In this case, it is retrieved a Property Map
   with the "entry-point" property, i.e., the URL of the MdO entry point
   for the corresponding network.



     HTTP/1.1 200 OK
     Content-Length: [TBD]
     Content-Type: multipart/related; boundary=example

     --example
     Content-Type: application/alto-endpointcost+json

     {
       "meta": {
         "cost-type": {
            "cost-mode": "array",
            "cost-metric": "ane-path"
          },
       },

       "cost-map": {
         "SAP1": {
           "SAP2": {
               "SAP1": {
                   "NF1": [
                     [ "AS1" ], [ "AS1", "AS2", "AS3" ]
                   ]
               },
               "NF1": {
                   "NF2": [
                     [ "AS1", "AS2" ], [ "AS3", "AS2" ]
                   ]
               },
               "NF2": {
                   "NF3": [
                     [ "AS2", "AS1" ], [ "AS2", "AS3" ]
                   ]



Lachos & Rothenberg      Expires January 3, 2019               [Page 13]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


               },
               "NF3": {
                   "SAP2": [
                     [ "AS1", "AS2", "AS3" ], [ "AS3" ]
                   ]
               }
           }
         }
       }
     }

     --example
     Content-Type: application/alto-propmap+json

     {
       "property-map": {
         "pid:AS1": { "entry-point": "http://172.25.0.10:8888/escape" },
         "pid:AS2": { "entry-point": "http://172.26.0.10:8888/escape" },
         "pid:AS3": { "entry-point": "http://172.27.0.10:8888/escape" }
       }
     }

     --example--



7.  Discussion

   In this section we analyze the benefits and open issues in our
   broker-assisted architecture.

7.1.  Benefits

   The broker-assisted orchestration has numerous benefits, such as:

   o  Avoid the distribution of topology and resource information in a
      peer-to-peer fashion (MdO-to-MdO)

   o  The (abstracted) information and offered resources, services are
      maintained in each local MdO.

   o  Allow domains without physical infrastructure (hence without BGP
      or BGP-LS) to advertise their capabilities.

   o  An ALTO-based privacy-preserving information model to provide
      computing, storage and networking resource info.





Lachos & Rothenberg      Expires January 3, 2019               [Page 14]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   o  An MdO discovery method to determine the underlying network graph
      and a potential set of paths before bilateral negotiation between
      MdOs is started.

7.2.  Open Issues

   Although the broker-assisted information exchange has several
   advantages, it also raises some questions which we try to answer from
   our lessons learned.

   o  What kind of organization will manage and support the operation of
      a broker entity?  If a broker is used to exchange information,
      then how does one ensure that the data delivered amongst the
      operators by this 3rd party has not been changed?

      *  The broker entity must be trusted by each operator since it
         stores and handles sensitive information.  For example, future
         deployment of SDN at IXPs can be used as a trusted third-party
         platform to support rich business models between different
         operators [DRAFT-HHSFC].

   o  In the case of peer-to-peer information exchange model, an MdO
      failure concerns only the domain where the failure occurs, other
      peers can perform the information exchange without any limitation.
      However, If any error occurs in the broker entity the information
      exchange among all involved ASes will be impacted.  How avoid this
      single point of failure?

      *  The broker entity maintains a centralized database.  Local
         restoration/replication options may be applied.

   o  The MdO information exchange depends on the policies.  Operators
      have a preference to share a different view about its compute and
      network resources towards different operators.  For example, a
      detailed view for the operators that are belonging to same
      operator group and a high-level information towards the other
      operators.  How is the fine-grained/coarse-grained information
      exchange handled?.

      *  It requires much more complex database handling and information
         exchange with the MdOs depending on the policies.

8.  IANA Considerations

   This document includes no request to IANA.






Lachos & Rothenberg      Expires January 3, 2019               [Page 15]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


9.  Security Considerations

   TBD.

10.  Acknowledgments

   This work is supported by the Innovation Center of Ericsson S.A.,
   Brazil (grant agreement UNI.64).

   Thank you to Robert Szabo (Ericsson Research, Hungary) for the
   contribution and substantial feedback and suggestions in this
   document.

   Many thanks to Richard Yang, Dawn Chan, Jensen Zhang, Shawn Lin, Qiao
   Xiang, Sabine Randriamasy for their feedback on this draft.

11.  References

11.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997,
              <http://xml.resource.org/public/rfc/html/rfc2119.html>.

   [RFC7285]  Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S.,
              Previdi, S., Roome, W., Shalunov, S., and R. Woundy,
              "Application-Layer Traffic Optimization (ALTO) Protocol",
              RFC 7285, DOI 10.17487/RFC7285, September 2014,
              <https://www.rfc-editor.org/info/rfc7285>.

   [RFC8189]  Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost
              Application-Layer Traffic Optimization (ALTO)", RFC 8189,
              DOI 10.17487/RFC8189, October 2017,
              <https://www.rfc-editor.org/info/rfc8189>.

11.2.  Informative References

   [DRAFT-HHSFC]
              Li, G., Li, G., Li, T., Xu, Q., and H. Zhou, "Hybrid
              Hierarchical Multi-Domain Service Function chaining",
              draft-li-sfc-hhsfc-04 (work in progress), April 2018.

   [DRAFT-PM]
              Roome, W., Chen, S., Randriamasy, S., Yang, Y., and J.
              Zhang, "Unified Properties for the ALTO Protocol", draft-
              ietf-alto-unified-props-new-03 (work in progress), March
              2018.




Lachos & Rothenberg      Expires January 3, 2019               [Page 16]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   [DRAFT-PV]
              Bernstein, G., Chen, S., Gao, K., Lee, Y., Roome, W.,
              Scharf, M., Yang, Y., and J. Zhang, "ALTO Extension: Path
              Vector Cost Type", draft-ietf-alto-path-vector-03 (work in
              progress), March 2018.

   [ETSI-NFV-DEF]
              ETSI, "Network Functions Virtualisation (NFV); Terminology
              for Main Concepts in NFV V1.3.1", Jan 2018,
              <https://docbox.etsi.org/isg/nfv/open/Publications_pdf/
              Specs-Reports/NFV%20003v1.3.1%20-%20GR%20-%20Terminology%2
              0for%20Main%20Concepts%20in%20NFV.pdf>.

   [ETSI-NFV-MANO]
              ETSI, "Network Functions Virtualisation (NFV) Management
              and Orchestration V1.1.1", Dec 2014,
              <http://www.etsi.org/deliver/etsi_gs/NFV-
              MAN/001_099/001/01.01.01_60/gs_NFV-MAN001v010101p.pdf>.

   [ETSI-NFV-MANO-MDO]
              ETSI, "Network Functions Virtualisation (NFV) Release 3;
              Management and Orchestration; Report on architecture
              options to support multiple administrative domains
              V3.1.1", Jan 2018, <http://www.etsi.org/deliver/etsi_gr/
              NFV-IFA/001_099/028/03.01.01_60/
              gr_NFV-IFA028v030101p.pdf>.

   [H2020.5GEX]
              Bernardos, C., Dugeon, O., Galis, A., Morris, D., Simon,
              C., and R. Szabo, "5G Exchange (5GEx)--Multi-domain
              Orchestration for Software Defined Infrastructures",
              focus vol. 4, no.5, p.2, 2015.

   [H2020.5GEX.ESCAPE]
              5GEx Project, "ESCAPE: Extensible Service ChAin
              Prototyping Environment", 2015,
              <https://github.com/5GExchange/escape>.

   [ICAF]     Demchenko, Y., Makkes, M., Strijkers, R., Ngo, C., and C.
              Laat, "Intercloud Architecture Framework for Heterogeneous
              Multi-Provider Cloud based Infrastructure Services
              Provisioning", International Journal of Next-Generation
              Computing vol. 4, no.2, 2013.








Lachos & Rothenberg      Expires January 3, 2019               [Page 17]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   [PPP-5:2013]
              5G-PPP, "Advanced 5G Network Infrastructure for the Future
              Internet", 2013, <https://5g-ppp.eu/wp-
              content/uploads/2014/02/Advanced-5G-Network-
              Infrastructure-PPP-in-H2020_Final_November-2013.pdf>.

   [T-NOVA]   FP7 project T-NOVA, "T-NOVA Project, Network Functions as
              a Service over Virtualised Infrastructures", 2014,
              <http://www.t-nova.eu/>.

   [TELEFONICA.NET.TOPO]
              Telefonica I+D, "Netphony-Topology", 2016,
              <https://github.com/telefonicaid/netphony-topology>.

   [TOSCA]    OASIS, "TOSCA: Topology and Orchestration Specification
              for Cloud Applications V1.0", 2013, <http://docs.oasis-
              open.org/tosca/TOSCA/v1.0/os/TOSCA-v1.0-os.pdf>.

   [UNIFY.NFFG]
              UNIFY Deliverable D3.2a, "Network Function Forwarding
              Graph specification", 2015, <http://www.fp7-
              unify.eu/files/fp7-unify-eu-docs/Results/Deliverables/
              UNIGY_D3.2a_NFFG%20Specification.pdf>.

   [VITAL]    VITAL PROJECT H2020, "VITAL -- VIrtualized hybrid
              satellite-TerrestriAl systems for resilient and fLexible
              future networks", 2015, <http://www.ict-vital.eu/>.

11.3.  URIs

   [1] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.1

   [2] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.1

   [3] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.1.2

   [4] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.3.6

Appendix A.  Proof of Concept Use Case Implementation

   A strawman use case scenario has been implemented following the
   architectural proposal of the 5GEx project [H2020.5GEX].  It refers
   to an E2ENS orchestration involving three administrative domains.




Lachos & Rothenberg      Expires January 3, 2019               [Page 18]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   As shown in Figure 2, each administrative domain has an MdO (MdO-AS1,
   MdO-AS2, and MdO-AS3) to coordinate resource and/or service
   orchestration at multi-operator level via interface I2 APIs.  For the
   orchestration within the same administrative domain, each MdO uses
   emulated DOs with emulated I3 interfaces, since no data-plane is
   present.  DOs use static configuration files to load local
   information about resources (I3-RC) and topology (I3-RT).  The
   different MdO components are based on existing open source tools such
   as ESCAPE [H2020.5GEX.ESCAPE] (Service/Resource Orchestrator) and
   Netphony-topology [TELEFONICA.NET.TOPO] (Resource Topology) and run
   in Docker containers on a single computer.  Besides, MdOs expose I1
   interfaces to the tenants who request services and/or slices which
   should follow a Network Function Forwarding Graph (NFFG) [UNIFY.NFFG]
   format.

   In case of the broker layer, the IdR and IdT components use a UNIFY
   Virtualizer API [UNIFY.NFFG] (broker-based I2-RC API) and a REST API
   (broker-based I2-RT API) respectively, in order to create the
   hierarchical databases.  Regarding the IdT, the administrative domain
   2 is a transit provider so that the domain-level topology computed
   is: AS1-AS2-AS3.  From the inter-domain information are created the
   two different ALTO Map Services: (i) Property Map and (ii) Cost Map.





























Lachos & Rothenberg      Expires January 3, 2019               [Page 19]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


                            +----------------------------------------+
                            |       +---------------+    BROKER LAYER|
                       XXXXXXXXXXXXXX  ALTO Server  |                |
                       X    |       +--------+----+-+                |
                       X    |               /      \                 |
                       X    | +-----------+/+--+  +-\-------------+  |
                       X    | | Inter-domain   |  | Inter-domain  |  |
       +-----------+   X    | | Topology (IdT) |  | Resource (IdR)|  |
       | Service   |   X    | +----------------+  +---------------+  |
       | Graph (SG)|   X    +---------^-^----------------^--^--------+
       | Request   |   X              * *    .............  .
       +-----+-----+   X              * *    . ..............
             |         X              * *    . .              MdO-AS3
           I1|         X              * *    . .      +--------------+
             |         X     MdO-AS1  * *    . .      |              |
       +-----|---------X-----------+  * *    . .      |     MdO-AS2  |
       |     |                     |  * *    . .   +---------------+ |
       | +---v-------------------+ |  * *    . .   | +-----------+ | |
       | |                       | |  * *    . .   | |           | | |
       | |  Network Service Orch.| |  * *    . .   | |   NSO     | | |
       | |  (NSO)                | |  * *    . .   | |           | | |
       | +-----------------------+ |  * *    . .   | +-----------+ | |
       |                           |  * *    . .   |               | |
       | +---------+               |  * *    . .   | +---+         | |
       | | Resource........................... .   | |   |         | |
       | | Topoloy |               |  * *      .......RT |         | |
       | | (RT)    | +-----------+ |  * *          | |   |         | |
       | +---------+ |Resource   | |  * *          | +---+   +---+ | |
       |             |Orch.      | |  * **********************   | | |
       |             |(RO)       ******            |         |RO | +-+
       |             +-----------+ |               |         |   | |
       |                           |<------------->|         +---+ |
       +---------------------------+      I2       +-----+---------+
                  /     \                                |
               I3/       \                               |I3
       +-------+---+  +-----------+                +-----------+
       | Domain    |  | Domain    |                | Domain    |
       | Orch (DO) |  | Orch (DO) |                | Orch (DO) |
       +-----------+  +-----------+                +-----------+
                                            Legend:
                                            XXX ALTO Protocol
                                            ... broker-based I2-RT API
                                            *** broker-based I2-RC API


               Figure 2: Broker-assisted 5GEx Info Exchange





Lachos & Rothenberg      Expires January 3, 2019               [Page 20]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


   The Property Map includes property values grouped by Autonomous
   System (AS).  Such values are SAPs, NFs and the 5GEx Entry Point
   (e.g., the URL of the ESCAPE orchestrator).  An example of the
   Property Map in our prototype is:

   +-----+------------+-------+--------------+-----+-----+-------+-----+
   |     |   Entry    |  Port | Capabilities | CPU | MEM |  Stor | ... |
   |     |   Point    |  SAP  |              |     |     |  age  |     |
   +-----+------------+-------+--------------+-----+-----+-------+-----+
   | AS1 | http://... |  SAP1 |  {NF1, NF3}  |  50 |  60 |   70  | ... |
   | AS2 | http://... |   -   |    {NF2}     |  10 |  20 |   30  | ... |
   | AS3 | http://... |  SAP2 |  {NF1, NF3}  |  80 |  90 |  100  | ... |
   +-----+------------+-------+--------------+-----+-----+-------+-----+

                        Table 1: ALTO Property Map

   The Cost Map defines a path vector as an array of ASes, representing
   the AS-level topological distance for a given E2ENS request.  Path
   vector constraints (as described in the Multi-Cost Map [RFC8189]) can
   be applied to restricts the response to costs that satisfy a list of
   simple predicates.

   Table 2 below shows a brief example of an SG request and its path
   vector response containing a list of potential providers to be
   traversed to deliver such service.  Every AS path is computed from
   the inter-domain topology information in the IdT module.  In our
   scenario, MdO-AS2 is a transit provider, so that the domain-level
   topology map is AS1<->AS2<->AS3.

   +--------------------+----------------------------------------------+
   | Service Graph (SG) |                Path(s) Vector                |
   |      Request       |                                              |
   +--------------------+----------------------------------------------+
   | SAP1->NF1->NF2->NF | 1:{AS1:SAP1->AS1:NF1->AS2:NF2->AS3:NF3->AS3: |
   |      3->SAP2       |                    SAP2}                     |
   |                    | 2:{AS1:SAP1->AS1:NF1->AS2:NF2->AS1:NF3->AS2- |
   |                    |                  >AS3:SAP2}                  |
   |                    | 3:{AS1:SAP1->AS2->AS3:NF1->AS2:NF2->AS3:NF3- |
   |                    |                  AS3:SAP2}                   |
   |                    | 4:{AS1:SAP1->AS2->AS3:NF1->AS2:NF2->AS1:NF3- |
   |                    |               >AS2->AS3:SAP2}                |
   +--------------------+----------------------------------------------+

                          Table 2: ALTO Cost Map







Lachos & Rothenberg      Expires January 3, 2019               [Page 21]


Internet-Draft    ALTO-based Multi-domain Orchestration        July 2018


Authors' Addresses

   Danny Alex Lachos Perez
   University of Campinas
   Av. Albert Einstein 400
   Campinas, Sao Paulo  13083-970
   Brazil

   Email: dlachosp@dca.fee.unicamp.br
   URI:   https://intrig.dca.fee.unicamp.br/danny-lachos/


   Christian Esteve Rothenberg
   University of Campinas
   Av. Albert Einstein 400
   Campinas, Sao Paulo  13083-970
   Brazil

   Email: chesteve@dca.fee.unicamp.br
   URI:   https://intrig.dca.fee.unicamp.br/christian/































Lachos & Rothenberg      Expires January 3, 2019               [Page 22]


Html markup produced by rfcmarkup 1.127, available from https://tools.ietf.org/tools/rfcmarkup/