[Docs] [txt|pdf] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 02 03

DetNet                                                         H. Wang
Internet Draft                                                 P. Wang
Intended status: Standards Track                              C. Zhang
Expires: December 11, 2018                                     Y. Yang
                                               Chongqing University of
                                          Posts and Telecommunications
                                                          June 9, 2018

         Joint Scheduling Architecture for Deterministic Industrial
                          Field/Backhaul Networks


   The joint scheduling between industrial field network and backhaul
   network is important to satisfy the requirements of deterministic
   delay for data flows in factories. This document describes a joint
   scheduling architecture for deterministic industrial field/backhaul
   networks. Taking WIA-PA, an international standard about industrial
   wireless field network, and IPv6-based backhaul network as an
   example, this document depicts how the joint scheduling architecture
   works in detail.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on December 11, 2018.

Wang, et al.          Expires December 11, 2018               [Page 1]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.

Table of Contents

   1. Introduction ................................................ 2
   2. Joint Scheduling Architecture ............................... 3
      2.1. Distributed Architecture ............................... 4
      2.2. Centralized Architecture ............................... 5
      2.3. Joint Scheduling Architecture .......................... 6
   3. Joint Scheduling Scheme ..................................... 8
      3.1. WIA-PA Network Joint Scheduling ........................ 9
      3.2. Protocol Conversion .................................... 9
      3.3. Industrial Backhaul Network Scheduling ................ 11
      3.4. Bandwidth guarantee method ............................ 13
   4. Security Considerations .................................... 13
   5. IANA Considerations ........................................ 13
   6. References ................................................. 13
      6.1. Normative References .................................. 13
      6.2. Informative References ................................ 13
   Authors' Addresses ............................................ 15

1. Introduction

   Deterministic transmission is one of essential elements of
   industrial network. Thus, applying deterministic network into the
   whole industrial network has attracted a lot of attention recently.
   Deterministic network is also focused on the industrial field
   networks, such as ISA100.11a [IEC62734], WirelessHART [IEC62591] and
   WIA-PA [IEC62601]. Most industrial field networks are in a middle or
   small size, and the network coverage is constant. Therefore, An
   industrial backhaul network is needed to connect different types of
   industrial field networks. Meanwhile, the deterministic transmission
   of the industrial backhaul network should also be guaranteed.

Wang, et al.          Expires December 11, 2018               [Page 2]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

   However, there are little considerations about joint scheduling
   between industrial backhaul network and industrial field network.

   The emerging Software Defined Networks (SDN) technology on the
   Internet brings a new choice to solve joint scheduling problem. SDN
   has been proposed as a new network architecture in recent years. The
   network architecture separates the network control plane from the
   forwarding plane, which brings a revolution for the network
   architecture. By separating control plane from forwarding plane ,and
   the open communication protocol, SDN breaks the closure of
   traditional network device provider. Besides, open interfaces and
   free programmability also make network management more efficient and

   In document [I-D.bas-usecase-detnet] and [I-D.finn-detnet-
   architecture] submitted by the IETF DetNet working group,
   deterministic network based on Ethernet has been researched already.
   They propose a network architecture based on SDN technology that can
   accurately control the transmission of data streams. However, the
   characteristics of the industrial backhaul network and the actual
   condition of industrial field deterministic networks are not
   considered. Firstly, the data that transmits in industrial backhaul
   network is highly sensitive to transmission delay. Secondly, the
   existing deterministic networks have been widely deployed in
   industrial field environment, thus the direct replacement for
   original networks will consume many workers and material resources.

   Based on existing research in document [I-D.finn-detnet-
   architecture], this document proposes a joint scheduling
   architecture for deterministic industrial networks. It will firstly
   replace the industrial backhaul networks and other non-deterministic
   networks located in industrial networks with deterministic Ethernet
   network. Then this document proposes a joint scheduler based on SDN
   technology. By deploying the deterministic network in complete
   industrial network, it can realize the end-to-end deterministic
   scheduling between different industrial field networks.

2. Joint Scheduling Architecture

   There are many types of network controllers in industrial networks,
   which constitute the control plane of the whole industrial network
   together. The control plane is very important in the entire network,
   especially when it refers to cross-domain transmission of time-
   sensitive data. The control plane architecture affects the
   performance of the network greatly. It is becoming a hot research on
   how to give full play to the performance of their respective
   networks when multiple controllers are in the joint work.

Wang, et al.          Expires December 11, 2018               [Page 3]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

   However, there is no unified standard for the joint architecture of
   multiple controllers in the industry presently. The mainstream of
   architecture includes distributed architecture and centralized

2.1. Distributed Architecture

   Distributed architecture is known as East-West architecture. In the
   architecture, the status of all network controllers are equal, these
   controllers connect with each other to form an unstructured network,
   and implement cross-domain transmission task by exchanging
   information, as shown in Figure 1.

            |                                                      |
    Application +--------+        +--------+          +-------+    |
    Plane       |   APP  |        |   APP  |          |  APP  |    |
            |   +----+---+        +----+---+          +---+---+    |
            |        |                 |                  |        |
                     |                 |                  |
                     |                 |                  |
             |       |                 |                  |        |
    Control  |  +----+-----+      +----+-----+       +----+-----+  |
    Plane    |  |Controller|----->|Controller|------>|Controller|  |
             |  |          |<-----|          |<------|          |  |
             |  +----------+      +----------+       +----------+  |
             |       |                 |                   |       |
                     |                 |                   |
                     |                 |                   |
                +----+-----+      +----+-----+        +----+-----+
    Forwarding  |  WIA-PA  |----->| backhaul |------->|  WIA-PA  |
    Plane       |  network |<-----|  network |<-------|  network |
                +----------+      +----------+        +----------+
                   Figure 1. Distributed Architecture
In distributed architecture, controller can exchange different network
topologies and the accessibility of information by east-west interface,
and each controller can establish a global network topology. From a
global network perspective, each controller is equal, thus it can serve
as a server role as well as the ability to start deterministic cross-
network transmission task.

Wang, et al.          Expires December 11, 2018               [Page 4]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

2.2. Centralized Architecture

   Centralized architecture is also called a vertical multi-level
   architecture. In this architecture, the control plane is divided
   into two parts, one part is the basic control plane, which is
   composed of a variety of network controllers, and another is a main
   network controller, which is responsible for controlling the basic
   control plane. The detailed description of centralized architecture
   is shown in Figure 2.

            |                                                     |
    Application  +--------+       +--------+          +-------+   |
    Plane   |    |   APP  |       |   APP  |          |  APP  |   |
            |    +--------+       +----+---+          +-------+   |
            |                          |                          |
            |                          |                          |
            |                   +------+-----+                    |
            |       +-----------|    Main    |----------+         |
            |       |           | Controller |          |         |
    Control |       |           +------+-----+          |         |
    Plane   |       |                  |                |         |
               +----+-----+       +----+-----+     +----+-----+   |
            |  |Controller|       |Controller|     |Controller|   |
            |  +----+-----+       +----+-----+     +----+-----+   |
            |       |                  |                |         |
                    |                  |                |
                    |                  |                |
               +----+-----+       +----+-----+     +----+-----+
    Forwarding |  WIA-PA  |------>| backhaul |---->|  WIA-PA  |
     Plane     |  network |<------|  network |<----|  network |
               +----------+       +----------+     +----------+

                   Figure 2. Centralized Architecture

   The centralized architecture does not need to expand east-west
   interface. It only needs to establish a connection with the basic
   controllers using southbound interface. After the connection is

Wang, et al.          Expires December 11, 2018               [Page 5]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

   established, the main controller obtains the every topology of
   network domain by the API interface provided by the basic
   controllers, and storages global network topology. Main controller
   can also assign tasks to basic controllers by API interface.

2.3. Joint Scheduling Architecture

   In practical application, distributed architecture not only needs to
   extend the east-west interface, but also maintains a global network
   topology in each controller. Only each controller maintains such a
   global network topology, can it ensure the deterministic control of
   whole network.

   For deterministic industrial network, the scale of network is not
   very large. Besides, in industrial backhaul network, a single SDN
   controller is sufficient to meet the demands of control. If
   centralized architecture is directly applied to an industrial
   network, it will not only be unable to make full use of advantages
   of the multi-controller architecture, but also cause unnecessary
   information interaction between controllers wasting network resource.

   Considering the problems existing in above two architectures, this
   document proposes a joint scheduling architecture based on the
   architecture document [I-D.finn-detnet-architecture]. The
   architecture is a semi-distributed structure in which the main
   controller can manage and control the basic controller, but the
   basic controller also has local network management capabilities, and
   the east-west interface between the controllers is also reserved.

   The joint scheduling architecture is optimized according to the
   characteristics of deterministic industrial network. A single SDN
   controller can unite the WIA-PA network system manager to manage the
   entire industrial network, and provide support for the deterministic
   scheduling of cross-network data transmission through industrial
   backhaul network located in different WIA-PA networks.

Wang, et al.          Expires December 11, 2018               [Page 6]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

             |                                                     |
    Application   +--------+       +--------+       +-------+      |
    Plane         |   APP  |       |   APP  |       |  APP  |      |
             |    +--------+       +----+---+       +-------+      |
             |                          |                          |
            |                           |                          |
    Control |  +--------------+    +----+-----+   +--------------+ |
    Plane   |  |    WIA-PA    |----|   SDN    |---|   WIA-PA     | |
            |  |System Manager|    |Controller|   |System Manager| |
            |  +------+-------+    +----+-----+   +-------+------+ |
            |         |                 |                 |        |
                      |                 |                 |
                      |                 |                 |
                 +----+-----+      +----+-----+     +-----+----+
    Forwarding   |  WIA-PA  |----->| backhaul |---->|  WIA-PA  |
    Plane        |  network |<-----| network  |<----|  network |
                 +----------+      +----------+     +----------+

                  Figure 3. Joint scheduling architecture

   Figure 3 depicts the joint scheduling architecture. The architecture
   can be mainly classified into three planes:

   o Forwarding plane: this plane contains various types of network
      equipment belonged to different networks. It is the physical
      entities for network transmission. To obtain the desired network
      functions from network manager, devices should abstract their own
      resources to provide to its network manager respectively.

   o Control plane: this plane is composed by WIA-PA system manager
      and SDN controller. Joint scheduler is integrated into the SDN
      controller by a way of plugin, and other WIA-PA system managers
      communicate with joint scheduler by establishing a connection
      with the SDN controller. Meanwhile, joint scheduler implement the
      management of industrial backhaul network by directly invoking
      the corresponding module of SDN controller.

Wang, et al.          Expires December 11, 2018               [Page 7]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

   o Application plane: this plane provides users with a unified
      interface about many variety of resources for the whole network.
      At the same time, it also provides an intuitive and user-friendly
      interface, which shields the complex network information of the

   When application plane triggers a joint scheduling task, SDN
   controller calculates path and allocates resource according to the
   task request from the application plane. Upon finishing calculation,
   SDN controller sends them using the unified joint scheduling
   interface to corresponding network manager, and then the network
   manager sends them to the industrial field network.

   Based on joint scheduling architecture, joint scheduler can realize
   control and scheduling for the entire industrial network, thus it
   can provide a real-time guarantee for each data stream.

3.  Joint Scheduling Scheme

   Taking WIA-PA network and IPv6-based backhaul network as an example,
   this section describes how the joint scheduling architecture works.
   Existing WIA-PA scheduling scheme only applies to WIA-PA field
   network. Scheduling scheme will fail once data transfers to backhaul
   networks. Joint scheduling scheme is an innovation and expansion
   compared to WIA-PA scheduling scheme.

   Firstly, in order to enable WIA-PA data to be transmitted and
   identified in industrial backhaul networks, joint scheduling scheme
   need to support deterministic scheduling to industrial backhaul

   Secondly, by conducting an optimization for original WIA-PA
   scheduling scheme, original scheduling scheme can work with joint
   scheduler, and simultaneously be applied to cross-domain network.

   Thirdly, due to the specificity of cross-border transmission
   services, the joint scheduling scheme for WIA-PA network VCR_ID and
   route ID is reclassified.

   Finally, due to system manager allocates short address to field
   device based on WIA-PA network address information independently.
   Thus the short address of field device in entire industrial network
   is uncertain. In order to identify the field device belonged to
   different network domains, the network identifier (PAN_ID) is
   applied to the joint scheduling scheme to identify different WIA-PA

Wang, et al.          Expires December 11, 2018               [Page 8]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

   After the SDN controller initiates joint scheduling module, WIA-PA
   system manager will actively establish a connection with the united
   scheduler. After the scheduler receives a cross-border transmission
   request, joint scheduler will send a request for obtaining topology
   information and node information to WIA-PA System Manager. Then, the
   scheduler will assign paths and network resources according to this
   information by pre-defined scheduling algorithm. After the path and
   network resources have been calculated, joint scheduler will
   configure and deploy networks by the corresponding network

3.1. WIA-PA Network Joint Scheduling

   In joint scheduling process, the joint scheduler communicates with
   WIA-PA system manager by invoking the east-west interfaces to
   implement route deployment and resource allocation to the WIA-PA
   network. The system manager will configure communication resources
   for the corresponding gateway device, router and field equipment
   when it receives configuration commands from joint scheduler. Then
   the system manager constructs a reply message based on the relevant
   network information and returns it to the joint scheduler by the
   east-west interface. It is worth noting that both the WIA-PA system
   manager and the joint scheduler need to support the parsing and
   construction of network messages in the east-west interface.

3.2. Protocol Conversion

   For cross-domain transmission, industrial backhaul network is
   different from WIA-PA network which is not an IP-based Ethernet.
   Protocol conversion for WIA-PA packet in gateway is needed when data
   generated from WIA-PA network needs to transmit to another field
   network through industrial backhaul. In this scheme, the WIA-PA
   gateway will translate the WIA-PA frames into IPv6 packets, and fill
   the VCR of WIA-PA frames and the priority information into the IPv6
   header. Meanwhile, according to the joint scheduling scheme, joint
   scheduler is able to recognize the WIA-PA data stream and allocate
   resources according to the types of WIA-PA frame.

Wang, et al.          Expires December 11, 2018               [Page 9]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

          | Start |
        |  Receiving  |
        |data packets |
        /-------------\           +--------------+
       /Whether is the \          | Forwarded to |
       |  management   |-- Yes -->|  the system  |
       \     data      /          |    manager   |
        \-------------/           +--------------+
        | Resolution|
        |   Packet  |
        /------------\          /------------\        +-------------+
       /   Find the   \        /   Find the   \       | Encapsulate |
       |corresponding |- Yes ->| corresponding|- Yes->|   and sent  |
       \     VCR      /        \ IPv6 address /       |  IPv6 packet|
        \------------/          \------------/        +-------------+
             |                         |                      |
             No                        No                     |
             |                         |                      |
         +--------+                    |                      |
         |  End   |<-------------------+----------------------+

          Figure 4. The process of protocol conversion in gateway

   As shown in Figure 4, the WIA-PA gateway will receive the address
   mapping configuration command from joint scheduler. Then VCR tables
   and IPv6 address-mapping tables will be formed according to this
   information. When gateway receives WIA-PA packets, it will firstly
   parse Route ID, Object ID and Instance ID, and find corresponding
   VCR from VCR tables. Meanwhile, the gateway finds the corresponding
   IPv6 address according to Route ID in IPv6 address mapping table.
   Then, gateway begins to encapsulate WIA-PA packets based on IPv6
   format, fill VCR_ID in IPv6 header flow label field and the priority
   of WIA-PA packet in IPv6 header fields.

Wang, et al.          Expires December 11, 2018              [Page 10]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

   When receiving IPv6 packets from industrial backhaul networks,
   gateway will recognize VCR_ID from IPv6 packet header, and obtain
   packet VCR according to the VCR ID in VCR table, then replace it
   with the information of original packet.

3.3. Industrial Backhaul Network Scheduling

   In deterministic network based on SDN, in order to achieve
   communication with gateway, joint scheduler needs to configure the
   paths for SDN switches in the industrial backhaul network. Meanwhile,
   joint scheduler can classify different types of WIA-PA data streams
   by matching IPv6 flow label field. Meanwhile, according to the
   priority information in IPv6 header field and VCR_ID type, joint
   scheduling can allocate the necessary resources to communication and
   ensure that important data flow is not affected when adding new data
   flows in existing network. It can also monitor the real-time data
   flow. To guarantee the real-time performance of critical data flows,
   redundant paths are also considered when necessary. The scheduling
   process of industrial backhaul network is shown in Figure 5.

Wang, et al.          Expires December 11, 2018              [Page 11]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

                   | Start |
                |Obtain network|
                |   topology   |
              |    Calculate the  |
              | path and allocates|<--------------+
              |    resources      |               |
              +-------------------+               |
                      |                           |
                 +----------+                     |
                 |Query path|                     |
                 +----------+                     |
                      |                           |
                /---------------\         +----------------+
               /whether the path \        |  Calculate the |
               |meets the resource|- No ->|   weight and   |
               \  requirements   /        | adjustment path|
                \---------------/         +----------------+
                | Deployment |
                | flow table |
                   | End |

      Figure 5. The scheduling process of industrial backhaul network

   When receiving the request for service, the joint scheduler will
   calculate route information and resource allocation. Once the path
   and resource allocation are determined, joint scheduling will
   confirm whether the resource and path are capable of meeting
   business requirements by the inside module of SDN controller. If it
   meets, then the flow table is deployed by SDN controller. Otherwise,
   the path information and resource allocation are recalculated to
   choose the correct paths to transmit data flow.

Wang, et al.          Expires December 11, 2018              [Page 12]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

3.4. Bandwidth guarantee method

   Bandwidth guarantee method is implemented on the basis of joint
   scheduling mechanism, in order to solve the problem that industrial
   backhaul networks can not identify fine-grained and cross-network
   data transmission. By filling the priority information and the
   RouteID defined in WIA-PA network into the IPv6 header, the SDN
   controller can not only identify cross-network transmission of the
   WIA-PA data stream, but also obtain priority information about the
   WIA-PA data stream. In industrial backhaul network, the SDN switch
   employs the mechanism of priority queue to allocate network
   bandwidth. Thus SND switch can distribute WIA-PA data streams into
   corresponding queues of ports according to the received flow table
   from SDN controller. Therefore, the bandwidth of data stream is

   By using the above methods, joint scheduling mechanism can
   distinguish data streams in a fine-grained way and guarantee
   bandwidth when data transmits in industrial backhaul network. For
   example, real-time data in WIA-PA network is sensitive to delay,
   thus it should be allocated more bandwidth to reduce transmission
   delay. For not urgent data, it can be assigned less bandwidth to
   reserve bandwidth for real-time data. Meanwhile, SDN controller can
   flexibly adjust bandwidth allocation strategy to relieve network

4. Security Considerations

5. IANA Considerations

   This memo includes no request to IANA.

6. References

6.1. Normative References

6.2. Informative References

           ISA/IEC, "ISA100.11a, Wireless Systems for Automation,
           also IEC 62734", 2011, <http://www.isa100wci.org/enUS/

Wang, et al.          Expires December 11, 2018              [Page 13]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

           IEC, "Industrial Communication Networks -
           Wireless Communication Network and Communication Profiles
           - WirelessHART - IEC 62591", 2010,

           IEC, "Industrial networks - Wireless communication network
           and communication profiles - WIA-PA - IEC 62601", 2015, <

           Finn, N. and P. Thubert, "Deterministic Networking Problem
           Statement", draft-finn-detnet-problem-statement-05 (work in
           progress), March 2016.

           Finn, N., Thubert, P., and M. Teener, "Deterministic
           Networking Architecture", draft-finn-detnet-architecture-08
           (work in progress), August 2016.

           Kaneko, Y., Toshiba and Das, S, "Building Automation Use
           Cases and Requirements for Deterministic Networking", draft-
           bas-usecase-detnet-00 (work in progress), October 2015.

Wang, et al.          Expires December 11, 2018              [Page 14]

 Internet-Draft        DetNet Backhaul Architecture           June 2018

Authors' Addresses

   Heng Wang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065

  Phone: (86)-23-6248-7845
   Email: wangheng@cqupt.edu.cn

   Ping Wang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065

   Phone: (86)-23-6246-1061
   Email: wangping@cqupt.edu.cn

   Chang Zhang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065

   Phone: (86)-23-6246-1061
   Email: zc910522@126.com

   Yi Yang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065

   Phone: (86)-23-6246-1061
   Email: 15023705316@163.com

Wang, et al.          Expires December 11, 2018              [Page 15]

Html markup produced by rfcmarkup 1.129b, available from https://tools.ietf.org/tools/rfcmarkup/