[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00 01

Internet-Draft                                Bhuvaneswaran Vengainathan
Network Working Group                                        Anton Basil
Intended Status: Informational                        Veryx Technologies
Expires: September 2014                                   Vishwas Manral
                                                              Ionos Corp
                                                          March 24, 2014


    Benchmarking Methodology for OpenFlow SDN Controller Performance
             draft-bhuvan-bmwg-of-controller-benchmarking-00

Abstract

   This document discusses and defines a number of tests that may be
   used to measure the performance characteristics of OpenFlow (OF) SDN
   controller. In addition to defining the tests, this document
   also describes specific formats for reporting the results of the
   tests.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF). Note that other groups may also distribute
   working documents as Internet-Drafts. The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time. It is inappropriate to use Internet-Drafts
   as reference material or to cite them other than as "work in
   progress.

   This Internet-Draft will expire on September 20, 2014.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors. All rights reserved.










Bhuvan                Expires September 20, 2014                [Page 1]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.

Table of Contents

   1.  Introduction   . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  3
   3.  Scope    . . . . . . . . . . . . . . . . . . . . . . . . . . .  4
   4.  Test Setup   . . . . . . . . . . . . . . . . . . . . . . . . .  4
   4.1.  Test Setup for Standalone Controller   . . . . . . . . . . .  4
   4.1.1.  Proactive Mode   . . . . . . . . . . . . . . . . . . . . .  4
   4.1.2.  Reactive Mode   . . . . . . . . . . . . . . .  . . . . . .  5
   4.2.  Test Setup for Controller Teaming    . . . . . . . . . . . .  5
   4.2.1.  Proactive Mode   . . . . . . . . . . . . . . . . . . . . .  5
   4.2.2.  Reactive Mode  . . . . . . . . . . . . . . . . . . . . . .  6
   5.  Test Considerations  . . . . . . . . . . . . . . . . . . . . .  6
   5.1.  Virtual Switches/Applications  . . . . . . . . . . . . . . .  6
   5.2.  Test Traffic Requirements  . . . . . . . . . . . . . . . . .  6
   5.3.  Southbound Connection Setup Requirements   . . . . . . . . .  7
   5.3.1.  OpenFlow Channel Setup   . . . . . . . . . . . . . . . . .  7
   5.3.2.  Test Recommendations   . . . . . . . . . . . . . . . . . .  8
   6.  Benchmarking Tests   . . . . . . . . . . . . . . . . . . . . .  8
     6.1  Performance   . . . . . . . . . . . . . . . . . . . . . . .  8
       6.1.1  Flow setup rate   . . . . . . . . . . . . . . . . . . .  8
         6.1.1.1  Flow setup rate under constant network load   . . .  8
         6.1.1.2  Flow setup rate under incremental network load  . . 10
       6.1.2  Flow setup delay  . . . . . . . . . . . . . . . . . . . 12
         6.1.2.1  Flow setup delay under constant network load  . . . 12
         6.1.2.2  Flow setup delay under incremental network load . . 13
         6.1.2.3  Flow setup delay under stress network load  . . . . 14
       6.1.3  End-End flow setup duration   . . . . . . . . . . . . . 16
     6.2  Scalability   . . . . . . . . . . . . . . . . . . . . . . . 18
       6.2.1  OpenFlow connections capacity   . . . . . . . . . . . . 17
       6.2.2  Switch scalable limit   . . . . . . . . . . . . . . . . 19
       6.2.3  Flow scalable limit   . . . . . . . . . . . . . . . . . 20
     6.3  Reliability   . . . . . . . . . . . . . . . . . . . . . . . 21
       6.3.1  Errored OpenFlow connections handling   . . . . . . . . 21
       6.3.2  Denial of service handling  . . . . . . . . . . . . . . 23
       6.3.3  Controller failover time  . . . . . . . . . . . . . . . 24
       6.3.4  Data path re-convergence time   . . . . . . . . . . . . 24




Bhuvan                Expires September 20, 2014                [Page 2]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   7.  References   . . . . . . . . . . . . . . . . . . . . . . . . . 26
     7.1.  Normative References   . . . . . . . . . . . . . . . . . . 26
     7.2.  Informative References    . . . . . . . . . . . . . . . .  26
   8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 26
   9.  Security Considerations  . . . . . . . . . . . . . . . . . . . 26
   10.  Appendix A - Test setup and test applicability  . . . . . . . 26
   11.  Appendix B - OpenFlow protocol overview   . . . . . . . . . . 27
     11.1.  Protocol Overview   . . . . . . . . . . . . . . . . . . . 27
     11.2.  Messages Overview   . . . . . . . . . . . . . . . . . . . 27
     11.3.  Connection Overview   . . . . . . . . . . . . . . . . . . 28
   12.  Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . 28

1. Introduction

   This document provides methodologies for benchmarking OpenFlow SDN
   controller performance. This includes benchmarking controller
   for flow performance, scalability and reliability. In addition to
   defining tests, this document also describes specific formats for
   reporting test results. The results of these tests will provide
   the vendor and user quantitative data with which to evaluate the
   controller performance.

   Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
   this document are to be interpreted as described in RFC 2119.

2. Terminology

   Reactive Flow Setup: Controller can add, update or delete flow
   entries in flow tables of a switch in response to packets received
   from the switch

   Proactive Flow Setup: Controller can add, update or delete flow
   entries in flow tables of a switch based on trigger from
   controller's management plane.

   REST: Representational state transfer

   OFR: OpenFlow response messages










Bhuvan                Expires September 20, 2014                [Page 3]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


3. Scope

   SDN defines logically centralized control plane to enable network
   programmability and agility. The logically centralized control
   plane can be a composition of single SDN controller (standalone) or
   a cluster of SDN controllers (controller teaming) enabling network
   programmability through one or more southbound interfaces. OpenFlow
   is one among the protocol widely used for southbound communication.
   This document defines methodologies for benchmarking OpenFlow based
   SDN controller performance independent of composition.

4. Test Setup

   Test configurations defined in this document will be confined to
   standalone controller and cluster of controllers supporting
   proactive and/or reactive mode of flow setup. Not all tests
   described in this document are to be performed for all test setups.
   These tests are performed by simulating real network scenarios in a
   lab environment. Appendix A lists the test applicability for each
   test setup. All of the tests that are relevant to an SDN controller
   SHOULD be performed.

4.1. Test Setup for Standalone Controller

4.1.1. Proactive Mode

                          --------------------
                         |    Orchestration   |
                          --------------------
                                   |
                                   | (Northbound interface)
                         -----------------------
                        |     SDN Controller    |
                        |          (DUT)        |
                         -----------------------
                                   | (Southbound interface)
                                   |
                       ---------------------------
                      |            |              |
                  ----------    ----------    ----------
                 |  SDN     |  |   SDN    |..|   SDN    |
                 | Switch 1 |  | Switch 2 |  | Switch n |
                  ----------    ----------    ----------

                                  Figure 1






Bhuvan                Expires September 20, 2014                [Page 4]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


4.1.2. Reactive Mode

                         -----------------------
                        |     SDN Controller    |
                        |          (DUT)        |
                         -----------------------
                                   | (Southbound interface)
                                   |
                       ---------------------------
                      |            |              |
                  ----------    ----------    ----------
                 |  SDN     |  |   SDN    |..|   SDN    |
                 | Switch 1 |  | Switch 2 |  | Switch n |
                  ----------    ----------    ----------

                                Figure 2

4.2. Test Setup for Controller Teaming

   In controller clustering, the connectivity and the communication
   between controllers are outside the scope of this document.

4.2.1. Proactive Mode
                       --------------------
                      | Orchestration      |
                       --------------------
                                |
                                |
                                | (Northbound interface)
        ---------------------------------------------------------
       |  ------------------             ------------------      |
       | | SDN Controller 1 | <--E/W--> | SDN Controller n |     |
       |  ------------------             ------------------      |
        ---------------------------------------------------------
                                | (Southbound interface)
                                |
                    ---------------------------
                   |            |              |
               ----------    ----------    ----------
              |  SDN     |  |   SDN    |..|   SDN    |
              | Switch 1 |  | Switch 2 |  | Switch n |
               ----------    ----------    ----------

                           Figure 3







Bhuvan                Expires September 20, 2014                [Page 5]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


4.2.2. Reactive Mode

        ---------------------------------------------------------
       |  ------------------             ------------------      |
       | | SDN Controller 1 | <--E/W--> | SDN Controller n |     |
       |  ------------------             ------------------      |
        ---------------------------------------------------------
                                | (Southbound interface)
                                |
                    ---------------------------
                   |            |              |
              ----------    ----------    ----------
             |  SDN     |  |   SDN    |..|   SDN    |
             | Switch 1 |  | Switch 2 |  | Switch n |
              ----------    ----------    ----------

                           Figure 4

5. Test Considerations

5.1. Virtual Switches/Applications

   SDN controller performance testing involves connections and flows
   setup from multiple switches, hosts and/or applications. These
   entities may not necessarily be a real physical hardware and
   can be emulated in a single hardware platform. The test report
   MUST indicate the number of switches and hosts used in the
   test.

5.2. Test Traffic Requirements

   While the key function of an SDN controller is to program the data
   plane, the SDN controller may use Layer 2, Layer 3 or Layer 4
   fields for programming the data plane. The author understands that
   it will take a considerable period of time to perform tests for
   various traffic types and packet sizes. We believe that the results
   are worth the effort. For the purposes of benchmarking SDN
   controller performance, the document references traffic type IP of
   size 128 bytes












Bhuvan                Expires September 20, 2014                [Page 6]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


5.3. Southbound Connection Setup Requirements

   The methodologies defined in this document are independent of
   OpenFlow version and channel setup ways and connection type (e.g.,
   TCP/TLS or SSL).However this document describes a number of channel
   setup ways, and recommends some of the channel setup ways and
   connection type over which the tests MUST be carried out and
   measure the performance.

5.3.1. OpenFlow Channel Setup

   This section describes three ways of OpenFlow channel setup.

5.3.1.1. Single Connection

   By default, a single OpenFlow channel is established between each
   OpenFlow switch and controller when a single controller
   (standalone) manages a network.

5.3.1.2. Multiple Connections

   Multiple OpenFlow channels are established from each OpenFlow
   switch to multiple controllers when multiple controllers are
   clustered and manage a network In such case, each OpenFlow switch
   can employ any one of the below modes.

   Active-Passive:
   In this mode, one controller takes a MASTER role and other
   controllers in the cluster take SLAVE role. The selection process
   is out of this document scope. The OpenFlow channel between the
   switch and master controller is in active state and others are in
   standby state. Any asynchronous communications from the switch have
   to be sent on the active channel.

   Active-Active:
   In this mode, all controllers take role as EQUAL. The OpenFlow
   channel between the switch and all controllers are in active state.
   Communication between OpenFlow switch and controller cluster can be
   sent on any active channels.

5.3.1.3. Auxiliary Connection

   Additional connections apart from the primary connection (as
   described in section 5.3.1.1. and 5.3.1.2.) can be established
   between a switch to a controller in order to improve the switch
   processing performance.





Bhuvan                Expires September 20, 2014                [Page 7]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


5.3.2. Test Recommendations

   For standalone controller, the performance tests MUST be carried on
   'single connection' using 'TCP'. Tests over other connections like
   'auxiliary connections' are optional, depending on a specific
   support of the product.

   For controller teaming, the performance tests MUST be carried on
   'multiple connections - active-passive mode' using 'TCP'. Tests
   over other connections like 'multiple connections - active-active',
   'auxiliary connections' are optional, depending on a specific
   support of the product.

6.  Benchmarking Tests

6.1  Performance

6.1.1  Flow setup rate

   Maximum number of flows setup by a controller, expressed in flows
   per second.

   This metric is measured in two modes, using constant network load
   and incremental network load.

6.1.1.1  Flow setup rate under constant network load

6.1.1.1.1  Objective

   To measure the total number of OpenFlow flow responses (Packet-
   Out/Flow-Mod) received from the controller under constant network
   load (fixed number of switches and flows)

6.1.1.1.2  Setup Parameters

   The following parameters MUST be defined:

   Network setup parameters:

      Number of Switches - Defines the number of OpenFlow switch
      connections established with controller.

      Number of Flows - Defines unique number of flows setup on each
      OpenFlow connections.







Bhuvan                Expires September 20, 2014                [Page 8]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   OpenFlow setup parameters:

      OpenFlow Version - Defines OpenFlow protocol version used for
      OpenFlow connection setup.

      Channel Type - Defines OpenFlow connection setup type used for
      OpenFlow connection setup, for example TCP or TLS.

   Test setup parameters:

      Test Iterations - Defines the number of times the test need to
      be repeated.

      Test Duration - Defines the duration of test iteration,
      expressed in seconds.

6.1.1.1.3  Procedure

   Reactive Mode Flow Setup:

      Establish OpenFlow connection from all the specified number of
      switches. Load the controller with Packet-In messages send from
      all switches for a configured number of flows and for a specified
      test duration. Single Packet-In message can have one or more
      flow headers in its payload.

   Proactive Mode Flow Setup:

      Establish OpenFlow connection from all the specified number of
      switches. Load the controller with configured number of flows
      through the northbound interface of controller for a specified
      test duration.

   The test SHOULD be repeated a number of times for each of the above
   procedure and record the results. It is recommended to give
   sufficient time for the controller to flush all the pending
   response between test iteration. On the completion of the test,
   gracefully close all the OpenFlow sessions established with
   controller.












Bhuvan                Expires September 20, 2014                [Page 9]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.1.1.1.4  Measurement

                                  Total (OFRi)
   Flow setup rate (FSRi)       = ---------------
                                  Test Duration

                                  SUM (FSR1, FSR2,..FSRn)
   Average flow setup rate      = ------------------------
                                  Test Iterations

   Maximum flow setup rate      = Max (FSR1,FSR2,..FSRn)

   Minimum flow setup rate      = Min (FSR1,FSR2,..FSRn)

   Where,
   FSR1 is the average OpenFlow responses received in 1st iteration
   and FSRn is the average OpenFlow responses received in nth
   iteration.

   Measurement MUST take into account all the Packet-Outs embedded in
   a single OpenFlow response message (Packet-Out/Flow-Mod).

6.1.1.1.5  Reporting Format

   The results of the flow setup rate under constant network load test
   for each of the test procedure SHOULD be plotted as a graph. If
   this is done then the X axis MUST be the test iteration number The
   Y axis MUST be the OpenFlow responses per second. The Y axis value
   range should be between the minimum and maximum OpenFlow setup
   rate. Plot the flow setup rate results obtained on each test
   iteration in the graph. The maximum, average and minimum flow setup
   rate should be reported at the bottom of the graph. The test report
   MUST indicate whether learning was enabled or disabled.

   Also report the total number of frames transmitted, number of valid
   responses and error responses received for each iteration in a
   table.

6.1.1.2  Flow setup rate under incremental network load

6.1.1.2.1  Objective

   To measure the total OpenFlow flow responses (Packet-Out/Flow-Mod)
   received from the controller for incremental network load
   (increasing number of switches and flows).

6.1.1.2.2  Setup Parameters

   Use the same setup parameters as defined in section 6.1.1.1.2


Bhuvan                Expires September 20, 2014               [Page 10]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.1.1.2.3  Procedure

   Reactive Mode Flow Setup:

      Establish OpenFlow connection from 10% of the specified number
      of switches. Load controller with Packet-In messages send from
      10% of switches for a configured number of flows and for a
      specified sample interval. Single Packet-In message can have one
      or more flow headers in its payload.

   The test SHOULD be repeated with 20% of the specified number of
   switches in the next iteration and continue the test until the
   configured number of switches is included in the test. On the
   completion of the test, gracefully close all the OpenFlow sessions
   established with controller.

6.1.1.2.4  Measurement

                                  Total (OFRi)
   Flow setup rate (FSRi)       = ---------------
                                  Test Duration

   Maximum flow setup rate      = Max (FSR1,FSR2,..FSR10)


   Minimum flow setup rate      = Min (FSR1,FSR2,..FSR10)

   Where,
   FSR1 is the average OpenFlow responses received in 1st iteration
   and FSR10 is the average OpenFlow responses received in 10th
   iteration.

   Measurement MUST take into account all the Packet-Outs embedded in
   a single OpenFlow response message (Packet-Out/Flow-Mod).

6.1.1.2.5  Reporting Format

   The results of the flow setup rate under incremental network load
   test SHOULD be plotted as a graph. If this is done then the X axis
   MUST be the number of switches per test iteration The Y axis MUST
   be the OpenFlow responses per second. The Y axis value range should
   be between the minimum and maximum OpenFlow setup rate. Plot the
   flow setup rate results obtained on each test iteration in the
   graph. The maximum and minimum flow setup rate should be reported
   at the bottom of the graph. The test report MUST indicate whether
   learning was enabled or disabled.





Bhuvan                Expires September 20, 2014               [Page 11]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   Also report the total number of frames transmitted, number of valid
   responses and error responses received for each iteration in a
   table.

6.1.2  Flow setup delay

   Time taken by the controller to setup a flow, expressed in
   milliseconds.

   This metric is measured in three modes, using constant network
   load, incremental network load and stress network load.

6.1.2.1  Flow setup delay under constant network load

6.1.2.1.1  Objective

   To measure the time taken by controller to setup a flow under low
   network load.

6.1.2.1.2  Setup Parameters

   Use the same setup parameters as defined in section 6.1.1.1.2

6.1.2.1.3  Procedure

   Reactive Mode Flow Setup:

      Establish OpenFlow connection from all the specified number of
      switches. Send Packet-In messages from each of the switches one
      at a time and wait for the response before soliciting the next
      flow request.  Single Packet-In message MUST have one flow
      header in its payload. This process is repeated as many times as
      possible within the configured sample time interval.

   The test SHOULD be repeated a number of times and record the
   values. On the completion of the test, gracefully close all the
   OpenFlow sessions established with controller.

6.1.2.1.4  Measurement
                                  Test Duration
   Flow setup delay (FSDi)      = ---------------
                                  Total (OFRi)

                                  SUM (FSD1, FSD2,..FSDn)
   Average flow setup delay     = -------------------------
                                  Test Iterations





Bhuvan                Expires September 20, 2014               [Page 12]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   Maximum flow setup delay     = Max (FSD1,FSD2,..FSDn)

   Minimum flow setup delay     = Min (FSD1,FSD2,..FSDn)

   Where,
   FSD1 is the average time taken to setup a flow in first iteration
   and
   FSDn is the average time taken to setup a flow in nth iteration.

6.1.2.1.5  Reporting Format

   The results of the flow setup delay under constant network load
   test SHOULD be plotted as a graph. If this is done then the X axis
   MUST be the test iteration number The Y axis MUST be the Flow setup
   delay in milliseconds. The Y axis value range should be between the
   minimum and maximum OpenFlow setup delay. Plot the flow setup delay
   results obtained on each test iteration in the graph. The maximum,
   average and minimum flow setup delay should also be reported at the
   bottom of the graph. The test report MUST indicate whether learning
   was enabled or disabled.

   Also report the total number of frames transmitted, number of valid
   responses and error responses received for each iteration in a
   table.

6.1.2.2  Flow setup delay under incremental network load

6.1.2.2.1  Objective

   To measure the amount of time taken by controller to setup a flow
   under gradual increase of network load.

6.1.2.2.2  Setup Parameters

   Use the same setup parameters as defined in section 6.1.1.1.2

6.1.2.2.3  Procedure

   Reactive Mode Flow Setup:

      Establish OpenFlow connection from 10% of the specified number
      of switches. Send Packet-In messages from each of the switches
      one at a time and wait for the response before soliciting the
      next flow request.  Single Packet-In message MUST have one flow
      header in its payload. This process is repeated as many times as
      possible within the configured sample time interval.





Bhuvan                Expires September 20, 2014               [Page 13]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   The test SHOULD be repeated with 20% of the specified number of
   switches in the next iteration and continue the test until the
   configured number of switches are included in the test. On the
   completion of the test, gracefully close all the OpenFlow sessions
   established with controller.

6.1.2.2.4  Measurement

                                  Test Duration
   Flow setup delay (FSDi)      = ---------------
                                  Total (OFRi)

   Maximum flow setup delay     = Max (FSD1,FSD2,..FSD10)

   Minimum flow setup delay     = Min (FSD1,FSD2,..FSD10)

   Where,
   FSD1 is the average time taken to setup a flow in first iteration
   and
   FSD10 is the average time taken to setup a flow in 10th iteration.

6.1.2.2.5  Reporting Format

   The results of the flow setup delay under incremental network load
   test SHOULD be plotted as a graph. If this is done then the X axis
   MUST be the number of switches per test iteration. The Y axis MUST
   be the Flow setup delay in milliseconds. The Y axis value range
   should be between the minimum and maximum OpenFlow setup delay.
   Plot the flow setup delay results obtained on each test iteration
   in the graph. The maximum and minimum flow setup delay should also
   be reported at the bottom of the graph. The test report MUST
   indicate whether learning was enabled or disabled.

   Also report the total number of frames transmitted, number of valid
   responses and error responses received for each iteration in a
   table.

6.1.2.3  Flow setup delay under stress network load

6.1.2.3.1  Objective

   To measure the amount of time taken by controller to setup a flow
   under high network load.

6.1.2.3.2  Setup Parameters

   Use the same setup parameters as defined in section 6.1.1.1.2



Bhuvan                Expires September 20, 2014               [Page 14]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.1.2.3.3  Procedure

   Reactive Mode Flow Setup:

      Establish OpenFlow connection from all the specified number of
      switches. Reserve one switch as a test switch and send Packet-In
      messages from other switches for a configured number of flows
      for a specified sample interval. The Packet-In message sent from
      other switches can have one or more flow headers in its payload.

      From the test switch, send Packet-In message one at a time and
      wait for the response before soliciting the next flow request.
      At regular frequency, insert time stamp to the Packet-In message
      and measure the round trip time it takes for the flow to get
      added in the switch. This process is repeated as many times as
      possible within the configured sample time interval.

   The test SHOULD be repeated a number of times and record the
   values. On the completion of the test, gracefully close all the
   OpenFlow sessions established with controller.

6.1.2.3.4  Measurement

                                  Test Duration
   Flow setup delay (FSDi)      = -------------------------
                                  Total (OFRi on test switch)

                                  SUM (FSD1, FSD2,..FSDn)
   Average flow setup delay     = ------------------------
                                  Test Iterations

   Maximum flow setup delay     = Max (FSD1,FSD2,..FSDn)

   Minimum flow setup delay     = Min (FSD1,FSD2,..FSDn)

   Where,
   FSD1 is the average time taken to setup a flow in first iteration
   and
   FSDn is the average time taken to setup a flow in nth iteration.












Bhuvan                Expires September 20, 2014               [Page 15]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.1.2.3.5  Reporting Format

   The results of the flow setup delay under constant network load
   test SHOULD be plotted as a graph. If this is done then the X axis
   MUST be the test iteration number The Y axis MUST be the Flow setup
   delay in milliseconds. The Y axis value range should be between the
   minimum and maximum OpenFlow setup delay. Plot the flow setup delay
   results obtained on each test iteration in the graph. The maximum,
   average and minimum flow setup delay should also be reported at the
   bottom of the graph. The test report MUST indicate whether learning
   was enabled or disabled.

   Also report the total number of frames transmitted, number of valid
   responses and error responses received for each iteration in a
   table.

6.1.3  End-End flow setup duration

6.1.3.1  Objective

   To determine the time taken by the controller to setup a flow
   between a source to a destination, expressed in milliseconds.

6.1.3.2  Setup Parameters

   The following parameters MUST be defined:

   Network setup parameters:

      Number of switches - Total number of switches inter-connected in
      a topology.

   OpenFlow setup parameters:

      OpenFlow Version - Defines OpenFlow protocol version used for
      OpenFlow connection setup.

      Channel Type - Defines OpenFlow connection setup type used for
      OpenFlow connection setup, for example TCP or TLS.












Bhuvan                Expires September 20, 2014               [Page 16]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.1.3.4  Procedure

   Inter connect the specified number of switches in linear or tree
   fashion. Establish OpenFlow connection from all switches. To ensure
   that the controller completes the network discovery process, send
   traffic stream from a source connected on one end of the network to
   the destination connected on the other end of network and verify
   that the traffic reaches the destination properly. Prior sending
   the traffic, send gratuitous ARP to the controller for the
   destination.

   On completion of above verification, send a traffic stream to a
   different destination with the timestamp inserted on each frame.

   On completion of the test, gracefully close all the OpenFlow
   connection established with the controller.

   The test SHOULD be repeated by varying the number of inter-
   connected switches.

6.1.3.5  Measurement

   Flow setup delay = Flow transmit time at the source - Flow
   reception time at the destination.

6.1.3.6  Reporting Format

   The test results MUST be reported in a table format with a row for
   number of switches and a column for Flow setup delay.

6.2 Scalability

6.2.1 OpenFlow connections capacity

6.2.1.1  Objective

   Maximum number of concurrent OpenFlow connections supported by a
   controller.

6.2.1.2  Setup Parameters

   The following parameters MUST be defined:

   Network setup parameters:

      Connection Setup Rate - Maximum number of TCP connection requests
      accept per second.




Bhuvan                Expires September 20, 2014               [Page 17]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   OpenFlow setup parameters:

      OpenFlow Version - Defines OpenFlow protocol version used for
      OpenFlow connection setup.

      Channel Type - Defines OpenFlow connection setup type used for
      OpenFlow connection setup, for example TCP or TLS.

   Test setup parameters:

      Test Iterations - Defines the number of times the test need to be
      repeated.

6.2.1.3  Procedure

   Establish OpenFlow connection with the controller at the specified
   connection setup rate until the controller drops the OpenFlow
   connection request.

   To validate each connection, send a Packet-In message after the
   OpenFlow connection has been established. On the completion of the
   test, gracefully close all the OpenFlow sessions established with
   controller.

   The test SHOULD be repeated a number of times. Each iteration,
   record total number of responses received from the controller and
   the test duration, in milliseconds.

6.2.1.4  Measurement

   Maximum Concurrent OpenFlow Connections = Max (OFR1, OFR2, .. OFRn)

   Minimum OpenFlow Connection Establishment Time, in milliseconds

                        Test duration of Min (OFR1, OFR2, .. OFRn)
                      = ------------------------------------------
                        Min (OFR1, OFR2, .. OFRn)

6.2.1.5  Reporting Format

   The results of the OpenFlow connection capacity test SHOULD be
   plotted as a graph. If this is done then the X axis MUST be the
   test iteration number The Y axis MUST be the OpenFlow connection
   capacity. Plot the OpenFlow responses obtained on each test
   iteration in the graph. The Maximum Concurrent OpenFlow Connections
   and Minimum OpenFlow Connection Establishment Time should be
   reported at the bottom of the graph.




Bhuvan                Expires September 20, 2014               [Page 18]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.2.2  Switch scalable limit

6.2.2.1  Objective

   To determine the number of switches a controller can optimally
   manage.

6.2.2.2  Setup Parameters

   The following parameters MUST be defined:

   Network setup parameters:

      Acceptable Flow Setup Delay - Acceptable flow response time,
      expressed in milliseconds.

      Connection Setup Rate - Maximum number of TCP connection requests
      accept per second.

   Test setup parameters:

      Test Iterations - Defines the number of times the test need to be
      repeated.

   OpenFlow setup parameters:

      OpenFlow Version - Defines OpenFlow protocol version used for
      OpenFlow connection setup.

      Channel Type - Defines OpenFlow connection setup type used for
      OpenFlow connection setup, for example TCP or TLS.

6.2.2.3  Procedure

   Establish OpenFlow connection with the controller at the specified
   connection setup rate. Reserve one switch as a test switch and send
   Packet-In messages from other switches at a constant load. The
   Packet-In message sent from other switches can have one or more
   flow headers in its payload.

   From the test switch, send Packet-In message one at a time and wait
   for the response before soliciting the next flow request.  At
   regular frequency, insert time stamp to the Packet-In message and
   measure the round trip time it takes for the flow to get added in
   the switch. Compare the measured flow round trip time with the
   configured acceptable flow setup delay value at each step.





Bhuvan                Expires September 20, 2014               [Page 19]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   The above step should be continued until the measured response time
   is lower or equal to the configured response time. On the
   completion of the test, gracefully close all the OpenFlow sessions
   established with controller.

   The test SHOULD be repeated with varying number of network loads.

6.2.2.4  Measurement

   Switch Scalable Limit = Number of switch connections active at the
   step when the measured value starts exceeding the acceptable value

6.2.2.5  Reporting Format

   The test report MUST note the switch scalable limit factor and the
   offered network load in terms of flows for every iteration. The
   results SHOULD be reported in the format of a table with a row for
   offered network load and columns for number of switch connections
   attempted and number of active switch connections (Switch Scalable
   Limit).

6.2.3 Flow scalable limit

6.2.3.1  Objective

   To determine the controller OpenFlow table capacity

6.2.3.2  Setup Parameters

   The following parameters MUST be defined:

   Network setup parameters:

      Number of Flows - Defines unique number of flows send on the
      established OpenFlow connection at the beginning of the test.

   OpenFlow setup parameters:

      OpenFlow Version - Defines OpenFlow protocol version used for
      OpenFlow connection setup.

      Channel Type - Defines OpenFlow connection setup type used for
      OpenFlow connection setup, for example TCP or TLS.








Bhuvan                Expires September 20, 2014               [Page 20]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.2.3.3  Procedure

   Establish OpenFlow connection from a switch with the controller.
   Send Packet-In messages from the established connection for a
   constant number of flows and measure the Flow-Mod messages received
   from the controller. Make sure that the controller has learnt all
   flows destination prior the test. One way to achieve this is by
   sending Packet-In messages with gratuitous ARP frame for all flows.

   The Packet-In message sent on each connection can have one or more
   flow headers in its payload.

   Repeat the iteration by varying the number of flows until the
   maximum flow request, at which no flow response loss occurs, is
   found. Since the controller may employ aging time mechanism for the
   flow, the controller flow aging time SHOULD be set to a maximum
   possible value

   On the completion of the test, gracefully close the OpenFlow
   session established with controller.

6.2.3.4  Measurement

   Flow Scalable Limit:

      Maximum flow request, expressed in packets per second, at which
      no flow response loss is detected.

6.2.3.5  Reporting Format

   The test report MUST note the offered load and the responses
   received on each iteration. The results SHOULD be reported in the
   format of a table with a row for the offered load and a column for
   the number of responses received.

6.3  Reliability

6.3.1  Errored OpenFlow connections handling

6.3.1.1  Objective

   To characterize the behavior of the controller when presented with
   a combination of both legal and Illegal OpenFlow messages.








Bhuvan                Expires September 20, 2014               [Page 21]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.3.1.2  Setup Parameters

   The following parameters MUST be defined:

   Network setup parameters:

      Number of Switches - Defines the number of OpenFlow switch
      connections established with controller.

      Number of Flows - Defines unique number of flows setup on each
      OpenFlow connections.

   OpenFlow setup parameters:

      OpenFlow Version - Defines OpenFlow protocol version used for
      OpenFlow connection setup.

      Channel Type - Defines OpenFlow connection setup type used for
      OpenFlow connection setup, for example TCP or TLS.

   Test setup parameters:

      Test Iterations - Defines the number of times the test need to be
      repeated.

      Test Duration - Defines the duration of test iteration, expressed
      in seconds.

6.3.1.3  Procedure

   Establish OpenFlow connection from all the specified number of
   switches. Load controller with Packet-In messages send from all
   switches for a configured number of flows and for a specified test
   duration. On each connection, after every 5% of flows transmission,
   send an invalid Packet-In message (message with wrong header field
   value). Single Packet-In message can have one or more flow headers
   in its payload.

   The test SHOULD be repeated a number of times by varying the
   offered network load (number of switches) and record the results.
   On the completion of the test, gracefully close all the OpenFlow
   sessions established with controller.

6.3.1.4 Measurement

   Average response rate:

   Total number of OpenFlow responses received divided by the test
   duration, expressed in responses per second.


Bhuvan                Expires September 20, 2014               [Page 22]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.3.1.5 Reporting Format

   The test report MUST note the number of switches, average offered
   load (correct and incorrect messages) and the average responses
   received. The results SHOULD be reported in the format of a table
   with a row for the number of switches and columns for the average
   offered load and average responses received.

6.3.2  Denial of service handling

6.3.2.1 Objective

   To determine the effect of a denial of service attack on a
   controller OpenFlow connection establishment rates. The denial of
   service handling test MUST be run after obtaining baseline
   measurements from section 6.3.

6.3.2.2  Setup Parameters

   Use the same setup parameters defined in section 6.2.1.2

   In addition, the following setup parameters MUST be defined:

   OpenFlow connection attack rate - Rate at which the TCP connections
   are established without sending any OpenFlow messages.

6.3.2.3  Procedure

   Use the same procedure as defined in section 6.2.1.3. In addition,
   establish TCP connections at the specified OpenFlow attack rate
   without performing any Hello exchanges.

6.3.2.4  Measurement

   Perform the same measurements as defined in section 6.2.1.4. In
   addition, track the number of TCP connections established without
   performing any Hello exchanges.

6.3.2.5  Reporting Format

   The test SHOULD use the same format as defined in section 6.2.1.5. In
   addition, the test SHOULD report the number of OpenFlow connections
   attempted without performing any Hello exchanges at the bottom of
   the report.







Bhuvan                Expires September 20, 2014               [Page 23]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.3.3  Controller failover time

6.3.3.1  Objective

   To determine the time taken to switch from one controller to
   another when the controllers are teamed and the master controller
   fails.

6.3.3.2  Setup Parameters

   Use the same setup parameters defined in section 6.1.3.2.

6.3.3.3  Procedure

   Establish OpenFlow connection with MASTER and SLAVE controllers.
   Duplicate Packet-In message and send on both connections one at a
   time. Insert timestamp to each Packet-In message sent on both
   connections. During the test, bring down the master controller.

   The test SHOULD be continued until first flow response is received
   from the new master. The test SHOULD be repeated a number of times
   by varying the number of switches.

6.3.3.4  Measurement

   Failover Time = (Time at which last response received from the old
   master) - (Time at which the first response received from the new
   master)

   Packet Losses = Total number of unique requests sent - Total number
   of unique responses received.

6.3.3.5  Reporting Format

   The test MUST note the number of flow requests sent on each
   connection and the number of responses received. The results SHOULD
   be reported in the format of a table with a row for the number of
   switches and columns for number of requests sent, responses
   received, packet losses and failover time.

6.3.4  Data path re-convergence time

6.3.4.1  Objective

   To determine the time taken to re-route a flow by the controller
   when there is a failure in the existing flow path.





Bhuvan                Expires September 20, 2014               [Page 24]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


6.3.4.2 Setup Parameters

   Use the same setup parameters defined in section 6.1.3.2

6.3.4.3  Procedure

   Inter connect the specified number of switches in tree fashion.
   Establish OpenFlow connection from all switches. To ensure that the
   controller completes the network discovery process, send traffic
   stream from a source connected on one end of the network to the
   destination connected on the other end of network and verify that
   the traffic reaches the destination properly. Prior sending the
   traffic, send gratuitous ARP to the controller for learning the
   destination.

   In addition, ensure that the topology has a redundant path from a
   source to destination. Send a traffic stream for different
   destination with sequence number inserted in each of the frame.
   During the course of the test, bring down the link of the traffic
   path.

   On completion of the test, gracefully close all the OpenFlow
   connection established with the controller. Repeat the test with
   varying number of inter-connected switches.

6.3.4.4  Measurement

   Failover time:

      Delta between the receive timestamp of the errored sequence
      packet (first packet received after re-convergence) and the last
      valid packet, expressed in milliseconds.

   Packet Loss:

      Difference between the sequence number of errored sequence packet
      and the last valid packet.

6.3.4.5  Reporting Format

   The test results MUST be reported in a table format with a row for
   number of switches and a column for failover time and packet
   losses.








Bhuvan                Expires September 20, 2014               [Page 25]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


7. References

7.1.  Normative References

   [1]  OpenFlow Switch Specification, Version 1.4.0 (Wire Protocol
   0x05), October 14, 2013

7.2.  Informative References

8.  IANA Considerations

   This document does not have any IANA requests.

9.  Security Considerations

   The primary goal of this document is to provide methodologies in
   benchmarking OpenFlow controller's performance with respect to
   southbound communication. There may arise some performance and
   security issues while interacting through northbound interfaces,
   assessment of such issues are outside the scope of this document.

10.  Appendix A - Test setup and test applicability

   +-------------------------------------------------------------------+
   |Benchmarking Tests     |Standalone Controller|Controller Teaming   |
   |                       |---------------------|---------------------|
   |                       |Proactive |Reactive  |Proactive |Reactive  |
   +-------------------------------------------------------------------+
   |1. Flow Setup Rate     |          |          |          |          |
   |   - Constant Loadi    |Applicable|Applicable|Applicable|Applicable|
   |   - Incremental Load  | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |2. Flow Setup Delay    |          |          |          |          |
   |   - Constant Load     | NA       |Applicable| NA       |Applicable|
   |   - Incremental Load  | NA       |Applicable| NA       |Applicable|
   |   - Stress Load       | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |
   |3. OpenFlow Connections|          |          |          |          |
   |   Capacity            | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |4. Switch Scalable     |          |          |          |          |
   |   Limit               | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |5. Flow Scalable Limit | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |6. Errored OpenFlow    |          |          |          |          |
   |   Connections Handling| NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |



Bhuvan                Expires September 20, 2014               [Page 26]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


   |7. Denial of Service   |          |          |          |          |
   |   Handling            | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |8. End-End Flow Setup  |          |          |          |          |
   |   Duration            | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |9. Controller Failover |          |          |          |          |
   |   Time                | NA       |Applicable| NA       |Applicable|
   |                       |          |          |          |          |
   |10. Datapath           |          |          |          |          |
   |    Re-convergence Time| NA       |Applicable| NA       |Applicable|
   +-------------------------------------------------------------------+

11. Appendix B - OpenFlow protocol overview

11.1. Protocol Overview

   OpenFlow is an open standard protocol defined by Open Networking
   Foundation (ONF), used for programming the forwarding plane of
   network switches or routers via a centralized controller.

11.2. Messages Overview

   OpenFlow protocol supports three messages types namely controller-
   to-switch, asynchronous and symmetric.

   Controller-to-switch messages are initiated by the controller and
   used to directly manage or inspect the state of the switch. These
   messages allow controllers to query/configure the switch (Features,
   Configuration messages), collect information from switch (Read-
   State message), send packets on specified port of switch (Packet-
   out message), and modify switch forwarding plane and state (Modify-
   State, Role-Request messages etc.).

   Asynchronous messages are generated by the switch without a
   controller soliciting them. These messages allow switches to update
   controllers to denote an arrival of new flow (Packet-in), switch
   state change (Flow-Removed, Port-status) and error (Error).

   Symmetric messages are generated in either direction without
   solicitation. These messages allow switches and controllers to set
   up connection (Hello), verify for liveness (Echo) and offer
   additional functionalities (Experimenter).








Bhuvan                Expires September 20, 2014               [Page 27]


Internet Draft     OF Controller Benchmarking Methodology     March 2014


11.3. Connection Overview

   OpenFlow channel is used to exchange OpenFlow message between an
   OpenFlow switch and an OpenFlow controller. The OpenFlow channel
   connection can be setup using plain TCP or TLS. By default, a
   switch establishes single connection with SDN controller. A switch
   may establish multiple parallel connections to single controller
   (auxiliary connection) or multiple controllers to handle controller
   failures and load balancing.

12. Authors' Addresses

   Bhuvaneswaran Vengainathan
   Veryx Technologies Inc.
   1 International Plaza, Suite 550
   Philadelphia
   PA 19113

   Email: bhuvaneswaran.vengainathan@veryxtech.com

   Anton Basil
   Veryx Technologies Inc.
   1 International Plaza, Suite 550
   Philadelphia
   PA 19113

   Email: anton.basil@veryxtech.com

   Vishwas Manral
   Ionos Corp,
   4100 Moorpark Ave,
   San Jose, CA

   Email: vishwas@ionosnetworks.com















Bhuvan                Expires September 20, 2014               [Page 28]


Html markup produced by rfcmarkup 1.122, available from https://tools.ietf.org/tools/rfcmarkup/