draft-ietf-bmwg-sdn-controller-benchmark-meth-09.txt   rfc8456.txt 
Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies
Expires: November 25, 2018 Mark Tassinari
Hewlett-Packard
Vishwas Manral
Nano Sec
Sarah Banks
VSS Monitoring
May 25, 2018
Benchmarking Methodology for SDN Controller Performance Internet Engineering Task Force (IETF) V. Bhuvaneswaran
draft-ietf-bmwg-sdn-controller-benchmark-meth-09 Request for Comments: 8456 A. Basil
Category: Informational Veryx Technologies
ISSN: 2070-1721 M. Tassinari
Hewlett Packard Enterprise
V. Manral
NanoSec
S. Banks
VSS Monitoring
October 2018
Abstract Benchmarking Methodology for Software-Defined Networking (SDN)
Controller Performance
This document defines methodologies for benchmarking control plane Abstract
performance of SDN controllers. SDN controller is a core component
in software-defined networking architecture that controls the
network behavior. SDN controllers have been implemented with many
varying designs in order to achieve their intended network
functionality. Hence, the authors have taken the approach of
considering an SDN controller as a black box, defining the
methodology in a manner that is agnostic to protocols and network
services supported by controllers. The intent of this document is to
provide a method to measure the performance of all controller
implementations.
Status of this Memo This document defines methodologies for benchmarking the control-
plane performance of Software-Defined Networking (SDN) Controllers.
The SDN Controller is a core component in the SDN architecture that
controls the behavior of the network. SDN Controllers have been
implemented with many varying designs in order to achieve their
intended network functionality. Hence, the authors of this document
have taken the approach of considering an SDN Controller to be a
black box, defining the methodology in a manner that is agnostic to
protocols and network services supported by controllers. This
document provides a method for measuring the performance of all
controller implementations.
This Internet-Draft is submitted in full conformance with the Status of This Memo
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This document is not an Internet Standards Track specification; it is
Task Force (IETF). Note that other groups may also distribute published for informational purposes.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six This document is a product of the Internet Engineering Task Force
months and may be updated, replaced, or obsoleted by other documents (IETF). It represents the consensus of the IETF community. It has
at any time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress. Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on November 25, 2018. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8456.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with respect
respect to this document. Code Components extracted from this to this document. Code Components extracted from this document must
document must include Simplified BSD License text as described in include Simplified BSD License text as described in Section 4.e of
Section 4.e of the Trust Legal Provisions and are provided without the Trust Legal Provisions and are provided without warranty as
warranty as described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction...................................................4 1. Introduction ....................................................4
2. Scope..........................................................4 1.1. Conventions Used in This Document ..........................4
3. Test Setup.....................................................4 2. Scope ...........................................................4
3.1. Test setup - Controller working in Standalone Mode........5 3. Test Setup ......................................................4
3.2. Test setup - Controller working in Cluster Mode...........6 3.1. Test Setup - Controller Operating in Standalone Mode .......5
4. Test Considerations............................................7 3.2. Test Setup - Controller Operating in Cluster Mode ..........6
4.1. Network Topology..........................................7 4. Test Considerations .............................................7
4.2. Test Traffic..............................................7 4.1. Network Topology ...........................................7
4.3. Test Emulator Requirements................................7 4.2. Test Traffic ...............................................7
4.4. Connection Setup..........................................7 4.3. Test Emulator Requirements .................................7
4.5. Measurement Point Specification and Recommendation........8 4.4. Connection Setup ...........................................8
4.6. Connectivity Recommendation...............................8 4.5. Measurement Point Specification and Recommendation .........9
4.7. Test Repeatability........................................8 4.6. Connectivity Recommendation ................................9
4.8. Test Reporting............................................8 4.7. Test Repeatability .........................................9
5. Benchmarking Tests.............................................9 4.8. Test Reporting .............................................9
5.1. Performance...............................................9 5. Benchmarking Tests .............................................11
5.1.1. Network Topology Discovery Time......................9 5.1. Performance ...............................................11
5.1.2. Asynchronous Message Processing Time................11 5.1.1. Network Topology Discovery Time ....................11
5.1.3. Asynchronous Message Processing Rate................12 5.1.2. Asynchronous Message Processing Time ...............13
5.1.4. Reactive Path Provisioning Time.....................15 5.1.3. Asynchronous Message Processing Rate ...............14
5.1.5. Proactive Path Provisioning Time....................16 5.1.4. Reactive Path Provisioning Time ....................17
5.1.6. Reactive Path Provisioning Rate.....................18 5.1.5. Proactive Path Provisioning Time ...................19
5.1.7. Proactive Path Provisioning Rate....................19 5.1.6. Reactive Path Provisioning Rate ....................21
5.1.8. Network Topology Change Detection Time..............21 5.1.7. Proactive Path Provisioning Rate ...................23
5.2. Scalability..............................................22 5.1.8. Network Topology Change Detection Time .............25
5.2.1. Control Session Capacity............................22 5.2. Scalability ...............................................26
5.2.2. Network Discovery Size..............................23 5.2.1. Control Sessions Capacity ..........................26
5.2.3. Forwarding Table Capacity...........................24 5.2.2. Network Discovery Size .............................27
5.3. Security.................................................26 5.2.3. Forwarding Table Capacity ..........................29
5.3.1. Exception Handling..................................26
5.3.2. Denial of Service Handling..........................27
5.4. Reliability..............................................29
5.4.1. Controller Failover Time............................29
5.4.2. Network Re-Provisioning Time........................30
6. References....................................................32
6.1. Normative References.....................................32
6.2. Informative References...................................32
7. IANA Considerations...........................................32
8. Security Considerations.......................................32
9. Acknowledgments...............................................33
Appendix A Benchmarking Methodology using OpenFlow Controllers..34
A.1. Protocol Overview........................................34
A.2. Messages Overview........................................34
A.3. Connection Overview......................................34
A.4. Performance Benchmarking Tests...........................35
A.4.1. Network Topology Discovery Time.....................35
A.4.2. Asynchronous Message Processing Time................36
A.4.3. Asynchronous Message Processing Rate................37
A.4.4. Reactive Path Provisioning Time.....................38
A.4.5. Proactive Path Provisioning Time....................39
A.4.6. Reactive Path Provisioning Rate.....................40
A.4.7. Proactive Path Provisioning Rate....................41
A.4.8. Network Topology Change Detection Time..............42
A.5. Scalability..............................................43
A.5.1. Control Sessions Capacity...........................43
A.5.2. Network Discovery Size..............................43
A.5.3. Forwarding Table Capacity...........................44
A.6. Security.................................................46
A.6.1. Exception Handling..................................46
A.6.2. Denial of Service Handling..........................47
A.7. Reliability..............................................49
A.7.1. Controller Failover Time............................49
A.7.2. Network Re-Provisioning Time........................50
Authors' Addresses...............................................53
1. Introduction 5.3. Security ..................................................31
5.3.1. Exception Handling .................................31
5.3.2. Handling Denial-of-Service Attacks .................32
5.4. Reliability ...............................................34
5.4.1. Controller Failover Time ...........................34
5.4.2. Network Re-provisioning Time .......................36
6. IANA Considerations ............................................37
7. Security Considerations ........................................38
8. References .....................................................38
8.1. Normative References ......................................38
8.2. Informative References ....................................38
Appendix A. Benchmarking Methodology Using OpenFlow Controllers ...39
A.1. Protocol Overview ..........................................39
A.2. Messages Overview ..........................................39
A.3. Connection Overview ........................................39
A.4. Performance Benchmarking Tests .............................40
A.4.1. Network Topology Discovery Time ........................40
A.4.2. Asynchronous Message Processing Time ...................42
A.4.3. Asynchronous Message Processing Rate ...................43
A.4.4. Reactive Path Provisioning Time ........................44
A.4.5. Proactive Path Provisioning Time .......................46
A.4.6. Reactive Path Provisioning Rate ........................47
A.4.7. Proactive Path Provisioning Rate .......................49
A.4.8. Network Topology Change Detection Time .................50
A.5. Scalability ................................................51
A.5.1. Control Sessions Capacity ..............................51
A.5.2. Network Discovery Size .................................52
A.5.3. Forwarding Table Capacity ..............................54
A.6. Security ...................................................55
A.6.1. Exception Handling .....................................55
A.6.2. Handling Denial-of-Service Attacks .....................57
A.7. Reliability ................................................59
A.7.1. Controller Failover Time ...............................59
A.7.2. Network Re-provisioning Time ...........................61
Acknowledgments ...................................................63
Authors' Addresses ................................................64
This document provides generic methodologies for benchmarking SDN 1. Introduction
controller performance. An SDN controller may support many
This document provides generic methodologies for benchmarking
Software-Defined Networking (SDN) Controller performance. To achieve
the desired functionality, an SDN Controller may support many
northbound and southbound protocols, implement a wide range of northbound and southbound protocols, implement a wide range of
applications, and work solely, or as a group to achieve the desired applications, and work either alone or as part of a group. This
functionality. This document considers an SDN controller as a black document considers an SDN Controller to be a black box, regardless of
box, regardless of design and implementation. The tests defined in design and implementation. The tests defined in this document can be
the document can be used to benchmark SDN controller for used to benchmark an SDN Controller for performance, scalability,
performance, scalability, reliability and security independent of reliability, and security, independently of northbound and southbound
northbound and southbound protocols. Terminology related to protocols. Terminology related to benchmarking SDN Controllers is
benchmarking SDN controllers is described in the companion described in the companion terminology document [RFC8455]. These
terminology document [I-D.sdn-controller-benchmark-term]. These tests can be performed on an SDN Controller running as a virtual
tests can be performed on an SDN controller running as a virtual machine (VM) instance or on a bare metal server. This document is
machine (VM) instance or on a bare metal server. This document is intended for those who want to measure an SDN Controller's
intended for those who want to measure the SDN controller performance as well as compare the performance of various SDN
performance as well as compare various SDN controllers performance. Controllers.
Conventions used in this document 1.1. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in "OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
2. Scope 2. Scope
This document defines methodology to measure the networking metrics This document defines a methodology for measuring the networking
of SDN controllers. For the purpose of this memo, the SDN controller metrics of SDN Controllers. For the purpose of this memo, the SDN
is a function that manages and controls Network Devices. Any SDN Controller is a function that manages and controls Network Devices.
controller without a control capability is out of scope for this Any SDN Controller without a control capability is out of scope for
memo. The tests defined in this document enable benchmarking of SDN this memo. The tests defined in this document enable the
Controllers in two ways; as a standalone controller and as a cluster benchmarking of SDN Controllers in two ways: standalone mode
of homogeneous controllers. These tests are recommended for (a standalone controller) and cluster mode (a cluster of homogeneous
execution in lab environments rather than in live network controllers). These tests are recommended for execution in lab
deployments. Performance benchmarking of a federation of environments rather than in live network deployments. Performance
controllers, set of SDN controllers managing different domains, is benchmarking of a federation of controllers (i.e., a set of SDN
beyond the scope of this document. Controllers) managing different domains, is beyond the scope of this
document.
3. Test Setup 3. Test Setup
The tests defined in this document enable measurement of an SDN As noted above, the tests defined in this document enable the
controller's performance in standalone mode and cluster mode. This measurement of an SDN Controller's performance in standalone mode and
section defines common reference topologies that are later referred cluster mode. This section defines common reference topologies that
to in individual tests. are referred to in individual tests described later in this document.
3.1. Test setup - Controller working in Standalone Mode 3.1. Test Setup - Controller Operating in Standalone Mode
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| Application Plane Test Emulator | | Application-Plane Test Emulator |
| | | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | Application | | Service | | | | Application | | Service | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | | |
+-----------------------------+(I2)-------------------------+ +-----------------------------+(I2)-------------------------+
| |
| (Northbound interfaces) | (Northbound Interface)
+-------------------------------+ +-------------------------------+
| +----------------+ | | +----------------+ |
| | SDN Controller | | | | SDN Controller | |
| +----------------+ | | +----------------+ |
| | | |
| Device Under Test (DUT) | | Device Under Test (DUT) |
+-------------------------------+ +-------------------------------+
| (Southbound interfaces) | (Southbound Interface)
| |
+-----------------------------+(I1)-------------------------+ +-----------------------------+(I1)-------------------------+
| | | |
| +-----------+ +-----------+ | | +-----------+ +-------------+ |
| | Network | | Network | | | | Network | | Network | |
| | Device 2 |--..-| Device n-1| | | | Device 2 |--..-| Device n - 1| |
| +-----------+ +-----------+ | | +-----------+ +-------------+ |
| / \ / \ | | / \ / \ |
| / \ / \ | | / \ / \ |
| l0 / X \ ln | | l0 / X \ ln |
| / / \ \ | | / / \ \ |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| | Network | | Network | | | | Network | | Network | |
| | Device 1 |..| Device n | | | | Device 1 |..| Device n | |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| | | | | | | |
| +---------------+ +---------------+ | | +---------------+ +---------------+ |
| | Test Traffic | | Test Traffic | | | | Test Traffic | | Test Traffic | |
| | Generator | | Generator | | | | Generator | | Generator | |
| | (TP1) | | (TP2) | | | | (TP1) | | (TP2) | |
| +---------------+ +---------------+ | | +---------------+ +---------------+ |
| | | |
| Forwarding Plane Test Emulator | | Forwarding-Plane Test Emulator |
+-----------------------------------------------------------+ +-----------------------------------------------------------+
Figure 1 Figure 1
3.2. Test setup - Controller working in Cluster Mode 3.2. Test Setup - Controller Operating in Cluster Mode
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| Application Plane Test Emulator | | Application-Plane Test Emulator |
| | | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | Application | | Service | | | | Application | | Service | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | | |
+-----------------------------+(I2)-------------------------+ +-----------------------------+(I2)-------------------------+
| |
| (Northbound interfaces) | (Northbound Interface)
+---------------------------------------------------------+ +---------------------------------------------------------+
| | | |
| ------------------ ------------------ | | +------------------+ +------------------+ |
| | SDN Controller 1 | <--E/W--> | SDN Controller n | | | | SDN Controller 1 | <--E/W--> | SDN Controller n | |
| ------------------ ------------------ | | +------------------+ +------------------+ |
| | | |
| Device Under Test (DUT) | | Device Under Test (DUT) |
+---------------------------------------------------------+ +---------------------------------------------------------+
| (Southbound interfaces) | (Southbound Interface)
| |
+-----------------------------+(I1)-------------------------+ +-----------------------------+(I1)-------------------------+
| | | |
| +-----------+ +-----------+ | | +-----------+ +-------------+ |
| | Network | | Network | | | | Network | | Network | |
| | Device 2 |--..-| Device n-1| | | | Device 2 |--..-| Device n - 1| |
| +-----------+ +-----------+ | | +-----------+ +-------------+ |
| / \ / \ | | / \ / \ |
| / \ / \ | | / \ / \ |
| l0 / X \ ln | | l0 / X \ ln |
| / / \ \ | | / / \ \ |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| | Network | | Network | | | | Network | | Network | |
| | Device 1 |..| Device n | | | | Device 1 |..| Device n | |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| | | | | | | |
| +---------------+ +---------------+ | | +---------------+ +---------------+ |
| | Test Traffic | | Test Traffic | | | | Test Traffic | | Test Traffic | |
| | Generator | | Generator | | | | Generator | | Generator | |
| | (TP1) | | (TP2) | | | | (TP1) | | (TP2) | |
| +---------------+ +---------------+ | | +---------------+ +---------------+ |
| | | |
| Forwarding Plane Test Emulator | | Forwarding-Plane Test Emulator |
+-----------------------------------------------------------+ +-----------------------------------------------------------+
Figure 2 Figure 2
4. Test Considerations 4. Test Considerations
4.1. Network Topology 4.1. Network Topology
The test cases SHOULD use Leaf-Spine topology with at least 2 The test cases SHOULD use Leaf-Spine topology with at least two
Network Devices in the topology for benchmarking. The test traffic Network Devices in the topology for benchmarking. Test traffic
generators TP1 and TP2 SHOULD be connected to the leaf Network generators TP1 and TP2 SHOULD be connected to the leaf Network
Device 1 and the leaf Network Device n. To achieve a complete Device 1 and the leaf Network Device n. To achieve a complete
performance characterization of the SDN controller, it is performance characterization of the SDN Controller, it is recommended
recommended that the controller be benchmarked for many network that the controller be benchmarked for many network topologies and a
topologies and a varying number of Network Devices. Further, care varying number of Network Devices. Further, care should be taken to
should be taken to make sure that a loop prevention mechanism is make sure that a loop-prevention mechanism is enabled in either the
enabled either in the SDN controller, or in the network when the SDN Controller or the network when the topology contains redundant
topology contains redundant network paths. network paths.
4.2. Test Traffic 4.2. Test Traffic
Test traffic is used to notify the controller about the asynchronous Test traffic is used to notify the controller about the asynchronous
arrival of new flows. The test cases SHOULD use frame sizes of 128, arrival of new flows. The test cases SHOULD use frame sizes of 128,
512 and 1508 bytes for benchmarking. Tests using jumbo frames are 512, and 1508 bytes for benchmarking. Tests using jumbo frames are
optional. optional.
4.3. Test Emulator Requirements 4.3. Test Emulator Requirements
The Test Emulator SHOULD time stamp the transmitted and received The test emulator SHOULD timestamp the transmitted and received
control messages to/from the controller on the established network control messages to/from the controller on the established network
connections. The test cases use these values to compute the connections. The test cases use these values to compute the
controller processing time. controller processing time.
4.4. Connection Setup 4.4. Connection Setup
There may be controller implementations that support unencrypted and There may be controller implementations that support unencrypted and
encrypted network connections with Network Devices. Further, the encrypted network connections with Network Devices. Further, the
controller may have backward compatibility with Network Devices controller may be backward compatible with Network Devices running
running older versions of southbound protocols. It may be useful to older versions of southbound protocols. It may be useful to measure
measure the controller performance with one or more applicable the controller's performance with one or more applicable connection
connection setup methods defined below. For cases with encrypted setup methods defined below. For cases with encrypted communications
communications between the controller and the switch, key management between the controller and the switch, key management and key
and key exchange MUST take place before any performance or benchmark exchange MUST take place before any performance or benchmark
measurements. measurements.
1. Unencrypted connection with Network Devices, running same 1. Unencrypted connection with Network Devices, running the same
protocol version. protocol version.
2. Unencrypted connection with Network Devices, running different
protocol versions.
Example:
a. Controller running current protocol version and switch 2. Unencrypted connection with Network Devices, running different
running older protocol version protocol versions.
b. Controller running older protocol version and switch
running current protocol version
3. Encrypted connection with Network Devices, running same
protocol version
4. Encrypted connection with Network Devices, running different
protocol versions.
Example:
a. Controller running current protocol version and switch
running older protocol version
b. Controller running older protocol version and switch
running current protocol version
4.5. Measurement Point Specification and Recommendation Examples:
The measurement accuracy depends on several factors including the a. Controller running current protocol version and switch
point of observation where the indications are captured. For running older protocol version.
example, the notification can be observed at the controller or test
emulator. The test operator SHOULD make the observations/
measurements at the interfaces of test emulator unless it is
explicitly mentioned otherwise in the individual test. In any case,
the locations of measurement points MUST be reported.
4.6. Connectivity Recommendation b. Controller running older protocol version and switch
running current protocol version.
The SDN controller in the test setup SHOULD be connected directly 3. Encrypted connection with Network Devices, running the same
with the forwarding and the management plane test emulators to avoid protocol version.
any delays or failure introduced by the intermediate devices during
benchmarking tests. When the controller is implemented as a virtual
machine, details of the physical and logical connectivity MUST be
reported.
4.7. Test Repeatability 4. Encrypted connection with Network Devices, running different
protocol versions.
To increase the confidence in measured result, it is recommended Examples:
that each test RECOMMENDED be repeated a minimum of 10 times.
4.8. Test Reporting a. Controller running current protocol version and switch
running older protocol version.
b. Controller running older protocol version and switch
running current protocol version.
4.5. Measurement Point Specification and Recommendation
The accuracy of the measurements depends on several factors,
including the point of observation where the indications are
captured. For example, the notification can be observed at the
controller or test emulator. The test operator SHOULD make the
observations/measurements at the interfaces of the test emulator,
unless explicitly specified otherwise in the individual test. In any
case, the locations of measurement points MUST be reported.
4.6. Connectivity Recommendation
The SDN Controller in the test setup SHOULD be connected directly
with the forwarding-plane and management-plane test emulators to
avoid any delays or failure introduced by the intermediate devices
during benchmarking tests. When the controller is implemented as a
virtual machine, details of the physical and logical connectivity
MUST be reported.
4.7. Test Repeatability
To increase confidence in the measured results, it is recommended
that each test SHOULD be repeated a minimum of 10 times.
4.8. Test Reporting
Each test has a reporting format that contains some global and Each test has a reporting format that contains some global and
identical reporting components, and some individual components that identical reporting components, and some individual components that
are specific to individual tests. The following test configuration are specific to individual tests. The following parameters for test
parameters and controller settings parameters MUST be reflected in configuration and controller settings MUST be reflected in the test
the test report. report.
Test Configuration Parameters: Test Configuration Parameters:
1. Controller name and version 1. Controller name and version
2. Northbound protocols and versions
3. Southbound protocols and versions
4. Controller redundancy mode (Standalone or Cluster Mode)
5. Connection setup (Unencrypted or Encrypted)
6. Network Device Type (Physical or Virtual or Emulated)
7. Number of Nodes
8. Number of Links
9. Dataplane Test Traffic Type
10. Controller System Configuration (e.g., Physical or Virtual
Machine, CPU, Memory, Caches, Operating System, Interface
Speed, Storage)
11. Reference Test Setup (e.g., Section 3.1 etc.,)
Controller Settings Parameters: 2. Northbound protocols and versions
1. Topology re-discovery timeout
2. Controller redundancy mode (e.g., active-standby etc.,)
3. Controller state persistence enabled/disabled
To ensure the repeatability of test, the following capabilities of 3. Southbound protocols and versions
test emulator SHOULD be reported
1. Maximum number of Network Devices that the forwarding plane 4. Controller redundancy mode (standalone or cluster mode)
emulates
2. Control message processing time (e.g., Topology Discovery
Messages)
One way to determine the above two values are to simulate the 5. Connection setup (unencrypted or encrypted)
required control sessions and messages from the control plane.
5. Benchmarking Tests 6. Network Device type (physical, virtual, or emulated)
5.1. Performance 7. Number of nodes
8. Number of links
5.1.1. Network Topology Discovery Time 9. Data-plane test traffic type
Objective: 10. Controller system configuration (e.g., physical or virtual
machine, CPU, memory, caches, operating system, interface
speed, storage)
The time taken by controller(s) to determine the complete network 11. Reference test setup (e.g., the setup shown in Section 3.1)
topology, defined as the interval starting with the first discovery
message from the controller(s) at its Southbound interface, ending
with all features of the static topology determined.
Reference Test Setup: Parameters for Controller Settings:
The test SHOULD use one of the test setups described in section 3.1 1. Topology rediscovery timeout
or section 3.2 of this document.
Prerequisite: 2. Controller redundancy mode (e.g., active-standby)
1. The controller MUST support network discovery. 3. Controller state persistence enabled/disabled
2. Tester should be able to retrieve the discovered topology
information either through the controller's management interface,
or northbound interface to determine if the discovery was
successful and complete.
3. Ensure that the controller's topology re-discovery timeout has
been set to the maximum value to avoid initiation of re-discovery
process in the middle of the test.
Procedure: To ensure the repeatability of the test, the following capabilities
of the test emulator SHOULD be reported:
1. Ensure that the controller is operational, its network 1. Maximum number of Network Devices that the forwarding plane
applications, northbound and southbound interfaces are up and emulates
running.
2. Establish the network connections between controller and Network
Devices.
3. Record the time for the first discovery message (Tm1) received
from the controller at forwarding plane test emulator interface
I1.
4. Query the controller every t seconds (RECOMMENDED value for t is
3) to obtain the discovered network topology information through
the northbound interface or the management interface and compare
it with the deployed network topology information.
5. Stop the trial when the discovered topology information matches
the deployed network topology, or when the discovered topology
information return the same details for 3 consecutive queries.
6. Record the time last discovery message (Tmn) sent to controller
from the forwarding plane test emulator interface (I1) when the
trial completed successfully. (e.g., the topology matches).
Measurement: 2. Control message processing time (e.g., topology discovery
messages)
Topology Discovery Time Tr1 = Tmn-Tm1. One way to determine the above two values is to simulate the required
control sessions and messages from the control plane.
Tr1 + Tr2 + Tr3 .. Trn 5. Benchmarking Tests
Average Topology Discovery Time (TDm) = -----------------------
Total Trials
SUM[SQUAREOF(Tri-TDm)]
Topology Discovery Time Variance (TDv) ----------------------
Total Trials -1
Reporting Format: 5.1. Performance
The Topology Discovery Time results MUST be reported in the format 5.1.1. Network Topology Discovery Time
of a table, with a row for each successful iteration. The last row
of the table indicates the Topology Discovery Time variance and the
previous row indicates the average Topology Discovery Time.
If this test is repeated with varying number of nodes over the same Objective:
topology, the results SHOULD be reported in the form of a graph. The
X coordinate SHOULD be the Number of nodes (N), the Y coordinate
SHOULD be the average Topology Discovery Time.
5.1.2. Asynchronous Message Processing Time Measure the time taken by the controller(s) to determine the
complete network topology, defined as the interval starting with
the first discovery message from the controller(s) at its
southbound interface and ending with all features of the static
topology determined.
Objective: Reference Test Setup:
The time taken by controller(s) to process an asynchronous message, This test SHOULD use one of the test setups illustrated in
defined as the interval starting with an asynchronous message from a Section 3.1 or Section 3.2 of this document.
network device after the discovery of all the devices by the
controller(s), ending with a response message from the controller(s)
at its Southbound interface.
Reference Test Setup: Prerequisites:
This test SHOULD use one of the test setup described in section 3.1 1. The controller MUST support network discovery.
or section 3.2 of this document.
Prerequisite: 2. The tester should be able to retrieve the discovered topology
information through either the controller's management
interface or northbound interface to determine if the discovery
was successful and complete.
1. The controller MUST have successfully completed the network 3. Ensure that the controller's topology rediscovery timeout has
topology discovery for the connected Network Devices. been set to the maximum value, to avoid initiation of the
rediscovery process in the middle of the test.
Procedure: Procedure:
1. Generate asynchronous messages from every connected Network 1. Ensure that the controller is operational and that its network
Device, to the SDN controller, one at a time in series from the applications, northbound interface, and southbound interface
forwarding plane test emulator for the trial duration. are up and running.
2. Record every request transmit time (T1) and the corresponding
response received time (R1) at the forwarding plane test emulator
interface (I1) for every successful message exchange.
Measurement: 2. Establish the network connections between the controller and
the Network Devices.
SUM{Ri} - SUM{Ti} 3. Record the time for the first discovery message (Tm1) received
Asynchronous Message Processing Time Tr1 = ----------------------- from the controller at the forwarding-plane test emulator
Nrx interface (I1).
Where Nrx is the total number of successful messages exchanged 4. Query the controller every t seconds (the RECOMMENDED value for
t is 3) to obtain the discovered network topology information
through the northbound interface or the management interface,
and compare it with the deployed network topology information.
Tr1 + Tr2 + Tr3..Trn 5. Stop the trial when the discovered topology information matches
Average Asynchronous Message Processing Time = -------------------- the deployed network topology or when the discovered topology
information returns the same details for three consecutive
queries.
6. Record the time for the last discovery message (Tmn) sent to
the controller from the forwarding-plane test emulator
interface (I1) when the trial completes successfully (e.g.,
when the topology matches).
Measurements:
Topology Discovery Time (DT1) = Tmn - Tm1
DT1 + DT2 + DT3 .. DTn
Average Topology Discovery Time (TDm) = -----------------------
Total Trials
SUM[SQUAREOF(DTi - TDm)]
Topology Discovery Time Variance (TDv) = ------------------------
Total Trials - 1
Reporting Format:
The Topology Discovery Time results MUST be reported in tabular
format, with a row for each successful iteration. The last row of
the table indicates the Topology Discovery Time variance, and the
previous row indicates the Average Topology Discovery Time.
If this test is repeated with a varying number of nodes over the
same topology, the results SHOULD be reported in the form of a
graph. The X coordinate SHOULD be the number of nodes (N), and
the Y coordinate SHOULD be the Average Topology Discovery Time.
5.1.2. Asynchronous Message Processing Time
Objective:
Measure the time taken by the controller(s) to process an
asynchronous message, defined as the interval starting with an
asynchronous message from a Network Device after the discovery of
all the devices by the controller(s) and ending with a response
message from the controller(s) at its southbound interface.
Reference Test Setup:
This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
Prerequisite:
The controller MUST have successfully completed the network
topology discovery for the connected Network Devices.
Procedure:
1. Generate asynchronous messages from every connected Network
Device to the SDN Controller, one at a time in series from the
forwarding-plane test emulator for the Trial Duration.
2. Record every request transmit time (T1) and the corresponding
response received time (R1) at the forwarding-plane test
emulator interface (I1) for every successful message exchange.
Measurements:
Asynchronous Message Processing Time (APT1) =
SUM{Ri} - SUM{Ti}
-----------------------
Nrx
Where Nrx is the total number of successful messages exchanged.
Average Asynchronous Message Processing Time =
APT1 + APT2 + APT3 .. APTn
--------------------------
Total Trials Total Trials
Asynchronous Message Processing Time Variance (TAMv) = Asynchronous Message Processing Time Variance (TAMv) =
SUM[SQUAREOF(APTi - TAMm)]
--------------------------
Total Trials - 1
SUM[SQUAREOF(Tri-TAMm)] Where TAMm is the Average Asynchronous Message Processing Time.
----------------------
Total Trials -1
Where TAMm is the Average Asynchronous Message Processing Time. Reporting Format:
Reporting Format: The Asynchronous Message Processing Time results MUST be reported
in tabular format, with a row for each iteration. The last row of
the table indicates the Asynchronous Message Processing Time
variance, and the previous row indicates the Average Asynchronous
Message Processing Time.
The Asynchronous Message Processing Time results MUST be reported in The report SHOULD capture the following information, in addition
the format of a table with a row for each iteration. The last row of to the configuration parameters captured per Section 4.8:
the table indicates the Asynchronous Message Processing Time
variance and the previous row indicates the average Asynchronous
Message Processing Time.
The report SHOULD capture the following information in addition to - Successful messages exchanged (Nrx)
the configuration parameters captured in section 4.8.
- Successful messages exchanged (Nrx) - Percentage of unsuccessful messages exchanged, computed
using the formula ((1 - Nrx/Ntx) * 100), where Ntx is the
total number of messages transmitted to the controller
- Percentage of unsuccessful messages exchanged, computed using the If this test is repeated with a varying number of nodes with the
formula (1 - Nrx/Ntx) * 100), Where Ntx is the total number of same topology, the results SHOULD be reported in the form of a
messages transmitted to the controller. graph. The X coordinate SHOULD be the number of nodes (N), and
the Y coordinate SHOULD be the Average Asynchronous Message
Processing Time.
If this test is repeated with varying number of nodes with same 5.1.3. Asynchronous Message Processing Rate
topology, the results SHOULD be reported in the form of a graph. The
X coordinate SHOULD be the Number of nodes (N), the Y coordinate
SHOULD be the average Asynchronous Message Processing Time.
5.1.3. Asynchronous Message Processing Rate Objective:
Objective: Measure the number of responses to asynchronous messages (a new
flow arrival notification message, link down, etc.) for which the
controller(s) performed processing and replied with a valid and
productive (non-trivial) response message.
Measure the number of responses to asynchronous messages (such as Using a single procedure, this test will measure the following two
new flow arrival notification message, link down, etc.) for which benchmarks on the Asynchronous Message Processing Rate (see
the controller(s) performed processing and replied with a valid and Section 2.3.1.3 of [RFC8455]):
productive (non-trivial) response message
This test will measure two benchmarks on Asynchronous Message 1. Maximum Asynchronous Message Processing Rate
Processing Rate using a single procedure. The two benchmarks are
(see section 2.3.1.3 of [I-D.sdn-controller-benchmark-term]):
1. Loss-free Asynchronous Message Processing Rate 2. Loss-Free Asynchronous Message Processing Rate
2. Maximum Asynchronous Message Processing Rate Here, two benchmarks are determined through a series of trials
where the number of messages sent to the controller(s) and the
responses received from the controller(s) are counted over the
Trial Duration. The message response rate and the Message Loss
Ratio are calculated for each trial.
Here two benchmarks are determined through a series of trials where Reference Test Setup:
the number of messages are sent to the controller(s), and the
responses from the controller(s) are counted over the trial
duration. The message response rate and the message loss ratio are
calculated for each trial.
Reference Test Setup: This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
The test SHOULD use one of the test setups described in section 3.1 Prerequisites:
or section 3.2 of this document.
Prerequisite: 1. The controller(s) MUST have successfully completed the network
topology discovery for the connected Network Devices.
1. The controller(s) MUST have successfully completed the network 2. Choose and record the Trial Duration (Td), the sending rate
topology discovery for the connected Network Devices. STEP size, the tolerance on equality for two consecutive trials
2. Choose and record the Trial Duration (Td), the sending rate step- (P%), and the maximum possible message-sending rate (Ntx1/Td).
size (STEP), the tolerance on equality for two consecutive trials
(P%),and the maximum possible message sending rate (Ntx1/Td).
Procedure: Procedure:
1. Generate asynchronous messages continuously at the maximum 1. Generate asynchronous messages continuously at the maximum
possible rate on the established connections from all the possible rate on the established connections from all the
emulated/simulated Network Devices for the given trial Duration emulated/simulated Network Devices for the given Trial
(Td). Duration (Td).
2. Record the total number of responses received from the controller
(Nrx1) as well as the number of messages sent (Ntx1) to the
controller within the trial duration (Td).
3. Calculate the Asynchronous Message Processing Rate (Tr1) and the
Message Loss Ratio (Lr1). Ensure that the controller(s) have
returned to normal operation.
4. Repeat the trial by reducing the asynchronous message sending rate
used in last trial by the STEP size.
5. Continue repeating the trials and reducing the sending rate until
both the maximum value of Nrxn (number of responses received from
the controller) and the Nrxn corresponding to zero loss ratio have
been found.
6. The trials corresponding to the benchmark levels MUST be repeated
using the same asynchronous message rates until the responses
received from the controller are equal (+/-P%) for two consecutive
trials.
7. Record the number of responses received from the controller (Nrxn)
as well as the number of messages sent (Ntxn) to the controller in
the last trial.
Measurement: 2. Record the total number of responses received (Nrx1) from the
controller as well as the number of messages sent (Ntx1) to the
controller within the Trial Duration (Td).
Nrxn 3. Calculate the Asynchronous Message Processing Rate (APR1) and
Asynchronous Message Processing Rate Trn = ----- the Message Loss Ratio (Lr1). Ensure that the controller(s)
Td has returned to normal operation.
Maximum Asynchronous Message Processing Rate = MAX(Trn) for all n 4. Repeat the trial by reducing the asynchronous message-sending
rate used in the last trial by the STEP size.
Nrxn 5. Continue repeating the trials and reducing the sending rate
Asynchronous Message Loss Ratio Lrn = 1 - ----- until both the maximum value of Nrxn (number of responses
Ntxn received from the controller) and the Nrxn corresponding to a
Loss Ratio of zero have been found.
Loss-free Asynchronous Message Processing Rate = MAX(Trn) given 6. The trials corresponding to the benchmark levels MUST be
Lrn=0 repeated using the same asynchronous message rates until the
responses received from the controller are equal (+/-P%) for
two consecutive trials.
Reporting Format: 7. Record the number of responses received (Nrxn) from the
controller as well as the number of messages sent (Ntxn) to the
controller in the last trial.
The Asynchronous Message Processing Rate results MUST be reported in Measurements:
the format of a table with a row for each trial.
The table should report the following information in addition to the Nrxn
configuration parameters captured in section 4.8, with columns: Asynchronous Message Processing Rate (APRn) = -----
Td
- Offered rate (Ntxn/Td) Maximum Asynchronous Message Processing Rate = MAX(APRn) for all n
- Asynchronous Message Processing Rate (Trn) Nrxn
Asynchronous Message Loss Ratio (Lrn) = 1 - -----
Ntxn
- Loss Ratio (Lr) Loss-Free Asynchronous Message Processing Rate = MAX(APRn)
given Lrn = 0
- Benchmark at this iteration (blank for none, Maximum, Loss-Free) Reporting Format:
The results MAY be presented in the form of a graph. The X axis The Asynchronous Message Processing Rate results MUST be reported
SHOULD be the Offered rate, and dual Y axes would represent in tabular format, with a row for each trial.
Asynchronous Message Processing Rate and Loss Ratio, respectively.
If this test is repeated with varying number of nodes over same The table should report the following information, in addition to
topology, the results SHOULD be reported in the form of a graph. The the configuration parameters captured per Section 4.8, with
X axis SHOULD be the Number of nodes (N), the Y axis SHOULD be the columns:
Asynchronous Message Processing Rate. Both the Maximum and the Loss-
Free Rates should be plotted for each N.
5.1.4. Reactive Path Provisioning Time - Offered rate (Ntxn/Td)
Objective: - Asynchronous Message Processing Rate (APRn)
The time taken by the controller to setup a path reactively between - Loss Ratio (Lr)
source and destination node, defined as the interval starting with
the first flow provisioning request message received by the
controller(s) at its Southbound interface, ending with the last flow
provisioning response message sent from the controller(s) at its
Southbound interface.
Reference Test Setup: - Benchmark at this iteration (blank for none, Maximum
Asynchronous Message Processing Rate, Loss-Free Asynchronous
Message Processing Rate)
The test SHOULD use one of the test setups described in section 3.1 The results MAY be presented in the form of a graph. The X axis
or section 3.2 of this document. The number of Network Devices in SHOULD be the offered rate, and dual Y axes would represent the
the path is a parameter of the test that may be varied from 2 to Asynchronous Message Processing Rate and the Loss Ratio,
maximum discovery size in repetitions of this test. respectively.
Prerequisite: If this test is repeated with a varying number of nodes over the
same topology, the results SHOULD be reported in the form of a
graph. The X axis SHOULD be the number of nodes (N), and the
Y axis SHOULD be the Asynchronous Message Processing Rate. Both
the Maximum Asynchronous Message Processing Rate and the Loss-Free
Asynchronous Message Processing Rate should be plotted for each N.
1. The controller MUST contain the network topology information for 5.1.4. Reactive Path Provisioning Time
the deployed network topology.
2. The controller should have the knowledge about the location of
destination endpoint for which the path has to be provisioned.
This can be achieved through dynamic learning or static
provisioning.
3. Ensure that the default action for 'flow miss' in Network Device
is configured to 'send to controller'.
4. Ensure that each Network Device in a path requires the controller
to make the forwarding decision while paving the entire path.
Procedure: Objective:
1. Send a single traffic stream from the test traffic generator TP1 Measure the time taken by the controller to set up a path
to test traffic generator TP2. reactively between source and destination nodes, defined as the
2. Record the time of the first flow provisioning request message interval starting with the first flow provisioning request message
sent to the controller (Tsf1) from the Network Device at the received by the controller(s) at its southbound interface and
forwarding plane test emulator interface (I1). ending with the last flow provisioning response message sent from
3. Wait for the arrival of first traffic frame at the Traffic the controller(s) at its southbound interface.
Endpoint TP2 or the expiry of trial duration (Td).
4. Record the time of the last flow provisioning response message
received from the controller (Tdf1) to the Network Device at the
forwarding plane test emulator interface (I1).
Measurement: Reference Test Setup:
Reactive Path Provisioning Time Tr1 = Tdf1-Tsf1. This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document. The number of
Network Devices in the path is a parameter of the test that may be
varied from two to the maximum discovery size in repetitions of
this test.
Tr1 + Tr2 + Tr3 .. Trn Prerequisites:
Average Reactive Path Provisioning Time = -----------------------
Total Trials
SUM[SQUAREOF(Tri-TRPm)] 1. The controller MUST contain the network topology information
Reactive Path Provisioning Time Variance(TRPv) --------------------- for the deployed network topology.
Total Trials -1
Where TRPm is the Average Reactive Path Provisioning Time. 2. The controller should know the location of the destination
endpoint for which the path has to be provisioned. This can be
achieved through dynamic learning or static provisioning.
Reporting Format: 3. Ensure that the default action for "flow miss" in the Network
Device is configured to "send to controller".
The Reactive Path Provisioning Time results MUST be reported in the 4. Ensure that each Network Device in a path requires the
format of a table with a row for each iteration. The last row of the controller to make the forwarding decision while paving the
table indicates the Reactive Path Provisioning Time variance and the entire path.
previous row indicates the Average Reactive Path Provisioning Time.
The report should capture the following information in addition to Procedure:
the configuration parameters captured in section 4.8.
- Number of Network Devices in the path 1. Send a single traffic stream from test traffic generator TP1 to
test traffic generator TP2.
5.1.5. Proactive Path Provisioning Time 2. Record the time of the first flow provisioning request message
sent to the controller (Tsf1) from the Network Device at the
forwarding-plane test emulator interface (I1).
Objective: 3. Wait for the arrival of the first traffic frame at the endpoint
(i.e., test traffic generator TP2) or the expiry of the Trial
Duration (Td).
The time taken by the controller to setup a path proactively between 4. Record the time of the last flow provisioning response message
source and destination node, defined as the interval starting with received from the controller (Tdf1) to the Network Device at
the first proactive flow provisioned in the controller(s) at its the forwarding-plane test emulator interface (I1).
Northbound interface, ending with the last flow provisioning
response message sent from the controller(s) at its Southbound
interface.
Reference Test Setup: Measurements:
The test SHOULD use one of the test setups described in section 3.1 Reactive Path Provisioning Time (RPT1) = Tdf1 - Tsf1
or section 3.2 of this document.
Prerequisite: Average Reactive Path Provisioning Time =
RPT1 + RPT2 + RPT3 .. RPTn
--------------------------
Total Trials
1. The controller MUST contain the network topology information for Reactive Path Provisioning Time Variance (TRPv) =
the deployed network topology. SUM[SQUAREOF(RPTi - TRPm)]
2. The controller should have the knowledge about the location of --------------------------
destination endpoint for which the path has to be provisioned. Total Trials - 1
This can be achieved through dynamic learning or static
provisioning.
3. Ensure that the default action for flow miss in Network Device is
'drop'.
Procedure: Where TRPm is the Average Reactive Path Provisioning Time.
1. Send a single traffic stream from test traffic generator TP1 to Reporting Format:
TP2.
2. Install the flow entries to reach from test traffic generator TP1
to the test traffic generator TP2 through controller's northbound
or management interface.
3. Wait for the arrival of first traffic frame at the test traffic
generator TP2 or the expiry of trial duration (Td).
4. Record the time when the proactive flow is provisioned in the
Controller (Tsf1) at the management plane test emulator interface
I2.
5. Record the time of the last flow provisioning message received
from the controller (Tdf1) at the forwarding plane test emulator
interface I1.
Measurement: The Reactive Path Provisioning Time results MUST be reported in
tabular format, with a row for each iteration. The last row of
the table indicates the Reactive Path Provisioning Time variance,
and the previous row indicates the Average Reactive Path
Provisioning Time.
Proactive Flow Provisioning Time Tr1 = Tdf1-Tsf1. The report should capture the following information, in addition
to the configuration parameters captured per Section 4.8:
Tr1 + Tr2 + Tr3 .. Trn - Number of Network Devices in the path
Average Proactive Path Provisioning Time = -----------------------
Total Trials
SUM[SQUAREOF(Tri-TPPm)] 5.1.5. Proactive Path Provisioning Time
Proactive Path Provisioning Time Variance(TPPv) --------------------
Total Trials -1
Where TPPm is the Average Proactive Path Provisioning Time. Objective:
Reporting Format: Measure the time taken by the controller to set up a path
proactively between source and destination nodes, defined as the
interval starting with the first proactive flow provisioned in the
controller(s) at its northbound interface and ending with the last
flow provisioning response message sent from the controller(s) at
its southbound interface.
The Proactive Path Provisioning Time results MUST be reported in the Reference Test Setup:
format of a table with a row for each iteration. The last row of the
table indicates the Proactive Path Provisioning Time variance and
the previous row indicates the Average Proactive Path Provisioning
Time.
The report should capture the following information in addition to This test SHOULD use one of the test setups illustrated in
the configuration parameters captured in section 4.8. Section 3.1 or Section 3.2 of this document.
- Number of Network Devices in the path Prerequisites:
5.1.6. Reactive Path Provisioning Rate 1. The controller MUST contain the network topology information
for the deployed network topology.
Objective: 2. The controller should know the location of the destination
endpoint for which the path has to be provisioned. This can be
achieved through dynamic learning or static provisioning.
The maximum number of independent paths a controller can 3. Ensure that the default action for "flow miss" in the Network
concurrently establish per second between source and destination Device is "drop".
nodes reactively, defined as the number of paths provisioned per
second by the controller(s) at its Southbound interface for the flow
provisioning requests received for path provisioning at its
Southbound interface between the start of the test and the expiry of
given trial duration.
Reference Test Setup: Procedure:
The test SHOULD use one of the test setups described in section 3.1 1. Send a single traffic stream from test traffic generator TP1 to
or section 3.2 of this document. test traffic generator TP2.
Prerequisite: 2. Install the flow entries so that the traffic travels from test
traffic generator TP1 until it reaches test traffic
generator TP2 through the controller's northbound interface or
management interface.
1. The controller MUST contain the network topology information for 3. Wait for the arrival of the first traffic frame at test traffic
the deployed network topology. generator TP2 or the expiry of the Trial Duration (Td).
2. The controller should have the knowledge about the location of
destination addresses for which the paths have to be provisioned.
This can be achieved through dynamic learning or static
provisioning.
3. Ensure that the default action for 'flow miss' in Network Device
is configured to 'send to controller'.
4. Ensure that each Network Device in a path requires the controller
to make the forwarding decision while provisioning the entire
path.
Procedure: 4. Record the time when the proactive flow is provisioned in the
controller (Tsf1) at the management-plane test emulator
interface (I2).
1. Send traffic with unique source and destination addresses from 5. Record the time of the last flow provisioning message received
test traffic generator TP1. from the controller (Tdf1) at the forwarding-plane test
2. Record total number of unique traffic frames (Ndf) received at the emulator interface (I1).
test traffic generator TP2 within the trial duration (Td).
Measurement: Measurements:
Ndf Proactive Flow Provisioning Time (PPT1) = Tdf1 - Tsf1
Reactive Path Provisioning Rate Tr1 = ------
Td
Tr1 + Tr2 + Tr3 .. Trn Average Proactive Path Provisioning Time =
Average Reactive Path Provisioning Rate = ------------------------ PPT1 + PPT2 + PPT3 .. PPTn
Total Trials --------------------------
Total Trials
SUM[SQUAREOF(Tri-RPPm)] Proactive Path Provisioning Time Variance (TPPv) =
Reactive Path Provisioning Rate Variance(RPPv) -------------------- SUM[SQUAREOF(PPTi - TPPm)]
Total Trials -1 --------------------------
Total Trials - 1
Where RPPm is the Average Reactive Path Provisioning Rate. Where TPPm is the Average Proactive Path Provisioning Time.
Reporting Format: Reporting Format:
The Reactive Path Provisioning Rate results MUST be reported in the The Proactive Path Provisioning Time results MUST be reported in
format of a table with a row for each iteration. The last row of the tabular format, with a row for each iteration. The last row of
table indicates the Reactive Path Provisioning Rate variance and the the table indicates the Proactive Path Provisioning Time variance,
previous row indicates the Average Reactive Path Provisioning Rate. and the previous row indicates the Average Proactive Path
Provisioning Time.
The report should capture the following information in addition to The report should capture the following information, in addition
the configuration parameters captured in section 4.8. to the configuration parameters captured per Section 4.8:
- Number of Network Devices in the path - Number of Network Devices in the path
- Offered rate 5.1.6. Reactive Path Provisioning Rate
5.1.7. Proactive Path Provisioning Rate Objective:
Objective: Measure the maximum number of independent paths a controller can
concurrently establish per second between source and destination
nodes reactively, defined as the number of paths provisioned per
second by the controller(s) at its southbound interface for the
flow provisioning requests received for path provisioning at its
southbound interface between the start of the test and the expiry
of the given Trial Duration.
Measure the maximum number of independent paths a controller can Reference Test Setup:
concurrently establish per second between source and destination
nodes proactively, defined as the number of paths provisioned per
second by the controller(s) at its Southbound interface for the
paths requested in its Northbound interface between the start of the
test and the expiry of given trial duration. The measurement is
based on dataplane observations of successful path activation
Reference Test Setup: This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
The test SHOULD use one of the test setups described in section 3.1 Prerequisites:
or section 3.2 of this document.
Prerequisite: 1. The controller MUST contain the network topology information
for the deployed network topology.
1. The controller MUST contain the network topology information for 2. The controller should know the location of destination
the deployed network topology. addresses for which the paths have to be provisioned. This can
be achieved through dynamic learning or static provisioning.
2. The controller should have the knowledge about the location of 3. Ensure that the default action for "flow miss" in the Network
destination addresses for which the paths have to be provisioned. Device is configured to "send to controller".
This can be achieved through dynamic learning or static
provisioning.
3. Ensure that the default action for flow miss in Network Device is 4. Ensure that each Network Device in a path requires the
'drop'. controller to make the forwarding decision while provisioning
the entire path.
Procedure: Procedure:
1. Send traffic continuously with unique source and destination 1. Send traffic with unique source and destination addresses from
addresses from test traffic generator TP1. test traffic generator TP1.
2. Install corresponding flow entries to reach from simulated 2. Record the total number of unique traffic frames (Ndf) received
sources at the test traffic generator TP1 to the simulated at test traffic generator TP2 within the Trial Duration (Td).
destinations at test traffic generator TP2 through controller's
northbound or management interface.
3. Record total number of unique traffic frames received Ndf) at the Measurements:
test traffic generator TP2 within the trial duration (Td).
Measurement: Ndf
Reactive Path Provisioning Rate (RPR1) = ------
Td
Ndf Average Reactive Path Provisioning Rate =
Proactive Path Provisioning Rate Tr1 = ------ RPR1 + RPR2 + RPR3 .. RPRn
Td --------------------------
Total Trials
Tr1 + Tr2 + Tr3 .. Trn Reactive Path Provisioning Rate Variance (RPPv) =
Average Proactive Path Provisioning Rate = ----------------------- SUM[SQUAREOF(RPRi - RPPm)]
Total Trials --------------------------
Total Trials - 1
SUM[SQUAREOF(Tri-PPPm)] Where RPPm is the Average Reactive Path Provisioning Rate.
Proactive Path Provisioning Rate Variance(PPPv) --------------------
Total Trials -1
Where PPPm is the Average Proactive Path Provisioning Rate. Reporting Format:
Reporting Format: The Reactive Path Provisioning Rate results MUST be reported in
tabular format, with a row for each iteration. The last row of
the table indicates the Reactive Path Provisioning Rate variance,
and the previous row indicates the Average Reactive Path
Provisioning Rate.
The Proactive Path Provisioning Rate results MUST be reported in the The report should capture the following information, in addition
format of a table with a row for each iteration. The last row of the to the configuration parameters captured per Section 4.8:
table indicates the Proactive Path Provisioning Rate variance and
the previous row indicates the Average Proactive Path Provisioning
Rate.
The report should capture the following information in addition to - Number of Network Devices in the path
the configuration parameters captured in section 4.8.
- Number of Network Devices in the path - Offered rate
- Offered rate 5.1.7. Proactive Path Provisioning Rate
5.1.8. Network Topology Change Detection Time Objective:
Objective: Measure the maximum number of independent paths a controller can
concurrently establish per second between source and destination
nodes proactively, defined as the number of paths provisioned per
second by the controller(s) at its southbound interface for the
paths requested in its northbound interface between the start of
the test and the expiry of the given Trial Duration. The
measurement is based on data-plane observations of successful path
activation.
The amount of time required for the controller to detect any changes Reference Test Setup:
in the network topology, defined as the interval starting with the
notification message received by the controller(s) at its Southbound
interface, ending with the first topology rediscovery messages sent
from the controller(s) at its Southbound interface.
Reference Test Setup: This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
The test SHOULD use one of the test setups described in section 3.1 Prerequisites:
or section 3.2 of this document.
Prerequisite: 1. The controller MUST contain the network topology information
for the deployed network topology.
1. The controller MUST have successfully discovered the network 2. The controller should know the location of destination
topology information for the deployed network topology. addresses for which the paths have to be provisioned. This can
be achieved through dynamic learning or static provisioning.
2. The periodic network discovery operation should be configured to 3. Ensure that the default action for "flow miss" in the Network
twice the Trial duration (Td) value. Device is "drop".
Procedure: Procedure:
1. Trigger a topology change event by bringing down an active 1. Send traffic continuously with unique source and destination
Network Device in the topology. addresses from test traffic generator TP1.
2. Record the time when the first topology change notification is 2. Install corresponding flow entries so that the traffic travels
sent to the controller (Tcn) at the forwarding plane test emulator from simulated sources at test traffic generator TP1 until it
interface (I1). reaches the simulated destinations at test traffic
generator TP2 through the controller's northbound interface or
management interface.
3. Stop the trial when the controller sends the first topology re- 3. Record the total number of unique traffic frames (Ndf) received
discovery message to the Network Device or the expiry of trial at test traffic generator TP2 within the Trial Duration (Td).
duration (Td).
4. Record the time when the first topology re-discovery message is Measurements:
received from the controller (Tcd) at the forwarding plane test
emulator interface (I1)
Measurement: Ndf
Proactive Path Provisioning Rate (PPR1) = ------
Td
Network Topology Change Detection Time Tr1 = Tcd-Tcn. Average Proactive Path Provisioning Rate =
PPR1 + PPR2 + PPR3 .. PPRn
--------------------------
Total Trials
Tr1 + Tr2 + Tr3 .. Trn Proactive Path Provisioning Rate Variance (PPPv) =
Average Network Topology Change Detection Time = ------------------ SUM[SQUAREOF(PPRi - PPPm)]
Total Trials -------------------------
Total Trials - 1
Network Topology Change Detection Time Variance(NTDv) = Where PPPm is the Average Proactive Path Provisioning Rate.
SUM[SQUAREOF(Tri-NTDm)] Reporting Format:
-----------------------
Total Trials -1
Where NTDm is the Average Network Topology Change Detection Time. The Proactive Path Provisioning Rate results MUST be reported in
tabular format, with a row for each iteration. The last row of
the table indicates the Proactive Path Provisioning Rate variance,
and the previous row indicates the Average Proactive Path
Provisioning Rate.
Reporting Format: The report should capture the following information, in addition
to the configuration parameters captured per Section 4.8:
The Network Topology Change Detection Time results MUST be reported - Number of Network Devices in the path
in the format of a table with a row for each iteration. The last row
of the table indicates the Network Topology Change Detection Time
variance and the previous row indicates the average Network Topology
Change Time.
5.2. Scalability - Offered rate
5.2.1. Control Session Capacity 5.1.8. Network Topology Change Detection Time
Objective: Objective:
Measure the maximum number of control sessions the controller can Measure the amount of time taken by the controller to detect any
maintain, defined as the number of sessions that the controller can changes in the network topology, defined as the interval starting
accept from network devices, starting with the first control with the notification message received by the controller(s) at its
session, ending with the last control session that the controller(s) southbound interface and ending with the first topology
accepts at its Southbound interface. rediscovery message sent from the controller(s) at its southbound
interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 This test SHOULD use one of the test setups illustrated in
or section 3.2 of this document. Section 3.1 or Section 3.2 of this document.
Procedure: Prerequisites:
1. Establish control connection with controller from every Network 1. The controller MUST have successfully discovered the network
Device emulated in the forwarding plane test emulator. topology information for the deployed network topology.
2. Stop the trial when the controller starts dropping the control
connections.
3. Record the number of successful connections established with the
controller (CCn) at the forwarding plane test emulator.
Measurement: 2. The periodic network discovery operation should be configured
to twice the Trial Duration (Td) value.
Control Sessions Capacity = CCn. Procedure:
Reporting Format: 1. Trigger a topology change event by bringing down an active
Network Device in the topology.
The Control Session Capacity results MUST be reported in addition to 2. Record the time when the first topology change notification is
the configuration parameters captured in section 4.8. sent to the controller (Tcn) at the forwarding-plane test
emulator interface (I1).
5.2.2. Network Discovery Size 3. Stop the trial when the controller sends the first topology
rediscovery message to the Network Device or the expiry of the
Trial Duration (Td).
Objective: 4. Record the time when the first topology rediscovery message is
received from the controller (Tcd) at the forwarding-plane test
emulator interface (I1).
Measure the network size (number of nodes, links and hosts) that a Measurements:
controller can discover, defined as the size of a network that the
controller(s) can discover, starting from a network topology given
by the user for discovery, ending with the topology that the
controller(s) could successfully discover.
Reference Test Setup: Network Topology Change Detection Time (TDT1) = Tcd - Tcn
The test SHOULD use one of the test setups described in section 3.1 Average Network Topology Change Detection Time =
or section 3.2 of this document. TDT1 + TDT2 + TDT3 .. TDTn
--------------------------
Total Trials
Prerequisite: Network Topology Change Detection Time Variance (NTDv) =
SUM[SQUAREOF(TDTi - NTDm)]
--------------------------
Total Trials - 1
1. The controller MUST support automatic network discovery. Where NTDm is the Average Network Topology Change
Detection Time.
2. Tester should be able to retrieve the discovered topology Reporting Format:
information either through controller's management interface or
northbound interface.
Procedure: The Network Topology Change Detection Time results MUST be
reported in tabular format, with a row for each iteration. The
last row of the table indicates the Network Topology Change
Detection Time variance, and the previous row indicates the
Average Network Topology Change Detection Time.
1. Establish the network connections between controller and network 5.2. Scalability
nodes.
2. Query the controller every t seconds (RECOMMENDED value for t is
30) to obtain the discovered network topology information through
the northbound interface or the management interface.
3. Stop the trial when the discovered network topology information
remains the same as that of last two query responses.
4. Compare the obtained network topology information with the
deployed network topology information.
5. If the comparison is successful, increase the number of nodes by 1
and repeat the trial.
If the comparison is unsuccessful, decrease the number of nodes by
1 and repeat the trial.
6. Continue the trial until the comparison of step 5 is successful.
7. Record the number of nodes for the last trial run (Ns) where the
topology comparison was successful.
Measurement: 5.2.1. Control Sessions Capacity
Network Discovery Size = Ns. Objective:
Reporting Format: Measure the maximum number of control sessions the controller can
maintain, defined as the number of sessions that the controller
can accept from Network Devices, starting with the first control
session and ending with the last control session that the
controller(s) accepts at its southbound interface.
The Network Discovery Size results MUST be reported in addition to Reference Test Setup:
the configuration parameters captured in section 4.8.
5.2.3. Forwarding Table Capacity This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
Objective: Prerequisites:
Measure the maximum number of flow entries a controller can manage None
in its Forwarding table.
Reference Test Setup: Procedure:
The test SHOULD use one of the test setups described in section 3.1 1. Establish control connections with the controller from every
or section 3.2 of this document. Network Device emulated in the forwarding-plane test emulator.
Prerequisite: 2. Stop the trial when the controller starts dropping the control
connections.
1. The controller Forwarding table should be empty. 3. Record the number of successful connections established (CCn)
2. Flow Idle time MUST be set to higher or infinite value. with the controller at the forwarding-plane test emulator.
3. The controller MUST have successfully completed network topology
discovery.
4. Tester should be able to retrieve the forwarding table information
either through controller's management interface or northbound
interface.
Procedure: Measurement:
Reactive Flow Provisioning Mode: Control Sessions Capacity = CCn
1. Send bi-directional traffic continuously with unique source and Reporting Format:
destination addresses from test traffic generators TP1 and TP2 at
the asynchronous message processing rate of controller.
2. Query the controller at a regular interval (e.g., 5 seconds) for
the number of learned flow entries from its northbound interface.
3. Stop the trial when the retrieved value is constant for three
consecutive iterations and record the value received from the last
query (Nrp).
Proactive Flow Provisioning Mode: The Control Sessions Capacity results MUST be reported in addition
to the configuration parameters captured per Section 4.8.
1. Install unique flows continuously through controller's northbound 5.2.2. Network Discovery Size
or management interface until a failure response is received from
the controller.
2. Record the total number of successful responses (Nrp).
Note: Objective:
Some controller designs for proactive flow provisioning mode may Measure the network size (number of nodes, links, and hosts) that
require the switch to send flow setup requests in order to generate a controller can discover, defined as the size of a network that
flow setup responses. In such cases, it is recommended to generate the controller(s) can discover, starting with a network topology
bi-directional traffic for the provisioned flows. provided by the user for discovery and ending with the number of
nodes, links, and hosts that the controller(s) were able to
successfully discover.
Measurement: Reference Test Setup:
Proactive Flow Provisioning Mode: This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
Max Flow Entries = Total number of flows provisioned (Nrp) Prerequisites:
Reactive Flow Provisioning Mode: 1. The controller MUST support automatic network discovery.
Max Flow Entries = Total number of learned flow entries (Nrp) 2. The tester should be able to retrieve the discovered topology
Forwarding Table Capacity = Max Flow Entries. information through either the controller's management
interface or northbound interface.
Reporting Format: Procedure:
The Forwarding Table Capacity results MUST be tabulated with the 1. Establish the network connections between the controller and
following information in addition to the configuration parameters the network nodes.
captured in section 4.8.
- Provisioning Type (Proactive/Reactive) 2. Query the controller every t seconds (the RECOMMENDED value for
t is 30) to obtain the discovered network topology information
through the northbound interface or the management interface.
5.3. Security 3. Stop the trial when the discovered network topology information
remains the same as that of the last two query responses.
5.3.1. Exception Handling 4. Compare the obtained network topology information with the
deployed network topology information.
Objective: 5. If the comparison is successful, increase the number of nodes
by 1 and repeat the trial.
If the comparison is unsuccessful, decrease the number of nodes
by 1 and repeat the trial.
Determine the effect of handling error packets and notifications on 6. Continue the trial until the comparison (step 5) is successful.
performance tests. The impact MUST be measured for the following
performance tests
a. Path Provisioning Rate 7. Record the number of nodes for the last trial run (Ns) where
the topology comparison was successful.
b. Path Provisioning Time Measurement:
c. Network Topology Change Detection Time Network Discovery Size = Ns
Reference Test Setup: Reporting Format:
The test SHOULD use one of the test setups described in section 3.1 The Network Discovery Size results MUST be reported in addition to
or section 3.2 of this document. the configuration parameters captured per Section 4.8.
Prerequisite: 5.2.3. Forwarding Table Capacity
1. This test MUST be performed after obtaining the baseline Objective:
measurement results for the above performance tests.
2. Ensure that the invalid messages are not dropped by the
intermediate devices connecting the controller and Network
Devices.
Procedure: Measure the maximum number of flow entries a controller can manage
in its Forwarding Table.
1. Perform the above listed performance tests and send 1% of messages Reference Test Setup:
from the Asynchronous Message Processing Rate as invalid messages
from the connected Network Devices emulated at the forwarding
plane test emulator.
2. Perform the above listed performance tests and send 2% of messages
from the Asynchronous Message Processing Rate as invalid messages
from the connected Network Devices emulated at the forwarding
plane test emulator.
Note: This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
Invalid messages can be frames with incorrect protocol fields or any Prerequisites:
form of failure notifications sent towards controller.
Measurement: 1. The controller's Forwarding Table should be empty.
Measurement MUST be done as per the equation defined in the 2. "Flow idle time" MUST be set to a higher or infinite value.
corresponding performance test measurement section.
Reporting Format: 3. The controller MUST have successfully completed network
topology discovery.
The Exception Handling results MUST be reported in the format of 4. The tester should be able to retrieve the Forwarding Table
table with a column for each of the below parameters and row for information through either the controller's management
each of the listed performance tests. interface or northbound interface.
- Without Exceptions Procedures:
- With 1% Exceptions o Reactive Flow Provisioning Mode:
- With 2% Exceptions 1. Send bidirectional traffic continuously with unique source
and destination addresses from test traffic generators TP1
and TP2 at the Asynchronous Message Processing Rate of the
controller.
5.3.2. Denial of Service Handling 2. Query the controller at a regular interval (e.g., every
5 seconds) for the number of learned flow entries from its
northbound interface.
Objective: 3. Stop the trial when the retrieved value is constant for
three consecutive iterations, and record the value received
from the last query (Nrp).
Determine the effect of handling DoS attacks on performance and o Proactive Flow Provisioning Mode:
scalability tests the impact MUST be measured for the following
tests:
a. Path Provisioning Rate 1. Install unique flows continuously through the controller's
northbound interface or management interface until a failure
response is received from the controller.
b. Path Provisioning Time 2. Record the total number of successful responses (Nrp).
c. Network Topology Change Detection Time
d. Network Discovery Size Note:
Reference Test Setup: Some controller designs for Proactive Flow Provisioning mode
may require the switch to send flow setup requests in order to
generate flow setup responses. In such cases, it is
recommended to generate bidirectional traffic for the
provisioned flows.
The test SHOULD use one of the test setups described in section 3.1 Measurements:
or section 3.2 of this document.
Prerequisite: Proactive Flow Provisioning Mode:
This test MUST be performed after obtaining the baseline measurement Max Flow Entries = Total number of flows provisioned (Nrp)
results for the above tests.
Procedure: Reactive Flow Provisioning Mode:
1. Perform the listed tests and launch a DoS attack towards Max Flow Entries = Total number of learned flow entries (Nrp)
controller while the trial is running.
Note: Forwarding Table Capacity = Max Flow Entries
DoS attacks can be launched on one of the following interfaces. Reporting Format:
a. Northbound (e.g., Query for flow entries continuously on The Forwarding Table Capacity results MUST be tabulated with the
northbound interface) following information, in addition to the configuration parameters
b. Management (e.g., Ping requests to controller's management captured per Section 4.8:
interface)
c. Southbound (e.g., TCP SYN messages on southbound interface)
Measurement: - Provisioning Type (Proactive/Reactive)
Measurement MUST be done as per the equation defined in the 5.3. Security
corresponding test's measurement section.
Reporting Format: 5.3.1. Exception Handling
The DoS Attacks Handling results MUST be reported in the format of Objective:
table with a column for each of the below parameters and row for
each of the listed tests.
- Without any attacks Determine the effects of handling error packets and notifications
on performance tests. The impact MUST be measured for the
following performance tests:
- With attacks 1. Path Provisioning Rate
The report should also specify the nature of attack and the 2. Path Provisioning Time
interface.
5.4. Reliability 3. Network Topology Change Detection Time
5.4.1. Controller Failover Time Reference Test Setup:
Objective: This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
The time taken to switch from an active controller to the backup Prerequisites:
controller, when the controllers work in redundancy mode and the
active controller fails, defined as the interval starting with the
active controller bringing down, ending with the first re-discovery
message received from the new controller at its Southbound
interface.
Reference Test Setup: 1. This test MUST be performed after obtaining the baseline
measurement results for the performance tests listed above.
The test SHOULD use the test setup described in section 3.2 of this 2. Ensure that the invalid messages are not dropped by the
document. intermediate devices connecting the controller and Network
Devices.
Prerequisite: Procedure:
1. Master controller election MUST be completed. 1. Perform the above-listed performance tests, and send 1% of the
2. Nodes are connected to the controller cluster as per the messages from the Asynchronous Message Processing Rate test
Redundancy Mode (RM). (Section 5.1.3) as invalid messages from the connected Network
3. The controller cluster should have successfully completed the Devices emulated at the forwarding-plane test emulator.
network topology discovery.
4. The Network Device MUST send all new flows to the controller when
it receives from the test traffic generator.
5. Controller should have learned the location of destination (D1) at
TP2.
Procedure: 2. Perform the above-listed performance tests, and send 2% of the
messages from the Asynchronous Message Processing Rate test
(Section 5.1.3) as invalid messages from the connected Network
Devices emulated at the forwarding-plane test emulator.
1. Send uni-directional traffic continuously with incremental Note:
sequence number and source addresses from test traffic generator
TP1 at the rate that the controller processes without any drops.
2. Ensure that there are no packet drops observed at TP2.
3. Bring down the active controller.
4. Stop the trial when a first frame received on TP2 after failover
operation.
5. Record the time at which the last valid frame received (T1) at
test traffic generator TP2 before sequence error and the first
valid frame received (T2) after the sequence error at TP2
Measurement: Invalid messages can be frames with incorrect protocol fields or
any form of failure notifications sent towards the controller.
Controller Failover Time = (T2 - T1) Measurements:
Packet Loss = Number of missing packet sequences. Measurements MUST be done as per the equation defined in the
"Measurements" section of the corresponding test listed under
"Objective".
Reporting Format: Reporting Format:
The Controller Failover Time results MUST be tabulated with the The Exception Handling results MUST be reported in tabular format,
following information. with a column for each of the below parameters and row for each of
the above-listed performance tests:
- Number of cluster nodes - Without Exceptions
- Redundancy mode - With 1% Exceptions
- Controller Failover Time - With 2% Exceptions
- Packet Loss 5.3.2. Handling Denial-of-Service Attacks
- Cluster keep-alive interval Objective:
5.4.2. Network Re-Provisioning Time Determine the effects of handling DoS attacks on performance and
scalability tests. The impact MUST be measured for the following
tests:
Objective: 1. Path Provisioning Rate
The time taken to re-route the traffic by the Controller, when there 2. Path Provisioning Time
is a failure in existing traffic paths, defined as the interval
starting from the first failure notification message received by the
controller, ending with the last flow re-provisioning message sent
by the controller at its Southbound interface.
Reference Test Setup: 3. Network Topology Change Detection Time
This test SHOULD use one of the test setup described in section 3.1 4. Network Discovery Size
or section 3.2 of this document.
Prerequisite: Reference Test Setup:
1. Network with the given number of nodes and redundant paths MUST be
deployed.
2. Ensure that the controller MUST have knowledge about the location
of test traffic generators TP1 and TP2.
3. Ensure that the controller does not pre-provision the alternate This test SHOULD use one of the test setups illustrated in
path in the emulated Network Devices at the forwarding plane test Section 3.1 or Section 3.2 of this document.
emulator.
Procedure: Prerequisite:
1. Send bi-directional traffic continuously with unique sequence This test MUST be performed after obtaining the baseline
number from TP1 and TP2. measurement results for the performance tests listed above.
2. Bring down a link or switch in the traffic path.
3. Stop the trial after receiving first frame after network re-
convergence.
4. Record the time of last received frame prior to the frame loss at
TP2 (TP2-Tlfr) and the time of first frame received after the
frame loss at TP2 (TP2-Tffr). There must be a gap in sequence
numbers of these frames
5. Record the time of last received frame prior to the frame loss at
TP1 (TP1-Tlfr) and the time of first frame received after the
frame loss at TP1 (TP1-Tffr).
Measurement: Procedure:
Forward Direction Path Re-Provisioning Time (FDRT) Perform the above-listed tests, and launch a DoS attack towards
= (TP2-Tffr - TP2-Tlfr) the controller while the trial is running.
Reverse Direction Path Re-Provisioning Time (RDRT) Note: DoS attacks can be launched on one of the following
= (TP1-Tffr - TP1-Tlfr) interfaces:
Network Re-Provisioning Time = (FDRT+RDRT)/2 1. Northbound (e.g., query for flow entries continuously on the
northbound interface)
Forward Direction Packet Loss = Number of missing sequence frames 2. Management (e.g., Ping requests to the controller's
at TP1 management interface)
Reverse Direction Packet Loss = Number of missing sequence frames 3. Southbound (e.g., TCP SYN messages on the southbound
at TP2 interface)
Reporting Format: Measurements:
The Network Re-Provisioning Time results MUST be tabulated with the Measurements MUST be done as per the equation defined in the
following information. "Measurements" section of the corresponding test listed under
"Objective".
- Number of nodes in the primary path Reporting Format:
- Number of nodes in the alternate path The results regarding the handling of DoS attacks MUST be reported
in tabular format, with a column for each of the below parameters
and a row for each of the above-listed tests.
- Network Re-Provisioning Time - Without any attacks
- Forward Direction Packet Loss
- Reverse Direction Packet Loss - With attacks
6. References The report should also specify the nature of the attack and the
interface in question.
6.1. Normative References 5.4. Reliability
[RFC2119] S. Bradner, "Key words for use in RFCs to Indicate 5.4.1. Controller Failover Time
Requirement Levels", RFC 2119, March 1997.
[RFC8174] B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC Objective:
2119 Key Words", RFC 8174, May 2017.
[I-D.sdn-controller-benchmark-term] Bhuvaneswaran.V, Anton Basil, Measure the time taken to switch from an active controller to the
Mark.T, Vishwas Manral, Sarah Banks, "Terminology for backup controller when the controllers work in redundancy mode and
Benchmarking SDN Controller Performance", the active controller fails, defined as the interval starting when
draft-ietf-bmwg-sdn-controller-benchmark-term-10 the active controller is brought down and ending with the first
(Work in progress), May 25, 2018 rediscovery message received from the new controller at its
southbound interface.
6.2. Informative References Reference Test Setup:
[OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" This test SHOULD use the test setup illustrated in Section 3.2 of
Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. this document.
7. IANA Considerations Prerequisites:
This document does not have any IANA requests. 1. Master controller election MUST be completed.
8. Security Considerations 2. Nodes are connected to the controller cluster per the
implemented redundancy mode (e.g., active-standby).
Benchmarking tests described in this document are limited to the 3. The controller cluster should have successfully completed the
performance characterization of controllers in a lab environment network topology discovery.
with isolated network.
4. The Network Device MUST send all new flows to the controller
when it receives them from the test traffic generator.
5. The controller should have learned the location of the
destination (D1) at test traffic generator TP2.
Procedure:
1. Send unidirectional traffic continuously with incremental
sequence numbers and source addresses from test traffic
generator TP1 at the rate at which the controller can process
the traffic without any drops.
2. Ensure that there are no packet drops observed at test traffic
generator TP2.
3. Bring down the active controller.
4. Stop the trial when the first frame after the failover
operation is received on test traffic generator TP2.
5. Record the time at which the last valid frame was received (T1)
at test traffic generator TP2 before the sequence error and the
time at which the first valid frame was received (T2) after the
sequence error at test traffic generator TP2.
Measurements:
Controller Failover Time = (T2 - T1)
Packet Loss = Number of missing packet sequences
Reporting Format:
The Controller Failover Time results MUST be tabulated with the
following information:
- Number of cluster nodes
- Redundancy mode
- Controller Failover Time
- Packet Loss
- Cluster keep-alive interval
5.4.2. Network Re-provisioning Time
Objective:
Measure the time taken by the controller to reroute traffic when
there is a failure in existing traffic paths, defined as the
interval starting with the first failure notification message
received by the controller and ending with the last flow
re-provisioning message sent by the controller at its southbound
interface.
Reference Test Setup:
This test SHOULD use one of the test setups illustrated in
Section 3.1 or Section 3.2 of this document.
Prerequisites:
1. A network with a specified number of nodes and redundant paths
MUST be deployed.
2. The controller MUST know the location of test traffic
generators TP1 and TP2.
3. Ensure that the controller does not pre-provision the alternate
path in the emulated Network Devices at the forwarding-plane
test emulator.
Procedure:
1. Send bidirectional traffic continuously with a unique sequence
number from test traffic generators TP1 and TP2.
2. Bring down a link or switch in the traffic path.
3. Stop the trial after receiving the first frame after network
reconvergence.
4. Record the time of the last received frame prior to the frame
loss at test traffic generator TP2 (TP2-Tlfr) and the time of
the first frame received after the frame loss at test traffic
generator TP2 (TP2-Tffr). There must be a gap in sequence
numbers of these frames.
5. Record the time of the last received frame prior to the frame
loss at test traffic generator TP1 (TP1-Tlfr) and the time of
the first frame received after the frame loss at test traffic
generator TP1 (TP1-Tffr).
Measurements:
Forward Direction Path Re-provisioning Time (FDRT)
= (TP2-Tffr - TP2-Tlfr)
Reverse Direction Path Re-provisioning Time (RDRT)
= (TP1-Tffr - TP1-Tlfr)
Network Re-provisioning Time = (FDRT + RDRT)/2
Forward Direction Packet Loss = Number of missing sequence frames
at test traffic generator TP1
Reverse Direction Packet Loss = Number of missing sequence frames
at test traffic generator TP2
Reporting Format:
The Network Re-provisioning Time results MUST be tabulated with
the following information:
- Number of nodes in the primary path
- Number of nodes in the alternate path
- Network Re-provisioning Time
- Forward Direction Packet Loss
- Reverse Direction Packet Loss
6. IANA Considerations
This document has no IANA actions.
7. Security Considerations
The benchmarking tests described in this document are limited to the
performance characterization of controllers in a lab environment with
isolated networks.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network, or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the controller. solely on measurements observable external to the controller.
Special capabilities SHOULD NOT exist in the controller specifically Special capabilities SHOULD NOT exist in the controller specifically
for benchmarking purposes. Any implications for network security for benchmarking purposes. Any implications for network security
arising from the controller SHOULD be identical in the lab and in arising from the controller SHOULD be identical in the lab and in
production networks. production networks.
9. Acknowledgments 8. References
The authors would like to thank the following individuals for 8.1. Normative References
providing their valuable comments to the earlier versions of this
document: Al Morton (AT&T), Sandeep Gangadharan (HP), M. Georgescu
(NAIST), Andrew McGregor (Google), Scott Bradner , Jay Karthik
(Cisco), Ramakrishnan (Dell), Khasanov Boris (Huawei), Brian
Castelli (Spirent)
This document was prepared using 2-Word-v2.0.template.dot. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
Appendix A Benchmarking Methodology using OpenFlow Controllers [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
RFC 2119 Key Words", BCP 14, RFC 8174,
DOI 10.17487/RFC8174, May 2017,
<https://www.rfc-editor.org/info/rfc8174>.
This section gives an overview of OpenFlow protocol and provides [RFC8455] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
test methodology to benchmark SDN controllers supporting OpenFlow and S. Banks, "Terminology for Benchmarking
southbound protocol. OpenFlow protocol is used as an example to Software-Defined Networking (SDN) Controller Performance",
illustrate the methodologies defined in this document. RFC 8455, DOI 10.17487/RFC8455, October 2018,
<https://www.rfc-editor.org/info/rfc8455>.
A.1. Protocol Overview 8.2. Informative References
OpenFlow is an open standard protocol defined by Open Networking [OpenFlow-Spec]
Foundation (ONF)[ OpenFlow Switch Specification], used for ONF, "OpenFlow Switch Specification" Version 1.4.0 (Wire
programming the forwarding plane of network switches or routers via Protocol 0x05), October 2013,
a centralized controller. <https://www.opennetworking.org/wp-content/
uploads/2014/10/openflow-spec-v1.4.0.pdf>.
A.2. Messages Overview Appendix A. Benchmarking Methodology Using OpenFlow Controllers
OpenFlow protocol supports three messages types namely controller- This section gives an overview of the OpenFlow protocol
to-switch, asynchronous and symmetric. [OpenFlow-Spec] and provides a test methodology for benchmarking SDN
Controllers supporting the OpenFlow southbound protocol. The
OpenFlow protocol is used as an example to illustrate the
methodologies defined in this document.
A.1. Protocol Overview
OpenFlow [OpenFlow-Spec] is an open standard protocol defined by the
Open Networking Foundation (ONF) and used for programming the
forwarding plane of network switches or routers via a centralized
controller.
A.2. Messages Overview
The OpenFlow protocol supports three message types -- namely,
controller-to-switch, asynchronous, and symmetric.
Controller-to-switch messages are initiated by the controller and Controller-to-switch messages are initiated by the controller and
used to directly manage or inspect the state of the switch. These used to directly manage or inspect the state of the switch. These
messages allow controllers to query/configure the switch (Features, messages allow controllers to query/configure the switch ("features"
Configuration messages), collect information from switch (Read-State messages, configuration messages), collect information from a switch
message), send packets on specified port of switch (Packet-out (Read-State messages), send packets on a specified port of a switch
message), and modify switch forwarding plane and state (Modify- (OFPT_PACKET_OUT messages), and modify the switch forwarding plane
State, Role-Request messages etc.). and state (Modify-State messages, Role-Request messages, etc.).
Asynchronous messages are generated by the switch without a Asynchronous messages are generated by the switch without a
controller soliciting them. These messages allow switches to update controller soliciting them. These messages allow switches to update
controllers to denote an arrival of new flow (Packet-in), switch controllers to denote an arrival of a new flow (OFPT_PACKET_IN
state change (Flow-Removed, Port-status) and error (Error). messages), switch state changes ("flow-removed" messages, port-status
messages), and errors (Error messages).
Symmetric messages are generated in either direction without Symmetric messages are generated in either direction without
solicitation. These messages allow switches and controllers to set solicitation. These messages allow switches and controllers to set
up connection (Hello), verify for liveness (Echo) and offer up a connection (Hello messages), verify liveness (Echo messages),
additional functionalities (Experimenter). and offer additional functionalities (Experimenter messages).
A.3. Connection Overview A.3. Connection Overview
OpenFlow channel is used to exchange OpenFlow message between an The OpenFlow channel is used to exchange OpenFlow messages between an
OpenFlow switch and an OpenFlow controller. The OpenFlow channel OpenFlow switch and an OpenFlow controller. The OpenFlow channel
connection can be setup using plain TCP or TLS. By default, a switch connection can be set up using plain TCP or TLS. By default, a
establishes single connection with SDN controller. A switch may switch establishes a single connection with the SDN Controller. A
establish multiple parallel connections to single controller switch may establish multiple parallel connections to a single
(auxiliary connection) or multiple controllers to handle controller controller (auxiliary connection) or multiple controllers to handle
failures and load balancing. controller failures and load balancing.
A.4. Performance Benchmarking Tests A.4. Performance Benchmarking Tests
A.4.1. Network Topology Discovery Time A.4.1. Network Topology Discovery Time
Procedure: Procedure:
Network Devices OpenFlow SDN Network Devices OpenFlow SDN
Controller Application Controller Application
| | | | | |
| |<Initialize controller | | |<Initialize controller |
| |app.,NB and SB interfaces> | | |app., NB and SB interfaces>|
| | | | | |
|<Deploy network with | | |<Deploy network with | |
| given no. of OF switches> | | | given no. of OF switches> | |
| | | | | |
| OFPT_HELLO Exchange | | | OFPT_HELLO Exchange | |
|<-------------------------->| | |<-------------------------->| |
| | | | | |
| PACKET_OUT with LLDP | | | OFPT_PACKET_OUT with LLDP| |
| to all switches | | | to all switches| |
(Tm1)|<---------------------------| | (Tm1)|<---------------------------| |
| | | | | |
| PACKET_IN with LLDP| | | OFPT_PACKET_IN with LLDP| |
| rcvd from switch-1| | | rcvd from Switch 1| |
|--------------------------->| | |--------------------------->| |
| | | | | |
| PACKET_IN with LLDP| | | OFPT_PACKET_IN with LLDP| |
| rcvd from switch-2| | | rcvd from Switch 2| |
|--------------------------->| | |--------------------------->| |
| . | | | . | |
| . | | | . | |
| | | | | |
| PACKET_IN with LLDP| | | OFPT_PACKET_IN with LLDP| |
| rcvd from switch-n| | | rcvd from Switch n| |
(Tmn)|--------------------------->| | (Tmn)|--------------------------->| |
| | | | | |
| | <Wait for the expiry | | | <Wait for the expiry of|
| | of Trial duration (Td)>| | | the Trial Duration (Td)>|
| | | | | |
| | Query the controller for| | | Query the controller for|
| | discovered n/w topo.(Di)| | | discovered n/w topo. (Di)|
| |<--------------------------| | |<--------------------------|
| | | | | |
| | <Compare the discovered | | | <Compare the discovered|
| | & offered n/w topology>| | | n/w topology and the|
| | offered n/w topology>|
| | | | | |
Legend: Legend:
NB: Northbound NB: Northbound
SB: Southbound SB: Southbound
OF: OpenFlow OF: OpenFlow
Tm1: Time of reception of first LLDP message from controller OFP: OpenFlow Protocol
Tmn: Time of last LLDP message sent to controller LLDP: Link-Layer Discovery Protocol
Tm1: Time of reception of first LLDP message from controller
Tmn: Time of last LLDP message sent to controller
Discussion: Discussion:
The Network Topology Discovery Time can be obtained by calculating The Network Topology Discovery Time can be obtained by calculating
the time difference between the first PACKET_OUT with LLDP message the time difference between the first OFPT_PACKET_OUT with an LLDP
received from the controller (Tm1) and the last PACKET_IN with LLDP message received from the controller (Tm1) and the last
message sent to the controller (Tmn) when the comparison is OFPT_PACKET_IN with an LLDP message sent to the controller (Tmn)
successful. when the comparison is successful.
A.4.2. Asynchronous Message Processing Time A.4.2. Asynchronous Message Processing Time
Procedure: Procedure:
Network Devices OpenFlow SDN Network Devices OpenFlow SDN
Controller Application Controller Application
| | | | | |
|PACKET_IN with single | | |OFPT_PACKET_IN with single | |
|OFP match header | | |OFP match header | |
(T0)|--------------------------->| | (T0)|--------------------------->| |
| | | | | |
| PACKET_OUT with single OFP | | |OFPT_PACKET_OUT with single | |
| action header | | | OFP action header | |
(R0)|<---------------------------| | (R0)|<---------------------------| |
| . | | | . | |
| . | | | . | |
| . | | | . | |
| | | | | |
|PACKET_IN with single OFP | | |OFPT_PACKET_IN with single | |
|match header | | |OFP match header | |
(Tn)|--------------------------->| | (Tn)|--------------------------->| |
| | | | | |
| PACKET_OUT with single OFP | | |OFPT_PACKET_OUT with single | |
| action header| | | OFP action header | |
(Rn)|<---------------------------| | (Rn)|<---------------------------| |
| | | | | |
|<Wait for the expiry of | | |<Wait for the expiry of the | |
|Trial duration> | | |Trial Duration> | |
| | | | | |
|<Record the number of | | |<Record the number of | |
|PACKET_INs/PACKET_OUTs | | |OFPT_PACKET_INs/ | |
|Exchanged (Nrx)> | | |OFPT_PACKET_OUTs | |
|exchanged (Nrx)> | |
| | | | | |
Legend: Legend:
T0,T1, ..Tn are PACKET_IN messages transmit timestamps. T0,T1, ..Tn: transmit timestamps of OFPT_PACKET_IN messages
R0,R1, ..Rn are PACKET_OUT messages receive timestamps. R0,R1, ..Rn: receive timestamps of OFPT_PACKET_OUT messages
Nrx : Number of successful PACKET_IN/PACKET_OUT message Nrx: Number of successful OFPT_PACKET_IN/OFPT_PACKET_OUT
exchanges message exchanges
Discussion: Discussion:
The Asynchronous Message Processing Time will be obtained by sum of The Asynchronous Message Processing Time will be obtained by
((R0-T0),(R1-T1)..(Rn - Tn))/ Nrx. calculating the sum of ((R0 - T0),(R1 - T1)..(Rn - Tn))/Nrx.
A.4.3. Asynchronous Message Processing Rate A.4.3. Asynchronous Message Processing Rate
Procedure: Procedure:
Network Devices OpenFlow SDN Network Devices OpenFlow SDN
Controller Application Controller Application
| | | | | |
|PACKET_IN with single OFP | | |OFPT_PACKET_IN with single | |
|match headers | | |OFP match header | |
|--------------------------->| | |--------------------------->| |
| | | | | |
| PACKET_OUT with single | | |OFPT_PACKET_OUT with single | |
| OFP action headers| | | OFP action header | |
|<---------------------------| | |<---------------------------| |
| | | | | |
| . | | | . | |
| . | | | . | |
| . | | | . | |
| | | | | |
|PACKET_IN with single OFP | | |OFPT_PACKET_IN with single | |
|match headers | | |OFP match header | |
|--------------------------->| | |--------------------------->| |
| | | | | |
| PACKET_OUT with single | | |OFPT_PACKET_OUT with single | |
| OFP action headers| | | OFP action header | |
|<---------------------------| | |<---------------------------| |
| | | | | |
|<Repeat the steps until the | | |<Repeat the steps until | |
|expiry of Trial Duration> | | |the expiry of the | |
| | | |Trial Duration> | |
|<Record the number of OFP | | | | |
(Ntx1)|match headers sent> | | |<Record the number of OFP | |
| | | (Ntx1)|match headers sent> | |
|<Record the number of OFP | | | | |
(Nrx1)|action headers rcvd> | | |<Record the number of OFP | |
| | | (Nrx1)|action headers rcvd> | |
| | |
Note: The Ntx1 on initial trials should be greater than Nrx1 and Note: The Ntx1 on initial trials should be greater than Nrx1.
repeat the trials until the Nrxn for two consecutive trials equeal Repeat the trials until the Nrxn for two consecutive trials equals
to (+/-P%). (+/-P%).
Discussion: Discussion:
This test will measure two benchmarks using single procedure. 1) The Using a single procedure, this test will measure two benchmarks:
Maximum Asynchronous Message Processing Rate will be obtained by
calculating the maximum PACKET OUTs (Nrxn) received from the
controller(s) across n trials. 2) The Loss-free Asynchronous Message
Processing Rate will be obtained by calculating the maximum PACKET
OUTs received from controller (s) when Loss Ratio equals zero. The
loss ratio is obtained by 1 - Nrxn/Ntxn
A.4.4. Reactive Path Provisioning Time 1. The Maximum Asynchronous Message Processing Rate will be
obtained by calculating the maximum OFPT_PACKET_OUTs (Nrxn)
received from the controller(s) across n trials.
Procedure: 2. The Loss-Free Asynchronous Message Processing Rate will be
obtained by calculating the maximum OFPT_PACKET_OUTs
received from the controller(s) when the Loss Ratio equals
zero. The Loss Ratio is obtained by calculating
1 - Nrxn/Ntxn.
Test Traffic Test Traffic Network Devices OpenFlow A.4.4. Reactive Path Provisioning Time
Generator TP1 Generator TP2 Controller
| | | |
| |G-ARP (D1) | |
| |--------------------->| |
| | | |
| | |PACKET_IN(D1) |
| | |------------------>|
| | | |
|Traffic (S1,D1) | |
(Tsf1)|----------------------------------->| |
| | | |
| | | |
| | | |
| | |PACKET_IN(S1,D1) |
| | |------------------>|
| | | |
| | | FLOW_MOD(D1) |
| | |<------------------|
| | | |
| |Traffic (S1,D1) | |
| (Tdf1)|<---------------------| |
| | | |
Legend: Procedure:
G-ARP: Gratuitous ARP message. Test Traffic Test Traffic Network Devices OpenFlow
Tsf1: Time of first frame sent from TP1 Generator TP1 Generator TP2 Controller
Tdf1: Time of first frame received from TP2 | | | |
| |G-ARP (D1) | |
| |--------------------->| |
| | | |
| | |OFPT_PACKET_IN(D1) |
| | |-------------------->|
| | | |
|Traffic (S1,D1) | |
(Tsf1)|----------------------------------->| |
| | | |
| | | |
| | | |
| | |OFPT_PACKET_IN(S1,D1)|
| | |-------------------->|
| | | |
| | | FLOW_MOD(D1) |
| | |<--------------------|
| | | |
| |Traffic (S1,D1) | |
| (Tdf1)|<---------------------| |
| | | |
Discussion: Legend:
The Reactive Path Provisioning Time can be obtained by finding the G-ARP: Gratuitous ARP message
time difference between the transmit and receive time of the traffic Tsf1: Time of first frame sent from TP1
(Tsf1-Tdf1). Tdf1: Time of first frame received from TP2
A.4.5. Proactive Path Provisioning Time Discussion:
Procedure: The Reactive Path Provisioning Time can be obtained by finding the
time difference between the transmit and receive times of the
traffic (Tsf1 - Tdf1).
Test Traffic Test Traffic Network Devices OpenFlow SDN A.4.5. Proactive Path Provisioning Time
Generator TP1 Generator TP2 Controller Application
| | | | |
| | | | |
| | | | <Install flow|
| | | | for S1,D1> |
| |G-ARP (D1) | | |
| |-------------->| | |
| | | | |
| | |PACKET_IN(D1) | |
| | |--------------->| |
| | | | |
|Traffic (S1,D1) | | |
Tsf1)|---------------------------->| | |
| | | | |
| | | FLOW_MOD(D1) | |
| | |<---------------| |
| | | | |
| |Traffic (S1,D1)| | |
| (Tdf1)|<--------------| | |
| | | | |
Legend: Procedure:
G-ARP: Gratuitous ARP message. Test Traffic Test Traffic Network Devices OpenFlow SDN
Tsf1: Time of first frame sent from TP1 Generator TP1 Generator TP2 Controller Application
Tdf1: Time of first frame received from TP2 | | | | |
| | | | |
| | | |<Install flow|
| | | | for S1,D1> |
| |G-ARP (D1) | | |
| |-------------->| | |
| | | | |
| | |OFPT_PACKET_IN(D1)| |
| | |----------------->| |
| | | | |
|Traffic (S1,D1) | | |
(Tsf1)|--------------------------->| | |
| | | | |
| | | FLOW_MOD(D1) | |
| | |<-----------------| |
| | | | |
| |Traffic (S1,D1)| | |
| (Tdf1)|<--------------| | |
| | | | |
Discussion: Legend:
The Proactive Path Provisioning Time can be obtained by finding the G-ARP: Gratuitous ARP message
time difference between the transmit and receive time of the traffic Tsf1: Time of first frame sent from TP1
(Tsf1-Tdf1). Tdf1: Time of first frame received from TP2
A.4.6. Reactive Path Provisioning Rate Discussion:
Procedure: The Proactive Path Provisioning Time can be obtained by finding
the time difference between the transmit and receive times of the
traffic (Tsf1 - Tdf1).
A.4.6. Reactive Path Provisioning Rate
Procedure:
Test Traffic Test Traffic Network Devices OpenFlow Test Traffic Test Traffic Network Devices OpenFlow
Generator TP1 Generator TP2 Controller Generator TP1 Generator TP2 Controller
| | | | | | | |
| | | | | | | |
| | | | | | | |
| |G-ARP (D1..Dn) | | | |G-ARP (D1..Dn) | |
| |--------------------| | | |--------------------| |
| | | | | | | |
| | |PACKET_IN(D1..Dn) | | | |OFPT_PACKET_IN(D1..Dn)|
| | |--------------------->| | | |--------------------->|
| | | | | | | |
|Traffic (S1..Sn,D1..Dn) | | |Traffic (S1..Sn,D1..Dn) | |
|--------------------------------->| | |--------------------------------->| |
| | | | | | | |
| | |PACKET_IN(S1.Sn,D1.Dn)| | | |OFPT_PACKET_IN(S1..Sn,|
| | | D1..Dn)|
| | |--------------------->| | | |--------------------->|
| | | | | | | |
| | | FLOW_MOD(S1) | | | | FLOW_MOD(S1) |
| | |<---------------------| | | |<---------------------|
| | | | | | | |
| | | FLOW_MOD(D1) | | | | FLOW_MOD(D1) |
| | |<---------------------| | | |<---------------------|
| | | | | | | |
| | | FLOW_MOD(S2) | | | | FLOW_MOD(S2) |
| | |<---------------------| | | |<---------------------|
skipping to change at page 41, line 10 skipping to change at page 48, line 5
| | | | | | | |
| | | FLOW_MOD(Dn) | | | | FLOW_MOD(Dn) |
| | |<---------------------| | | |<---------------------|
| | | | | | | |
| | Traffic (S1..Sn, | | | | Traffic (S1..Sn, | |
| | D1..Dn)| | | | D1..Dn)| |
| |<-------------------| | | |<-------------------| |
| | | | | | | |
| | | | | | | |
Legend: Legend:
G-ARP: Gratuitous ARP G-ARP: Gratuitous ARP message
D1..Dn: Destination Endpoint 1, Destination Endpoint 2 .... D1..Dn: Destination Endpoint 1, Destination Endpoint 2 ...,
Destination Endpoint n Destination Endpoint n
S1..Sn: Source Endpoint 1, Source Endpoint 2 .., Source S1..Sn: Source Endpoint 1, Source Endpoint 2 ...,
Endpoint n Source Endpoint n
Discussion: Discussion:
The Reactive Path Provisioning Rate can be obtained by finding the The Reactive Path Provisioning Rate can be obtained by finding the
total number of frames received at TP2 after the trial duration. total number of frames received at test traffic generator TP2
after the Trial Duration.
A.4.7. Proactive Path Provisioning Rate A.4.7. Proactive Path Provisioning Rate
Procedure: Procedure:
Test Traffic Test Traffic Network Devices OpenFlow SDN Test Traffic Test Traffic Network Devices OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1..Dn) | | | | |G-ARP (D1..Dn) | | |
| |-------------->| | | | |--------------->| | |
| | | | | | | | | |
| | |PACKET_IN(D1.Dn)| | | | |OFPT_PACKET_IN | |
| | |--------------->| | | | | (D1..Dn)| |
| | | | | | | |---------------->| |
|Traffic (S1..Sn,D1..Dn) | | | | | | | |
Tsf1)|---------------------------->| | | |Traffic (S1..Sn,D1..Dn) | | |
| | | | | (Tsf1)|---------------------------->| | |
| | | | <Install flow| | | | | |
| | | | for S1,D1> | | | | |<Install flow|
| | | | | | | | | for S1,D1> |
| | | | . | | | | | |
| | | | <Install flow| | | | | . |
| | | | for Sn,Dn> | | | | |<Install flow|
| | | | | | | | | for Sn,Dn> |
| | | FLOW_MOD(S1) | | | | | | |
| | |<---------------| | | | | FLOW_MOD(S1) | |
| | | | | | | |<----------------| |
| | | FLOW_MOD(D1) | | | | | | |
| | |<---------------| | | | | FLOW_MOD(D1) | |
| | | | | | | |<----------------| |
| | | . | | | | | | |
| | | FLOW_MOD(Sn) | | | | | . | |
| | |<---------------| | | | | FLOW_MOD(Sn) | |
| | | | | | | |<----------------| |
| | | FLOW_MOD(Dn) | | | | | | |
| | |<---------------| | | | | FLOW_MOD(Dn) | |
| | | | | | | |<----------------| |
| |Traffic (S1.Sn,| | | | | | | |
| | D1.Dn)| | | | |Traffic (S1..Sn,| | |
| (Tdf1)|<--------------| | | | | D1..Dn)| | |
| | | | | | (Tdf1)|<---------------| | |
| | | | |
Legend: Legend:
G-ARP: Gratuitous ARP G-ARP: Gratuitous ARP message
D1..Dn: Destination Endpoint 1, Destination Endpoint 2 .... D1..Dn: Destination Endpoint 1, Destination Endpoint 2 ...,
Destination Endpoint n Destination Endpoint n
S1..Sn: Source Endpoint 1, Source Endpoint 2 .., Source S1..Sn: Source Endpoint 1, Source Endpoint 2 ...,
Endpoint n Source Endpoint n
Discussion: Discussion:
The Proactive Path Provisioning Rate can be obtained by finding the The Proactive Path Provisioning Rate can be obtained by finding
total number of frames received at TP2 after the trial duration the total number of frames received at test traffic generator TP2
after the Trial Duration.
A.4.8. Network Topology Change Detection Time A.4.8. Network Topology Change Detection Time
Procedure: Procedure:
Network Devices OpenFlow SDN Network Devices OpenFlow SDN
Controller Application Controller Application
| | | | | |
| | <Bring down a link in | | | <Bring down a link in |
| | switch S1>| | | Switch S1>|
| | | | | |
T0 |PORT_STATUS with link down | | T0 |PORT_STATUS with link down | |
| from S1 | | | from S1 | |
|--------------------------->| | |--------------------------->| |
| | | | | |
|First PACKET_OUT with LLDP | | |First OFPT_PACKET_OUT with | |
|to OF Switch | | |LLDP to OF switch | |
T1 |<---------------------------| | T1 |<---------------------------| |
| | | | | |
| | <Record time of 1st | | | <Record time of first|
| | PACKET_OUT with LLDP T1>| | | OFPT_PACKET_OUT with|
| | LLDP T1>|
| | |
Discussion: Discussion:
The Network Topology Change Detection Time can be obtained by The Network Topology Change Detection Time can be obtained by
finding the difference between the time the OpenFlow switch S1 sends finding the difference between the time that OpenFlow Switch S1
the PORT_STATUS message (T0) and the time that the OpenFlow sends the PORT_STATUS message (T0) and the time that the OpenFlow
controller sends the first topology re-discovery message (T1) to controller sends the first topology rediscovery message (T1) to
OpenFlow switches. OpenFlow switches.
A.5. Scalability A.5. Scalability
A.5.1. Control Sessions Capacity A.5.1. Control Sessions Capacity
Procedure: Procedure:
Network Devices OpenFlow Network Devices OpenFlow
Controller Controller
| | | |
| OFPT_HELLO Exchange for Switch 1 | | OFPT_HELLO Exchange for Switch 1 |
|<------------------------------------->| |<------------------------------------->|
| | | |
| OFPT_HELLO Exchange for Switch 2 | | OFPT_HELLO Exchange for Switch 2 |
|<------------------------------------->| |<------------------------------------->|
| . | | . |
| . | | . |
| . | | . |
| OFPT_HELLO Exchange for Switch n | | OFPT_HELLO Exchange for Switch n |
|X<----------------------------------->X| |X<----------------------------------->X|
| | | |
Discussion: Discussion:
The value of Switch n-1 will provide Control Sessions Capacity. The value of Switch (n - 1) will provide the Control Sessions
Capacity.
A.5.2. Network Discovery Size A.5.2. Network Discovery Size
Procedure: Procedure:
Network Devices OpenFlow SDN Network Devices OpenFlow SDN
Controller Application Controller Application
| | | | | |
| | <Deploy network with | | | <Deploy network with |
| |given no. of OF switches N>| | |given no. of OF switches N>|
| | | | | |
| OFPT_HELLO Exchange | | | OFPT_HELLO Exchange | |
|<-------------------------->| | |<-------------------------->| |
| | | | | |
| PACKET_OUT with LLDP | | | OFPT_PACKET_OUT with LLDP| |
| to all switches | | | to all switches | |
|<---------------------------| | |<---------------------------| |
| | | | | |
| PACKET_IN with LLDP| | | OFPT_PACKET_IN with LLDP| |
| rcvd from switch-1| | | rcvd from Switch 1| |
|--------------------------->| | |--------------------------->| |
| | | | | |
| PACKET_IN with LLDP| | | OFPT_PACKET_IN with LLDP| |
| rcvd from switch-2| | | rcvd from Switch 2| |
|--------------------------->| | |--------------------------->| |
| . | | | . | |
| . | | | . | |
| | | | | |
| PACKET_IN with LLDP| | | OFPT_PACKET_IN with LLDP| |
| rcvd from switch-n| | | rcvd from Switch n| |
|--------------------------->| | |--------------------------->| |
| | | | | |
| | <Wait for the expiry | | | <Wait for the expiry of|
| | of Trial duration (Td)>| | | the Trial Duration (Td)>|
| | | | | |
| | Query the controller for| | | Query the controller for|
| | discovered n/w topo.(N1)| | | discovered n/w topo. (N1)|
| |<--------------------------| | |<--------------------------|
| | | | | |
| | <If N1==N repeat Step 1 | | | <If N1==N, repeat Step 1|
| |with N+1 nodes until N1<N >| | | with N + 1 nodes|
| | until N1<N >|
| | | | | |
| | <If N1<N repeat Step 1 | | | <If N1<N, repeat Step 1 |
| | with N=N1 nodes once and | | | with N=N1 nodes once and |
| | exit> | | | exit> |
| | | | | |
Legend: Legend:
n/w topo: Network Topology n/w topo: Network topology
OF: OpenFlow OF: OpenFlow
Discussion: Discussion:
The value of N1 provides the Network Discovery Size value. The trial The value of N1 provides the Network Discovery Size value. The
duration can be set to the stipulated time within which the user Trial Duration can be set to the stipulated time within which the
expects the controller to complete the discovery process. user expects the controller to complete the discovery process.
A.5.3. Forwarding Table Capacity A.5.3. Forwarding Table Capacity
Procedure:
Test Traffic Network Devices OpenFlow SDN Procedure:
Generator TP1 Controller Application
| | | | Test Traffic Network Devices OpenFlow SDN
| | | | Generator TP1 Controller Application
|G-ARP (H1..Hn) | | | | | | |
|----------------->| | | | | | |
| | | | |G-ARP (H1..Hn) | | |
| |PACKET_IN(D1..Dn) | | |---------------->| | |
| |------------------>| | | | | |
| | | | | |OFPT_PACKET_IN(D1..Dn)| |
| | |<Wait for 5 secs>| | |--------------------->| |
| | | | | | | |
| | | <Query for FWD | | | |<Wait for 5 secs>|
| | | entry> |(F1) | | | |
| | | | | | | <Query for FWD |
| | |<Wait for 5 secs>| | | | entry> |(F1)
| | | | | | | |
| | | <Query for FWD | | | |<Wait for 5 secs>|
| | | entry> |(F2) | | | |
| | | | | | | <Query for FWD |
| | |<Wait for 5 secs>| | | | entry> |(F2)
| | | | | | | |
| | | <Query for FWD | | | |<Wait for 5 secs>|
| | | entry> |(F3) | | | |
| | | | | | | <Query for FWD |
| | | <Repeat Step 2 | | | | entry> |(F3)
| | |until F1==F2==F3>| | | | |
| | | | | | | <Repeat Step 2 |
| | |until F1==F2==F3>|
| | | |
Legend: Legend:
G-ARP: Gratuitous ARP G-ARP: Gratuitous ARP message
H1..Hn: Host 1 .. Host n H1..Hn: Host 1 .. Host n
FWD: Forwarding Table FWD: Forwarding Table
Discussion: Discussion:
Query the controller forwarding table entries for multiple times Query the controller's Forwarding Table entries multiple times,
until the three consecutive queries return the same value. The last until three consecutive queries return the same value. The last
value retrieved from the controller will provide the Forwarding value retrieved from the controller will provide the Forwarding
Table Capacity value. The query interval is user configurable. The 5 Table Capacity value. The query interval is user configurable.
seconds shown in this example is for representational purpose. The interval of 5 seconds shown in this example is for
representational purposes.
A.6. Security A.6. Security
A.6.1. Exception Handling A.6.1. Exception Handling
Procedure: Procedure:
Test Traffic Test Traffic Network Devices OpenFlow SDN Test Traffic Test Traffic Network Devices OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1..Dn) | | | | |G-ARP (D1..Dn) | | |
| |------------------>| | | | |--------------->| | |
| | | | | | | | | |
| | |PACKET_IN(D1..Dn)| | | | |OFPT_PACKET_IN(D1..Dn)| |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
|Traffic (S1..Sn,D1..Dn) | | | |Traffic (S1..Sn,D1..Dn) | | |
|----------------------------->| | | |-------------------------->| | |
| | | | | | | | | |
| | |PACKET_IN(S1..Sa,| | | | |OFPT_PACKET_IN(S1..Sa,| |
| | | D1..Da)| | | | | D1..Da)| |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
| | |PACKET_IN(Sa+1.. | | | | |OFPT_PACKET_IN | |
| | |.Sn,Da+1..Dn) | | | | | (Sa+1..Sn,| |
| | |(1% incorrect OFP| | | | | Da+1..Dn)| |
| | | Match header)| | | | | (1% incorrect OFP| |
| | |---------------->| | | | | match header)| |
| | | | | | | |--------------------->| |
| | | FLOW_MOD(D1..Dn)| | | | | | |
| | |<----------------| | | | | FLOW_MOD(D1..Dn)| |
| | | | | | | |<---------------------| |
| | | FLOW_MOD(S1..Sa)| | | | | | |
| | | OFP headers| | | | | FLOW_MOD(S1..Sa)| |
| | |<----------------| | | | | OFP headers| |
| | | | | | | |<---------------------| |
| |Traffic (S1..Sa, | | | | | | | |
| | D1..Da)| | | | |Traffic (S1..Sa,| | |
| |<------------------| | | | | D1..Da)| | |
| | | | | | |<---------------| | |
| | | | <Wait for | | | | | |
| | | | Test | | | | | <Wait for the|
| | | | Duration>| | | | | expiry of the|
| | | | | | | | | Trial|
| | | | <Record Rx| | | | | Duration>|
| | | | frames at| | | | | |
| | | | TP2 (Rn1)>| | | | | <Record Rx|
| | | | | | | | | frames at|
| | | | <Repeat | | | | | TP2 (Rn1)>|
| | | | Step1 with | | | | | |
| | | |2% incorrect| | | | | <Repeat |
| | | | PACKET_INs>| | | | | Step 1 with|
| | | | | | | | | 2% incorrect|
| | | | <Record Rx| | | | |OFPT_PACKET_INs>|
| | | | frames at| | | | | |
| | | | TP2 (Rn2)>| | | | | <Record Rx|
| | | | | | | | | frames at|
| | | | TP2 (Rn2)>|
Legend: Legend:
G-ARP: Gratuitous ARP G-ARP: Gratuitous ARP message
PACKET_IN(Sa+1..Sn,Da+1..Dn): OpenFlow PACKET_IN with wrong OFPT_PACKET_IN(Sa+1..Sn,Da+1..Dn): OFPT_PACKET_IN with
version number wrong version number
Rn1: Total number of frames received at Test Port 2 with Rn1: Total number of frames received at Test Port 2
1% incorrect frames with 1% incorrect frames
Rn2: Total number of frames received at Test Port 2 with Rn2: Total number of frames received at Test Port 2
2% incorrect frames with 2% incorrect frames
Discussion: Discussion:
The traffic rate sent towards OpenFlow switch from Test Port 1 The traffic rate sent towards the OpenFlow switch from Test Port 1
should be 1% higher than the Path Programming Rate. Rn1 will provide should be 1% higher than the Path Programming Rate. Rn1 will
the Path Provisioning Rate of controller at 1% of incorrect frames provide the Path Provisioning Rate of the controller when 1% of
handling and Rn2 will provide the Path Provisioning Rate of incorrect frames are received, and Rn2 will provide the Path
controller at 2% of incorrect frames handling. Provisioning Rate of the controller when 2% of incorrect frames
are received.
The procedure defined above provides test steps to determine the The procedure defined above provides test steps to determine the
effect of handling error packets on Path Programming Rate. Same effects of handling error packets on the Path Programming Rate.
procedure can be adopted to determine the effects on other The same procedure can be adapted to determine the effects on
performance tests listed in this benchmarking tests. other performance tests listed in this benchmarking test.
A.6.2. Denial of Service Handling A.6.2. Handling Denial-of-Service Attacks
Procedure: Procedure:
Test Traffic Test Traffic Network Devic OpenFlow SDN Test Traffic Test Traffic Network Device OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1..Dn) | | | | |G-ARP (D1..Dn) | | |
| |------------------>| | | | |---------------->| | |
| | | | | | | | | |
| | |PACKET_IN(D1..Dn)| | | | |OFPT_PACKET_IN(D1..Dn)| |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
|Traffic (S1..Sn,D1..Dn) | | | |Traffic (S1..Sn,D1..Dn) | | |
|----------------------------->| | | |--------------------------->| | |
| | | | | | | | | |
| | |PACKET_IN(S1..Sn,| | | | |OFPT_PACKET_IN(S1..Sn,| |
| | | D1..Dn)| | | | | D1..Dn)| |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
| | |TCP SYN Attack | | | | |TCP SYN attack | |
| | |from a switch | | | | |from a switch | |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
| | |FLOW_MOD(D1..Dn) | | | | |FLOW_MOD(D1..Dn) | |
| | |<----------------| | | | |<---------------------| |
| | | | | | | | | |
| | | FLOW_MOD(S1..Sn)| | | | | FLOW_MOD(S1..Sn) | |
| | | OFP headers| | | | | OFP headers | |
| | |<----------------| | | | |<---------------------| |
| | | | | | | | | |
| |Traffic (S1..Sn, | | | | |Traffic (S1..Sn, | | |
| | D1..Dn)| | | | | D1..Dn) | | |
| |<------------------| | | | |<----------------| | |
| | | | | | | | | |
| | | | <Wait for | | | | |<Wait for the|
| | | | Test | | | | |expiry of the|
| | | | Duration>| | | | | Trial|
| | | | | | | | | Duration>|
| | | | <Record Rx| | | | | |
| | | | frames at| | | | | <Record Rx|
| | | | TP2 (Rn1)>| | | | | frames at|
| | | | | | | | | TP2 (Rn1)>|
| | | | |
Legend: Legend:
G-ARP: Gratuitous ARP G-ARP: Gratuitous ARP message
Discussion: Discussion:
TCP SYN attack should be launched from one of the emulated/simulated A TCP SYN attack should be launched from one of the
OpenFlow Switch. Rn1 provides the Path Programming Rate of emulated/simulated OpenFlow switches. Rn1 provides the Path
controller uponhandling denial of service attack. Programming Rate of the controller upon handling a denial-of-
service attack.
The procedure defined above provides test steps to determine the The procedure defined above provides test steps to determine the
effect of handling denial of service on Path Programming Rate. Same effects of handling denial of service on the Path Programming
procedure can be adopted to determine the effects on other Rate. The same procedure can be adapted to determine the effects
performance tests listed in this benchmarking tests. on other performance tests listed in this benchmarking test.
A.7. Reliability A.7. Reliability
A.7.1. Controller Failover Time A.7.1. Controller Failover Time
Procedure: Procedure:
Test Traffic Test Traffic Network Device OpenFlow SDN Test Traffic Test Traffic Network Device OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1) | | | | |G-ARP (D1) | | |
| |------------>| | | | |-------------->| | |
| | | | | | | | | |
| | |PACKET_IN(D1) | | | | |OFPT_PACKET_IN(D1) | |
| | |---------------->| | | | |---------------------->| |
| | | | | | | | | |
|Traffic (S1..Sn,D1) | | | |Traffic (S1..Sn,D1) | | |
|-------------------------->| | | |--------------------------->| | |
| | | | | | | | | |
| | | | | | | | | |
| | |PACKET_IN(S1,D1) | | | | |OFPT_PACKET_IN(S1,D1) | |
| | |---------------->| | | | |---------------------->| |
| | | | | | | | | |
| | |FLOW_MOD(D1) | | | | |FLOW_MOD(D1) | |
| | |<----------------| | | | |<----------------------| |
| | |FLOW_MOD(S1) | | | | |FLOW_MOD(S1) | |
| | |<----------------| | | | |<----------------------| |
| | | | | | | | | |
| |Traffic (S1,D1)| | | | |Traffic (S1,D1)| | |
| |<------------| | | | |<--------------| | |
| | | | | | | | | |
| | |PACKET_IN(S2,D1) | | | | |OFPT_PACKET_IN(S2,D1) | |
| | |---------------->| | | | |---------------------->| |
| | | | | | | | | |
| | |FLOW_MOD(S2) | | | | |FLOW_MOD(S2) | |
| | |<----------------| | | | |<----------------------| |
| | | | | | | | | |
| | |PACKET_IN(Sn-1,D1)| | | | |OFPT_PACKET_IN | |
| | |---------------->| | | | | (Sn-1,D1) | |
| | | | | | | |---------------------->| |
| | |PACKET_IN(Sn,D1) | | | | | | |
| | |---------------->| | | | |OFPT_PACKET_IN(Sn,D1) | |
| | | . | | | | |---------------------->| |
| | | . |<Bring down the| | | | . | |
| | | . |active control-| | | | . |<Bring down |
| | | | ler> | | | | . | the active |
| | | FLOW_MOD(Sn-1) | | | | | | controller> |
| | | <-X----------| | | | | FLOW_MOD(Sn-1) | |
| | | | | | | | X<-----------------| |
| | |FLOW_MOD(Sn) | | | | | | |
| | |<----------------| | | | |FLOW_MOD(Sn) | |
| | | | | | | |<----------------------| |
| |Traffic (Sn,D1)| | | | | | | |
| |<------------| | | | |Traffic (Sn,D1)| | |
| | | | | | |<--------------| | |
| | | |<Stop the test | | | | | |
| | | |after recv. | | | | |<Stop the |
| | | |traffic upon | | | | |test after |
| | | | failure> | | | | |recv. traffic|
| | | |upon |
| | | |failure> |
Legend: Legend:
G-ARP: Gratuitous ARP. G-ARP: Gratuitous ARP message
Discussion: Discussion:
The time difference between the last valid frame received before the The time difference between the last valid frame received before
traffic loss and the first frame received after the traffic loss the traffic loss and the first frame received after the traffic
will provide the controller failover time. loss will provide the Controller Failover Time.
If there is no frame loss during controller failover time, the If there is no frame loss during the Controller Failover Time, the
controller failover time can be deemed negligible. Controller Failover Time can be deemed negligible.
A.7.2. Network Re-Provisioning Time A.7.2. Network Re-provisioning Time
Procedure: Procedure:
Test Traffic Test Traffic Network Devices OpenFlow SDN Test Traffic Test Traffic Network Devices OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1) | | | | |G-ARP (D1) | | |
| |-------------->| | | | |--------------->| | |
| | | | | | | | | |
| | |PACKET_IN(D1) | | | | |OFPT_PACKET_IN(D1) | |
| | |---------------->| | | | |--------------------->| |
| G-ARP (S1) | | | | |G-ARP (S1) | | |
|---------------------------->| | | |----------------------------->| | |
| | | | | | | | | |
| | |PACKET_IN(S1) | | | | |OFPT_PACKET_IN(S1) | |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
|Traffic (S1,D1,Seq.no (1..n))| | | |Traffic (S1,D1,Seq. no (1..n))| | |
|---------------------------->| | | |----------------------------->| | |
| | | | | | | | | |
| | |PACKET_IN(S1,D1) | | | | |OFPT_PACKET_IN(S1,D1) | |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
| |Traffic (D1,S1,| | | | | Traffic (D1,S1,| | |
| | Seq.no (1..n))| | | | | Seq. no (1..n))| | |
| |-------------->| | | | |--------------->| | |
| | | | | | | | | |
| | |PACKET_IN(D1,S1) | | | | |OFPT_PACKET_IN(D1,S1) | |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
| | |FLOW_MOD(D1) | | | | |FLOW_MOD(D1) | |
| | |<----------------| | | | |<---------------------| |
| | | | | | | | | |
| | |FLOW_MOD(S1) | | | | |FLOW_MOD(S1) | |
| | |<----------------| | | | |<---------------------| |
| | | | | | | | | |
| |Traffic (S1,D1,| | | | | Traffic (S1,D1,| | |
| | Seq.no(1))| | | | | Seq. no(1))| | |
| |<--------------| | | | |<---------------| | |
| | | | | | | | | |
| |Traffic (S1,D1,| | | | | Traffic (S1,D1,| | |
| | Seq.no(2))| | | | | Seq. no(2))| | |
| |<--------------| | | | |<---------------| | |
| | | | | | | | | |
| | | | | | | | | |
| Traffic (D1,S1,Seq.no(1))| | | | Traffic (D1,S1,Seq. no(1))| | |
|<----------------------------| | | |<-----------------------------| | |
| | | | | | | | | |
| Traffic (D1,S1,Seq.no(2))| | | | Traffic (D1,S1,Seq. no(2))| | |
|<----------------------------| | | |<-----------------------------| | |
| | | | | | | | | |
| Traffic (D1,S1,Seq.no(x))| | | | Traffic (D1,S1,Seq. no(x))| | |
|<----------------------------| | | |<-----------------------------| | |
| | | | | | | | | |
| |Traffic (S1,D1,| | | | | Traffic (S1,D1,| | |
| | Seq.no(x))| | | | | Seq. no(x))| | |
| |<--------------| | | | |<---------------| | |
| | | | | | | | | |
| | | | | | | | | |
| | | | <Bring down | | | | | <Bring down |
| | | | the switch in| | | | | the switch in|
| | | |active traffic| | | | | the active|
| | | | path> | | | | | traffic path>|
| | | | | | | | | |
| | |PORT_STATUS(Sa) | | | | |PORT_STATUS(Sa) | |
| | |---------------->| | | | |--------------------->| |
| | | | | | | | | |
| |Traffic (S1,D1,| | | | | Traffic (S1,D1,| | |
| | Seq.no(n-1))| | | | | Seq. no(n - 1))| | |
| | X<-----------| | | | | X<------------| | |
| | | | | | | | | |
| Traffic (D1,S1,Seq.no(n-1))| | | |Traffic (D1,S1,Seq. no(n - 1))| | |
| X------------------------| | | | X<------------------------| | |
| | | | | | | | | |
| | | | | | | | | |
| | |FLOW_MOD(D1) | | | | |FLOW_MOD(D1) | |
| | |<----------------| | | | |<---------------------| |
| | | | | | | | | |
| | |FLOW_MOD(S1) | | | | |FLOW_MOD(S1) | |
| | |<----------------| | | | |<---------------------| |
| | | | | | | | | |
| Traffic (D1,S1,Seq.no(n))| | | | Traffic (D1,S1,Seq. no(n))| | |
|<----------------------------| | | |<-----------------------------| | |
| | | | | | | | | |
| |Traffic (S1,D1,| | | | | Traffic (S1,D1,| | |
| | Seq.no(n))| | | | | Seq. no(n))| | |
| |<--------------| | | | |<---------------| | |
| | | | | | | | | |
| | | |<Stop the test| | | | |<Stop the test|
| | | | after recv. | | | | | after recv. |
| | | | traffic upon| | | | | traffic upon|
| | | | failover> | | | | | failover> |
Legend:
Legend:
G-ARP: Gratuitous ARP message. G-ARP: Gratuitous ARP message
Seq.no: Sequence number. Seq. no: Sequence number
Sa: Neighbor switch of the switch that was brought down. Sa: Neighbor switch of the switch that was brought down
Discussion: Discussion:
The time difference between the last valid frame received before the The time difference between the last valid frame received before
traffic loss (Packet number with sequence number x) and the first the traffic loss (packet with sequence number x) and the first
frame received after the traffic loss (packet with sequence number frame received after the traffic loss (packet with sequence
n) will provide the network path re-provisioning time. number n) will provide the Network Re-provisioning Time.
Note that the trial is valid only when the controller provisions the Note that the trial is valid only when the controller provisions
alternate path upon network failure. the alternate path upon network failure.
Acknowledgments
The authors would like to thank the following individuals for
providing their valuable comments regarding the earlier draft
versions of this document: Al Morton (AT&T), Sandeep Gangadharan
(HP), M. Georgescu (NAIST), Andrew McGregor (Google), Scott Bradner,
Jay Karthik (Cisco), Ramki Krishnan (VMware), Boris Khasanov
(Huawei), and Brian Castelli (Spirent).
Authors' Addresses Authors' Addresses
Bhuvaneswaran Vengainathan Bhuvaneswaran Vengainathan
Veryx Technologies Inc. Veryx Technologies Inc.
1 International Plaza, Suite 550 1 International Plaza, Suite 550
Philadelphia Philadelphia, PA 19113
PA 19113 United States of America
Email: bhuvaneswaran.vengainathan@veryxtech.com Email: bhuvaneswaran.vengainathan@veryxtech.com
Anton Basil Anton Basil
Veryx Technologies Inc. Veryx Technologies Inc.
1 International Plaza, Suite 550 1 International Plaza, Suite 550
Philadelphia Philadelphia, PA 19113
PA 19113 United States of America
Email: anton.basil@veryxtech.com Email: anton.basil@veryxtech.com
Mark Tassinari Mark Tassinari
Hewlett-Packard, Hewlett Packard Enterprise
8000 Foothills Blvd, 8000 Foothills Blvd.
Roseville, CA 95747 Roseville, CA 95747
United States of America
Email: mark.tassinari@hpe.com Email: mark.tassinari@hpe.com
Vishwas Manral Vishwas Manral
Nano Sec, NanoSec Co
CA 3350 Thomas Rd.
Santa Clara, CA 95054
United States of America
Email: vishwas.manral@gmail.com Email: vishwas.manral@gmail.com
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
930 De Guigne Drive, 930 De Guigne Drive
Sunnyvale, CA Sunnyvale, CA 94085
United States of America
Email: sbanks@encrypted.net Email: sbanks@encrypted.net
 End of changes. 501 change blocks. 
1673 lines changed or deleted 1840 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/