draft-ietf-bmwg-sdn-controller-benchmark-meth-02.txt   draft-ietf-bmwg-sdn-controller-benchmark-meth-03.txt 
Internet-Draft Bhuvaneswaran Vengainathan Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies Intended Status: Informational Veryx Technologies
Expires: January 8, 2017 Mark Tassinari Expires: June 8, 2017 Mark Tassinari
Hewlett-Packard Hewlett-Packard
Vishwas Manral Vishwas Manral
Nano Sec Nano Sec
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
July 8, 2016 January 8, 2017
Benchmarking Methodology for SDN Controller Performance Benchmarking Methodology for SDN Controller Performance
draft-ietf-bmwg-sdn-controller-benchmark-meth-02 draft-ietf-bmwg-sdn-controller-benchmark-meth-03
Abstract Abstract
This document defines the methodologies for benchmarking control This document defines the methodologies for benchmarking control
plane performance of SDN controllers. Terminology related to plane performance of SDN controllers. Terminology related to
benchmarking SDN controllers is described in the companion benchmarking SDN controllers is described in the companion
terminology document. SDN controllers have been implemented with terminology document. SDN controllers have been implemented with
many varying designs in order to achieve their intended network many varying designs in order to achieve their intended network
functionality. Hence, the authors have taken the approach of functionality. Hence, the authors have taken the approach of
considering an SDN controller as a black box, defining the considering an SDN controller as a black box, defining the
skipping to change at page 1, line 45 skipping to change at page 1, line 45
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current. Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as reference at any time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress. material or to cite them other than as "work in progress.
This Internet-Draft will expire on January 8, 2017. This Internet-Draft will expire on June 8, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License. warranty as described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction ................................................ 3 1. Introduction...................................................4
2. Scope ....................................................... 4 2. Scope..........................................................4
3. Test Setup .................................................. 4 3. Test Setup.....................................................4
3.1. Test setup - Controller working in Standalone Mode....... 5 3.1. Test setup - Controller working in Standalone Mode........5
3.2. Test setup - Controller working in Cluster Mode.......... 6 3.2. Test setup - Controller working in Cluster Mode...........6
4. Test Considerations ......................................... 7 4. Test Considerations............................................7
4.1. Network Topology ........................................ 7 4.1. Network Topology..........................................7
4.2. Test Traffic ........................................... 7 4.2. Test Traffic..............................................7
4.3. Connection Setup ........................................ 7 4.3. Test Emulator Requirements................................7
4.4. Measurement Point Specification and Recommendation....... 8 4.4. Connection Setup..........................................7
4.5. Connectivity Recommendation ............................. 8 4.5. Measurement Point Specification and Recommendation........8
4.6. Test Repeatability ...................................... 8 4.6. Connectivity Recommendation...............................8
5. Benchmarking Tests .......................................... 9 4.7. Test Repeatability........................................8
5.1. Performance ............................................ 9 5. Benchmarking Tests.............................................9
5.1.1. Network Topology Discovery Time .................... 9 5.1. Performance...............................................9
5.1.2. Asynchronous Message Processing Time .............. 11 5.1.1. Network Topology Discovery Time......................9
5.1.3. Asynchronous Message Processing Rate .............. 12 5.1.2. Asynchronous Message Processing Time................11
5.1.4. Reactive Path Provisioning Time ................... 14 5.1.3. Asynchronous Message Processing Rate................12
5.1.5. Proactive Path Provisioning Time .................. 15 5.1.4. Reactive Path Provisioning Time.....................14
5.1.6. Reactive Path Provisioning Rate ................... 16 5.1.5. Proactive Path Provisioning Time....................15
5.1.7. Proactive Path Provisioning Rate .................. 18 5.1.6. Reactive Path Provisioning Rate.....................17
5.1.8. Network Topology Change Detection Time ............ 19 5.1.7. Proactive Path Provisioning Rate....................18
5.2. 6.2 Scalability ........................................ 20 5.1.8. Network Topology Change Detection Time..............20
5.2.1. Control Session Capacity .......................... 20 5.2. Scalability..............................................21
5.2.2. Network Discovery Size ............................ 21 5.2.1. Control Session Capacity............................21
5.2.3. 6.2.3 Forwarding Table Capacity ................... 22 5.2.2. Network Discovery Size..............................22
5.3. 6.3 Security .......................................... 24 5.2.3. Forwarding Table Capacity...........................23
5.3.1. 6.3.1 Exception Handling .......................... 24 5.3. Security.................................................24
5.3.2. Denial of Service Handling ........................ 25 5.3.1. Exception Handling..................................24
5.3.2. Denial of Service Handling..........................26
5.4. Reliability ........................................... 26 5.4. Reliability..............................................27
5.4.1. Controller Failover Time .......................... 26 5.4.1. Controller Failover Time............................27
5.4.2. Network Re-Provisioning Time ...................... 28 5.4.2. Network Re-Provisioning Time........................28
6. References ................................................. 29 6. References....................................................30
6.1. Normative References ................................... 29 6.1. Normative References.....................................30
6.2. Informative References ................................. 30 6.2. Informative References...................................31
7. IANA Considerations ........................................ 30 7. IANA Considerations...........................................31
8. Security Considerations ..................................... 30 8. Security Considerations.......................................31
9. Acknowledgments ............................................ 31 9. Acknowledgments...............................................31
Appendix A. Example Test Topologies ............................ 32 Appendix A. Example Test Topologies..............................33
A.1. Leaf-Spine Topology - Three Tier Network Architecture... 32 A.1. Leaf-Spine Topology - Three Tier Network Architecture....33
A.2. Leaf-Spine Topology - Two Tier Network Architecture..... 32 A.2. Leaf-Spine Topology - Two Tier Network Architecture......33
Appendix B. Benchmarking Methodology using OpenFlow Controllers. 33 Appendix B. Benchmarking Methodology using OpenFlow Controllers..34
B.1. Protocol Overview ...................................... 33 B.1. Protocol Overview........................................34
B.2. Messages Overview ...................................... 33 B.2. Messages Overview........................................34
B.3. Connection Overview .................................... 33 B.3. Connection Overview......................................34
B.4. Performance Benchmarking Tests ......................... 34 B.4. Performance Benchmarking Tests...........................35
B.4.1. Network Topology Discovery Time ................... 34 B.4.1. Network Topology Discovery Time.....................35
B.4.2. Asynchronous Message Processing Time .............. 35 B.4.2. Asynchronous Message Processing Time................36
B.4.3. Asynchronous Message Processing Rate .............. 36 B.4.3. Asynchronous Message Processing Rate................37
B.4.4. Reactive Path Provisioning Time ................... 37 B.4.4. Reactive Path Provisioning Time.....................38
B.4.5. Proactive Path Provisioning Time .................. 38 B.4.5. Proactive Path Provisioning Time....................39
B.4.6. Reactive Path Provisioning Rate ................... 39 B.4.6. Reactive Path Provisioning Rate.....................40
B.4.7. Proactive Path Provisioning Rate .................. 40 B.4.7. Proactive Path Provisioning Rate....................41
B.4.8. Network Topology Change Detection Time ............ 41 B.4.8. Network Topology Change Detection Time..............42
B.5. Scalability ........................................... 42 B.5. Scalability..............................................43
B.5.1. Control Sessions Capacity ......................... 42 B.5.1. Control Sessions Capacity...........................43
B.5.2. Network Discovery Size ............................ 42 B.5.2. Network Discovery Size..............................43
B.5.3. Forwarding Table Capacity ......................... 43 B.5.3. Forwarding Table Capacity...........................44
B.6. Security .............................................. 45 B.6. Security.................................................46
B.6.1. Exception Handling ................................ 45 B.6.1. Exception Handling..................................46
B.6.2. Denial of Service Handling ........................ 46 B.6.2. Denial of Service Handling..........................47
B.7. Reliability ........................................... 48 B.7. Reliability..............................................49
B.7.1. Controller Failover Time .......................... 48 B.7.1. Controller Failover Time............................49
B.7.2. Network Re-Provisioning Time ...................... 49 B.7.2. Network Re-Provisioning Time........................50
Authors' Addresses ............................................ 52 Authors' Addresses...............................................53
1. Introduction 1. Introduction
This document provides generic methodologies for benchmarking SDN This document provides generic methodologies for benchmarking SDN
controller performance. An SDN controller may support many controller performance. An SDN controller may support many
northbound and southbound protocols, implement a wide range of northbound and southbound protocols, implement a wide range of
applications, and work solely, or as a group to achieve the desired applications, and work solely, or as a group to achieve the desired
functionality. This document considers an SDN controller as a black functionality. This document considers an SDN controller as a black
box, regardless of design and implementation. The tests defined in box, regardless of design and implementation. The tests defined in
the document can be used to benchmark SDN controller for the document can be used to benchmark SDN controller for
skipping to change at page 4, line 18 skipping to change at page 4, line 29
SDN controllers performance. SDN controllers performance.
Conventions used in this document Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119. document are to be interpreted as described in RFC 2119.
2. Scope 2. Scope
This document defines methodology to measure the networking metrics 3. This document defines methodology to measure the networking metrics
of SDN controllers. For the purpose of this memo, the SDN controller of SDN controllers. For the purpose of this memo, the SDN controller
is a function that manages and controls Network Devices. Any SDN is a function that manages and controls Network Devices. Any SDN
controller without a control capability is out of scope for this controller without a control capability is out of scope for this
memo. The tests defined in this document enable benchmarking of SDN memo. The tests defined in this document enable benchmarking of SDN
Controllers in two ways; as a standalone controller and as a cluster Controllers in two ways; as a standalone controller and as a cluster
of homogeneous controllers. These tests are recommended for of homogeneous controllers. These tests are recommended for
execution in lab environments rather than in live network execution in lab environments rather than in live network
deployments. Performance benchmarking of a federation of controllers deployments. Performance benchmarking of a federation of controllers
is beyond the scope of this document. is beyond the scope of this document. Test Setup
3. Test Setup
The tests defined in this document enable measurement of an SDN The tests defined in this document enable measurement of an SDN
controllers performance in standalone mode and cluster mode. This controllers performance in standalone mode and cluster mode. This
section defines common reference topologies that are later referred section defines common reference topologies that are later referred
to in individual tests. to in individual tests (Additional forwarding Plane topologies are
provided in Appendix A).
3.1. Test setup - Controller working in Standalone Mode 3.1. Test setup - Controller working in Standalone Mode
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| Application Plane Test Emulator | | Application Plane Test Emulator |
| | | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | Application | | Service | | | | Application | | Service | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | | |
+-----------------------------+(I2)-------------------------+ +-----------------------------+(I2)-------------------------+
| |
| |
| (Northbound interface) | (Northbound interfaces)
+-------------------------------+ +-------------------------------+
| +----------------+ | | +----------------+ |
| | SDN Controller | | | | SDN Controller | |
| +----------------+ | | +----------------+ |
| | | |
| Device Under Test (DUT) | | Device Under Test (DUT) |
+-------------------------------+ +-------------------------------+
| (Southbound interface) | (Southbound interfaces)
| |
| |
+-----------------------------+(I1)-------------------------+ +-----------------------------+(I1)-------------------------+
| | | |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| | Network |l1 ln-1| Network | | | | Network |l1 ln-1| Network | |
| | Device 1 |---- .... ----| Device n | | | | Device 1 |---- .... ----| Device n | |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| |l0 |ln | | |l0 |ln |
| | | | | | | |
skipping to change at page 6, line 17 skipping to change at page 6, line 17
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| Application Plane Test Emulator | | Application Plane Test Emulator |
| | | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | Application | | Service | | | | Application | | Service | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | | |
+-----------------------------+(I2)-------------------------+ +-----------------------------+(I2)-------------------------+
| |
| |
| (Northbound interface) | (Northbound interfaces)
+---------------------------------------------------------+ +---------------------------------------------------------+
| | | |
| ------------------ ------------------ | | ------------------ ------------------ |
| | SDN Controller 1 | <--E/W--> | SDN Controller n | | | | SDN Controller 1 | <--E/W--> | SDN Controller n | |
| ------------------ ------------------ | | ------------------ ------------------ |
| | | |
| Device Under Test (DUT) | | Device Under Test (DUT) |
+---------------------------------------------------------+ +---------------------------------------------------------+
| (Southbound interface) | (Southbound interfaces)
| |
| |
+-----------------------------+(I1)-------------------------+ +-----------------------------+(I1)-------------------------+
| | | |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| | Network |l1 ln-1| Network | | | | Network |l1 ln-1| Network | |
| | Device 1 |---- .... ----| Device n | | | | Device 1 |---- .... ----| Device n | |
| +-----------+ +-----------+ | | +-----------+ +-----------+ |
| |l0 |ln | | |l0 |ln |
| | | | | | | |
skipping to change at page 7, line 18 skipping to change at page 7, line 18
The test cases SHOULD use Leaf-Spine topology with at least 1 The test cases SHOULD use Leaf-Spine topology with at least 1
Network Device in the topology for benchmarking. The test traffic Network Device in the topology for benchmarking. The test traffic
generators TP1 and TP2 SHOULD be connected to the first and the last generators TP1 and TP2 SHOULD be connected to the first and the last
leaf Network Device. If a test case uses test topology with 1 leaf Network Device. If a test case uses test topology with 1
Network Device, the test traffic generators TP1 and TP2 SHOULD be Network Device, the test traffic generators TP1 and TP2 SHOULD be
connected to the same node. However to achieve a complete connected to the same node. However to achieve a complete
performance characterization of the SDN controller, it is performance characterization of the SDN controller, it is
recommended that the controller be benchmarked for many network recommended that the controller be benchmarked for many network
topologies and a varying number of Network Devices. This document topologies and a varying number of Network Devices. This document
includes a few sample test topologies, defined in Section 10 - includes two sample test topologies, defined in Section 10 -
Appendix A for reference. Further, care should be taken to make sure Appendix A for reference. Further, care should be taken to make sure
that a loop prevention mechanism is enabled either in the SDN that a loop prevention mechanism is enabled either in the SDN
controller, or in the network when the topology contains redundant controller, or in the network when the topology contains redundant
network paths. network paths.
4.2. Test Traffic 4.2. Test Traffic
Test traffic is used to notify the controller about the arrival of Test traffic is used to notify the controller about the asynchronous
new flows. The test cases SHOULD use multiple frame sizes as arrival of new flows. The test cases SHOULD use frame sizes of 128,
recommended in RFC2544 for benchmarking. 512 and 1508 bytes for benchmarking. Testing using jumbo frames are
optional.
4.3. Test Emulator Requirements 4.3. Test Emulator Requirements
The Test Emulator SHOULD time stamp the transmitted and received The Test Emulator SHOULD time stamp the transmitted and received
control messages to/from the controller on the established network control messages to/from the controller on the established network
connections. The test cases use these values to compute the connections. The test cases use these values to compute the
controller processing time. controller processing time.
4.4. Connection Setup 4.4. Connection Setup
There may be controller implementations that support unencrypted and There may be controller implementations that support unencrypted and
encrypted network connections with Network Devices. Further, the encrypted network connections with Network Devices. Further, the
controller may have backward compatibility with Network Devices controller may have backward compatibility with Network Devices
running older versions of southbound protocols. It is recommended running older versions of southbound protocols. It may be useful to
that the controller performance be measured with one or more measure the controller performance be measured with one or more
applicable connection setup methods defined below. applicable connection setup methods defined below.
1.Unencrypted connection with Network Devices, running same 1. Unencrypted connection with Network Devices, running same
protocol version. protocol version.
2.Unencrypted connection with Network Devices, running different 2. Unencrypted connection with Network Devices, running different
protocol versions. protocol versions.
Example: Example:
a.Controller running current protocol version and switch a. Controller running current protocol version and switch
running older protocol version running older protocol version
b.Controller running older protocol version and switch b. Controller running older protocol version and switch
running current protocol version running current protocol version
3.Encrypted connection with Network Devices, running same 3. Encrypted connection with Network Devices, running same
protocol version protocol version
4.Encrypted connection with Network Devices, running different 4. Encrypted connection with Network Devices, running different
protocol versions. protocol versions.
Example: Example:
a.Controller running current protocol version and switch a. Controller running current protocol version and switch
running older protocol version running older protocol version
b.Controller running older protocol version and switch b. Controller running older protocol version and switch
running current protocol version running current protocol version
4.5. Measurement Point Specification and Recommendation 4.5. Measurement Point Specification and Recommendation
The measurement accuracy depends on several factors including the The measurement accuracy depends on several factors including the
point of observation where the indications are captured. For point of observation where the indications are captured. For
example, the notification can be observed at the controller or test example, the notification can be observed at the controller or test
emulator. The test operator SHOULD make the observations/ emulator. The test operator SHOULD make the observations/
measurements at the interfaces of test emulator unless it is measurements at the interfaces of test emulator unless it is
explicitly mentioned otherwise in the individual test. explicitly mentioned otherwise in the individual test. In any case,
the locations of measurement points MUST be reported.
4.6. Connectivity Recommendation 4.6. Connectivity Recommendation
The SDN controller in the test setup SHOULD be connected directly The SDN controller in the test setup SHOULD be connected directly
with the forwarding and the management plane test emulators to avoid with the forwarding and the management plane test emulators to avoid
any delays or failure introduced by the intermediate devices during any delays or failure introduced by the intermediate devices during
benchmarking tests. benchmarking tests. When the controller is implemented as a virtual
machine, details of the physical and logical connectivity MUST be
reported.
4.7. Test Repeatability 4.7. Test Repeatability
To increase the confidence in measured result, it is recommended To increase the confidence in measured result, it is recommended
that each test SHOULD be repeated a minimum of 10 times. that each test SHOULD be repeated a minimum of 10 times.
Test Reporting Test Reporting
Each test has a reporting format that contains some global and Each test has a reporting format that contains some global and
identical reporting components, and some individual components that identical reporting components, and some individual components that
are specific to individual tests. The following test configuration are specific to individual tests. The following test configuration
parameters and controller settings parameters MUST be reflected in parameters and controller settings parameters MUST be reflected in
the test report. the test report.
Test Configuration Parameters: Test Configuration Parameters:
1.Controller name and version 1. Controller name and version
2.Northbound protocols and versions 2. Northbound protocols and versions
3.Southbound protocols and versions 3. Southbound protocols and versions
4.Controller redundancy mode (Standalone or Cluster Mode) 4. Controller redundancy mode (Standalone or Cluster Mode)
5.Connection setup (Unencrypted or Encrypted) 5. Connection setup (Unencrypted or Encrypted)
6.Network Topology (Mesh or Tree or Linear) 6. Network Topology (Mesh or Tree or Linear)
7.Network Device Type (Physical or Virtual or Emulated) 7. Network Device Type (Physical or Virtual or Emulated)
8.Number of Nodes 8. Number of Nodes
9.Number of Links 9. Number of Links
10.Test Traffic Type 10. Dataplane Test Traffic Type
11.Controller System Configuration (e.g., CPU, Memory, Operating 11. Controller System Configuration (e.g., Physical or Virtual
System, Interface Speed etc.,) Machine, CPU, Memory, Caches, Operating System, Interface
12.Reference Test Setup (e.g., Section 3.1 etc.,) Speed, Storage)
12. Reference Test Setup (e.g., Section 3.1 etc.,)
Controller Settings Parameters: Controller Settings Parameters:
1.Topology re-discovery timeout 1. Topology re-discovery timeout
2.Controller redundancy mode (e.g., active-standby etc.,) 2. Controller redundancy mode (e.g., active-standby etc.,)
3. Controller state persistence enabled/disabled
To ensure the repeatability of test, the following capabilities of To ensure the repeatability of test, the following capabilities of
test emulator SHOULD be reported test emulator SHOULD be reported
1.Maximum number of Network Devices that the forwarding plane 1. Maximum number of Network Devices that the forwarding plane
emulates emulates
2.Control message processing time (e.g., Topology Discovery 2. Control message processing time (e.g., Topology Discovery
Messages) Messages)
One way to determine the above two values are to simulate the One way to determine the above two values are to simulate the
required control sessions and messages from the control plane. required control sessions and messages from the control plane.
5. Benchmarking Tests 5. Benchmarking Tests
5.1. Performance 5.1. Performance
5.1.1. Network Topology Discovery Time 5.1.1. Network Topology Discovery Time
skipping to change at page 9, line 47 skipping to change at page 10, line 8
Objective: Objective:
The time taken by controller(s) to determine the complete network The time taken by controller(s) to determine the complete network
topology, defined as the interval starting with the first discovery topology, defined as the interval starting with the first discovery
message from the controller(s) at its Southbound interface, ending message from the controller(s) at its Southbound interface, ending
with all features of the static topology determined. with all features of the static topology determined.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST support network discovery. 1. The controller MUST support network discovery.
2. Tester should be able to retrieve the discovered topology 2. Tester should be able to retrieve the discovered topology
information either through the controller's management interface, information either through the controller's management interface,
or northbound interface to determine if the discovery was or northbound interface to determine if the discovery was
successful and complete. successful and complete.
3. Ensure that the controller's topology re-discovery timeout has 3. Ensure that the controller's topology re-discovery timeout has
been set to the maximum value to avoid initiation of re-discovery been set to the maximum value to avoid initiation of re-discovery
process in the middle of the test. process in the middle of the test.
Procedure: Procedure:
1. Ensure that the controller is operational, its network 1. Ensure that the controller is operational, its network
applications, northbound and southbound interfaces are up and applications, northbound and southbound interfaces are up and
running. running.
2. Establish the network connections between controller and Network 2. Establish the network connections between controller and Network
Devices. Devices.
3. Record the time for the first discovery message (Tm1) received 3. Record the time for the first discovery message (Tm1) received
from the controller at forwarding plane test emulator interface from the controller at forwarding plane test emulator interface
I1. I1.
4. Query the controller every 3 seconds to obtain the discovered 4. Query the controller every 3 seconds to obtain the discovered
network topology information through the northbound interface or network topology information through the northbound interface or
the management interface and compare it with the deployed network the management interface and compare it with the deployed network
topology information. topology information.
5. Stop the test when the discovered topology information matches the 5. Stop the test when the discovered topology information matches the
deployed network topology, or when the discovered topology deployed network topology, or when the discovered topology
information for 3 consecutive queries return the same details. information for 3 consecutive queries return the same details.
6. Record the time last discovery message (Tmn) sent to controller 6. Record the time last discovery message (Tmn) sent to controller
from the forwarding plane test emulator interface (I1) when the from the forwarding plane test emulator interface (I1) when the
test completed successfully. (e.g., the topology matches). test completed successfully. (e.g., the topology matches).
Measurement: Measurement:
Topology Discovery Time Tr1 = Tmn-Tm1. Topology Discovery Time Tr1 = Tmn-Tm1.
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Topology Discovery Time = ----------------------- Average Topology Discovery Time = -----------------------
Total Test Iterations Total Test Iterations
Reporting Format: Reporting Format:
skipping to change at page 11, line 28 skipping to change at page 11, line 34
The time taken by controller(s) to process an asynchronous message, The time taken by controller(s) to process an asynchronous message,
defined as the interval starting with an asynchronous message from a defined as the interval starting with an asynchronous message from a
network device after the discovery of all the devices by the network device after the discovery of all the devices by the
controller(s), ending with a response message from the controller(s) controller(s), ending with a response message from the controller(s)
at its Southbound interface. at its Southbound interface.
Reference Test Setup: Reference Test Setup:
This test SHOULD use one of the test setup described in section 3.1 This test SHOULD use one of the test setup described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1.The controller MUST have completed the network topology discovery 1. The controller MUST have successfully completed the network
for the connected Network Devices. topology discovery for the connected Network Devices.
Procedure: Procedure:
1.Generate asynchronous messages from every connected Network 1. Generate asynchronous messages from every connected Network
Device, to the SDN controller, one at a time in series from the Device, to the SDN controller, one at a time in series from the
forwarding plane test emulator for the test duration. forwarding plane test emulator for the test duration.
2.Record every request transmit (T1) timestamp and the 2. Record every request transmit (T1) timestamp and the
corresponding response (R1) received timestamp at the corresponding response (R1) received timestamp at the
forwarding plane test emulator interface (I1) for every forwarding plane test emulator interface (I1) for every
successful message exchange. successful message exchange.
Measurement: Measurement:
(R1-T1) + (R2-T2)..(Rn-Tn) (R1-T1) + (R2-T2)..(Rn-Tn)
Asynchronous Message Processing Time Tr1 = ----------------------- Asynchronous Message Processing Time Tr1 = -----------------------
Nrx Nrx
skipping to change at page 12, line 35 skipping to change at page 12, line 44
topologies, the results SHOULD be reported in the form of a graph. topologies, the results SHOULD be reported in the form of a graph.
The X coordinate SHOULD be the Topology Type, the Y coordinate The X coordinate SHOULD be the Topology Type, the Y coordinate
SHOULD be the average Asynchronous Message Processing Time. SHOULD be the average Asynchronous Message Processing Time.
5.1.3. Asynchronous Message Processing Rate 5.1.3. Asynchronous Message Processing Rate
Objective: Objective:
The maximum number of asynchronous messages (session aliveness check The maximum number of asynchronous messages (session aliveness check
message, new flow arrival notification message etc.) that the message, new flow arrival notification message etc.) that the
controller(s) can process, defined as the number of asynchronous controller(s) can process, defined as the iteration starting with
messages the controller(s) can process at its Southbound interface sending asynchronous messages to the controller (s) at the maximum
between the start of the test and the expiry of given test duration possible rate and ending with an iteration that the controller(s)
processes the received asynchronous messages without dropping.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST have completed the network topology discovery 1. The controller MUST have successfully completed the network
for the connected Network Devices. topology discovery for the connected Network Devices.
Procedure: Procedure:
1. Generate asynchronous messages continuously at the maximum 1. Generate asynchronous messages continuously at the maximum
possible rate on the established connections from all the possible rate on the established connections from all the
connected Network Devices in the forwarding plane test emulator emulated/simulated Network Devices for the given Test Duration
for the Test Duration (Td). (Td).
2. Record the total number of responses received from the controller 2. Record the total number of responses received from the controller
(Nrx) as well as the number of messages sent(Ntx) to the (Nrx1) as well as the number of messages sent (Ntx1) to the
controller within the test duration(Td) at the forwarding plane controller within the test duration(Td).
test emulator interface (I1). 3. Repeat the test by generating the asynchronous messages equal to
the number of responses received from the controller in last
iteration for the given test duration (Td).
4. Test MUST be repeated until the generated asynchronous messages
and the responses received from the controller are equal for two
consecutive iterations.
5. Record the number of responses received from the controller (Nrxn)
as well as the number of messages sent(Ntxn) to the controller in
the last test iteration.
Measurement: Measurement:
Nrx Nrxn
Asynchronous Message Processing Rate Tr1 = ----- Asynchronous Message Processing Rate Tr1 = -----
Td Td
Tr1 + Tr2 + Tr3..Trn Tr1 + Tr2 + Tr3..Trn
Average Asynchronous Message Processing Rate= -------------------- Average Asynchronous Message Processing Rate= --------------------
Total Test Iterations Total Test Iterations
Loss Ratio = (Ntx-Nrx)/100.
Reporting Format: Reporting Format:
The Asynchronous Message Processing Rate results MUST be reported in The Asynchronous Message Processing Rate results MUST be reported in
the format of a table with a row for each iteration. The last row of the format of a table with a row for each iteration. The last row of
the table indicates the average Asynchronous Message Processing the table indicates the average Asynchronous Message Processing
Rate. Rate.
The report should capture the following information in addition to The report should capture the following information in addition to
the configuration parameters captured in section 5. the configuration parameters captured in section 5.
skipping to change at page 14, line 12 skipping to change at page 14, line 29
The X coordinate SHOULD be the Topology Type, the Y coordinate The X coordinate SHOULD be the Topology Type, the Y coordinate
SHOULD be the average Asynchronous Message Processing Rate. SHOULD be the average Asynchronous Message Processing Rate.
5.1.4. Reactive Path Provisioning Time 5.1.4. Reactive Path Provisioning Time
Objective: Objective:
The time taken by the controller to setup a path reactively between The time taken by the controller to setup a path reactively between
source and destination node, defined as the interval starting with source and destination node, defined as the interval starting with
the first flow provisioning request message received by the the first flow provisioning request message received by the
controller(s), ending with the last flow provisioning response controller(s) at its Southbound interface, ending with the last flow
message sent from the controller(s) at it Southbound interface. provisioning response message sent from the controller(s) at its
Southbound interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A. The
number of Network Devices in the path is a parameter of the test
that may be varied from 2 to maximum discovery size in repetitions
of this test.
Prerequisite: Prerequisite:
1. The controller MUST contain the network topology information for 1. The controller MUST contain the network topology information for
the deployed network topology. the deployed network topology.
2. The controller should have the knowledge about the location of 2. The controller should have the knowledge about the location of
destination endpoint for which the path has to be provisioned. destination endpoint for which the path has to be provisioned.
This can be achieved through dynamic learning or static This can be achieved through dynamic learning or static
provisioning. provisioning.
3. Ensure that the default action for 'flow miss' in Network Device 3. Ensure that the default action for 'flow miss' in Network Device
is configured to 'send to controller'. is configured to 'send to controller'.
4. Ensure that each Network Device in a path requires the controller 4. Ensure that each Network Device in a path requires the controller
to make the forwarding decision while paving the entire path. to make the forwarding decision while paving the entire path.
Procedure: Procedure:
1. Send a single traffic stream from the test traffic generator TP1 1. Send a single traffic stream from the test traffic generator TP1
to test traffic generator TP2. to test traffic generator TP2.
2. Record the time of the first flow provisioning request message 2. Record the time of the first flow provisioning request message
sent to the controller (Tsf1) from the Network Device at the sent to the controller (Tsf1) from the Network Device at the
forwarding plane test emulator interface (I1). forwarding plane test emulator interface (I1).
3. Wait for the arrival of first traffic frame at the Traffic 3. Wait for the arrival of first traffic frame at the Traffic
Endpoint TP2 or the expiry of test duration (Td). Endpoint TP2 or the expiry of test duration (Td).
4. Record the time of the last flow provisioning response message 4. Record the time of the last flow provisioning response message
received from the controller (Tdf1) to the Network Device at the received from the controller (Tdf1) to the Network Device at the
forwarding plane test emulator interface (I1). forwarding plane test emulator interface (I1).
Measurement: Measurement:
Reactive Path Provisioning Time Tr1 = Tdf1-Tsf1. Reactive Path Provisioning Time Tr1 = Tdf1-Tsf1.
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Reactive Path Provisioning Time = ----------------------- Average Reactive Path Provisioning Time = -----------------------
Total Test Iterations Total Test Iterations
Reporting Format: Reporting Format:
The Reactive Path Provisioning Time results MUST be reported in the The Reactive Path Provisioning Time results MUST be reported in the
format of a table with a row for each iteration. The last row of the format of a table with a row for each iteration. The last row of the
table indicates the Average Reactive Path Provisioning Time table indicates the Average Reactive Path Provisioning Time
The report should capture the following information in addition to The report should capture the following information in addition to
the configuration parameters captured in section 5. the configuration parameters captured in section 5.
- Number of Network Devices in the path - Number of Network Devices in the path
5.1.5. Proactive Path Provisioning Time 5.1.5. Proactive Path Provisioning Time
Objective: Objective:
The time taken by the controller to setup a path proactively between The time taken by the controller to setup a path proactively between
source and destination node, defined as the interval starting with source and destination node, defined as the interval starting with
the first proactive flow provisioned in the controller(s) at its the first proactive flow provisioned in the controller(s) at its
Northbound interface, ending with the last flow provisioning Northbound interface, ending with the last flow provisioning
response message sent from the controller(s) at it Southbound response message sent from the controller(s) at it Southbound
interface. interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST contain the network topology information for 1. The controller MUST contain the network topology information for
the deployed network topology. the deployed network topology.
2. The controller should have the knowledge about the location of 2. The controller should have the knowledge about the location of
destination endpoint for which the path has to be provisioned. destination endpoint for which the path has to be provisioned.
This can be achieved through dynamic learning or static This can be achieved through dynamic learning or static
provisioning. provisioning.
3. Ensure that the default action for flow miss in Network Device is 3. Ensure that the default action for flow miss in Network Device is
'drop'. 'drop'.
Procedure: Procedure:
1. Send a single traffic stream from test traffic generator TP1 to 1. Send a single traffic stream from test traffic generator TP1 to
TP2. TP2.
2. Install the flow entries to reach from test traffic generator TP1 2. Install the flow entries to reach from test traffic generator TP1
to the test traffic generator TP2 through controller's northbound to the test traffic generator TP2 through controller's northbound
or management interface. or management interface.
3. Wait for the arrival of first traffic frame at the test traffic 3. Wait for the arrival of first traffic frame at the test traffic
generator TP2 or the expiry of test duration (Td). generator TP2 or the expiry of test duration (Td).
4. Record the time when the proactive flow is provisioned in the 4. Record the time when the proactive flow is provisioned in the
Controller (Tsf1) at the management plane test emulator interface Controller (Tsf1) at the management plane test emulator interface
I2. I2.
5. Record the time of the last flow provisioning message received 5. Record the time of the last flow provisioning message received
from the controller (Tdf1) at the forwarding plane test emulator from the controller (Tdf1) at the forwarding plane test emulator
interface I1. interface I1.
Measurement: Measurement:
Proactive Flow Provisioning Time Tr1 = Tdf1-Tsf1. Proactive Flow Provisioning Time Tr1 = Tdf1-Tsf1.
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Proactive Path Provisioning Time = ----------------------- Average Proactive Path Provisioning Time = -----------------------
Total Test Iterations Total Test Iterations
Reporting Format: Reporting Format:
skipping to change at page 17, line 8 skipping to change at page 17, line 30
The maximum number of independent paths a controller can The maximum number of independent paths a controller can
concurrently establish between source and destination nodes concurrently establish between source and destination nodes
reactively, defined as the number of paths provisioned by the reactively, defined as the number of paths provisioned by the
controller(s) at its Southbound interface for the flow provisioning controller(s) at its Southbound interface for the flow provisioning
requests received for path provisioning at its Southbound interface requests received for path provisioning at its Southbound interface
between the start of the test and the expiry of given test duration. between the start of the test and the expiry of given test duration.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST contain the network topology information for 1. The controller MUST contain the network topology information for
the deployed network topology. the deployed network topology.
2. The controller should have the knowledge about the location of 2. The controller should have the knowledge about the location of
destination addresses for which the paths have to be provisioned. destination addresses for which the paths have to be provisioned.
This can be achieved through dynamic learning or static This can be achieved through dynamic learning or static
provisioning. provisioning.
3. Ensure that the default action for 'flow miss' in Network Device 3. Ensure that the default action for 'flow miss' in Network Device
is configured to 'send to controller'. is configured to 'send to controller'.
4. Ensure that each Network Device in a path requires the controller 4. Ensure that each Network Device in a path requires the controller
to make the forwarding decision while provisioning the entire to make the forwarding decision while provisioning the entire
path. path.
Procedure: Procedure:
1. Send traffic with unique source and destination addresses from 1. Send traffic with unique source and destination addresses from
test traffic generator TP1. test traffic generator TP1.
2. Record total number of unique traffic frames (Ndf) received at the 2. Record total number of unique traffic frames (Ndf) received at the
test traffic generator TP2 within the test duration (Td). test traffic generator TP2 within the test duration (Td).
Measurement: Measurement:
Ndf Ndf
Reactive Path Provisioning Rate Tr1 = ------ Reactive Path Provisioning Rate Tr1 = ------
Td Td
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Reactive Path Provisioning Rate = ------------------------ Average Reactive Path Provisioning Rate = ------------------------
Total Test Iterations Total Test Iterations
skipping to change at page 18, line 4 skipping to change at page 18, line 25
Reporting Format: Reporting Format:
The Reactive Path Provisioning Rate results MUST be reported in the The Reactive Path Provisioning Rate results MUST be reported in the
format of a table with a row for each iteration. The last row of the format of a table with a row for each iteration. The last row of the
table indicates the Average Reactive Path Provisioning Rate. table indicates the Average Reactive Path Provisioning Rate.
The report should capture the following information in addition to The report should capture the following information in addition to
the configuration parameters captured in section 5. the configuration parameters captured in section 5.
- Number of Network Devices in the path - Number of Network Devices in the path
- Offered rate - Offered rate
5.1.7. Proactive Path Provisioning Rate 5.1.7. Proactive Path Provisioning Rate
Objective: Objective:
Measure the maximum number of independent paths a controller can Measure the maximum rate of independent paths a controller can
concurrently establish between source and destination nodes concurrently establish between source and destination nodes
proactively, defined as the number of paths provisioned by the proactively, defined as the number of paths provisioned by the
controller(s) at its Southbound interface for the paths provisioned controller(s) at its Southbound interface for the paths requested in
in its Northbound interface between the start of the test and the its Northbound interface between the start of the test and the
expiry of given test duration . expiry of given test duration . The measurement is based on
dataplane observations of successful path activation
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST contain the network topology information for 1. The controller MUST contain the network topology information for
the deployed network topology. the deployed network topology.
2. The controller should have the knowledge about the location of 2. The controller should have the knowledge about the location of
destination addresses for which the paths have to be provisioned. destination addresses for which the paths have to be provisioned.
This can be achieved through dynamic learning or static This can be achieved through dynamic learning or static
provisioning. provisioning.
skipping to change at page 19, line 41 skipping to change at page 20, line 18
The amount of time required for the controller to detect any changes The amount of time required for the controller to detect any changes
in the network topology, defined as the interval starting with the in the network topology, defined as the interval starting with the
notification message received by the controller(s) at its Southbound notification message received by the controller(s) at its Southbound
interface, ending with the first topology rediscovery messages sent interface, ending with the first topology rediscovery messages sent
from the controller(s) at its Southbound interface. from the controller(s) at its Southbound interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST have discovered the network topology 1. The controller MUST have successfully discovered the network
information for the deployed network topology. topology information for the deployed network topology.
2. The periodic network discovery operation should be configured to 2. The periodic network discovery operation should be configured to
twice the Test duration (Td) value. twice the Test duration (Td) value.
Procedure: Procedure:
1. Trigger a topology change event by bringing down an active 1. Trigger a topology change event by bringing down an active
Network Device in the topology. Network Device in the topology.
2. Record the time when the first topology change notification is 2. Record the time when the first topology change notification is
skipping to change at page 20, line 36 skipping to change at page 21, line 11
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Network Topology Change Detection Time = ------------------ Average Network Topology Change Detection Time = ------------------
Total Test Iterations Total Test Iterations
Reporting Format: Reporting Format:
The Network Topology Change Detection Time results MUST be reported The Network Topology Change Detection Time results MUST be reported
in the format of a table with a row for each iteration. The last in the format of a table with a row for each iteration. The last
row of the table indicates the average Network Topology Change Time. row of the table indicates the average Network Topology Change Time.
5.2. 6.2 Scalability 5.2. Scalability
5.2.1. Control Session Capacity 5.2.1. Control Session Capacity
Objective: Objective:
Measure the maximum number of control sessions the controller can Measure the maximum number of control sessions the controller can
maintain, defined as the number of sessions that the controller can maintain, defined as the number of sessions that the controller can
accept from network devices, starting with the first control accept from network devices, starting with the first control
session, ending with the last control session that the controller(s) session, ending with the last control session that the controller(s)
accepts at its Southbound interface. accepts at its Southbound interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Procedure: Procedure:
1. Establish control connection with controller from every Network 1. Establish control connection with controller from every Network
Device emulated in the forwarding plane test emulator. Device emulated in the forwarding plane test emulator.
2. Stop the test when the controller starts dropping the control 2. Stop the test when the controller starts dropping the control
connection. connections.
3. Record the number of successful connections established with the 3. Record the number of successful connections established with the
controller (CCn) at the forwarding plane test emulator. controller (CCn) at the forwarding plane test emulator.
Measurement: Measurement:
Control Sessions Capacity = CCn. Control Sessions Capacity = CCn.
Reporting Format: Reporting Format:
The Control Session Capacity results MUST be reported in addition to The Control Session Capacity results MUST be reported in addition to
the configuration parameters captured in section 5. the configuration parameters captured in section 5.
skipping to change at page 21, line 41 skipping to change at page 22, line 18
Measure the network size (number of nodes, links and hosts) that a Measure the network size (number of nodes, links and hosts) that a
controller can discover, defined as the size of a network that the controller can discover, defined as the size of a network that the
controller(s) can discover, starting from a network topology given controller(s) can discover, starting from a network topology given
by the user for discovery, ending with the topology that the by the user for discovery, ending with the topology that the
controller(s) could successfully discover. controller(s) could successfully discover.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST support automatic network discovery. 1. The controller MUST support automatic network discovery.
2. Tester should be able to retrieve the discovered topology 2. Tester should be able to retrieve the discovered topology
information either through controller's management interface or information either through controller's management interface or
northbound interface. northbound interface.
Procedure: Procedure:
1. Establish the network connections between controller and network 1. Establish the network connections between controller and network
nodes. nodes.
2. Query the controller for the discovered network topology 2. Query the controller for the discovered network topology
information and compare it with the deployed network topology information and compare it with the deployed network topology
information. information.
3. 3a. Increase the number of nodes by 1 when the comparison is 3. 3a. Increase the number of nodes by 1 when the comparison is
successful and repeat the test. successful and repeat the test.
4. 3b. Decrease the number of nodes by 1 when the comparison fails 4. 3b. Decrease the number of nodes by 1 when the comparison fails
and repeat the test. and repeat the test.
5. Continue the test until the comparison of step 3b is successful. 5. Continue the test until the comparison of step 3b is successful.
6. Record the number of nodes for the last iteration (Ns) where the 6. Record the number of nodes for the last iteration (Ns) where the
topology comparison was successful. topology comparison was successful.
Measurement: Measurement:
Network Discovery Size = Ns. Network Discovery Size = Ns.
Reporting Format: Reporting Format:
The Network Discovery Size results MUST be reported in addition to The Network Discovery Size results MUST be reported in addition to
the configuration parameters captured in section 5. the configuration parameters captured in section 5.
5.2.3. 6.2.3 Forwarding Table Capacity 5.2.3. Forwarding Table Capacity
Objective: Objective:
Measure the maximum number of flow entries a controller can manage Measure the maximum number of flow entries a controller can manage
in its Forwarding table. in its Forwarding table.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller Forwarding table should be empty. 1. The controller Forwarding table should be empty.
2. Flow Idle time MUST be set to higher or infinite value. 2. Flow Idle time MUST be set to higher or infinite value.
3. The controller MUST have completed network topology discovery. 3. The controller MUST have successfully completed network topology
discovery.
4. Tester should be able to retrieve the forwarding table information 4. Tester should be able to retrieve the forwarding table information
either through controller's management interface or northbound either through controller's management interface or northbound
interface. interface.
Procedure: Procedure:
Reactive Flow Provisioning Mode: Reactive Flow Provisioning Mode:
1. Send bi-directional traffic continuously with unique source and/or 1. Send bi-directional traffic continuously with unique source and/or
destination addresses from test traffic generators TP1 and TP2 at destination addresses from test traffic generators TP1 and TP2 at
the asynchronous message processing rate of controller. the asynchronous message processing rate of controller.
2. Query the controller at a regular interval (e.g., 5 seconds) for 2. Query the controller at a regular interval (e.g., 5 seconds) for
the number of learnt flow entries from its northbound interface. the number of learnt flow entries from its northbound interface.
3. Stop the test when the retrieved value is constant for three 3. Stop the test when the retrieved value is constant for three
consecutive iterations and record the value received from the last consecutive iterations and record the value received from the last
query (Nrp). query (Nrp).
Proactive Flow Provisioning Mode: Proactive Flow Provisioning Mode:
1. Install unique flows continuously through controller's northbound 1. Install unique flows continuously through controller's northbound
or management interface until a failure response is received from or management interface until a failure response is received from
the controller. the controller.
2. Record the total number of successful responses (Nrp). 2. Record the total number of successful responses (Nrp).
Note: Note:
Some controller designs for proactive flow provisioning mode may Some controller designs for proactive flow provisioning mode may
require the switch to send flow setup requests in order to generate require the switch to send flow setup requests in order to generate
flow setup responses. In such cases, it is recommended to generate flow setup responses. In such cases, it is recommended to generate
bi-directional traffic for the provisioned flows. bi-directional traffic for the provisioned flows.
Measurement: Measurement:
skipping to change at page 24, line 5 skipping to change at page 24, line 27
Forwarding Table Capacity = Max Flow Entries. Forwarding Table Capacity = Max Flow Entries.
Reporting Format: Reporting Format:
The Forwarding Table Capacity results MUST be tabulated with the The Forwarding Table Capacity results MUST be tabulated with the
following information in addition to the configuration parameters following information in addition to the configuration parameters
captured in section 5. captured in section 5.
- Provisioning Type (Proactive/Reactive) - Provisioning Type (Proactive/Reactive)
5.3. 6.3 Security 5.3. Security
5.3.1. 6.3.1 Exception Handling 5.3.1. Exception Handling
Objective: Objective:
Determine the effect of handling error packets and notifications on Determine the effect of handling error packets and notifications on
performance tests. The impact MUST be measured for the following performance tests. The impact MUST be measured for the following
performance tests performance tests
a. Path Provisioning Rate a. Path Provisioning Rate
b. Path Provisioning Time b. Path Provisioning Time
c. Network Topology Change Detection Time c. Network Topology Change Detection Time
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. This test MUST be performed after obtaining the baseline 1. This test MUST be performed after obtaining the baseline
measurement results for the above performance tests. measurement results for the above performance tests.
2. Ensure that the invalid messages are not dropped by the 2. Ensure that the invalid messages are not dropped by the
intermediate devices connecting the controller and Network intermediate devices connecting the controller and Network
Devices. Devices.
Procedure: Procedure:
1. Perform the above listed performance tests and send 1% of messages 1. Perform the above listed performance tests and send 1% of messages
from the Asynchronous Message Processing Rate as invalid messages from the Asynchronous Message Processing Rate as invalid messages
from the connected Network Devices emulated at the forwarding from the connected Network Devices emulated at the forwarding
plane test emulator. plane test emulator.
2. Perform the above listed performance tests and send 2% of messages 2. Perform the above listed performance tests and send 2% of messages
from the Asynchronous Message Processing Rate as invalid messages from the Asynchronous Message Processing Rate as invalid messages
from the connected Network Devices emulated at the forwarding from the connected Network Devices emulated at the forwarding
plane test emulator. plane test emulator.
Note: Note:
Invalid messages can be frames with incorrect protocol fields or any Invalid messages can be frames with incorrect protocol fields or any
form of failure notifications sent towards controller. form of failure notifications sent towards controller.
Measurement: Measurement:
Measurement MUST be done as per the equation defined in the Measurement MUST be done as per the equation defined in the
corresponding performance test measurement section. corresponding performance test measurement section.
skipping to change at page 25, line 44 skipping to change at page 26, line 24
b. Path Provisioning Time b. Path Provisioning Time
c. Network Topology Change Detection Time c. Network Topology Change Detection Time
d. Network Discovery Size d. Network Discovery Size
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
This test MUST be performed after obtaining the baseline measurement This test MUST be performed after obtaining the baseline measurement
results for the above tests. results for the above tests.
Procedure: Procedure:
1. Perform the listed tests and launch a DoS attack towards 1. Perform the listed tests and launch a DoS attack towards
controller while the test is running. controller while the test is running.
Note: Note:
DoS attacks can be launched on one of the following interfaces. DoS attacks can be launched on one of the following interfaces.
a. Northbound (e.g., Sending a huge number of requests on a. Northbound (e.g., Sending a huge number of requests on
northbound interface) northbound interface)
b. Management (e.g., Ping requests to controller's management b. Management (e.g., Ping requests to controller's management
interface) interface)
c. Southbound (e.g., TCP SYNC messages on southbound interface) c. Southbound (e.g., TCP SYNC messages on southbound interface)
Measurement: Measurement:
Measurement MUST be done as per the equation defined in the Measurement MUST be done as per the equation defined in the
corresponding test's measurement section. corresponding test's measurement section.
Reporting Format: Reporting Format:
The DoS Attacks Handling results MUST be reported in the format of The DoS Attacks Handling results MUST be reported in the format of
table with a column for each of the below parameters and row for table with a column for each of the below parameters and row for
skipping to change at page 27, line 10 skipping to change at page 27, line 34
The time taken to switch from an active controller to the backup The time taken to switch from an active controller to the backup
controller, when the controllers work in redundancy mode and the controller, when the controllers work in redundancy mode and the
active controller fails, defined as the interval starting with the active controller fails, defined as the interval starting with the
active controller bringing down, ending with the first re-discovery active controller bringing down, ending with the first re-discovery
message received from the new controller at its Southbound message received from the new controller at its Southbound
interface. interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use the test setup described in section 3.2 of this The test SHOULD use the test setup described in section 3.2 of this
document. document in combination with Appendix A.
Prerequisite: Prerequisite:
1. Master controller election MUST be completed. 1. Master controller election MUST be completed.
2. Nodes are connected to the controller cluster as per the 2. Nodes are connected to the controller cluster as per the
Redundancy Mode (RM). Redundancy Mode (RM).
3. The controller cluster should have completed the network topology 3. The controller cluster should have successfully completed the
discovery. network topology discovery.
4. The Network Device MUST send all new flows to the controller when 4. The Network Device MUST send all new flows to the controller when
it receives from the test traffic generator. it receives from the test traffic generator.
5. Controller should have learnt the location of destination (D1) at 5. Controller should have learnt the location of destination (D1) at
test traffic generator TP2. test traffic generator TP2.
Procedure: Procedure:
1. Send uni-directional traffic continuously with incremental 1. Send uni-directional traffic continuously with incremental
sequence number and source addresses from test traffic generator sequence number and source addresses from test traffic generator
TP1 at the rate that the controller processes without any drops. TP1 at the rate that the controller processes without any drops.
2. Ensure that there are no packet drops observed at the test traffic 2. Ensure that there are no packet drops observed at the test traffic
generator TP2. generator TP2.
3. Bring down the active controller. 3. Bring down the active controller.
4. Stop the test when a first frame received on TP2 after failover 4. Stop the test when a first frame received on TP2 after failover
operation. operation.
5. Record the time at which the last valid frame received (T1) at 5. Record the time at which the last valid frame received (T1) at
test traffic generator TP2 before sequence error and the first test traffic generator TP2 before sequence error and the first
valid frame received (T2) after the sequence error at TP2 valid frame received (T2) after the sequence error at TP2
Measurement: Measurement:
Controller Failover Time = (T2 - T1) Controller Failover Time = (T2 - T1)
Packet Loss = Number of missing packet sequences. Packet Loss = Number of missing packet sequences.
Reporting Format: Reporting Format:
The Controller Failover Time results MUST be tabulated with the The Controller Failover Time results MUST be tabulated with the
skipping to change at page 28, line 4 skipping to change at page 28, line 31
Controller Failover Time = (T2 - T1) Controller Failover Time = (T2 - T1)
Packet Loss = Number of missing packet sequences. Packet Loss = Number of missing packet sequences.
Reporting Format: Reporting Format:
The Controller Failover Time results MUST be tabulated with the The Controller Failover Time results MUST be tabulated with the
following information. following information.
- Number of cluster nodes - Number of cluster nodes
- Redundancy mode - Redundancy mode
- Controller Failover - Controller Failover Time
- Time Packet Loss - Packet Loss
- Cluster keep-alive interval - Cluster keep-alive interval
5.4.2. Network Re-Provisioning Time 5.4.2. Network Re-Provisioning Time
Objective: Objective:
The time taken to re-route the traffic by the Controller, when there The time taken to re-route the traffic by the Controller, when there
is a failure in existing traffic paths, defined as the interval is a failure in existing traffic paths, defined as the interval
starting from the first failure notification message received by the starting from the first failure notification message received by the
controller, ending with the last flow re-provisioning message sent controller, ending with the last flow re-provisioning message sent
by the controller at its Southbound interface. by the controller at its Southbound interface.
Reference Test Setup: Reference Test Setup:
This test SHOULD use one of the test setup described in section 3.1 This test SHOULD use one of the test setup described in section 3.1
or section 3.2 of this document. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. Network with the given number of nodes and redundant paths MUST be 1. Network with the given number of nodes and redundant paths MUST be
deployed. deployed.
2. Ensure that the controller MUST have knowledge about the location 2. Ensure that the controller MUST have knowledge about the location
of test traffic generators TP1 and TP2. of test traffic generators TP1 and TP2.
3. Ensure that the controller does not pre-provision the alternate 3. Ensure that the controller does not pre-provision the alternate
path in the emulated Network Devices at the forwarding plane test path in the emulated Network Devices at the forwarding plane test
emulator. emulator.
Procedure: Procedure:
1. Send bi-directional traffic continuously with unique sequence 1. Send bi-directional traffic continuously with unique sequence
number from TP1 and TP2. number from TP1 and TP2.
2. Bring down a link or switch in the traffic path. 2. Bring down a link or switch in the traffic path.
3. Stop the test after receiving first frame after network re- 3. Stop the test after receiving first frame after network re-
convergence. convergence.
4. Record the time of last received frame prior to the frame loss at 4. Record the time of last received frame prior to the frame loss at
TP2 (TP2-Tlfr) and the time of first frame received after the TP2 (TP2-Tlfr) and the time of first frame received after the
frame loss at TP2 (TP2-Tffr). frame loss at TP2 (TP2-Tffr). There must be a gap in sequence
numbers of these frames
5. Record the time of last received frame prior to the frame loss at 5. Record the time of last received frame prior to the frame loss at
TP1 (TP1-Tlfr) and the time of first frame received after the TP1 (TP1-Tlfr) and the time of first frame received after the
frame loss at TP1 (TP1-Tffr). frame loss at TP1 (TP1-Tffr).
Measurement: Measurement:
Forward Direction Path Re-Provisioning Time (FDRT) Forward Direction Path Re-Provisioning Time (FDRT)
= (TP2-Tffr - TP2-Tlfr) = (TP2-Tffr - TP2-Tlfr)
Reverse Direction Path Re-Provisioning Time (RDRT) Reverse Direction Path Re-Provisioning Time (RDRT)
= (TP1-Tffr - TP1-Tlfr) = (TP1-Tffr - TP1-Tlfr)
Network Re-Provisioning Time = (FDRT+RDRT)/2 Network Re-Provisioning Time = (FDRT+RDRT)/2
skipping to change at page 30, line 23 skipping to change at page 30, line 48
[RFC5440] JP. Vasseur, JL. Le Roux, "Path Computation Element (PCE) [RFC5440] JP. Vasseur, JL. Le Roux, "Path Computation Element (PCE)
Communication Protocol (PCEP)", RFC 5440, March 2009. Communication Protocol (PCEP)", RFC 5440, March 2009.
[OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" [OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification"
Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.
[I-D.sdn-controller-benchmark-term] Bhuvaneswaran.V, Anton Basil, [I-D.sdn-controller-benchmark-term] Bhuvaneswaran.V, Anton Basil,
Mark.T, Vishwas Manral, Sarah Banks, "Terminology for Mark.T, Vishwas Manral, Sarah Banks, "Terminology for
Benchmarking SDN Controller Performance", Benchmarking SDN Controller Performance",
draft-ietf-bmwg-sdn-controller-benchmark-term-02 draft-ietf-bmwg-sdn-controller-benchmark-term-03
(Work in progress), July 8, 2016 (Work in progress), January 8, 2017
6.2. Informative References 6.2. Informative References
[I-D.i2rs-architecture] A. Atlas, J. Halpern, S. Hares, D. Ward, [I-D.i2rs-architecture] A. Atlas, J. Halpern, S. Hares, D. Ward,
T. Nadeau, "An Architecture for the Interface to the T. Nadeau, "An Architecture for the Interface to the
Routing System", draft-ietf-i2rs-architecture-09 Routing System", draft-ietf-i2rs-architecture-09
(Work in progress), March 6, 2015 (Work in progress), March 6, 2015
[OpenContrail] Ankur Singla, Bruno Rijsman, "OpenContrail [OpenContrail] Ankur Singla, Bruno Rijsman, "OpenContrail
Architecture Documentation", Architecture Documentation",
skipping to change at page 31, line 5 skipping to change at page 31, line 29
7. IANA Considerations 7. IANA Considerations
This document does not have any IANA requests. This document does not have any IANA requests.
8. Security Considerations 8. Security Considerations
Benchmarking tests described in this document are limited to the Benchmarking tests described in this document are limited to the
performance characterization of controller in lab environment with performance characterization of controller in lab environment with
isolated network. isolated network.
The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test
traffic into a production network, or misroute traffic to the test
management network.
Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the controller.
Special capabilities SHOULD NOT exist in the controller specifically
for benchmarking purposes. Any implications for network security
arising from the controller SHOULD be identical in the lab and in
production networks
9. Acknowledgments 9. Acknowledgments
The authors would like to thank the following individuals for The authors would like to thank the following individuals for
providing their valuable comments to the earlier versions of this providing their valuable comments to the earlier versions of this
document: Al Morton (AT&T), Sandeep Gangadharan (HP), M. Georgescu document: Al Morton (AT&T), Sandeep Gangadharan (HP), M. Georgescu
(NAIST), Andrew McGregor (Google), Scott Bradner (Harvard (NAIST), Andrew McGregor (Google), Scott Bradner (Harvard
University), Jay Karthik (Cisco), Ramakrishnan (Dell), Khasanov University), Jay Karthik (Cisco), Ramakrishnan (Dell), Khasanov
Boris (Huawei), Brian Castelli (Spirent) Boris (Huawei), Brian Castelli (Spirent)
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
Appendix A. Example Test Topologies Appendix A. Example Test Topologies
A.1. Leaf-Spine Topology - Three Tier Network Architecture A.1. Leaf-Spine Topology - Three Tier Network Architecture
+----------+ +----------+
| SDN | | SDN |
| Node | (Core) | Node | (Core)
+----------+ +----------+
/ \ / \
/ \ / \
+------+ +------+ +------+ +------+
| SDN | | SDN | (Spine) | SDN | | SDN | (Spine)
| Node |.. | Node | | Node |.. | Node |
+------+ +------+ +------+ +------+
/ \ / \ / \ / \
/ \ / \ / \ / \
l1 / / \ ln-1 l1 / / \ ln-1
/ / \ \ / / \ \
+--------+ +-------+ +--------+ +-------+
| SDN | | SDN | | SDN | | SDN |
| Node |.. | Node | (Leaf) | Node |.. | Node | (Leaf)
+--------+ +-------+ +--------+ +-------+
A.2. Leaf-Spine Topology - Two Tier Network Architecture A.2. Leaf-Spine Topology - Two Tier Network Architecture
+------+ +------+ +------+ +------+
| SDN | | SDN | (Spine) | SDN | | SDN | (Spine)
| Node |.. | Node | | Node |.. | Node |
+------+ +------+ +------+ +------+
/ \ / \ / \ / \
/ \ / \ / \ / \
l1 / / \ ln-1 l1 / / \ ln-1
/ / \ \ / / \ \
+--------+ +-------+ +--------+ +-------+
| SDN | | SDN | | SDN | | SDN |
| Node |.. | Node | (Leaf) | Node |.. | Node | (Leaf)
+--------+ +-------+ +--------+ +-------+
Appendix B. Benchmarking Methodology using OpenFlow Controllers Appendix B. Benchmarking Methodology using OpenFlow Controllers
This section gives an overview of OpenFlow protocol and provides This section gives an overview of OpenFlow protocol and provides
test methodology to benchmark SDN controllers supporting OpenFlow test methodology to benchmark SDN controllers supporting OpenFlow
southbound protocol. southbound protocol.
B.1. Protocol Overview B.1. Protocol Overview
OpenFlow is an open standard protocol defined by Open Networking OpenFlow is an open standard protocol defined by Open Networking
Foundation (ONF), used for programming the forwarding plane of Foundation (ONF), used for programming the forwarding plane of
network switches or routers via a centralized controller. network switches or routers via a centralized controller.
B.2. Messages Overview B.2. Messages Overview
OpenFlow protocol supports three messages types namely controller- OpenFlow protocol supports three messages types namely controller-
to-switch, asynchronous and symmetric. to-switch, asynchronous and symmetric.
Controller-to-switch messages are initiated by the controller and Controller-to-switch messages are initiated by the controller and
used to directly manage or inspect the state of the switch. These used to directly manage or inspect the state of the switch. These
messages allow controllers to query/configure the switch (Features, messages allow controllers to query/configure the switch (Features,
Configuration messages), collect information from switch (Read-State Configuration messages), collect information from switch (Read-State
message), send packets on specified port of switch (Packet-out message), send packets on specified port of switch (Packet-out
message), and modify switch forwarding plane and state (Modify- message), and modify switch forwarding plane and state (Modify-
skipping to change at page 33, line 40 skipping to change at page 34, line 40
Asynchronous messages are generated by the switch without a Asynchronous messages are generated by the switch without a
controller soliciting them. These messages allow switches to update controller soliciting them. These messages allow switches to update
controllers to denote an arrival of new flow (Packet-in), switch controllers to denote an arrival of new flow (Packet-in), switch
state change (Flow-Removed, Port-status) and error (Error). state change (Flow-Removed, Port-status) and error (Error).
Symmetric messages are generated in either direction without Symmetric messages are generated in either direction without
solicitation. These messages allow switches and controllers to set solicitation. These messages allow switches and controllers to set
up connection (Hello), verify for liveness (Echo) and offer up connection (Hello), verify for liveness (Echo) and offer
additional functionalities (Experimenter). additional functionalities (Experimenter).
B.3. Connection Overview B.3. Connection Overview
OpenFlow channel is used to exchange OpenFlow message between an OpenFlow channel is used to exchange OpenFlow message between an
OpenFlow switch and an OpenFlow controller. The OpenFlow channel OpenFlow switch and an OpenFlow controller. The OpenFlow channel
connection can be setup using plain TCP or TLS. By default, a switch connection can be setup using plain TCP or TLS. By default, a switch
establishes single connection with SDN controller. A switch may establishes single connection with SDN controller. A switch may
establish multiple parallel connections to single controller establish multiple parallel connections to single controller
(auxiliary connection) or multiple controllers to handle controller (auxiliary connection) or multiple controllers to handle controller
failures and load balancing. failures and load balancing.
B.4. Performance Benchmarking Tests B.4. Performance Benchmarking Tests
B.4.1. Network Topology Discovery Time B.4.1. Network Topology Discovery Time
Procedure: Procedure:
Network Devices OpenFlow SDN Network Devices OpenFlow SDN
Controller Application Controller Application
| | | | | |
| |<Initialize controller | | |<Initialize controller |
| |app.,NB and SB interfaces> | | |app.,NB and SB interfaces> |
skipping to change at page 42, line 13 skipping to change at page 43, line 13
| | PACKET_OUT with LLDP T1>| | | PACKET_OUT with LLDP T1>|
Discussion: Discussion:
The Network Topology Change Detection Time can be obtained by The Network Topology Change Detection Time can be obtained by
finding the difference between the time the OpenFlow switch S1 sends finding the difference between the time the OpenFlow switch S1 sends
the PORT_STATUS message (T0) and the time that the OpenFlow the PORT_STATUS message (T0) and the time that the OpenFlow
controller sends the first topology re-discovery message (T1) to controller sends the first topology re-discovery message (T1) to
OpenFlow switches. OpenFlow switches.
B.5. Scalability B.5. Scalability
B.5.1. Control Sessions Capacity B.5.1. Control Sessions Capacity
Procedure: Procedure:
Network Devices OpenFlow Network Devices OpenFlow
Controller Controller
| | | |
| OFPT_HELLO Exchange for Switch 1 | | OFPT_HELLO Exchange for Switch 1 |
|<------------------------------------->| |<------------------------------------->|
skipping to change at page 45, line 5 skipping to change at page 46, line 5
FWD: Forwarding Table FWD: Forwarding Table
Discussion: Discussion:
Query the controller forwarding table entries for multiple times Query the controller forwarding table entries for multiple times
until the three consecutive queries return the same value. The last until the three consecutive queries return the same value. The last
value retrieved from the controller will provide the Forwarding value retrieved from the controller will provide the Forwarding
Table Capacity value. The query interval is user configurable. The 5 Table Capacity value. The query interval is user configurable. The 5
seconds shown in this example is for representational purpose. seconds shown in this example is for representational purpose.
B.6. Security B.6. Security
B.6.1. Exception Handling B.6.1. Exception Handling
Procedure: Procedure:
Test Traffic Test Traffic Network Devices OpenFlow SDN Test Traffic Test Traffic Network Devices OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1..Dn) | | | | |G-ARP (D1..Dn) | | |
| |------------------>| | | | |------------------>| | |
skipping to change at page 48, line 5 skipping to change at page 49, line 5
TCP SYN attack should be launched from one of the emulated/simulated TCP SYN attack should be launched from one of the emulated/simulated
OpenFlow Switch. Rn1 provides the Path Programming Rate of OpenFlow Switch. Rn1 provides the Path Programming Rate of
controller uponhandling denial of service attack. controller uponhandling denial of service attack.
The procedure defined above provides test steps to determine the The procedure defined above provides test steps to determine the
effect of handling denial of service on Path Programming Rate. Same effect of handling denial of service on Path Programming Rate. Same
procedure can be adopted to determine the effects on other procedure can be adopted to determine the effects on other
performance tests listed in this benchmarking tests. performance tests listed in this benchmarking tests.
B.7. Reliability B.7. Reliability
B.7.1. Controller Failover Time B.7.1. Controller Failover Time
Procedure: Procedure:
Test Traffic Test Traffic Network Device OpenFlow SDN Test Traffic Test Traffic Network Device OpenFlow SDN
Generator TP1 Generator TP2 Controller Application Generator TP1 Generator TP2 Controller Application
| | | | | | | | | |
| |G-ARP (D1) | | | | |G-ARP (D1) | | |
| |------------>| | | | |------------>| | |
 End of changes. 143 change blocks. 
281 lines changed or deleted 315 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/