draft-ietf-bmwg-sdn-controller-benchmark-term-10.txt   rfc8455.txt 
Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies
Expires: November 25, 2018 Mark Tassinari
Hewlett-Packard
Vishwas Manral
Nano Sec
Sarah Banks
VSS Monitoring
May 25, 2018
Terminology for Benchmarking SDN Controller Performance Internet Engineering Task Force (IETF) V. Bhuvaneswaran
draft-ietf-bmwg-sdn-controller-benchmark-term-10 Request for Comments: 8455 A. Basil
Category: Informational Veryx Technologies
ISSN: 2070-1721 M. Tassinari
Hewlett Packard Enterprise
V. Manral
NanoSec
S. Banks
VSS Monitoring
October 2018
Abstract Terminology for Benchmarking Software-Defined Networking (SDN)
Controller Performance
This document defines terminology for benchmarking an SDN Abstract
controller's control plane performance. It extends the terminology
already defined in RFC 7426 for the purpose of benchmarking SDN
controllers. The terms provided in this document help to benchmark
SDN controller's performance independent of the controller's
supported protocols and/or network services.
Status of this Memo This document defines terminology for benchmarking a Software-Defined
Networking (SDN) controller's control-plane performance. It extends
the terminology already defined in RFC 7426 for the purpose of
benchmarking SDN Controllers. The terms provided in this document
help to benchmark an SDN Controller's performance independently of
the controller's supported protocols and/or network services.
This Internet-Draft is submitted in full conformance with the Status of This Memo
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This document is not an Internet Standards Track specification; it is
Task Force (IETF). Note that other groups may also distribute published for informational purposes.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six This document is a product of the Internet Engineering Task Force
months and may be updated, replaced, or obsoleted by other documents (IETF). It represents the consensus of the IETF community. It has
at any time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress. Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on November 25, 2018. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
https://www.rfc-editor.org/info/rfc8455.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with respect
respect to this document. Code Components extracted from this to this document. Code Components extracted from this document must
document must include Simplified BSD License text as described in include Simplified BSD License text as described in Section 4.e of
Section 4.e of the Trust Legal Provisions and are provided without the Trust Legal Provisions and are provided without warranty as
warranty as described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction...................................................4 1. Introduction ....................................................3
2. Term Definitions...............................................4 1.1. Conventions Used in This Document ..........................3
2.1. SDN Terms.................................................4 2. Term Definitions ................................................4
2.1.1. Flow.................................................4 2.1. SDN Terms ..................................................4
2.1.2. Northbound Interface.................................5 2.1.1. Flow ................................................4
2.1.3. Southbound Interface.................................5 2.1.2. Northbound Interface ................................4
2.1.4. Controller Forwarding Table..........................5 2.1.3. Southbound Interface ................................5
2.1.5. Proactive Flow Provisioning Mode.....................6 2.1.4. Controller Forwarding Table .........................5
2.1.6. Reactive Flow Provisioning Mode......................6 2.1.5. Proactive Flow Provisioning Mode ....................5
2.1.7. Path.................................................7 2.1.6. Reactive Flow Provisioning Mode .....................6
2.1.8. Standalone Mode......................................7 2.1.7. Path ................................................6
2.1.9. Cluster/Redundancy Mode..............................7 2.1.8. Standalone Mode .....................................6
2.1.10. Asynchronous Message................................8 2.1.9. Cluster/Redundancy Mode .............................7
2.1.11. Test Traffic Generator..............................8 2.1.10. Asynchronous Message ...............................7
2.1.12. Leaf-Spine Topology.................................9 2.1.11. Test Traffic Generator .............................7
2.2. Test Configuration/Setup Terms............................9 2.1.12. Leaf-Spine Topology ................................8
2.2.1. Number of Network Devices............................9 2.2. Test Configuration/Setup Terms .............................8
2.2.2. Trial Repetition.....................................9 2.2.1. Number of Network Devices ...........................8
2.2.3. Trial Duration......................................10 2.2.2. Trial Repetition ....................................8
2.2.4. Number of Cluster nodes.............................10 2.2.3. Trial Duration ......................................9
2.3. Benchmarking Terms.......................................10 2.2.4. Number of Cluster Nodes .............................9
2.3.1. Performance.........................................11 2.3. Benchmarking Terms .........................................9
2.3.1.1. Network Topology Discovery Time................11 2.3.1. Performance .........................................9
2.3.1.2. Asynchronous Message Processing Time...........11 2.3.1.1. Network Topology Discovery Time ............9
2.3.1.3. Asynchronous Message Processing Rate...........12 2.3.1.2. Asynchronous Message Processing Time ......10
2.3.1.4. Reactive Path Provisioning Time................13 2.3.1.3. Asynchronous Message Processing Rate ......10
2.3.1.5. Proactive Path Provisioning Time...............13 2.3.1.4. Reactive Path Provisioning Time ...........11
2.3.1.6. Reactive Path Provisioning Rate................14 2.3.1.5. Proactive Path Provisioning Time ..........12
2.3.1.7. Proactive Path Provisioning Rate...............14 2.3.1.6. Reactive Path Provisioning Rate ...........12
2.3.1.8. Network Topology Change Detection Time.........15 2.3.1.7. Proactive Path Provisioning Rate ..........13
2.3.2. Scalability.........................................16 2.3.1.8. Network Topology Change Detection Time ....13
2.3.2.1. Control Sessions Capacity......................16
2.3.2.2. Network Discovery Size.........................16
2.3.2.3. Forwarding Table Capacity......................17
2.3.3. Security............................................17
2.3.3.1. Exception Handling.............................17
2.3.3.2. Denial of Service Handling.....................18
2.3.4. Reliability.........................................18
2.3.4.1. Controller Failover Time.......................18
2.3.4.2. Network Re-Provisioning Time...................19
3. Test Setup....................................................19
3.1. Test setup - Controller working in Standalone Mode.......20
3.2. Test setup - Controller working in Cluster Mode..........21
4. Test Coverage.................................................22
5. References....................................................23
5.1. Normative References.....................................23
5.2. Informative References...................................23
6. IANA Considerations...........................................23
7. Security Considerations.......................................23
8. Acknowledgements..............................................24
9. Authors' Addresses............................................24
1. Introduction 2.3.2. Scalability ........................................14
2.3.2.1. Control Sessions Capacity .................14
2.3.2.2. Network Discovery Size ....................14
2.3.2.3. Forwarding Table Capacity .................15
2.3.3. Security ...........................................15
2.3.3.1. Exception Handling ........................15
2.3.3.2. Handling Denial-of-Service Attacks ........16
2.3.4. Reliability ........................................16
2.3.4.1. Controller Failover Time ..................16
2.3.4.2. Network Re-provisioning Time ..............17
3. Test Setup .....................................................17
3.1. Test Setup - Controller Operating in Standalone Mode ......18
3.2. Test Setup - Controller Operating in Cluster Mode .........19
4. Test Coverage ..................................................20
5. IANA Considerations ............................................21
6. Security Considerations ........................................21
7. Normative References ...........................................21
Acknowledgments ...................................................22
Authors' Addresses ................................................23
Software Defined Networking (SDN) is a networking architecture in 1. Introduction
Software-Defined Networking (SDN) is a networking architecture in
which network control is decoupled from the underlying forwarding which network control is decoupled from the underlying forwarding
function and is placed in a centralized location called the SDN function and is placed in a centralized location called the SDN
controller. The SDN controller provides an abstraction of the Controller. The SDN Controller provides an abstraction of the
underlying network and offers a global view of the overall network underlying network and offers a global view of the overall network to
to applications and business logic. Thus, an SDN controller provides applications and business logic. Thus, an SDN Controller provides
the flexibility to program, control, and manage network behaviour the flexibility to program, control, and manage network behavior
dynamically through northbound and southbound interfaces. Since the dynamically through northbound and southbound interfaces. Since the
network controls are logically centralized, the need to benchmark network controls are logically centralized, the need to benchmark the
the SDN controller performance becomes significant. This document SDN Controller's performance becomes significant. This document
defines terms to benchmark various controller designs for defines terms to benchmark various controller designs for
performance, scalability, reliability and security, independent of performance, scalability, reliability, and security, independently of
northbound and southbound protocols. A mechanism for benchmarking northbound and southbound protocols. A mechanism for benchmarking
the performance of SDN controllers is defined in the companion the performance of SDN Controllers is defined in the companion
methodology document [I-D.sdn-controller-benchmark-meth]. These two methodology document [RFC8456]. These two documents provide methods
documents provide a method to measure and evaluate the performance for measuring and evaluating the performance of various controller
of various controller implementations. implementations.
Conventions used in this document 1.1. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in "OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
2. Term Definitions 2. Term Definitions
2.1. SDN Terms 2.1. SDN Terms
The terms defined in this section are extensions to the terms The terms defined in this section are extensions to the terms defined
defined in [RFC7426] "Software-Defined Networking (SDN): Layers and in [RFC7426] ("Software-Defined Networking (SDN): Layers and
Architecture Terminology". That RFC should be referred before Architecture Terminology"). Readers should refer to [RFC7426] before
attempting to make use of this document. attempting to make use of this document.
2.1.1. Flow 2.1.1. Flow
Definition:
The definition of Flow is same as microflows defined in [RFC4689]
Section 3.1.5.
Discussion:
A flow can be set of packets having same source address, destination
address, source port and destination port, or any of these
combinations.
Measurement Units:
N/A
See Also:
None
2.1.2. Northbound Interface
Definition:
The definition of northbound interface is same the Service Interface
defined in [RFC7426].
Discussion:
The northbound interface allows SDN applications and orchestration
systems to program and retrieve the network information through the
SDN controller.
Measurement Units:
N/A
See Also:
None
2.1.3. Southbound Interface
Definition:
The southbound interface is the application programming interface
provided by the SDN controller to interact with the SDN nodes.
Discussion:
Southbound interface enables controller to interact with the SDN
nodes in the network for dynamically defining the traffic forwarding
behaviour.
Measurement Units:
N/A
See Also:
None
2.1.4. Controller Forwarding Table
Definition:
A controller forwarding table contains flow entries learned in one
of two ways: first, entries could be learned from traffic received
through the data plane, or second, these entries could be statically
provisioned on the controller and distributed to devices via the
southbound interface.
Discussion:
The controller forwarding table has an aging mechanism which will be
applied only for dynamically learned entries.
Measurement Units:
N/A
See Also:
None
2.1.5. Proactive Flow Provisioning Mode
Definition:
Controller programming flows in Network Devices based on the flow
entries provisioned through controller's northbound interface.
Discussion:
Network orchestration systems and SDN applications can define the
network forwarding behaviour by programming the controller using
proactive flow provisioning. The controller can then program the
Network Devices with the pre-provisioned entries.
Measurement Units: Definition:
N/A The definition of "flow" is the same as the definition of
"microflows" provided in Section 3.1.5 of [RFC4689].
See Also: Discussion:
None A flow can be a set of packets having the same source address,
destination address, source port, and destination port, or any
combination of these items.
2.1.6. Reactive Flow Provisioning Mode Measurement Units:
N/A
Definition: 2.1.2. Northbound Interface
Controller programming flows in Network Devices based on the traffic
received from Network Devices through controller's southbound
interface
Discussion: Definition:
The SDN controller dynamically decides the forwarding behaviour The definition of "northbound interface" is the same as the
based on the incoming traffic from the Network Devices. The definition of "service interface" provided in [RFC7426].
controller then programs the Network Devices using Reactive Flow
Provisioning.
Measurement Units: Discussion:
N/A The northbound interface allows SDN applications and orchestration
systems to program and retrieve the network information through
the SDN Controller.
See Also: Measurement Units:
None N/A
2.1.7. Path 2.1.3. Southbound Interface
Definition: Definition:
Refer to Section 5 in [RFC2330] The southbound interface is the application programming interface
provided by the SDN Controller to interact with the SDN nodes.
Discussion: Discussion:
None The southbound interface enables the controller to interact with
the SDN nodes in the network for dynamically defining the traffic
forwarding behavior.
Measurement Units: Measurement Units:
N/A N/A
See Also: 2.1.4. Controller Forwarding Table
None
2.1.8. Standalone Mode Definition:
A controller Forwarding Table contains flow entries learned in one
of two ways: first, entries can be learned from traffic received
through the data plane, or second, these entries can be statically
provisioned on the controller and distributed to devices via the
southbound interface.
Definition: Discussion:
Single controller handling all control plane functionalities without The controller Forwarding Table has an aging mechanism that will
redundancy, or the ability to provide high availability and/or be applied only for dynamically learned entries.
automatic failover.
Discussion: Measurement Units:
In standalone mode, one controller manages one or more network N/A
domains.
Measurement Units: 2.1.5. Proactive Flow Provisioning Mode
N/A
See Also: Definition:
None Controller programming flows in Network Devices based on the flow
entries provisioned through the controller's northbound interface.
2.1.9. Cluster/Redundancy Mode Discussion:
Network orchestration systems and SDN applications can define the
network forwarding behavior by programming the controller, using
Proactive Flow Provisioning. The controller can then program the
Network Devices with the pre-provisioned entries.
Definition: Measurement Units:
A group of 2 or more controllers handling all control plane N/A
functionalities.
Discussion: 2.1.6. Reactive Flow Provisioning Mode
In cluster mode, multiple controllers are teamed together for the
purpose of load sharing and/or high availability. The controllers in
the group may work in active/standby (master/slave) or active/active
(equal) mode depending on the intended purpose.
Measurement Units: Definition:
N/A Controller programming flows in Network Devices based on the
traffic received from Network Devices through the controller's
southbound interface.
See Also: Discussion:
None The SDN Controller dynamically decides the forwarding behavior
based on the incoming traffic from the Network Devices. The
controller then programs the Network Devices, using Reactive Flow
Provisioning.
2.1.10. Asynchronous Message Measurement Units:
N/A
Definition: 2.1.7. Path
Any message from the Network Device that is generated for network
events.
Discussion: Definition:
Control messages like flow setup request and response message is Refer to Section 5 in [RFC2330].
classified as asynchronous message. The controller has to return a
response message. Note that the Network Device will not be in
blocking mode and continues to send/receive other control messages.
Measurement Units: Discussion:
N/A None
See Also: Measurement Units:
None N/A
2.1.11. Test Traffic Generator 2.1.8. Standalone Mode
Definition: Definition:
Test Traffic Generator is an entity that generates/receives network A single controller handles all control-plane functionalities
traffic. without redundancy, and it is unable to provide high availability
and/or automatic failover.
Discussion: Discussion:
Test Traffic Generator typically connects with Network Devices to In standalone mode, one controller manages one or more network
send/receive real-time network traffic. domains.
Measurement Units: Measurement Units:
N/A N/A
See Also: 2.1.9. Cluster/Redundancy Mode
None
2.1.12. Leaf-Spine Topology Definition:
In this mode, a group of two or more controllers handles all
control-plane functionalities.
Definition: Discussion:
Leaf-Spine is a two layered network topology, where a series of In cluster mode, multiple controllers are teamed together for the
leaf switches, form the access layer, are fully meshed to a series purpose of load sharing and/or high availability. The controllers
of spine switches that form the backbone layer. in the group may operate in active/standby (master/slave) or
active/active (equal) mode, depending on the intended purpose.
Discussion: Measurement Units:
In Leaf-Spine Topology, every leaf switch is connected to each of N/A
the spine switches in the topology.
Measurement Units: 2.1.10. Asynchronous Message
N/A
See Also: Definition:
None Any message from the Network Device that is generated for network
events.
2.2. Test Configuration/Setup Terms Discussion:
Control messages like flow setup request and response messages are
classified as asynchronous messages. The controller has to return
a response message. Note that the Network Device will not be in
blocking mode and continues to send/receive other control
messages.
2.2.1. Number of Network Devices Measurement Units:
N/A
Definition: 2.1.11. Test Traffic Generator
The number of Network Devices present in the defined test topology.
Discussion: Definition:
The Network Devices defined in the test topology can be deployed The test traffic generator is an entity that generates/receives
using real hardware or emulated in hardware platforms. network traffic.
Measurement Units: Discussion:
Number of network devices The test traffic generator typically connects with Network Devices
to send/receive real-time network traffic.
See Also: Measurement Units:
None N/A
2.2.2. Trial Repetition 2.1.12. Leaf-Spine Topology
Definition: Definition:
The number of times the test needs to be repeated. "Leaf-Spine" is a two-layered network topology, where a series of
leaf switches that form the access layer are fully meshed to a
series of spine switches that form the backbone layer.
Discussion: Discussion:
The test needs to be repeated for multiple iterations to obtain a In the Leaf-Spine topology, every leaf switch is connected to each
reliable metric. It is recommended that this test SHOULD be of the spine switches in the topology.
performed for at least 10 iterations to increase the confidence in
measured result.
Measurement Units: Measurement Units:
Number of trials N/A
See Also: 2.2. Test Configuration/Setup Terms
None
2.2.3. Trial Duration 2.2.1. Number of Network Devices
Definition: Definition:
Defines the duration of test trials for each iteration. The number of Network Devices present in the defined test
topology.
Discussion: Discussion:
Trial duration forms the basis for stop criteria for benchmarking The Network Devices defined in the test topology can be deployed
tests. Trials not completed within this time interval is considered using real hardware or can be emulated in hardware platforms.
as incomplete.
Measurement Units: Measurement Units:
Seconds Number of Network Devices.
See Also:
None
2.2.4. Number of Cluster nodes
Definition:
Defines the number of controllers present in the controller cluster.
Discussion: 2.2.2. Trial Repetition
This parameter is relevant when testing the controller performance
in clustering/teaming mode. The number of nodes in the cluster MUST
be greater than 1.
Measurement Units: Definition:
Number of controller nodes The number of times the test needs to be repeated.
See Also: Discussion:
None The test needs to be repeated for multiple iterations to obtain a
reliable metric. It is recommended that this test SHOULD be
performed for at least 10 iterations to increase confidence in the
measured results.
2.3. Benchmarking Terms Measurement Units:
Number of trials.
This section defines metrics for benchmarking the SDN controller. 2.2.3. Trial Duration
The procedure to perform the defined metrics is defined in the
accompanying methodology document[I-D.sdn-controller-benchmark-meth]
2.3.1. Performance Definition:
Defines the duration of test trials for each iteration.
2.3.1.1. Network Topology Discovery Time Discussion:
The Trial Duration forms the basis for "stop" criteria for
benchmarking tests. Trials not completed within this time
interval are considered incomplete.
Definition: Measurement Units:
The time taken by controller(s) to determine the complete network Seconds.
topology, defined as the interval starting with the first discovery
message from the controller(s) at its Southbound interface, ending
with all features of the static topology determined.
Discussion: 2.2.4. Number of Cluster Nodes
Network topology discovery is key for the SDN controller to
provision and manage the network. So it is important to measure how
quickly the controller discovers the topology to learn the current
network state. This benchmark is obtained by presenting a network
topology (Tree, Mesh or Linear) with the given number of nodes to
the controller and wait for the discovery process to complete. It is
expected that the controller supports network discovery mechanism
and uses protocol messages for its discovery process.
Measurement Units: Definition:
Milliseconds Defines the number of controllers present in the controller
cluster.
See Also: Discussion:
None This parameter is relevant when testing the controller's
performance in clustering/teaming mode. The number of nodes in
the cluster MUST be greater than 1.
2.3.1.2. Asynchronous Message Processing Time Measurement Units:
Number of controller nodes.
Definition: 2.3. Benchmarking Terms
The time taken by controller(s) to process an asynchronous message,
defined as the interval starting with an asynchronous message from a
network device after the discovery of all the devices by the
controller(s), ending with a response message from the controller(s)
at its Southbound interface.
Discussion: This section defines metrics for benchmarking the SDN Controller.
For SDN to support dynamic network provisioning, it is important to The procedure for performing the defined metrics is defined in the
measure how quickly the controller responds to an event triggered companion methodology document [RFC8456].
from the network. The event could be any notification messages
generated by a Network Device upon arrival of a new flow, link down
etc. This benchmark is obtained by sending asynchronous messages
from every connected Network Devices one at a time for the defined
trial duration. This test assumes that the controller will respond
to the received asynchronous message.
Measurement Units: 2.3.1. Performance
Milliseconds
See Also: 2.3.1.1. Network Topology Discovery Time
None
2.3.1.3. Asynchronous Message Processing Rate Definition:
The time taken by the controller(s) to determine the complete
network topology, defined as the interval starting with the first
discovery message from the controller(s) at its southbound
interface and ending with all features of the static topology
determined.
Definition: Discussion:
The number responses to asynchronous messages per second (such as Network topology discovery is key for the SDN Controller to
new flow arrival notification message, link down, etc.) for which provision and manage the network, so it is important to measure
the controller(s) performed processing and replied with a valid and how quickly the controller discovers the topology to learn the
productive (non-trivial) response message. current network state. This benchmark is obtained by presenting a
network topology (tree, mesh, or linear) with a specified number
of nodes to the controller and waiting for the discovery process
to complete. It is expected that the controller supports a
network discovery mechanism and uses protocol messages for its
discovery process.
Discussion: Measurement Units:
As SDN assures flexible network and agile provisioning, it is Milliseconds.
important to measure how many network events (such as new flow
arrival notification message, link down, etc.) the controller can
handle at a time. This benchmark is measured by sending asynchronous
messages from every connected Network Device at the rate that the
controller processes (without dropping them). This test assumes that
the controller responds to all the received asynchronous messages
(the messages can be designed to elicit individual responses).
When sending asynchronous messages to the controller(s) at high 2.3.1.2. Asynchronous Message Processing Time
rates, some messages or responses may be discarded or corrupted and
require retransmission to controller(s). Therefore, a useful
qualification on Asynchronous Message Processing Rate is whether the
in-coming message count equals the response count in each trial.
This is called the Loss-free Asynchronous Message Processing Rate.
Note that several of the early controller benchmarking tools did not Definition:
consider lost messages, and instead report the maximum response The time taken by the controller(s) to process an asynchronous
rate. This is called the Maximum Asynchronous Message Processing message, defined as the interval starting with an asynchronous
Rate. message from a Network Device after the discovery of all the
devices by the controller(s) and ending with a response message
from the controller(s) at its southbound interface.
To characterize both the Loss-free and Maximum Rates, a test could Discussion:
begin the first trial by sending asynchronous messages to the For SDN to support dynamic network provisioning, it is important
controller(s) at the maximum possible rate and record the message to measure how quickly the controller responds to an event
reply rate and the message loss rate. The message sending rate is triggered from the network. The event can be any notification
then decreased by the step-size. The message reply rate and the messages generated by a Network Device upon arrival of a new flow,
message loss rate are recorded. The test ends with a trial where the link down, etc. This benchmark is obtained by sending
controller(s) processes the all asynchronous messages sent without asynchronous messages from every connected Network Device one at a
loss. This is the Loss-free Asynchronous Message Processing Rate. time for the defined Trial Duration. This test assumes that the
controller will respond to the received asynchronous messages.
The trial where the controller(s) produced the maximum response rate Measurement Units:
is the Maximum Asynchronous Message Processing Rate. Of course, the Milliseconds.
first trial could begin at a low sending rate with zero lost
responses, and increase until the Loss-free and Maximum Rates are
discovered.
Measurement Units: 2.3.1.3. Asynchronous Message Processing Rate
Messages processed per second.
See Also: Definition:
None The number of responses to asynchronous messages per second (a new
flow arrival notification message, link down, etc.) for which the
controller(s) performed processing and replied with a valid and
productive (non-trivial) response message.
2.3.1.4. Reactive Path Provisioning Time Discussion:
As SDN assures a flexible network and agile provisioning, it is
important to measure how many network events (a new flow arrival
notification message, link down, etc.) the controller can handle
at a time. This benchmark is measured by sending asynchronous
messages from every connected Network Device at the rate that the
controller processes (without dropping them). This test assumes
that the controller responds to all the received asynchronous
messages (the messages can be designed to elicit individual
responses).
Definition: When sending asynchronous messages to the controller(s) at high
The time taken by the controller to setup a path reactively between rates, some messages or responses may be discarded or corrupted
source and destination node, defined as the interval starting with and require retransmission to controller(s). Therefore, a useful
the first flow provisioning request message received by the qualification on the Asynchronous Message Processing Rate is
controller(s), ending with the last flow provisioning response whether the incoming message count equals the response count in
message sent from the controller(s) at its Southbound interface. each trial. This is called the Loss-Free Asynchronous Message
Processing Rate.
Discussion: Note that several of the early controller benchmarking tools did
As SDN supports agile provisioning, it is important to measure how not consider lost messages and instead report the maximum response
fast that the controller provisions an end-to-end flow in the rate. This is called the Maximum Asynchronous Message Processing
dataplane. The benchmark is obtained by sending traffic from a Rate.
source endpoint to the destination endpoint, finding the time
difference between the first and the last flow provisioning message
exchanged between the controller and the Network Devices for the
traffic path.
Measurement Units: To characterize both the Loss-Free Asynchronous Message Processing
Milliseconds. Rate and the Maximum Asynchronous Message Processing Rate, a test
can begin the first trial by sending asynchronous messages to the
controller(s) at the maximum possible rate and can then record the
message reply rate and the message loss rate. The message-sending
rate is then decreased by the STEP size. The message reply rate
and the message loss rate are recorded. The test ends with a
trial where the controller(s) processes all of the asynchronous
messages sent without loss. This is the Loss-Free Asynchronous
Message Processing Rate.
See Also: The trial where the controller(s) produced the maximum response
None rate is the Maximum Asynchronous Message Processing Rate. Of
course, the first trial can begin at a low sending rate with zero
lost responses and then increase the rate until the Loss-Free
Asynchronous Message Processing Rate and the Maximum Asynchronous
Message Processing Rate are discovered.
2.3.1.5. Proactive Path Provisioning Time Measurement Units:
Messages processed per second.
Definition: 2.3.1.4. Reactive Path Provisioning Time
The time taken by the controller to proactively setup a path between
source and destination node, defined as the interval starting with
the first proactive flow provisioned in the controller(s) at its
Northbound interface, ending with the last flow provisioning command
message sent from the controller(s) at its Southbound interface.
Discussion: Definition:
The time taken by the controller to set up a path reactively
between source and destination nodes, defined as the interval
starting with the first flow provisioning request message received
by the controller(s) and ending with the last flow provisioning
response message sent from the controller(s) at its southbound
interface.
For SDN to support pre-provisioning of traffic path from Discussion:
application, it is important to measure how fast that the controller As SDN supports agile provisioning, it is important to measure how
provisions an end-to-end flow in the dataplane. The benchmark is fast the controller provisions an end-to-end flow in the
obtained by provisioning a flow on controller's northbound interface data plane. The benchmark is obtained by sending traffic from a
for the traffic to reach from a source to a destination endpoint, source endpoint to the destination endpoint and finding the time
finding the time difference between the first and the last flow difference between the first and last flow provisioning message
provisioning message exchanged between the controller and the exchanged between the controller and the Network Devices for the
Network Devices for the traffic path. traffic path.
Measurement Units: Measurement Units:
Milliseconds. Milliseconds.
See Also: 2.3.1.5. Proactive Path Provisioning Time
None
2.3.1.6. Reactive Path Provisioning Rate Definition:
The time taken by the controller to proactively set up a path
between source and destination nodes, defined as the interval
starting with the first proactive flow provisioned in the
controller(s) at its northbound interface and ending with the last
flow provisioning command message sent from the controller(s) at
its southbound interface.
Definition: Discussion:
The maximum number of independent paths a controller can For SDN to support pre-provisioning of the traffic path from the
concurrently establish per second between source and destination application, it is important to measure how fast the controller
nodes reactively, defined as the number of paths provisioned per provisions an end-to-end flow in the data plane. The benchmark is
second by the controller(s) at its Southbound interface for the flow obtained by provisioning a flow on the controller's northbound
provisioning requests received for path provisioning at its interface for the traffic to reach from a source to a destination
Southbound interface between the start of the trial and the expiry endpoint and finding the time difference between the first and
of given trial duration. last flow provisioning message exchanged between the controller
and the Network Devices for the traffic path.
Discussion: Measurement Units:
For SDN to support agile traffic forwarding, it is important to Milliseconds.
measure how many end-to-end flows that the controller could setup in
the dataplane. This benchmark is obtained by sending traffic each
with unique source and destination pairs from the source Network
Device and determine the number of frames received at the
destination Network Device.
Measurement Units: 2.3.1.6. Reactive Path Provisioning Rate
Paths provisioned per second.
See Also: Definition:
None The maximum number of independent paths a controller can
concurrently establish per second between source and destination
nodes reactively, defined as the number of paths provisioned per
second by the controller(s) at its southbound interface for the
flow provisioning requests received for path provisioning at its
southbound interface between the start of the trial and the expiry
of the given Trial Duration.
2.3.1.7. Proactive Path Provisioning Rate Discussion:
For SDN to support agile traffic forwarding, it is important to
measure how many end-to-end flows the controller can set up in the
data plane. This benchmark is obtained by sending each traffic
flow with unique source and destination pairs from the source
Network Device and determining the number of frames received at
the destination Network Device.
Definition: Measurement Units:
Measure the maximum number of independent paths a controller can Paths provisioned per second.
concurrently establish per second between source and destination
nodes proactively, defined as the number of paths provisioned per
second by the controller(s) at its Southbound interface for the
paths provisioned in its Northbound interface between the start of
the trial and the expiry of given trial duration.
Discussion: 2.3.1.7. Proactive Path Provisioning Rate
For SDN to support pre-provisioning of traffic path for a larger
network from the application, it is important to measure how many
end-to-end flows that the controller could setup in the dataplane.
This benchmark is obtained by sending traffic each with unique
source and destination pairs from the source Network Device. Program
the flows on controller's northbound interface for traffic to reach
from each of the unique source and destination pairs and determine
the number of frames received at the destination Network Device.
Measurement Units: Definition:
Paths provisioned per second. The maximum number of independent paths a controller can
concurrently establish per second between source and destination
nodes proactively, defined as the number of paths provisioned per
second by the controller(s) at its southbound interface for the
paths provisioned in its northbound interface between the start of
the trial and the expiry of the given Trial Duration.
See Also: Discussion:
None For SDN to support pre-provisioning of the traffic path for a
larger network from the application, it is important to measure
how many end-to-end flows the controller can set up in the
data plane. This benchmark is obtained by sending each traffic
flow with unique source and destination pairs from the source
Network Device. Program the flows on the controller's northbound
interface for traffic to reach from each of the unique source and
destination pairs, and determine the number of frames received at
the destination Network Device.
2.3.1.8. Network Topology Change Detection Time Measurement Units:
Paths provisioned per second.
Definition: 2.3.1.8. Network Topology Change Detection Time
The amount of time required for the controller to detect any changes
in the network topology, defined as the interval starting with the
notification message received by the controller(s) at its Southbound
interface, ending with the first topology rediscovery messages sent
from the controller(s) at its Southbound interface.
Discussion: Definition:
In order for the controller to support fast network failure The amount of time taken by the controller to detect any changes
recovery, it is critical to measure how fast the controller is able in the network topology, defined as the interval starting with the
to detect any network-state change events. This benchmark is notification message received by the controller(s) at its
obtained by triggering a topology change event and measuring the southbound interface and ending with the first topology
time controller takes to detect and initiate a topology re-discovery rediscovery messages sent from the controller(s) at its southbound
process. interface.
Measurement Units: Discussion:
Milliseconds In order for the controller to support fast network failure
recovery, it is critical to measure how fast the controller is
able to detect any network-state change events. This benchmark is
obtained by triggering a topology change event and measuring the
time the controller takes to detect and initiate a topology
rediscovery process.
See Also: Measurement Units:
None Milliseconds.
2.3.2. Scalability 2.3.2. Scalability
2.3.2.1. Control Sessions Capacity 2.3.2.1. Control Sessions Capacity
Definition: Definition:
Measure the maximum number of control sessions the controller can The maximum number of control sessions the controller can
maintain, defined as the number of sessions that the controller can maintain, defined as the number of sessions that the controller
accept from network devices, starting with the first control can accept from Network Devices, starting with the first control
session, ending with the last control session that the controller(s) session and ending with the last control session that the
accepts at its Southbound interface. controller(s) accepts at its southbound interface.
Discussion: Discussion:
Measuring the controller's control sessions capacity is important to Measuring the controller's Control Sessions Capacity is important
determine the controller's system and bandwidth resource for determining the controller's system and bandwidth resource
requirements. This benchmark is obtained by establishing control requirements. This benchmark is obtained by establishing a
session with the controller from each of the Network Device until it control session with the controller from each of the Network
fails. The number of sessions that were successfully established Devices until the controller fails. The number of sessions that
will provide the Control Sessions Capacity. were successfully established will provide the Control Sessions
Capacity.
Measurement Units: Measurement Units:
Maximum number of control sessions Maximum number of control sessions.
See Also: 2.3.2.2. Network Discovery Size
None
2.3.2.2. Network Discovery Size Definition:
The network size (number of nodes and links) that a controller can
discover, defined as the size of a network that the controller(s)
can discover, starting with a network topology provided by the
user for discovery and ending with the number of nodes and links
that the controller(s) can successfully discover.
Definition: Discussion:
Measure the network size (number of nodes and links) that a Measuring the maximum network size that the controller can
controller can discover, defined as the size of a network that the discover is key to optimal network planning. This benchmark is
controller(s) can discover, starting from a network topology given obtained by presenting an initial set of Network Devices for
by the user for discovery, ending with the topology that the discovery to the controller. Based on the initial discovery, the
controller(s) could successfully discover. number of Network Devices is increased or decreased to determine
the maximum number of nodes and links that the controller can
discover.
Discussion: Measurement Units:
For optimal network planning, it is key to measure the maximum Maximum number of network nodes and links.
network size that the controller can discover. This benchmark is
obtained by presenting an initial set of Network Devices for
discovery to the controller. Based on the initial discovery, the
number of Network Devices is increased or decreased to determine the
maximum nodes that the controller can discover.
Measurement Units: 2.3.2.3. Forwarding Table Capacity
Maximum number of network nodes and links
See Also: Definition:
None The maximum number of flow entries that a controller can manage in
its Forwarding Table.
2.3.2.3. Forwarding Table Capacity Discussion:
It is important to measure the capacity of a controller's
Forwarding Table to determine the number of flows that the
controller can forward without flooding or dropping any traffic.
This benchmark is obtained by continuously presenting the
controller with new flow entries through the Reactive Flow
Provisioning mode or the Proactive Flow Provisioning mode until
the Forwarding Table becomes full. The maximum number of nodes
that the controller can hold in its Forwarding Table will provide
the Forwarding Table Capacity.
Definition: Measurement Units:
The maximum number of flow entries that a controller can manage in Maximum number of flow entries managed.
its Forwarding table.
Discussion: 2.3.3. Security
It is significant to measure the capacity of controller's Forwarding
Table to determine the number of flows that controller could forward
without flooding/dropping. This benchmark is obtained by
continuously presenting the controller with new flow entries through
reactive or proactive flow provisioning mode until the forwarding
table becomes full. The maximum number of nodes that the controller
can hold in its Forwarding Table will provide Forwarding Table
Capacity.
Measurement Units: 2.3.3.1. Exception Handling
Maximum number of flow entries managed.
See Also: Definition:
None To determine the effect of handling error packets and
notifications on performance tests.
2.3.3. Security Discussion:
This benchmark is to be performed after obtaining the baseline
measurement results for the performance tests defined in
Section 2.3.1. This benchmark determines the deviation from the
baseline performance due to the handling of error or failure
messages from the connected Network Devices.
2.3.3.1. Exception Handling Measurement Units:
Deviation from baseline metrics while handling Exceptions.
Definition: 2.3.3.2. Handling Denial-of-Service Attacks
To determine the effect of handling error packets and notifications
on performance tests.
Discussion: Definition:
This benchmark test is to be performed after obtaining the baseline To determine the effect of handling denial-of-service (DoS)
performance of the performance tests defined in Section 2.3.1. This attacks on performance and scalability tests.
benchmark determines the deviation from the baseline performance due
to the handling of error or failure messages from the connected
Network Devices.
Measurement Units: Discussion:
Deviation of baseline metrics while handling Exceptions. This benchmark is to be performed after obtaining the baseline
measurement results for the performance and scalability tests
defined in Sections 2.3.1 and 2.3.2. This benchmark determines
the deviation from the baseline performance due to the handling of
DoS attacks on the controller.
See Also: Measurement Units:
None Deviation from baseline metrics while handling DoS attacks.
2.3.3.2. Denial of Service Handling 2.3.4. Reliability
Definition: 2.3.4.1. Controller Failover Time
To determine the effect of handling denial of service (DoS) attacks
on performance and scalability tests.
Discussion: Definition:
This benchmark test is to be performed after obtaining the baseline The time taken to switch from an active controller to the backup
performance of the performance and scalability tests defined in controller when the controllers operate in redundancy mode and the
section 2.3.1 and section 2.3.2. This benchmark determines the active controller fails, defined as the interval starting when the
deviation from the baseline performance due to the handling of active controller is brought down and ending with the first
denial of service attacks on controller. rediscovery message received from the new controller at its
southbound interface.
Measurement Units: Discussion:
Deviation of baseline metrics while handling Denial of Service This benchmark determines the impact of provisioning new flows
Attacks. when controllers are teamed together and the active controller
fails.
See Also: Measurement Units:
None Milliseconds.
2.3.4. Reliability 2.3.4.2. Network Re-provisioning Time
2.3.4.1. Controller Failover Time Definition:
The time taken by the controller to reroute traffic when there is
a failure in existing traffic paths, defined as the interval
starting with the first failure notification message received by
the controller and ending with the last flow re-provisioning
message sent by the controller at its southbound interface.
Definition: Discussion:
The time taken to switch from an active controller to the backup This benchmark determines the controller's re-provisioning ability
controller, when the controllers work in redundancy mode and the upon network failures and makes the following assumptions:
active controller fails, defined as the interval starting with the
active controller bringing down, ending with the first re-discovery
message received from the new controller at its Southbound
interface.
Discussion: 1. The network topology supports a redundant path between the
This benchmark determines the impact of provisioning new flows when source and destination endpoints.
controllers are teamed and the active controller fails.
Measurement Units: 2. The controller does not pre-provision the redundant path.
Milliseconds.
See Also: Measurement Units:
None Milliseconds.
2.3.4.2. Network Re-Provisioning Time 3. Test Setup
Definition: This section provides common reference topologies that are referred
The time taken to re-route the traffic by the Controller, when there to in individual tests defined in the companion methodology document
is a failure in existing traffic paths, defined as the interval [RFC8456].
starting from the first failure notification message received by the
controller, ending with the last flow re-provisioning message sent
by the controller at its Southbound interface.
Discussion: 3.1. Test Setup - Controller Operating in Standalone Mode
This benchmark determines the controller's re-provisioning ability
upon network failures. This benchmark test assumes the following:
1. Network topology supports redundant path between source and
destination endpoints.
2. Controller does not pre-provision the redundant path.
Measurement Units: +-----------------------------------------------------------+
Milliseconds. | Application-Plane Test Emulator |
| |
| +-----------------+ +-------------+ |
| | Application | | Service | |
| +-----------------+ +-------------+ |
| |
+-----------------------------+(I2)-------------------------+
|
| (Northbound Interface)
+-------------------------------+
| +----------------+ |
| | SDN Controller | |
| +----------------+ |
| |
| Device Under Test (DUT) |
+-------------------------------+
| (Southbound Interface)
|
+-----------------------------+(I1)-------------------------+
| |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 2 |--..-| Device n-1| |
| +-----------+ +-----------+ |
| / \ / \ |
| / \ / \ |
| l0 / X \ ln |
| / / \ \ |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 1 |..| Device n | |
| +-----------+ +-----------+ |
| | | |
| +---------------+ +---------------+ |
| | Test Traffic | | Test Traffic | |
| | Generator | | Generator | |
| | (TP1) | | (TP2) | |
| +---------------+ +---------------+ |
| |
| Forwarding-Plane Test Emulator |
+-----------------------------------------------------------+
See Also: Figure 1
None
3. Test Setup 3.2. Test Setup - Controller Operating in Cluster Mode
This section provides common reference topologies that are later +-----------------------------------------------------------+
referred to in individual tests defined in the companion methodology | Application-Plane Test Emulator |
document. | |
| +-----------------+ +-------------+ |
| | Application | | Service | |
| +-----------------+ +-------------+ |
| |
+-----------------------------+(I2)-------------------------+
|
| (Northbound Interface)
+---------------------------------------------------------+
| |
| +------------------+ +------------------+ |
| | SDN Controller 1 | <--E/W--> | SDN Controller n | |
| +------------------+ +------------------+ |
| |
| Device Under Test (DUT) |
+---------------------------------------------------------+
| (Southbound Interface)
|
+-----------------------------+(I1)-------------------------+
| |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 2 |--..-| Device n-1| |
| +-----------+ +-----------+ |
| / \ / \ |
| / \ / \ |
| l0 / X \ ln |
| / / \ \ |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 1 |..| Device n | |
| +-----------+ +-----------+ |
| | | |
| +---------------+ +---------------+ |
| | Test Traffic | | Test Traffic | |
| | Generator | | Generator | |
| | (TP1) | | (TP2) | |
| +---------------+ +---------------+ |
| |
| Forwarding-Plane Test Emulator |
+-----------------------------------------------------------+
3.1. Test setup - Controller working in Standalone Mode Figure 2
+-----------------------------------------------------------+ 4. Test Coverage
| Application Plane Test Emulator |
| |
| +-----------------+ +-------------+ |
| | Application | | Service | |
| +-----------------+ +-------------+ |
| |
+-----------------------------+(I2)-------------------------+
|
| (Northbound interface)
+-------------------------------+
| +----------------+ |
| | SDN Controller | |
| +----------------+ |
| |
| Device Under Test (DUT) |
+-------------------------------+
| (Southbound interface)
|
+-----------------------------+(I1)-------------------------+
| |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 2 |--..-| Device n-1| |
| +-----------+ +-----------+ |
| / \ / \ |
| / \ / \ |
| l0 / X \ ln |
| / / \ \ |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 1 |..| Device n | |
| +-----------+ +-----------+ |
| | | |
| +---------------+ +---------------+ |
| | Test Traffic | | Test Traffic | |
| | Generator | | Generator | |
| | (TP1) | | (TP2) | |
| +---------------+ +---------------+ |
| |
| Forwarding Plane Test Emulator |
+-----------------------------------------------------------+
Figure 1 +-------------------------------------------------------------------+
| Lifecycle | Speed | Scalability | Reliability |
+------------+-------------------+---------------+------------------+
| | 1. Network |1. Network | |
| | Topology | Discovery | |
| | Discovery | Size | |
| | Time | | |
| | | | |
| | 2. Reactive Path | | |
| | Provisioning | | |
| | Time | | |
| | | | |
| | 3. Proactive Path | | |
| Setup | Provisioning | | |
| | Time | | |
| | | | |
| | 4. Reactive Path | | |
| | Provisioning | | |
| | Rate | | |
| | | | |
| | 5. Proactive Path | | |
| | Provisioning | | |
| | Rate | | |
| | | | |
+------------+-------------------+---------------+------------------+
| | 1. Maximum |1. Control |1. Network |
| | Asynchronous | Sessions | Topology |
| | Message | Capacity | Change |
| | Processing Rate| | Detection Time |
| | |2. Forwarding | |
| | 2. Loss-Free | Table |2. Exception |
| | Asynchronous | Capacity | Handling |
| | Message | | |
| Operational| Processing Rate| |3. Handling |
| | | | Denial-of- |
| | 3. Asynchronous | | Service Attacks|
| | Message | | |
| | Processing Time| |4. Network |
| | | | Re-provisioning|
| | | | Time |
| | | | |
+------------+-------------------+---------------+------------------+
| Teardown | | |1. Controller |
| | | | Failover Time |
+------------+-------------------+---------------+------------------+
3.2. Test setup - Controller working in Cluster Mode 5. IANA Considerations
+-----------------------------------------------------------+ This document has no IANA actions.
| Application Plane Test Emulator |
| |
| +-----------------+ +-------------+ |
| | Application | | Service | |
| +-----------------+ +-------------+ |
| |
+-----------------------------+(I2)-------------------------+
|
| (Northbound interface)
+---------------------------------------------------------+
| |
| ------------------ ------------------ |
| | SDN Controller 1 | <--E/W--> | SDN Controller n | |
| ------------------ ------------------ |
| |
| Device Under Test (DUT) |
+---------------------------------------------------------+
| (Southbound interface)
|
+-----------------------------+(I1)-------------------------+
| |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 2 |--..-| Device n-1| |
| +-----------+ +-----------+ |
| / \ / \ |
| / \ / \ |
| l0 / X \ ln |
| / / \ \ |
| +-----------+ +-----------+ |
| | Network | | Network | |
| | Device 1 |..| Device n | |
| +-----------+ +-----------+ |
| | | |
| +---------------+ +---------------+ |
| | Test Traffic | | Test Traffic | |
| | Generator | | Generator | |
| | (TP1) | | (TP2) | |
| +---------------+ +---------------+ |
| |
| Forwarding Plane Test Emulator |
+-----------------------------------------------------------+
Figure 2 6. Security Considerations
4. Test Coverage The benchmarking tests described in this document are limited to the
performance characterization of controllers in a lab environment with
isolated networks.
+ -----------------------------------------------------------------+ The benchmarking network topology will be an independent test setup
| Lifecycle | Speed | Scalability | Reliability | and MUST NOT be connected to devices that may forward the test
+ -----------+-------------------+---------------+-----------------+ traffic into a production network or misroute traffic to the test
| | 1. Network Topolo-|1. Network | | management network.
| | -gy Discovery | Discovery | |
| | Time | Size | |
| | | | |
| | 2. Reactive Path | | |
| | Provisioning | | |
| | Time | | |
| | | | |
| | 3. Proactive Path | | |
| | Provisioning | | |
| Setup | Time | | |
| | | | |
| | 4. Reactive Path | | |
| | Provisioning | | |
| | Rate | | |
| | | | |
| | 5. Proactive Path | | |
| | Provisioning | | |
| | Rate | | |
| | | | |
+------------+-------------------+---------------+-----------------+
| | 1. Maximum |1. Control |1. Network |
| | Asynchronous | Sessions | Topology |
| | Message Proces-| Capacity | Change |
| | -sing Rate | | Detection Time|
| | |2. Forwarding | |
| | 2. Loss-Free | Table |2. Exception |
| | Asynchronous | Capacity | Handling |
| | Message Proces-| | |
| Operational| -sing Rate | |3. Denial of |
| | | | Service |
| | 3. Asynchronous | | Handling |
| | Message Proces-| | |
| | -sing Time | |4. Network Re- |
| | | | Provisioning |
| | | | Time |
| | | | |
+------------+-------------------+---------------+-----------------+
| | | | |
| Tear Down | | |1. Controller |
| | | | Failover Time |
+------------+-------------------+---------------+-----------------+
5. References Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the controller.
5.1. Normative References Special capabilities SHOULD NOT exist in the controller specifically
for benchmarking purposes. Any implications for network security
arising from the controller SHOULD be identical in the lab and in
production networks.
[RFC7426] E. Haleplidis, K. Pentikousis, S. Denazis, J. Hadi Salim, 7. Normative References
D. Meyer, O. Koufopavlou "Software-Defined Networking
(SDN): Layers and Architecture Terminology", RFC 7426,
January 2015.
[RFC4689] S. Poretsky, J. Perser, S. Erramilli, S. Khurana [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
"Terminology for Benchmarking Network-layer Traffic Requirement Levels", BCP 14, RFC 2119,
Control Mechanisms", RFC 4689, October 2006. DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC2330] V. Paxson, G. Almes, J. Mahdavi, M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. DOI 10.17487/RFC2330, May 1998,
<https://www.rfc-editor.org/info/rfc2330>.
[RFC2119] S. Bradner, "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997.
[RFC8174] B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", RFC 8174, May 2017.
[I-D.sdn-controller-benchmark-meth] Bhuvaneswaran.V, Anton Basil,
Mark.T, Vishwas Manral, Sarah Banks "Benchmarking
Methodology for SDN Controller Performance",
draft-ietf-bmwg-sdn-controller-benchmark-meth-09
(Work in progress), May 25, 2018
5.2. Informative References
[OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification"
Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.
6. IANA Considerations [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
"Terminology for Benchmarking Network-layer Traffic
Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689,
October 2006, <https://www.rfc-editor.org/info/rfc4689>.
This document does not have any IANA requests. [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S.,
Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software-
Defined Networking (SDN): Layers and Architecture
Terminology", RFC 7426, DOI 10.17487/RFC7426,
January 2015, <https://www.rfc-editor.org/info/rfc7426>.
7. Security Considerations [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
RFC 2119 Key Words", BCP 14, RFC 8174,
DOI 10.17487/RFC8174, May 2017,
<https://www.rfc-editor.org/info/rfc8174>.
Security issues are not discussed in this memo. [RFC8456] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
and S. Banks, "Benchmarking Methodology for Software-
Defined Networking (SDN) Controller Performance",
RFC 8456, DOI 10.17487/RFC8456, October 2018,
<https://www.rfc-editor.org/info/rfc8456>.
8. Acknowledgements Acknowledgments
The authors would like to acknowledge Al Morton (AT&T) for the The authors would like to acknowledge Al Morton (AT&T) for his
significant contributions to the earlier versions of this document. significant contributions to the earlier draft versions of this
The authors would like to thank the following individuals for document. The authors would like to thank the following individuals
providing their valuable comments to the earlier versions of this for providing their valuable comments to the earlier draft versions
document: Sandeep Gangadharan (HP), M. Georgescu (NAIST), Andrew of this document: Sandeep Gangadharan (HP), M. Georgescu (NAIST),
McGregor (Google), Scott Bradner , Jay Karthik (Cisco), Ramakrishnan Andrew McGregor (Google), Scott Bradner, Jay Karthik (Cisco),
(Dell), Khasanov Boris (Huawei). Ramki Krishnan (VMware), and Boris Khasanov (Huawei).
9. Authors' Addresses Authors' Addresses
Bhuvaneswaran Vengainathan Bhuvaneswaran Vengainathan
Veryx Technologies Inc. Veryx Technologies Inc.
1 International Plaza, Suite 550 1 International Plaza, Suite 550
Philadelphia Philadelphia, PA 19113
PA 19113 United States of America
Email: bhuvaneswaran.vengainathan@veryxtech.com Email: bhuvaneswaran.vengainathan@veryxtech.com
Anton Basil Anton Basil
Veryx Technologies Inc. Veryx Technologies Inc.
1 International Plaza, Suite 550 1 International Plaza, Suite 550
Philadelphia Philadelphia, PA 19113
PA 19113 United States of America
Email: anton.basil@veryxtech.com Email: anton.basil@veryxtech.com
Mark Tassinari Mark Tassinari
Hewlett-Packard, Hewlett Packard Enterprise
8000 Foothills Blvd, 8000 Foothills Blvd.
Roseville, CA 95747 Roseville, CA 95747
United States of America
Email: mark.tassinari@hpe.com Email: mark.tassinari@hpe.com
Vishwas Manral Vishwas Manral
Nano Sec,CA NanoSec Co
3350 Thomas Rd.
Santa Clara, CA 95054
United States of America
Email: vishwas.manral@gmail.com Email: vishwas.manral@gmail.com
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
930 De Guigne Drive, 930 De Guigne Drive
Sunnyvale, CA Sunnyvale, CA 94085
United States of America
Email: sbanks@encrypted.net Email: sbanks@encrypted.net
 End of changes. 190 change blocks. 
816 lines changed or deleted 761 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/