draft-ietf-bmwg-sdn-controller-benchmark-term-09.txt   draft-ietf-bmwg-sdn-controller-benchmark-term-10.txt 
Internet-Draft Bhuvaneswaran Vengainathan Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies Intended Status: Informational Veryx Technologies
Expires: August 25, 2018 Mark Tassinari Expires: November 25, 2018 Mark Tassinari
Hewlett-Packard Hewlett-Packard
Vishwas Manral Vishwas Manral
Nano Sec Nano Sec
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
February 25, 2018 May 25, 2018
Terminology for Benchmarking SDN Controller Performance Terminology for Benchmarking SDN Controller Performance
draft-ietf-bmwg-sdn-controller-benchmark-term-09 draft-ietf-bmwg-sdn-controller-benchmark-term-10
Abstract Abstract
This document defines terminology for benchmarking an SDN This document defines terminology for benchmarking an SDN
controller's control plane performance. It extends the terminology controller's control plane performance. It extends the terminology
already defined in RFC 7426 for the purpose of benchmarking SDN already defined in RFC 7426 for the purpose of benchmarking SDN
controllers. The terms provided in this document help to benchmark controllers. The terms provided in this document help to benchmark
SDN controller's performance independent of the controller's SDN controller's performance independent of the controller's
supported protocols and/or network services. A mechanism for supported protocols and/or network services.
benchmarking the performance of SDN controllers is defined in the
companion methodology document I-D sdn-controller-benchmark-meth.
These two documents provide a standard mechanism to measure and
evaluate the performance of various controller implementations.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current. Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as reference at any time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress. material or to cite them other than as "work in progress.
This Internet-Draft will expire on August 25, 2018. This Internet-Draft will expire on November 25, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 4, line 14 skipping to change at page 4, line 14
1. Introduction 1. Introduction
Software Defined Networking (SDN) is a networking architecture in Software Defined Networking (SDN) is a networking architecture in
which network control is decoupled from the underlying forwarding which network control is decoupled from the underlying forwarding
function and is placed in a centralized location called the SDN function and is placed in a centralized location called the SDN
controller. The SDN controller provides an abstraction of the controller. The SDN controller provides an abstraction of the
underlying network and offers a global view of the overall network underlying network and offers a global view of the overall network
to applications and business logic. Thus, an SDN controller provides to applications and business logic. Thus, an SDN controller provides
the flexibility to program, control, and manage network behaviour the flexibility to program, control, and manage network behaviour
dynamically through standard interfaces. Since the network controls dynamically through northbound and southbound interfaces. Since the
are logically centralized, the need to benchmark the SDN controller network controls are logically centralized, the need to benchmark
performance becomes significant. This document defines terms to the SDN controller performance becomes significant. This document
benchmark various controller designs for performance, scalability, defines terms to benchmark various controller designs for
reliability and security, independent of northbound and southbound performance, scalability, reliability and security, independent of
protocols. northbound and southbound protocols. A mechanism for benchmarking
the performance of SDN controllers is defined in the companion
methodology document [I-D.sdn-controller-benchmark-meth]. These two
documents provide a method to measure and evaluate the performance
of various controller implementations.
Conventions used in this document Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in "OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
2. Term Definitions 2. Term Definitions
2.1. SDN Terms 2.1. SDN Terms
The terms defined in this section are extensions to the terms The terms defined in this section are extensions to the terms
defined in [RFC7426] "Software-Defined Networking (SDN): Layers and defined in [RFC7426] "Software-Defined Networking (SDN): Layers and
Architecture Terminology". This RFC should be referred before Architecture Terminology". That RFC should be referred before
attempting to make use of this document. attempting to make use of this document.
2.1.1. Flow 2.1.1. Flow
Definition: Definition:
The definition of Flow is same as microflows defined in [RFC4689] The definition of Flow is same as microflows defined in [RFC4689]
Section 3.1.5. Section 3.1.5.
Discussion: Discussion:
A flow can be set of packets having same source address, destination A flow can be set of packets having same source address, destination
skipping to change at page 9, line 34 skipping to change at page 9, line 34
2.2.1. Number of Network Devices 2.2.1. Number of Network Devices
Definition: Definition:
The number of Network Devices present in the defined test topology. The number of Network Devices present in the defined test topology.
Discussion: Discussion:
The Network Devices defined in the test topology can be deployed The Network Devices defined in the test topology can be deployed
using real hardware or emulated in hardware platforms. using real hardware or emulated in hardware platforms.
Measurement Units: Measurement Units:
N/A Number of network devices
See Also: See Also:
None None
2.2.2. Trial Repetition 2.2.2. Trial Repetition
Definition: Definition:
The number of times the test needs to be repeated. The number of times the test needs to be repeated.
Discussion: Discussion:
The test needs to be repeated for multiple iterations to obtain a The test needs to be repeated for multiple iterations to obtain a
reliable metric. It is recommended that this test SHOULD be reliable metric. It is recommended that this test SHOULD be
performed for at least 10 iterations to increase the confidence in performed for at least 10 iterations to increase the confidence in
measured result. measured result.
Measurement Units: Measurement Units:
N/A Number of trials
See Also: See Also:
None None
2.2.3. Trial Duration 2.2.3. Trial Duration
Definition: Definition:
Defines the duration of test trials for each iteration. Defines the duration of test trials for each iteration.
Discussion: Discussion:
Trial duration forms the basis for stop criteria for benchmarking Trial duration forms the basis for stop criteria for benchmarking
tests. Trials not completed within this time interval is considered tests. Trials not completed within this time interval is considered
as incomplete. as incomplete.
Measurement Units: Measurement Units:
seconds Seconds
See Also: See Also:
None None
2.2.4. Number of Cluster nodes 2.2.4. Number of Cluster nodes
Definition: Definition:
Defines the number of controllers present in the controller cluster. Defines the number of controllers present in the controller cluster.
Discussion: Discussion:
This parameter is relevant when testing the controller performance This parameter is relevant when testing the controller performance
in clustering/teaming mode. The number of nodes in the cluster MUST in clustering/teaming mode. The number of nodes in the cluster MUST
be greater than 1. be greater than 1.
Measurement Units: Measurement Units:
N/A Number of controller nodes
See Also: See Also:
None None
2.3. Benchmarking Terms 2.3. Benchmarking Terms
This section defines metrics for benchmarking the SDN controller. This section defines metrics for benchmarking the SDN controller.
The procedure to perform the defined metrics is defined in the The procedure to perform the defined metrics is defined in the
accompanying methodology document[I-D.sdn-controller-benchmark-meth] accompanying methodology document[I-D.sdn-controller-benchmark-meth]
skipping to change at page 11, line 26 skipping to change at page 11, line 26
Network topology discovery is key for the SDN controller to Network topology discovery is key for the SDN controller to
provision and manage the network. So it is important to measure how provision and manage the network. So it is important to measure how
quickly the controller discovers the topology to learn the current quickly the controller discovers the topology to learn the current
network state. This benchmark is obtained by presenting a network network state. This benchmark is obtained by presenting a network
topology (Tree, Mesh or Linear) with the given number of nodes to topology (Tree, Mesh or Linear) with the given number of nodes to
the controller and wait for the discovery process to complete. It is the controller and wait for the discovery process to complete. It is
expected that the controller supports network discovery mechanism expected that the controller supports network discovery mechanism
and uses protocol messages for its discovery process. and uses protocol messages for its discovery process.
Measurement Units: Measurement Units:
milliseconds Milliseconds
See Also: See Also:
None None
2.3.1.2. Asynchronous Message Processing Time 2.3.1.2. Asynchronous Message Processing Time
Definition: Definition:
The time taken by controller(s) to process an asynchronous message, The time taken by controller(s) to process an asynchronous message,
defined as the interval starting with an asynchronous message from a defined as the interval starting with an asynchronous message from a
network device after the discovery of all the devices by the network device after the discovery of all the devices by the
controller(s), ending with a response message from the controller(s) controller(s), ending with a response message from the controller(s)
at its Southbound interface. at its Southbound interface.
Discussion: Discussion:
For SDN to support dynamic network provisioning, it is important to For SDN to support dynamic network provisioning, it is important to
measure how quickly the controller responds to an event triggered measure how quickly the controller responds to an event triggered
from the network. The event could be any notification messages from the network. The event could be any notification messages
generated by an Network Device upon arrival of a new flow, link down generated by a Network Device upon arrival of a new flow, link down
etc. This benchmark is obtained by sending asynchronous messages etc. This benchmark is obtained by sending asynchronous messages
from every connected Network Devices one at a time for the defined from every connected Network Devices one at a time for the defined
trial duration. This test assumes that the controller will respond trial duration. This test assumes that the controller will respond
to the received asynchronous message. to the received asynchronous message.
Measurement Units: Measurement Units:
milliseconds Milliseconds
See Also: See Also:
None None
2.3.1.3. Asynchronous Message Processing Rate 2.3.1.3. Asynchronous Message Processing Rate
Definition: Definition:
The number responses to asynchronous messages (such as new flow The number responses to asynchronous messages per second (such as
arrival notification message, etc.) for which the controller(s) new flow arrival notification message, link down, etc.) for which
performed processing and replied with a valid and productive (non- the controller(s) performed processing and replied with a valid and
trivial) response message. productive (non-trivial) response message.
Discussion: Discussion:
As SDN assures flexible network and agile provisioning, it is As SDN assures flexible network and agile provisioning, it is
important to measure how many network events the controller can important to measure how many network events (such as new flow
handle at a time. This benchmark is obtained by sending asynchronous arrival notification message, link down, etc.) the controller can
handle at a time. This benchmark is measured by sending asynchronous
messages from every connected Network Device at the rate that the messages from every connected Network Device at the rate that the
controller processes (without dropping them). This test assumes that controller processes (without dropping them). This test assumes that
the controller responds to all the received asynchronous messages the controller responds to all the received asynchronous messages
(the messages can be designed to elicit individual responses). (the messages can be designed to elicit individual responses).
When sending asynchronous messages to the controller(s) at high When sending asynchronous messages to the controller(s) at high
rates, some messages or responses may be discarded or corrupted and rates, some messages or responses may be discarded or corrupted and
require retransmission to controller(s). Therefore, a useful require retransmission to controller(s). Therefore, a useful
qualification on Asynchronous Message Processing Rate is whether the qualification on Asynchronous Message Processing Rate is whether the
in-coming message count equals the response count in each trial. in-coming message count equals the response count in each trial.
skipping to change at page 13, line 36 skipping to change at page 13, line 36
Discussion: Discussion:
As SDN supports agile provisioning, it is important to measure how As SDN supports agile provisioning, it is important to measure how
fast that the controller provisions an end-to-end flow in the fast that the controller provisions an end-to-end flow in the
dataplane. The benchmark is obtained by sending traffic from a dataplane. The benchmark is obtained by sending traffic from a
source endpoint to the destination endpoint, finding the time source endpoint to the destination endpoint, finding the time
difference between the first and the last flow provisioning message difference between the first and the last flow provisioning message
exchanged between the controller and the Network Devices for the exchanged between the controller and the Network Devices for the
traffic path. traffic path.
Measurement Units: Measurement Units:
milliseconds. Milliseconds.
See Also: See Also:
None None
2.3.1.5. Proactive Path Provisioning Time 2.3.1.5. Proactive Path Provisioning Time
Definition: Definition:
The time taken by the controller to proactively setup a path between The time taken by the controller to proactively setup a path between
source and destination node, defined as the interval starting with source and destination node, defined as the interval starting with
the first proactive flow provisioned in the controller(s) at its the first proactive flow provisioned in the controller(s) at its
skipping to change at page 14, line 15 skipping to change at page 14, line 15
For SDN to support pre-provisioning of traffic path from For SDN to support pre-provisioning of traffic path from
application, it is important to measure how fast that the controller application, it is important to measure how fast that the controller
provisions an end-to-end flow in the dataplane. The benchmark is provisions an end-to-end flow in the dataplane. The benchmark is
obtained by provisioning a flow on controller's northbound interface obtained by provisioning a flow on controller's northbound interface
for the traffic to reach from a source to a destination endpoint, for the traffic to reach from a source to a destination endpoint,
finding the time difference between the first and the last flow finding the time difference between the first and the last flow
provisioning message exchanged between the controller and the provisioning message exchanged between the controller and the
Network Devices for the traffic path. Network Devices for the traffic path.
Measurement Units: Measurement Units:
milliseconds. Milliseconds.
See Also: See Also:
None None
2.3.1.6. Reactive Path Provisioning Rate 2.3.1.6. Reactive Path Provisioning Rate
Definition: Definition:
The maximum number of independent paths a controller can The maximum number of independent paths a controller can
concurrently establish per second between source and destination concurrently establish per second between source and destination
nodes reactively, defined as the number of paths provisioned per nodes reactively, defined as the number of paths provisioned per
skipping to change at page 15, line 34 skipping to change at page 15, line 34
2.3.1.8. Network Topology Change Detection Time 2.3.1.8. Network Topology Change Detection Time
Definition: Definition:
The amount of time required for the controller to detect any changes The amount of time required for the controller to detect any changes
in the network topology, defined as the interval starting with the in the network topology, defined as the interval starting with the
notification message received by the controller(s) at its Southbound notification message received by the controller(s) at its Southbound
interface, ending with the first topology rediscovery messages sent interface, ending with the first topology rediscovery messages sent
from the controller(s) at its Southbound interface. from the controller(s) at its Southbound interface.
Discussion: Discussion:
In order to for the controller to support fast network failure In order for the controller to support fast network failure
recovery, it is critical to measure how fast the controller is able recovery, it is critical to measure how fast the controller is able
to detect any network-state change events. This benchmark is to detect any network-state change events. This benchmark is
obtained by triggering a topology change event and measuring the obtained by triggering a topology change event and measuring the
time controller takes to detect and initiate a topology re-discovery time controller takes to detect and initiate a topology re-discovery
process. process.
Measurement Units: Measurement Units:
milliseconds Milliseconds
See Also: See Also:
None None
2.3.2. Scalability 2.3.2. Scalability
2.3.2.1. Control Sessions Capacity 2.3.2.1. Control Sessions Capacity
Definition: Definition:
Measure the maximum number of control sessions the controller can Measure the maximum number of control sessions the controller can
skipping to change at page 16, line 25 skipping to change at page 16, line 25
Discussion: Discussion:
Measuring the controller's control sessions capacity is important to Measuring the controller's control sessions capacity is important to
determine the controller's system and bandwidth resource determine the controller's system and bandwidth resource
requirements. This benchmark is obtained by establishing control requirements. This benchmark is obtained by establishing control
session with the controller from each of the Network Device until it session with the controller from each of the Network Device until it
fails. The number of sessions that were successfully established fails. The number of sessions that were successfully established
will provide the Control Sessions Capacity. will provide the Control Sessions Capacity.
Measurement Units: Measurement Units:
N/A Maximum number of control sessions
See Also: See Also:
None None
2.3.2.2. Network Discovery Size 2.3.2.2. Network Discovery Size
Definition: Definition:
Measure the network size (number of nodes and links) that a Measure the network size (number of nodes and links) that a
controller can discover, defined as the size of a network that the controller can discover, defined as the size of a network that the
controller(s) can discover, starting from a network topology given controller(s) can discover, starting from a network topology given
skipping to change at page 16, line 48 skipping to change at page 16, line 48
Discussion: Discussion:
For optimal network planning, it is key to measure the maximum For optimal network planning, it is key to measure the maximum
network size that the controller can discover. This benchmark is network size that the controller can discover. This benchmark is
obtained by presenting an initial set of Network Devices for obtained by presenting an initial set of Network Devices for
discovery to the controller. Based on the initial discovery, the discovery to the controller. Based on the initial discovery, the
number of Network Devices is increased or decreased to determine the number of Network Devices is increased or decreased to determine the
maximum nodes that the controller can discover. maximum nodes that the controller can discover.
Measurement Units: Measurement Units:
N/A Maximum number of network nodes and links
See Also: See Also:
None None
2.3.2.3. Forwarding Table Capacity 2.3.2.3. Forwarding Table Capacity
Definition: Definition:
The maximum number of flow entries that a controller can manage in The maximum number of flow entries that a controller can manage in
its Forwarding table. its Forwarding table.
skipping to change at page 17, line 46 skipping to change at page 17, line 46
on performance tests. on performance tests.
Discussion: Discussion:
This benchmark test is to be performed after obtaining the baseline This benchmark test is to be performed after obtaining the baseline
performance of the performance tests defined in Section 2.3.1. This performance of the performance tests defined in Section 2.3.1. This
benchmark determines the deviation from the baseline performance due benchmark determines the deviation from the baseline performance due
to the handling of error or failure messages from the connected to the handling of error or failure messages from the connected
Network Devices. Network Devices.
Measurement Units: Measurement Units:
N/A Deviation of baseline metrics while handling Exceptions.
See Also: See Also:
None None
2.3.3.2. Denial of Service Handling 2.3.3.2. Denial of Service Handling
Definition: Definition:
To determine the effect of handling denial of service (DoS) attacks To determine the effect of handling denial of service (DoS) attacks
on performance and scalability tests. on performance and scalability tests.
Discussion: Discussion:
This benchmark test is to be performed after obtaining the baseline This benchmark test is to be performed after obtaining the baseline
performance of the performance and scalability tests defined in performance of the performance and scalability tests defined in
section 2.3.1 and section 2.3.1. This benchmark determines the section 2.3.1 and section 2.3.2. This benchmark determines the
deviation from the baseline performance due to the handling of deviation from the baseline performance due to the handling of
denial of service attacks on controller. denial of service attacks on controller.
Measurement Units: Measurement Units:
Deviation of baseline metrics while handling Denial of Service Deviation of baseline metrics while handling Denial of Service
Attacks. Attacks.
See Also: See Also:
None None
skipping to change at page 18, line 38 skipping to change at page 18, line 38
Definition: Definition:
The time taken to switch from an active controller to the backup The time taken to switch from an active controller to the backup
controller, when the controllers work in redundancy mode and the controller, when the controllers work in redundancy mode and the
active controller fails, defined as the interval starting with the active controller fails, defined as the interval starting with the
active controller bringing down, ending with the first re-discovery active controller bringing down, ending with the first re-discovery
message received from the new controller at its Southbound message received from the new controller at its Southbound
interface. interface.
Discussion: Discussion:
This benchmark determine the impact of provisioning new flows when This benchmark determines the impact of provisioning new flows when
controllers are teamed and the active controller fails. controllers are teamed and the active controller fails.
Measurement Units: Measurement Units:
milliseconds. Milliseconds.
See Also: See Also:
None None
2.3.4.2. Network Re-Provisioning Time 2.3.4.2. Network Re-Provisioning Time
Definition: Definition:
The time taken to re-route the traffic by the Controller, when there The time taken to re-route the traffic by the Controller, when there
is a failure in existing traffic paths, defined as the interval is a failure in existing traffic paths, defined as the interval
starting from the first failure notification message received by the starting from the first failure notification message received by the
controller, ending with the last flow re-provisioning message sent controller, ending with the last flow re-provisioning message sent
by the controller at its Southbound interface . by the controller at its Southbound interface.
Discussion: Discussion:
This benchmark determines the controller's re-provisioning ability This benchmark determines the controller's re-provisioning ability
upon network failures. This benchmark test assumes the following: upon network failures. This benchmark test assumes the following:
1. Network topology supports redundant path between source and 1. Network topology supports redundant path between source and
destination endpoints. destination endpoints.
2. Controller does not pre-provision the redundant path. 2. Controller does not pre-provision the redundant path.
Measurement Units: Measurement Units:
milliseconds. Milliseconds.
See Also: See Also:
None None
3. Test Setup 3. Test Setup
This section provides common reference topologies that are later This section provides common reference topologies that are later
referred to in individual tests defined in the companion methodology referred to in individual tests defined in the companion methodology
document. document.
skipping to change at page 23, line 31 skipping to change at page 23, line 31
[RFC2119] S. Bradner, "Key words for use in RFCs to Indicate [RFC2119] S. Bradner, "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
[RFC8174] B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC [RFC8174] B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", RFC 8174, May 2017. 2119 Key Words", RFC 8174, May 2017.
[I-D.sdn-controller-benchmark-meth] Bhuvaneswaran.V, Anton Basil, [I-D.sdn-controller-benchmark-meth] Bhuvaneswaran.V, Anton Basil,
Mark.T, Vishwas Manral, Sarah Banks "Benchmarking Mark.T, Vishwas Manral, Sarah Banks "Benchmarking
Methodology for SDN Controller Performance", Methodology for SDN Controller Performance",
draft-ietf-bmwg-sdn-controller-benchmark-meth-08 draft-ietf-bmwg-sdn-controller-benchmark-meth-09
(Work in progress), February 25, 2018 (Work in progress), May 25, 2018
5.2. Informative References 5.2. Informative References
[OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" [OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification"
Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.
6. IANA Considerations 6. IANA Considerations
This document does not have any IANA requests. This document does not have any IANA requests.
 End of changes. 29 change blocks. 
43 lines changed or deleted 44 lines changed or added

This html diff was produced by rfcdiff 1.46. The latest version is available from http://tools.ietf.org/tools/rfcdiff/