draft-ietf-bmwg-sdn-controller-benchmark-term-04.txt   draft-ietf-bmwg-sdn-controller-benchmark-term-05.txt 
Internet-Draft Bhuvaneswaran Vengainathan Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies Intended Status: Informational Veryx Technologies
Expires: December 29, 2017 Mark Tassinari Expires: April 01, 2018 Mark Tassinari
Hewlett-Packard Hewlett-Packard
Vishwas Manral Vishwas Manral
Nano Sec Nano Sec
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
June 29, 2017 October 01, 2017
Terminology for Benchmarking SDN Controller Performance Terminology for Benchmarking SDN Controller Performance
draft-ietf-bmwg-sdn-controller-benchmark-term-04 draft-ietf-bmwg-sdn-controller-benchmark-term-05
Abstract Abstract
This document defines terminology for benchmarking an SDN This document defines terminology for benchmarking an SDN
controller's control plane performance. It extends the terminology controller's control plane performance. It extends the terminology
already defined in RFC 7426 for the purpose of benchmarking SDN already defined in RFC 7426 for the purpose of benchmarking SDN
controllers. The terms provided in this document help to benchmark controllers. The terms provided in this document help to benchmark
SDN controller's performance independent of the controller's SDN controller's performance independent of the controller's
supported protocols and/or network services. A mechanism for supported protocols and/or network services. A mechanism for
benchmarking the performance of SDN controllers is defined in the benchmarking the performance of SDN controllers is defined in the
skipping to change at page 1, line 44 skipping to change at page 1, line 44
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current. Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as reference at any time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress. material or to cite them other than as "work in progress.
This Internet-Draft will expire on December 29, 2017. This Internet-Draft will expire on April 01, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 37 skipping to change at page 2, line 37
2.1.3. Controller Forwarding Table..........................5 2.1.3. Controller Forwarding Table..........................5
2.1.4. Proactive Flow Provisioning Mode.....................5 2.1.4. Proactive Flow Provisioning Mode.....................5
2.1.5. Reactive Flow Provisioning Mode......................6 2.1.5. Reactive Flow Provisioning Mode......................6
2.1.6. Path.................................................6 2.1.6. Path.................................................6
2.1.7. Standalone Mode......................................6 2.1.7. Standalone Mode......................................6
2.1.8. Cluster/Redundancy Mode..............................7 2.1.8. Cluster/Redundancy Mode..............................7
2.1.9. Asynchronous Message.................................7 2.1.9. Asynchronous Message.................................7
2.1.10. Test Traffic Generator..............................8 2.1.10. Test Traffic Generator..............................8
2.2. Test Configuration/Setup Terms............................8 2.2. Test Configuration/Setup Terms............................8
2.2.1. Number of Network Devices............................8 2.2.1. Number of Network Devices............................8
2.2.2. Test Iterations......................................8 2.2.2. Trails...............................................8
2.2.3. Test Duration........................................9 2.2.3. Trail Duration.......................................9
2.2.4. Number of Cluster nodes..............................9 2.2.4. Number of Cluster nodes..............................9
2.3. Benchmarking Terms.......................................10 2.3. Benchmarking Terms.......................................10
2.3.1. Performance.........................................10 2.3.1. Performance.........................................10
2.3.1.1. Network Topology Discovery Time................10 2.3.1.1. Network Topology Discovery Time................10
2.3.1.2. Asynchronous Message Processing Time...........10 2.3.1.2. Asynchronous Message Processing Time...........10
2.3.1.3. Asynchronous Message Processing Rate...........11 2.3.1.3. Asynchronous Message Processing Rate...........11
2.3.1.4. Reactive Path Provisioning Time................11 2.3.1.4. Reactive Path Provisioning Time................12
2.3.1.5. Proactive Path Provisioning Time...............12 2.3.1.5. Proactive Path Provisioning Time...............12
2.3.1.6. Reactive Path Provisioning Rate................12 2.3.1.6. Reactive Path Provisioning Rate................13
2.3.1.7. Proactive Path Provisioning Rate...............13 2.3.1.7. Proactive Path Provisioning Rate...............14
2.3.1.8. Network Topology Change Detection Time.........13 2.3.1.8. Network Topology Change Detection Time.........14
2.3.2. Scalability.........................................14
2.3.2.1. Control Sessions Capacity......................14 2.3.2. Scalability.........................................15
2.3.2.1. Control Sessions Capacity......................15
2.3.2.2. Network Discovery Size.........................15 2.3.2.2. Network Discovery Size.........................15
2.3.2.3. Forwarding Table Capacity......................15 2.3.2.3. Forwarding Table Capacity......................16
2.3.3. Security............................................16 2.3.3. Security............................................16
2.3.3.1. Exception Handling.............................16 2.3.3.1. Exception Handling.............................16
2.3.3.2. Denial of Service Handling.....................16 2.3.3.2. Denial of Service Handling.....................17
2.3.4. Reliability.........................................17 2.3.4. Reliability.........................................17
2.3.4.1. Controller Failover Time.......................17 2.3.4.1. Controller Failover Time.......................17
2.3.4.2. Network Re-Provisioning Time...................17 2.3.4.2. Network Re-Provisioning Time...................18
3. Test Setup....................................................18 3. Test Setup....................................................18
3.1. Test setup - Controller working in Standalone Mode.......18 3.1. Test setup - Controller working in Standalone Mode.......18
3.2. Test setup - Controller working in Cluster Mode..........19 3.2. Test setup - Controller working in Cluster Mode..........19
4. Test Coverage.................................................20 4. Test Coverage.................................................20
5. References....................................................21 5. References....................................................21
5.1. Normative References.....................................21 5.1. Normative References.....................................21
5.2. Informative References...................................21 5.2. Informative References...................................22
6. IANA Considerations...........................................21 6. IANA Considerations...........................................22
7. Security Considerations.......................................22 7. Security Considerations.......................................22
8. Acknowledgements..............................................22 8. Acknowledgements..............................................22
9. Authors' Addresses............................................22 9. Authors' Addresses............................................22
1. Introduction 1. Introduction
Software Defined Networking (SDN) is a networking architecture in Software Defined Networking (SDN) is a networking architecture in
which network control is decoupled from the underlying forwarding which network control is decoupled from the underlying forwarding
function and is placed in a centralized location called the SDN function and is placed in a centralized location called the SDN
controller. The SDN controller abstracts the underlying network and controller. The SDN controller abstracts the underlying network and
offers a global view of the overall network to applications and offers a global view of the overall network to applications and
business logic. Thus, an SDN controller provides the flexibility to business logic. Thus, an SDN controller provides the flexibility to
program, control, and manage network behaviour dynamically through program, control, and manage network behaviour dynamically through
standard interfaces. Since the network controls are logically standard interfaces. Since the network controls are logically
centralized, the need to benchmark the SDN controller performance centralized, the need to benchmark the SDN controller performance
becomes significant. This document defines terms to benchmark becomes significant. This document defines terms to benchmark
various controller designs for performance, scalability, reliability various controller designs for performance, scalability, reliability
and security, independent of northbound and southbound protocols. and security, independent of northbound and southbound protocols.
The methodologies are defined in [I-D.sdn-controller-benchmark-meth].
Conventions used in this document Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119. document are to be interpreted as described in RFC 2119.
2. Term Definitions 2. Term Definitions
2.1. SDN Terms 2.1. SDN Terms
skipping to change at page 5, line 9 skipping to change at page 5, line 9
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
None None
2.1.2. Northbound Interface 2.1.2. Northbound Interface
Definition: Definition:
The definition of northbound interface is same Service Interface The definition of northbound interface is same Service Interface
defined in [RFC7426]. defined in [RFC7426] .
Discussion: Discussion:
The northbound interface allows SDN applications and orchestration The northbound interface allows SDN applications and orchestration
systems to program and retrieve the network information through the systems to program and retrieve the network information through the
SDN controller. SDN controller.
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
skipping to change at page 6, line 33 skipping to change at page 6, line 33
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
None None
2.1.6. Path 2.1.6. Path
Definition: Definition:
Refer to Section 5 in [RFC2330]. Refer to Section 5 in [RFC2330]
Discussion: Discussion:
None None
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
None None
skipping to change at page 8, line 43 skipping to change at page 8, line 43
Discussion: Discussion:
The Network Devices defined in the test topology can be deployed The Network Devices defined in the test topology can be deployed
using real hardware or emulated in hardware platforms. using real hardware or emulated in hardware platforms.
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
None None
2.2.2. Test Iterations 2.2.2. Trails
Definition: Definition:
The number of times the test needs to be repeated. The number of times the test needs to be repeated.
Discussion: Discussion:
The test needs to be repeated for multiple iterations to obtain a The test needs to be repeated for multiple iterations to obtain a
reliable metric. It is recommended that this test SHOULD be reliable metric. It is recommended that this test SHOULD be
performed for at least 10 iterations to increase the confidence in performed for at least 10 iterations to increase the confidence in
measured result. measured result.
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
None None
2.2.3. Test Duration 2.2.3. Trail Duration
Definition: Definition:
Defines the duration of test trails for each iteration. Defines the duration of test trails for each iteration.
Discussion: Discussion:
Test duration forms the basis for stop criteria for benchmarking Trail duration forms the basis for stop criteria for benchmarking
tests. Test not completed within this time interval is considered as tests. Trail not completed within this time interval is considered
incomplete. as incomplete.
Measurement Units: Measurement Units:
seconds seconds
See Also: See Also:
None None
2.2.4. Number of Cluster nodes 2.2.4. Number of Cluster nodes
Definition: Definition:
skipping to change at page 10, line 9 skipping to change at page 10, line 9
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
None None
2.3. Benchmarking Terms 2.3. Benchmarking Terms
This section defines metrics for benchmarking the SDN controller. This section defines metrics for benchmarking the SDN controller.
The procedure to perform the defined metrics is defined in the The procedure to perform the defined metrics is defined in the
accompanying methodology document. accompanying methodology document [I-D.sdn-controller-benchmark-meth]
2.3.1. Performance 2.3.1. Performance
2.3.1.1. Network Topology Discovery Time 2.3.1.1. Network Topology Discovery Time
Definition: Definition:
The time taken by controller(s) to determine the complete network The time taken by controller(s) to determine the complete network
topology, defined as the interval starting with the first discovery topology, defined as the interval starting with the first discovery
message from the controller(s) at its Southbound interface, ending message from the controller(s) at its Southbound interface, ending
with all features of the static topology determined. with all features of the static topology determined.
skipping to change at page 11, line 7 skipping to change at page 11, line 8
controller(s), ending with a response message from the controller(s) controller(s), ending with a response message from the controller(s)
at its Southbound interface. at its Southbound interface.
Discussion: Discussion:
For SDN to support dynamic network provisioning, it is important to For SDN to support dynamic network provisioning, it is important to
measure how quickly the controller responds to an event triggered measure how quickly the controller responds to an event triggered
from the network. The event could be any notification messages from the network. The event could be any notification messages
generated by an Network Device upon arrival of a new flow, link down generated by an Network Device upon arrival of a new flow, link down
etc. This benchmark is obtained by sending asynchronous messages etc. This benchmark is obtained by sending asynchronous messages
from every connected Network Devices one at a time for the defined from every connected Network Devices one at a time for the defined
test duration. This test assumes that the controller will respond to trail duration. This test assumes that the controller will respond
the received asynchronous message. to the received asynchronous message.
Measurement Units: Measurement Units:
milliseconds milliseconds
See Also: See Also:
None None
2.3.1.3. Asynchronous Message Processing Rate 2.3.1.3. Asynchronous Message Processing Rate
Definition: Definition:
The maximum number of asynchronous messages (session aliveness check The number responses to asynchronous messages (such as new flow
message, new flow arrival notification message etc.) that the arrival notification message, etc.) for which the controller(s)
controller(s) can process, defined as the iteration starting with performed processing and replied with a valid and productive (non-
sending asynchronous messages to the controller (s) at the maximum trivial) response message.
possible rate and ending with an iteration that the controller(s)
processes the received asynchronous messages without dropping.
Discussion: Discussion:
As SDN assures flexible network and agile provisioning, it is As SDN assures flexible network and agile provisioning, it is
important to measure how many network events that the controller can important to measure how many network events that the controller can
handle at a time. This benchmark is obtained by sending asynchronous handle at a time. This benchmark is obtained by sending asynchronous
messages from every connected Network Devices at the rate that the messages from every connected Network Devices at the rate that the
controller processes without dropping. This test assumes that the controller processes without dropping. This test assumes that the
controller will respond to all the received asynchronous messages. controller responds to all the received asynchronous messages (the
messages can be designed to elicit individual responses).
When sending asynchronous messages to the controller(s) at high
rates, some messages or responses may be discarded or corrupted and
require retransmission to controller(s). Therefore, a useful
qualification on Asynchronous Message Processing Rate is whether the
in-coming message count equals the response count in each trial.
This is called the Loss-free Asynchronous Message Processing Rate.
Note that several of the early controller benchmarking tools did not
consider lost messages, and instead report the maximum response
rate. This is called the Maximum Asynchronous Message Processing
Rate.
To characterize both the Loss-free and Maximum Rates, a test could
begin the first trial by sending asynchronous messages to the
controller (s) at the maximum possible rate and record the message
reply rate and the message loss rate. The message sending rate is
then decreased by the step-size. The message reply rate and the
message loss rate are recorded. The test ends with a trial where the
controller(s) processes the all asynchronous messages sent without
loss. This is the Loss-free Asynchronous Message Processing Rate.
The trial where the controller(s) produced the maximum response rate
is the Maximum Asynchronous Message Processing Rate. Of course, the
first trial could begin at a low sending rate with zero lost
responses, and increase until the Loss-free and Maximum Rates are
discovered.
Measurement Units: Measurement Units:
Messages processed per second. Messages processed per second.
See Also: See Also:
None None
2.3.1.4. Reactive Path Provisioning Time 2.3.1.4. Reactive Path Provisioning Time
Definition: Definition:
skipping to change at page 13, line 5 skipping to change at page 13, line 33
None None
2.3.1.6. Reactive Path Provisioning Rate 2.3.1.6. Reactive Path Provisioning Rate
Definition: Definition:
The maximum number of independent paths a controller can The maximum number of independent paths a controller can
concurrently establish between source and destination nodes concurrently establish between source and destination nodes
reactively, defined as the number of paths provisioned by the reactively, defined as the number of paths provisioned by the
controller(s) at its Southbound interface for the flow provisioning controller(s) at its Southbound interface for the flow provisioning
requests received for path provisioning at its Southbound interface requests received for path provisioning at its Southbound interface
between the start of the test and the expiry of given test duration between the start of the trail and the expiry of given trail
duration
Discussion: Discussion:
For SDN to support agile traffic forwarding, it is important to For SDN to support agile traffic forwarding, it is important to
measure how many end-to-end flows that the controller could setup in measure how many end-to-end flows that the controller could setup in
the dataplane. This benchmark is obtained by sending traffic each the dataplane. This benchmark is obtained by sending traffic each
with unique source and destination pairs from the source Network with unique source and destination pairs from the source Network
Device and determine the number of frames received at the Device and determine the number of frames received at the
destination Network Device. destination Network Device.
Measurement Units: Measurement Units:
skipping to change at page 13, line 28 skipping to change at page 14, line 12
See Also: See Also:
None None
2.3.1.7. Proactive Path Provisioning Rate 2.3.1.7. Proactive Path Provisioning Rate
Definition: Definition:
Measure the maximum number of independent paths a controller can Measure the maximum number of independent paths a controller can
concurrently establish between source and destination nodes concurrently establish between source and destination nodes
proactively, defined as the number of paths provisioned by the proactively, defined as the number of paths provisioned by the
controller(s) at its Southbound interface for the paths provisioned controller(s) at its Southbound interface for the paths provisioned
in its Northbound interface between the start of the test and the in its Northbound interface between the start of the trail and the
expiry of given test duration expiry of given trail duration
Discussion: Discussion:
For SDN to support pre-provisioning of traffic path for a larger For SDN to support pre-provisioning of traffic path for a larger
network from the application, it is important to measure how many network from the application, it is important to measure how many
end-to-end flows that the controller could setup in the dataplane. end-to-end flows that the controller could setup in the dataplane.
This benchmark is obtained by sending traffic each with unique This benchmark is obtained by sending traffic each with unique
source and destination pairs from the source Network Device. Program source and destination pairs from the source Network Device. Program
the flows on controller's northbound interface for traffic to reach the flows on controller's northbound interface for traffic to reach
from each of the unique source and destination pairs and determine from each of the unique source and destination pairs and determine
the number of frames received at the destination Network Device. the number of frames received at the destination Network Device.
skipping to change at page 21, line 27 skipping to change at page 22, line 8
"Terminology for Benchmarking Network-layer Traffic "Terminology for Benchmarking Network-layer Traffic
Control Mechanisms", RFC 4689, October 2006. Control Mechanisms", RFC 4689, October 2006.
[RFC2330] V. Paxson, G. Almes, J. Mahdavi, M. Mathis, [RFC2330] V. Paxson, G. Almes, J. Mahdavi, M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. May 1998.
[I-D.sdn-controller-benchmark-meth] Bhuvaneswaran.V, Anton Basil, [I-D.sdn-controller-benchmark-meth] Bhuvaneswaran.V, Anton Basil,
Mark.T, Vishwas Manral, Sarah Banks "Benchmarking Mark.T, Vishwas Manral, Sarah Banks "Benchmarking
Methodology for SDN Controller Performance", Methodology for SDN Controller Performance",
draft-ietf-bmwg-sdn-controller-benchmark-meth-04 draft-ietf-bmwg-sdn-controller-benchmark-meth-05
(Work in progress), June 8, 2017 (Work in progress), October 1, 2017
5.2. Informative References 5.2. Informative References
[OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" [OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification"
Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.
6. IANA Considerations 6. IANA Considerations
This document does not have any IANA requests. This document does not have any IANA requests.
 End of changes. 24 change blocks. 
40 lines changed or deleted 67 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/