< draft-ietf-bmwg-sdn-controller-benchmark-meth-06.txt   draft-ietf-bmwg-sdn-controller-benchmark-meth-07.txt >
Internet-Draft Bhuvaneswaran Vengainathan Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies Intended Status: Informational Veryx Technologies
Expires: May 16, 2018 Mark Tassinari Expires: June 10, 2018 Mark Tassinari
Hewlett-Packard Hewlett-Packard
Vishwas Manral Vishwas Manral
Nano Sec Nano Sec
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
November 16, 2017 January 10, 2018
Benchmarking Methodology for SDN Controller Performance Benchmarking Methodology for SDN Controller Performance
draft-ietf-bmwg-sdn-controller-benchmark-meth-06 draft-ietf-bmwg-sdn-controller-benchmark-meth-07
Abstract Abstract
This document defines the methodologies for benchmarking control This document defines the methodologies for benchmarking control
plane performance of SDN controllers. Terminology related to plane performance of SDN controllers. Terminology related to
benchmarking SDN controllers is described in the companion benchmarking SDN controllers is described in the companion
terminology document. SDN controllers have been implemented with terminology document. SDN controllers have been implemented with
many varying designs in order to achieve their intended network many varying designs in order to achieve their intended network
functionality. Hence, the authors have taken the approach of functionality. Hence, the authors have taken the approach of
considering an SDN controller as a black box, defining the considering an SDN controller as a black box, defining the
skipping to change at page 1, line 45 skipping to change at page 1, line 45
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current. Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as reference at any time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress. material or to cite them other than as "work in progress.
This Internet-Draft will expire on May 16, 2018. This Internet-Draft will expire on June 10, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License. warranty as described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction...................................................4 1. Introduction...................................................4
2. Scope..........................................................4 2. Scope..........................................................4
3. Test Setup.....................................................5 3. Test Setup..........................Error! Bookmark not defined.
3.1. Test setup - Controller working in Standalone Mode........5 3.1. Test setup - Controller working in Standalone Mode........5
3.2. Test setup - Controller working in Cluster Mode...........6 3.2. Test setup - Controller working in Cluster Mode...........6
4. Test Considerations............................................7 4. Test Considerations............................................7
4.1. Network Topology..........................................7 4.1. Network Topology..........................................7
4.2. Test Traffic..............................................7 4.2. Test Traffic..............................................7
4.3. Test Emulator Requirements................................7 4.3. Test Emulator Requirements................................7
4.4. Connection Setup..........................................7 4.4. Connection Setup..........................................7
4.5. Measurement Point Specification and Recommendation........8 4.5. Measurement Point Specification and Recommendation........8
4.6. Connectivity Recommendation...............................8 4.6. Connectivity Recommendation...............................8
4.7. Test Repeatability........................................8 4.7. Test Repeatability........................................8
skipping to change at page 4, line 38 skipping to change at page 4, line 38
This document defines methodology to measure the networking metrics This document defines methodology to measure the networking metrics
of SDN controllers. For the purpose of this memo, the SDN controller of SDN controllers. For the purpose of this memo, the SDN controller
is a function that manages and controls Network Devices. Any SDN is a function that manages and controls Network Devices. Any SDN
controller without a control capability is out of scope for this controller without a control capability is out of scope for this
memo. The tests defined in this document enable benchmarking of SDN memo. The tests defined in this document enable benchmarking of SDN
Controllers in two ways; as a standalone controller and as a cluster Controllers in two ways; as a standalone controller and as a cluster
of homogeneous controllers. These tests are recommended for of homogeneous controllers. These tests are recommended for
execution in lab environments rather than in live network execution in lab environments rather than in live network
deployments. Performance benchmarking of a federation of controllers deployments. Performance benchmarking of a federation of controllers
is beyond the scope of this document. Test Setup is beyond the scope of this document.
3. Test Setup
The tests defined in this document enable measurement of an SDN The tests defined in this document enable measurement of an SDN
controllers performance in standalone mode and cluster mode. This controller's performance in standalone mode and cluster mode. This
section defines common reference topologies that are later referred section defines common reference topologies that are later referred
to in individual tests (Additional forwarding Plane topologies are to in individual tests (Additional forwarding Plane topologies are
provided in Appendix A). provided in Appendix A).
3. Test Setup
3.1. Test setup - Controller working in Standalone Mode 3.1. Test setup - Controller working in Standalone Mode
+-----------------------------------------------------------+ +-----------------------------------------------------------+
| Application Plane Test Emulator | | Application Plane Test Emulator |
| | | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | Application | | Service | | | | Application | | Service | |
| +-----------------+ +-------------+ | | +-----------------+ +-------------+ |
| | | |
+-----------------------------+(I2)-------------------------+ +-----------------------------+(I2)-------------------------+
skipping to change at page 7, line 28 skipping to change at page 7, line 28
includes two sample test topologies, defined in Section 10 - includes two sample test topologies, defined in Section 10 -
Appendix A for reference. Further, care should be taken to make sure Appendix A for reference. Further, care should be taken to make sure
that a loop prevention mechanism is enabled either in the SDN that a loop prevention mechanism is enabled either in the SDN
controller, or in the network when the topology contains redundant controller, or in the network when the topology contains redundant
network paths. network paths.
4.2. Test Traffic 4.2. Test Traffic
Test traffic is used to notify the controller about the asynchronous Test traffic is used to notify the controller about the asynchronous
arrival of new flows. The test cases SHOULD use frame sizes of 128, arrival of new flows. The test cases SHOULD use frame sizes of 128,
512 and 1508 bytes for benchmarking. Testing using jumbo frames are 512 and 1508 bytes for benchmarking. Tests using jumbo frames are
optional. optional.
4.3. Test Emulator Requirements 4.3. Test Emulator Requirements
The Test Emulator SHOULD time stamp the transmitted and received The Test Emulator SHOULD time stamp the transmitted and received
control messages to/from the controller on the established network control messages to/from the controller on the established network
connections. The test cases use these values to compute the connections. The test cases use these values to compute the
controller processing time. controller processing time.
4.4. Connection Setup 4.4. Connection Setup
There may be controller implementations that support unencrypted and There may be controller implementations that support unencrypted and
encrypted network connections with Network Devices. Further, the encrypted network connections with Network Devices. Further, the
controller may have backward compatibility with Network Devices controller may have backward compatibility with Network Devices
running older versions of southbound protocols. It may be useful to running older versions of southbound protocols. It may be useful to
measure the controller performance be measured with one or more measure the controller performance with one or more applicable
applicable connection setup methods defined below. connection setup methods defined below.
1. Unencrypted connection with Network Devices, running same 1. Unencrypted connection with Network Devices, running same
protocol version. protocol version.
2. Unencrypted connection with Network Devices, running different 2. Unencrypted connection with Network Devices, running different
protocol versions. protocol versions.
Example: Example:
a. Controller running current protocol version and switch a. Controller running current protocol version and switch
running older protocol version running older protocol version
b. Controller running older protocol version and switch b. Controller running older protocol version and switch
skipping to change at page 10, line 37 skipping to change at page 10, line 37
Devices. Devices.
3. Record the time for the first discovery message (Tm1) received 3. Record the time for the first discovery message (Tm1) received
from the controller at forwarding plane test emulator interface from the controller at forwarding plane test emulator interface
I1. I1.
4. Query the controller every 3 seconds to obtain the discovered 4. Query the controller every 3 seconds to obtain the discovered
network topology information through the northbound interface or network topology information through the northbound interface or
the management interface and compare it with the deployed network the management interface and compare it with the deployed network
topology information. topology information.
5. Stop the trial when the discovered topology information matches 5. Stop the trial when the discovered topology information matches
the deployed network topology, or when the discovered topology the deployed network topology, or when the discovered topology
information for 3 consecutive queries return the same details. information return the same details for 3 consecutive queries.
6. Record the time last discovery message (Tmn) sent to controller 6. Record the time last discovery message (Tmn) sent to controller
from the forwarding plane test emulator interface (I1) when the from the forwarding plane test emulator interface (I1) when the
trial completed successfully. (e.g., the topology matches). trial completed successfully. (e.g., the topology matches).
Measurement: Measurement:
Topology Discovery Time Tr1 = Tmn-Tm1. Topology Discovery Time Tr1 = Tmn-Tm1.
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Topology Discovery Time = ----------------------- Average Topology Discovery Time = -----------------------
skipping to change at page 13, line 36 skipping to change at page 13, line 36
(P%),and the maximum possible message sending rate (Ntx1/Td). (P%),and the maximum possible message sending rate (Ntx1/Td).
Procedure: Procedure:
1. Generate asynchronous messages continuously at the maximum 1. Generate asynchronous messages continuously at the maximum
possible rate on the established connections from all the possible rate on the established connections from all the
emulated/simulated Network Devices for the given trial Duration emulated/simulated Network Devices for the given trial Duration
(Td). (Td).
2. Record the total number of responses received from the controller 2. Record the total number of responses received from the controller
(Nrx1) as well as the number of messages sent (Ntx1) to the (Nrx1) as well as the number of messages sent (Ntx1) to the
controller within the trial duration(Td). controller within the trial duration (Td).
3. Calculate the Asynchronous Message Processing Rate (Tr1) and 3. Calculate the Asynchronous Message Processing Rate (Tr1) and
the Message Loss Ratio (Lr1). Ensure that the controller(s) have the Message Loss Ratio (Lr1). Ensure that the controller(s) have
returned to normal operation. returned to normal operation.
4. Repeat the trial by reducing the asynchronous message sending rate 4. Repeat the trial by reducing the asynchronous message sending rate
used in last trial by the STEP size. used in last trial by the STEP size.
5. Continue repeating the trials and reducing the sending rate until 5. Continue repeating the trials and reducing the sending rate until
both the maximum value of Nrxn and the Nrxn corresponding to zero both the maximum value of Nrxn and the Nrxn corresponding to zero
loss ratio have been found. loss ratio have been found.
6. The trials corresponding to the benchmark levels MUST be repeated 6. The trials corresponding to the benchmark levels MUST be repeated
using the same asynchronous message rates until the responses using the same asynchronous message rates until the responses
skipping to change at page 16, line 17 skipping to change at page 16, line 17
Reactive Path Provisioning Time Tr1 = Tdf1-Tsf1. Reactive Path Provisioning Time Tr1 = Tdf1-Tsf1.
Tr1 + Tr2 + Tr3 .. Trn Tr1 + Tr2 + Tr3 .. Trn
Average Reactive Path Provisioning Time = ----------------------- Average Reactive Path Provisioning Time = -----------------------
Total Trials Total Trials
Reporting Format: Reporting Format:
The Reactive Path Provisioning Time results MUST be reported in the The Reactive Path Provisioning Time results MUST be reported in the
format of a table with a row for each iteration. The last row of the format of a table with a row for each iteration. The last row of the
table indicates the Average Reactive Path Provisioning Time table indicates the Average Reactive Path Provisioning Time.
The report should capture the following information in addition to The report should capture the following information in addition to
the configuration parameters captured in section 5. the configuration parameters captured in section 5.
- Number of Network Devices in the path - Number of Network Devices in the path
5.1.5. Proactive Path Provisioning Time 5.1.5. Proactive Path Provisioning Time
Objective: Objective:
The time taken by the controller to setup a path proactively between The time taken by the controller to setup a path proactively between
source and destination node, defined as the interval starting with source and destination node, defined as the interval starting with
the first proactive flow provisioned in the controller(s) at its the first proactive flow provisioned in the controller(s) at its
Northbound interface, ending with the last flow provisioning Northbound interface, ending with the last flow provisioning
response message sent from the controller(s) at it Southbound response message sent from the controller(s) at its Southbound
interface. interface.
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document in combination with Appendix A. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST contain the network topology information for 1. The controller MUST contain the network topology information for
skipping to change at page 19, line 21 skipping to change at page 19, line 21
5.1.7. Proactive Path Provisioning Rate 5.1.7. Proactive Path Provisioning Rate
Objective: Objective:
Measure the maximum rate of independent paths a controller can Measure the maximum rate of independent paths a controller can
concurrently establish between source and destination nodes concurrently establish between source and destination nodes
proactively, defined as the number of paths provisioned by the proactively, defined as the number of paths provisioned by the
controller(s) at its Southbound interface for the paths requested in controller(s) at its Southbound interface for the paths requested in
its Northbound interface between the start of the test and the its Northbound interface between the start of the test and the
expiry of given trial duration . The measurement is based on expiry of given trial duration. The measurement is based on
dataplane observations of successful path activation dataplane observations of successful path activation
Reference Test Setup: Reference Test Setup:
The test SHOULD use one of the test setups described in section 3.1 The test SHOULD use one of the test setups described in section 3.1
or section 3.2 of this document in combination with Appendix A. or section 3.2 of this document in combination with Appendix A.
Prerequisite: Prerequisite:
1. The controller MUST contain the network topology information for 1. The controller MUST contain the network topology information for
skipping to change at page 24, line 28 skipping to change at page 24, line 28
interface. interface.
Procedure: Procedure:
Reactive Flow Provisioning Mode: Reactive Flow Provisioning Mode:
1. Send bi-directional traffic continuously with unique source and/or 1. Send bi-directional traffic continuously with unique source and/or
destination addresses from test traffic generators TP1 and TP2 at destination addresses from test traffic generators TP1 and TP2 at
the asynchronous message processing rate of controller. the asynchronous message processing rate of controller.
2. Query the controller at a regular interval (e.g., 5 seconds) for 2. Query the controller at a regular interval (e.g., 5 seconds) for
the number of learnt flow entries from its northbound interface. the number of learned flow entries from its northbound interface.
3. Stop the trial when the retrieved value is constant for three 3. Stop the trial when the retrieved value is constant for three
consecutive iterations and record the value received from the last consecutive iterations and record the value received from the last
query (Nrp). query (Nrp).
Proactive Flow Provisioning Mode: Proactive Flow Provisioning Mode:
1. Install unique flows continuously through controller's northbound 1. Install unique flows continuously through controller's northbound
or management interface until a failure response is received from or management interface until a failure response is received from
the controller. the controller.
2. Record the total number of successful responses (Nrp). 2. Record the total number of successful responses (Nrp).
skipping to change at page 25, line 7 skipping to change at page 25, line 7
bi-directional traffic for the provisioned flows. bi-directional traffic for the provisioned flows.
Measurement: Measurement:
Proactive Flow Provisioning Mode: Proactive Flow Provisioning Mode:
Max Flow Entries = Total number of flows provisioned (Nrp) Max Flow Entries = Total number of flows provisioned (Nrp)
Reactive Flow Provisioning Mode: Reactive Flow Provisioning Mode:
Max Flow Entries = Total number of learnt flow entries (Nrp) Max Flow Entries = Total number of learned flow entries (Nrp)
Forwarding Table Capacity = Max Flow Entries. Forwarding Table Capacity = Max Flow Entries.
Reporting Format: Reporting Format:
The Forwarding Table Capacity results MUST be tabulated with the The Forwarding Table Capacity results MUST be tabulated with the
following information in addition to the configuration parameters following information in addition to the configuration parameters
captured in section 5. captured in section 5.
- Provisioning Type (Proactive/Reactive) - Provisioning Type (Proactive/Reactive)
skipping to change at page 28, line 35 skipping to change at page 28, line 35
Prerequisite: Prerequisite:
1. Master controller election MUST be completed. 1. Master controller election MUST be completed.
2. Nodes are connected to the controller cluster as per the 2. Nodes are connected to the controller cluster as per the
Redundancy Mode (RM). Redundancy Mode (RM).
3. The controller cluster should have successfully completed the 3. The controller cluster should have successfully completed the
network topology discovery. network topology discovery.
4. The Network Device MUST send all new flows to the controller when 4. The Network Device MUST send all new flows to the controller when
it receives from the test traffic generator. it receives from the test traffic generator.
5. Controller should have learnt the location of destination (D1) at 5. Controller should have learned the location of destination (D1) at
test traffic generator TP2. test traffic generator TP2.
Procedure: Procedure:
1. Send uni-directional traffic continuously with incremental 1. Send uni-directional traffic continuously with incremental
sequence number and source addresses from test traffic generator sequence number and source addresses from test traffic generator
TP1 at the rate that the controller processes without any drops. TP1 at the rate that the controller processes without any drops.
2. Ensure that there are no packet drops observed at the test traffic 2. Ensure that there are no packet drops observed at the test traffic
generator TP2. generator TP2.
3. Bring down the active controller. 3. Bring down the active controller.
skipping to change at page 31, line 17 skipping to change at page 31, line 17
- Reverse Direction Packet Loss - Reverse Direction Packet Loss
6. References 6. References
6.1. Normative References 6.1. Normative References
[I-D.sdn-controller-benchmark-term] Bhuvaneswaran.V, Anton Basil, [I-D.sdn-controller-benchmark-term] Bhuvaneswaran.V, Anton Basil,
Mark.T, Vishwas Manral, Sarah Banks, "Terminology for Mark.T, Vishwas Manral, Sarah Banks, "Terminology for
Benchmarking SDN Controller Performance", Benchmarking SDN Controller Performance",
draft-ietf-bmwg-sdn-controller-benchmark-term-06 draft-ietf-bmwg-sdn-controller-benchmark-term-07
(Work in progress), November 16, 2017 (Work in progress), January 10, 2018
6.2. Informative References 6.2. Informative References
[OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" [OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification"
Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. Version 1.4.0 (Wire Protocol 0x05), October 14, 2013.
7. IANA Considerations 7. IANA Considerations
This document does not have any IANA requests. This document does not have any IANA requests.
8. Security Considerations 8. Security Considerations
Benchmarking tests described in this document are limited to the Benchmarking tests described in this document are limited to the
performance characterization of controller in lab environment with performance characterization of controllers in a lab environment
isolated network. with isolated network.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network, or misroute traffic to the test traffic into a production network, or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the controller. solely on measurements observable external to the controller.
Special capabilities SHOULD NOT exist in the controller specifically Special capabilities SHOULD NOT exist in the controller specifically
for benchmarking purposes. Any implications for network security for benchmarking purposes. Any implications for network security
arising from the controller SHOULD be identical in the lab and in arising from the controller SHOULD be identical in the lab and in
production networks production networks.
9. Acknowledgments 9. Acknowledgments
The authors would like to thank the following individuals for The authors would like to thank the following individuals for
providing their valuable comments to the earlier versions of this providing their valuable comments to the earlier versions of this
document: Al Morton (AT&T), Sandeep Gangadharan (HP), M. Georgescu document: Al Morton (AT&T), Sandeep Gangadharan (HP), M. Georgescu
(NAIST), Andrew McGregor (Google), Scott Bradner (Harvard (NAIST), Andrew McGregor (Google), Scott Bradner (Harvard
University), Jay Karthik (Cisco), Ramakrishnan (Dell), Khasanov University), Jay Karthik (Cisco), Ramakrishnan (Dell), Khasanov
Boris (Huawei), Brian Castelli (Spirent) Boris (Huawei), Brian Castelli (Spirent)
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
Appendix A. Example Test Topologies Appendix A. Example Test Topologies
A.1. Leaf-Spine Topology - Three Tier Network Architecture A.1. Leaf-Spine Topology - Three Tier Network Architecture
+----------+ +----------+
| SDN | | SDN |
| Node | (Core) | Node | (Core)
+----------+ +----------+
/ \ / \
/ \ / \
+------+ +------+ +------+ +------+
 End of changes. 23 change blocks. 
27 lines changed or deleted 27 lines changed or added

This html diff was produced by rfcdiff 1.46. The latest version is available from http://tools.ietf.org/tools/rfcdiff/