draft-ietf-bmwg-traffic-management-06.txt   rfc7640.txt 
Network Working Group B. Constantine
Internet Draft JDSU
Intended status: Informational R. Krishnan
Expires: February 2016 Dell Inc.
June 9, 2015
Traffic Management Benchmarking Internet Engineering Task Force (IETF) B. Constantine
draft-ietf-bmwg-traffic-management-06.txt Request for Comments: 7640 JDSU
Category: Informational R. Krishnan
ISSN: 2070-1721 Dell Inc.
September 2015
Status of this Memo Traffic Management Benchmarking
This Internet-Draft is submitted in full conformance with the Abstract
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This framework describes a practical methodology for benchmarking the
Task Force (IETF). Note that other groups may also distribute traffic management capabilities of networking devices (i.e.,
working documents as Internet-Drafts. The list of current Internet- policing, shaping, etc.). The goals are to provide a repeatable test
Drafts is at http://datatracker.ietf.org/drafts/current/. method that objectively compares performance of the device's traffic
management capabilities and to specify the means to benchmark traffic
management with representative application traffic.
Internet-Drafts are draft documents valid for a maximum of six months Status of This Memo
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 9, 2015. This document is not an Internet Standards Track specification; it is
published for informational purposes.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc7640.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Abstract
This framework describes a practical methodology for benchmarking the
traffic management capabilities of networking devices (i.e. policing,
shaping, etc.). The goal is to provide a repeatable test method that
objectively compares performance of the device's traffic management
capabilities and to specify the means to benchmark traffic management
with representative application traffic.
Table of Contents Table of Contents
1. Introduction...................................................4 1. Introduction ....................................................3
1.1. Traffic Management Overview...............................4 1.1. Traffic Management Overview ................................3
1.2. DUT Lab Configuration and Testing Overview................5 1.2. Lab Configuration and Testing Overview .....................5
2. Conventions used in this document..............................7 2. Conventions Used in This Document ...............................6
3. Scope and Goals................................................8 3. Scope and Goals .................................................7
4. Traffic Benchmarking Metrics...................................9 4. Traffic Benchmarking Metrics ...................................10
4.1. Metrics for Stateless Traffic Tests......................10 4.1. Metrics for Stateless Traffic Tests .......................10
4.2. Metrics for Stateful Traffic Tests.......................11 4.2. Metrics for Stateful Traffic Tests ........................12
5. Tester Capabilities...........................................12 5. Tester Capabilities ............................................13
5.1. Stateless Test Traffic Generation........................13 5.1. Stateless Test Traffic Generation .........................13
5.1.1. Burst Hunt with Stateless Traffic...................13 5.1.1. Burst Hunt with Stateless Traffic ..................14
5.2. Stateful Test Pattern Generation.........................13 5.2. Stateful Test Pattern Generation ..........................14
5.2.1. TCP Test Pattern Definitions........................15 5.2.1. TCP Test Pattern Definitions .......................15
6. Traffic Benchmarking Methodology..............................16 6. Traffic Benchmarking Methodology ...............................17
6.1. Policing Tests...........................................16 6.1. Policing Tests ............................................17
6.1.1 Policer Individual Tests................................17 6.1.1. Policer Individual Tests ...........................18
6.1.2 Policer Capacity Tests..............................18 6.1.2. Policer Capacity Tests .............................19
6.1.2.1 Maximum Policers on Single Physical Port..........19 6.1.2.1. Maximum Policers on Single Physical Port ..20
6.1.2.2 Single Policer on All Physical Ports..............20 6.1.2.2. Single Policer on All Physical Ports ......22
6.1.2.3 Maximum Policers on All Physical Ports............21 6.1.2.3. Maximum Policers on All Physical Ports ....22
6.2. Queue/Scheduler Tests....................................21 6.2. Queue/Scheduler Tests .....................................23
6.2.1 Queue/Scheduler Individual Tests........................21 6.2.1. Queue/Scheduler Individual Tests ...................23
6.2.1.1 Testing Queue/Scheduler with Stateless Traffic....21 6.2.1.1. Testing Queue/Scheduler with
6.2.1.2 Testing Queue/Scheduler with Stateful Traffic.....23 Stateless Traffic .........................23
6.2.2 Queue / Scheduler Capacity Tests......................25 6.2.1.2. Testing Queue/Scheduler with
6.2.2.1 Multiple Queues / Single Port Active..............25 Stateful Traffic ..........................25
6.2.2.1.1 Strict Priority on Egress Port..................26 6.2.2. Queue/Scheduler Capacity Tests .....................28
6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ).....26 6.2.2.1. Multiple Queues, Single Port Active .......28
6.2.2.2 Single Queue per Port / All Ports Active..........27 6.2.2.1.1. Strict Priority on
6.2.2.3 Multiple Queues per Port, All Ports Active........27 Egress Port ....................28
6.3. Shaper tests.............................................28 6.2.2.1.2. Strict Priority + WFQ on
6.3.1 Shaper Individual Tests...............................28 Egress Port ....................29
6.3.1.1 Testing Shaper with Stateless Traffic.............29 6.2.2.2. Single Queue per Port, All Ports Active ...30
6.3.1.2 Testing Shaper with Stateful Traffic..............30 6.2.2.3. Multiple Queues per Port, All
6.3.2 Shaper Capacity Tests.................................32 Ports Active ..............................31
6.3.2.1 Single Queue Shaped, All Physical Ports Active....32 6.3. Shaper Tests ..............................................32
6.3.2.2 All Queues Shaped, Single Port Active.............32 6.3.1. Shaper Individual Tests ............................32
6.3.2.3 All Queues Shaped, All Ports Active...............33 6.3.1.1. Testing Shaper with Stateless Traffic .....33
6.4. Concurrent Capacity Load Tests...........................34 6.3.1.2. Testing Shaper with Stateful Traffic ......34
7. Security Considerations.......................................34 6.3.2. Shaper Capacity Tests ..............................36
8. IANA Considerations...........................................34 6.3.2.1. Single Queue Shaped, All Physical
9. References....................................................35 Ports Active ..............................37
9.1. Normative References.....................................35 6.3.2.2. All Queues Shaped, Single Port Active .....37
9.2. Informative References...................................35 6.3.2.3. All Queues Shaped, All Ports Active .......39
Appendix A: Open Source Tools for Traffic Management Testing.....36
Appendix B: Stateful TCP Test Patterns...........................37
Acknowledgments..................................................41
Authors' Addresses...............................................42
1. Introduction 6.4. Concurrent Capacity Load Tests ............................40
7. Security Considerations ........................................40
8. References .....................................................41
8.1. Normative References ......................................41
8.2. Informative References ....................................42
Appendix A. Open Source Tools for Traffic Management Testing ......44
Appendix B. Stateful TCP Test Patterns ............................45
Acknowledgments ...................................................51
Authors' Addresses ................................................51
Traffic management (i.e. policing, shaping, etc.) is an increasingly 1. Introduction
Traffic management (i.e., policing, shaping, etc.) is an increasingly
important component when implementing network Quality of Service important component when implementing network Quality of Service
(QoS). (QoS).
There is currently no framework to benchmark these features There is currently no framework to benchmark these features, although
although some standards address specific areas which are described some standards address specific areas as described in Section 1.1.
in Section 1.1.
This draft provides a framework to conduct repeatable traffic This document provides a framework to conduct repeatable traffic
management benchmarks for devices and systems in a lab environment. management benchmarks for devices and systems in a lab environment.
Specifically, this framework defines the methods to characterize Specifically, this framework defines the methods to characterize the
the capacity of the following traffic management features in network capacity of the following traffic management features in network
devices; classification, policing, queuing / scheduling, and devices: classification, policing, queuing/scheduling, and traffic
traffic shaping. shaping.
This benchmarking framework can also be used as a test procedure to This benchmarking framework can also be used as a test procedure to
assist in the tuning of traffic management parameters before service assist in the tuning of traffic management parameters before service
activation. In addition to Layer 2/3 (Ethernet / IP) benchmarking, activation. In addition to Layer 2/3 (Ethernet/IP) benchmarking,
Layer 4 (TCP) test patterns are proposed by this draft in order to Layer 4 (TCP) test patterns are proposed by this document in order to
more realistically benchmark end-user traffic. more realistically benchmark end-user traffic.
1.1. Traffic Management Overview 1.1. Traffic Management Overview
In general, a device with traffic management capabilities performs In general, a device with traffic management capabilities performs
the following functions: the following functions:
- Traffic classification: identifies traffic according to various - Traffic classification: identifies traffic according to various
configuration rules (for example IEEE 802.1Q Virtual LAN (VLAN), configuration rules (for example, IEEE 802.1Q Virtual LAN (VLAN),
Differential Services Code Point (DSCP) etc.) and marks this traffic Differentiated Services Code Point (DSCP)) and marks this traffic
internally to the network device. Multiple external priorities internally to the network device. Multiple external priorities
(DSCP, 802.1p, etc.) can map to the same priority in the device. (DSCP, 802.1p, etc.) can map to the same priority in the device.
- Traffic policing: limits the rate of traffic that enters a network
device according to the traffic classification. If the traffic
exceeds the provisioned limits, the traffic is either dropped or
remarked and forwarded onto to the next network device
- Traffic Scheduling: provides traffic classification within the
network device by directing packets to various types of queues and
applies a dispatching algorithm to assign the forwarding sequence
of packets
- Traffic shaping: a traffic control technique that actively buffers
and smooths the output rate in an attempt to adapt bursty traffic
to the configured limits
- Active Queue Management (AQM): AQM involves monitoring the status
of internal queues and proactively dropping (or remarking) packets,
which causes hosts using congestion-aware protocols to back-off and
in turn alleviate queue congestion [AQM-RECO]. On the other hand,
classic traffic management techniques reactively drop (or remark)
packets based on queue full condition. The benchmarking scenarios
for AQM are different and is outside of the scope of this testing
framework.
Even though AQM is outside of scope of this framework, it should be - Traffic policing: limits the rate of traffic that enters a network
noted that the TCP metrics and TCP test patterns (defined in Sections device according to the traffic classification. If the traffic
4.2 and 5.2, respectively) could be useful to test new AQM exceeds the provisioned limits, the traffic is either dropped or
algorithms (targeted to alleviate buffer bloat). Examples of these remarked and forwarded onto the next network device.
algorithms include code1 and pie (draft-ietf-aqm-code1 and
draft-ietf-aqm-pie). - Traffic scheduling: provides traffic classification within the
network device by directing packets to various types of queues and
applies a dispatching algorithm to assign the forwarding sequence
of packets.
- Traffic shaping: controls traffic by actively buffering and
smoothing the output rate in an attempt to adapt bursty traffic to
the configured limits.
- Active Queue Management (AQM): involves monitoring the status of
internal queues and proactively dropping (or remarking) packets,
which causes hosts using congestion-aware protocols to "back off"
and in turn alleviate queue congestion [RFC7567]. On the other
hand, classic traffic management techniques reactively drop (or
remark) packets based on queue-full conditions. The benchmarking
scenarios for AQM are different and are outside the scope of this
testing framework.
Even though AQM is outside the scope of this framework, it should be
noted that the TCP metrics and TCP test patterns (defined in
Sections 4.2 and 5.2, respectively) could be useful to test new AQM
algorithms (targeted to alleviate "bufferbloat"). Examples of these
algorithms include Controlled Delay [CoDel] and Proportional Integral
controller Enhanced [PIE].
The following diagram is a generic model of the traffic management The following diagram is a generic model of the traffic management
capabilities within a network device. It is not intended to capabilities within a network device. It is not intended to
represent all variations of manufacturer traffic management represent all variations of manufacturer traffic management
capabilities, but provide context to this test framework. capabilities, but it provides context for this test framework.
|----------| |----------------| |--------------| |----------| |----------| |----------------| |--------------| |----------|
| | | | | | | | | | | | | | | |
|Interface | |Ingress Actions | |Egress Actions| |Interface | |Interface | |Ingress Actions | |Egress Actions| |Interface |
|Input | |(classification,| |(scheduling, | |Output | |Ingress | |(classification,| |(scheduling, | |Egress |
|Queues | | marking, | | shaping, | |Queues | |Queues | | marking, | | shaping, | |Queues |
| |-->| policing or |-->| active queue |-->| | | |-->| policing, or |-->| active queue |-->| |
| | | shaping) | | management | | | | | | shaping) | | management, | | |
| | | | | remarking) | | | | | | | | remarking) | | |
|----------| |----------------| |--------------| |----------| |----------| |----------------| |--------------| |----------|
Figure 1: Generic Traffic Management capabilities of a Network Device Figure 1: Generic Traffic Management Capabilities of a Network Device
Ingress actions such as classification are defined in [RFC4689] Ingress actions such as classification are defined in [RFC4689] and
and include IP addresses, port numbers, DSCP, etc. In terms of include IP addresses, port numbers, and DSCP. In terms of marking,
marking, [RFC2697] and [RFC2698] define a single rate and dual rate, [RFC2697] and [RFC2698] define a Single Rate Three Color Marker and a
three color marker, respectively. Two Rate Three Color Marker, respectively.
The Metro Ethernet Forum (MEF) specifies policing and shaping in The Metro Ethernet Forum (MEF) specifies policing and shaping in
terms of Ingress and Egress Subscriber/Provider Conditioning terms of ingress and egress subscriber/provider conditioning
Functions in MEF12.1 [MEF-12.1]; Ingress and Bandwidth Profile functions as described in MEF 12.2 [MEF-12.2], as well as ingress and
attributes in MEF10.2 [MEF-10.2] and MEF 26 [MEF-26]. bandwidth profile attributes as described in MEF 10.3 [MEF-10.3] and
MEF 26.1 [MEF-26.1].
1.2 Lab Configuration and Testing Overview 1.2. Lab Configuration and Testing Overview
The following is the description of the lab set-up for the traffic The following diagram shows the lab setup for the traffic management
management tests: tests:
+--------------+ +-------+ +----------+ +-----------+ +--------------+ +-------+ +----------+ +-----------+
| Transmitting | | | | | | Receiving | | Transmitting | | | | | | Receiving |
| Test Host | | | | | | Test Host | | Test Host | | | | | | Test Host |
| |-----| Device|---->| Network |--->| | | |-----| Device|---->| Network |--->| |
| | | Under | | Delay | | | | | | Under | | Delay | | |
| | | Test | | Emulator | | | | | | Test | | Emulator | | |
| |<----| |<----| |<---| | | |<----| |<----| |<---| |
| | | | | | | | | | | | | | | |
+--------------+ +-------+ +----------+ +-----------+ +--------------+ +-------+ +----------+ +-----------+
As shown in the test diagram, the framework supports uni-directional Figure 2: Lab Setup for Traffic Management Tests
and bi-directional traffic management tests (where the transmitting
As shown in the test diagram, the framework supports unidirectional
and bidirectional traffic management tests (where the transmitting
and receiving roles would be reversed on the return path). and receiving roles would be reversed on the return path).
This testing framework describes the tests and metrics for each of This testing framework describes the tests and metrics for each of
the following traffic management functions: the following traffic management functions:
- Classification
- Policing
- Queuing / Scheduling
- Shaping
The tests are divided into individual and rated capacity tests. - Classification
The individual tests are intended to benchmark the traffic management
- Policing
- Queuing/scheduling
- Shaping
The tests are divided into individual and rated capacity tests. The
individual tests are intended to benchmark the traffic management
functions according to the metrics defined in Section 4. The functions according to the metrics defined in Section 4. The
capacity tests verify traffic management functions under the load of capacity tests verify traffic management functions under the load of
many simultaneous individual tests and their flows. many simultaneous individual tests and their flows.
This involves concurrent testing of multiple interfaces with the This involves concurrent testing of multiple interfaces with the
specific traffic management function enabled, and increasing load to specific traffic management function enabled, and increasing the load
the capacity limit of each interface. to the capacity limit of each interface.
As an example: a device is specified to be capable of shaping on all For example, a device is specified to be capable of shaping on all of
of its egress ports. The individual test would first be conducted to its egress ports. The individual test would first be conducted to
benchmark the specified shaping function against the metrics defined benchmark the specified shaping function against the metrics defined
in section 4. Then the capacity test would be executed to test the in Section 4. Then, the capacity test would be executed to test the
shaping function concurrently on all interfaces and with maximum shaping function concurrently on all interfaces and with maximum
traffic load. traffic load.
The Network Delay Emulator (NDE) is required for TCP stateful tests The Network Delay Emulator (NDE) is required for TCP stateful tests
in order to allow TCP to utilize a significant size TCP window in its in order to allow TCP to utilize a TCP window of significant size in
control loop. its control loop.
Also note that the Network Delay Emulator (NDE) SHOULD be passive in Note also that the NDE SHOULD be passive in nature (e.g., a fiber
nature such as a fiber spool. This is recommended to eliminate the spool). This is recommended to eliminate the potential effects that
potential effects that an active delay element (i.e. test impairment an active delay element (i.e., test impairment generator) may have on
generator) may have on the test flows. In the case where a fiber the test flows. In the case where a fiber spool is not practical due
spool is not practical due to the desired latency, an active NDE MUST to the desired latency, an active NDE MUST be independently verified
be independently verified to be capable of adding the configured to be capable of adding the configured delay without loss. In other
delay without loss. In other words, the DUT would be removed and the words, the Device Under Test (DUT) would be removed and the NDE
NDE performance benchmarked independently. performance benchmarked independently.
Note that the NDE SHOULD be used only as emulated delay. Most NDEs Note that the NDE SHOULD be used only as emulated delay. Most NDEs
allow for per flow delay actions, emulating QoS prioritization. For allow for per-flow delay actions, emulating QoS prioritization. For
this framework, the NDE's sole purpose is simply to add delay to all this framework, the NDE's sole purpose is simply to add delay to all
packets (emulate network latency). So to benchmark the performance of packets (emulate network latency). So, to benchmark the performance
the NDE, maximum offered load should be tested against the following of the NDE, the maximum offered load should be tested against the
frame sizes: 128, 256, 512, 768, 1024, 1500,and 9600 bytes. The delay following frame sizes: 128, 256, 512, 768, 1024, 1500, and
accuracy at each of these packet sizes can then be used to calibrate 9600 bytes. The delay accuracy at each of these packet sizes can
the range of expected Bandwidth Delay Product (BDP) for the TCP then be used to calibrate the range of expected Bandwidth-Delay
stateful tests. Product (BDP) for the TCP stateful tests.
2. Conventions used in this document 2. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. document are to be interpreted as described in [RFC2119].
The following acronyms are used: The following acronyms are used:
AQM: Active Queue Management AQM: Active Queue Management
BB: Bottleneck Bandwidth
BDP: Bandwidth Delay Product
BSA: Burst Size Achieved
CBS: Committed Burst Size BB: Bottleneck Bandwidth
CIR: Committed Information Rate BDP: Bandwidth-Delay Product
DUT: Device Under Test BSA: Burst Size Achieved
EBS: Excess Burst Size CBS: Committed Burst Size
CIR: Committed Information Rate
EIR: Excess Information Rate DUT: Device Under Test
NDE: Network Delay Emulator EBS: Excess Burst Size
SP: Strict Priority Queuing EIR: Excess Information Rate
QL: Queue Length NDE: Network Delay Emulator
QoS: Quality of Service QL: Queue Length
RTH: Receiving Test Host QoS: Quality of Service
RTT: Round Trip Time RTT: Round-Trip Time
SBB: Shaper Burst Bytes SBB: Shaper Burst Bytes
SBI: Shaper Burst Interval SBI: Shaper Burst Interval
SR: Shaper Rate SP: Strict Priority
SSB: Send Socket Buffer SR: Shaper Rate
Tc: CBS Time Interval SSB: Send Socket Buffer
Te: EBS Time Interval SUT: System Under Test
Ti Transmission Interval Ti: Transmission Interval
TTH: Transmitting Test Host TTP: TCP Test Pattern
TTP: TCP Test Pattern
TTPET: TCP Test Pattern Execution Time TTPET: TCP Test Pattern Execution Time
3. Scope and Goals 3. Scope and Goals
The scope of this work is to develop a framework for benchmarking and The scope of this work is to develop a framework for benchmarking and
testing the traffic management capabilities of network devices in the testing the traffic management capabilities of network devices in the
lab environment. These network devices may include but are not lab environment. These network devices may include but are not
limited to: limited to:
- Switches (including Layer 2/3 devices)
- Routers
- Firewalls
- General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.)
- Switches (including Layer 2/3 devices)
- Routers
- Firewalls
- General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.)
Essentially, any network device that performs traffic management as Essentially, any network device that performs traffic management as
defined in section 1.1 can be benchmarked or tested with this defined in Section 1.1 can be benchmarked or tested with this
framework. framework.
The primary goal is to assess the maximum forwarding performance The primary goal is to assess the maximum forwarding performance
deemed to be within the provisioned traffic limits that a network deemed to be within the provisioned traffic limits that a network
device can sustain without dropping or impairing packets, or device can sustain without dropping or impairing packets, and without
compromising the accuracy of multiple instances of traffic compromising the accuracy of multiple instances of traffic management
management functions. This is the benchmark for comparison between functions. This is the benchmark for comparison between devices.
devices.
Within this framework, the metrics are defined for each traffic Within this framework, the metrics are defined for each traffic
management test but do not include pass / fail criterion, which is management test but do not include pass/fail criteria, which are not
not within the charter of BMWG. This framework provides the test within the charter of the BMWG. This framework provides the test
methods and metrics to conduct repeatable testing, which will methods and metrics to conduct repeatable testing, which will provide
provide the means to compare measured performance between DUTs. the means to compare measured performance between DUTs.
As mentioned in section 1.2, these methods describe the individual As mentioned in Section 1.2, these methods describe the individual
tests and metrics for several management functions. It is also within tests and metrics for several management functions. It is also
scope that this framework will benchmark each function in terms of within scope that this framework will benchmark each function in
overall rated capacity. This involves concurrent testing of multiple terms of overall rated capacity. This involves concurrent testing of
interfaces with the specific traffic management function enabled, up multiple interfaces with the specific traffic management function
to the capacity limit of each interface. enabled, up to the capacity limit of each interface.
It is not within scope of this framework to specify the procedure for It is not within the scope of this framework to specify the procedure
testing multiple configurations of traffic management functions for testing multiple configurations of traffic management functions
concurrently. The multitudes of possible combinations is almost concurrently. The multitudes of possible combinations are almost
unbounded and the ability to identify functional "break points" unbounded, and the ability to identify functional "break points"
would be almost impossible. would be almost impossible.
However, section 6.4 provides suggestions for some profiles of However, Section 6.4 provides suggestions for some profiles of
concurrent functions that would be useful to benchmark. The key concurrent functions that would be useful to benchmark. The key
requirement for any concurrent test function is that tests MUST requirement for any concurrent test function is that tests MUST
produce reliable and repeatable results. produce reliable and repeatable results.
Also, it is not within scope to perform conformance testing. Tests Also, it is not within scope to perform conformance testing. Tests
defined in this framework benchmark the traffic management functions defined in this framework benchmark the traffic management functions
according to the metrics defined in section 4 and do not address any according to the metrics defined in Section 4 and do not address any
conformance to standards related to traffic management. conformance to standards related to traffic management.
The current specifications don't specify exact behavior or The current specifications don't specify exact behavior or
implementation and the specifications that do exist (cited in implementation, and the specifications that do exist (cited in
Section 1.1) allow implementations to vary w.r.t. short term rate Section 1.1) allow implementations to vary with regard to short-term
accuracy and other factors. This is a primary driver for this rate accuracy and other factors. This is a primary driver for this
framework: to provide an objective means to compare vendor traffic framework: to provide an objective means to compare vendor traffic
management functions. management functions.
Another goal is to devise methods that utilize flows with Another goal is to devise methods that utilize flows with congestion-
congestion-aware transport (TCP) as part of the traffic load and aware transport (TCP) as part of the traffic load and still produce
still produce repeatable results in the isolated test environment. repeatable results in the isolated test environment. This framework
This framework will derive stateful test patterns (TCP or will derive stateful test patterns (TCP or application layer) that
application layer) that can also be used to further benchmark the can also be used to further benchmark the performance of applicable
performance of applicable traffic management techniques such as traffic management techniques such as queuing/scheduling and traffic
queuing / scheduling and traffic shaping. In cases where the shaping. In cases where the network device is stateful in nature
network device is stateful in nature (i.e. firewall, etc.), (i.e., firewall, etc.), stateful test pattern traffic is important to
stateful test pattern traffic is important to test along with test, along with stateless UDP traffic in specific test scenarios
stateless, UDP traffic in specific test scenarios (i.e. (i.e., applications using TCP transport and UDP VoIP, etc.).
applications using TCP transport and UDP VoIP, etc.).
As mentioned earlier in the document, repeatability of test results As mentioned earlier in this document, repeatability of test results
is critical, especially considering the nature of stateful TCP is critical, especially considering the nature of stateful TCP
traffic. To this end, the stateful tests will use TCP test patterns traffic. To this end, the stateful tests will use TCP test patterns
to emulate applications. This framework also provides guidelines to emulate applications. This framework also provides guidelines for
for application modeling and open source tools to achieve the application modeling and open source tools to achieve the repeatable
repeatable stimulus. And finally, TCP metrics from [RFC6349] MUST stimulus. Finally, TCP metrics from [RFC6349] MUST be measured for
be measured for each stateful test and provide the means to compare each stateful test and provide the means to compare each repeated
each repeated test. test.
Even though the scope is targeted to TCP applications (i.e. Web, Even though this framework targets the testing of TCP applications
Email, database, etc.), the framework could be applied to SCTP (i.e., web, email, database, etc.), it could also be applied to the
in terms of test patterns. WebRTC, SS7 signaling, and 3gpp are Stream Control Transmission Protocol (SCTP) in terms of test
SCTP-based applications that could be modeled with this patterns. WebRTC, Signaling System 7 (SS7) signaling, and 3GPP are
framework to benchmark SCTP's effect on traffic management SCTP-based applications that could be modeled with this framework to
performance. benchmark SCTP's effect on traffic management performance.
Also note that currently, this framework does not address tcpcrypt Note that at the time of this writing, this framework does not
(encrypted TCP) test patterns, although the metrics defined in address tcpcrypt (encrypted TCP) test patterns, although the metrics
Section 4.2 can still be used since the metrics are based on TCP defined in Section 4.2 can still be used because the metrics are
retransmission and RTT measurements (versus any of the payload). based on TCP retransmission and RTT measurements (versus any of the
Thus if tcpcrypt becomes popular, it would be natural for payload). Thus, if tcpcrypt becomes popular, it would be natural for
benchmarkers to consider encrypted TCP patterns and include them benchmarkers to consider encrypted TCP patterns and include them in
in test cases. test cases.
4. Traffic Benchmarking Metrics 4. Traffic Benchmarking Metrics
The metrics to be measured during the benchmarks are divided into two The metrics to be measured during the benchmarks are divided into two
(2) sections: packet layer metrics used for the stateless traffic (2) sections: packet-layer metrics used for the stateless traffic
testing and TCP layer metrics used for the stateful traffic testing and TCP-layer metrics used for the stateful traffic testing.
testing.
4.1. Metrics for Stateless Traffic Tests 4.1. Metrics for Stateless Traffic Tests
Stateless traffic measurements require that sequence number and Stateless traffic measurements require that a sequence number and
time-stamp be inserted into the payload for lost packet analysis. timestamp be inserted into the payload for lost-packet analysis.
Delay analysis may be achieved by insertion of timestamps directly Delay analysis may be achieved by insertion of timestamps directly
into the packets or timestamps stored elsewhere (packet captures). into the packets or timestamps stored elsewhere (packet captures).
This framework does not specify the packet format to carry sequence This framework does not specify the packet format to carry sequence
number or timing information. number or timing information.
However,[RFC4737] and [RFC4689] provide recommendations However, [RFC4737] and [RFC4689] provide recommendations for sequence
for sequence tracking along with definitions of in-sequence and tracking, along with definitions of in-sequence and out-of-order
out-of-order packets. packets.
The following are the metrics that MUST be measured during the
stateless traffic benchmarking components of the tests:
- Burst Size Achieved (BSA): for the traffic policing and network
queue tests, the tester will be configured to send bursts to test
either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of
a policer or the queue / buffer size configured in the DUT. The
Burst Size Achieved metric is a measure of the actual burst size
received at the egress port of the DUT with no lost packets. As an
example, the configured CBS of a DUT is 64KB and after the burst
test, only a 63 KB can be achieved without packet loss. Then 63KB is
the BSA. Also, the average Packet Delay Variation (PDV see below) as
experienced by the packets sent at the BSA burst size should be
recorded. This metric shall be reported in units of bytes, KBytes,
or MBytes.
- Lost Packets(LP): For all traffic management tests, the tester will
transmit the test packets into the DUT ingress port and the number of
packets received at the egress port will be measured. The difference
between packets transmitted into the ingress port and received at the
egress port is the number of lost packets as measured at the egress
port. These packets must have unique identifiers such that only the
test packets are measured. For cases where multiple flows are
transmitted from ingress to egress port (e.g. IP conversations), each
flow must have sequence numbers within the test packets stream.
[RFC6703] and [RFC2680] describe the need to establish the The following metrics MUST be measured during the stateless traffic
time threshold to wait before a packet is declared as lost, and this benchmarking components of the tests:
threshold MUST be reported with the results. This metric shall be
reported as an integer number which cannot be negative. (see:
http://tools.ietf.org/html/rfc6703#section-4.1)
- Out of Order (OOO): in additions to the LP metric, the test - Burst Size Achieved (BSA): For the traffic policing and network
packets must be monitored for sequence. [RFC4689] defines the queue tests, the tester will be configured to send bursts to test
general function of sequence tracking, as well as definitions either the Committed Burst Size (CBS) or Excess Burst Size (EBS)
for in-sequence and out-of-order packets. Out-of-order packets of a policer or the queue/buffer size configured in the DUT. The
will be counted per [RFC4737]. This metric shall be reported as BSA metric is a measure of the actual burst size received at the
an integer number which cannot be negative. egress port of the DUT with no lost packets. For example, the
configured CBS of a DUT is 64 KB, and after the burst test, only a
63 KB burst can be achieved without packet loss. Then, 63 KB is
the BSA. Also, the average Packet Delay Variation (PDV) (see
below) as experienced by the packets sent at the BSA burst size
should be recorded. This metric SHALL be reported in units of
bytes, KB, or MB.
- Packet Delay (PD): the Packet Delay metric is the difference - Lost Packets (LP): For all traffic management tests, the tester
between the timestamp of the received egress port packets and the will transmit the test packets into the DUT ingress port, and the
packets transmitted into the ingress port and specified in [RFC1242]. number of packets received at the egress port will be measured.
The transmitting host and receiving host time must be in The difference between packets transmitted into the ingress port
time sync using NTP , GPS, etc. This metric SHALL be reported as a and received at the egress port is the number of lost packets as
real number of seconds, where a negative measurement usually measured at the egress port. These packets must have unique
indicates a time synchronization problem between test devices. identifiers such that only the test packets are measured. For
cases where multiple flows are transmitted from the ingress port
to the egress port (e.g., IP conversations), each flow must have
sequence numbers within the stream of test packets.
- Packet Delay Variation (PDV): the Packet Delay Variation metric is [RFC6703] and [RFC2680] describe the need to establish the time
the variation between the timestamp of the received egress port threshold to wait before a packet is declared as lost. This
packets and specified in [RFC5481]. Note that per [RFC5481], threshold MUST be reported, with the results reported as an integer
this PDV is the variation of one-way delay across many packets in number that cannot be negative.
the traffic flow. Per the measurement formula in [RFC5481], select
the high percentile of 99% and units of measure will be a real
number of seconds (negative is not possible for PDV and would
indicate a measurement error).
- Shaper Rate (SR): The SR represents the average DUT output - Out-of-Sequence (OOS): In addition to the LP metric, the test
rate (bps) over the test interval. The Shaper Rate is only packets must be monitored for sequence. [RFC4689] defines the
applicable to the traffic shaping tests. general function of sequence tracking, as well as definitions for
in-sequence and out-of-order packets. Out-of-order packets will
be counted per [RFC4737]. This metric SHALL be reported as an
integer number that cannot be negative.
- Shaper Burst Bytes (SBB): A traffic shaper will emit packets in - Packet Delay (PD): The PD metric is the difference between the
different size "trains"; these are frames "back-to-back", respect timestamp of the received egress port packets and the packets
the mandatory inter-frame gap. This metric characterizes the method transmitted into the ingress port, as specified in [RFC1242]. The
by which the shaper emits traffic. Some shapers transmit larger transmitting host and receiving host time must be in time sync
bursts per interval, and a burst of 1 packet would apply to the (achieved by using NTP, GPS, etc.). This metric SHALL be reported
extreme case of a shaper sending a CBR stream of single packets. as a real number of seconds, where a negative measurement usually
This metric SHALL be reported in units of bytes, KBytes, or MBytes. indicates a time synchronization problem between test devices.
Shaper Burst Bytes is only applicable to thetraffic shaping tests.
- Shaper Burst Interval(SBI): the SBI is the time between shaper - Packet Delay Variation (PDV): The PDV metric is the variation
emitted bursts and is measured at the DUT egress port. This metric between the timestamp of the received egress port packets, as
shall be reported as an real number of seconds. Shaper Burst specified in [RFC5481]. Note that per [RFC5481], this PDV is the
Interval is only applicable to the traffic shaping tests, variation of one-way delay across many packets in the traffic
flow. Per the measurement formula in [RFC5481], select the high
percentile of 99%, and units of measure will be a real number of
seconds (a negative value is not possible for the PDV and would
indicate a measurement error).
4.2. Metrics for Stateful Traffic Tests - Shaper Rate (SR): The SR represents the average DUT output rate
(bps) over the test interval. The SR is only applicable to the
traffic-shaping tests.
The stateful metrics will be based on [RFC6349] TCP metrics and - Shaper Burst Bytes (SBB): A traffic shaper will emit packets in
MUST include: "trains" of different sizes; these frames are emitted "back-to-
back" with respect to the mandatory interframe gap. This metric
characterizes the method by which the shaper emits traffic. Some
shapers transmit larger bursts per interval, and a burst of
one packet would apply to the less common case of a shaper sending
a constant-bitrate stream of single packets. This metric SHALL be
reported in units of bytes, KB, or MB. The SBB metric is only
applicable to the traffic-shaping tests.
- TCP Test Pattern Execution Time (TTPET): [RFC6349] defined the TCP - Shaper Burst Interval (SBI): The SBI is the time between bursts
Transfer Time for bulk transfers, which is simply the measured time emitted by the shaper and is measured at the DUT egress port.
to transfer bytes across single or concurrent TCP connections. The This metric SHALL be reported as a real number of seconds. The
TCP test patterns used in traffic management tests will include bulk SBI is only applicable to the traffic-shaping tests.
transfer and interactive applications. The interactive patterns
include instances such as HTTP business applications, database
applications, etc. The TTPET will be the measure of the time for a
single execution of a TCP Test Pattern (TTP). Average, minimum, and
maximum times will be measured or calculated and expressed as a real
number of seconds.
An example would be an interactive HTTP TTP session which should take 4.2. Metrics for Stateful Traffic Tests
5 seconds on a GigE network with 0.5 millisecond latency. During ten
(10) executions of this TTP, the TTPET results might be: average of
6.5 seconds, minimum of 5.0 seconds, and maximum of 7.9 seconds.
- TCP Efficiency: after the execution of the TCP Test Pattern, TCP The stateful metrics will be based on [RFC6349] TCP metrics and MUST
Efficiency represents the percentage of Bytes that were not include:
retransmitted.
Transmitted Bytes - Retransmitted Bytes - TCP Test Pattern Execution Time (TTPET): [RFC6349] defined the TCP
Transfer Time for bulk transfers, which is simply the measured
time to transfer bytes across single or concurrent TCP
connections. The TCP test patterns used in traffic management
tests will include bulk transfer and interactive applications.
The interactive patterns include instances such as HTTP business
applications and database applications. The TTPET will be the
measure of the time for a single execution of a TCP Test Pattern
(TTP). Average, minimum, and maximum times will be measured or
calculated and expressed as a real number of seconds.
TCP Efficiency % = --------------------------------------- X 100 An example would be an interactive HTTP TTP session that should take
5 seconds on a GigE network with 0.5-millisecond latency. During ten
(10) executions of this TTP, the TTPET results might be an average of
6.5 seconds, a minimum of 5.0 seconds, and a maximum of 7.9 seconds.
Transmitted Bytes - TCP Efficiency: After the execution of the TTP, TCP Efficiency
represents the percentage of bytes that were not retransmitted.
Transmitted Bytes are the total number of TCP Bytes to be transmitted Transmitted Bytes - Retransmitted Bytes
including the original and the retransmitted Bytes. These TCP Efficiency % = --------------------------------------- X 100
retransmitted bytes should be recorded from the sender's TCP/IP stack Transmitted Bytes
perspective, to avoid any misinterpretation that a reordered packet "Transmitted Bytes" is the total number of TCP bytes to be
is a retransmitted packet (as may be the case with packet decode transmitted, including the original bytes and the retransmitted
interpretation). bytes. To avoid any misinterpretation that a reordered packet is a
retransmitted packet (as may be the case with packet decode
interpretation), these retransmitted bytes should be recorded from
the perspective of the sender's TCP/IP stack.
- Buffer Delay: represents the increase in RTT during a TCP test - Buffer Delay: Buffer Delay represents the increase in RTT during a
versus the baseline DUT RTT (non congested, inherent latency). RTT TCP test versus the baseline DUT RTT (non-congested, inherent
and the technique to measure RTT (average versus baseline) are latency). RTT and the technique to measure RTT (average versus
defined in [RFC6349]. Referencing [RFC6349], the average RTT is baseline) are defined in [RFC6349]. Referencing [RFC6349], the
derived from the total of all measured RTTs during the actual test average RTT is derived from the total of all measured RTTs during
sampled at every second divided by the test duration in seconds. the actual test sampled at every second divided by the test
duration in seconds.
Total RTTs during transfer Total RTTs during transfer
Average RTT during transfer = ----------------------------- Average RTT during transfer = ------------------------------
Transfer duration in seconds Transfer duration in seconds
Average RTT during Transfer - Baseline RTT Average RTT during transfer - Baseline RTT
Buffer Delay % = ------------------------------------------ X 100 Buffer Delay % = ------------------------------------------ X 100
Baseline RTT Baseline RTT
Note that even though this was not explicitly stated in [RFC6349], Note that even though this was not explicitly stated in [RFC6349],
retransmitted packets should not be used in RTT measurements. retransmitted packets should not be used in RTT measurements.
Also, the test results should record the average RTT in millisecond Also, the test results should record the average RTT in milliseconds
across the entire test duration and number of samples. across the entire test duration, as well as the number of samples.
5. Tester Capabilities 5. Tester Capabilities
The testing capabilities of the traffic management test environment The testing capabilities of the traffic management test environment
are divided into two (2) sections: stateless traffic testing and are divided into two (2) sections: stateless traffic testing and
stateful traffic testing stateful traffic testing.
5.1. Stateless Test Traffic Generation 5.1. Stateless Test Traffic Generation
The test device MUST be capable of generating traffic at up to the The test device MUST be capable of generating traffic at up to the
link speed of the DUT. The test device must be calibrated to verify link speed of the DUT. The test device must be calibrated to verify
that it will not drop any packets. The test device's inherent PD and that it will not drop any packets. The test device's inherent PD and
PDV must also be calibrated and subtracted from the PD and PDV PDV must also be calibrated and subtracted from the PD and PDV
metrics. The test device must support the encapsulation to be metrics. The test device must support the encapsulation to be
tested such as IEEE 802.1Q VLAN, IEEE 802.1ad Q-in-Q, Multiprotocol tested, e.g., IEEE 802.1Q VLAN, IEEE 802.1ad Q-in-Q, Multiprotocol
Label Switching (MPLS), etc. Also, the test device must allow Label Switching (MPLS). Also, the test device must allow control of
control of the classification techniques defined in [RFC4689] the classification techniques defined in [RFC4689] (e.g., IP address,
(i.e. IP address, DSCP, TOS, etc classification). DSCP, classification of Type of Service).
The open source tool "iperf" can be used to generate stateless UDP The open source tool "iperf" can be used to generate stateless UDP
traffic and is discussed in Appendix A. Since iperf is a software traffic and is discussed in Appendix A. Since iperf is a software-
based tool, there will be performance limitations at higher link based tool, there will be performance limitations at higher link
speeds (e.g. GigE, 10 GigE, etc.). Careful calibration of any test speeds (e.g., 1 GigE, 10 GigE). Careful calibration of any test
environment using iperf is important. At higher link speeds, it is environment using iperf is important. At higher link speeds, using
recommended to use hardware based packet test equipment. hardware-based packet test equipment is recommended.
5.1.1 Burst Hunt with Stateless Traffic 5.1.1. Burst Hunt with Stateless Traffic
A central theme for the traffic management tests is to benchmark the A central theme for the traffic management tests is to benchmark the
specified burst parameter of traffic management function, since burst specified burst parameter of a traffic management function, since
burst parameters listed in Service Level Agreements (SLAs) are
parameters of SLAs are specified in bytes. For testing efficiency, specified in bytes. For testing efficiency, including a burst hunt
it is recommended to include a burst hunt feature, which automates feature is recommended, as this feature automates the manual process
the manual process of determining the maximum burst size which can of determining the maximum burst size that can be supported by a
be supported by a traffic management function. traffic management function.
The burst hunt algorithm should start at the target burst size The burst hunt algorithm should start at the target burst size
(maximum burst size supported by the traffic management function) (maximum burst size supported by the traffic management function) and
and will send single bursts until it can determine the largest burst will send single bursts until it can determine the largest burst that
that can pass without loss. If the target burst size passes, then can pass without loss. If the target burst size passes, then the
the test is complete. The hunt aspect occurs when the target burst test is complete. The "hunt" aspect occurs when the target burst
size is not achieved; the algorithm will drop down to a configured size is not achieved; the algorithm will drop down to a configured
minimum burst size and incrementally increase the burst until the minimum burst size and incrementally increase the burst until the
maximum burst supported by the DUT is discovered. The recommended maximum burst supported by the DUT is discovered. The recommended
granularity of the incremental burst size increase is 1 KB. granularity of the incremental burst size increase is 1 KB.
Optionally for a policer function and if the burst size passes, the For a policer function, if the burst size passes, the burst should be
burst should be increased by increments of 1 KB to verify that the increased by increments of 1 KB to verify that the policer is truly
policer is truly configured properly (or enabled at all). configured properly (or enabled at all).
5.2. Stateful Test Pattern Generation 5.2. Stateful Test Pattern Generation
The TCP test host will have many of the same attributes as the TCP The TCP test host will have many of the same attributes as the TCP
test host defined in [RFC6349]. The TCP test device may be a test host defined in [RFC6349]. The TCP test device may be a
standard computer or a dedicated communications test instrument. In standard computer or a dedicated communications test instrument. In
both cases, it must be capable of emulating both a client and a both cases, it must be capable of emulating both a client and a
server. server.
For any test using stateful TCP test traffic, the Network Delay For any test using stateful TCP test traffic, the Network Delay
Emulator (NDE function from the lab set-up diagram) must be used in Emulator (the NDE function as shown in the lab setup diagram in
order to provide a meaningful BDP. As referenced in section 2, the Section 1.2) must be used in order to provide a meaningful BDP. As
target traffic rate and configured RTT MUST be verified independently discussed in Section 1.2, the target traffic rate and configured RTT
using just the NDE for all stateful tests (to ensure the NDE can MUST be verified independently, using just the NDE for all stateful
delay without loss). tests (to ensure that the NDE can add delay without inducing any
packet loss).
The TCP test host MUST be capable to generate and receive stateful The TCP test host MUST be capable of generating and receiving
TCP test traffic at the full link speed of the DUT. As a general stateful TCP test traffic at the full link speed of the DUT. As a
rule of thumb, testing TCP Throughput at rates greater than 500 Mbps general rule of thumb, testing TCP throughput at rates greater than
may require high performance server hardware or dedicated hardware 500 Mbps may require high-performance server hardware or dedicated
based test tools. hardware-based test tools.
The TCP test host MUST allow adjusting both Send and Receive Socket The TCP test host MUST allow the adjustment of both Send and Receive
Buffer sizes. The Socket Buffers must be large enough to fill the Socket Buffer sizes. The Socket Buffers must be large enough to fill
BDP for bulk transfer TCP test application traffic. the BDP for bulk transfer of TCP test application traffic.
Measuring RTT and retransmissions per connection will generally Measuring RTT and retransmissions per connection will generally
require a dedicated communications test instrument. In the absence of require a dedicated communications test instrument. In the absence
dedicated hardware based test tools, these measurements may need to of dedicated hardware-based test tools, these measurements may need
be conducted with packet capture tools, i.e. conduct TCP Throughput to be conducted with packet capture tools; i.e., conduct TCP
tests and analyze RTT and retransmissions in packet captures. throughput tests, and analyze RTT and retransmissions in packet
captures.
The TCP implementation used by the test host MUST be specified in The TCP implementation used by the test host MUST be specified in the
the test results (e.g. TCP New Reno, TCP options supported, etc.). test results (e.g., TCP New Reno, TCP options supported).
Additionally, the test results SHALL provide specific congestion Additionally, the test results SHALL provide specific congestion
control algorithm details, as per [RFC3148]. control algorithm details, as per [RFC3148].
While [RFC6349] defined the means to conduct throughput tests of TCP While [RFC6349] defined the means to conduct throughput tests of TCP
bulk transfers, the traffic management framework will extend TCP test bulk transfers, the traffic management framework will extend TCP test
execution into interactive TCP application traffic. Examples include execution into interactive TCP application traffic. Examples include
email, HTTP, business applications, etc. This interactive traffic is email, HTTP, and business applications. This interactive traffic is
bi-directional and can be chatty, meaning many turns in traffic bidirectional and can be chatty, meaning many turns in traffic
communication during the course of a transaction (versus the communication during the course of a transaction (versus the
relatively uni-directional flow of bulk transfer applications). relatively unidirectional flow of bulk transfer applications).
The test device must not only support bulk TCP transfer application The test device must not only support bulk TCP transfer application
traffic but MUST also support chatty traffic. A valid stress test traffic but MUST also support chatty traffic. A valid stress test
SHOULD include both traffic types. This is due to the non-uniform, SHOULD include both traffic types. This is due to the non-uniform,
bursty nature of chatty applications versus the relatively uniform bursty nature of chatty applications versus the relatively uniform
nature of bulk transfers (the bulk transfer smoothly stabilizes to nature of bulk transfers (the bulk transfer smoothly stabilizes to
equilibrium state under lossless conditions). equilibrium state under lossless conditions).
While iperf is an excellent choice for TCP bulk transfer testing, While iperf is an excellent choice for TCP bulk transfer testing, the
the netperf open source tool provides the ability to control the "netperf" open source tool provides the ability to control client and
client and server request / response behavior. The netperf-wrapper server request/response behavior. The netperf-wrapper tool is a
tool is a Python wrapper to run multiple simultaneous netperf Python script that runs multiple simultaneous netperf instances and
instances and aggregate the results. Appendix A provides an overview aggregates the results. Appendix A provides an overview of
of netperf / netperf-wrapper and another open source application netperf/netperf-wrapper, as well as iperf. As with any software-
emulation tools, iperf. As with any software based tool, the based tool, the performance must be qualified to the link speed to be
performance must be qualified to the link speed to be tested. tested. Hardware-based test equipment should be considered for
Hardware-based test equipment should be considered for reliable reliable results at higher link speeds (e.g., 1 GigE, 10 GigE).
results at higher links speeds (e.g. 1 GigE, 10 GigE).
5.2.1. TCP Test Pattern Definitions 5.2.1. TCP Test Pattern Definitions
As mentioned in the goals of this framework, techniques are defined As mentioned in the goals of this framework, techniques are defined
to specify TCP traffic test patterns to benchmark traffic to specify TCP traffic test patterns to benchmark traffic management
management technique(s) and produce repeatable results. Some technique(s) and produce repeatable results. Some network devices,
network devices such as firewalls, will not process stateless test such as firewalls, will not process stateless test traffic; this is
traffic which is another reason why stateful TCP test traffic must another reason why stateful TCP test traffic must be used.
be used.
An application could be fully emulated up to Layer 7, however this An application could be fully emulated up to Layer 7; however, this
framework proposes that stateful TCP test patterns be used in order framework proposes that stateful TCP test patterns be used in order
to provide granular and repeatable control for the benchmarks. The to provide granular and repeatable control for the benchmarks. The
following diagram illustrates a simple Web Browsing application following diagram illustrates a simple web-browsing application
(HTTP). (HTTP).
GET url GET URL
Client ------------------------> Web
Web 200 OK 100ms |
Browser <------------------------ Server Client -------------------------> Web
|
Web 200 OK 100 ms |
|
Browser <------------------------- Server
In this example, the Client Web Browser (Client) requests a URL and Figure 3: Simple Flow Diagram for a Web Application
then the Web Server delivers the web page content to the Client
(after a Server delay of 100 millisecond). This asynchronous,
"request/response" behavior is intrinsic to most TCP based
applications such as Email (SMTP), File Transfers (FTP and SMB),
Database (SQL), Web Applications (SOAP), REST, etc. The impact to In this example, the Client Web Browser (client) requests a URL, and
the network elements is due to the multitudes of Clients and the then the Web Server delivers the web page content to the client
variety of bursty traffic, which stresses traffic management (after a server delay of 100 milliseconds). This asynchronous
functions. The actual emulation of the specific application "request/response" behavior is intrinsic to most TCP-based
protocols is not required and TCP test patterns can be defined to applications, such as email (SMTP), file transfers (FTP and Server
mimic the application network traffic flows and produce repeatable Message Block (SMB)), database (SQL), web applications (SOAP), and
results. Representational State Transfer (REST). The impact on the network
elements is due to the multitudes of clients and the variety of
bursty traffic, which stress traffic management functions. The
actual emulation of the specific application protocols is not
required, and TCP test patterns can be defined to mimic the
application network traffic flows and produce repeatable results.
Application modeling techniques have been proposed in Application modeling techniques have been proposed in
"3GPP2 C.R1002-0 v1.0" and provides examples to model the behavior of [3GPP2-C_R1002-A], which provides examples to model the behavior of
HTTP, FTP, and WAP applications at the TCP layer. The models have HTTP, FTP, and Wireless Application Protocol (WAP) applications at
been defined with various mathematical distributions for the the TCP layer. The models have been defined with various
Request/Response bytes and inter-request gap times. The model mathematical distributions for the request/response bytes and
definition format described in this work are the basis for the inter-request gap times. The model definition formats described in
guidelines provides in Appendix B and are also similar to formats [3GPP2-C_R1002-A] are the basis for the guidelines provided in
used by network modeling tools. Packet captures can also be used to Appendix B and are also similar to formats used by network modeling
characterize application traffic and specify some of the test tools. Packet captures can also be used to characterize application
patterns listed in Appendix B. traffic and specify some of the test patterns listed in Appendix B.
This framework does not specify a fixed set of TCP test patterns, but This framework does not specify a fixed set of TCP test patterns but
does provide test cases that SHOULD be performed in Appendix B. Some does provide test cases that SHOULD be performed; see Appendix B.
of these examples reflect those specified in "draft-ietf-bmwg-ca- Some of these examples reflect those specified in [CA-Benchmark],
bench-meth-04" which suggests traffic mixes for a variety of which suggests traffic mixes for a variety of representative
representative application profiles. Other examples are simply application profiles. Other examples are simply well-known
well-known application traffic types such as HTTP. application traffic types such as HTTP.
6. Traffic Benchmarking Methodology 6. Traffic Benchmarking Methodology
The traffic benchmarking methodology uses the test set-up from The traffic benchmarking methodology uses the test setup from
section 2 and metrics defined in section 4. Section 1.2 and metrics defined in Section 4.
Each test SHOULD compare the network device's internal statistics Each test SHOULD compare the network device's internal statistics
(available via command line management interface, SNMP, etc.) to the (available via command line management interface, SNMP, etc.) to the
measured metrics defined in section 4. This evaluates the accuracy measured metrics defined in Section 4. This evaluates the accuracy
of the internal traffic management counters under individual test of the internal traffic management counters under individual test
conditions and capacity test conditions that are defined in each conditions and capacity test conditions as defined in Sections 4.1
subsection. This comparison is not intended to compare real-time and 4.2. This comparison is not intended to compare real-time
statistics, but the cumulative statistics reported after the test statistics, but rather the cumulative statistics reported after the
has completed and device counters have updated (it is common for test has completed and device counters have updated (it is common for
device counters to update after a 10 second or greater interval). device counters to update after an interval of 10 seconds or more).
From a device configuration standpoint, scheduling and shaping From a device configuration standpoint, scheduling and shaping
functionality can be applied to logical ports such Link Aggregation functionality can be applied to logical ports (e.g., Link Aggregation
(LAG). This would result in the same scheduling and shaping (LAG)). This would result in the same scheduling and shaping
configuration applied to all the member physical ports. The focus of configuration applied to all of the member physical ports. The focus
this draft is only on tests at a physical port level. of this document is only on tests at a physical-port level.
The following sections provide the objective, procedure, metrics, and The following sections provide the objective, procedure, metrics, and
reporting format for each test. For all test steps, the following reporting format for each test. For all test steps, the following
global parameters must be specified: global parameters must be specified:
Test Runs (Tr). Defines the number of times the test needs to be run Test Runs (Tr):
to ensure accurate and repeatable results. The recommended value is The number of times the test needs to be run to ensure accurate
a minimum of 10. and repeatable results. The recommended value is a minimum
of 10.
Test Duration (Td). Defines the duration of a test iteration, Test Duration (Td):
expressed in seconds. The recommended minimum value is 60 seconds. The duration of a test iteration, expressed in seconds. The
recommended minimum value is 60 seconds.
The variability in the test results MUST be measured between Test The variability in the test results MUST be measured between test
Runs and if the variation is characterized as a significant portion runs, and if the variation is characterized as a significant portion
of the measured values, the next step may be to revise the methods to of the measured values, the next step may be to revise the methods to
achieve better consistency. achieve better consistency.
6.1. Policing Tests 6.1. Policing Tests
A policer is defined as the entity performing the policy function. A policer is defined as the entity performing the policy function.
The intent of the policing tests is to verify the policer performance The intent of the policing tests is to verify the policer performance
(i.e. CIR-CBS and EIR-EBS parameters). The tests will verify that the (i.e., CIR/CBS and EIR/EBS parameters). The tests will verify that
network device can handle the CIR with CBS and the EIR with EBS and the network device can handle the CIR with CBS and the EIR with EBS,
will use back-back packet testing concepts from [RFC2544] (but and will use back-to-back packet-testing concepts as described in
adapted to burst size algorithms and terminology). Also [MEF-14], [RFC2544] (but adapted to burst size algorithms and terminology).
[MEF-19], and [MEF-37] provide some basis for specific components of Also, [MEF-14], [MEF-19], and [MEF-37] provide some bases for
this test. The burst hunt algorithm defined in section 5.1.1 can specific components of this test. The burst hunt algorithm defined
also be used to automate the measurement of the CBS value. in Section 5.1.1 can also be used to automate the measurement of the
CBS value.
The tests are divided into two (2) sections; individual policer The tests are divided into two (2) sections: individual policer tests
tests and then full capacity policing tests. It is important to and then full-capacity policing tests. It is important to benchmark
benchmark the basic functionality of the individual policer then the basic functionality of the individual policer and then proceed
proceed into the fully rated capacity of the device. This capacity into the fully rated capacity of the device. This capacity may
may include the number of policing policies per device and the include the number of policing policies per device and the number of
number of policers simultaneously active across all ports. policers simultaneously active across all ports.
6.1.1 Policer Individual Tests 6.1.1. Policer Individual Tests
Objective: Objective:
Test a policer as defined by [RFC4115] or MEF 10.2, depending upon Test a policer as defined by [RFC4115] or [MEF-10.3], depending
the equipment's specification. In addition to verifying that the upon the equipment's specification. In addition to verifying that
policer allows the specified CBS and EBS bursts to pass, the policer the policer allows the specified CBS and EBS bursts to pass, the
test MUST verify that the policer will remark or drop excess, and policer test MUST verify that the policer will remark or drop
pass traffic at the specified CBS/EBS values. excess packets, and pass traffic at the specified CBS/EBS values.
Test Summary: Test Summary:
Policing tests should use stateless traffic. Stateful TCP test Policing tests should use stateless traffic. Stateful TCP test
traffic will generally be adversely affected by a policer in the traffic will generally be adversely affected by a policer in the
absence of traffic shaping. So while TCP traffic could be used, absence of traffic shaping. So, while TCP traffic could be used,
it is more accurate to benchmark a policer with stateless traffic. it is more accurate to benchmark a policer with stateless traffic.
As an example for [RFC4115], consider a CBS and EBS of 64KB and CIR As an example of a policer as defined by [RFC4115], consider a
and EIR of 100 Mbps on a 1GigE physical link (in color-blind mode). CBS/EBS of 64 KB and CIR/EIR of 100 Mbps on a 1 GigE physical link
A stateless traffic burst of 64KB would be sent into the policer at (in color-blind mode). A stateless traffic burst of 64 KB would
the GigE rate. This equates to approximately a 0.512 millisecond be sent into the policer at the GigE rate. This equates to an
burst time (64 KB at 1 GigE). The traffic generator must space these approximately 0.512-millisecond burst time (64 KB at 1 GigE). The
bursts to ensure that the aggregate throughput does not exceed the traffic generator must space these bursts to ensure that the
CIR. The Ti between the bursts would equal CBS * 8 / CIR = 5.12 aggregate throughput does not exceed the CIR. The Ti between the
millisecond in this example. bursts would equal CBS * 8 / CIR = 5.12 milliseconds in this
example.
Test Metrics: Test Metrics:
The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL The metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV)
be measured at the egress port and recorded. SHALL be measured at the egress port and recorded.
Procedure: Procedure:
1. Configure the DUT policing parameters for the desired CIR/EIR and 1. Configure the DUT policing parameters for the desired CIR/EIR
CBS/EBS values to be tested and CBS/EBS values to be tested.
2. Configure the tester to generate a stateless traffic burst equal 2. Configure the tester to generate a stateless traffic burst
to CBS and an interval equal to Ti (CBS in bits / CIR) equal to CBS and an interval equal to Ti (CBS in bits/CIR).
3. Compliant Traffic Step: Generate bursts of CBS + EBS traffic into 3. Compliant Traffic Test: Generate bursts of CBS + EBS traffic
the policer ingress port and measure the metrics defined in into the policer ingress port, and measure the metrics defined
section 4.1 (BSA, LP. OOS, PD, and PDV) at the egress port and in Section 4.1 (BSA, LP, OOS, PD, and PDV) at the egress port
across the entire Td (default 60 seconds duration) and across the entire Td (default 60-second duration).
4. Excess Traffic Test: Generate bursts of greater than CBS + EBS 4. Excess Traffic Test: Generate bursts of greater than CBS + EBS
limit traffic into the policer ingress port and verify that the bytes into the policer ingress port, and verify that the
policer only allowed the BSA bytes to exit the egress. The excess policer only allowed the BSA bytes to exit the egress. The
burst MUST be recorded and the recommended value is 1000 bytes. excess burst MUST be recorded; the recommended value is
Additional tests beyond the simple color-blind example might 1000 bytes. Additional tests beyond the simple color-blind
include: color-aware mode, configurations where EIR is greater example might include color-aware mode, configurations where
than CIR, etc. EIR is greater than CIR, etc.
Reporting Format: Reporting Format:
The policer individual report MUST contain all results for each The policer individual report MUST contain all results for each
CIR/EIR/CBS/EBS test run and a recommended format is as follows: CIR/EIR/CBS/EBS test run. A recommended format is as follows:
******************************************************** ***********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: CIR, EIR, CBS, EBS Test Configuration Summary: Tr, Td
The results table should contain entries for each test run, (Test #1 DUT Configuration Summary: CIR, EIR, CBS, EBS
to Test #Tr).
Compliant Traffic Test: BSA, LP, OOS, PD, and PDV The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):
Excess Traffic Test: BSA - Compliant Traffic Test: BSA, LP, OOS, PD, and PDV
********************************************************
6.1.2 Policer Capacity Tests - Excess Traffic Test: BSA
***********************************************************
6.1.2. Policer Capacity Tests
Objective: Objective:
The intent of the capacity tests is to verify the policer performance The intent of the capacity tests is to verify the policer
in a scaled environment with multiple ingress customer policers on performance in a scaled environment with multiple ingress customer
multiple physical ports. This test will benchmark the maximum number policers on multiple physical ports. This test will benchmark the
of active policers as specified by the device manufacturer. maximum number of active policers as specified by the device
manufacturer.
Test Summary: Test Summary:
The specified policing function capacity is generally expressed in The specified policing function capacity is generally expressed in
terms of the number of policers active on each individual physical terms of the number of policers active on each individual physical
port as well as the number of unique policer rates that are utilized. port as well as the number of unique policer rates that are
For all of the capacity tests, the benchmarking test procedure and utilized. For all of the capacity tests, the benchmarking test
report format described in Section 6.1.1 for a single policer MUST procedure and reporting format described in Section 6.1.1 for a
be applied to each of the physical port policers. single policer MUST be applied to each of the physical-port
policers.
As an example, a Layer 2 switching device may specify that each of For example, a Layer 2 switching device may specify that each of
the 32 physical ports can be policed using a pool of policing service the 32 physical ports can be policed using a pool of policing
policies. The device may carry a single customer's traffic on each service policies. The device may carry a single customer's
physical port and a single policer is instantiated per physical port. traffic on each physical port, and a single policer is
Another possibility is that a single physical port may carry multiple instantiated per physical port. Another possibility is that a
customers, in which case many customer flows would be policed single physical port may carry multiple customers, in which case
concurrently on an individual physical port (separate policers per many customer flows would be policed concurrently on an individual
customer on an individual port). physical port (separate policers per customer on an individual
port).
Test Metrics: Test Metrics:
The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL The metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV)
be measured at the egress port and recorded. SHALL be measured at the egress port and recorded.
The following sections provide the specific test scenarios, The following sections provide the specific test scenarios,
procedures, and reporting formats for each policer capacity test. procedures, and reporting formats for each policer capacity test.
6.1.2.1 Maximum Policers on Single Physical Port Test 6.1.2.1. Maximum Policers on Single Physical Port
Test Summary: Test Summary:
The first policer capacity test will benchmark a single physical The first policer capacity test will benchmark a single physical
port, maximum policers on that physical port. port, with maximum policers on that physical port.
Assume multiple categories of ingress policers at rates r1, r2,...rn. Assume multiple categories of ingress policers at rates
There are multiple customers on a single physical port. Each customer r1, r2, ..., rn. There are multiple customers on a single
could be represented by a single tagged vlan, double tagged vlan, physical port. Each customer could be represented by a
VPLS instance etc. Each customer is mapped to a different policer. single-tagged VLAN, a double-tagged VLAN, a Virtual Private LAN
Each of the policers can be of rates r1, r2,..., rn. Service (VPLS) instance, etc. Each customer is mapped to a
different policer. Each of the policers can be of rates
r1, r2, ..., rn.
An example configuration would be An example configuration would be
- Y1 customers, policer rate r1
- Y2 customers, policer rate r2
- Y3 customers, policer rate r3
...
- Yn customers, policer rate rn
Some bandwidth on the physical port is dedicated for other traffic - Y1 customers, policer rate r1
non customer traffic); this includes network control protocol
traffic. There is a separate policer for the other traffic. Typical
deployments have 3 categories of policers; there may be some
deployments with more or less than 3 categories of ingress
policers.
Test Procedure: - Y2 customers, policer rate r2
1. Configure the DUT policing parameters for the desired CIR/EIR and
CBS/EBS values for each policer rate (r1-rn) to be tested
2. Configure the tester to generate a stateless traffic burst equal - Y3 customers, policer rate r3
to CBS and an interval equal to TI (CBS in bits/CIR) for each
customer stream (Y1 - Yn). The encapsulation for each customer
must also be configured according to the service tested (VLAN,
VPLS, IP mapping, etc.).
3. Compliant Traffic Step: Generate bursts of CBS + EBS traffic into ...
the policer ingress port for each customer traffic stream and
measure the metrics defined in section 4.1 (BSA, LP, OOS, PD, and
PDV) at the egress port for each stream and across the entire Td
(default 30 seconds duration)
4. Excess Traffic Test: Generate bursts of greater than CBS + EBS - Yn customers, policer rate rn
limit traffic into the policer ingress port for each customer Some bandwidth on the physical port is dedicated for other traffic
traffic stream and verify that the policer only allowed the BSA (i.e., other than customer traffic); this includes network control
bytes to exit the egress for each stream. The excess burst MUST protocol traffic. There is a separate policer for the other
recorded and the recommended value is 1000 bytes. traffic. Typical deployments have three categories of policers;
there may be some deployments with more or less than three
categories of ingress policers.
Procedure:
1. Configure the DUT policing parameters for the desired CIR/EIR
and CBS/EBS values for each policer rate (r1-rn) to be tested.
2. Configure the tester to generate a stateless traffic burst
equal to CBS and an interval equal to Ti (CBS in bits/CIR) for
each customer stream (Y1-Yn). The encapsulation for each
customer must also be configured according to the service
tested (VLAN, VPLS, IP mapping, etc.).
3. Compliant Traffic Test: Generate bursts of CBS + EBS traffic
into the policer ingress port for each customer traffic stream,
and measure the metrics defined in Section 4.1 (BSA, LP, OOS,
PD, and PDV) at the egress port for each stream and across the
entire Td (default 30-second duration).
4. Excess Traffic Test: Generate bursts of greater than CBS + EBS
bytes into the policer ingress port for each customer traffic
stream, and verify that the policer only allowed the BSA bytes
to exit the egress for each stream. The excess burst MUST be
recorded; the recommended value is 1000 bytes.
Reporting Format: Reporting Format:
The policer individual report MUST contain all results for each The policer individual report MUST contain all results for each
CIR/EIR/CBS/EBS test run, per customer traffic stream. CIR/EIR/CBS/EBS test run, per customer traffic stream. A
recommended format is as follows:
A recommended format is as follows: *****************************************************************
******************************************************** Test Configuration Summary: Tr, Td
Test Configuration Summary: Tr, Td
Customer traffic stream Encapsulation: Map each stream to VLAN, Customer Traffic Stream Encapsulation: Map each stream to VLAN,
VPLS, IP address VPLS, IP address
DUT Configuration Summary per Customer Traffic Stream: CIR, EIR, DUT Configuration Summary per Customer Traffic Stream: CIR, EIR,
CBS, EBS CBS, EBS
The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):
The results table should contain entries for each test run, (Test #1 - Customer Stream Y1-Yn (see note) Compliant Traffic Test:
to Test #Tr). BSA, LP, OOS, PD, and PDV
Customer Stream Y1-Yn (see note), Compliant Traffic Test: BSA, LP, - Customer Stream Y1-Yn (see note) Excess Traffic Test: BSA
OOS, PD, and PDV
Customer Stream Y1-Yn (see note), Excess Traffic Test: BSA *****************************************************************
********************************************************
Note: For each test run, there will be a two (2) rows for each Note: For each test run, there will be two (2) rows for each
customer stream, the compliant traffic result and the excess traffic customer stream: the Compliant Traffic Test result and the Excess
result. Traffic Test result.
6.1.2.2 Single Policer on All Physical Ports 6.1.2.2. Single Policer on All Physical Ports
Test Summary: Test Summary:
The second policer capacity test involves a single Policer function The second policer capacity test involves a single policer
per physical port with all physical ports active. In this test, function per physical port with all physical ports active. In
there is a single policer per physical port. The policer can have this test, there is a single policer per physical port. The
one of the rates r1, r2,.., rn. All the physical ports in the policer can have one of the rates r1, r2, ..., rn. All of the
networking device are active. physical ports in the networking device are active.
Procedure: Procedure:
The procedure is identical to 6.1.1, the configured parameters must The procedure for this test is identical to the procedure listed
be reported per port and the test report must include results per in Section 6.1.1. The configured parameters must be reported
measured egress port per port, and the test report must include results per measured
egress port.
6.1.2.3 Maximum Policers on All Physical Ports 6.1.2.3. Maximum Policers on All Physical Ports
Finally the third policer capacity test involves a combination of the The third policer capacity test is a combination of the first and
first and second capacity test, namely maximum policers active per second capacity tests, i.e., maximum policers active per physical
physical port and all physical ports are active. port and all physical ports active.
Procedure: Procedure:
Uses the procedural method from 6.1.2.1 and the configured parameters The procedure for this test is identical to the procedure listed
must be reported per port and the test report must include per stream in Section 6.1.2.1. The configured parameters must be reported
results per measured egress port. per port, and the test report must include per-stream results per
measured egress port.
6.2. Queue and Scheduler Tests 6.2. Queue/Scheduler Tests
Queues and traffic Scheduling are closely related in that a queue's Queues and traffic scheduling are closely related in that a queue's
priority dictates the manner in which the traffic scheduler priority dictates the manner in which the traffic scheduler transmits
transmits packets out of the egress port. packets out of the egress port.
Since device queues / buffers are generally an egress function, this Since device queues/buffers are generally an egress function, this
test framework will discuss testing at the egress (although the test framework will discuss testing at the egress (although the
technique can be applied to ingress side queues). technique can be applied to ingress-side queues).
Similar to the policing tests, the tests are divided into two Similar to the policing tests, these tests are divided into two
sections; individual queue/scheduler function tests and then full sections: individual queue/scheduler function tests and then
capacity tests. full-capacity tests.
6.2.1 Queue/Scheduler Individual Tests Overview 6.2.1. Queue/Scheduler Individual Tests
The various types of scheduling techniques include FIFO, Strict The various types of scheduling techniques include FIFO, Strict
Priority (SP), Weighted Fair Queueing (WFQ) along with other Priority (SP) queuing, and Weighted Fair Queuing (WFQ), along with
variations. This test framework recommends to test at a minimum other variations. This test framework recommends testing with a
of three techniques although it is the discretion of the tester minimum of three techniques, although benchmarking other
to benchmark other device scheduling algorithms. device-scheduling algorithms is left to the discretion of the tester.
6.2.1.1 Queue/Scheduler with Stateless Traffic Test 6.2.1.1. Testing Queue/Scheduler with Stateless Traffic
Objective: Objective:
Verify that the configured queue and scheduling technique can Verify that the configured queue and scheduling technique can
handle stateless traffic bursts up to the queue depth. handle stateless traffic bursts up to the queue depth.
Test Summary: Test Summary:
A network device queue is memory based unlike a policing function, A network device queue is memory based, unlike a policing
which is token or credit based. However, the same concepts from function, which is token or credit based. However, the same
section 6.1 can be applied to testing network device queues. concepts from Section 6.1 can be applied to testing network device
queues.
The device's network queue should be configured to the desired size The device's network queue should be configured to the desired
in KB (queue length, QL) and then stateless traffic should be size in KB (i.e., Queue Length (QL)), and then stateless traffic
transmitted to test this QL. should be transmitted to test this QL.
A queue should be able to handle repetitive bursts with the A queue should be able to handle repetitive bursts with the
transmission gaps proportional to the bottleneck bandwidth. This transmission gaps proportional to the Bottleneck Bandwidth (BB).
gap is referred to as the transmission interval (Ti). Ti can The transmission gap is referred to here as the transmission
be defined for the traffic bursts and is based off of the QL and interval (Ti). The Ti can be defined for the traffic bursts and
Bottleneck Bandwidth (BB) of the egress interface. is based on the QL and BB of the egress interface.
Ti = QL * 8 / BB Ti = QL * 8 / BB
Note that this equation is similar to the Ti required for Note that this equation is similar to the Ti required for
transmission into a policer (QL = CBS, BB = CIR). Also note that the transmission into a policer (QL = CBS, BB = CIR). Note also that
burst hunt algorithm defined in section 5.1.1 can also be used to the burst hunt algorithm defined in Section 5.1.1 can also be used
automate the measurement of the queue value. to automate the measurement of the queue value.
The stateless traffic burst shall be transmitted at the link speed The stateless traffic burst SHALL be transmitted at the link speed
and spaced within the Ti time interval. The metrics defined in and spaced within the transmission interval (Ti). The metrics
section 4.1 shall be measured at the egress port and recorded; the defined in Section 4.1 SHALL be measured at the egress port and
primary result is to verify the BSA and that no packets are dropped. recorded; the primary intent is to verify the BSA and verify that
no packets are dropped.
The scheduling function must also be characterized to benchmark the The scheduling function must also be characterized to benchmark
device's ability to schedule the queues according to the priority. the device's ability to schedule the queues according to the
An example would be 2 levels of priority including SP and FIFO priority. An example would be two levels of priority that include
queueing. Under a flow load greater the egress port speed, the SP and FIFO queuing. Under a flow load greater than the egress
higher priority packets should be transmitted without drops (and port speed, the higher-priority packets should be transmitted
also maintain low latency), while the lower priority (or best without drops (and also maintain low latency), while the lower-
effort) queue may be dropped. priority (or best-effort) queue may be dropped.
Test Metrics: Test Metrics:
The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL The metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV)
be measured at the egress port and recorded. SHALL be measured at the egress port and recorded.
Procedure: Procedure:
1. Configure the DUT queue length (QL) and scheduling technique 1. Configure the DUT QL and scheduling technique parameters (FIFO,
(FIFO, SP, etc) parameters SP, etc.).
2. Configure the tester to generate a stateless traffic burst equal 2. Configure the tester to generate a stateless traffic burst
to QL and an interval equal to Ti (QL in bits/BB) equal to QL and an interval equal to Ti (QL in bits/BB).
3. Generate bursts of QL traffic into the DUT and measure the 3. Generate bursts of QL traffic into the DUT, and measure the
metrics defined in section 4.1 (LP, OOS, PD, and PDV) at the metrics defined in Section 4.1 (LP, OOS, PD, and PDV) at the
egress port and across the entire Td (default 30 seconds egress port and across the entire Td (default 30-second
duration) duration).
Report Format: Reporting Format:
The Queue/Scheduler Stateless Traffic individual report MUST contain The Queue/Scheduler Stateless Traffic individual report MUST
all results for each QL/BB test run and a recommended format is as contain all results for each QL/BB test run. A recommended format
follows: is as follows:
******************************************************** ****************************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Scheduling technique, BB and QL Test Configuration Summary: Tr, Td
The results table should contain entries for each test run DUT Configuration Summary: Scheduling technique (i.e., FIFO, SP,
as follows, WFQ, etc.), BB, and QL
The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):
(Test #1 to Test #Tr). - LP, OOS, PD, and PDV
- LP, OOS, PD, and PDV ****************************************************************
********************************************************
6.2.1.2 Testing Queue/Scheduler with Stateful Traffic 6.2.1.2. Testing Queue/Scheduler with Stateful Traffic
Objective: Objective:
Verify that the configured queue and scheduling technique can handle Verify that the configured queue and scheduling technique can
stateful traffic bursts up to the queue depth. handle stateful traffic bursts up to the queue depth.
Test Background and Summary: Test Background and Summary:
To provide a more realistic benchmark and to test queues in layer 4 To provide a more realistic benchmark and to test queues in
devices such as firewalls, stateful traffic testing is recommended Layer 4 devices such as firewalls, stateful traffic testing is
for the queue tests. Stateful traffic tests will also utilize the recommended for the queue tests. Stateful traffic tests will also
Network Delay Emulator (NDE) from the network set-up configuration in utilize the Network Delay Emulator (NDE) from the network setup
section 2. configuration in Section 1.2.
The BDP of the TCP test traffic must be calibrated to the QL of the The BDP of the TCP test traffic must be calibrated to the QL of
device queue. Referencing [RFC6349], the BDP is equal to: the device queue. Referencing [RFC6349], the BDP is equal to:
BB * RTT / 8 (in bytes) BB * RTT / 8 (in bytes)
The NDE must be configured to an RTT value which is large enough to The NDE must be configured to an RTT value that is large enough to
allow the BDP to be greater than QL. An example test scenario is allow the BDP to be greater than QL. An example test scenario is
defined below: defined below:
- Ingress link = GigE - Ingress link = GigE
- Egress link = 100 Mbps (BB)
- QL = 32KB
RTT(min) = QL * 8 / BB and would equal 2.56 ms (and the BDP = 32KB) - Egress link = 100 Mbps (BB)
In this example, one (1) TCP connection with window size / SSB of - QL = 32 KB
32KB would be required to test the QL of 32KB. This Bulk Transfer
Test can be accomplished using iperf as described in Appendix A.
Two types of TCP tests MUST be performed: Bulk Transfer test and RTT(min) = QL * 8 / BB and would equal 2.56 ms
Micro Burst Test Pattern as documented in Appendix B. The Bulk (and the BDP = 32 KB)
Transfer Test only bursts during the TCP Slow Start (or Congestion
Avoidance) state, while the Micro Burst test emulates application
layer bursting which may occur any time during the TCP connection.
Other tests types SHOULD include: Simple Web Site, Complex Web Site, In this example, one (1) TCP connection with window size / SSB of
Business Applications, Email, SMB/CIFS File Copy (which are also 32 KB would be required to test the QL of 32 KB. This Bulk
documented in Appendix B). Transfer Test can be accomplished using iperf, as described in
Appendix A.
Test Metrics: Two types of TCP tests MUST be performed: the Bulk Transfer Test
The test results will be recorded per the stateful metrics defined in and the Micro Burst Test Pattern, as documented in Appendix B.
section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), The Bulk Transfer Test only bursts during the TCP Slow Start (or
TCP Efficiency, and Buffer Delay. Congestion Avoidance) state, while the Micro Burst Test Pattern
emulates application-layer bursting, which may occur any time
during the TCP connection.
Procedure: Other types of tests SHOULD include the following: simple web
sites, complex web sites, business applications, email, and
SMB/CIFS (Common Internet File System) file copy (all of which are
also documented in Appendix B).
1. Configure the DUT queue length (QL) and scheduling technique Test Metrics:
(FIFO, SP, etc) parameters The test results will be recorded per the stateful metrics defined
in Section 4.2 -- primarily the TCP Test Pattern Execution Time
(TTPET), TCP Efficiency, and Buffer Delay.
2. Configure the test generator* with a profile of an emulated Procedure:
application traffic mixture 1. Configure the DUT QL and scheduling technique parameters (FIFO,
SP, etc.).
- The application mixture MUST be defined in terms of percentage 2. Configure the test generator* with a profile of an emulated
of the total bandwidth to be tested application traffic mixture.
- The rate of transmission for each application within the mixture - The application mixture MUST be defined in terms of
MUST be also be configurable percentage of the total bandwidth to be tested.
* The test generator MUST be capable of generating precise TCP - The rate of transmission for each application within the
test patterns for each application specified, to ensure repeatable mixture MUST also be configurable.
results.
3. Generate application traffic between the ingress (client side) and * To ensure repeatable results, the test generator MUST be
egress (server side) ports of the DUT and measure application capable of generating precise TCP test patterns for each
throughput the metrics (TTPET, TCP Efficiency, and Buffer Delay), application specified.
per application stream and at the ingress and egress port (across 3. Generate application traffic between the ingress (client side)
the entire Td, default 60 seconds duration). and egress (server side) ports of the DUT, and measure the
metrics (TTPET, TCP Efficiency, and Buffer Delay) per
application stream and at the ingress and egress ports (across
the entire Td, default 60-second duration).
Concerning application measurements, a couple of items require A couple of items require clarification concerning application
clarification. An application session may be comprised of a single measurements: an application session may be comprised of a single
TCP connection or multiple TCP connections. TCP connection or multiple TCP connections.
For the single TCP connection application sessions, the application If an application session utilizes a single TCP connection, the
thoughput / metrics have a 1-1 relationship to the TCP connection application throughput/metrics have a 1-1 relationship to the TCP
measurements. connection measurements.
If an application session (i.e. HTTP-based application) utilizes If an application session (e.g., an HTTP-based application)
multiple TCP connections, then all of the TCP connections are utilizes multiple TCP connections, then all of the TCP connections
aggregated in the application throughput measurement / metrics for are aggregated in the application throughput measurement/metrics
that application. for that application.
Then there is the case of mulitlple instances of an application Then, there is the case of multiple instances of an application
session (i.e. multiple FTPs emulating multiple clients). In this session (i.e., multiple FTPs emulating multiple clients). In this
situation, the test should measure / record each FTP application situation, the test should measure/record each FTP application
session independently, tabulating the minimum, maximum, and average session independently, tabulating the minimum, maximum, and
for all FTP sessions. average for all FTP sessions.
Finally, application throughput measurements are based on Layer 4 Finally, application throughput measurements are based on Layer 4
TCP throughput and do not include bytes retransmitted. The TCP TCP throughput and do not include bytes retransmitted. The TCP
Efficiency metric MUST be measured during the test and provides a Efficiency metric MUST be measured during the test, because it
measure of "goodput" during each test. provides a measure of "goodput" during each test.
Reporting Format: Reporting Format:
The Queue/Scheduler Stateful Traffic individual report MUST contain The Queue/Scheduler Stateful Traffic individual report MUST
all results for each traffic scheduler and QL/BB test run and a contain all results for each traffic scheduler and QL/BB test run.
recommended format is as follows: A recommended format is as follows:
******************************************************** ******************************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Scheduling technique, BB and QL Test Configuration Summary: Tr, Td
Application Mixture and Intensities: this is the percent configured DUT Configuration Summary: Scheduling technique (i.e., FIFO, SP,
of each application type WFQ, etc.), BB, and QL
The results table should contain entries for each test run with Application Mixture and Intensities: These are the percentages
minimum, maximum, and average per application session as follows, configured for each application type.
(Test #1 to Test #Tr)
- Per Application Throughout (bps) and TTPET The results table should contain entries for each test run, with
- Per Application Bytes In and Bytes Out minimum, maximum, and average per application session, as follows
- Per Application TCP Efficiency, and Buffer Delay (Test #1 to Test #Tr):
********************************************************
6.2.2 Queue / Scheduler Capacity Tests - Throughput (bps) and TTPET for each application session
- Bytes In and Bytes Out for each application session
- TCP Efficiency and Buffer Delay for each application session
******************************************************************
6.2.2. Queue/Scheduler Capacity Tests
Objective: Objective:
The intent of these capacity tests is to benchmark queue/scheduler The intent of these capacity tests is to benchmark queue/scheduler
performance in a scaled environment with multiple queues/schedulers performance in a scaled environment with multiple
active on multiple egress physical ports. This test will benchmark queues/schedulers active on multiple egress physical ports. These
the maximum number of queues and schedulers as specified by the tests will benchmark the maximum number of queues and schedulers
device manufacturer. Each priority in the system will map to a as specified by the device manufacturer. Each priority in the
separate queue. system will map to a separate queue.
Test Metrics: Test Metrics:
The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL The metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV)
be measured at the egress port and recorded. SHALL be measured at the egress port and recorded.
The following sections provide the specific test scenarios, The following sections provide the specific test scenarios,
procedures, and reporting formats for each queue / scheduler capacity procedures, and reporting formats for each queue/scheduler capacity
test. test.
6.2.2.1 Multiple Queues / Single Port Active 6.2.2.1. Multiple Queues, Single Port Active
For the first scheduler / queue capacity test, multiple queues per For the first queue/scheduler capacity test, multiple queues per port
port will be tested on a single physical port. In this case, will be tested on a single physical port. In this case, all of the
all the queues (typically 8) are active on a single physical port. queues (typically eight) are active on a single physical port.
Traffic from multiple ingress physical ports are directed to the Traffic from multiple ingress physical ports is directed to the same
same egress physical port which will cause oversubscription on the egress physical port. This will cause oversubscription on the egress
egress physical port. physical port.
There are many types of priority schemes and combinations of There are many types of priority schemes and combinations of
priorities that are managed by the scheduler. The following priorities that are managed by the scheduler. The following sections
sections specify the priority schemes that should be tested. specify the priority schemes that should be tested.
6.2.2.1.1 Strict Priority on Egress Port 6.2.2.1.1. Strict Priority on Egress Port
Test Summary: Test Summary:
For this test, Strict Priority (SP) scheduling on the egress For this test, SP scheduling on the egress physical port should be
physical port should be tested and the benchmarking methodology tested, and the benchmarking methodologies specified in
specified in section 6.2.1.1 and 6.2.1.2 (procedure, metrics, Sections 6.2.1.1 (stateless) and 6.2.1.2 (stateful) (procedure,
and reporting format) should be applied here. For a given metrics, and reporting format) should be applied here. For a
priority, each ingress physical port should get a fair share of given priority, each ingress physical port should get a fair share
the egress physical port bandwidth. of the egress physical-port bandwidth.
Since this is a capacity test, the configuration and report Since this is a capacity test, the configuration and report
results format from 6.2.1.1 and 6.2.1.2 MUST also include: results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also
include:
Configuration: Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate
/ mixture for stateful traffic for each physical ingress port
Report results: - The number of physical ingress ports active during the test
- For each ingress port traffic stream, the achieved throughput
rate and metrics at the egress port
6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ) on Egress Port - The classification marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic
rate/mixture for stateful traffic for each physical
ingress port
Report Results:
- For each ingress port traffic stream, the achieved throughput
rate and metrics at the egress port
6.2.2.1.2. Strict Priority + WFQ on Egress Port
Test Summary: Test Summary:
For this test, Strict Priority (SP) and Weighted Fair Queue (WFQ) For this test, SP and WFQ should be enabled simultaneously in the
should be enabled simultaneously in the scheduler but on a single scheduler, but on a single egress port. The benchmarking
egress port. The benchmarking methodology specified in Section methodologies specified in Sections 6.2.1.1 (stateless) and
6.2.1.2 (stateful) (procedure, metrics, and reporting format)
should be applied here. Additionally, the egress port
bandwidth-sharing among weighted queues should be proportional to
the assigned weights. For a given priority, each ingress physical
port should get a fair share of the egress physical-port
bandwidth.
6.2.1.1 and 6.2.1.2 (procedure, metrics, and reporting format) Since this is a capacity test, the configuration and report
should be applied here. Additionally, the egress port bandwidth results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also
sharing among weighted queues should be proportional to the assigned include:
weights. For a given priority, each ingress physical port should get
a fair share of the egress physical port bandwidth.
Since this is a capacity test, the configuration and report results Configuration:
format from 6.2.1.1 and 6.2.1.2 MUST also include:
Configuration: - The number of physical ingress ports active during the test
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
Report results: - The classification marking (DSCP, VLAN, etc.) for each physical
- For each ingress port traffic stream, the achieved throughput rate ingress port
and metrics at each queue of the egress port queue (both the SP
and WFQ queue).
Example: - The traffic rate for stateful traffic and the traffic
- Egress Port SP Queue: throughput and metrics for ingress streams rate/mixture for stateful traffic for each physical
1-n ingress port
- Egress Port WFQ Queue: throughput and metrics for ingress streams
1-n
6.2.2.2 Single Queue per Port / All Ports Active Report Results:
- For each ingress port traffic stream, the achieved throughput
rate and metrics at each queue of the egress port queue (both
the SP and WFQ)
Example:
- Egress Port SP Queue: throughput and metrics for ingress
streams 1-n
- Egress Port WFQ: throughput and metrics for ingress streams 1-n
6.2.2.2. Single Queue per Port, All Ports Active
Test Summary: Test Summary:
Traffic from multiple ingress physical ports are directed to the Traffic from multiple ingress physical ports is directed to the
same egress physical port, which will cause oversubscription on the same egress physical port. This will cause oversubscription on
egress physical port. Also, the same amount of traffic is directed the egress physical port. Also, the same amount of traffic is
to each egress physical port. directed to each egress physical port.
The benchmarking methodology specified in Section 6.2.1.1 The benchmarking methodologies specified in Sections 6.2.1.1
and 6.2.1.2 (procedure, metrics, and reporting format) should be (stateless) and 6.2.1.2 (stateful) (procedure, metrics, and
applied here. Each ingress physical port should get a fair share of reporting format) should be applied here. Each ingress physical
the egress physical port bandwidth. Additionally, each egress port should get a fair share of the egress physical-port
physical port should receive the same amount of traffic. bandwidth. Additionally, each egress physical port should receive
the same amount of traffic.
Since this is a capacity test, the configuration and report results Since this is a capacity test, the configuration and report
format from 6.2.1.1 and 6.2.1.2 MUST also include: results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also
include:
Configuration: Configuration:
- The number of ingress ports active during the test
- The number of egress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
Report results: - The number of ingress ports active during the test
- For each egress port, the achieved throughput rate and metrics at
the egress port queue for each ingress port stream.
Example: - The number of egress ports active during the test
- Egress Port 1: throughput and metrics for ingress streams 1-n
- Egress Port n: throughput and metrics for ingress streams 1-n
6.2.2.3 Multiple Queues per Port, All Ports Active - The classification marking (DSCP, VLAN, etc.) for each physical
ingress port
Traffic from multiple ingress physical ports are directed to all - The traffic rate for stateful traffic and the traffic
queues of each egress physical port, which will cause rate/mixture for stateful traffic for each physical
oversubscription on the egress physical ports. Also, the same ingress port
amount of traffic is directed to each egress physical port.
The benchmarking methodology specified in Section 6.2.1.1 Report Results:
and 6.2.1.2 (procedure, metrics, and reporting format) should be
applied here. For a given priority, each ingress physical port
should get a fair share of the egress physical port bandwidth.
Additionally, each egress physical port should receive the same
amount of traffic.
Since this is a capacity test, the configuration and report results - For each egress port, the achieved throughput rate and metrics
format from 6.2.1.1 and 6.2.1.2 MUST also include: at the egress port queue for each ingress port stream
Configuration: Example:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
Report results: - Egress Port 1: throughput and metrics for ingress streams 1-n
- For each egress port, the achieved throughput rate and metrics at
each egress port queue for each ingress port stream.
Example: - Egress Port n: throughput and metrics for ingress streams 1-n
- Egress Port 1, SP Queue: throughput and metrics for ingress
streams 1-n
- Egress Port 2, WFQ Queue: throughput and metrics for ingress
streams 1-n
.
.
- Egress Port n, SP Queue: throughput and metrics for ingress
streams 1-n
- Egress Port n, WFQ Queue: throughput and metrics for ingress
streams 1-n
6.3. Shaper tests 6.2.2.3. Multiple Queues per Port, All Ports Active
A traffic shaper is memory based like a queue, but with the added Test Summary:
intelligence of an active traffic scheduler. The same concepts from Traffic from multiple ingress physical ports is directed to all
section 6.2 (Queue testing) can be applied to testing network device queues of each egress physical port. This will cause
shaper. oversubscription on the egress physical ports. Also, the same
amount of traffic is directed to each egress physical port.
Again, the tests are divided into two sections; individual shaper The benchmarking methodologies specified in Sections 6.2.1.1
benchmark tests and then full capacity shaper benchmark tests. (stateless) and 6.2.1.2 (stateful) (procedure, metrics, and
reporting format) should be applied here. For a given priority,
each ingress physical port should get a fair share of the egress
physical-port bandwidth. Additionally, each egress physical port
should receive the same amount of traffic.
6.3.1 Shaper Individual Tests Overview Since this is a capacity test, the configuration and report
results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also
include:
Configuration:
- The number of physical ingress ports active during the test
- The classification marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic
rate/mixture for stateful traffic for each physical
ingress port
Report Results:
- For each egress port, the achieved throughput rate and metrics
at each egress port queue for each ingress port stream
Example:
- Egress Port 1, SP Queue: throughput and metrics for ingress
streams 1-n
- Egress Port 2, WFQ: throughput and metrics for ingress
streams 1-n
...
- Egress Port n, SP Queue: throughput and metrics for ingress
streams 1-n
- Egress Port n, WFQ: throughput and metrics for ingress
streams 1-n
6.3. Shaper Tests
Like a queue, a traffic shaper is memory based, but with the added
intelligence of an active traffic scheduler. The same concepts as
those described in Section 6.2 (queue testing) can be applied to
testing a network device shaper.
Again, the tests are divided into two sections: individual shaper
benchmark tests and then full-capacity shaper benchmark tests.
6.3.1. Shaper Individual Tests
A traffic shaper generally has three (3) components that can be A traffic shaper generally has three (3) components that can be
configured: configured:
- Ingress Queue bytes - Ingress Queue bytes
- Shaper Rate, bps
- Burst Committed (Bc) and Burst Excess (Be), bytes
The Ingress Queue holds burst traffic and the shaper then meters - Shaper Rate (SR), bps
traffic out of the egress port according to the Shaper Rate and
Bc/Be parameters. Shapers generally transmit into policers, so
the idea is for the emitted traffic to conform to the policer's
limits.
6.3.1.1 Testing Shaper with Stateless Traffic - Burst Committed (Bc) and Burst Excess (Be), bytes
The Ingress Queue holds burst traffic, and the shaper then meters
traffic out of the egress port according to the SR and Bc/Be
parameters. Shapers generally transmit into policers, so the idea is
for the emitted traffic to conform to the policer's limits.
6.3.1.1. Testing Shaper with Stateless Traffic
Objective: Objective:
Test a shaper by transmitting stateless traffic bursts into the Test a shaper by transmitting stateless traffic bursts into the
shaper ingress port and verifying that the egress traffic is shaped shaper ingress port and verifying that the egress traffic is
according to the shaper traffic profile. shaped according to the shaper traffic profile.
Test Summary: Test Summary:
The stateless traffic must be burst into the DUT ingress port and The stateless traffic must be burst into the DUT ingress port and
not exceed the Ingress Queue. The burst can be a single burst or not exceed the Ingress Queue. The burst can be a single burst or
multiple bursts. If multiple bursts are transmitted, then the multiple bursts. If multiple bursts are transmitted, then the
Ti (Time interval) must be large enough so that the Shaper Rate is transmission interval (Ti) must be large enough so that the SR is
not exceeded. An example will clarify single and multiple burst not exceeded. An example will clarify single-burst and multiple-
test cases. burst test cases.
In the example, the shaper's ingress and egress ports are both full In this example, the shaper's ingress and egress ports are both
duplex Gigabit Ethernet. The Ingress Queue is configured to be full-duplex Gigabit Ethernet. The Ingress Queue is configured to
512,000 bytes, the Shaper Rate (SR) = 50 Mbps, and both Bc/Be be 512,000 bytes, the SR = 50 Mbps, and both Bc and Be are
configured to be 32,000 bytes. For a single burst test, the configured to be 32,000 bytes. For a single-burst test, the
transmitting test device would burst 512,000 bytes maximum into the transmitting test device would burst 512,000 bytes maximum into
ingress port and then stop transmitting. the ingress port and then stop transmitting.
If a multiple burst test is to be conducted, then the burst bytes If a multiple-burst test is to be conducted, then the burst bytes
divided by the time interval between the 512,000 byte bursts must divided by the transmission interval between the 512,000-byte
not exceed the Shaper Rate. The time interval (Ti) must adhere to bursts must not exceed the SR. The transmission interval (Ti)
a similar formula as described in section 6.2.1.1 for queues, namely: must adhere to a formula similar to the formula described in
Section 6.2.1.1 for queues, namely:
Ti = Ingress Queue x 8 / Shaper Rate Ti = Ingress Queue * 8 / SR
For the example from the previous paragraph, Ti between bursts must For the example from the previous paragraph, the Ti between bursts
be greater than 82 millisecond (512,000 bytes x 8 / 50,000,000 bps). must be greater than 82 milliseconds (512,000 bytes * 8 /
This yields an average rate of 50 Mbps so that an Input Queue 50,000,000 bps). This yields an average rate of 50 Mbps so that
would not overflow. an Ingress Queue would not overflow.
Test Metrics: Test Metrics:
- The metrics defined in section 4.1 (LP, OOS, PDV, SR, SBB, SBI) The metrics defined in Section 4.1 (LP, OOS, PDV, SR, SBB, and
SHALL be measured at the egress port and recorded. SBI) SHALL be measured at the egress port and recorded.
Procedure: Procedure:
1. Configure the DUT shaper ingress queue length (QL) and shaper 1. Configure the DUT shaper ingress QL and shaper egress rate
egress rate parameters (SR, Bc, Be) parameters parameters (SR, Bc, Be).
2. Configure the tester to generate a stateless traffic burst equal 2. Configure the tester to generate a stateless traffic burst
to QL and an interval equal to Ti (QL in bits/BB) equal to QL and an interval equal to Ti (QL in bits/BB).
3. Generate bursts of QL traffic into the DUT and measure the metrics 3. Generate bursts of QL traffic into the DUT, and measure the
defined in section 4.1 (LP, OOS, PDV, SR, SBB, SBI) at the egress metrics defined in Section 4.1 (LP, OOS, PDV, SR, SBB, and SBI)
port and across the entire Td (default 30 seconds duration) at the egress port and across the entire Td (default 30-second
duration).
Report Format: Reporting Format:
The Shaper Stateless Traffic individual report MUST contain all The Shaper Stateless Traffic individual report MUST contain all
results for each QL/SR test run and a recommended format is as results for each QL/SR test run. A recommended format is as
follows: follows:
********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Ingress Burst Rate, QL, SR ***********************************************************
The results table should contain entries for each test run as Test Configuration Summary: Tr, Td
follows,(Test #1 to Test #Tr).
- LP, OOS, PDV, SR, SBB, SBI DUT Configuration Summary: Ingress Burst Rate, QL, SR
********************************************************
6.3.1.2 Testing Shaper with Stateful Traffic The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):
- LP, OOS, PDV, SR, SBB, and SBI
***********************************************************
6.3.1.2. Testing Shaper with Stateful Traffic
Objective: Objective:
Test a shaper by transmitting stateful traffic bursts into the shaper Test a shaper by transmitting stateful traffic bursts into the
ingress port and verifying that the egress traffic is shaped shaper ingress port and verifying that the egress traffic is
according to the shaper traffic profile. shaped according to the shaper traffic profile.
Test Summary: Test Summary:
To provide a more realistic benchmark and to test queues in layer 4 To provide a more realistic benchmark and to test queues in
devices such as firewalls, stateful traffic testing is also Layer 4 devices such as firewalls, stateful traffic testing is
recommended for the shaper tests. Stateful traffic tests will also also recommended for the shaper tests. Stateful traffic tests
utilize the Network Delay Emulator (NDE) from the network set-up will also utilize the Network Delay Emulator (NDE) from the
configuration in section 2. network setup configuration in Section 1.2.
The BDP of the TCP test traffic must be calculated as described in
section 6.2.2. To properly stress network buffers and the traffic
shaping function, the cumulative TCP window should exceed the BDP
which will stress the shaper. BDP factors of 1.1 to 1.5 are
recommended, but the values are the discretion of the tester and
should be documented.
The cumulative TCP Window Sizes* (RWND at the receiving end & CWND The BDP of the TCP test traffic must be calculated as described in
at the transmitting end) equates to: Section 6.2.1.2. To properly stress network buffers and the
traffic-shaping function, the TCP window size (which is the
minimum of the TCP RWND and sender socket) should be greater than
the BDP, which will stress the shaper. BDP factors of 1.1 to 1.5
are recommended, but the values are left to the discretion of the
tester and should be documented.
TCP window size* for each connection x number of connections The cumulative TCP window sizes* (RWND at the receiving end and
CWND at the transmitting end) equates to the TCP window size* for
each connection, multiplied by the number of connections.
* as described in section 3 of [RFC6349], the SSB MUST be large * As described in Section 3 of [RFC6349], the SSB MUST be large
enough to fill the BDP enough to fill the BDP.
Example, if the BDP is equal to 256 Kbytes and a connection size of For example, if the BDP is equal to 256 KB and a connection size
64Kbytes is used for each connection, then it would require four (4) of 64 KB is used for each connection, then it would require four
connections to fill the BDP and 5-6 connections (over subscribe the (4) connections to fill the BDP and 5-6 connections (oversubscribe
BDP) to stress test the traffic shaping function. the BDP) to stress-test the traffic-shaping function.
Two types of TCP tests MUST be performed: Bulk Transfer test and Two types of TCP tests MUST be performed: the Bulk Transfer Test
Micro Burst Test Pattern as documented in Appendix B. The Bulk and the Micro Burst Test Pattern, as documented in Appendix B.
Transfer Test only bursts during the TCP Slow Start (or Congestion The Bulk Transfer Test only bursts during the TCP Slow Start (or
Avoidance) state, while the Micro Burst test emulates application Congestion Avoidance) state, while the Micro Burst Test Pattern
layer bursting which may any time during the TCP connection. emulates application-layer bursting, which may occur any time
during the TCP connection.
Other tests types SHOULD include: Simple Web Site, Complex Web Site, Other types of tests SHOULD include the following: simple web
Business Applications, Email, SMB/CIFS File Copy (which are also sites, complex web sites, business applications, email, and
documented in Appendix B). SMB/CIFS file copy (all of which are also documented in
Appendix B).
Test Metrics: Test Metrics:
The test results will be recorded per the stateful metrics defined in The test results will be recorded per the stateful metrics defined
section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), in Section 4.2 -- primarily the TCP Test Pattern Execution Time
TCP Efficiency, and Buffer Delay. (TTPET), TCP Efficiency, and Buffer Delay.
Procedure: Procedure:
1. Configure the DUT shaper ingress queue length (QL) and shaper 1. Configure the DUT shaper ingress QL and shaper egress rate
egress rate parameters (SR, Bc, Be) parameters parameters (SR, Bc, Be).
2. Configure the test generator* with a profile of an emulated 2. Configure the test generator* with a profile of an emulated
application traffic mixture application traffic mixture.
- The application mixture MUST be defined in terms of percentage - The application mixture MUST be defined in terms of
of the total bandwidth to be tested percentage of the total bandwidth to be tested.
- The rate of transmission for each application within the mixture - The rate of transmission for each application within the
MUST be also be configurable mixture MUST also be configurable.
* The test generator MUST be capable of generating precise TCP * To ensure repeatable results, the test generator MUST be
test patterns for each application specified, to ensure repeatable capable of generating precise TCP test patterns for each
results. application specified.
3. Generate application traffic between the ingress (client side) and 3. Generate application traffic between the ingress (client side)
egress (server side) ports of the DUT and measure the metrics and egress (server side) ports of the DUT, and measure the
(TTPET, TCP Efficiency, and Buffer Delay) per application stream metrics (TTPET, TCP Efficiency, and Buffer Delay) per
and at the ingress and egress port (across the entire Td, default application stream and at the ingress and egress ports (across
30 seconds duration). the entire Td, default 30-second duration).
Reporting Format: Reporting Format:
The Shaper Stateful Traffic individual report MUST contain all The Shaper Stateful Traffic individual report MUST contain all
results for each traffic scheduler and QL/SR test run and a results for each traffic scheduler and QL/SR test run. A
recommended format is as follows: recommended format is as follows:
******************************************************** ******************************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Ingress Burst Rate, QL, SR Test Configuration Summary: Tr, Td
Application Mixture and Intensities: this is the percent configured DUT Configuration Summary: Ingress Burst Rate, QL, SR
of each application type
The results table should contain entries for each test run with
minimum, maximum, and average per application session as follows,
(Test #1 to Test #Tr)
- Per Application Throughout (bps) and TTPET Application Mixture and Intensities: These are the percentages
- Per Application Bytes In and Bytes Out configured for each application type.
- Per Application TCP Efficiency, and Buffer Delay
********************************************************
6.3.2 Shaper Capacity Tests The results table should contain entries for each test run, with
minimum, maximum, and average per application session, as follows
(Test #1 to Test #Tr):
- Throughput (bps) and TTPET for each application session
- Bytes In and Bytes Out for each application session
- TCP Efficiency and Buffer Delay for each application session
******************************************************************
6.3.2. Shaper Capacity Tests
Objective: Objective:
The intent of these scalability tests is to verify shaper performance The intent of these scalability tests is to verify shaper
in a scaled environment with shapers active on multiple queues on performance in a scaled environment with shapers active on
multiple egress physical ports. This test will benchmark the maximum multiple queues on multiple egress physical ports. These tests
number of shapers as specified by the device manufacturer. will benchmark the maximum number of shapers as specified by the
device manufacturer.
The following sections provide the specific test scenarios, The following sections provide the specific test scenarios,
procedures, and reporting formats for each shaper capacity test. procedures, and reporting formats for each shaper capacity test.
6.3.2.1 Single Queue Shaped, All Physical Ports Active 6.3.2.1. Single Queue Shaped, All Physical Ports Active
Test Summary: Test Summary:
The first shaper capacity test involves per port shaping, all The first shaper capacity test involves per-port shaping with all
physical ports active. Traffic from multiple ingress physical ports physical ports active. Traffic from multiple ingress physical
are directed to the same egress physical port and this will cause ports is directed to the same egress physical port. This will
oversubscription on the egress physical port. Also, the same amount cause oversubscription on the egress physical port. Also, the
of traffic is directed to each egress physical port. same amount of traffic is directed to each egress physical port.
The benchmarking methodology specified in Section 6.3.1 (procedure, The benchmarking methodologies specified in Sections 6.3.1.1
metrics, and reporting format) should be applied here. Since this is (stateless) and 6.3.1.2 (stateful) (procedure, metrics, and
a capacity test, the configuration and report results format from reporting format) should be applied here. Since this is a
6.3.1 MUST also include: capacity test, the configuration and report results format (see
Section 6.3.1) MUST also include:
Configuration: Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
- The shaped egress ports shaper parameters (QL, SR, Bc, Be)
Report results: - The number of physical ingress ports active during the test
- For each active egress port, the achieved throughput rate and
shaper metrics for each ingress port traffic stream
Example: - The classification marking (DSCP, VLAN, etc.) for each physical
- Egress Port 1: throughput and metrics for ingress streams 1-n ingress port
- Egress Port n: throughput and metrics for ingress streams 1-n
6.3.2.2 All Queues Shaped, Single Port Active - The traffic rate for stateful traffic and the traffic
rate/mixture for stateful traffic for each physical
ingress port
- The shaped egress port shaper parameters (QL, SR, Bc, Be)
Report Results:
- For each active egress port, the achieved throughput rate and
shaper metrics for each ingress port traffic stream
Example:
- Egress Port 1: throughput and metrics for ingress streams 1-n
- Egress Port n: throughput and metrics for ingress streams 1-n
6.3.2.2. All Queues Shaped, Single Port Active
Test Summary: Test Summary:
The second shaper capacity test is conducted with all queues actively The second shaper capacity test is conducted with all queues
shaping on a single physical port. The benchmarking methodology actively shaping on a single physical port. The benchmarking
described in per port shaping test (previous section) serves as the methodology described in the per-port shaping test
foundation for this. Additionally, each of the SP queues on the (Section 6.3.2.1) serves as the foundation for this.
egress physical port is configured with a shaper. For the highest Additionally, each of the SP queues on the egress physical port is
priority queue, the maximum amount of bandwidth available is limited configured with a shaper. For the highest-priority queue, the
by the bandwidth of the shaper. For the lower priority queues, the maximum amount of bandwidth available is limited by the bandwidth
maximum amount of bandwidth available is limited by the bandwidth of of the shaper. For the lower-priority queues, the maximum amount
the shaper and traffic in higher priority queues. of bandwidth available is limited by the bandwidth of the shaper
and traffic in higher-priority queues.
The benchmarking methodology specified in Section 6.3.1 (procedure, The benchmarking methodologies specified in Sections 6.3.1.1
metrics, and reporting format) should be applied here. Since this is (stateless) and 6.3.1.2 (stateful) (procedure, metrics, and
a capacity test, the configuration and report results format from reporting format) should be applied here. Since this is a
6.3.1 MUST also include: capacity test, the configuration and report results format (see
Section 6.3.1) MUST also include:
Configuration: Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate/mixture
for stateful traffic for each physical ingress port
- For the active egress port, each shaper queue parameters (QL, SR,
Bc, Be)
Report results: - The number of physical ingress ports active during the test
- For each queue of the active egress port, the achieved throughput
rate and shaper metrics for each ingress port traffic stream
Example: - The classification marking (DSCP, VLAN, etc.) for each physical
- Egress Port High Priority Queue: throughput and metrics for ingress port
ingress streams 1-n
- Egress Port Lower Priority Queue: throughput and metrics for
ingress streams 1-n
6.3.2.3 All Queues Shaped, All Ports Active - The traffic rate for stateful traffic and the traffic
rate/mixture for stateful traffic for each physical
ingress port
- For the active egress port, each of the following shaper queue
parameters: QL, SR, Bc, Be
Report Results:
- For each queue of the active egress port, the achieved
throughput rate and shaper metrics for each ingress port
traffic stream
Example:
- Egress Port High-Priority Queue: throughput and metrics for
ingress streams 1-n
- Egress Port Lower-Priority Queue: throughput and metrics for
ingress streams 1-n
6.3.2.3. All Queues Shaped, All Ports Active
Test Summary: Test Summary:
And for the third shaper capacity test (which is a combination of the For the third shaper capacity test (which is a combination of the
tests in the previous two sections),all queues will be actively tests listed in Sections 6.3.2.1 and 6.3.2.2), all queues will be
shaping and all physical ports active. actively shaping and all physical ports active.
The benchmarking methodology specified in Section 6.3.1 (procedure, The benchmarking methodologies specified in Sections 6.3.1.1
metrics, and reporting format) should be applied here. Since this is (stateless) and 6.3.1.2 (stateful) (procedure, metrics, and
a capacity test, the configuration and report results format from reporting format) should be applied here. Since this is a
6.3.1 MUST also include: capacity test, the configuration and report results format (see
Section 6.3.1) MUST also include:
Configuration: Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
- For each of the active egress ports, shaper port and per queue
parameters(QL, SR, Bc, Be)
Report results: - The number of physical ingress ports active during the test
- For each queue of each active egress port, the achieved throughput
rate and shaper metrics for each ingress port traffic stream
Example: - The classification marking (DSCP, VLAN, etc.) for each physical
- Egress Port 1 High Priority Queue: throughput and metrics for ingress port
ingress streams 1-n
- Egress Port 1 Lower Priority Queue: throughput and metrics for
ingress streams 1-n
.
- Egress Port n High Priority Queue: throughput and metrics for
ingress streams 1-n
- Egress Port n Lower Priority Queue: throughput and metrics for
ingress streams 1-n
6.4 Concurrent Capacity Load Tests - The traffic rate for stateful traffic and the traffic
rate/mixture for stateful traffic for each physical
ingress port
As mentioned in the scope of this document, it is impossible to - For each of the active egress ports: shaper port parameters and
per-queue parameters (QL, SR, Bc, Be)
Report Results:
- For each queue of each active egress port, the achieved
throughput rate and shaper metrics for each ingress port
traffic stream
Example:
- Egress Port 1, High-Priority Queue: throughput and metrics for
ingress streams 1-n
- Egress Port 1, Lower-Priority Queue: throughput and metrics for
ingress streams 1-n
...
- Egress Port n, High-Priority Queue: throughput and metrics for
ingress streams 1-n
- Egress Port n, Lower-Priority Queue: throughput and metrics for
ingress streams 1-n
6.4. Concurrent Capacity Load Tests
As mentioned in Section 3 of this document, it is impossible to
specify the various permutations of concurrent traffic management specify the various permutations of concurrent traffic management
functions that should be tested in a device for capacity testing. functions that should be tested in a device for capacity testing.
However, some profiles are listed below which may be useful However, some profiles are listed below that may be useful for
to test under capacity as well: testing multiple configurations of traffic management functions:
- Policers on ingress and queuing on egress - Policers on ingress and queuing on egress
- Policers on ingress and shapers on egress (not intended for a
flow to be policed then shaped, these would be two different
flows tested at the same time)
- etc.
The test procedures and reporting formatting from the previous - Policers on ingress and shapers on egress (not intended for a flow
sections may be modified to accommodate the capacity test profile. to be policed and then shaped; these would be two different flows
tested at the same time)
7. Security Considerations The test procedures and reporting formats from Sections 6.1, 6.2,
and 6.3 may be modified to accommodate the capacity test profile.
7. Security Considerations
Documents of this type do not directly affect the security of the Documents of this type do not directly affect the security of the
Internet or of corporate networks as long as benchmarking is not Internet or of corporate networks as long as benchmarking is not
performed on devices or systems connected to production networks. performed on devices or systems connected to production networks.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black box" basis, relying
solely on measurements observable external to the DUT/SUT. solely on measurements observable external to the DUT/SUT.
Special capabilities SHOULD NOT exist in the DUT/SUT specifically for Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
benchmarking purposes. Any implications for network security arising benchmarking purposes. Any implications for network security arising
from the DUT/SUT SHOULD be identical in the lab and in production from the DUT/SUT SHOULD be identical in the lab and in production
networks. networks.
8. IANA Considerations 8. References
This document does not REQUIRE an IANA registration for ports 8.1. Normative References
dedicated to the TCP testing described in this document.
9. References [3GPP2-C_R1002-A]
3rd Generation Partnership Project 2, "cdma2000 Evaluation
Methodology", Version 1.0, Revision A, May 2009,
<http://www.3gpp2.org/public_html/specs/
C.R1002-A_v1.0_Evaluation_Methodology.pdf>.
9.1. Normative References [RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
July 1991, <http://www.rfc-editor.org/info/rfc1242>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC2119, March 1997. Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC1242] S. Bradner, "Benchmarking Terminology for Network [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Interconnection Devices," RFC1242 July 1991 Network Interconnect Devices", RFC 2544,
DOI 10.17487/RFC2544, March 1999,
<http://www.rfc-editor.org/info/rfc2544>.
[RFC2544] S. Bradner, "Benchmarking Methodology for Network [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Interconnect Devices," RFC2544 March 1999 Packet Loss Metric for IPPM", RFC 2680,
DOI 10.17487/RFC2680, September 1999,
<http://www.rfc-editor.org/info/rfc2680>.
[RFC3148] M. Mathis et al., A Framework for Defining Empirical [RFC3148] Mathis, M. and M. Allman, "A Framework for Defining
Bulk Transfer Capacity Metrics," RFC3148 July 2001 Empirical Bulk Transfer Capacity Metrics", RFC 3148,
DOI 10.17487/RFC3148, July 2001,
<http://www.rfc-editor.org/info/rfc3148>.
[RFC5481] A. Morton et al., "Packet Delay Variation Applicability [RFC4115] Aboul-Magd, O. and S. Rabie, "A Differentiated Service
Statement," RFC5481 March 2009 Two-Rate, Three-Color Marker with Efficient Handling of
in-Profile Traffic", RFC 4115, DOI 10.17487/RFC4115,
July 2005, <http://www.rfc-editor.org/info/rfc4115>.
[RFC6703] A. Morton et al., "Reporting IP Network Performance [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
Metrics: Different Points of View." RFC 6703 August 2012 "Terminology for Benchmarking Network-layer Traffic
Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689,
October 2006, <http://www.rfc-editor.org/info/rfc4689>.
[RFC2680] G. Almes et al., "A One-way Packet Loss Metric for IPPM," [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
RFC2680 September 1999 S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
DOI 10.17487/RFC4737, November 2006,
<http://www.rfc-editor.org/info/rfc4737>.
[RFC4689] S. Poretsky et al., "Terminology for Benchmarking [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Network-layer Traffic Control Mechanisms," RFC4689, Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
October 2006 March 2009, <http://www.rfc-editor.org/info/rfc5481>.
[RFC4737] A. Morton et al., "Packet Reordering Metrics," RFC4737, [RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage,
February 2006 "Framework for TCP Throughput Testing", RFC 6349,
DOI 10.17487/RFC6349, August 2011,
<http://www.rfc-editor.org/info/rfc6349>.
[RFC4115] O. Aboul-Magd et al., "A Differentiated Service Two-Rate, [RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting
Three-Color Marker with Efficient Handling of in-Profile Traffic." IP Network Performance Metrics: Different Points of View",
RFC4115 July 2005 RFC 6703, DOI 10.17487/RFC6703, August 2012,
<http://www.rfc-editor.org/info/rfc6703>.
[RFC6349] Barry Constantine et al., "Framework for TCP Throughput [SPECweb2009]
Testing," RFC6349, August 2011 Standard Performance Evaluation Corporation (SPEC),
"SPECweb2009 Release 1.20 Benchmark Design Document",
April 2010, <https://www.spec.org/web2009/docs/design/
SPECweb2009_Design.html>.
9.2. Informative References 8.2. Informative References
[RFC2697] J. Heinanen et al., "A Single Rate Three Color Marker," [CA-Benchmark]
RFC2697, September 1999 Hamilton, M. and S. Banks, "Benchmarking Methodology for
Content-Aware Network Devices", Work in Progress,
draft-ietf-bmwg-ca-bench-meth-04, February 2013.
[RFC2698] J. Heinanen et al., "A Two Rate Three Color Marker, " [CoDel] Nichols, K., Jacobson, V., McGregor, A., and J. Iyengar,
RFC2698, September 1999 "Controlled Delay Active Queue Management", Work in
Progress, draft-ietf-aqm-codel-01, April 2015.
[AQM-RECO] Fred Baker et al., "IETF Recommendations Regarding [MEF-10.3] Metro Ethernet Forum, "Ethernet Services Attributes
Active Queue Management," August 2014, Phase 3", MEF 10.3, October 2013,
https://datatracker.ietf.org/doc/draft-ietf-aqm- <https://www.mef.net/Assets/Technical_Specifications/
recommendation/ PDF/MEF_10.3.pdf>.
[MEF-10.2] "MEF 10.2: Ethernet Services Attributes Phase 2," October [MEF-12.2] Metro Ethernet Forum, "Carrier Ethernet Network
2009, http://metroethernetforum.org/PDF_Documents/ Architecture Framework -- Part 2: Ethernet Services
technical-specifications/MEF10.2.pdf Layer", MEF 12.2, May 2014,
<https://www.mef.net/Assets/Technical_Specifications/
PDF/MEF_12.2.pdf>.
[MEF-12.1] "MEF 12.1: Carrier Ethernet Network Architecture [MEF-14] Metro Ethernet Forum, "Abstract Test Suite for Traffic
Framework -- Management Phase 1", MEF 14, November 2005,
Part 2: Ethernet Services Layer - Base Elements," April <https://www.mef.net/Assets/
2010, https://www.metroethernetforum.org/Assets/Technical Technical_Specifications/PDF/MEF_14.pdf>.
_Specifications/PDF/MEF12.1.pdf
[MEF-26] "MEF 26: External Network Network Interface (ENNI) - [MEF-19] Metro Ethernet Forum, "Abstract Test Suite for UNI
Phase 1,"January 2010, http://www.metroethernetforum.org Type 1", MEF 19, April 2007, <https://www.mef.net/Assets/
/PDF_Documents/technical-specifications/MEF26.pdf Technical_Specifications/PDF/MEF_19.pdf>.
[MEF-14] "Abstract Test Suite for Traffic Management Phase 1, [MEF-26.1] Metro Ethernet Forum, "External Network Network Interface
https://www.metroethernetforum.org/Assets (ENNI) - Phase 2", MEF 26.1, January 2012,
/Technical_Specifications/PDF/MEF_14.pdf <http://www.mef.net/Assets/Technical_Specifications/
PDF/MEF_26.1.pdf>.
[MEF-19] "Abstract Test Suite for UNII Type 1", [MEF-37] Metro Ethernet Forum, "Abstract Test Suite for ENNI",
https://www.metroethernetforum.org/Assets MEF 37, January 2012, <https://www.mef.net/Assets/
/Technical_Specifications/PDF/MEF_19.pdf Technical_Specifications/PDF/MEF_37.pdf>.
[MEF-37] "Abstract Test Suite for ENNI", [PIE] Pan, R., Natarajan, P., Baker, F., White, G., VerSteeg,
https://www.metroethernetforum.org/Assets B., Prabhu, M., Piglione, C., and V. Subramanian, "PIE: A
/Technical_Specifications/PDF/MEF_37.pdf Lightweight Control Scheme To Address the Bufferbloat
Problem", Work in Progress, draft-ietf-aqm-pie-02,
August 2015.
Appendix A: Open Source Tools for Traffic Management Testing [RFC2697] Heinanen, J. and R. Guerin, "A Single Rate Three Color
Marker", RFC 2697, DOI 10.17487/RFC2697, September 1999,
<http://www.rfc-editor.org/info/rfc2697>.
[RFC2698] Heinanen, J. and R. Guerin, "A Two Rate Three Color
Marker", RFC 2698, DOI 10.17487/RFC2698, September 1999,
<http://www.rfc-editor.org/info/rfc2698>.
[RFC7567] Baker, F., Ed., and G. Fairhurst, Ed., "IETF
Recommendations Regarding Active Queue Management",
BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015,
<http://www.rfc-editor.org/info/rfc7567>.
Appendix A. Open Source Tools for Traffic Management Testing
This framework specifies that stateless and stateful behaviors SHOULD This framework specifies that stateless and stateful behaviors SHOULD
both be tested. Some open source tools that can be used to both be tested. Some open source tools that can be used to
accomplish many of the tests proposed in this framework are: accomplish many of the tests proposed in this framework are iperf,
iperf, netperf (with netperf-wrapper),uperf, TMIX, netperf (with netperf-wrapper), the "uperf" tool, Tmix,
TCP-incast-generator, and D-ITG (Distributed Internet Traffic TCP-incast-generator, and D-ITG (Distributed Internet Traffic
Generator). Generator).
Iperf can generate UDP or TCP based traffic; a client and server must iperf can generate UDP-based or TCP-based traffic; a client and
both run the iperf software in the same traffic mode. The server is server must both run the iperf software in the same traffic mode.
set up to listen and then the test traffic is controlled from the The server is set up to listen, and then the test traffic is
client. Both uni-directional and bi-directional concurrent testing controlled from the client. Both unidirectional and bidirectional
are supported. concurrent testing are supported.
The UDP mode can be used for the stateless traffic testing. The The UDP mode can be used for the stateless traffic testing. The
target bandwidth, packet size, UDP port, and test duration can be target bandwidth, packet size, UDP port, and test duration can be
controlled. A report of bytes transmitted, packets lost, and delay controlled. A report of bytes transmitted, packets lost, and delay
variation are provided by the iperf receiver. variation is provided by the iperf receiver.
Iperf (TCP mode), TCP-incast-generator, and D-ITG can be used for iperf (TCP mode), TCP-incast-generator, and D-ITG can be used for
stateful traffic testing to test bulk transfer traffic. The TCP stateful traffic testing to test bulk transfer traffic. The TCP
Window size (which is actually the SSB), the number of connections, window size (which is actually the SSB), number of connections,
the packet size, TCP port and the test duration can be controlled. packet size, TCP port, and test duration can be controlled. A report
A report of bytes transmitted and throughput achieved are provided of bytes transmitted and throughput achieved is provided by the iperf
by the iperf sender, while TCP-incast-generator and D-ITG provide sender, while TCP-incast-generator and D-ITG provide even more
even more statistics. statistics.
Netperf is a software application that provides network bandwidth netperf is a software application that provides network bandwidth
testing between two hosts on a network. It supports Unix domain testing between two hosts on a network. It supports UNIX domain
sockets, TCP, SCTP, DLPI and UDP via BSD Sockets. Netperf provides sockets, TCP, SCTP, and UDP via BSD Sockets. netperf provides a
a number of predefined tests e.g. to measure bulk (unidirectional) number of predefined tests, e.g., to measure bulk (unidirectional)
data transfer or request response performance data transfer or request/response performance
http://en.wikipedia.org/wiki/Netperf). Netperf-wrapper is a Python (http://en.wikipedia.org/wiki/Netperf). netperf-wrapper is a Python
script that runs multiple simultaneous netperf instances and script that runs multiple simultaneous netperf instances and
aggregate the results. aggregates the results.
uperf uses a description (or model) of an application mixture and uperf uses a description (or model) of an application mixture. It
the tool generates the load according to the model desciptor. uperf generates the load according to the model descriptor. uperf is more
is more flexible than Netperf in it's ability to generate request flexible than netperf in its ability to generate request/response
/ response application behavior within a single TCP connection. The application behavior within a single TCP connection. The application
application model descriptor can be based off of empirical data, but model descriptor can be based on empirical data, but at the time of
currently the import of packet captures is not directly supported. this writing, the import of packet captures is not directly
supported.
Tmix is another application traffic emulation tool and uses packet Tmix is another application traffic emulation tool. It uses packet
captures directly to create the traffic profile. The packet trace is captures directly to create the traffic profile. The packet trace is
'reverse compiled' into a source-level characterization, called a "reverse compiled" into a source-level characterization, called a
connection vector, of each TCP connection present in the trace. While "connection vector", of each TCP connection present in the trace.
most widely used in ns2 simulation environment, TMix also runs on While most widely used in ns2 simulation environments, Tmix also runs
Linux hosts. on Linux hosts.
These open source tool's traffic generation capabilities facilitate The traffic generation capabilities of these open source tools
the emulation of the TCP test patterns which are discussed in facilitate the emulation of the TCP test patterns discussed in
Appendix B. Appendix B.
Appendix B: Stateful TCP Test Patterns Appendix B. Stateful TCP Test Patterns
This framework recommends at a minimum the following TCP test This framework recommends at a minimum the following TCP test
patterns since they are representative of real world application patterns, since they are representative of real-world application
traffic (section 5.2.1 describes some methods to derive other traffic (Section 5.2.1 describes some methods to derive other
application-based TCP test patterns). application-based TCP test patterns).
- Bulk Transfer: generate concurrent TCP connections whose aggregate - Bulk Transfer: Generate concurrent TCP connections whose aggregate
number of in-flight data bytes would fill the BDP. Guidelines number of in-flight data bytes would fill the BDP. Guidelines
from [RFC6349] are used to create this TCP traffic pattern. from [RFC6349] are used to create this TCP traffic pattern.
- Micro Burst: generate precise burst patterns within a single or - Micro Burst: Generate precise burst patterns within a single TCP
multiple TCP connections(s). The idea is for TCP to establish connection or multiple TCP connections. The idea is for TCP to
equilibrium and then burst application bytes at defined sizes. The establish equilibrium and then burst application bytes at defined
test tool must allow the burst size and burst time interval to be sizes. The test tool must allow the burst size and burst time
configurable. interval to be configurable.
- Web Site Patterns: The HTTP traffic model from - Web Site Patterns: The HTTP traffic model shown in Table 4.1.3-1
"3GPP2 C.R1002-0 v1.0" is referenced (Table 4.1.3.2.1) to develop of [3GPP2-C_R1002-A] demonstrates a way to develop these TCP test
these TCP test patterns. In summary, the HTTP traffic model consists patterns. In summary, the HTTP traffic model consists of the
of the following parameters: following parameters:
- Main object size (Sm)
- Embedded object size (Se)
- Number of embedded objects per page (Nd)
- Client processing time (Tcp)
- Server processing time (Tsp)
Web site test patterns are illustrated with the following examples: - Main object size (Sm)
- Simple Web Site: mimic the request / response and object - Embedded object size (Se)
download behavior of a basic web site (small company).
- Complex Web Site: mimic the request / response and object
download behavior of a complex web site (ecommerce site).
Referencing the HTTP traffic model parameters , the following table - Number of embedded objects per page (Nd)
was derived (by analysis and experimentation) for Simple and Complex
Web site TCP test patterns:
Simple Complex - Client processing time (Tcp)
Parameter Web Site Web Site
-----------------------------------------------------
Main object Ave. = 10KB Ave. = 300KB
size (Sm) Min. = 100B Min. = 50KB
Max. = 500KB Max. = 2MB
Embedded object Ave. = 7KB Ave. = 10KB - Server processing time (Tsp)
size (Se) Min. = 50B Min. = 100B
Max. = 350KB Max. = 1MB
Number of embedded Ave. = 5 Ave. = 25 Web site test patterns are illustrated with the following examples:
objects per page (Nd) Min. = 2 Min. = 10
Max. = 10 Max. = 50
Client processing Ave. = 3s Ave. = 10s - Simple web site: Mimic the request/response and object download
time (Tcp)* Min. = 1s Min. = 3s behavior of a basic web site (small company).
Max. = 10s Max. = 30s
Server processing Ave. = 5s Ave. = 8s - Complex web site: Mimic the request/response and object download
time (Tsp)* Min. = 1s Min. = 2s behavior of a complex web site (eCommerce site).
Max. = 15s Max. = 30s
* The client and server processing time is distributed across the Referencing the HTTP traffic model parameters, the following table
transmission / receipt of all of the main and embedded objects was derived (by analysis and experimentation) for simple web site and
complex web site TCP test patterns:
Simple Complex
Parameter Web Site Web Site
-----------------------------------------------------
Main object Ave. = 10KB Ave. = 300KB
size (Sm) Min. = 100B Min. = 50KB
Max. = 500KB Max. = 2MB
Embedded object Ave. = 7KB Ave. = 10KB
size (Se) Min. = 50B Min. = 100B
Max. = 350KB Max. = 1MB
Number of embedded Ave. = 5 Ave. = 25
objects per page (Nd) Min. = 2 Min. = 10
Max. = 10 Max. = 50
Client processing Ave. = 3s Ave. = 10s
time (Tcp)* Min. = 1s Min. = 3s
Max. = 10s Max. = 30s
Server processing Ave. = 5s Ave. = 8s
time (Tsp)* Min. = 1s Min. = 2s
Max. = 15s Max. = 30s
* The client and server processing time is distributed across the
transmission/receipt of all of the main and embedded objects.
To be clear, the parameters in this table are reasonable guidelines To be clear, the parameters in this table are reasonable guidelines
for the TCP test pattern traffic generation. The test tool can use for the TCP test pattern traffic generation. The test tool can use
fixed parameters for simpler tests and mathematical distributions for fixed parameters for simpler tests and mathematical distributions for
more complex tests. However, the test pattern must be repeatable to more complex tests. However, the test pattern must be repeatable to
ensure that the benchmark results can be reliably compared. ensure that the benchmark results can be reliably compared.
- Inter-active Patterns: While Web site patterns are inter-active - Interactive Patterns: While web site patterns are interactive to a
to a degree, they mainly emulate the downloading of various degree, they mainly emulate the downloading of web sites of
complexity web sites. Inter-active patterns are more chatty in varying complexity. Interactive patterns are more chatty in
nature since there is alot of user interaction with the servers. nature, since there is a lot of user interaction with the servers.
Examples include business applications such as Peoplesoft, Oracle Examples include business applications such as PeopleSoft and
and consumer applications such as Facebook, IM, etc. For the inter- Oracle, and consumer applications such as Facebook and IM. For
active patterns, the packet capture technique was used to the interactive patterns, the packet capture technique was used to
characterize some business applications and also the email characterize some business applications and also the email
application. application.
In summary, an inter-active application can be described by the In summary, an interactive application can be described by the
following parameters: following parameters:
- Client message size (Scm)
- Number of Client messages (Nc)
- Server response size (Srs)
- Number of server messages (Ns)
- Client processing time (Tcp)
- Server processing Time (Tsp)
- File size upload (Su)*
- File size download (Sd)*
* The file size parameters account for attachments uploaded or - Client message size (Scm)
downloaded and may not be present in all inter-active applications
- Number of client messages (Nc)
- Server response size (Srs)
- Number of server messages (Ns)
- Client processing time (Tcp)
- Server processing time (Tsp)
- File size upload (Su)*
- File size download (Sd)*
* The file size parameters account for attachments uploaded or
downloaded and may not be present in all interactive applications.
Again using packet capture as a means to characterize, the following Again using packet capture as a means to characterize, the following
table reflects the guidelines for Simple Business Application, table reflects the guidelines for simple business applications,
Complex Business Application, eCommerce, and Email Send / Receive: complex business applications, eCommerce, and email Send/Receive:
Simple Complex Simple Complex
Parameter Biz. App. Biz. App eCommerce* Email Business Business
Parameter Application Application eCommerce* Email
-------------------------------------------------------------------- --------------------------------------------------------------------
Client message Ave. = 450B Ave. = 2KB Ave. = 1KB Ave. = 200B Client message Ave. = 450B Ave. = 2KB Ave. = 1KB Ave. = 200B
size (Scm) Min. = 100B Min. = 500B Min. = 100B Min. = 100B size (Scm) Min. = 100B Min. = 500B Min. = 100B Min. = 100B
Max. = 1.5KB Max. = 100KB Max. = 50KB Max. = 1KB Max. = 1.5KB Max. = 100KB Max. = 50KB Max. = 1KB
Number of client Ave. = 10 Ave. = 100 Ave. = 20 Ave. = 10 Number of client Ave. = 10 Ave. = 100 Ave. = 20 Ave. = 10
messages (Nc) Min. = 5 Min. = 50 Min. = 10 Min. = 5 messages (Nc) Min. = 5 Min. = 50 Min. = 10 Min. = 5
Max. = 25 Max. = 250 Max. = 100 Max. = 25 Max. = 25 Max. = 250 Max. = 100 Max. = 25
Client processing Ave. = 10s Ave. = 30s Ave. = 15s Ave. = 5s Client processing Ave. = 10s Ave. = 30s Ave. = 15s Ave. = 5s
time (Tcp)** Min. = 3s Min. = 3s Min. = 5s Min. = 3s time (Tcp)** Min. = 3s Min. = 3s Min. = 5s Min. = 3s
Max. = 30s Max. = 60s Max. = 120s Max. = 45s Max. = 30s Max. = 60s Max. = 120s Max. = 45s
Server response Ave. = 2KB Ave. = 5KB Ave. = 8KB Ave. = 200B Server response Ave. = 2KB Ave. = 5KB Ave. = 8KB Ave. = 200B
skipping to change at page 40, line 5 skipping to change at page 48, line 37
Max. = 100KB Max. = 1MB Max. = 50KB Max. = 750B Max. = 100KB Max. = 1MB Max. = 50KB Max. = 750B
Number of server Ave. = 50 Ave. = 200 Ave. = 100 Ave. = 15 Number of server Ave. = 50 Ave. = 200 Ave. = 100 Ave. = 15
messages (Ns) Min. = 10 Min. = 25 Min. = 15 Min. = 5 messages (Ns) Min. = 10 Min. = 25 Min. = 15 Min. = 5
Max. = 200 Max. = 1000 Max. = 500 Max. = 40 Max. = 200 Max. = 1000 Max. = 500 Max. = 40
Server processing Ave. = 0.5s Ave. = 1s Ave. = 2s Ave. = 4s Server processing Ave. = 0.5s Ave. = 1s Ave. = 2s Ave. = 4s
time (Tsp)** Min. = 0.1s Min. = 0.5s Min. = 1s Min. = 0.5s time (Tsp)** Min. = 0.1s Min. = 0.5s Min. = 1s Min. = 0.5s
Max. = 5s Max. = 20s Max. = 10s Max. = 15s Max. = 5s Max. = 20s Max. = 10s Max. = 15s
Complex Business Application, eCommerce, and Email Send / Receive File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB
(continued):
Simple Complex
Parameter Biz. App. Biz. App eCommerce* Email
--------------------------------------------------------------------
File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB
upload (Su) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB upload (Su) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB
Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB
File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB File size Ave. = 50KB Ave. = 100KB Ave. = N/A Ave. = 100KB
download (Sd) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB download (Sd) Min. = 2KB Min. = 10KB Min. = N/A Min. = 20KB
Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB Max. = 200KB Max. = 2MB Max. = N/A Max. = 10MB
* eCommerce used a combination of packet capture techniques and * eCommerce used a combination of packet capture techniques and
reference traffic flows from "SPECweb2009" (need proper reference) reference traffic flows as described in [SPECweb2009].
** The client and server processing time is distributed across the
transmission / receipt of all of messages. Client processing time
consists mainly of the delay between user interactions (not machine
processing).
And again, the parameters in this table are the guidelines for the ** The client and server processing time is distributed across the
TCP test pattern traffic generation. The test tool can use fixed transmission/receipt of all of the messages. The client
processing time consists mainly of the delay between user
interactions (not machine processing).
Again, the parameters in this table are the guidelines for the TCP
test pattern traffic generation. The test tool can use fixed
parameters for simpler tests and mathematical distributions for more parameters for simpler tests and mathematical distributions for more
complex tests. However, the test pattern must be repeatable to complex tests. However, the test pattern must be repeatable to
ensure that the benchmark results can be reliably compared. ensure that the benchmark results can be reliably compared.
- SMB/CIFS File Copy: mimic a network file copy, both read and write. - SMB/CIFS file copy: Mimic a network file copy, both read and
As opposed to FTP which is a bulk transfer and is only flow write. As opposed to FTP, which is a bulk transfer and is only
controlled via TCP, SMB/CIFS divides a file into application blocks flow-controlled via TCP, SMB/CIFS divides a file into application
and utilizes application level handshaking in addition to blocks and utilizes application-level handshaking in addition to
TCP flow control. TCP flow control.
In summary, an SMB/CIFS file copy can be described by the following In summary, an SMB/CIFS file copy can be described by the following
parameters: parameters:
- Client message size (Scm)
- Number of client messages (Nc)
- Server response size (Srs)
- Number of Server messages (Ns)
- Client processing time (Tcp)
- Server processing time (Tsp)
- Block size (Sb)
The client and server messages are SMB control messages. The Block - Client message size (Scm)
size is the data portion of th file transfer.
Again using packet capture as a means to characterize the following - Number of client messages (Nc)
- Server response size (Srs)
- Number of server messages (Ns)
- Client processing time (Tcp)
- Server processing time (Tsp)
- Block size (Sb)
The client and server messages are SMB control messages. The block
size is the data portion of the file transfer.
Again using packet capture as a means to characterize, the following
table reflects the guidelines for SMB/CIFS file copy: table reflects the guidelines for SMB/CIFS file copy:
SMB SMB/CIFS
Parameter File Copy Parameter File Copy
------------------------------ --------------------------------
Client message Ave. = 450B Client message Ave. = 450B
size (Scm) Min. = 100B size (Scm) Min. = 100B
Max. = 1.5KB Max. = 1.5KB
Number of client Ave. = 10
messages (Nc) Min. = 5
Max. = 25
Client processing Ave. = 1ms
time (Tcp) Min. = 0.5ms
Max. = 2
Server response Ave. = 2KB
size (Srs) Min. = 500B
Max. = 100KB
Number of server Ave. = 10
messages (Ns) Min. = 10
Max. = 200
Server processing Ave. = 1ms
time (Tsp) Min. = 0.5ms
Max. = 2ms
Block Ave. = N/A
Size (Sb)* Min. = 16KB
Max. = 128KB
*Depending upon the tested file size, the block size will be Number of client Ave. = 10
transferred n number of times to complete the example. An example messages (Nc) Min. = 5
would be a 10 MB file test and 64KB block size. In this case 160 Max. = 25
blocks would be transferred after the control channel is opened
between the client and server. Client processing Ave. = 1ms
time (Tcp) Min. = 0.5ms
Max. = 2
Server response Ave. = 2KB
size (Srs) Min. = 500B
Max. = 100KB
Number of server Ave. = 10
messages (Ns) Min. = 10
Max. = 200
Server processing Ave. = 1ms
time (Tsp) Min. = 0.5ms
Max. = 2ms
Block Ave. = N/A
size (Sb)* Min. = 16KB
Max. = 128KB
* Depending upon the tested file size, the block size will be
transferred "n" number of times to complete the example. An
example would be a 10 MB file test and 64 KB block size. In
this case, 160 blocks would be transferred after the control
channel is opened between the client and server.
Acknowledgments Acknowledgments
We would like to thank Al Morton for his continuous review and We would like to thank Al Morton for his continuous review and
invaluable input to the document. We would also like to thank invaluable input to this document. We would also like to thank Scott
Scott Bradner for providing guidance early in the drafts Bradner for providing guidance early in this document's conception,
conception in the area of benchmarking scope of traffic management in the area of the benchmarking scope of traffic management
functions. Additionally, we would like to thank Tim Copley for this functions. Additionally, we would like to thank Tim Copley for his
original input and David Taht, Gory Erg, Toke Hoiland-Jorgensen for original input, as well as David Taht, Gory Erg, and Toke
their review and input for the AQM group. And for the formal reviews Hoiland-Jorgensen for their review and input for the AQM group.
of this document, we would like to thank Gilles Forget, Also, for the formal reviews of this document, we would like to thank
Vijay Gurbani, Reinhard Schrage, and Bhuvaneswaran Vengainathan Gilles Forget, Vijay Gurbani, Reinhard Schrage, and Bhuvaneswaran
Vengainathan.
Authors' Addresses Authors' Addresses
Barry Constantine Barry Constantine
JDSU, Test and Measurement Division JDSU, Test and Measurement Division
Germantown, MD 20876-7100, USA Germantown, MD 20876-7100
Phone: +1 240 404 2227 United States
Phone: +1-240-404-2227
Email: barry.constantine@jdsu.com Email: barry.constantine@jdsu.com
Ram(Ramki) Krishnan Ram (Ramki) Krishnan
Dell Inc. Dell Inc.
Santa Clara, CA 95054, USA Santa Clara, CA 95054
Phone: +001-408-406-7890 United States
Phone: +1-408-406-7890
Email: ramkri123@gmail.com Email: ramkri123@gmail.com
 End of changes. 417 change blocks. 
1406 lines changed or deleted 1597 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/