draft-ietf-bmwg-traffic-management-00.txt   draft-ietf-bmwg-traffic-management-01.txt 
Network Working Group B. Constantine Network Working Group B. Constantine
Internet Draft JDSU Internet Draft JDSU
Intended status: Informational T. Copley Intended status: Informational T. Copley
Expires: January 2015 Level-3 Expires: May 2015 Level-3
August 10, 2014 R. Krishnan November 12, 2014 R. Krishnan
Brocade Communications Brocade Communications
Traffic Management Benchmarking Traffic Management Benchmarking
draft-ietf-bmwg-traffic-management-00.txt draft-ietf-bmwg-traffic-management-01.txt
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 10, 2015. This Internet-Draft will expire on May 12, 2015.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 4, line 9 skipping to change at page 4, line 9
9. Conclusions...................................................24 9. Conclusions...................................................24
10. References...................................................24 10. References...................................................24
10.1. Normative References....................................25 10.1. Normative References....................................25
10.2. Informative References..................................25 10.2. Informative References..................................25
11. Acknowledgments..............................................25 11. Acknowledgments..............................................25
1. Introduction 1. Introduction
Traffic management (i.e. policing, shaping, etc.) is an increasingly Traffic management (i.e. policing, shaping, etc.) is an increasingly
important component when implementing network Quality of Service important component when implementing network Quality of Service
(QoS). There is currently no framework to benchmark these features (QoS).
although some standards address specific areas. This draft provides
a framework to conduct repeatable traffic management benchmarks for
devices and systems in a lab environment.
Specifically, this framework defines the methods to characterize the There is currently no framework to benchmark these features
capacity of the following traffic management features in network although some standards address specific areas which are described
in Section 1.1.
This draft provides a framework to conduct repeatable traffic
management benchmarks for devices and systems in a lab environment.
Specifically, this framework defines the methods to characterize
the capacity of the following traffic management features in network
devices; classification, policing, queuing / scheduling, and devices; classification, policing, queuing / scheduling, and
traffic shaping. traffic shaping.
This benchmarking framework can also be used as a test procedure to This benchmarking framework can also be used as a test procedure to
assist in the tuning of traffic management parameters before service assist in the tuning of traffic management parameters before service
activation. In addition to Layer 2/3 benchmarking, Layer 4 test activation. In addition to Layer 2/3 (Ethernet / IP) benchmarking,
patterns are proposed by this draft in order to more realistically Layer 4 (TCP) test patterns are proposed by this draft in order to
benchmark end-user traffic. more realistically benchmark end-user traffic.
1.1. Traffic Management Overview 1.1. Traffic Management Overview
In general, a device with traffic management capabilities performs In general, a device with traffic management capabilities performs
the following functions: the following functions:
- Traffic classification: identifies traffic according to various - Traffic classification: identifies traffic according to various
configuration rules (i.e. VLAN, DSCP, etc.) and marks this traffic configuration rules for example IEEE 802.1Q Virtual LAN (VLAN),
Differential Services Code Point (DSCP) etc. and marks this traffic
internally to the network device. Multiple external priorities internally to the network device. Multiple external priorities
(DSCP, 802.1p, etc.) can map to the same priority in the device. (DSCP, 802.1p, etc.) can map to the same priority in the device.
- Traffic policing: limits the rate of traffic that enters a network - Traffic policing: limits the rate of traffic that enters a network
device according to the traffic classification. If the traffic device according to the traffic classification. If the traffic
exceeds the contracted limits, the traffic is either dropped or exceeds the provisioned limits, the traffic is either dropped or
remarked and sent onto to the next network device remarked and forwarded onto to the next network device
- Traffic Scheduling: provides traffic classification within the - Traffic Scheduling: provides traffic classification within the
network device by directing packets to various types of queues and network device by directing packets to various types of queues and
applies a dispatching algorithm to assign the forwarding sequence applies a dispatching algorithm to assign the forwarding sequence
of packets of packets
- Traffic shaping: a traffic control technique that actively buffers - Traffic shaping: a traffic control technique that actively buffers
and meters the output rate in an attempt to adapt bursty traffic and smooths the output rate in an attempt to adapt bursty traffic
to the configured limits to the configured limits
- Active Queue Management (AQM): monitors the status of internal - Active Queue Management (AQM):
queues and actively drops (or re-marks) packets, which causes hosts AQM involves monitoring the status of internal queues and proactively
using congestion-aware protocols to back-off and in turn can dropping (or remarking) packets, which causes hosts using
alleviate queue congestion. Note that AQM is outside of the scope congestion-aware protocols to back-off and in turn alleviate queue
of this testing framework. congestion [AQM-RECO]. On the other hand, classic traffic management
techniques reactively drop (or remark) packets based on queue full
condition. The benchmarking scenarios for AQM are different and is
outside of the scope of this testing framework.
The following diagram is a generic model of the traffic management The following diagram is a generic model of the traffic management
capabilities within a network device. It is not intended to capabilities within a network device. It is not intended to
represent all variations of manufacturer traffic management represent all variations of manufacturer traffic management
capabilities, but provide context to this test framework. capabilities, but provide context to this test framework.
|----------| |----------------| |--------------| |----------| |----------| |----------------| |--------------| |----------|
| | | | | | | | | | | | | | | |
|Interface | |Ingress Actions | |Egress Actions| |Interface | |Interface | |Ingress Actions | |Egress Actions| |Interface |
|Input | |(classification,| |(scheduling, | |Output | |Input | |(classification,| |(scheduling, | |Output |
|Queues | | marking, | | shaping, | |Queues | |Queues | | marking, | | shaping, | |Queues |
| |-->| policing or |-->| active queue |-->| | | |-->| policing or |-->| active queue |-->| |
| | | shaping) | | management | | | | | | shaping) | | management | | |
| | | | | re-marking) | | | | | | | | remarking) | | |
|----------| |----------------| |--------------| |----------| |----------| |----------------| |--------------| |----------|
Figure 1: Generic Traffic Management capabilities of a Network Device Figure 1: Generic Traffic Management capabilities of a Network Device
Ingress actions such as classification are defined in RFC 4689 and Ingress actions such as classification are defined in RFC 4689 [RFC4689]
include IP addresses, port numbers, DSCP, etc. In terms of marking, and include IP addresses, port numbers, DSCP, etc. In terms of marking,
RFC 2697 and RFC 2698 define a single rate and dual rate, three color RFC 2697 [RFC2697] and RFC 2698 [RFC2698] define a single rate and dual
marker, respectively. rate, three color marker, respectively.
The MEF specifies policing and shaping in terms of Ingress and Egress The Metro Ethernet Forum (MEF) specifies policing and shaping in terms
Subscriber/Provider Conditioning Functions in MEF12.1; Ingress and of Ingress and Egress Subscriber/Provider Conditioning Functions in
Bandwidth Profile attributes in MEF 10.2 and MEF 26. MEF12.1 [MEF-12.1]; Ingress and Bandwidth Profile attributes in MEF10.2
[MEF-10.2] and MEF 26 [MEF-26].
1.2 DUT Lab Configuration and Testing Overview 1.2 Lab Configuration and Testing Overview
The following is the description of the lab set-up for the traffic The following is the description of the lab set-up for the traffic
management tests: management tests:
+--------------+ +-------+ +----------+ +-----------+ +--------------+ +-------+ +----------+ +-----------+
| Transmitting | | | | | | Receiving | | Transmitting | | | | | | Receiving |
| Test Host | | | | | | Test Host | | Test Host | | | | | | Test Host |
| |-----| DUT |---->| Network |--->| | | |-----| Device|---->| Network |--->| |
| | | | | Delay | | | | | | Under | | Delay | | |
| | | | | Emulator | | | | | | Test | | Emulator | | |
| |<----| |<----| |<---| | | |<----| |<----| |<---| |
| | | | | | | | | | | | | | | |
+--------------+ +-------+ +----------+ +-----------+ +--------------+ +-------+ +----------+ +-----------+
As shown in the test diagram, the framework supports uni-directional As shown in the test diagram, the framework supports uni-directional
and bi-directional traffic management tests. and bi-directional traffic management tests (where the transmitting
and receiving roles would be reversed on the return path).
This testing framework describes the tests and metrics for each of This testing framework describes the tests and metrics for each of
the following traffic management functions: the following traffic management functions:
- Policing - Policing
- Queuing / Scheduling - Queuing / Scheduling
- Shaping - Shaping
The tests are divided into individual tests and rated capacity tests. The tests are divided into individual and rated capacity tests.
The individual tests are intended to benchmark the traffic management The individual tests are intended to benchmark the traffic management
functions according to the metrics defined in Section 4. The functions according to the metrics defined in Section 4. The
capacity tests verify traffic management functions under full load. capacity tests verify traffic management functions under the load of
many simultaneous individual tests and their flows.
This involves concurrent testing of multiple interfaces with the This involves concurrent testing of multiple interfaces with the
specific traffic management function enabled, and doing so to the specific traffic management function enabled, and increasing load to
capacity limit of each interface. the capacity limit of each interface.
As an example: a device is specified to be capable of shaping on all As an example: a device is specified to be capable of shaping on all
of it's egress ports. The individual test would first be conducted to of its egress ports. The individual test would first be conducted to
benchmark the advertised shaping function against the metrics defined benchmark the specified shaping function against the metrics defined
in section 4. Then the capacity test would be executed to test the in section 4. Then the capacity test would be executed to test the
shaping function concurrently on all interfaces and with maximum shaping function concurrently on all interfaces and with maximum
traffic load. traffic load.
The Network Delay Emulator (NDE) is a requirement for the TCP The Network Delay Emulator (NDE) is required for TCP stateful tests
stateful tests, which require network delay to allow TCP to fully in order to allow TCP to utilize a significant size TCP window in its
open the TCP window. Also note that the Network Delay Emulator (NDE) control loop.
should be passive in nature such as a fiber spool. This is
recommended to eliminate the potential effects that an active delay Also note that the Network Delay Emulator (NDE) should be passive in
element (i.e. test impairment generator) may have on the test flows. nature such as a fiber spool. This is recommended to eliminate the
In the case that a fiber spool is not practical due to the desired potential effects that an active delay element (i.e. test impairment
latency, an active NDE must be independently verified to be capable generator) may have on the test flows. In the case where a fiber
of adding the configured delay without loss. In other words, the spool is not practical due to the desired latency, an active NDE must
DUT would be removed and the NDE performance benchmarked be independently verified to be capable of adding the configured delay
independently. without loss. In other words, the DUT would be removed and the NDE
performance benchmarked independently.
Note the NDE should be used in "full pipe" delay mode. Most NDEs Note the NDE should be used in "full pipe" delay mode. Most NDEs
allow for per flow delay actions, emulating QoS prioritization. For allow for per flow delay actions, emulating QoS prioritization. For
this framework, the NDE's sole purpose is simply to add delay to all this framework, the NDE's sole purpose is simply to add delay to all
packets (emulate network latency). So to benchmark the performance of packets (emulate network latency). So to benchmark the performance of
the NDE, maximum offered load should be tested against the following the NDE, maximum offered load should be tested against the following
frame sizes: 128, 256, 512, 768, 1024, 1500,and 9600 bytes. The delay frame sizes: 128, 256, 512, 768, 1024, 1500,and 9600 bytes. The delay
accuracy at each of these packet sizes can then be used to calibrate accuracy at each of these packet sizes can then be used to calibrate
the range of expected BDPs for the TCP stateful tests. the range of expected Bandwidth Delay Product (BDP) for the TCP stateful
tests.
2. Conventions used in this document 2. Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
The following acronyms are used: The following acronyms are used:
AQM: Active Queue Management
BB: Bottleneck Bandwidth BB: Bottleneck Bandwidth
BDP: Bandwidth Delay Product BDP: Bandwidth Delay Product
BSA: Burst Size Achieved BSA: Burst Size Achieved
CBS: Committed Burst Size CBS: Committed Burst Size
CIR: Committed Information Rate CIR: Committed Information Rate
skipping to change at page 7, line 37 skipping to change at page 7, line 39
EIR: Excess Information Rate EIR: Excess Information Rate
NDE: Network Delay Emulator NDE: Network Delay Emulator
SP: Strict Priority Queuing SP: Strict Priority Queuing
QL: Queue Length QL: Queue Length
QoS: Quality of Service QoS: Quality of Service
RED: Random Early Discard RTH: Receiving Test Host
RTT: Round Trip Time RTT: Round Trip Time
SBB: Shaper Burst Bytes SBB: Shaper Burst Bytes
SBI: Shaper Burst Interval SBI: Shaper Burst Interval
SR: Shaper Rate SR: Shaper Rate
SSB: Send Socket Buffer SSB: Send Socket Buffer
skipping to change at page 8, line 4 skipping to change at page 7, line 54
SBI: Shaper Burst Interval SBI: Shaper Burst Interval
SR: Shaper Rate SR: Shaper Rate
SSB: Send Socket Buffer SSB: Send Socket Buffer
Tc: CBS Time Interval Tc: CBS Time Interval
Te: EBS Time Interval Te: EBS Time Interval
Ti Transmission Interval Ti Transmission Interval
TTH: Transmitting Test Host
TTP: TCP Test Pattern TTP: TCP Test Pattern
TTPET: TCP Test Pattern Execution Time TTPET: TCP Test Pattern Execution Time
WRED: Weighted Random Early Discard
3. Scope and Goals 3. Scope and Goals
The scope of this work is to develop a framework for benchmarking and The scope of this work is to develop a framework for benchmarking and
testing the traffic management capabilities of network devices in the testing the traffic management capabilities of network devices in the
lab environment. These network devices may include but are not lab environment. These network devices may include but are not
limited to: limited to:
- Switches (including Layer 2/3 devices) - Switches (including Layer 2/3 devices)
- Routers - Routers
- Firewalls - Firewalls
- General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.) - General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.)
Essentially, any network device that performs traffic management as Essentially, any network device that performs traffic management as
defined in section 1.1 can be benchmarked or tested with this defined in section 1.1 can be benchmarked or tested with this
framework. framework.
The primary goal is to assess the maximum forwarding performance that The primary goal is to assess the maximum forwarding performance deemed
a network device can sustain without dropping or impairing packets, to be within the provisioned traffic limits that a network device can
or compromising the accuracy of multiple instances of traffic sustain without dropping or impairing packets, or compromising the
management functions. This is the benchmark for comparison between accuracy of multiple instances of traffic management functions. This
devices. is the benchmark for comparison between devices.
Within this framework, the metrics are defined for each traffic Within this framework, the metrics are defined for each traffic
management test but do not include pass / fail criterion, which is management test but do not include pass / fail criterion, which is
not within the charter of BMWG. This framework provides the test not within the charter of BMWG. This framework provides the test
methods and metrics to conduct repeatable testing, which will methods and metrics to conduct repeatable testing, which will
provide the means to compare measured performance between DUTs. provide the means to compare measured performance between DUTs.
As mentioned in section 1.2, this framework describes the individual As mentioned in section 1.2, these methods describe the individual
tests and metrics for several management functions. It is also within tests and metrics for several management functions. It is also within
scope that this framework will benchmark each function in terms of scope that this framework will benchmark each function in terms of
overall rated capacity. This involves concurrent testing of multiple overall rated capacity. This involves concurrent testing of multiple
interfaces with the specific traffic management function enabled, up interfaces with the specific traffic management function enabled, up
to the capacity limit of each interface. to the capacity limit of each interface.
It is not within scope of this framework to specify the procedure for It is not within scope of this of this framework to specify the
testing multiple traffic management functions concurrently. The procedure for testing multiple configurations of traffic management
multitudes of possible combinations is almost unbounded and the functions concurrently. The multitudes of possible combinations is
ability to identify functional "break points" would be most times almost unbounded and the ability to identify functional "break points"
impossible. would be almost impossible.
However, section 6.4 provides suggestions for some profiles of However, section 6.4 provides suggestions for some profiles of
concurrent functions that would be useful to benchmark. The key concurrent functions that would be useful to benchmark. The key
requirement for any concurrent test function is that tests must requirement for any concurrent test function is that tests must
produce reliable and repeatable results. produce reliable and repeatable results.
Also, it is not within scope to perform conformance testing. Tests Also, it is not within scope to perform conformance testing. Tests
defined in this framework benchmark the traffic management functions defined in this framework benchmark the traffic management functions
according to the metrics defined in section 4 and do not address any according to the metrics defined in section 4 and do not address any
conformance to standards related to traffic management. Traffic conformance to standards related to traffic management. The current
management specifications largely do not exist and this is a prime specifications don't specify exact behavior or implementation and the
driver for this framework; to provide an objective means to compare specifications that do exist (cited in section 1.1) allow
vendor traffic management functions. implementations to vary w.r.t. short term rate accuracy and other
factors. This is a primary driver for this framework with the key
goal to provide an objective means to compare vendor traffic
management functions.
Another goal is to devise methods that utilize flows with Another goal is to devise methods that utilize flows with
congestion-aware transport (TCP) as part of the traffic load and congestion-aware transport (TCP) as part of the traffic load and
still produce repeatable results in the isolated test environment. still produce repeatable results in the isolated test environment.
This framework will derive stateful test patterns (TCP or This framework will derive stateful test patterns (TCP or
application layer) that can also be used to further benchmark the application layer) that can also be used to further benchmark the
performance of applicable traffic management techniques such as performance of applicable traffic management techniques such as
queuing / scheduling and traffic shaping. In cases where the queuing / scheduling and traffic shaping. In cases where the
network device is stateful in nature (i.e. firewall, etc.), network device is stateful in nature (i.e. firewall, etc.),
stateful test pattern traffic is important to test along with stateful test pattern traffic is important to test along with
stateless, UDP traffic in specific test scenarios (i.e. stateless, UDP traffic in specific test scenarios (i.e.
applications using TCP transport and UDP VoIP, etc.) applications using TCP transport and UDP VoIP, etc.)
And finally, this framework will provide references to open source As mentioned earlier in the document, repeatability of test results
tools that can be used to provide stateless and/or stateful is critical, especially considering the nature of stateful TCP traffic.
traffic generation emulation. To this end, the stateful tests will use TCP test patterns to emulate
applications. This framework also provides guidelines for application
modeling and open source tools to achieve the repeatable stimulus.
And finally, TCP metrics from RFC 6349 are specified to report for
each stateful test and provide the means to compare each repeated
test.
4. Traffic Benchmarking Metrics 4. Traffic Benchmarking Metrics
The metrics to be measured during the benchmarks are divided into two The metrics to be measured during the benchmarks are divided into two
(2) sections: packet layer metrics used for the stateless traffic (2) sections: packet layer metrics used for the stateless traffic
testing and segment layer metrics used for the stateful traffic testing and TCP layer metrics used for the stateful traffic
testing. testing.
4.1. Metrics for Stateless Traffic Tests 4.1. Metrics for Stateless Traffic Tests
For the stateless traffic tests, the metrics are defined at the layer
3 packet level versus layer 2 packet level for consistency.
Stateless traffic measurements require that sequence number and Stateless traffic measurements require that sequence number and
time-stamp be inserted into the payload for lost packet analysis. time-stamp be inserted into the payload for lost packet analysis.
Delay analysis may be achieved by insertion of timestamps directly Delay analysis may be achieved by insertion of timestamps directly
into the packets or timestamps stored elsewhere (packet captures). into the packets or timestamps stored elsewhere (packet captures).
This framework does not specify the packet format to carry sequence This framework does not specify the packet format to carry sequence
number or timing information. However, RFC 4689 provides number or timing information.
recommendations for sequence tracking along with definitions of
in-sequence and out-of-order packets. However, RFC 4737 [RFC4737] and RFC 4689 provide recommendations
for sequence tracking along with definitions of in-sequence and
out-of-order packets.
The following are the metrics to be used during the stateless traffic The following are the metrics to be used during the stateless traffic
benchmarking components of the tests: benchmarking components of the tests:
- Burst Size Achieved (BSA): for the traffic policing and network - Burst Size Achieved (BSA): for the traffic policing and network
queue tests, the tester will be configured to send bursts to test queue tests, the tester will be configured to send bursts to test
either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of
a policer or the queue / buffer size configured in the DUT. The a policer or the queue / buffer size configured in the DUT. The
Burst Size Achieved metric is a measure of the actual burst size Burst Size Achieved metric is a measure of the actual burst size
received at the egress port of the DUT with no lost packets. As an received at the egress port of the DUT with no lost packets. As an
skipping to change at page 10, line 35 skipping to change at page 10, line 26
BSA. Also, the average Packet Delay Variation (PDV see below) as BSA. Also, the average Packet Delay Variation (PDV see below) as
experienced by the packets sent at the BSA burst size should be experienced by the packets sent at the BSA burst size should be
recorded. recorded.
- Lost Packets (LP): For all traffic management tests, the tester will - Lost Packets (LP): For all traffic management tests, the tester will
transmit the test packets into the DUT ingress port and the number of transmit the test packets into the DUT ingress port and the number of
packets received at the egress port will be measured. The difference packets received at the egress port will be measured. The difference
between packets transmitted into the ingress port and received at the between packets transmitted into the ingress port and received at the
egress port is the number of lost packets as measured at the egress egress port is the number of lost packets as measured at the egress
port. These packets must have unique identifiers such that only the port. These packets must have unique identifiers such that only the
test packets are measured. RFC 4737 and RFC 2680 describe the need to test packets are measured. For cases where multiple flows are
to establish the time threshold to wait before a packet is declared transmitted from ingress to egress port (e.g. IP conversations), each
as lost. packet as lost, and this threshold MUST be reported with flow must have sequence numbers within the test packets stream.
the results.
RFC 4737 and RFC 2680 [RFC2680] describe the need to to establish the
time threshold to wait before a packet is declared as lost. packet as
lost, and this threshold MUST be reported with the results.
- Out of Sequence (OOS): in additions to the LP metric, the test - Out of Sequence (OOS): in additions to the LP metric, the test
packets must be monitored for sequence and the out-of-sequence (OOS) packets must be monitored for sequence and the out-of-sequence (OOS)
packets. RFC 4689 defines the general function of sequence tracking, as packets. RFC 4689 defines the general function of sequence tracking, as
well as definitions for in-sequence and out-of-order packets. Out-of- well as definitions for in-sequence and out-of-order packets. Out-of-
order packets will be counted per RFC 4737 and RFC 2680. order packets will be counted per RFC 4737 and RFC 2680.
- Packet Delay (PD): the Packet Delay metric is the difference between - Packet Delay (PD): the Packet Delay metric is the difference between
the timestamp of the received egress port packets and the packets the timestamp of the received egress port packets and the packets
transmitted into the ingress port and specified in RFC 2285. transmitted into the ingress port and specified in RFC 2285. The
transmitting host and receiving host time must be in time sync using
NTP , GPS, etc.
- Packet Delay Variation (PDV): the Packet Delay Variation metric is - Packet Delay Variation (PDV): the Packet Delay Variation metric is
the variation between the timestamp of the received egress port the variation between the timestamp of the received egress port
packets and specified in RFC 5481. packets and specified in RFC 5481. Note that per RFC 5481, this PDV
is the variation of one-way delay across many packets in the traffic
flow.
- Shaper Rate (SR): the Shaper Rate is only applicable to the - Shaper Rate (SR): the Shaper Rate is only applicable to the
traffic shaping tests. The SR represents the average egress output traffic shaping tests. The SR represents the average egress output
rate (bps) over the test interval. rate (bps) over the test interval.
- Shaper Burst Bytes (SBB): the Shaper Burst Bytes is - Shaper Burst Bytes (SBB): the Shaper Burst Bytes is only applicable
only applicable to the traffic shaping tests. A traffic shaper will to the traffic shaping tests. A traffic shaper will emit packets in
emit packets in different size "trains" (bytes back-to-back). This different size "trains" (bytes back-to-back). This metric
metric characterizes the method by which the shaper emits traffic. characterizes the method by which the shaper emits traffic. Some
Some shapers transmit larger bursts per interval, while other shapers shapers transmit larger bursts per interval, and a burst of 1 packet
may transmit a single frame at the CIR rate (two extreme examples). would apply to the extreme case of a shaper sending a CBR stream of
single packets.
- Shaper Burst Interval(SBI): the interval is only applicable to the - Shaper Burst Interval(SBI): the interval is only applicable to the
traffic shaping tests and again is the time between a shaper emitted traffic shaping tests and again is the time between shaper emitted
bursts. bursts.
4.2. Metrics for Stateful Traffic Tests 4.2. Metrics for Stateful Traffic Tests
The stateful metrics will be based on RFC 6349 TCP metrics and will The stateful metrics will be based on RFC 6349 [RFC 6349] TCP metrics and will
include: include:
- TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP
Transfer Time for bulk transfers, which is simply the measured time Transfer Time for bulk transfers, which is simply the measured time
to transfer bytes across single or concurrent TCP connections. The to transfer bytes across single or concurrent TCP connections. The
TCP test patterns used in traffic management tests will include bulk TCP test patterns used in traffic management tests will include bulk
transfer and interactive applications. The interactive patterns include transfer and interactive applications. The interactive patterns include
instances such as HTTP business applications, database applications, instances such as HTTP business applications, database applications,
etc. The TTPET will be the measure of the time for a single execution etc. The TTPET will be the measure of the time for a single execution
of a TCP Test Pattern (TTP). Average, minimum, and maximum times will of a TCP Test Pattern (TTP). Average, minimum, and maximum times will
skipping to change at page 11, line 34 skipping to change at page 11, line 34
across the entire test duration and number of samples. across the entire test duration and number of samples.
5. Tester Capabilities 5. Tester Capabilities
The testing capabilities of the traffic management test environment The testing capabilities of the traffic management test environment
are divided into two (2) sections: stateless traffic testing and are divided into two (2) sections: stateless traffic testing and
stateful traffic testing stateful traffic testing
5.1. Stateless Test Traffic Generation 5.1. Stateless Test Traffic Generation
The test set must be capable of generating traffic at up to the The test device must be capable of generating traffic at up to the
link speed of the DUT. The test set must be calibrated to verify link speed of the DUT. The test device must be calibrated to verify
that it will not drop any packets. The test set's inherent PD and PDV that it will not drop any packets. The test device's inherent PD and
must also be calibrated and subtracted from the PD and PDV metrics. PDV must also be calibrated and subtracted from the PD and PDV metrics.
The test set must support the encapsulation to be tested such as The test device must support the encapsulation to be tested such as
VLAN, Q-in-Q, MPLS, etc. Also, the test set must allow control of IEEE 802.1Q VLAN, IEEE 802.1ad Q-in-Q, Multiprotocol Label Switching
the classification techniques defined in RFC 4689 (i.e. IP address, (MPLS), etc. Also, the test device must allow control of the
DSCP, TOS, etc classification). classification techniques defined in RFC 4689 (i.e. IP address, DSCP,
TOS, etc classification).
The open source tool "iperf" can be used to generate stateless UDP The open source tool "iperf" can be used to generate stateless UDP
traffic and is discussed in Appendix A. Since iperf is a software traffic and is discussed in Appendix A. Since iperf is a software
based tool, there will be performance limitations at higher link based tool, there will be performance limitations at higher link
speeds (e.g. GigE, 10 GigE, etc.). Careful calibration of any test speeds (e.g. GigE, 10 GigE, etc.). Careful calibration of any test
environment using iperf is important. At higher link speeds, it is environment using iperf is important. At higher link speeds, it is
recommended to use hardware based packet test equipment. recommended to use hardware based packet test equipment.
5.1.1 Burst Hunt with Stateless Traffic 5.1.1 Burst Hunt with Stateless Traffic
skipping to change at page 12, line 36 skipping to change at page 12, line 36
truly configured properly (or enabled at all). truly configured properly (or enabled at all).
5.2. Stateful Test Pattern Generation 5.2. Stateful Test Pattern Generation
The TCP test host will have many of the same attributes as the TCP test The TCP test host will have many of the same attributes as the TCP test
host defined in RFC 6349. The TCP test device may be a standard host defined in RFC 6349. The TCP test device may be a standard
computer or a dedicated communications test instrument. In both cases, computer or a dedicated communications test instrument. In both cases,
it must be capable of emulating both a client and a server. it must be capable of emulating both a client and a server.
For any test using stateful TCP test traffic, the Network Delay Emulator For any test using stateful TCP test traffic, the Network Delay Emulator
(NDE function from the lab set-up diagram) must be used in order to provide a (NDE function from the lab set-up diagram) must be used in order to
meaningful BDP. As referenced in section 2, the target traffic rate and provide a meaningful BDP. As referenced in section 2, the target
configured RTT must be verified independently using just the NDE for all traffic rate and configured RTT must be verified independently using
stateful tests (to ensure the NDE can delay without loss). just the NDE for all stateful tests (to ensure the NDE can delay without
loss).
The TCP test host must be capable to generate and receive stateful TCP The TCP test host must be capable to generate and receive stateful TCP
test traffic at the full link speed of the DUT. As a general rule of test traffic at the full link speed of the DUT. As a general rule of
thumb, testing TCP Throughput at rates greater than 500 Mbps may require thumb, testing TCP Throughput at rates greater than 500 Mbps may require
high performance server hardware or dedicated hardware based test tools. high performance server hardware or dedicated hardware based test tools.
The TCP test host must allow adjusting both Send and Receive Socket The TCP test host must allow adjusting both Send and Receive Socket
Buffer sizes. The Socket Buffers must be large enough to fill the BDP Buffer sizes. The Socket Buffers must be large enough to fill the BDP
for bulk transfer TCP test application traffic. for bulk transfer TCP test application traffic.
Measuring RTT and retransmissions per connection will generally require Measuring RTT and retransmissions per connection will generally require
a dedicated communications test instrument. In the absence of a dedicated communications test instrument. In the absence of
dedicated hardware based test tools, these measurements may need to be dedicated hardware based test tools, these measurements may need to be
conducted with packet capture tools, i.e. conduct TCP Throughput conducted with packet capture tools, i.e. conduct TCP Throughput
tests and analyze RTT and retransmissions in packet captures. tests and analyze RTT and retransmissions in packet captures.
The TCP implementation used by the test host must be specified in the The TCP implementation used by the test host must be specified in the
test results (i.e. OS version, i.e. LINUX OS kernel using TCP New Reno, test results (e.g. TCP New Reno,
TCP options supported, etc.). TCP options supported, etc.).
While RFC 6349 defined the means to conduct throughput tests of TCP bulk While RFC 6349 defined the means to conduct throughput tests of TCP bulk
transfers, the traffic management framework will extend TCP test transfers, the traffic management framework will extend TCP test
execution into interactive TCP application traffic. Examples include execution into interactive TCP application traffic. Examples include
email, HTTP, business applications, etc. This interactive traffic is email, HTTP, business applications, etc. This interactive traffic is
bi-directional and can be chatty. bi-directional and can be chatty.
The test device must not only support bulk TCP transfer application The test device must not only support bulk TCP transfer application
traffic but also chatty traffic. A valid stress test SHOULD include traffic but also chatty traffic. A valid stress test SHOULD include
both traffic types. This is due to the non-uniform, bursty nature of both traffic types. This is due to the non-uniform, bursty nature of
chatty applications versus the relatively uniform nature of bulk chatty applications versus the relatively uniform nature of bulk
transfers (the bulk transfer smoothly stabilizes to equilibrium state transfers (the bulk transfer smoothly stabilizes to equilibrium state
under lossless conditions). under lossless conditions).
While iperf is an excellent choice for TCP bulk transfer testing, the While iperf is an excellent choice for TCP bulk transfer testing, the
open source tool "Flowgrind" (referenced in Appendix A) is netperf open source tool provides the ability to control the client
client-server based and emulates interactive applications at the TCP and server request / response behavior. The netperf-wrapper tool is
layer. As with any software based tool, the performance must be a Python wrapper to run multiple simultaneous netperf instances and
qualified to the link speed to be tested. Hardware-based test equipment aggregate the results. Appendix A provides an overview of netperf /
should be considered for reliable results at higher links speeds (e.g. netperf-wrapper and another open source application emulation,
1 GigE, 10 GigE). Flowgrind. As with any software based tool, the performance must be
qualified to the link speed to be tested. Hardware-based test
equipment should be considered for reliable results at higher links
speeds (e.g. 1 GigE, 10 GigE).
5.2.1. TCP Test Pattern Definitions 5.2.1. TCP Test Pattern Definitions
As mentioned in the goals of this framework, techniques are defined As mentioned in the goals of this framework, techniques are defined
to specify TCP traffic test patterns to benchmark traffic to specify TCP traffic test patterns to benchmark traffic
management technique(s) and produce repeatable results. Some management technique(s) and produce repeatable results. Some
network devices such as firewalls, will not process stateless test network devices such as firewalls, will not process stateless test
traffic which is another reason why stateful TCP test traffic must traffic which is another reason why stateful TCP test traffic must
be used. be used.
skipping to change at page 14, line 6 skipping to change at page 14, line 6
GET url GET url
Client ------------------------> Web Client ------------------------> Web
Web 200 OK 100ms | Web 200 OK 100ms |
Browser <------------------------ Server Browser <------------------------ Server
In this example, the Client Web Browser (Client) requests a URL and In this example, the Client Web Browser (Client) requests a URL and
then the Web Server delivers the web page content to the Client then the Web Server delivers the web page content to the Client
(after a Server delay of 100 millisecond). This asynchronous, "request/ (after a Server delay of 100 millisecond). This asynchronous,
response" behavior is intrinsic to most TCP based applications such "request/response" behavior is intrinsic to most TCP based
as Email (SMTP), File Transfers (FTP and SMB), Database (SQL), Web applications such as Email (SMTP), File Transfers (FTP and SMB),
Applications (SOAP), REST, etc. The impact to the network elements is Database (SQL), Web Applications (SOAP), REST, etc. The impact to
due to the multitudes of Clients and the variety of bursty traffic, the network elements is due to the multitudes of Clients and the
which stresses traffic management functions. The actual emulation of variety of bursty traffic, which stresses traffic management functions.
the specific application protocols is not required and TCP test The actual emulation of the specific application protocols is not
patterns can be defined to mimic the application network traffic flows required and TCP test patterns can be defined to mimic the
and produce repeatable results. application network traffic flows and produce repeatable results.
There are two (2) techniques recommended by this framework to develop
standard TCP test patterns for traffic management benchmarking.
The first technique involves modeling, which have been described in
"3GPP2 C.R1002-0 v1.0" and describe the behavior of HTTP, FTP, and
WAP applications at the TCP layer. The models have been defined
with various mathematical distributions for the Request/Response
bytes and inter-request gap times. The Flowgrind tool (Appendix A)
supports many of the distributions and is a good choice as long as
the processing limits of the server platform are taken into
consideration.
The second technique is to conduct packet captures of the Application modeling techniques have been proposed in
applications to test and then to statefully play the application back "3GPP2 C.R1002-0 v1.0" and provides examples to model the behavior of
at the TCP layer. The TCP playback includes the request byte size, HTTP, FTP, and WAP applications at the TCP layer. The models have
response byte size, and inter-message gaps at both the client and the been defined with various mathematical distributions for the
server. The advantage of this method is that very realistic test Request/Response bytes and inter-request gap times.
patterns can be defined based on real world application traffic.
This framework does not specify a fixed set of TCP test patterns, but This framework does not specify a fixed set of TCP test patterns, but
does provide recommended test cases in Appendix B. Some of these examples does provide recommended test cases in Appendix B. Some of these
reflect those specified in "draft-ietf-bmwg-ca-bench-meth-04" which examples reflect those specified in "draft-ietf-bmwg-ca-bench-meth-04"
suggests traffic mixes for a variety of representative application which suggests traffic mixes for a variety of representative
profiles. Other examples are simply well known application traffic application profiles. Other examples are simply well-known
types. application traffic types such as HTTP.
6. Traffic Benchmarking Methodology 6. Traffic Benchmarking Methodology
The traffic benchmarking methodology uses the test set-up from The traffic benchmarking methodology uses the test set-up from
section 2 and metrics defined in section 4. Each test should be run section 2 and metrics defined in section 4.
for a minimum test time of 5 minutes.
Each test should compare the network device's internal statistics Each test should compare the network device's internal statistics
(available via command line management interface, SNMP, etc.) to the (available via command line management interface, SNMP, etc.) to the
measured metrics defined in section 4. This evaluates the accuracy measured metrics defined in section 4. This evaluates the accuracy
of the internal traffic management counters under individual test of the internal traffic management counters under individual test
conditions and capacity test conditions that are defined in each conditions and capacity test conditions that are defined in each
subsection. subsection.
From a device configuration standpoint, scheduling and shaping
functionality can be applied to logical ports such Link Aggregation
(LAG). This would result in the same scheduling and shaping
configuration applied to all the member physical ports. The focus of
this draft is only on tests at a physical port level.
The following sections provide the objective, procedure, metrics, and
reporting format for each test. For all test steps, the following
global parameters must be specified:
Test Runs (Tr). Defines the number of times the test needs to be run
to ensure accurate and repeatable results. The recommended value is 3.
Test Duration (Td). Defines the duration of a test iteration, expressed
in seconds. The recommended value it 60 seconds.
6.1. Policing Tests 6.1. Policing Tests
The intent of the policing tests is to verify the policer performance Policer is defined as the entity performing the policy function. The
intent of the policing tests is to verify the policer performance
(i.e. CIR-CBS and EIR-EBS parameters). The tests will verify that the (i.e. CIR-CBS and EIR-EBS parameters). The tests will verify that the
network device can handle the CIR with CBS and the EIR with EBS and network device can handle the CIR with CBS and the EIR with EBS and
will use back-back packet testing concepts from RFC 2544 (but adapted will use back-back packet testing concepts from RFC 2544 (but adapted
to burst size algorithms and terminology). Also MEF-14,19,37 provide to burst size algorithms and terminology). Also MEF-14,19,37 provide
some basis for specific components of this test. The burst hunt some basis for specific components of this test. The burst hunt
algorithm defined in section 5.1.1 can also be used to automate the algorithm defined in section 5.1.1 can also be used to automate the
measurement of the CBS value. measurement of the CBS value.
The tests are divided into two (2) sections; individual policer The tests are divided into two (2) sections; individual policer
tests and then full capacity policing tests. It is important to tests and then full capacity policing tests. It is important to
benchmark the basic functionality of the individual policer then benchmark the basic functionality of the individual policer then
proceed into the fully rated capacity of the device. This capacity may proceed into the fully rated capacity of the device. This capacity may
include the number of policing policies per device and the number of include the number of policing policies per device and the number of
policers simultaneously active across all ports. policers simultaneously active across all ports.
6.1.1 Policer Individual Tests 6.1.1 Policer Individual Tests
Objective:
Test a policer as defined by RFC 4115 or MEF 10.2, depending upon the
equipment's specification. In addition to verifying that the policer
allows the specified CBS and EBS bursts to pass, the policer test MUST
verify that the policer will remark or drop excess, and pass traffic at
the specified CBS/EBS values.
Test Summary:
Policing tests should use stateless traffic. Stateful TCP test traffic Policing tests should use stateless traffic. Stateful TCP test traffic
will generally be adversely affected by a policer in the absence of will generally be adversely affected by a policer in the absence of
traffic shaping. So while TCP traffic could be used, it is more traffic shaping. So while TCP traffic could be used, it is more
accurate to benchmark a policer with stateless traffic. accurate to benchmark a policer with stateless traffic.
The policer test shall test a policer as defined by RFC 4115 or As an example for RFC 4115, consider a CBS and EBS of 64KB and CIR and
MEF 10.2, depending upon the equipment's specification. As an example EIR of 100 Mbps on a 1GigE physical link (in color-blind mode). A
for RFC 4115, consider a CBS and EBS of 64KB and CIR and EIR of stateless traffic burst of 64KB would be sent into the policer at the
100 Mbps on a 1GigE physical link (in color-blind mode). A stateless GigE rate. This equates to approximately a 0.512 millisecond burst
traffic burst of 64KB would be sent into the policer at the GigE rate. time (64 KB at 1 GigE). The traffic generator must space these bursts
This equates to approximately a 0.512 millisecond burst time (64 KB at to ensure that the aggregate throughput does not exceed the CIR. The
1 GigE). The traffic generator must space these bursts to ensure that Ti between the bursts would equal CBS * 8 / CIR = 5.12 millisecond
the aggregate throughput does not exceed the CIR. The Ti between the in this example.
bursts would equal CBS * 8 / CIR = 5.12 millisecond in this example.
The metrics defined in section 4.1 shall be measured at the egress Test Metrics:
port and recorded. The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL
be measured at the egress port and recorded.
In addition to verifying that the policer allows the specified CBS Procedure:
and EBS bursts to pass, the policer test must verify that the policer 1. Configure the DUT policing parameters for the desired CIR/EIR and
will police at the specified CBS/EBS values. CBS/EBS values to be tested
For this portion of the test, the CBS/EBS value should be incremented 2. Configure the tester to generate a stateless traffic burst equal
by 1000 bytes higher than the configured CBS and that the egress port to CBS and an interval equal to Ti (CBS in bits / CIR)
measurements must show that the excess packets are dropped.
Additional tests beyond the simple color-blind example might include: 3. Compliant Traffic Step: Generate bursts of CBS + EBS traffic into
color-aware mode, configurations where EIR is greater than CIR, etc. the policer ingress port and measure the metrics defined in
section 4.1 (BSA, LP. OOS, PD, and PDV) at the egress port and across
the entire Td (default 60 seconds duration)
4. Excess Traffic Test: Generate bursts of greater than CBS + EBS limit
traffic into the policer ingress port and verify that the policer
only allowed the BSA bytes to exit the egress. The excess burst MUST
be recorded and the recommended value is 1000 bytes. Additional tests
beyond the simple color-blind example might include: color-aware mode,
configurations where EIR is greater than CIR, etc.
Reporting Format:
The policer individual report MUST contain all results for each
CIR/EIR/CBS/EBS test run and a recommended format is as follows:
********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: CIR, EIR, CBS, EBS
The results table should contain entries for each test run, (Test #1
to Test #Tr).
Compliant Traffic Test: BSA, LP, OOS, PD, and PDV
Excess Traffic Test: BSA
********************************************************
6.1.2 Policer Capacity Tests 6.1.2 Policer Capacity Tests
Objective:
The intent of the capacity tests is to verify the policer performance The intent of the capacity tests is to verify the policer performance
in a scaled environment with multiple ingress customer policers on in a scaled environment with multiple ingress customer policers on
multiple physical ports. This test will benchmark the maximum number multiple physical ports. This test will benchmark the maximum number
of active policers as specified by the device manufacturer. of active policers as specified by the device manufacturer.
Test Summary:
The specified policing function capacity is generally expressed in
terms of the number of policers active on each individual physical
port as well as the number of unique policer rates that are utilized.
For all of the capacity tests, the benchmarking test procedure and
report format described in Section 6.1.1 for a single policer MUST
be applied to each of the physical port policers.
As an example, a Layer 2 switching device may specify that each of the As an example, a Layer 2 switching device may specify that each of the
32 physical ports can be policed using a pool of policing service 32 physical ports can be policed using a pool of policing service
policies. The device may carry a single customer's traffic on each policies. The device may carry a single customer's traffic on each
physical port and a single policer is instantiated per physical port. physical port and a single policer is instantiated per physical port.
Another possibility is that a single physical port may carry multiple Another possibility is that a single physical port may carry multiple
customers, in which case many customer flows would be policed customers, in which case many customer flows would be policed
concurrently on an individual physical port (separate policers per concurrently on an individual physical port (separate policers per
customer on an individual port). customer on an individual port).
The specified policing function capacity is generally expressed in Test Metrics:
terms of the number of policers active on each individual physical The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL
port as well as the number of unique policer rates that are utilized. be measured at the egress port and recorded.
For all of the capacity tests, the benchmarking methodology described
in Section 6.1.1 for a single policer should be applied to each of
the physical port policers.
6.1.2.1 Maximum Policers on Single Physical Port The following sections provide the specific test scenarios,
procedures, and reporting formats for each policer capacity test.
6.1.2.1 Maximum Policers on Single Physical Port Test
Test Summary:
The first policer capacity test will benchmark a single physical port, The first policer capacity test will benchmark a single physical port,
maximum policers on that physical port. maximum policers on that physical port.
Assume multiple categories of ingress policers at rates r1, r2,...rn. Assume multiple categories of ingress policers at rates r1, r2,...rn.
There are multiple customers on a single physical port. Each customer There are multiple customers on a single physical port. Each customer
could be represented by a single tagged vlan, double tagged vlan, could be represented by a single tagged vlan, double tagged vlan,
VPLS instance etc. Each customer is mapped to a different policer. Each VPLS instance etc. Each customer is mapped to a different policer.
of the policers can be of rates r1, r2,..., rn. Each of the policers can be of rates r1, r2,..., rn.
An example configuration would be An example configuration would be
- Y1 customers, policer rate r1 - Y1 customers, policer rate r1
- Y2 customers, policer rate r2 - Y2 customers, policer rate r2
- Y3 customers, policer rate r3 - Y3 customers, policer rate r3
... ...
- Yn customers, policer rate rn - Yn customers, policer rate rn
Some bandwidth on the physical port is dedicated for other traffic (non Some bandwidth on the physical port is dedicated for other traffic (non
customer traffic); this includes network control protocol traffic. There customer traffic); this includes network control protocol traffic. There
is a separate policer for the other traffic. Typical deployments have 3 is a separate policer for the other traffic. Typical deployments have 3
categories of policers; there may be some deployments with more or less categories of policers; there may be some deployments with more or less
than 3 categories of ingress policers. than 3 categories of ingress policers.
Test Procedure:
1. Configure the DUT policing parameters for the desired CIR/EIR and
CBS/EBS values for each policer rate (r1-rn) to be tested
2. Configure the tester to generate a stateless traffic burst equal to
CBS and an interval equal to TI (CBS in bits/CIR) for each customer
stream (Y1 - Yn). The encapsulation for each customer must also be
configured according to the service tested (VLAN, VPLS, IP mapping,
etc.).
3. Compliant Traffic Step: Generate bursts of CBS + EBS traffic into the
policer ingress port for each customer traffic stream and measure the
metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) at the
egress port for each stream and across the entire Td (default 30
seconds duration)
4. Excess Traffic Test: Generate bursts of greater than CBS + EBS limit
traffic into the policer ingress port for each customer traffic
stream and verify that the policer only allowed the BSA bytes to exit
the egress for each stream. The excess burst MUST recorded and the
recommended value is 1000 bytes.
Reporting Format:
The policer individual report MUST contain all results for each
CIR/EIR/CBS/EBS test run, per customer traffic stream.
A recommended format is as follows:
********************************************************
Test Configuration Summary: Tr, Td
Customer traffic stream Encapsulation: Map each stream to VLAN,
VPLS, IP address
DUT Configuration Summary per Customer Traffic Stream: CIR, EIR,
CBS, EBS
The results table should contain entries for each test run, (Test #1
to Test #Tr).
Customer Stream Y1-Yn (see note), Compliant Traffic Test: BSA, LP,
OOS, PD, and PDV
Customer Stream Y1-Yn (see note), Excess Traffic Test: BSA
********************************************************
Note: For each test run, there will be a two (2) rows for each
customer stream, the compliant traffic result and the excess traffic
result.
6.1.2.2 Single Policer on All Physical Ports 6.1.2.2 Single Policer on All Physical Ports
Test Summary:
The second policer capacity test involves a single Policer function per The second policer capacity test involves a single Policer function per
physical port with all physical ports active. In this test, there is a physical port with all physical ports active. In this test, there is a
single policer per physical port. The policer can have one of the rates single policer per physical port. The policer can have one of the rates
r1, r2,.., rn. All the physical ports in the networking device are r1, r2,.., rn. All the physical ports in the networking device are
active. active.
Procedure:
The procedure is identical to 6.1.1, the configured parameters must be
reported per port and the test report must include results per
measured egress port
6.1.2.3 Maximum Policers on All Physical Ports 6.1.2.3 Maximum Policers on All Physical Ports
Finally the third policer capacity test involves a combination of the Finally the third policer capacity test involves a combination of the
first and second capacity test, namely maximum policers active per first and second capacity test, namely maximum policers active per
physical port and all physical ports are active . physical port and all physical ports are active.
Procedure:
Uses the procedural method from 6.1.2.1 and the configured parameters
must be reported per port and the test report must include per stream
results per measured egress port.
6.2. Queue and Scheduler Tests 6.2. Queue and Scheduler Tests
Queues and traffic Scheduling are closely related in that a queue's Queues and traffic Scheduling are closely related in that a queue's
priority dictates the manner in which the traffic scheduler's priority dictates the manner in which the traffic scheduler
transmits packets out of the egress port. transmits packets out of the egress port.
Since device queues / buffers are generally an egress function, this Since device queues / buffers are generally an egress function, this
test framework will discuss testing at the egress (although the test framework will discuss testing at the egress (although the
technique can be applied to ingress side queues). technique can be applied to ingress side queues).
Similar to the policing tests, the tests are divided into two Similar to the policing tests, the tests are divided into two
sections; individual queue/scheduler function tests and then full sections; individual queue/scheduler function tests and then full
capacity tests. capacity tests.
6.2.1 Queue/Scheduler Individual Tests 6.2.1 Queue/Scheduler Individual Tests Overview
The various types of scheduling techniques include FIFO, Strict The various types of scheduling techniques include FIFO, Strict
Priority (SP), Weighted Fair Queueing (WFQ) along with other Priority (SP), Weighted Fair Queueing (WFQ) along with other
variations. This test framework recommends to test at a minimum variations. This test framework recommends to test at a minimum
of three techniques although it is the discretion of the tester of three techniques although it is the discretion of the tester
to benchmark other device scheduling algorithms. to benchmark other device scheduling algorithms.
6.2.1.1 Testing Queue/Scheduler with Stateless Traffic 6.2.1.1 Queue/Scheduler with Stateless Traffic Test
Objective:
Verify that the configured queue and scheduling technique can
handle stateless traffic bursts up to the queue depth.
Test Summary:
A network device queue is memory based unlike a policing function, A network device queue is memory based unlike a policing function,
which is token or credit based. However, the same concepts from which is token or credit based. However, the same concepts from
section 6.1 can be applied to testing network device queues. section 6.1 can be applied to testing network device queues.
The device's network queue should be configured to the desired size The device's network queue should be configured to the desired size
in KB (queue length, QL) and then stateless traffic should be in KB (queue length, QL) and then stateless traffic should be
transmitted to test this QL. transmitted to test this QL.
A queue should be able to handle repetitive bursts with the A queue should be able to handle repetitive bursts with the
transmission gaps proportional to the bottleneck bandwidth. This transmission gaps proportional to the bottleneck bandwidth. This
skipping to change at page 18, line 18 skipping to change at page 20, line 68
result is to verify the BSA and that no packets are dropped. result is to verify the BSA and that no packets are dropped.
The scheduling function must also be characterized to benchmark the The scheduling function must also be characterized to benchmark the
device's ability to schedule the queues according to the priority. device's ability to schedule the queues according to the priority.
An example would be 2 levels of priority including SP and FIFO An example would be 2 levels of priority including SP and FIFO
queueing. Under a flow load greater the egress port speed, the queueing. Under a flow load greater the egress port speed, the
higher priority packets should be transmitted without drops (and higher priority packets should be transmitted without drops (and
also maintain low latency), while the lower priority (or best also maintain low latency), while the lower priority (or best
effort) queue may be dropped. effort) queue may be dropped.
Test Metrics:
The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL
be measured at the egress port and recorded.
Procedure:
1. Configure the DUT queue length (QL) and scheduling technique
(FIFO, SP, etc) parameters
2. Configure the tester to generate a stateless traffic burst equal
to QL and an interval equal to Ti (QL in bits/BB)
3. Generate bursts of QL traffic into the DUT and measure the
metrics defined in section 4.1 (LP, OOS, PD, and PDV) at the egress
port and across the entire Td (default 30 seconds duration)
Report Format:
The Queue/Scheduler Stateless Traffic individual report MUST contain
all results for each QL/BB test run and a recommended format is as
follows:
********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Scheduling technique, BB and QL
The results table should contain entries for each test run as follows,
(Test #1 to Test #Tr).
- LP, OOS, PD, and PDV
********************************************************
6.2.1.2 Testing Queue/Scheduler with Stateful Traffic 6.2.1.2 Testing Queue/Scheduler with Stateful Traffic
Objective:
Verify that the configured queue and scheduling technique can handle
stateless traffic bursts up to the queue depth.
Test Background and Summary:
To provide a more realistic benchmark and to test queues in layer 4 To provide a more realistic benchmark and to test queues in layer 4
devices such as firewalls, stateful traffic testing is recommended devices such as firewalls, stateful traffic testing is recommended
for the queue tests. Stateful traffic tests will also utilize the for the queue tests. Stateful traffic tests will also utilize the
Network Delay Emulator (NDE) from the network set-up configuration in Network Delay Emulator (NDE) from the network set-up configuration in
section 2. section 2.
The BDP of the TCP test traffic must be calibrated to the QL of the The BDP of the TCP test traffic must be calibrated to the QL of the
device queue. Referencing RFC 6349, the BDP is equal to: device queue. Referencing RFC 6349, the BDP is equal to:
BB * RTT / 8 (in bytes) BB * RTT / 8 (in bytes)
skipping to change at page 18, line 56 skipping to change at page 20, line 142
Two types of TCP tests must be performed: Bulk Transfer test and Micro Two types of TCP tests must be performed: Bulk Transfer test and Micro
Burst Test Pattern as documented in Appendix B. The Bulk Transfer Burst Test Pattern as documented in Appendix B. The Bulk Transfer
Test only bursts during the TCP Slow Start (or Congestion Avoidance) Test only bursts during the TCP Slow Start (or Congestion Avoidance)
state, while the Micro Burst test emulates application layer bursting state, while the Micro Burst test emulates application layer bursting
which may occur any time during the TCP connection. which may occur any time during the TCP connection.
Other tests types should include: Simple Web Site, Complex Web Site, Other tests types should include: Simple Web Site, Complex Web Site,
Business Applications, Email, SMB/CIFS File Copy (which are also Business Applications, Email, SMB/CIFS File Copy (which are also
documented in Appendix B). documented in Appendix B).
Test Metrics:
The test results will be recorded per the stateful metrics defined in The test results will be recorded per the stateful metrics defined in
section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), section 4.2, primarily the TCP Test Pattern Execution Time (TTPET),
TCP Efficiency, and Buffer Delay. TCP Efficiency, and Buffer Delay.
Procedure:
1. Configure the DUT queue length (QL) and scheduling technique
(FIFO, SP, etc) parameters
2. Configure the tester* to generate a profile of emulated of an
application traffic mixture
- The application mixture MUST be defined in terms of percentage
of the total bandwidth to be tested
- The rate of transmission for each application within the mixture
MUST be also be configurable
* The tester MUST be capable of generating a precise TCP test
patterns for each application specified, to ensure repeatable results.
3. Generate application traffic between the ingress (client side) and
egress (server side) ports of the DUT and measure the metrics (TTPET,
TCP Efficiency, and Buffer Delay) per application stream and at the
ingress and egress port (across the entire Td, default 60 seconds
duration).
Reporting Format:
The Queue/Scheduler Stateful Traffic individual report MUST contain all
results for each traffic scheduler and QL/BB test run and a recommended
format is as follows:
********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Scheduling technique, BB and QL
Application Mixture and Intensities: this is the percent configured of
each application type
The results table should contain entries for each test run as follows,
(Test #1 to Test #Tr).
- Per Application Throughout (bps) and TTPET
- Per Application Bytes In and Bytes Out
- Per Application TCP Efficiency, and Buffer Delay
********************************************************
6.2.2 Queue / Scheduler Capacity Tests 6.2.2 Queue / Scheduler Capacity Tests
Objective:
The intent of these capacity tests is to benchmark queue/scheduler The intent of these capacity tests is to benchmark queue/scheduler
performance in a scaled environment with multiple queues/schedulers performance in a scaled environment with multiple queues/schedulers
active on multiple egress physical ports. This test will benchmark active on multiple egress physical ports. This test will benchmark
the maximum number of queues and schedulers as specified by the the maximum number of queues and schedulers as specified by the
device manufacturer. Each priority in the system will map to a device manufacturer. Each priority in the system will map to a
separate queue. separate queue.
Test Metrics:
The metrics defined in section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL
be measured at the egress port and recorded.
The following sections provide the specific test scenarios, procedures,
and reporting formats for each queue / scheduler capacity test.
6.2.2.1 Multiple Queues / Single Port Active 6.2.2.1 Multiple Queues / Single Port Active
For the first scheduler / queue capacity test, multiple queues per For the first scheduler / queue capacity test, multiple queues per
port will be tested on a single physical port. In this case, port will be tested on a single physical port. In this case,
all the queues (typically 8) are active on a single physical port. all the queues (typically 8) are active on a single physical port.
Traffic from multiple ingress physical ports are directed to the Traffic from multiple ingress physical ports are directed to the
same egress physical port which will cause oversubscription on the same egress physical port which will cause oversubscription on the
egress physical port. egress physical port.
There are many types of priority schemes and combinations of There are many types of priority schemes and combinations of
priorities that are managed by the scheduler. The following priorities that are managed by the scheduler. The following
sections specify the priority schemes that should be tested. sections specify the priority schemes that should be tested.
6.2.2.1.1 Strict Priority on Egress Port 6.2.2.1.1 Strict Priority on Egress Port
Test Summary:
For this test, Strict Priority (SP) scheduling on the egress For this test, Strict Priority (SP) scheduling on the egress
physical port should be tested and the benchmarking methodology physical port should be tested and the benchmarking methodology
specified in section 6.2.1 should be applied here. For a given specified in section 6.2.1.1 and 6.2.1.2 (procedure, metrics,
and reporting format) should be applied here. For a given
priority, each ingress physical port should get a fair share of priority, each ingress physical port should get a fair share of
the egress physical port bandwidth. the egress physical port bandwidth.
TBD: RAMKI, do we need a concrete example?
Since this is a capacity test, the configuration and report
results format from 6.2.1.1 and 6.2.1.2 MUST also include:
Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate
/ mixture for stateful traffic for each physical ingress port
Report results:
- For each ingress port traffic stream, the achieved throughput
rate and metrics at the egress port
6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ) on Egress Port 6.2.2.1.2 Strict Priority + Weighted Fair Queue (WFQ) on Egress Port
Test Summary:
For this test, Strict Priority (SP) and Weighted Fair Queue (WFQ) For this test, Strict Priority (SP) and Weighted Fair Queue (WFQ)
should be enabled simultaneously in the scheduler but on a single should be enabled simultaneously in the scheduler but on a single
egress port. The benchmarking methodology specified in Section 6.2.1 egress port. The benchmarking methodology specified in Section
6.2.1.1 and 6.2.1.2 (procedure, metrics, and reporting format)
should be applied here. Additionally, the egress port bandwidth should be applied here. Additionally, the egress port bandwidth
sharing among weighted queues should be proportional to the assigned sharing among weighted queues should be proportional to the assigned
weights. For a given priority, each ingress physical port should get weights. For a given priority, each ingress physical port should get
a fair share of the egress physical port bandwidth. a fair share of the egress physical port bandwidth.
TBD: RAMKI, do we need a concrete example?
Since this is a capacity test, the configuration and report results
format from 6.2.1.1 and 6.2.1.2 MUST also include:
Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
Report results:
- For each ingress port traffic stream, the achieved throughput rate
and metrics at each queue of the egress port queue (both the SP
and WFQ queue).
Example:
- Egress Port SP Queue: throughput and metrics for ingress streams 1-n
- Egress Port WFQ Queue: throughput and metrics for ingress streams 1-n
6.2.2.2 Single Queue per Port / All Ports Active 6.2.2.2 Single Queue per Port / All Ports Active
Test Summary:
Traffic from multiple ingress physical ports are directed to the Traffic from multiple ingress physical ports are directed to the
same egress physical port, which will cause oversubscription on the same egress physical port, which will cause oversubscription on the
egress physical port. Also, the same amount of traffic is directed egress physical port. Also, the same amount of traffic is directed
to each egress physical port. to each egress physical port.
The benchmarking methodology specified in Section 6.2.1 should be The benchmarking methodology specified in Section 6.2.1.1
and 6.2.1.2 (procedure, metrics, and reporting format) should be
applied here. Each ingress physical port should get a fair share of applied here. Each ingress physical port should get a fair share of
the egress physical port bandwidth. Additionally, each egress the egress physical port bandwidth. Additionally, each egress
physical port should receive the same amount of traffic. physical port should receive the same amount of traffic.
Since this is a capacity test, the configuration and report results
format from 6.2.1.1 and 6.2.1.2 MUST also include:
Configuration:
- The number of ingress ports active during the test
- The number of egress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
Report results:
- For each egress port, the achieved throughput rate and metrics at
the egress port queue for each ingress port stream.
Example:
- Egress Port 1: throughput and metrics for ingress streams 1-n
- Egress Port n: throughput and metrics for ingress streams 1-n
6.2.2.3 Multiple Queues per Port, All Ports Active 6.2.2.3 Multiple Queues per Port, All Ports Active
Traffic from multiple ingress physical ports are directed to all Traffic from multiple ingress physical ports are directed to all
queues of each egress physical port, which will cause queues of each egress physical port, which will cause
oversubscription on the egress physical ports. Also, the same oversubscription on the egress physical ports. Also, the same
amount of traffic is directed to each egress physical port. amount of traffic is directed to each egress physical port.
The benchmarking methodology specified in Section 6.2.1 should be The benchmarking methodology specified in Section 6.2.1.1
and 6.2.1.2 (procedure, metrics, and reporting format) should be
applied here. For a given priority, each ingress physical port applied here. For a given priority, each ingress physical port
should get a fair share of the egress physical port bandwidth. should get a fair share of the egress physical port bandwidth.
Additionally, each egress physical port should receive the same Additionally, each egress physical port should receive the same
amount of traffic. amount of traffic.
Since this is a capacity test, the configuration and report results
format from 6.2.1.1 and 6.2.1.2 MUST also include:
Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate /
mixture for stateful traffic for each physical ingress port
Report results:
- For each egress port, the achieved throughput rate and metrics at
each egress port queue for each ingress port stream.
Example:
- Egress Port 1, SP Queue: throughput and metrics for ingress streams 1-n
- Egress Port 2, WFQ Queue: throughput and metrics for ingress streams 1-n
.
.
- Egress Port n, SP Queue: throughput and metrics for ingress streams 1-n
- Egress Port n, WFQ Queue: throughput and metrics for ingress streams 1-n
6.3. Shaper tests 6.3. Shaper tests
A traffic shaper is memory based like a queue, but with the added A traffic shaper is memory based like a queue, but with the added
intelligence of an active shaping element. The same concepts from intelligence of an active traffic scheduler. The same concepts from
section 6.2 (Queue testing) can be applied to testing network device section 6.2 (Queue testing) can be applied to testing network device
shaper. shaper.
Again, the tests are divided into two sections; individual shaper Again, the tests are divided into two sections; individual shaper
benchmark tests and then full capacity shaper benchmark tests. benchmark tests and then full capacity shaper benchmark tests.
6.3.1 Shaper Individual Tests 6.3.1 Shaper Individual Tests Overview
A traffic shaper generally has three (3) components that can be A traffic shaper generally has three (3) components that can be
configured: configured:
- Ingress Queue bytes - Ingress Queue bytes
- Shaper Rate, bps - Shaper Rate, bps
- Burst Committed (Bc) and Burst Excess (Be), bytes - Burst Committed (Bc) and Burst Excess (Be), bytes
The Ingress Queue holds burst traffic and the shaper then meters The Ingress Queue holds burst traffic and the shaper then meters
traffic out of the egress port according to the Shaper Rate and traffic out of the egress port according to the Shaper Rate and
Bc/Be parameters. Shapers generally transmit into policers, so Bc/Be parameters. Shapers generally transmit into policers, so
the idea is for the emitted traffic to conform to the policer's the idea is for the emitted traffic to conform to the policer's
limits. limits.
The stateless and stateful traffic test sections describe the
techniques to transmit bursts into the DUT's ingress port
and the metrics to benchmark at the shaper egress port.
6.3.1.1 Testing Shaper with Stateless Traffic 6.3.1.1 Testing Shaper with Stateless Traffic
Objective:
Test a shaper by transmitting stateless traffic bursts into the
shaper ingress port and verifying that the egress traffic is shaped
according to the shaper traffic profile.
Test Summary:
The stateless traffic must be burst into the DUT ingress port and The stateless traffic must be burst into the DUT ingress port and
not exceed the Ingress Queue. The burst can be a single burst or not exceed the Ingress Queue. The burst can be a single burst or
multiple bursts. If multiple bursts are transmitted, then the multiple bursts. If multiple bursts are transmitted, then the
Ti (Time interval) must be large enough so that the Shaper Rate is Ti (Time interval) must be large enough so that the Shaper Rate is
not exceeded. An example will clarify single and multiple burst not exceeded. An example will clarify single and multiple burst
test cases. test cases.
In the example, the shaper's ingress and egress ports are both full In the example, the shaper's ingress and egress ports are both full
duplex Gigabit Ethernet. The Ingress Queue is configured to be duplex Gigabit Ethernet. The Ingress Queue is configured to be
512,000 bytes, the Shaper Rate = 50 Mbps, and both Bc/Be configured 512,000 bytes, the Shaper Rate (SR) = 50 Mbps, and both Bc/Be configured
to be 32,000 bytes. For a single burst test, the transmitting test to be 32,000 bytes. For a single burst test, the transmitting test
device would burst 512,000 bytes maximum into the ingress port and device would burst 512,000 bytes maximum into the ingress port and
then stop transmitting. The egress port metrics from section 4.1 then stop transmitting.
will be recorded with particular emphasis on the LP, PDV, SBB, and
SBI metrics.
If a multiple burst test is to be conducted, then the burst bytes If a multiple burst test is to be conducted, then the burst bytes
divided by the time interval between the 512,000 byte bursts must divided by the time interval between the 512,000 byte bursts must
not exceed the Shaper Rate. The time interval (Ti) must adhere to not exceed the Shaper Rate. The time interval (Ti) must adhere to
a similar formula as described in section 6.2.1.1 for queues, namely: a similar formula as described in section 6.2.1.1 for queues, namely:
Ti = Ingress Queue x 8 / Shaper Rate Ti = Ingress Queue x 8 / Shaper Rate
So for the example from the previous paragraph, Ti between bursts must So for the example from the previous paragraph, Ti between bursts must
be greater than 82 millisecond (512,000 bytes x 8 / 50,000,000 bps). be greater than 82 millisecond (512,000 bytes x 8 / 50,000,000 bps).
This yields an average rate of 50 Mbps so that an Input Queue This yields an average rate of 50 Mbps so that an Input Queue
would not overflow. would not overflow.
Test Metrics:
- The metrics defined in section 4.1 (LP, OOS, PDV, SR, SBB, SBI) SHALL
be measured at the egress port and recorded.
Procedure:
1. Configure the DUT shaper ingress queue length (QL) and shaper
egress rate parameters (SR, Bc, Be) parameters
2. Configure the tester to generate a stateless traffic burst equal
to QL and an interval equal to Ti (QL in bits/BB)
3. Generate bursts of QL traffic into the DUT and measure the metrics
defined in section 4.1 (LP, OOS, PDV, SR, SBB, SBI) at the egress
port and across the entire Td (default 30 seconds duration)
Report Format:
The Shaper Stateless Traffic individual report MUST contain all results
for each QL/SR test run and a recommended format is as follows:
********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Ingress Burst Rate, QL, SR
The results table should contain entries for each test run as follows,
(Test #1 to Test #Tr).
- LP, OOS, PDV, SR, SBB, SBI
********************************************************
6.3.1.2 Testing Shaper with Stateful Traffic 6.3.1.2 Testing Shaper with Stateful Traffic
Objective:
Test a shaper by transmitting stateful traffic bursts into the shaper
ingress port and verifying that the egress traffic is shaped according
to the shaper traffic profile.
Test Summary:
To provide a more realistic benchmark and to test queues in layer 4 To provide a more realistic benchmark and to test queues in layer 4
devices such as firewalls, stateful traffic testing is also devices such as firewalls, stateful traffic testing is also
recommended for the shaper tests. Stateful traffic tests will also recommended for the shaper tests. Stateful traffic tests will also
utilize the Network Delay Emulator (NDE) from the network set-up utilize the Network Delay Emulator (NDE) from the network set-up
configuration in section 2. configuration in section 2.
The BDP of the TCP test traffic must be calculated as described in The BDP of the TCP test traffic must be calculated as described in
section 6.2.2. To properly stress network buffers and the traffic section 6.2.2. To properly stress network buffers and the traffic
shaping function, the cumulative TCP window should exceed the BDP shaping function, the cumulative TCP window should exceed the BDP
which will stress the shaper. BDP factors of 1.1 to 1.5 are which will stress the shaper. BDP factors of 1.1 to 1.5 are
skipping to change at page 21, line 64 skipping to change at page 29, line 35
Two types of TCP tests must be performed: Bulk Transfer test and Micro Two types of TCP tests must be performed: Bulk Transfer test and Micro
Burst Test Pattern as documented in Appendix B. The Bulk Transfer Burst Test Pattern as documented in Appendix B. The Bulk Transfer
Test only bursts during the TCP Slow Start (or Congestion Avoidance) Test only bursts during the TCP Slow Start (or Congestion Avoidance)
state, while the Micro Burst test emulates application layer bursting state, while the Micro Burst test emulates application layer bursting
which may any time during the TCP connection. which may any time during the TCP connection.
Other tests types should include: Simple Web Site, Complex Web Site, Other tests types should include: Simple Web Site, Complex Web Site,
Business Applications, Email, SMB/CIFS File Copy (which are also Business Applications, Email, SMB/CIFS File Copy (which are also
documented in Appendix B). documented in Appendix B).
Test Metrics:
The test results will be recorded per the stateful metrics defined in The test results will be recorded per the stateful metrics defined in
section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), section 4.2, primarily the TCP Test Pattern Execution Time (TTPET),
TCP Efficiency, and Buffer Delay. TCP Efficiency, and Buffer Delay.
Procedure:
1. Configure the DUT shaper ingress queue length (QL) and shaper
egress rate parameters (SR, Bc, Be) parameters
2. Configure the tester* to generate a profile of emulated of an
application traffic mixture
- The application mixture MUST be defined in terms of percentage
of the total bandwidth to be tested
- The rate of transmission for each application within the mixture
MUST be also be configurable
*The tester MUST be capable of generating precise TCP test patterns for
each application specified, to ensure repeatable results.
3. Generate application traffic between the ingress (client side) and
egress (server side) ports of the DUT and measure the metrics (TTPET,
TCP Efficiency, and Buffer Delay) per application stream and at the
ingress and egress port (across the entire Td, default 30 seconds
duration).
Reporting Format:
The Shaper Stateful Traffic individual report MUST contain all results
for each traffic scheduler and QL/SR test run and a recommended format
is as follows:
********************************************************
Test Configuration Summary: Tr, Td
DUT Configuration Summary: Ingress Burst Rate, QL, SR
Application Mixture and Intensities: this is the percent configured of
each application type
The results table should contain entries for each test run as follows,
(Test #1 to Test #Tr).
- Per Application Throughout (bps) and TTPET
- Per Application Bytes In and Bytes Out
- Per Application TCP Efficiency, and Buffer Delay
********************************************************
6.3.2 Shaper Capacity Tests 6.3.2 Shaper Capacity Tests
Objective:
The intent of these scalability tests is to verify shaper performance The intent of these scalability tests is to verify shaper performance
in a scaled environment with shapers active on multiple queues on in a scaled environment with shapers active on multiple queues on
multiple egress physical ports. This test will benchmark the maximum multiple egress physical ports. This test will benchmark the maximum
number of shapers as specified by the device manufacturer. number of shapers as specified by the device manufacturer.
For all of the capacity tests, the benchmarking methodology described The following sections provide the specific test scenarios, procedures,
in Section 6.3.1 for a single shaper should be applied to each of the and reporting formats for each shaper capacity test.
physical port and/or queue shapers.
6.3.2.1 Single Queue Shaped, All Physical Ports Active 6.3.2.1 Single Queue Shaped, All Physical Ports Active
Test Summary:
The first shaper capacity test involves per port shaping, all physical The first shaper capacity test involves per port shaping, all physical
ports active. Traffic from multiple ingress physical ports are directed ports active. Traffic from multiple ingress physical ports are directed
to the same egress physical port and this will cause oversubscription to the same egress physical port and this will cause oversubscription
on the egress physical port. Also, the same amount of traffic is on the egress physical port. Also, the same amount of traffic is
directed to each egress physical port. directed to each egress physical port.
The benchmarking methodology described in Section 6.3.1 should be The benchmarking methodology specified in Section 6.3.1 (procedure,
applied to each of the physical ports. Each ingress physical port metrics, and reporting format) should be applied here. Since this is a
should get a fair share of the egress physical port bandwidth. capacity test, the configuration and report results format from 6.3.1
MUST also include:
Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical ingress
port
- The traffic rate for stateful traffic and the traffic rate / mixture
for stateful traffic for each physical ingress port
- The shaped egress ports shaper parameters (QL, SR, Bc, Be)
Report results:
- For each active egress port, the achieved throughput rate and shaper
metrics for each ingress port traffic stream
Example:
- Egress Port 1: throughput and metrics for ingress streams 1-n
- Egress Port n: throughput and metrics for ingress streams 1-n
6.3.2.2 All Queues Shaped, Single Port Active 6.3.2.2 All Queues Shaped, Single Port Active
Test Summary:
The second shaper capacity test is conducted with all queues actively The second shaper capacity test is conducted with all queues actively
shaping on a single physical port. The benchmarking methodology shaping on a single physical port. The benchmarking methodology
described in per port shaping test (previous section) serves as the described in per port shaping test (previous section) serves as the
foundation for this. Additionally, each of the SP queues on the foundation for this. Additionally, each of the SP queues on the
egress physical port is configured with a shaper. For the highest egress physical port is configured with a shaper. For the highest
priority queue, the maximum amount of bandwidth available is limited priority queue, the maximum amount of bandwidth available is limited
by the bandwidth of the shaper. For the lower priority queues, the by the bandwidth of the shaper. For the lower priority queues, the
maximum amount of bandwidth available is limited by the bandwidth of maximum amount of bandwidth available is limited by the bandwidth of
the shaper and traffic in higher priority queues. the shaper and traffic in higher priority queues.
The benchmarking methodology specified in Section 6.3.1 (procedure,
metrics, and reporting format) should be applied here. Since this is
a capacity test, the configuration and report results format from
6.3.1 MUST also include:
Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical
ingress port
- The traffic rate for stateful traffic and the traffic rate/mixture
for stateful traffic for each physical ingress port
- For the active egress port, each shaper queue parameters (QL, SR, Bc, Be)
Report results:
- For each queue of the active egress port, the achieved throughput
rate and shaper metrics for each ingress port traffic stream
Example:
- Egress Port High Priority Queue: throughput and metrics for ingress streams 1-n
- Egress Port Lower Priority Queue: throughput and metrics for ingress streams 1-n
6.3.2.3 All Queues Shaped, All Ports Active 6.3.2.3 All Queues Shaped, All Ports Active
Test Summary:
And for the third shaper capacity test (which is a combination of the And for the third shaper capacity test (which is a combination of the
tests in the previous two sections),all queues will be actively tests in the previous two sections),all queues will be actively
shaping and all physical ports active. shaping and all physical ports active.
The benchmarking methodology specified in Section 6.3.1 (procedure, metrics,
and reporting format) should be applied here. Since this is a capacity test,
the configuration and report results format from 6.3.1 MUST also include:
Configuration:
- The number of physical ingress ports active during the test
- The classication marking (DSCP, VLAN, etc.) for each physical ingress port
- The traffic rate for stateful traffic and the traffic rate / mixture for
stateful traffic for each physical ingress port
- For each of the active egress ports, shaper port and per queue parameters
(QL, SR, Bc, Be)
Report results:
- For each queue of each active egress port, the achieved throughput rate
and shaper metrics for each ingress port traffic stream
Example:
- Egress Port 1 High Priority Queue: throughput and metrics for ingress streams 1-n
- Egress Port 1 Lower Priority Queue: throughput and metrics for ingress streams 1-n
.
.
- Egress Port n High Priority Queue: throughput and metrics for ingress streams 1-n
- Egress Port n Lower Priority Queue: throughput and metrics for ingress streams 1-n
6.4 Concurrent Capacity Load Tests 6.4 Concurrent Capacity Load Tests
As mentioned in the scope of this document, it is impossible to As mentioned in the scope of this document, it is impossible to
specify the various permutations of concurrent traffic management specify the various permutations of concurrent traffic management
functions that should be tested in a device for capacity testing. functions that should be tested in a device for capacity testing.
However, some profiles are listed below which may be useful However, some profiles are listed below which may be useful
to test under capacity as well: to test under capacity as well:
- Policers on ingress and queuing on egress - Policers on ingress and queuing on egress
- Policers on ingress and shapers on egress (not intended for a - Policers on ingress and shapers on egress (not intended for a
flow to be policed then shaped, these would be two different flow to be policed then shaped, these would be two different
flows tested at the same time) flows tested at the same time)
- etc. - etc.
The test procedures and reporting formatting from the previous sections may
be modified to accommodate the capacity test profile.
Appendix A: Open Source Tools for Traffic Management Testing Appendix A: Open Source Tools for Traffic Management Testing
This framework specifies that stateless and stateful behaviors should This framework specifies that stateless and stateful behaviors should
both be tested. Two (2) open source tools that can be used are iperf both be tested. Three (3) open source tools that can be used are
and Flowgrind to accomplish many of the tests proposed in this iperf, netperf (with netperf-wrapper),and Flowgrind to accomplish
framework. many of the tests proposed in this framework.
Iperf can generate UDP or TCP based traffic; a client and server must Iperf can generate UDP or TCP based traffic; a client and server must
both run the iperf software in the same traffic mode. The server is both run the iperf software in the same traffic mode. The server is
set up to listen and then the test traffic is controlled from the set up to listen and then the test traffic is controlled from the
client. Both uni-directional and bi-directional concurrent testing client. Both uni-directional and bi-directional concurrent testing
are supported. are supported.
The UDP mode can be used for the stateless traffic testing. The The UDP mode can be used for the stateless traffic testing. The
target bandwidth, packet size, UDP port, and test duration can be target bandwidth, packet size, UDP port, and test duration can be
controlled. A report of bytes transmitted, packets lost, and delay controlled. A report of bytes transmitted, packets lost, and delay
variation are provided by the iperf receiver. variation are provided by the iperf receiver.
The TCP mode can be used for stateful traffic testing to test bulk The TCP mode can be used for stateful traffic testing to test bulk
transfer traffic. The TCP Window size (which is actually the SSB), transfer traffic. The TCP Window size (which is actually the SSB),
the number of connections, the packet size, TCP port and the test the number of connections, the packet size, TCP port and the test
duration can be controlled. A report of bytes transmitted and duration can be controlled. A report of bytes transmitted and
throughput achieved are provided by the iperf sender. throughput achieved are provided by the iperf sender.
Netperf is a software application that provides network bandwidth
testing between two hosts on a network. It supports Unix domain
sockets, TCP, SCTP, DLPI and UDP via BSD Sockets.[1] Netperf provides
a number of predefined tests e.g. to measure bulk (unidirectional)
data transfer or request response performance (add reference to Wiki,
http://en.wikipedia.org/wiki/Netperf). Netperf-wrapper is a Python
script that runs multiple simultaneous netperf instances and
aggregate the results.
Flowgrind is a distributed network performance measurement tool. Flowgrind is a distributed network performance measurement tool.
Using the flowgrind controller, tests can be setup between hosts Using the flowgrind controller, tests can be setup between hosts
running flowgrind. For the purposes of this traffic management running flowgrind. For the purposes of this traffic management
testing framework, the key benefit of Flowgrind is that it can testing framework, the key benefit of Flowgrind is that it can
emulate non-bulk transfer applications such as HTTP, Email, etc. emulate non-bulk transfer applications such as HTTP, Email, etc.
This is due to fact that Flowgrind supports the concept of request
and response behavior while iperf does not.
Traffic generation options include the request size, response size, Traffic generation options include the request size, response size,
inter-request gap, and response time gap. Additionally, various inter-request gap, and response time gap. Additionally, various
distribution types are supported including constant, normal, distribution types are supported including constant, normal,
exponential, pareto, etc. These traffic generation parameters exponential, pareto, etc.
facilitate the emulation of some of the TCP test patterns
which are discussed in Appendix B.
Since these tools are software based, the host hardware must be Both netperf-wrapper and flowgrind's traffic generation parameters
qualified as capable of generating the target traffic loads facilitate the emulation of the TCP test patterns which are
without packet loss and within the packet delay variation threshold. discussed in Appendix B.
Appendix B: Stateful TCP Test Patterns Appendix B: Stateful TCP Test Patterns
This framework recommends at a minimum the following TCP test patterns This framework recommends at a minimum the following TCP test patterns
since they are representative of real world application traffic (section since they are representative of real world application traffic (section
5.2.1 describes some methods to derive other application-based TCP test 5.2.1 describes some methods to derive other application-based TCP test
patterns). patterns).
- Bulk Transfer: generate concurrent TCP connections whose aggregate - Bulk Transfer: generate concurrent TCP connections whose aggregate
number of in-flight data bytes would fill the BDP. Guidelines number of in-flight data bytes would fill the BDP. Guidelines
skipping to change at page 26, line 15 skipping to change at page 37, line 15
7. Security Considerations 7. Security Considerations
8. IANA Considerations 8. IANA Considerations
9. Conclusions 9. Conclusions
10. References 10. References
10.1. Normative References 10.1. Normative References
[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC 2119, March 1997.
[2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for Syntax
Specifications: ABNF", RFC 2234, Internet Mail Consortium and
Demon Internet Ltd., November 1997.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for [RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for
Syntax Specifications: ABNF", RFC 2234, Internet Mail Syntax Specifications: ABNF", RFC 2234, Internet Mail
Consortium and Demon Internet Ltd., November 1997. Consortium and Demon Internet Ltd., November 1997.
[RFC2680] G. Almes et al., "A One-way Packet Loss Metric for IPPM,"
RFC 2680 September 1999
[RFC2697] J. Heinanen et al., "A Single Rate Three Color Marker,"
RFC 2697, September 1999
[RFC2698] J. Heinanen et al., "A Two Rate Three Color Marker, "
RFC 2698, September 1999
[RFC4689] S. Poretsky et al., "Terminology for Benchmarking
Network-layer Traffic Control Mechanisms," RFC 4689,
October 2006
[RFC4737] A. Morton et al., "Packet Reordering Metrics," RFC 4737,
November 2006
[RFC6349] Barry Constantine et al., "Framework for TCP Throughput
Testing," RFC 6349, August 2011
[AQM-RECO] Fred Baker et al., "IETF Recommendations Regarding
Active Queue Management," August 2014,
https://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/
[MEF-10.2] "MEF 10.2: Ethernet Services Attributes Phase 2," October 2009,
http://metroethernetforum.org/PDF_Documents/technical-
specifications/MEF10.2.pdf
[MEF-12.1] "MEF 12.1: Carrier Ethernet Network Architecture Framework --
Part 2: Ethernet Services Layer - Base Elements," April 2010,
https://www.metroethernetforum.org/Assets/Technical_Specifications
/PDF/MEF12.1.pdf
[MEF-26] "MEF 26: External Network Network Interface (ENNI) - Phase 1,"
January 2010, http://www.metroethernetforum.org/PDF_Documents
/technical-specifications/MEF26.pdf
10.2. Informative References 10.2. Informative References
11. Acknowledgments 11. Acknowledgments
Authors' Addresses Authors' Addresses
Barry Constantine Barry Constantine
JDSU, Test and Measurement Division JDSU, Test and Measurement Division
Germantown, MD 20876-7100, USA Germantown, MD 20876-7100, USA
Phone: +1 240 404 2227 Phone: +1 240 404 2227
 End of changes. 118 change blocks. 
224 lines changed or deleted 729 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/