draft-ietf-bmwg-acc-bench-term-01.txt   draft-ietf-bmwg-acc-bench-term-02.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: April 2004 Expires in: August 2004
Scott Poretsky Scott Poretsky
Quarry Technologies Quarry Technologies
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
Ray Piatt February 2004
Cable and Wireless
October 2003
Terminology for Accelerated Stress Benchmarking Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-01.txt> <draft-ietf-bmwg-acc-bench-term-02.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 43 skipping to change at page 1, line 40
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
ABSTRACT ABSTRACT
This terminology document provides the terms to be used for This document provides the Terminology for performing Stress
benchmarking a router or switch under Accelerated Stress conditions. Benchmarking of networking devices. The three phases of the Stress
Terminology is provided for performing this benchmark using Test: Startup, Instability and Recovery are defined along with the
the Contol Plane, Data Plane, Management Plane, and Security benchmarks and configuration terms associated with the each phase.
Plane of the Device Under Test. Terms are defined for Also defined are the Benchmark Planes fundamental to stress testing
Configuration, Startup Conditions, Instability Conditions, Failure configuration, setup and measurement. The terminology is to be
Conditions, and Benchmarks. used with the companion framework and methodology documents.
Table of Contents Table of Contents
1. Introduction ............................................... 3 1. Introduction ............................................... 3
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Term definitions............................................ 3 3. Term definitions............................................ 3
3.1 Benchmark Planes.......................................... 3 3.1 General Terms............................................. 3
3.1.1 Control Plane........................................... 4 3.1.1 Benchmark Planes...................................... 3
3.1.2 Data Plane.............................................. 5 3.1.2 Configuration Sets.................................... 4
3.1.3 Management Plane........................................ 5 3.1.3 Startup Conditions.................................... 4
3.1.4 Security Plane.......................................... 6 3.1.4 Instability Conditions................................ 5
3.2 Configuration Sets........................................ 6 3.1.5 Aggregate Forwarding Rate............................. 6
3.2.1 Control Plane Configuration Set......................... 7 3.1.6 Controlled Session Loss............................... 6
3.2.2 Data Plane Configuration Set............................ 7 3.1.7 Uncontrolled Session Loss............................. 6
3.2.2.1 Traffic Profile....................................... 8 3.2 Benchmark Planes.......................................... 7
3.2.3 Management Plane Configuration Set...................... 8 3.2.1 Control Plane......................................... 7
3.2.4 Security Plane Configuration Set........................ 9 3.2.2 Data Plane............................................ 7
3.3 Startup Conditions........................................10 3.2.3 Management Plane...................................... 8
3.3.1 Control Plane Startup Conditions........................10 3.2.4 Security Plane........................................ 8
3.3.2 Security Plane Startup Conditions.......................11 3.3 Startup................................................... 9
3.4 Instability Conditions....................................11 3.3.1 Startup Phase......................................... 9
3.4.1 Control Plane Instability Conditions....................12 3.3.2 Benchmarks............................................10
3.4.2 Data Plane Instability Conditions.......................12 3.3.2.1 Stable Aggregate Forwarding Rate..................10
3.4.3 Management Plane Instability Conditions.................13 3.3.2.2 Stable Session Count..............................11
3.4.4 Security Plane Instability Conditions...................13 3.3.3 Control Plane.........................................11
3.5 Failure Conditions........................................14 3.3.3.1 Control Plane Configuration Set...................11
3.5.1 Unexpected Packet Loss..................................14 3.3.3.2 Control Plane Startup Conditions..................12
3.5.2 Unexpected Session Loss.................................14 3.3.4 Data Plane............................................12
3.5.3 Misrouted Packets.......................................15 3.3.4.1 Data Plane Configuration Set......................12
3.5.4 Access Denial...........................................15 3.3.4.2 Traffic Profile...................................13
3.5.5 Errored Management Value................................16 3.3.5 Management Plane......................................13
3.5.6 Security Plane Failure..................................16 3.3.5.1 Management Plane Configuration Set................13
3.6 Benchmarks................................................17 3.3.6 Security Plane........................................14
3.6.1 Success Threshold.......................................17 3.3.6.1 Security Plane Configuration Set..................14
3.6.2 Accelerated-Life Test Duration..........................17 3.3.6.2 Security Plane Startup Conditions.................15
4. Security Considerations.....................................18 3.4 Instability...............................................15
5. References..................................................18 3.4.1 Instability Phase.....................................15
6. Author's Address............................................18 3.4.2 Benchmarks............................................16
7. Full Copyright Statement....................................19 3.4.2.1 Unstable Aggregate Forwarding Rate................16
Appendix 1 - White Box Benchmarks..............................20 3.4.2.2 Degraded Aggregate Forwarding Rate................16
3.4.2.3 Average Degraded Aggregate Forwarding Rate........17
3.4.2.4 Unstable Uncontrolled Sessions Lost...............17
3.4.3 Instability Conditions................................18
3.4.3.1 Control Plane Instability Conditions..............18
3.4.3.2 Data Plane Instability Conditions.................18
3.4.3.3 Management Plane Instability Conditions...........19
3.4.3.4 Security Plane Instability Conditions.............19
3.5 Recovery..................................................20
3.5.1 Recovery Phase........................................20
3.5.2 Benchmarks............................................20
3.5.2.1 Recovered Aggregate Forwarding Rate...............20
3.5.2.2 Recovery Time.....................................21
3.5.2.3 Recovered Uncontrolled Sessions Lost..............21
4. Security Considerations.....................................22
5. References..................................................22
6. Author's Address............................................22
7. Full Copyright Statement....................................23
Appendix 1 - White Box Benchmarks..............................23
1. Introduction 1. Introduction
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured with
multiple protocols and security policies while forwarding traffic and multiple protocols and security policies while forwarding traffic and
being managed. To accurately benchmark a router for deployment it is being managed. To accurately benchmark a router for deployment it is
necessary to test that router in operational conditions by necessary to test that router in operational conditions by
simultaneously configuring and scaling network protocols and security simultaneously configuring and scaling network protocols and security
policies, forwarding traffic, and managing the device. It is helpful policies, forwarding traffic, and managing the device. It is helpful
to accelerate these network operational conditions so that the to accelerate these network operational conditions so that the
router under test can be benchmarked with faster test duration. router under test can be benchmarked with faster test duration.
Testing a router in accelerated network conditions is known as Testing a router in accelerated network conditions is known as
Accelerated Stress Testing. Accelerated Stress Testing.
This document defines the Benchmark Planes used for the Accelerated This document provides the Terminology for performing Stress
Life Benchmarking. These are the Control Plane, Data Plane, Benchmarking of networking devices. The three phases of the Stress
Management Plane, and Security Plane. For each plane, the Test: Startup, Instability and Recovery are defined along with the
Configuration Set, Startup Conditions, and Instability Conditions. benchmark and configuration terms associated with the each phase.
Terms are also defined for observing Failure Conditions and reporting Benchmarks for stress testing are defined using the Aggregate
Benchmarks. White Box benchmarks are provided in Appendix 1 for Forwarding Rate and control plane Session Count during each phase
additional DUT behavior measurements. of the test. Also defined are the Benchmark Planes fundamental to
stress testing configuration, setup and measurement. These are
the Control Plane, Data Plane, Management Plane and Security Plane
For each plane, the Configuration Set, Startup Conditions, and
Instability Conditions are defined. White Box benchmarks are
provided in Appendix 1 for additional DUT behavior measurements.
The terminology is to be used with the companion framework
document [6] and methodology document [7].
2. Existing definitions 2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this Devices" should be consulted before attempting to make use of this
document. document.
For the sake of clarity and continuity this RFC adopts the template For the sake of clarity and continuity this RFC adopts the template
for definitions set out in Section 2 of RFC 1242. Definitions are for definitions set out in Section 2 of RFC 1242. Definitions are
indexed and grouped together in sections for ease of reference. indexed and grouped together in sections for ease of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC 2119. this document are to be interpreted as described in RFC 2119.
3. Term definitions 3. Term definitions
3.1 General Terms
3.1 Benchmark Planes 3.1.1 Benchmark Planes
Definition: Definition:
The features, conditions, and behavior for the Accelerated Stress The features, conditions, and behavior for the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
There are four Benchmark Planes: Control Plane, Data Plane, There are four Benchmark Planes: Control Plane, Data Plane,
Management Plane, and Security Plane as shown in Figure 1. The Management Plane, and Security Plane as shown in Figure 1. The
Benchmark Planes define the Configuration, Startup Conditions, Benchmark Planes define the Configuration, Startup Conditions,
Instability Conditions, and Failure Conditions used for the test. Instability Conditions, and Failure Conditions used for the test.
skipping to change at page 4, line 25 skipping to change at page 4, line 31
| ----------- | | ----------- |
| | | |
| ___________ | | ___________ |
| | Data | | | | Data | |
|--->| Plane |<---| |--->| Plane |<---|
| | | |
----------- -----------
Figure 1. Router Accelerated Stress Benchmarking Planes Figure 1. Router Accelerated Stress Benchmarking Planes
Measurement units: N/A Measurement units:
N/A
Issues: None Issues:
None
See Also: See Also:
Control Plane Control Plane
Data Plane Data Plane
Management Plane Management Plane
Security Plane Security Plane
3.1.1 Control Plane 3.1.2 Configuration Sets
Definition:
The features and scaling limits used during the Accelerated Stress
Benchmarking.
Discussion:
There are four Configuration Sets: Control Plane Configuration Set,
Data Plane Configuration Set, Management Plane Configuration Set,
and Security Plane Configuration Set.
Measurement units:
N/A
Issues:
None
See Also:
Control Plane Configuration Set
Data Plane Configuration Set
Management Plane Configuration Set
Security Plane Configuration Set
3.1.3 Startup Conditions
Definition:
Test conditions that occur at the start of the Accelerated
Life Benchmark to establish conditions for the remainder of
the test.
Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions are defined for the Control
Plane and Security Plane.
Measurement units:
N/A
Issues:
None
See Also:
Control Plane Startup Conditions
Data Plane Startup Conditions
Management Plane Startup Conditions
Security Plane Startup Conditions
3.1.4 Instability Conditions
Definition:
Test conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT.
Discussion:
Instability Conditions are applied to the DUT after the
Startup Conditions have completed. Instability Conditions
occur for the Control Plane, Data Plane, Management Plane,
and Security Plane.
Measurement units:
N/A
Issues:
None
See Also:
Control Plane Instability Conditions
Data Plane Instability Conditions
Management Plane Instability Conditions
Security Plane Instability Conditions
3.1.5 Aggregate Forwarding Rate
Definition:
Sum of forwarding rates for all interfaces on the
DUT during the Startup Phase.
Discussion:
Each interface of the DUT forwards traffic at some
measured rate. The Aggregate Forwarding Rate is the
sum of forwarding rates for all interfaces on the DUT.
Measurement units:
pps
Issues:
None
See Also:
Startup Phase
3.1.6 Controlled Session Loss
Definition:
Control Plane sessions that are intentionally brought
down during the Stress test.
Discussion:
The test equipment is able to control protocol
session state with the DUT.
Measurement units:
None
Issues:
None
See Also:
Uncontrolled Session Loss
3.1.7 Uncontrolled Session Loss
Definition:
Control Plane sessions that are in the down state
but were not intentionally brought down during the
Stress test.
Discussion:
The test equipment is able to control protocol
session state with the DUT. The test equipment
is also to monitor for sessions lost with the
DUT which the test equipment itself did not
intentionally bring down.
Measurement units:
N/A
Issues:
None
See Also:
Controlled Session Loss
3.2 Benchmark Planes
3.2.1 Control Plane
Definition: Definition:
The Description of the control protocols enabled for The Description of the control protocols enabled for
the Accelerated Stress Benchmarking. the Accelerated Stress Benchmarking.
Discussion: Discussion:
The Control Plane defines the Configuration, Startup The Control Plane defines the Configuration, Startup
Conditions, Instability Conditions, and Failure Conditions, Instability Conditions, and Failure
Conditions of the control protocols used for the test. Conditions of the control protocols used for the test.
Control Plane protocols may include routing protocols, Control Plane protocols may include routing protocols,
multicast protocols, and MPLS protocols. These can be multicast protocols, and MPLS protocols. These can be
enabled or disabled for a benchmark test. enabled or disabled for a benchmark test.
Measurement units: N/A Measurement units:
N/A
Issues: None Issues:
None
See Also: See Also:
Benchmark Planes Benchmark Planes
Control Plane Configuration Set Control Plane Configuration Set
Control Plane Startup Conditions Control Plane Startup Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Control Plane Failure Conditions Control Plane Failure Conditions
3.1.2 Data Plane
3.2.2 Data Plane
Definition: Definition:
The data traffic profile used for the Accelerated Stress The data traffic profile used for the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
The Data Plane defines the Configuration, Startup The Data Plane defines the Configuration, Startup
Conditions, Instability Conditions, and Failure Conditions, Instability Conditions, and Failure
Conditions of the data traffic used for the test. Conditions of the data traffic used for the test.
The Data Plane includes the traffic and interface The Data Plane includes the traffic and interface
skipping to change at page 5, line 27 skipping to change at page 8, line 22
Measurement Units: Measurement Units:
N/A N/A
See Also: See Also:
Benchmark Planes Benchmark Planes
Data Plane Configuration Set Data Plane Configuration Set
Data Plane Startup Conditions Data Plane Startup Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Data Plane Failure Conditions Data Plane Failure Conditions
3.1.3 Management Plane 3.2.3 Management Plane
Definition: Definition:
The Management features and tools used for the The Management features and tools used for the
Accelerated Stress Benchmarking. Accelerated Stress Benchmarking.
Discussion: Discussion:
A key component of the Accelerated Stress Benchmarking is the A key component of the Accelerated Stress Benchmarking is the
Management Plane to assess manageability of the router Management Plane to assess manageability of the router
under stress. The Management Plane defines the Configuration, under stress. The Management Plane defines the Configuration,
Startup Conditions, Instability Conditions, and Failure Startup Conditions, Instability Conditions, and Failure
skipping to change at page 6, line 4 skipping to change at page 8, line 49
Issues: Issues:
None None
See Also: See Also:
Benchmark Planes Benchmark Planes
Management Plane Configuration Set Management Plane Configuration Set
Management Plane Startup Conditions Management Plane Startup Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Management Plane Failure Conditions Management Plane Failure Conditions
3.1.4 Security Plane
3.2.4 Security Plane
Definition: Definition:
The Security features used during the Accelerated Stress The Security features used during the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
The Control Plane defines the Configuration, Startup The Control Plane defines the Configuration, Startup
Conditions, Instability Conditions, and Failure Conditions, Instability Conditions, and Failure
Conditions of the security features and protocols used for Conditions of the security features and protocols used for
the test. The Security Plane includes the ACLs, Firewall, the test. The Security Plane includes the ACLs, Firewall,
skipping to change at page 6, line 30 skipping to change at page 9, line 25
Issues: Issues:
None None
See Also: See Also:
Benchmark Planes Benchmark Planes
Security Plane Configuration Set Security Plane Configuration Set
Security Plane Startup Conditions Security Plane Startup Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
Security Plane Failure Conditions Security Plane Failure Conditions
3.2 Configuration Sets 3.3 Startup
3.3.1 Startup Phase
Definition
The portion of the benchmarking test in which the
Startup Conditions are generated with the DUT. This
begins with the attempt to establish the first session
and ends when the last Control Plane session is
established.
Discussion:
The Startup Phase is the first Phase of the
benchmarking test preceding the Instability Phase
and Recovery Phase. It is specified by the
Configuration Sets and Startup Conditions for
each Benchmark Plane. The Startup Phase ends
and Instability Phase may begin when the
Configuration Sets are achieved with the DUT.
Measurement Units:
None
Issues:
The 'last control plane session is established' may not
be a sufficient indicator that steady-state is achieved
and Instability Conditions can be applied to begin the
Instability Phase.
See Also:
Benchmark Plane
Configuration Sets
Startup Conditions
Instability Phase
Recovery Phase
3.3.2 Benchmarks
3.3.2.1 Stable Aggregate Forwarding Rate
Definition: Definition:
The features and scaling limits used during the Accelerated Stress Average rate of traffic forwarded by the DUT during the
Benchmarking. Startup Phase.
Discussion: Discussion:
There are four Configuration Sets: Control Plane Configuration Set, Stable Aggregate Forwarding Rate is the calculated
Data Plane Configuration Set, Management Plane Configuration Set, average the Aggregate Forwarding Rates measured during
and Security Plane Configuration Set. the Startup Phase. It is recommended that the
Aggregate Forwarding Rate is measured at one-second
intervals until the Startup Phase ends.
Measurement units: N/A Measurement units:
pps
Issues: None Issues:
The act of the DUT establishing the Startup Conditions
could influence the forwarding rate in certain
implementations so that this "baseline" for the
remainder of the test is lowered. The alternative is
to change the definition of Startup Aggregate
Forwarding Rate so that it measured during the Startup
Phase, but after Startup Conditions are achieved.
The disadvantage of this definition would be that it
loses measurement of any impact establishing Startup
Conditions would have on forwarding rate. When
comparing the Startup Aggregate Forwarding Rate benchmark
of two devices it is preferred to know the impact
establishing Startup Conditions has on Forwarding Rate.
The definition was therefore selected so that Stable
Aggregate Forwarding Rate is calculated from measurement
samples throughout the entire Startup Phase.
See Also: See Also:
Control Plane Configuration Set Instability Conditions
Data Plane Configuration Set Aggregate Forwarding Rate
Management Plane Configuration Set Stable Aggregate Forwarding Rate
Security Plane Configuration Set 3.3.2.2 Stable Session Count
3.2.1 Control Plane Configuration Set Definition:
Total number of control plane sessions/adjacencies
established and maintained by the DUT prior to
Instability Conditions being initiated.
Discussion:
This measurement should be made after the Control
Plane Startup Conditions are applied to the DUT.
Measurement units:
sessions
Issues:
None
See Also:
Instability Conditions
3.3.3 Control Plane
3.3.3.1 Control Plane Configuration Set
Definition: Definition:
The routing protocols and scaling values used for the Accelerated The routing protocols and scaling values used for the Accelerated
Life Benchmarking. Life Benchmarking.
Discussion: Discussion:
Control Plane Configuration Set is shown in Figure 2 and specifies Control Plane Configuration Set is shown in Figure 2 and specifies
the Routing Protocols, Multicast, and MPLS configuration. Specific the Routing Protocols, Multicast, and MPLS configuration. Specific
protocols can be enabled or disabled for a benchmark test. protocols can be enabled or disabled for a benchmark test.
Measurement units: None Measurement units:
N/A
Issues: None Issues:
None
____________ ____________ ____________ ____________ ____________ ____________
| Routing | | Multicast | | MPLS | | Routing | | Multicast | | MPLS |
| Protocols |___ | Protocols | __| Protocols | | Protocols |___ | Protocols | __| Protocols |
| | | | | | | | | | | | | | | |
------------ | ------------ | ------------ ------------ | ------------ | ------------
| | | | | |
| | | | | |
| \/ | | \/ |
| ___________ | | ___________ |
| | | | | | | |
|------->| DUT |<------| |------->| DUT |<------|
| | ``| |
----------- -----------
Figure 2. Control Plane Configuration Module Figure 2. Control Plane Configuration Module
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
Management Configuration Set Management Configuration Set
Security Configuration Set Security Configuration Set
3.2.2 Data Plane Configuration Set 3.3.3.2 Control Plane Startup Conditions
Definition:
Control Plane conditions that occur at the start
of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test.
Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions for the Control Plane include
session establishment rate, number of sessions established
and number of routes learned.
Measurement units:
N/A
Issues:
None
See Also:
Startup Conditions
Security Plane Startup Conditions
Control Plane Configuration Set
3.3.4 Data Plane
3.3.4.1 Data Plane Configuration Set
Definition: Definition:
The data traffic profile enabled for the Accelerated Stress The data traffic profile enabled for the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
Data Plane Configuration Set includes the Traffic Profile and Data Plane Configuration Set includes the Traffic Profile and
interfaces used for the Accelerated Stress Benchmarking. interfaces used for the Accelerated Stress Benchmarking.
Measurement Units: Measurement Units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Traffic Profile Traffic Profile
3.2.2.1 Traffic Profile 3.3.4.2 Traffic Profile
Definition Definition
The characteristics of the Offered Load to the DUT used for The characteristics of the Offered Load to the DUT used for
the Accelerated Stress Benchmarking. the Accelerated Stress Benchmarking.
Discussion Discussion
The Traffic Profile specifies the number of packet size(s), packet The Traffic Profile specifies the number of packet size(s), packet
rate per interface, number of flows, and encapsulation used for the offered load to the DUT. rate per interface, number of flows, and encapsulation used for the
offered load to the DUT.
Measurement Units: Measurement Units:
Traffic Profile is reported as follows: Traffic Profile is reported as follows:
Parameter Units Parameter Units
--------- ------ --------- ------
Packet Size(s) bytes Packet Size(s) bytes
Packet Rate(interface) array of packets per second Packet Rate(interface) array of packets per second
Number of Flows number Number of Flows number
Encapsulation(flow) array of encapsulation type Encapsulation(flow) array of encapsulation type
Issues: Issues:
None None
See Also: See Also:
Data Plane Configuration Set Data Plane Configuration Set
3.2.3 Management Plane Configuration Set 3.3.5 Management Plane
3.3.5.1 Management Plane Configuration Set
Definition: Definition:
The router management features enabled for the The router management features enabled for the
Accelerated Stress Test. Accelerated Stress Test.
Discussion: Discussion:
A key component of the Accelerated Stress Test is the Management Configuration Set to assess manageability of the router under A key component of the Accelerated Stress Test is the Management
stress. The Management Configuration Set defines the management configuration of the DUT. Features that are part of the Configuration Set to assess manageability of the router under
stress. The Management Configuration Set defines the management
configuration of the DUT. Features that are part of the
Management Configuration Set include SNMP, Logging/Debug, and Management Configuration Set include SNMP, Logging/Debug, and
Statistics Collection, and services such as FTP, as shown in Statistics Collection, and services such as FTP, as shown in
Figure 3. Figure 3.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
skipping to change at page 9, line 29 skipping to change at page 14, line 29
\/ \/
___________ ___________
| Packet | | Packet |
| Statistics| | Statistics|
| Collector | | Collector |
| | | |
----------- -----------
Figure 3. Management Plane Configuration Set Figure 3. Management Plane Configuration Set
3.2.4 Security Plane Configuration Set 3.3.6 Security Plane
3.3.6.1 Security Plane Configuration Set
Definition: Definition:
Security features and scaling enabled for the Accelerated Stress Security features and scaling enabled for the Accelerated Stress
Test. Test.
Discussion: Discussion:
The Security Plane Configuration Set includes the configuration The Security Plane Configuration Set includes the configuration
and scaling of ACLs, Firewall, IPsec, and User Access, as shown and scaling of ACLs, Firewall, IPsec, and User Access, as shown
in Figure 4. in Figure 4.
skipping to change at page 10, line 15 skipping to change at page 15, line 15
N/A N/A
Issues: Issues:
None None
See Also: See Also:
ACL Configuration Set ACL Configuration Set
Secure Protocol Configuration Set Secure Protocol Configuration Set
Password Login Configuration Set Password Login Configuration Set
3.3 Startup Conditions 3.3.6.2 Security Plane Startup Conditions
Definition: Definition:
Test conditions that occur at the start of the Accelerated Security Plane conditions that occur at the start
Life Benchmark to establish conditions for the remainder of of the Accelerated Stress Benchmarking to establish conditions
the test. for the remainder of the test.
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions are defined for the Control failure. Startup Conditions for the Security Plane include
Plane and Security Plane. session establishment rate, number of sessions established
and number of policies learned, and number of user access
sessions opened.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Startup Conditions Startup Conditions
Data Plane Startup Conditions Data Plane Startup Conditions
Management Plane Startup Conditions Management Plane Startup Conditions
Security Plane Startup Conditions Security Plane Startup Conditions
3.3.1 Control Plane Startup Conditions 3.4 Instability
3.4.1 Instability Phase
Definition: Definition:
Control Plane conditions that occur at the start The portion of the benchmarking test in which the
of the Accelerated Stress Benchmarking to establish conditions Instability Conditions are offered to the DUT.
for the remainder of the test.
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce The Instability Phase is the middle Phase of
failure. Startup Conditions for the Control Plane include of the benchmarking test following the Startup
session establishment rate, number of sessions established Phase and preceding the Recovery Phase.
and number of routes learned.
Measurement units: Measurement Units:
N/A None
Issues: Issues:
None None
See Also: See Also:
Startup Conditions Instability Conditions
Security Plane Startup Conditions Startup Phase
Control Plane Configuration Set Recovery Phase
3.3.2 Security Plane Startup Conditions 3.4.2 Benchmarks
3.4.2.1 Unstable Aggregate Forwarding Rate
Definition: Definition:
Security Plane conditions that occur at the start Rate of taffic forwarded by the DUT during the
of the Accelerated Stress Benchmarking to establish conditions Instability Phase.
for the remainder of the test.
Discussion: Discussion:
Startup Conditions may cause stress on the DUT and produce Unstable Aggregated Forwarding Rate is an instantaneous
failure. Startup Conditions for the Security Plane include measurement of the Aggregate Forwarding Rate during the
session establishment rate, number of sessions established Instability Phase. It is recommended that the
and number of policies learned, and number of user access Unstable Aggregate Forwarding Rate is measured at
sessions opened. one-second intervals.
Measurement units: Measurement units:
N/A pps
Issues: Issues:
None None
See Also: See Also:
Startup Conditions Instability Conditions
Data Plane Startup Conditions Aggregate Forwarding Rate
Management Plane Startup Conditions
Security Plane Startup Conditions
3.4 Instability Conditions 3.4.2.2 Degraded Aggregate Forwarding Rate
Definition: Definition:
Test conditions that occur during the Accelerated Stress The reduction in Aggregate Forwarding Rate during the
Benchmark to produce instability and stress the DUT. Instability Phase.
Discussion: Discussion:
Instability Conditions are experienced by the DUT after the The Degraded Aggregate Forwarding Rate is calculated
Startup Conditions have completed. Instability Conditions for each measurement of the Unstable Aggregate
occur for the Control Plane, Data Plane, Management Plane, Forwarding Rate. The Degraded Aggregate Forwarding
and Security Plane. Rate is calculated by subtracting each measurement
of the Unstable Aggregate Forwarding Rate from the
Stable Aggregate Forwarding Rate, such that
Degraded Forwarding Rate =
Stable Aggregate Forwarding Rate -
Unstable Aggregate Forwarding Rate
Ideally, the Degraded Aggregate Forwarding Rate is zero.
Measurement Units:
pps
Issues:
None
See Also:
Instability Phase
Unstable Aggregate Forwarding Rate
3.4.2.3 Average Degraded Aggregate Forwarding Rate
Definition
DUT Benchmark that is the calculated average of the
obtained Degraded Forwarding Rates.
Discussion:
Measurement Units:
pps
Issues:
None
See Also:
Degraded Aggregate Forwarding Rate
3.4.2.4 Unstable Uncontrolled Sessions Lost
Definition:
Control Plane sessions that are in the down state
but were not intentionally brought down during the
Instability Phase.
Discussion:
The test equipment is able to control protocol
session state with the DUT. The test equipment
is also to monitor for sessions lost with the
DUT which the test equipment itself did not
intentionally bring down.
Measurement units: Measurement units:
N/A sessions
Issues: Issues:
None None
See Also: See Also:
Control Plane Instability Conditions Controlled Session Loss
Data Plane Instability Conditions Uncontrolled Session Loss
Management Plane Instability Conditions 3.4.3 Instability Conditions
Security Plane Instability Conditions
3.4.1 Control Plane Instability Conditions 3.4.3.1 Control Plane Instability Conditions
Definition: Definition:
Control Plane conditions that occur during the Accelerated Stress Control Plane conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT. Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Control Plane Instability Conditions are experienced by the DUT Control Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Control Plane after the Startup Conditions have completed. Control Plane
Instability Conditions experienced by the DUT include session Instability Conditions experienced by the DUT include session
loss, route withdrawal, and route cost changes. loss, route withdrawal, and route cost changes.
skipping to change at page 12, line 34 skipping to change at page 18, line 30
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.4.2 Data Plane Instability Conditions 3.4.3.2 Data Plane Instability Conditions
Definition: Definition:
Data Plane conditions that occur during the Accelerated Stress Data Plane conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT. Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Data Plane Instability Conditions are experienced by the DUT Data Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Data Plane after the Startup Conditions have completed. Data Plane
Instability Conditions experienced by the DUT include interface Instability Conditions experienced by the DUT include interface
shutdown, link loss, and overloaded links. shutdown, link loss, and overloaded links.
skipping to change at page 13, line 4 skipping to change at page 19, line 4
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.4.3 Management Plane Instability Conditions 3.4.3.3 Management Plane Instability Conditions
Definition: Definition:
Management Plane conditions that occur during the Accelerated Management Plane conditions that occur during the Accelerated
Life Benchmark to produce instability and stress the DUT. Life Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Management Plane Instability Conditions are experienced by the DUT Management Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Management Plane after the Startup Conditions have completed. Management Plane
Instability Conditions experienced by the DUT include repeated Instability Conditions experienced by the DUT include repeated
FTP of large files. FTP of large files.
skipping to change at page 13, line 28 skipping to change at page 19, line 28
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Security Plane Instability Conditions Security Plane Instability Conditions
3.4.4 Security Plane Instability Conditions 3.4.3.4 Security Plane Instability Conditions
Definition: Definition:
Security Plane conditions that occur during the Accelerated Security Plane conditions that occur during the Accelerated
Life Benchmark to produce instability and stress the DUT. Life Benchmark to produce instability and stress the DUT.
Discussion: Discussion:
Security Plane Instability Conditions are experienced by the DUT Security Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Security Plane after the Startup Conditions have completed. Security Plane
Instability Conditions experienced by the DUT include session Instability Conditions experienced by the DUT include session
loss and policy changes. loss and uninitiated policy changes.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Control Plane Instability Conditions Control Plane Instability Conditions
Data Plane Instability Conditions Data Plane Instability Conditions
Management Plane Instability Conditions Management Plane Instability Conditions
3.5 Failure Conditions 3.5 Recovery
3.5.1 Unexpected Session Loss
Definition:
Session Loss due to Control Plane or Data Plane failure during the
Accelerated Stress Benchmarking.
Discussion:
Session loss will be observed due to Convergence Events produced
by the Instability Conditions. This is not considered a failure
during the Accelerated Stress Benchmarking. It is expected behavior.
Unexpected packet loss occurs due to Data Plane or Control Plane
failure.
Measurement units:
sessions
Issues:
None
See Also:
Unexpected Packet Loss
Misrouted Packets
Access Denial
Errored Management Values
Security Plane Failure
3.5.2 Unexpected Packet Loss
Definition:
Packet Loss due to Control Plane or Data Plane failure during the
Accelerated Stress Benchmarking.
Discussion:
Packet loss will be observed due to Convergence Events produced
by the Instability Conditions. This is not considered a failure
during the Accelerated Stress Benchmarking. It is expected behavior.
Unexpected packet loss occurs due to Data Plane or Control Plane
failure.
Measurement units:
packets
Issues:
None
See Also: 3.5.1 Recovery Phase
Unexpected Session Loss
Misrouted Packets
Access Denial
Errored Management Values
Security Plane Failure
3.5.3 Misrouted Packets
Definition: Definition:
Packets routed to the incorrect next-hop or LSP due to Control The portion of the benchmarking test in which the
Plane failure during the Accelerated Stress Benchmarking. Startup Conditions are generated with the DUT, but
the Instability Conditions are no longer offered to
the DUT.
Discussion: Discussion:
Packets will be rerouted to new next-hops and LSPs due to Convergence The Recovery Phase is the final Phase of the
Events produced by the Instability Conditions. This is not considered benchmarking test following the Startup Phase and
a failure during the Accelerated Stress Benchmarking. It is expected Instability Phase. Startup Conditions must not be
behavior. Unexpected Misrouted packets occur due to Control Plane Restarted.
failure.
Measurement units:
packets
Issues: Measurement Units:
None None
See Also:
Unexpected Packet Loss
Unexpected Session Loss
Access Denial
Errored Management Values
Security Plane Failure
3.5.4 Access Denial
Definition:
Inability to management the DUT due to the inability
to gain remote access to the DUT.
Discussion:
Handling of Instability Conditions may cause a Control Plane failure
of the DUT externally observable as the inability to remotely manage
the DUT. Whether using SNMP, Telnet, SSH, or FTP, failure to manage
the device constitues a failure.
Measurement units:
Failed session
Issues: Issues:
None None
See Also: See Also:
Unexpected Packet Loss Startup Conditions
Unexpected Session Loss Startup Phase
Misrouted Packets Instability Conditions
Errored Management Value Instability Phase
Security Plane Failure
3.5.5 Errored Management Value
Definition:
Incorrect SNMP object value or statistic during the Accelerated
Life Benchmark.
Discussion:
Management of a router relies upon accurate reporting of object
values and statistics. The Accelerated Stress Benchmarkinging validates
that the DUT is able to report correct values when experiencing
Instability Conditions.
Measurement units:
Name of incorrect object or statistic
Issues: 3.5.2 Benchmarks
None
See Also: 3.5.2.1 Recovered Aggregate Forwarding Rate
Unexpected Packet Loss
Unexpected Session Loss
Misrouted Packets
Access Denial
Security Plane Failure
Success Threshold
3.5.6 Security Plane Failure Definition
Definition: Rate of traffic forwarded by the DUT during the Recovery
Violation of Security Policy for packets directed to the DUT Phase.
or routed through the DUT.
Discussion: Discussion:
Security Policy is configured to permit and deny access to packets Recovered Aggregate Forwarding Rate is an instantaneous
destined for the DUT and packets to be routed through the DUT. measurement of the Aggregate Forwarding Rate during the
A violation of this security policy is when either a packet to Recovery Phase. It is recommended that the Recovered
be denied is permitted or a packet to be permitted is denied. Aggregate Forwarding Rate is measured at one-second
intervals. Ideally, each measurement of the Recovered
Aggregate Forwarding Rate equals the Stable Aggregate
Forwarding Rate because the Instability Conditions
do not exist in both the Startup and Recovery Phases.
Measurement units: Measurement Units:
packets pps
Issues: Issues:
None None
See Also: See Also:
Unexpected Packet Loss Aggregate Forwarding Rate
Unexpected Session Loss Recovery Phase
Misrouted Packets Recovered Aggregate Forwarding Rate
Access Denial Startup Phase
Errored Management Value Stable Aggregate Forwarding Rate
3.6 Benchmarks
3.6.1 Success Threshold 3.5.2.2 Recovery Time
Definition: Definition
The intended number of hours for the DUT to operate without failure The amount of time for the Recovered Aggregate Forwarding
during the Accelerated Stress Benchmarking. Rate to become equal to the Stable Aggregate Forwarding Rate.
Discussion: Discussion
Success Threshold is a user determined value of time. It is Recovery Time is measured beginning at the instant the
necessary to define the Success Threshold because it is possible Instability Phase ends until the Recovered Aggregate
for the DUT to experience zero failures for given Configuration Forwarding Rate equals the Stable Aggregate Forwarding
Sets, Startup Conditions, and Instability Conditions Rate for a minimum of 180 consecutive seconds.
Measurement units: Measurement Units:
Hours seconds
Issues: Issues:
None None
See Also: See Also:
Accelerated-Life Test Duration Recovered Aggregate Forwarding Rate
Stable Aggregate Forwarding Rate
3.6.2 Accelerated-Life Test Duration 3.5.2.3 Recovered Uncontrolled Control Plane Sessions Lost
Definition: Definition:
The run-time for the Accelerated Stress Test without the DUT Control Plane sessions that are in the down state
exhibiting an error in the control plane, data plane, management but were not intentionally brought down during the
plane, or security plane. Recovery Phase.
Discussion: Discussion:
For a successful test, the Test Duration will equal the The test equipment is able to control protocol
Success Threshold. In the event of any failure, the Test Duration session state with the DUT. The test equipment
is less than the Success Threshold. Reporting of the Test is also to monitor for sessions lost with the
Duration MUST inlcude the Failure Condition from those defined DUT which the test equipment itself did not
in this document. intentionally bring down.
Measurement units: Measurement units:
Hours sessions
Issues: Issues:
None None
See Also: See Also:
Success Threshold Controlled Session Loss
Uncontrolled Session Loss
4. Security Considerations 4. Security Considerations
Documents of this type do not directly effect the security of Documents of this type do not directly effect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
5. References 5. References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991. Interconnection Devices", RFC 1242, July 1991.
skipping to change at page 18, line 27 skipping to change at page 22, line 27
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] "Core Router Evaluation for Higher Availability", Scott [4] "Core Router Evaluation for Higher Availability", Scott
Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
[5] "Router Stress Testing to Validate Readiness for Network [5] "Router Stress Testing to Validate Readiness for Network
Deployment", Scott Poretsky, IEEE CQR 2003. Deployment", Scott Poretsky, IEEE CQR 2003.
[6] Poretsky, S. and Rao, S., "Framework for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-frame-01,
work in progress, February 2004.
[7] Poretsky, S. and Rao, S., "Methodology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-00,
work in progress, February 2004.
6. Author's Address 6. Author's Address
Scott Poretsky Scott Poretsky
Quarry Technologies Quarry Technologies
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@quarrytech.com EMail: sporetsky@quarrytech.com
Shankar Rao Shankar Rao
950 17th Street
Suite 1900
Qwest Communications Qwest Communications
Denver, CO 80210 Denver, CO
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: srao@qwest.net Email: shankar.rao@qwest.com
Ray Piatt
Cable and Wireless
11700 Plaza America Drive
Reston, VA 20190
USA
Phone: + 1 703 292 2113
Email: rpiatt@cw.net
7. Full Copyright Statement 7. Full Copyright Statement
Copyright (C) The Internet Society (1998). All Rights Copyright (C) The Internet Society (1998). All Rights
Reserved. Reserved.
This document and translations of it may be copied and This document and translations of it may be copied and
furnished to others, and derivative works that comment on or furnished to others, and derivative works that comment on or
otherwise explain it or assist in its implementation may be otherwise explain it or assist in its implementation may be
prepared, copied, published and distributed, in whole or in prepared, copied, published and distributed, in whole or in
skipping to change at page 20, line 6 skipping to change at page 23, line 35
not be revoked by the Internet Society or its successors or not be revoked by the Internet Society or its successors or
assigns. This document and the information contained herein is assigns. This document and the information contained herein is
provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. FOR A PARTICULAR PURPOSE.
Appendix 1. White Box Benchmarking Terminology Appendix 1. White Box Benchmarking Terminology
Appendix 1.1 Minimum Available Memory
Minimum Available Memory
Definition: Definition:
Minimum DUT Available Memory during the duration of the Minimum DUT Available Memory during the duration of the
Accelerated Stress Test. Accelerated Stress Test.
Discussion: Discussion:
It is necessary to monitor DUT memory to measure this It is necessary to monitor DUT memory to measure this
benchmark. benchmark.
Measurement units: Measurement units:
bytes bytes
Issues: Issues:
None None
See Also: See Also:
Maximum CPU Utilization Maximum CPU Utilization
Maximum CPU Utilization
Appendix 1.2 Maximum CPU Utilization
Definition: Definition:
Maximum DUT CPU utilization during the duration of the Maximum DUT CPU utilization during the duration of the
Accelerated Stress Test. Accelerated Stress Test.
Discussion: Discussion:
It is necessary to monitor DUT CPU Utilization to measure It is necessary to monitor DUT CPU Utilization to measure
this benchmark. this benchmark.
Measurement units: Measurement units:
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/