draft-ietf-bmwg-acc-bench-term-07.txt   draft-ietf-bmwg-acc-bench-term-08.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: April 2006 Expires in: September 2006
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
October 2005 March 2006
Terminology for Accelerated Stress Benchmarking Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-07.txt> <draft-ietf-bmwg-acc-bench-term-08.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 1, line 41 skipping to change at page 1, line 41
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2005). All Rights Reserved. Copyright (C) The Internet Society (2006). All Rights Reserved.
ABSTRACT ABSTRACT
This document provides the Terminology for performing Stress This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the Test: Startup, Instability and Recovery are defined along with the
benchmarks and configuration terms associated with the each phase. benchmarks and configuration terms associated with the each phase.
Also defined are the Benchmark Planes fundamental to stress testing Also defined are the Benchmark Planes fundamental to stress testing
configuration, setup and measurement. The terminology is to be configuration, setup and measurement. The terminology is to be
used with the companion framework and methodology documents. used with the companion framework and methodology documents.
skipping to change at page 2, line 39 skipping to change at page 2, line 39
3.3.4.2 Traffic Profile...................................13 3.3.4.2 Traffic Profile...................................13
3.3.5 Management Plane......................................14 3.3.5 Management Plane......................................14
3.3.5.1 Management Plane Configuration Set................14 3.3.5.1 Management Plane Configuration Set................14
3.3.6 Security Plane........................................15 3.3.6 Security Plane........................................15
3.3.6.1 Security Plane Configuration Set..................15 3.3.6.1 Security Plane Configuration Set..................15
3.3.6.2 Security Plane Startup Conditions.................16 3.3.6.2 Security Plane Startup Conditions.................16
3.4 Instability...............................................16 3.4 Instability...............................................16
3.4.1 Instability Phase.....................................16 3.4.1 Instability Phase.....................................16
3.4.2 Benchmarks............................................17 3.4.2 Benchmarks............................................17
3.4.2.1 Unstable Aggregate Forwarding Rate................17 3.4.2.1 Unstable Aggregate Forwarding Rate................17
3.4.2.2 Degraded Aggregate Forwarding Rate................17 3.4.2.2 Aggregate Forwarding Rate Degradation.............17
3.4.2.3 Average Degraded Aggregate Forwarding Rate........18 3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18
3.4.2.4 Unstable Latency..................................18 3.4.2.4 Unstable Latency..................................18
3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19
3.4.3 Instability Conditions................................19 3.4.3 Instability Conditions................................19
3.4.3.1 Control Plane Instability Conditions..............19 3.4.3.1 Control Plane Instability Conditions..............19
3.4.3.2 Data Plane Instability Conditions.................20 3.4.3.2 Data Plane Instability Conditions.................20
3.4.3.3 Management Plane Instability Conditions...........20 3.4.3.3 Management Plane Instability Conditions...........20
3.4.3.4 Security Plane Instability Conditions.............20 3.4.3.4 Security Plane Instability Conditions.............20
3.5 Recovery..................................................21 3.5 Recovery..................................................21
3.5.1 Recovery Phase........................................21 3.5.1 Recovery Phase........................................21
3.5.2 Benchmarks............................................21 3.5.2 Benchmarks............................................21
skipping to change at page 3, line 30 skipping to change at page 3, line 30
router under test can be benchmarked with faster test duration. router under test can be benchmarked with faster test duration.
Testing a router in accelerated network conditions is known as Testing a router in accelerated network conditions is known as
Accelerated Stress Testing. Accelerated Stress Testing.
This document provides the Terminology for performing Stress This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase. benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase Forwarding Rate and control plane Session Count during each phase
of the test. Also defined are the Benchmark Planes fundamental to of the test. For each plane, the Configuration Set, Startup
stress testing configuration, setup and measurement. These are Conditions, and Instability Conditions are defined. Also defined are
the Control Plane, Data Plane, Management Plane and Security Plane the Benchmark Planes fundamental to stress testing configuration, setup
For each plane, the Configuration Set, Startup Conditions, and and measurement. These are the Control Plane, Data Plane, Management
Instability Conditions are defined. White Box benchmarks are Plane and Security Plane Multiple benchmarks are made for each
provided in Appendix 1 for additional DUT behavior measurements. Benchmark Plane during each Phase. Benchmarks can be compared across
The terminology is to be used with the companion methodology multiple planes for the same DUT or at the same plane for 2 or more
document [4]. The sequence of phases, actions, and benchmarks DUTS. These benchmarks White Box benchmarks are provided in Appendix 1
are shown in Table 1. for additional DUT behavior measurements. The terminology is to be
used with the companion methodology document [4]. The sequence of
phases, actions, and benchmarks are shown in Table 1.
2. Existing definitions 2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this Devices" should be consulted before attempting to make use of this
document. For the sake of clarity and continuity this RFC adopts document. For the sake of clarity and continuity this RFC adopts
the template for definitions set out in Section 2 of RFC 1242. the template for definitions set out in Section 2 of RFC 1242.
Definitions are indexed and grouped together in sections for ease Definitions are indexed and grouped together in sections for ease
of reference. of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
skipping to change at page 17, line 36 skipping to change at page 17, line 36
Measurement units: Measurement units:
pps pps
Issues: Issues:
None None
See Also: See Also:
Instability Conditions Instability Conditions
Aggregate Forwarding Rate Aggregate Forwarding Rate
3.4.2.2 Degraded Aggregate Forwarding Rate 3.4.2.2 Aggregate Forwarding Rate Degradation
Definition: Definition:
The reduction in Aggregate Forwarding Rate during the The reduction in Aggregate Forwarding Rate during the
Instability Phase. Instability Phase.
Discussion: Discussion:
The Degraded Aggregate Forwarding Rate is calculated The Aggregate Forwarding Rate Degradation is calculated
for each measurement of the Unstable Aggregate for each measurement of the Unstable Aggregate Forwarding
Forwarding Rate. The Degraded Aggregate Forwarding Rate. The Aggregate Forwarding Rate Degradation is
Rate is calculated by subtracting each measurement calculated by subtracting each measurement of the Unstable
of the Unstable Aggregate Forwarding Rate from the Aggregate Forwarding Rate from the Stable Aggregate
Stable Aggregate Forwarding Rate, such that Forwarding Rate, such that
Degraded Forwarding Rate = Aggregate Forwarding Rate Degradation=
Stable Aggregate Forwarding Rate - Stable Aggregate Forwarding Rate -
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
Ideally, the Degraded Aggregate Forwarding Rate is zero. Ideally, the Aggregate Forwarding Rate Degradation is zero.
Measurement Units: Measurement Units:
pps pps
Stress Benchmarking Stress Benchmarking
Issues: Issues:
None None
See Also: See Also:
Instability Phase Instability Phase
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
3.4.2.3 Average Degraded Aggregate Forwarding Rate 3.4.2.3 Average Aggregate Forwarding Rate Degradation
Definition Definition
DUT Benchmark that is the calculated average of the DUT Benchmark that is the calculated average of the
obtained Degraded Forwarding Rates. obtained Degraded Forwarding Rates.
Discussion: Discussion:
Average Aggregate Forwarding Rate Degradation=
(Sum (Stable Aggregate Forwarding Rate) -
Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples
Measurement Units: Measurement Units:
pps pps
Issues: Issues:
None None
See Also: See Also:
Degraded Aggregate Forwarding Rate Aggregate Forwarding Rate Degradation
3.4.2.4 Unstable Latency 3.4.2.4 Unstable Latency
Definition: Definition:
The average increase in measured packet latency during The average increase in measured packet latency during
the Instability Phase compared to the Startup Phase. the Instability Phase compared to the Startup Phase.
Discussion: Discussion:
Latency SHOULD be measured at a fixed interval during the Latency SHOULD be measured at a fixed interval during the
Instability Phase. Unstable Latency is the difference Instability Phase. Unstable Latency is the difference
skipping to change at page 23, line 14 skipping to change at page 23, line 14
Stress Benchmarking Stress Benchmarking
Discussion Discussion
Recovery Time is measured beginning at the instant the Recovery Time is measured beginning at the instant the
Instability Phase ends until the Recovered Aggregate Instability Phase ends until the Recovered Aggregate
Forwarding Rate equals the Stable Aggregate Forwarding Forwarding Rate equals the Stable Aggregate Forwarding
Rate for a minimum duration of 180 consecutive seconds. Rate for a minimum duration of 180 consecutive seconds.
Measurement Units: Measurement Units:
seconds milliseconds
Issues: Issues:
None None
See Also: See Also:
Recovered Aggregate Forwarding Rate Recovered Aggregate Forwarding Rate
Stable Aggregate Forwarding Rate Stable Aggregate Forwarding Rate
3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost
skipping to change at page 24, line 32 skipping to change at page 24, line 32
5. Security Considerations 5. Security Considerations
Documents of this type do not directly effect the security of Documents of this type do not directly effect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
6. References 6. References
6.1 Normative References 6.1 Normative References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, October 1991. Interconnection Devices", RFC 1242, March 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Methodology for Accelerated [4] Poretsky, S. and Rao, S., "Methodology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-04, Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-04,
work in progress, October 2005. work in progress, March 2006.
[5] Bradner, S., "Key words for use in RFCs to Indicate Requirement [5] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", RFC 2119, March 1997. Levels", RFC 2119, March 1997.
6.2 Informative References 6.2 Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", Scott [NANOG25] "Core Router Evaluation for Higher Availability", Scott
skipping to change at page 26, line 8 skipping to change at page 26, line 8
Measurement units: % Measurement units: %
Issues: None Issues: None
See Also: See Also:
Minimum Available Memory Minimum Available Memory
Stress Benchmarking Stress Benchmarking
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided on an This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
 End of changes. 18 change blocks. 
31 lines changed or deleted 35 lines changed or added

This html diff was produced by rfcdiff 1.29, available from http://www.levkowetz.com/ietf/tools/rfcdiff/