draft-ietf-bmwg-acc-bench-meth-06.txt   draft-ietf-bmwg-acc-bench-meth-07.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: September 2007
Intended Status: Informational
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
October 2006
Methodology Guidelines for Methodology Guidelines for
Accelerated Stress Benchmarking Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-meth-06.txt> <draft-ietf-bmwg-acc-bench-meth-07.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 1, line 41 skipping to change at page 1, line 41
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The IETF Trust (2007).
ABSTRACT ABSTRACT
Routers in an operational network are configured with multiple Routers in an operational network are configured with multiple
protocols and security policies while simultaneously forwarding protocols and security policies while simultaneously forwarding
traffic and being managed. To accurately benchmark a router for traffic and being managed. To accurately benchmark a router for
deployment it is necessary to test the router under accelerated deployment it is necessary to test the router under accelerated
operational conditions, which is known as Stress Testing. This operational conditions, which is known as Stress Testing. This
document provides the Methodology Guidelines for performing document provides the Methodology Guidelines for performing
Accelerated Stress Benchmarking of networking devices. Accelerated Stress Benchmarking of networking devices.
Descriptions of Test Topology, Benchmarks and Reporting Format Descriptions of Test Topology, Benchmarks and Reporting Format
are provided in addition to procedures for conducting various are provided in addition to procedures for conducting various
test cases. The methodology is to be used with the companion test cases. The methodology is to be used with the companion
terminology document [4]. These guidelines can be used as the terminology document [4]. These guidelines can be used as the
for Accelerated Stress Benchmarking
basis for additional methodology documents that benchmark basis for additional methodology documents that benchmark
stress conditions for specific network technologies. stress conditions for specific network technologies.
for Accelerated Stress Benchmarking
Table of Contents Table of Contents
1. Introduction ............................................... 2 1. Introduction ............................................... 2
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Test Setup.................................................. 3 3. Test Setup.................................................. 3
3.1 Test Topologies............................................ 3 3.1 Test Topologies............................................ 3
3.2 Test Considerations........................................ 3 3.2 Test Considerations........................................ 3
3.3 Reporting Format........................................... 4 3.3 Reporting Format........................................... 4
3.3.1 Configuration Sets....................................... 5 3.3.1 Configuration Sets....................................... 5
3.3.2 Startup Conditions....................................... 6 3.3.2 Startup Conditions....................................... 6
3.3.3 Instability Conditions................................... 6 3.3.3 Instability Conditions................................... 6
skipping to change at page 2, line 32 skipping to change at page 2, line 35
7. Normative References........................................11 7. Normative References........................................11
8. Informative References......................................11 8. Informative References......................................11
9. Author's Address............................................12 9. Author's Address............................................12
1. Introduction 1. Introduction
Router testing benchmarks have consistently been made in a monolithic Router testing benchmarks have consistently been made in a monolithic
fashion wherein a single protocol or behavior is measured in an fashion wherein a single protocol or behavior is measured in an
isolated environment. It is important to know the limits for a isolated environment. It is important to know the limits for a
networking device's behavior for each protocol in isolation, however networking device's behavior for each protocol in isolation, however
this does not produce a reliable benchmark of the device's behavior this does not produce a reliable benchmark of the device's behavior
in an operational network. in an operational network. Routers in an operational network are
configured with multiple protocols and security policies while
Routers in an operational network are configured with multiple simultaneously forwarding traffic and being managed. To accurately
protocols and security policies while simultaneously forwarding benchmark a router for deployment it is necessary to test that router
traffic and being managed. To accurately benchmark a router for in operational conditions by simultaneously configuring and scaling
deployment it is necessary to test that router in operational network protocols and security policies, forwarding traffic, and
conditions by simultaneously configuring and scaling network managing the device. It is helpful to accelerate these network
protocols and security policies, forwarding traffic, and managing operational conditions with Instability Conditions [4] so that the
the device. It is helpful to accelerate these network operational networking devices are stress tested.
conditions with Instability Conditions [4] so that the networking
devices are stress tested.
This document provides the Methodology for performing Stress This document provides the Methodology for performing Stress
Benchmarking of networking devices. Descriptions of Test Topology, Benchmarking of networking devices. Descriptions of Test Topology,
Benchmarks and Reporting Format are provided in addition to Benchmarks and Reporting Format are provided in addition to
procedures for conducting various test cases. The methodology is procedures for conducting various test cases. The methodology is
to be used with the companion terminology document [4]. to be used with the companion terminology document [4].
Stress Testing of networking devices provides the following benefits: Stress Testing of networking devices provides the following benefits:
1. Evaluation of multiple protocols enabled simultaneously as 1. Evaluation of multiple protocols enabled simultaneously as
configured in deployed networks configured in deployed networks
2. Evaluation of system and software stability 2. Evaluation of system and software stability
3. Evaluation of manageability under stressful conditions 3. Evaluation of manageability under stressful conditions
4. Identification of buffer overflow conditions 4. Identification of buffer overflow conditions
5. Identification of software coding bugs such as: 5. Identification of software coding bugs such as:
a. Memory leaks a. Memory leaks
for Accelerated Stress Benchmarking
b. Suboptimal CPU utilization b. Suboptimal CPU utilization
c. Coding logic c. Coding logic
for Accelerated Stress Benchmarking
These benefits produce significant advantages for network operations: These benefits produce significant advantages for network operations:
1. Increased stability of routers and protocols 1. Increased stability of routers and protocols
2. Hardened routers to DoS attacks 2. Hardened routers to DoS attacks
3. Verified manageability under stress 3. Verified manageability under stress
4. Planning router resources for growth and scale 4. Planning router resources for growth and scale
2. Existing definitions 2. Existing definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 document are to be interpreted as described in BCP 14, RFC 2119
[6]. RFC 2119 defines the use of these key words to help make the [5]. RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this intent of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track document uses these keywords, this document is not a standards track
document. document.
Terms related to Accelerated Stress Benchmarking are defined in [4]. Terms related to Accelerated Stress Benchmarking are defined in [4].
3. Test Setup 3. Test Setup
3.1 Test Topologies 3.1 Test Topologies
Figure 1 shows the physical configuration to be used for the Figure 1 shows the physical configuration to be used for the
methodologies provided in this document. The number of interfaces methodologies provided in this document. The number of interfaces
skipping to change at page 4, line 19 skipping to change at page 4, line 19
___|Management | ___|Management |
| | | | | |
| ----------- | -----------
\/ \/
___________ ___________
| | | |
| DUT | | DUT |
|--->| |<---| |--->| |<---|
xN | ----------- | xN xN | ----------- | xN
interfaces | | interfaces interfaces | | interfaces
| ___________ | | |
| | | | | | | |
|--->| Tester |<---| |--->| Tester |<---|
| | | |
----------- -----------
Figure 1. Physical Configuration Figure 1. Physical Configuration
___________ ___________ ___________ ___________
| Control | | Management| | Control | | Management|
| Plane |___ ___| Plane | | Plane |___ ___| Plane |
skipping to change at page 8, line 30 skipping to change at page 8, line 30
5. Apply Instability Conditions 5. Apply Instability Conditions
6. Apply Instability Condition specific to test case. 6. Apply Instability Condition specific to test case.
7. Report Instability Benchmarks 7. Report Instability Benchmarks
8. Stop applying all Instability Conditions 8. Stop applying all Instability Conditions
9. Report Recovery Benchmarks 9. Report Recovery Benchmarks
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
Expected Results Expected Results
Ideally the Forwarding Rates, Latencies, and Session Counts will Ideally the Forwarding Rates, Latencies, and Session Counts will
be neasured to be the same at each phase. If no packet or be measured to be the same at each phase. If no packet or
session loss occurs then the Instability Conditions MAY be session loss occurs then the Instability Conditions MAY be
increased for a repeated iteration (step 10 of the procedure). increased for a repeated iteration (step 10 of the procedure).
Example Procedure Example Procedure
1. Report Configuration Set 1. Report Configuration Set
BGP Enabled BGP Enabled
10 EBGP Peers 10 EBGP Peers
30 IBGP Peers 30 IBGP Peers
skipping to change at page 11, line 8 skipping to change at page 11, line 8
5. Apply single Instability Condition 5. Apply single Instability Condition
6. Report Instability Benchmarks 6. Report Instability Benchmarks
7. Stop applying all Instability Condition 7. Stop applying all Instability Condition
8. Report Recovery Benchmarks 8. Report Recovery Benchmarks
9. Optional - Change Configuration Set and/or Instability 9. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
for Accelerated Stress Benchmarking for Accelerated Stress Benchmarking
Expected Results Expected Results
Ideally the Forwarding Rates, Latencies, and Session Counts will Ideally the Forwarding Rates, Latencies, and Session Counts will
be neasured to be the same at each phase. If no packet or session be measured to be the same at each phase. If no packet or session
loss occurs then the Instability Conditions MAY be increased loss occurs then the Instability Conditions MAY be increased
for a repeated iteration (step 10 of the procedure). for a repeated iteration (step 10 of the procedure).
5. IANA Considerations 5. IANA Considerations
This document requires no IANA considerations. This document requires no IANA considerations.
6. Security Considerations 6. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
skipping to change at page 11, line 33 skipping to change at page 11, line 33
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, October 1991. Interconnection Devices", RFC 1242, October 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, October 1998. Devices", RFC 2285, October 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Terminology for Accelerated [4] Poretsky, S. and Rao, S., "Terminology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-10, Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-11,
work in progress, October 2006. work in progress, March 2007.
[5] Bradner, S., "Key words for use in RFCs to Indicate [5] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
8. Informative References 8. Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", [NANOG25] "Core Router Evaluation for Higher Availability",
Scott Poretsky, NANOG 25, October 8, 2002, Toronto, CA. Scott Poretsky, NANOG 25, October 8, 2002, Toronto, CA.
[IEEECQR] "Router Stress Testing to Validate Readiness for [IEEECQR] "Router Stress Testing to Validate Readiness for
Network Deployment", Scott Poretsky, IEEE CQR 2003. Network Deployment", Scott Poretsky, IEEE CQR 2003.
[CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data
Plane Route Convergence", Plane Route Convergence",
draft-ietf-bmwg-igp-dataplane-conv-meth-11, work in draft-ietf-bmwg-igp-dataplane-conv-meth-11, work in
progress, October 2006. progress, March 2007.
for Accelerated Stress Benchmarking for Accelerated Stress Benchmarking
9. Author's Address 9. Author's Address
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
skipping to change at page 13, line 8 skipping to change at page 13, line 8
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 Denver, CO 80202
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
for Accelerated Stress Benchmarking for Accelerated Stress Benchmarking
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2006). Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided on an This document and the information contained herein are provided
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
Intellectual Property Intellectual Property
The IETF takes no position regarding the validity or scope of any The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be on the procedures with respect to rights in RFC documents can be
 End of changes. 17 change blocks. 
33 lines changed or deleted 34 lines changed or added

This html diff was produced by rfcdiff 1.33. The latest version is available from http://tools.ietf.org/tools/rfcdiff/