draft-ietf-bmwg-acc-bench-meth-03.txt   draft-ietf-bmwg-acc-bench-meth-04.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: January 2006 Expires in: April 2006
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
July 2005 October 2005
Methodology Guidelines for Methodology Guidelines for
Accelerated Stress Benchmarking Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-meth-03.txt> <draft-ietf-bmwg-acc-bench-meth-04.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
By submitting this Internet-Draft, I certify that any applicable
This document is an Internet-Draft and is in full conformance with patent or other IPR claims of which I am aware have been disclosed,
all provisions of Section 10 of RFC2026. and any of which I become aware will be disclosed, in accordance with
RFC 3668.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as
Drafts. Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six months
months and may be updated, replaced, or obsoleted by other and may be updated, replaced, or obsoleted by other documents at any
documents at any time. It is inappropriate to use Internet-Drafts time. It is inappropriate to use Internet-Drafts as reference
as reference material or to cite them other than as "work in material or to cite them other than as "work in progress."
progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2005). All Rights Reserved.
ABSTRACT ABSTRACT
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured
multiple protocols and security policies while forwarding traffic and With multiple protocols and security policies while forwarding
being managed. To accurately benchmark a router for deployment it is traffic and being managed. To accurately benchmark a router for
necessary that the router be tested in these simultaneous deployment it is necessary that the router be tested in these
operational conditions, which is known as Stress Testing. This simultaneous operational conditions, which is known as Stress
document provides the Methodology Guidelines for performing Stress Testing. This document provides the Methodology Guidelines for
Benchmarking of networking devices. Descriptions of Test Topology, performing Stress Benchmarking of networking devices.
Benchmarks and Reporting Format are provided in addition to procedures Descriptions of Test Topology, Benchmarks and Reporting Format
for conducting various test cases. The methodology is to be used with are provided in addition to procedures for conducting various
the companion terminology document [4]. These guidelines can be used test cases. The methodology is to be used with the companion
as the basis for additional methodology documents that benchmark terminology document [4]. These guidelines can be used as the
specific network technologies under accelerated stress. basis for additional methodology documents that benchmark specific
network technologies under accelerated stress.
Stress Benchmarking
Table of Contents Table of Contents
1. Introduction ............................................... 2 1. Introduction ............................................... 2
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Test Setup.................................................. 3 3. Test Setup.................................................. 3
3.1 Test Topologies............................................ 3 3.1 Test Topologies............................................ 3
3.2 Test Considerations........................................ 3 3.2 Test Considerations........................................ 3
3.3 Reporting Format........................................... 4 3.3 Reporting Format........................................... 4
3.3.1 Configuration Sets....................................... 5 3.3.1 Configuration Sets....................................... 5
3.3.2 Startup Conditions....................................... 6 3.3.2 Startup Conditions....................................... 6
3.3.3 Instability Conditions................................... 6 3.3.3 Instability Conditions................................... 6
3.3.4 Benchmarks............................................... 7 3.3.4 Benchmarks............................................... 7
4. Example Test Case Procedure................................. 7 4. Example Test Case Procedure................................. 7
5. Security Considerations..................................... 9 5. IANA Considerations......................................... 8
6. Normative References........................................ 9 6. Security Considerations..................................... 9
7. Informative References......................................10 7. Normative References........................................ 9
8. Author's Address............................................10 8. Informative References......................................10
9. Author's Address............................................10
1. Introduction 1. Introduction
Router testing benchmarks have consistently been made in a monolithic Router testing benchmarks have consistently been made in a monolithic
fashion wherein a single protocol or behavior is measured in an fashion wherein a single protocol or behavior is measured in an
isolated environment. It is important to know the limits for a isolated environment. It is important to know the limits for a
networking device's behavior for each protocol in isolation, however networking device's behavior for each protocol in isolation, however
this does not produce a reliable benchmark of the device's behavior this does not produce a reliable benchmark of the device's behavior
in an operational network. in an operational network.
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured with
skipping to change at page 3, line 4 skipping to change at page 3, line 4
Stress Testing of networking devices provides the following benefits: Stress Testing of networking devices provides the following benefits:
1. Evaluation of multiple protocols enabled simultaneously as 1. Evaluation of multiple protocols enabled simultaneously as
configured in deployed networks configured in deployed networks
2. Evaluation of System and Software Stability 2. Evaluation of System and Software Stability
3. Evaluation of Manageability under stressful conditions 3. Evaluation of Manageability under stressful conditions
4. Identification of Buffer Overflow conditions 4. Identification of Buffer Overflow conditions
5. Identification of Software Coding bugs such as: 5. Identification of Software Coding bugs such as:
a. Memory Leaks a. Memory Leaks
b. Suboptimal CPU Utilization b. Suboptimal CPU Utilization
c. Coding Logic c. Coding Logic
Stress Benchmarking
These benefits produce significant advantages for network operations: These benefits produce significant advantages for network operations:
1. Increased stability of routers and protocols 1. Increased stability of routers and protocols
2. Hardened routers to DoS attacks 2. Hardened routers to DoS attacks
3. Verified manageability under stress 3. Verified manageability under stress
4. Planning router resources for growth and scale 4. Planning router resources for growth and scale
2. Existing definitions 2. Existing definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 document are to be interpreted as described in BCP 14, RFC 2119
skipping to change at page 4, line 5 skipping to change at page 4, line 5
The Accelerated Stress Benchmarking test can be applied in The Accelerated Stress Benchmarking test can be applied in
service provider test environments to benchmark DUTs under service provider test environments to benchmark DUTs under
stress in an environment that is reflective of an operational stress in an environment that is reflective of an operational
network. A particular Configuration Set is defined and the network. A particular Configuration Set is defined and the
DUT is benchmarked using this configuration set and the DUT is benchmarked using this configuration set and the
Instability Conditions. Varying Configuration Sets and/or Instability Conditions. Varying Configuration Sets and/or
Instability Conditions applied in an iterative fashion can Instability Conditions applied in an iterative fashion can
provide an accurate characterization of the DUT provide an accurate characterization of the DUT
to help determine future network deployments. to help determine future network deployments.
Stress Benchmarking
___________ ___________
| DUT | | DUT |
___|Management | ___|Management |
| | | | | |
| ----------- | -----------
\/ \/
___________ ___________
| | | |
| DUT | | DUT |
|--->| |<---| |--->| |<---|
skipping to change at page 5, line 5 skipping to change at page 5, line 5
Figure 2. Logical Configuration Figure 2. Logical Configuration
3.3 Reporting Format 3.3 Reporting Format
Each methodology requires reporting of information for test Each methodology requires reporting of information for test
repeatability when benchmarking the same or different devices. repeatability when benchmarking the same or different devices.
The information that are the Configuration Sets, Instability The information that are the Configuration Sets, Instability
Conditions, and Benchmarks, as defined in [4]. Example Conditions, and Benchmarks, as defined in [4]. Example
reporting formats for each are provided below. reporting formats for each are provided below.
Stress Benchmarking
3.3.1 Configuration Sets 3.3.1 Configuration Sets
Configuration Sets may include and are not limited to the following Configuration Sets may include and are not limited to the following
examples. examples.
Example Routing Protocol Configuration Set- Example Routing Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
BGP Enabled/Disabled BGP Enabled/Disabled
Number of EBGP Peers Peers Number of EBGP Peers Peers
Number of IBGP Peers Peers Number of IBGP Peers Peers
skipping to change at page 5, line 29 skipping to change at page 5, line 31
Number of MBGP Installed Routes Routes Number of MBGP Installed Routes Routes
IGP Enabled/Disabled IGP Enabled/Disabled
IGP-TE Enabled/Disabled IGP-TE Enabled/Disabled
Number of IGP Adjacencies Adjacencies Number of IGP Adjacencies Adjacencies
Number of IGP Routes Routes Number of IGP Routes Routes
Number of Nodes per Area Nodes Number of Nodes per Area Nodes
Example MPLS Protocol Configuration Set- Example MPLS Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
MPLS-TE Enabled/Disabled MPLS-TE Enabled/Disabled
Number of Ingress Tunnels Tunnels Number of Tunnels as Ingress Tunnels
Number of Mid-Point Tunnels Tunnels Number of Tunnels as Mid-Point Tunnels
Number of Egress Tunnels Tunnels Number of Tunnels as Egress Tunnels
LDP Enabled/Disabled LDP Enabled/Disabled
Number of Sessions Sessions Number of Sessions Sessions
Number of FECs FECs Number of FECs FECs
Example Multicast Protocol Configuration Set- Example Multicast Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
PIM-SM Enabled/Disabled PIM-SM Enabled/Disabled
RP Enabled/Disabled RP Enabled/Disabled
Number of Multicast Groups Groups Number of Multicast Groups Groups
MSDP Enabled/Disabled MSDP Enabled/Disabled
skipping to change at page 6, line 4 skipping to change at page 6, line 4
Traffic Forwarding Enabled/Disabled Traffic Forwarding Enabled/Disabled
Aggregate Offered Load bps (or pps) Aggregate Offered Load bps (or pps)
Number of Ingress Interfaces number Number of Ingress Interfaces number
Number of Egress Interfaces number Number of Egress Interfaces number
TRAFFIC PROFILE TRAFFIC PROFILE
Packet Size(s) bytes Packet Size(s) bytes
Offered Load (interface) array of bps Offered Load (interface) array of bps
Number of Flows number Number of Flows number
Encapsulation(flow) array of encapsulation type Encapsulation(flow) array of encapsulation type
Stress Benchmarking
Management Configuration Set- Management Configuration Set-
PARAMETER UNITS PARAMETER UNITS
SNMP GET Rate SNMP Gets/minute SNMP GET Rate SNMP Gets/minute
Logging Enabled/Disabled Logging Enabled/Disabled
Protocol Debug Enabled/Disabled Protocol Debug Enabled/Disabled
Telnet Rate Sessions/Hour Telnet Rate Sessions/Hour
FTP Rate Sessions/Hour FTP Rate Sessions/Hour
Concurrent Telnet Sessions Sessions Concurrent Telnet Sessions Sessions
Concurrent FTP Session Sessions Concurrent FTP Session Sessions
Packet Statistics Collector Enabled/Disabled Packet Statistics Collector Enabled/Disabled
skipping to change at page 7, line 4 skipping to change at page 7, line 4
BGP Route Flap Rate routes per minutes BGP Route Flap Rate routes per minutes
IGP Route Flap Rate routes per minutes IGP Route Flap Rate routes per minutes
LSP Reroute Rate LSP per minute LSP Reroute Rate LSP per minute
Overloaded Links number Overloaded Links number
Amount Links Overloaded % of bandwidth Amount Links Overloaded % of bandwidth
FTP Rate Mb/minute FTP Rate Mb/minute
IPsec Tunnel Flap Rate tunnels per minute IPsec Tunnel Flap Rate tunnels per minute
Filter Policy Changes policies per hour Filter Policy Changes policies per hour
SSH Session Restart SSH sessions per hour SSH Session Restart SSH sessions per hour
Telnet Session Restart Telnet session per hour Telnet Session Restart Telnet session per hour
Stress Benchmarking
3.3.4 Benchmarks 3.3.4 Benchmarks
Benchmarks are as defined in [1] and listed as follow: Benchmarks are as defined in [1] and listed as follow:
PARAMETER UNITS PHASE PARAMETER UNITS PHASE
Stable Aggregate Forwarding Rate pps Startup Stable Aggregate Forwarding Rate pps Startup
Stable Latency seconds Startup Stable Latency seconds Startup
Stable Session Count sessions Startup Stable Session Count sessions Startup
Unstable Aggregate Forwarding Rate pps Instability Unstable Aggregate Forwarding Rate pps Instability
Degraded Aggregate Forwarding Rate pps Instability Degraded Aggregate Forwarding Rate pps Instability
Ave. Degraded Aggregate Forwarding Rate pps Instability Ave. Degraded Aggregate Forwarding Rate pps Instability
Unstable Latency seconds Instability Unstable Latency seconds Instability
Unstable Uncontrolled Sessions Lost sessions Instability Unstable Uncontrolled Sessions Lost sessions Instability
Recovered Aggregate Forwarding Rate pps Recovery Recovered Aggregate Forwarding Rate pps Recovery
skipping to change at page 8, line 4 skipping to change at page 8, line 4
Traffic Forwarding Enabled Traffic Forwarding Enabled
Aggregate Offered Load 10Gbps Aggregate Offered Load 10Gbps
30 Ingress Interfaces 30 Ingress Interfaces
30 Egress Interfaces 30 Egress Interfaces
Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes
Forwarding Rate[1..30] = 1Gbps Forwarding Rate[1..30] = 1Gbps
10000 Flows 10000 Flows
Encapsulation[1..5000] = IPv4 Encapsulation[1..5000] = IPv4
Encapsulation[5001.10000] = IPsec Encapsulation[5001.10000] = IPsec
Stress Benchmarking
Logging Enabled Logging Enabled
Protocol Debug Disabled Protocol Debug Disabled
SNMP Enabled SNMP Enabled
SSH Enabled SSH Enabled
20 Concurrent SSH Sessions 20 Concurrent SSH Sessions
FTP Enabled FTP Enabled
RADIUS Enabled RADIUS Enabled
TACACS Disabled TACACS Disabled
Packet Statistics Collector Enabled Packet Statistics Collector Enabled
skipping to change at page 9, line 5 skipping to change at page 9, line 5
Overloaded Links = 5 of 30 Overloaded Links = 5 of 30
Amount Links Overloaded = 20% Amount Links Overloaded = 20%
SNMP GETs = 1 per sec SNMP GETs = 1 per sec
SSH Restart Rate = 10 sessions per hour SSH Restart Rate = 10 sessions per hour
FTP Restart Rate = 10 transfers per hour FTP Restart Rate = 10 transfers per hour
FTP Transfer Rate = 100 Mbps FTP Transfer Rate = 100 Mbps
Statistics Sampling Rate = 1:1 packets Statistics Sampling Rate = 1:1 packets
6. Apply Instability Condition specific to test case. 6. Apply Instability Condition specific to test case.
Stress Benchmarking
7. Report Instability Benchmarks as follow: 7. Report Instability Benchmarks as follow:
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
Degraded Aggregate Forwarding Rate Degraded Aggregate Forwarding Rate
Ave. Degraded Aggregate Forwarding Rate Ave. Degraded Aggregate Forwarding Rate
Unstable Latency Unstable Latency
Unstable Uncontrolled Sessions Lost Unstable Uncontrolled Sessions Lost
It is RECOMMENDED that the benchmarks be measured and It is RECOMMENDED that the benchmarks be measured and
recorded at one-second intervals. recorded at one-second intervals.
skipping to change at page 9, line 30 skipping to change at page 9, line 32
Recovered Latency Recovered Latency
Recovery Time Recovery Time
Recovered Uncontrolled Sessions Lost Recovered Uncontrolled Sessions Lost
It is RECOMMENDED that the benchmarks be measured and It is RECOMMENDED that the benchmarks be measured and
recorded at one-second intervals. recorded at one-second intervals.
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
5. Security Considerations 5. IANA Considerations
This document requires no IANA considerations.
6. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
6. Normative References 7. Normative References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991. Interconnection Devices", RFC 1242, October 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Terminology for Accelerated [4] Poretsky, S. and Rao, S., "Terminology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-05, Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-07,
work in progress, July 2005. work in progress, October 2005.
Stress Benchmarking
[5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane [5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane
Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-06, Route Convergence",
work in progress, July 2005. draft-ietf-bmwg-igp-dataplane-conv-term-08, work in
progress, October 2005.
[6] Bradner, S., "Key words for use in RFCs to Indicate Requirement [6] Bradner, S., "Key words for use in RFCs to Indicate
Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
7. Informative References 8. Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", Scott [NANOG25] "Core Router Evaluation for Higher Availability",
Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
[IEEECQR] "Router Stress Testing to Validate Readiness for Network [IEEECQR] "Router Stress Testing to Validate Readiness for
Deployment", Scott Poretsky, IEEE CQR 2003. Network Deployment", Scott Poretsky, IEEE CQR 2003.
[CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data Plane [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data
Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05, Plane Route Convergence",
work in progress, July 2005. draft-ietf-bmwg-igp-dataplane-conv-meth-08, work in progress,
October 2005.
8. Author's Address 9. Author's Address
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@reefpoint.com EMail: sporetsky@reefpoint.com
Shankar Rao Shankar Rao
1801 California Street 1801 California Street
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 Denver, CO 80202
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Stress Benchmarking
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided on an This document and the information contained herein are provided on an
 End of changes. 32 change blocks. 
52 lines changed or deleted 79 lines changed or added

This html diff was produced by rfcdiff 1.27, available from http://www.levkowetz.com/ietf/tools/rfcdiff/