draft-ietf-bmwg-acc-bench-meth-05.txt   draft-ietf-bmwg-acc-bench-meth-06.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: December 2006
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
October 2006
Methodology Guidelines for Methodology Guidelines for
Accelerated Stress Benchmarking Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-meth-05.txt> <draft-ietf-bmwg-acc-bench-meth-06.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 1, line 42 skipping to change at page 1, line 44
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The Internet Society (2006).
ABSTRACT ABSTRACT
Routers in an operational network are simultaneously configured Routers in an operational network are configured with multiple
With multiple protocols and security policies while forwarding protocols and security policies while simultaneously forwarding
traffic and being managed. To accurately benchmark a router for traffic and being managed. To accurately benchmark a router for
deployment it is necessary that the router be tested in these deployment it is necessary to test the router under accelerated
simultaneous operational conditions, which is known as Stress operational conditions, which is known as Stress Testing. This
Testing. This document provides the Methodology Guidelines for document provides the Methodology Guidelines for performing
performing Acceleratd Stress Benchmarking of networking devices. Accelerated Stress Benchmarking of networking devices.
Descriptions of Test Topology, Benchmarks and Reporting Format Descriptions of Test Topology, Benchmarks and Reporting Format
are provided in addition to procedures for conducting various are provided in addition to procedures for conducting various
test cases. The methodology is to be used with the companion test cases. The methodology is to be used with the companion
terminology document [4]. These guidelines can be used as the terminology document [4]. These guidelines can be used as the
basis for additional methodology documents that benchmark basis for additional methodology documents that benchmark
stress conditions for specific network technologies. stress conditions for specific network technologies.
Stress Benchmarking for Accelerated Stress Benchmarking
Table of Contents Table of Contents
1. Introduction ............................................... 2 1. Introduction ............................................... 2
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Test Setup.................................................. 3 3. Test Setup.................................................. 3
3.1 Test Topologies............................................ 3 3.1 Test Topologies............................................ 3
3.2 Test Considerations........................................ 3 3.2 Test Considerations........................................ 3
3.3 Reporting Format........................................... 4 3.3 Reporting Format........................................... 4
3.3.1 Configuration Sets....................................... 5 3.3.1 Configuration Sets....................................... 5
3.3.2 Startup Conditions....................................... 6 3.3.2 Startup Conditions....................................... 6
3.3.3 Instability Conditions................................... 6 3.3.3 Instability Conditions................................... 6
3.3.4 Benchmarks............................................... 7 3.3.4 Benchmarks............................................... 7
4. Example Test Case Procedure................................. 7 4. Stress Test Procedure...................................... 8
5. IANA Considerations......................................... 8 4.1 General Methodology with Multiple Instability Conditions... 8
6. Security Considerations..................................... 9 4.2 General Methodology with a Single Instability Condition....10
7. Normative References........................................ 9 5. IANA Considerations.........................................11
8. Informative References......................................10 6. Security Considerations.....................................11
9. Author's Address............................................10 7. Normative References........................................11
8. Informative References......................................11
9. Author's Address............................................12
1. Introduction 1. Introduction
Router testing benchmarks have consistently been made in a monolithic Router testing benchmarks have consistently been made in a monolithic
fashion wherein a single protocol or behavior is measured in an fashion wherein a single protocol or behavior is measured in an
isolated environment. It is important to know the limits for a isolated environment. It is important to know the limits for a
networking device's behavior for each protocol in isolation, however networking device's behavior for each protocol in isolation, however
this does not produce a reliable benchmark of the device's behavior this does not produce a reliable benchmark of the device's behavior
in an operational network. in an operational network.
Routers in an operational network are simultaneously configured with Routers in an operational network are configured with multiple
multiple protocols and security policies while forwarding traffic protocols and security policies while simultaneously forwarding
and being managed. To accurately benchmark a router for deployment traffic and being managed. To accurately benchmark a router for
it is necessary to test that router in operational conditions by deployment it is necessary to test that router in operational
simultaneously configuring and scaling network protocols and security conditions by simultaneously configuring and scaling network
policies, forwarding traffic, and managing the device. It is helpful protocols and security policies, forwarding traffic, and managing
to accelerate these network operational conditions with Instability the device. It is helpful to accelerate these network operational
Conditions [4] so that the networking devices are stress tested. conditions with Instability Conditions [4] so that the networking
devices are stress tested.
This document provides the Methodology for performing Stress This document provides the Methodology for performing Stress
Benchmarking of networking devices. Descriptions of Test Topology, Benchmarking of networking devices. Descriptions of Test Topology,
Benchmarks and Reporting Format are provided in addition to Benchmarks and Reporting Format are provided in addition to
procedures for conducting various test cases. The methodology is procedures for conducting various test cases. The methodology is
to be used with the companion terminology document [4]. to be used with the companion terminology document [4].
Stress Testing of networking devices provides the following benefits: Stress Testing of networking devices provides the following benefits:
1. Evaluation of multiple protocols enabled simultaneously as 1. Evaluation of multiple protocols enabled simultaneously as
configured in deployed networks configured in deployed networks
2. Evaluation of System and Software Stability 2. Evaluation of system and software stability
3. Evaluation of Manageability under stressful conditions 3. Evaluation of manageability under stressful conditions
4. Identification of Buffer Overflow conditions 4. Identification of buffer overflow conditions
5. Identification of Software Coding bugs such as: 5. Identification of software coding bugs such as:
a. Memory Leaks a. Memory leaks
b. Suboptimal CPU Utilization b. Suboptimal CPU utilization
c. Coding Logic c. Coding logic
Stress Benchmarking for Accelerated Stress Benchmarking
These benefits produce significant advantages for network operations: These benefits produce significant advantages for network operations:
1. Increased stability of routers and protocols 1. Increased stability of routers and protocols
2. Hardened routers to DoS attacks 2. Hardened routers to DoS attacks
3. Verified manageability under stress 3. Verified manageability under stress
4. Planning router resources for growth and scale 4. Planning router resources for growth and scale
2. Existing definitions 2. Existing definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
skipping to change at page 3, line 32 skipping to change at page 3, line 32
3. Test Setup 3. Test Setup
3.1 Test Topologies 3.1 Test Topologies
Figure 1 shows the physical configuration to be used for the Figure 1 shows the physical configuration to be used for the
methodologies provided in this document. The number of interfaces methodologies provided in this document. The number of interfaces
between the tester and DUT will scale depending upon the number of between the tester and DUT will scale depending upon the number of
control protocol sessions and traffic forwarding interfaces. A control protocol sessions and traffic forwarding interfaces. A
separate device may be required to externally manage the device in separate device may be required to externally manage the device in
the case that the test equipment does not support such the case that the test equipment does not support such
functionality. Figure 2 shows the logical configuration for the functionality. Figure 2 shows the logical configuration for the
stress test methodologies. Each plane may be emulated by single or stress test methodologies. Each plane MAY be emulated by single or
multiple test equipment. multiple test equipment.
3.2 Test Considerations 3.2 Test Considerations
The Accelerated Stress Benchmarking test can be applied in The Accelerated Stress Benchmarking test can be applied in
service provider test environments to benchmark DUTs under service provider test environments to benchmark DUTs under
stress in an environment that is reflective of an operational stress in an environment that reflects conditions found in
network. A particular Configuration Set is defined and the an operational network. A particular Configuration Set is
DUT is benchmarked using this configuration set and the defined and the DUT is benchmarked using this configuration
Instability Conditions. Varying Configuration Sets and/or set and the Instability Conditions. Varying Configuration
Instability Conditions applied in an iterative fashion can Sets and/or Instability Conditions applied in an iterative
provide an accurate characterization of the DUT fashion can provide an accurate characterization of the DUT
to help determine future network deployments. to help determine future network deployments.
Stress Benchmarking For the management plane SNMP Gets SHOULD be performed
continuously. Management sessions SHOULD be open
simultaneously and be repeatedly open and closed using
access protocols such as telnet and SSH. Open management
sessions SHOULD have valid and invalid configuration and
show commands entered. For the security plane, tunnels
for protocols such as IPsec SHOULD be established and
flapped. Policies for Firewalls and ACLs SHOULD be
repeatedly added and removed via management sessions.
for Accelerated Stress Benchmarking
___________ ___________
| DUT | | DUT |
___|Management | ___|Management |
| | | | | |
| ----------- | -----------
\/ \/
___________ ___________
| | | |
| DUT | | DUT |
skipping to change at page 4, line 48 skipping to change at page 4, line 48
| | | |
| ___________ | | ___________ |
| | Data | | | | Data | |
|--->| Plane |<---| |--->| Plane |<---|
| | | |
----------- -----------
Figure 2. Logical Configuration Figure 2. Logical Configuration
3.3 Reporting Format 3.3 Reporting Format
Each methodology requires reporting of information for test Each methodology requires reporting of information for test
repeatability when benchmarking the same or different devices. repeatability when benchmarking the same or different devices.
The information that are the Configuration Sets, Instability The information that are the Configuration Sets, Instability
Conditions, and Benchmarks, as defined in [4]. Example Conditions, and Benchmarks, as defined in [4]. Example
reporting formats for each are provided below. reporting formats for each are provided below. Benchmarks
MUST be reported as provided below.
Stress Benchmarking for Accelerated Stress Benchmarking
3.3.1 Configuration Sets 3.3.1 Configuration Sets
Configuration Sets may include and are not limited to the following The minimum Configuration Set that MUST be used is as follows:
examples. PARAMETER UNITS
Number of IGP Adjacencies Adjacencies
Number of IGP Routes Routes
Number of Nodes per Area Nodes
Number of Areas per Node Areas
SNMP GET Rate SNMP Gets/minute
Telnet Establishment Rate Sessions/Hour
Concurrent Telnet Sessions Sessions
FTP Establishment Rate Sessions/Hour
Concurrent FTP Session Sessions
SSH Establishment Rate Sessions/Hour
Concurrent SSH sessions Sessions
DATA TRAFFIC
Traffic Forwarding Enabled/Disabled
Aggregate Offered Load bps (or pps)
Number of Ingress Interfaces interfaces
Number of Egress Interfaces interfaces
Packet Size(s) bytes
Offered Load (interface) array of bps
Number of Flows flows
Encapsulation(flow) array of encapsulation types
Configuration Sets MAY include and are not limited to the
following examples.
Example Routing Protocol Configuration Set- Example Routing Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
BGP Enabled/Disabled BGP Enabled/Disabled
Number of EBGP Peers Peers Number of EBGP Peers Peers
Number of IBGP Peers Peers Number of IBGP Peers Peers
Number of BGP Route Instances Routes Number of BGP Route Instances Routes
Number of BGP Installed Routes Routes Number of BGP Installed Routes Routes
MBGP Enabled/Disabled MBGP Enabled/Disabled
Number of MBGP Route Instances Routes Number of MBGP Route Instances Routes
Number of MBGP Installed Routes Routes Number of MBGP Installed Routes Routes
IGP Enabled/Disabled IGP Enabled/Disabled
IGP-TE Enabled/Disabled IGP-TE Enabled/Disabled
Number of IGP Adjacencies Adjacencies Number of IGP Adjacencies Adjacencies
Number of IGP Routes Routes Number of IGP Routes Routes
Number of Nodes per Area Nodes Number of Nodes per Area Nodes
Number of Areas per Node Areas
Example MPLS Protocol Configuration Set- Example MPLS Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
MPLS-TE Enabled/Disabled MPLS-TE Enabled/Disabled
Number of Tunnels as Ingress Tunnels Number of Tunnels as Ingress Tunnels
Number of Tunnels as Mid-Point Tunnels Number of Tunnels as Mid-Point Tunnels
Number of Tunnels as Egress Tunnels Number of Tunnels as Egress Tunnels
LDP Enabled/Disabled LDP Enabled/Disabled
Number of Sessions Sessions Number of Sessions Sessions
Number of FECs FECs Number of FECs FECs
for Accelerated Stress Benchmarking
Example Multicast Protocol Configuration Set- Example Multicast Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
PIM-SM Enabled/Disabled PIM-SM Enabled/Disabled
RP Enabled/Disabled RP Enabled/Disabled
Number of Multicast Groups Groups Number of Multicast Groups Groups
MSDP Enabled/Disabled MSDP Enabled/Disabled
Example Data Plane Configuration Set- Example Data Plane Configuration Set-
PARAMETER UNITS PARAMETER UNITS
Traffic Forwarding Enabled/Disabled Traffic Forwarding Enabled/Disabled
Aggregate Offered Load bps (or pps) Aggregate Offered Load bps (or pps)
Number of Ingress Interfaces number Number of Ingress Interfaces interfaces
Number of Egress Interfaces number Number of Egress Interfaces interfaces
TRAFFIC PROFILE TRAFFIC PROFILE
Packet Size(s) bytes Packet Size(s) bytes
Offered Load (interface) array of bps Offered Load (interface) array of bps
Number of Flows number Number of Flows flows
Encapsulation(flow) array of encapsulation type Encapsulation(flow) array of encapsulation type
Stress Benchmarking
Management Configuration Set- Example Management Configuration Set-
PARAMETER UNITS PARAMETER UNITS
SNMP GET Rate SNMP Gets/minute SNMP GET Rate SNMP Gets/minute
Logging Enabled/Disabled Logging Enabled/Disabled
Protocol Debug Enabled/Disabled Protocol Debug Enabled/Disabled
Telnet Rate Sessions/Hour Telnet Establishment Rate Sessions/Hour
FTP Rate Sessions/Hour
Concurrent Telnet Sessions Sessions Concurrent Telnet Sessions Sessions
FTP Establishment Rate Sessions/Hour
Concurrent FTP Session Sessions Concurrent FTP Session Sessions
SSH Establishment Rate Sessions/Hour
Concurrent SSH sessions Sessions
Packet Statistics Collector Enabled/Disabled Packet Statistics Collector Enabled/Disabled
Statistics Sampling Rate X:1 packets Statistics Sampling Rate X:1 packets
Security Configuration Set - Example Security Configuration Set -
PARAMETER UNITS PARAMETER UNITS
Packet Filters Enabled/Disabled Packet Filters Enabled/Disabled
Number of Filters For-Me number Number of Filters For-Me filters
Number of Filter Rules For-Me number Number of Filter Rules For-Me rules
Number of Traffic Filters number Number of Traffic Filters filters
Number of Traffic Filter Rules number Number of Traffic Filter Rules rules
IPsec tunnels number IPsec tunnels tunnels
SSH Enabled/Disabled
Number of simultaneous SSH sessions number
RADIUS Enabled/Disabled RADIUS Enabled/Disabled
TACACS Enabled/Disabled TACACS Enabled/Disabled
Example SIP Configuration Set -
PARAMETER UNITS
Session Rate Sessions per Second
Media Streams per Session Streams per session
Total Sessions Sessions
for Accelerated Stress Benchmarking
3.3.2 Startup Conditions 3.3.2 Startup Conditions
Startup Conditions may include and are not limited to the following Startup Conditions MAY include and are not limited to the following
examples: examples:
PARAMETER UNITS PARAMETER UNITS
EBGP peering sessions negotiated Total EBGP Sessions EBGP peering sessions negotiated Total EBGP Sessions
IBGP peering sessions negotiated Total IBGP Sessions IBGP peering sessions negotiated Total IBGP Sessions
ISIS adjacencies established Total ISIS Adjacencies ISIS adjacencies established Total ISIS Adjacencies
ISIS routes learned rate ISIS Routes per Second ISIS routes learned rate ISIS Routes per Second
IPsec tunnels negotiated Total IPsec Tunnels IPsec tunnels negotiated Total IPsec Tunnels
IPsec tunnel establishment rate IPsec tunnels per second IPsec tunnel establishment rate IPsec tunnels per second
3.3.3 Instability Conditions 3.3.3 Instability Conditions
Instability Conditions may include and are not limited to the Instability Conditions MAY include and are not limited to the
following examples: following examples:
PARAMETER UNITS PARAMETER UNITS
Interface Shutdown Cycling Rate interfaces per minute Interface Shutdown Cycling Rate interfaces per minute
ISIS Route Flap Rate routes per minutes ISIS Route Flap Rate routes per minutes
LSP Reroute Rate LSP per minute LSP Reroute Rate LSP per minute
Overloaded Links number Overloaded Links number
Amount Links Overloaded % of bandwidth Amount Links Overloaded % of bandwidth
FTP Rate Mb/minute FTP Rate Mb/minute
IPsec Tunnel Flap Rate tunnels per minute IPsec Tunnel Flap Rate tunnels per minute
Filter Policy Changes policies per hour Filter Policy Changes policies per hour
SSH Session Rate SSH sessions per hour SSH Session Rate SSH sessions per hour
Telnet Session Rate Telnet session per hour Telnet Session Rate Telnet session per hour
Command Entry Rate Commands per Hour Command Entry Rate Commands per Hour
Message Flood Rate Messages per second Message Flood Rate Messages
Stress Benchmarking
3.3.4 Benchmarks 3.3.4 Benchmarks
Benchmarks are as defined in [1] and listed as follow: Benchmarks are as defined in [4] and MUST be reported as follow:
PARAMETER UNITS PHASE PARAMETER UNITS PHASE
Stable Aggregate Forwarding Rate pps Startup Stable Aggregate Forwarding Rate pps Startup
Stable Latency seconds Startup Stable Latency seconds Startup
Stable Session Count sessions Startup Stable Session Count sessions Startup
Unstable Aggregate Forwarding Rate pps Instability Unstable Aggregate Forwarding Rate pps Instability
Degraded Aggregate Forwarding Rate pps Instability Degraded Aggregate Forwarding Rate pps Instability
Ave. Degraded Aggregate Forwarding Rate pps Instability Ave. Degraded Aggregate Forwarding Rate pps Instability
Unstable Latency seconds Instability Unstable Latency seconds Instability
Unstable Uncontrolled Sessions Lost sessions Instability Unstable Uncontrolled Sessions Lost sessions Instability
Recovered Aggregate Forwarding Rate pps Recovery Recovered Aggregate Forwarding Rate pps Recovery
Recovered Latency seconds Recovery Recovered Latency seconds Recovery
Recovery Time seconds Recovery Recovery Time seconds Recovery
Recovered Uncontrolled Sessions sessions Recovery Recovered Uncontrolled Sessions sessions Recovery
for Accelerated Stress Benchmarking
4. Stress Test Procedure
4.1 General Methodology with Multiple Instability Conditions
Objective
To benchmark the DUT under accelerated stress when there are
multiple instability conditions.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report Stability Benchmarks
5. Apply Instability Conditions
6. Apply Instability Condition specific to test case.
7. Report Instability Benchmarks
8. Stop applying all Instability Conditions
9. Report Recovery Benchmarks
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Expected Results
Ideally the Forwarding Rates, Latencies, and Session Counts will
be neasured to be the same at each phase. If no packet or
session loss occurs then the Instability Conditions MAY be
increased for a repeated iteration (step 10 of the procedure).
Example Procedure
4. Example Test Case Procedure
1. Report Configuration Set 1. Report Configuration Set
BGP Enabled BGP Enabled
10 EBGP Peers 10 EBGP Peers
30 IBGP Peers 30 IBGP Peers
500K BGP Route Instances 500K BGP Route Instances
160K BGP FIB Routes 160K BGP FIB Routes
ISIS Enabled ISIS Enabled
ISIS-TE Disabled ISIS-TE Disabled
skipping to change at page 7, line 45 skipping to change at page 9, line 4
10K ISIS Level-1 Routes 10K ISIS Level-1 Routes
250 ISIS Nodes per Area 250 ISIS Nodes per Area
MPLS Disabled MPLS Disabled
IP Multicast Disabled IP Multicast Disabled
IPsec Enabled IPsec Enabled
10K IPsec tunnels 10K IPsec tunnels
640 Firewall Policies 640 Firewall Policies
100 Firewall Rules per Policy 100 Firewall Rules per Policy
for Accelerated Stress Benchmarking
Traffic Forwarding Enabled Traffic Forwarding Enabled
Aggregate Offered Load 10Gbps Aggregate Offered Load 10Gbps
30 Ingress Interfaces 30 Ingress Interfaces
30 Egress Interfaces 30 Egress Interfaces
Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes
Forwarding Rate[1..30] = 1Gbps Forwarding Rate[1..30] = 1Gbps
10000 Flows 10000 Flows
Encapsulation[1..5000] = IPv4 Encapsulation[1..5000] = IPv4
Encapsulation[5001.10000] = IPsec Encapsulation[5001.10000] = IPsec
Stress Benchmarking
Logging Enabled Logging Enabled
Protocol Debug Disabled Protocol Debug Disabled
SNMP Enabled SNMP Enabled
SSH Enabled SSH Enabled
10 Concurrent SSH Sessions 10 Concurrent SSH Sessions
FTP Enabled FTP Enabled
RADIUS Enabled RADIUS Enabled
TACACS Disabled TACACS Disabled
Packet Statistics Collector Enabled Packet Statistics Collector Enabled
skipping to change at page 8, line 38 skipping to change at page 9, line 47
Stable Aggregate Forwarding Rate Stable Aggregate Forwarding Rate
Stable Latency Stable Latency
Stable Session Count Stable Session Count
It is RECOMMENDED that the benchmarks be measured and It is RECOMMENDED that the benchmarks be measured and
recorded at one-second intervals. recorded at one-second intervals.
5. Apply Instability Conditions 5. Apply Instability Conditions
Interface Shutdown Cycling Rate = 1 interface every 5 minutes Interface Shutdown Cycling Rate = 1 interface every 5
minutes
BGP Session Flap Rate = 1 session every 10 minutes BGP Session Flap Rate = 1 session every 10 minutes
BGP Route Flap Rate = 100 routes per minute BGP Route Flap Rate = 100 routes per minute
ISIS Route Flap Rate = 100 routes per minute ISIS Route Flap Rate = 100 routes per minute
IPsec Tunnel Flap Rate = 1 tunnel per minute IPsec Tunnel Flap Rate = 1 tunnel per minute
Overloaded Links = 5 of 30 Overloaded Links = 5 of 30
Amount Links Overloaded = 20% Amount Links Overloaded = 20%
SNMP GETs = 1 per sec SNMP GETs = 1 per sec
SSH Session Rate = 6 sessions per hour SSH Session Rate = 6 sessions per hour
SSH Session Duration = 10 minutes SSH Session Duration = 10 minutes
Command Rate via SSH = 20 commands per minute Command Rate via SSH = 20 commands per minute
for Accelerated Stress Benchmarking
FTP Restart Rate = 10 continuous transfers (Puts/Gets) FTP Restart Rate = 10 continuous transfers (Puts/Gets)
per hour per hour
FTP Transfer Rate = 100 Mbps FTP Transfer Rate = 100 Mbps
Statistics Sampling Rate = 1:1 packets Statistics Sampling Rate = 1:1 packets
RADIUS Server Loss Rate = 1 per Hour RADIUS Server Loss Rate = 1 per Hour
RADIUS Server Loss Duration = 3 seconds RADIUS Server Loss Duration = 3 seconds
6. Apply Instability Condition specific to test case. 6. Apply Instability Condition specific to test case.
Stress Benchmarking
7. Report Instability Benchmarks as follow: 7. Report Instability Benchmarks as follow:
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
Degraded Aggregate Forwarding Rate Degraded Aggregate Forwarding Rate
Ave. Degraded Aggregate Forwarding Rate Ave. Degraded Aggregate Forwarding Rate
Unstable Latency Unstable Latency
Unstable Uncontrolled Sessions Lost Unstable Uncontrolled Sessions Lost
It is RECOMMENDED that the benchmarks be measured and It is RECOMMENDED that the benchmarks be measured and
recorded at one-second intervals. recorded at one-second intervals.
skipping to change at page 9, line 32 skipping to change at page 10, line 40
Recovered Latency Recovered Latency
Recovery Time Recovery Time
Recovered Uncontrolled Sessions Lost Recovered Uncontrolled Sessions Lost
It is RECOMMENDED that the benchmarks be measured and It is RECOMMENDED that the benchmarks be measured and
recorded at one-second intervals. recorded at one-second intervals.
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
4.2 General Methodology with a Single Instability Condition
Objective
To benchmark the DUT under accelerated stress when there is a
single instability conditions.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report Stability Benchmarks
5. Apply single Instability Condition
6. Report Instability Benchmarks
7. Stop applying all Instability Condition
8. Report Recovery Benchmarks
9. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
for Accelerated Stress Benchmarking
Expected Results
Ideally the Forwarding Rates, Latencies, and Session Counts will
be neasured to be the same at each phase. If no packet or session
loss occurs then the Instability Conditions MAY be increased
for a repeated iteration (step 10 of the procedure).
5. IANA Considerations 5. IANA Considerations
This document requires no IANA considerations. This document requires no IANA considerations.
6. Security Considerations 6. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
7. Normative References 7. Normative References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, October 1991. Interconnection Devices", RFC 1242, October 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, October 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Terminology for Accelerated [4] Poretsky, S. and Rao, S., "Terminology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-09, Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-10,
work in progress, June 2006. work in progress, October 2006.
Stress Benchmarking
[5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane
Route Convergence",
draft-ietf-bmwg-igp-dataplane-conv-term-11, work in
progress, June 2006.
[6] Bradner, S., "Key words for use in RFCs to Indicate [5] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
8. Informative References 8. Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", [NANOG25] "Core Router Evaluation for Higher Availability",
Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Scott Poretsky, NANOG 25, October 8, 2002, Toronto, CA.
[IEEECQR] "Router Stress Testing to Validate Readiness for [IEEECQR] "Router Stress Testing to Validate Readiness for
Network Deployment", Scott Poretsky, IEEE CQR 2003. Network Deployment", Scott Poretsky, IEEE CQR 2003.
[CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data
Plane Route Convergence", Plane Route Convergence",
draft-ietf-bmwg-igp-dataplane-conv-meth-11, work in draft-ietf-bmwg-igp-dataplane-conv-meth-11, work in
progress, June 2006. progress, October 2006.
for Accelerated Stress Benchmarking
9. Author's Address 9. Author's Address
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@reefpoint.com EMail: sporetsky@reefpoint.com
Shankar Rao Shankar Rao
1801 California Street 1801 California Street
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 Denver, CO 80202
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Stress Benchmarking for Accelerated Stress Benchmarking
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2006). Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided on an This document and the information contained herein are provided on an
 End of changes. 47 change blocks. 
84 lines changed or deleted 179 lines changed or added

This html diff was produced by rfcdiff 1.33. The latest version is available from http://tools.ietf.org/tools/rfcdiff/