draft-ietf-bmwg-acc-bench-meth-00.txt   draft-ietf-bmwg-acc-bench-meth-01.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: January 2005 Expires in: April 2005
Scott Poretsky Scott Poretsky
Quarry Technologies Quarry Technologies
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
July 2004 October 2004
Methodology for Accelerated Stress Benchmarking Methodology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-meth-00.txt> <draft-ietf-bmwg-acc-bench-meth-01.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, I certify that any applicable By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed, or patent or other IPR claims of which I am aware have been disclosed, or
will be disclosed, and any of which I become aware will be disclosed, will be disclosed, and any of which I become aware will be disclosed,
in accordance with RFC 3668. in accordance with RFC 3668.
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
skipping to change at page 2, line 17 skipping to change at page 2, line 17
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Test Setup.................................................. 3 3. Test Setup.................................................. 3
3.1 Test Topologies............................................ 3 3.1 Test Topologies............................................ 3
3.2 Test Considerations........................................ 4 3.2 Test Considerations........................................ 4
3.3 Reporting Format........................................... 4 3.3 Reporting Format........................................... 4
3.3.1 Configuration Sets....................................... 4 3.3.1 Configuration Sets....................................... 4
3.3.2 Instability Conditions................................... 6 3.3.2 Instability Conditions................................... 6
3.3.3 Benchmarks............................................... 6 3.3.3 Benchmarks............................................... 6
4. Test Cases.................................................. 7 4. Test Cases.................................................. 7
4.1 Failed Primary EBGP Peer................................... 7 4.1 Failed Primary EBGP Peer................................... 7
4.2 BGP Route Explosion........................................ 7 4.2 Establish New EBGP Peer.................................... 7
4.3 Persistent BGP Flapping.................................... 8 4.3 BGP Route Explosion........................................ 8
4.4 DoS Attack................................................. 8 4.4 BGP Policy Configuration................................... 8
5. Security Considerations..................................... 9 4.5 Persistent BGP Flapping.................................... 9
6. References.................................................. 9 4.6 DoS Attack................................................. 9
7. Author's Address............................................ 9 5. Security Considerations..................................... 10
6. References.................................................. 10
7. Author's Address............................................ 11
1. Introduction 1. Introduction
Router testing benchmarks have consistently been made in a Router testing benchmarks have consistently been made in a
monolithic fashion wherein a single protocol or behavior is monolithic fashion wherein a single protocol or behavior is
measured in an isolated environment. It is important to know the measured in an isolated environment. It is important to know the
limits for a networking device's behavior for each protocol in isolation, limits for a networking device's behavior for each protocol in isolation,
however this does not produce a reliable benchmark of the device's however this does not produce a reliable benchmark of the device's
behavior in an operational network. behavior in an operational network.
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured with
skipping to change at page 4, line 30 skipping to change at page 4, line 30
----------- -----------
Figure 2. Logical Configuration Figure 2. Logical Configuration
3.2 Test Considerations 3.2 Test Considerations
The Accelerated Stress Benchmarking test can be applied in The Accelerated Stress Benchmarking test can be applied in
service provider test environments to benchmark DUTs under service provider test environments to benchmark DUTs under
stress in an environment that is reflective of an operational stress in an environment that is reflective of an operational
network. A particular Configuration Set is defined and the network. A particular Configuration Set is defined and the
DUT is benchmarked using this configuration set and the Instability Conditions. DUT is benchmarked using this configuration set and the
Varying Configuration Sets and/or Instability Conditions applied in an iterative Instability Conditions. Varying Configuration Sets and/or
fashion can provide an accurate characterization of the DUT Instability Conditions applied in an iterative fashion can
provide an accurate characterization of the DUT
to help determine future network deployments. to help determine future network deployments.
3.3 Reporting Format 3.3 Reporting Format
Each methodology requires reporting of information for test Each methodology requires reporting of information for test
repeatability when benchmarking the same or different devices. repeatability when benchmarking the same or different devices.
The information that are the Configuration Sets, Instability The information that are the Configuration Sets, Instability
Conditions, and Benchmarks, as defined in [6]. Example Conditions, and Benchmarks, as defined in [6]. Example
reporting formats for each are provided below. reporting formats for each are provided below.
skipping to change at page 6, line 28 skipping to change at page 6, line 27
PARAMETER UNITS PARAMETER UNITS
Interface Shutdown Cycling Rate interfaces per minute Interface Shutdown Cycling Rate interfaces per minute
BGP Session Flap Rate sessions per minute BGP Session Flap Rate sessions per minute
BGP Route Flap Rate routes per minutes BGP Route Flap Rate routes per minutes
IGP Route Flap Rate routes per minutes IGP Route Flap Rate routes per minutes
LSP Reroute Rate LSP per minute LSP Reroute Rate LSP per minute
Overloaded Links number Overloaded Links number
Amount Links Overloaded % of bandwidth Amount Links Overloaded % of bandwidth
FTP Rate Mb/minute FTP Rate Mb/minute
IPsec Session Loss sessions per minute IPsec Session Loss sessions per minute
Filter Policy Changes policies per minute Filter Policy Changes policies per hour
SSH Session Re-Start SSH sessions per minute SSH Session Restart SSH sessions per hour
Telnet Session Restart Telnet session per hour
3.3.3 Benchmarks 3.3.3 Benchmarks
PARAMETER UNITS PHASE
Stable Aggregate Forwarding Rate pps Startup
Stable Latency seconds Startup
Stable Session Count sessions Startup
Unstable Aggregate Forwarding Rate pps Instability
Degraded Aggregate Forwarding Rate pps Instability
Ave. Degraded Aggregate Forwarding Rate pps Instability
Unstable Latency seconds Instability
Unstable Uncontrolled Sessions Lost sessions Instability
Recovered Aggregate Forwarding Rate pps Recovery
Recovered Latency seconds Recovery
Recovery Time seconds Recovery
Recovered Uncontrolled Sessions Lost sessions Recovery
It is RECOMMENDED that Aggregate Forwarding Rates, Latencies, and
Session Losses be measured at one-second intervals. These same
benchmarks can also be used as Variability Benchmarks reported as
the differences between the Benchmarks for multiple iterations
with the same DUT. For the purpose of the Variability Benchmarks,
A more complete characterization of the DUT would be to apply
multiple test iterations for the same Configuration Sets and
Instability Conditions, measure the Variability Benchmarks, and
then vary the Configuration Set and/or Instability Conditions.
PARAMETER UNITS
Stable Aggregate Forwarding Rate pps
Stable Session Count sessions
Unstable Aggregate Forwarding Rate pps
Degraded Aggregate Forwarding Rate pps
Average Degraded Aggregate Forwarding Rate pps
Unstable Uncontrolled Sessions Lost sessions
Recovered Aggregate Forwarding Rate pps
Recovery Time seconds
Recovered Uncontrolled Sessions Lost sessions
4. Test Cases 4. Test Cases
4.1 Failed Primary EBGP Peer 4.1 Failed Primary EBGP Peer
Objective Objective
The purpose of this test is to benchmark the performance The purpose of this test is to benchmark the performance
of the DUT during stress conditions when losing an EBGP of the DUT during stress conditions when losing an EBGP
Peer from which most FIB routes have been learned. Peer from which most FIB routes have been learned.
Procedure Procedure
1. Report Configuration Set 1. Report Configuration Set
2. Begin Startup Conditions with the DUT 2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT 3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability) 4. Report benchmarks (for stability)
5. Apply Instability Conditions 5. Apply Instability Conditions
6. Remove link to EBGP peer with most FIB routes 6. Remove link to EBGP peer with most FIB routes. This SHOULD
be achieved by losing physical layer connectivity with
a local fiber pull. Loss of the peering session SHOULD
cause the DUT to withdraw 100,000 or greater routes.
7. Report benchmarks (for instability) 7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions 8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery) 9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
Results Results
It is expected that there will be significant packet loss It is expected that there will be significant packet loss
until the DUT converges from the lost EBGP link. Other DUT until the DUT converges from the lost EBGP link. Other DUT
operation should be stable without session loss or sustained operation should be stable without session loss or sustained
packet loss. Recovery time should not be infinite. packet loss. Recovery time should not be infinite.
4.2 BGP Route Explosion 4.2 Establish New EBGP Peer
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when establishing a
new EBGP Peer from which routes are learned.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Configure new EBGP peering session at DUT and peering router.
Physical and Data Link Layer connectivity SHOULD already exist
to perform this step. Establishment of the peering session
SHOULD result in the DUT learning 100,000 or greater routes from
the BGP peer and advertising 100,000 or greater routes to
the BGP peer
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be zero packet loss as the DUT
learns the new routes. Other DUT operation should be stable
without session loss or sustained packet loss.
4.3 BGP Route Explosion
Objective Objective
The purpose of this test is to benchmark the performance The purpose of this test is to benchmark the performance
of the DUT during stress conditions when there is BGP Route of the DUT during stress conditions when there is BGP Route
Explosion experienced in the network. Explosion experienced in the network.
Procedure Procedure
1. Report Configuration Set 1. Report Configuration Set
2. Begin Startup Conditions with the DUT 2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT 3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability) 4. Report benchmarks (for stability)
5. Apply Instability Conditions 5. Apply Instability Conditions
6. Advertise 1M BGP routes from a single EBGP peer. 6. Advertise 1M BGP routes to the DUT from a single EBGP
neighbor.
7. Report benchmarks (for instability) 7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions 8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery) 9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
Results Results
It is expected that there will be no additional packet loss from It is expected that there will be no additional packet loss from
the advertisement of duplicate routes from a single peer. Other the advertisement of routes from the BGP neighbor. Other
DUT operation should be stable without session loss. Recovery DUT operation should be stable without session loss.
time should not be infinite.
4.3 Persistent BGP Flapping 4.4 BGP Policy Configuration
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when there is continuous
reconfiguration of BGP Policy at the DUT.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Configure BGP Policy on the DUT for each established neighbor.
The BGP Policy SHOULD filter 25% of the routes learned from
that neighbor. Note that the specific policy configuration
to achieve the filtering may be device specific.
7. Every 30 minutes remove the BGP Policy configuration and then
configure it gain so that it is reapplied.
8. Report benchmarks (for instability)
9. Stop applying all Instability Conditions
10. Report benchmarks (for recovery)
11. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be no packet loss resulting from
the continuous configuration and removal of BGP Policy for BGP
neighbors. Other DUT operation should be stable without session
loss.
4.5 Persistent BGP Flapping
Objective Objective
The purpose of this test is to benchmark the performance The purpose of this test is to benchmark the performance
of the DUT during stress conditions when flapping BGP Peering of the DUT during stress conditions when flapping BGP Peering
sessions for an infinite period. sessions for an infinite period.
Procedure Procedure
1. Report Configuration Set 1. Report Configuration Set
2. Begin Startup Conditions with the DUT 2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT 3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability) 4. Report benchmarks (for stability)
5. Apply Instability Conditions 5. Apply Instability Conditions
6. Repeatedly flap an IBGP and an EBGP peering session 6. Repeatedly flap an IBGP and an EBGP peering session.
This SHOULD be achieved by losing physical layer connectivity
via a local fiber pull. Loss of the EBGP peering session SHOULD
cause the DUT to withdraw 10,000 or greater routes. Route Flap
Dampening SHOULD NOT be enabled.
7. Report benchmarks (for instability) 7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions 8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery) 9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
Results Results
It is expected that there will be significant packet loss It is expected that there will be significant packet loss
from repeated convergence events. Other DUT operation should be from repeated convergence events. Other DUT operation should be
stable without session loss. Recovery time should not be infinite. stable without session loss. Recovery time should not be infinite.
4.4 DoS Attack 4.6 DoS Attack
Objective Objective
The purpose of this test is to benchmark the performance The purpose of this test is to benchmark the performance of the
of the DUT during stress conditions while experiencing a DUT during stress conditions while experiencing a DoS attack.
DoS attack.
Procedure Procedure
1. Report Configuration Set 1. Report Configuration Set
2. Begin Startup Conditions with the DUT 2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT 3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability) 4. Report benchmarks (for stability)
5. Apply Instability Conditions 5. Apply Instability Conditions
6. Initiate DoS Attack against DUT 6. Initiate DoS Attack against DUT. It is RECOMMENDED that
the SYN Flood attack be used for the DoS attack.
7. Report benchmarks (for instability) 7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions 8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery) 9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
Results Results
DUT should be able to defend against DoS attack without additional DUT should be able to defend against DoS attack without additional
packet loss or session loss. Recovery time should be immediate. packet loss or session loss.
Open issue is definition of DoS Attack for the purpose of this test.
COuld any DoS Attack be used? Should DoS Attack be defined?
5. Security Considerations 5. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
6. References 6. References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991. Interconnection Devices", RFC 1242, October 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] "Core Router Evaluation for Higher Availability", Scott [4] "Core Router Evaluation for Higher Availability", Scott
Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
[5] "Router Stress Testing to Validate Readiness for Network [5] "Router Stress Testing to Validate Readiness for Network
Deployment", Scott Poretsky, IEEE CQR 2003. Deployment", Scott Poretsky, IEEE CQR 2003.
[6] Poretsky, S. and Rao, S., "Terminology for Accelerated [6] Poretsky, S. and Rao, S., "Terminology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-03, Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-04,
work in progress, July 2004. work in progress, October 2004.
7. Author's Address 7. Author's Address
Scott Poretsky Scott Poretsky
Quarry Technologies Quarry Technologies
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/