draft-ietf-bmwg-acc-bench-meth-02.txt   draft-ietf-bmwg-acc-bench-meth-03.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: October 2005 Expires in: January 2006
Scott Poretsky Scott Poretsky
Quarry Technologies Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
February 2005 July 2005
Methodology for Accelerated Stress Benchmarking Methodology Guidelines for
<draft-ietf-bmwg-acc-bench-meth-02.txt> Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-meth-03.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, I certify that any applicable By submitting this Internet-Draft, each author represents that any
patent or other IPR claims of which I am aware have been disclosed, or applicable patent or other IPR claims of which he or she is aware
will be disclosed, and any of which I become aware will be disclosed, have been or will be disclosed, and any of which he or she becomes
in accordance with RFC 3668. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 41 skipping to change at page 1, line 41
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in as reference material or to cite them other than as "work in
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
ABSTRACT ABSTRACT
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured with
multiple protocols and security policies while forwarding traffic and multiple protocols and security policies while forwarding traffic and
being managed. To accurately benchmark a router for deployment it is being managed. To accurately benchmark a router for deployment it is
necessary that the router be tested in these simultaneous necessary that the router be tested in these simultaneous
operational conditions, which is known as Stress Testing. This operational conditions, which is known as Stress Testing. This
document provides the Methodology for performing Stress Benchmarking document provides the Methodology Guidelines for performing Stress
of networking devices. Descriptions of Test Topology, Benchmarks and Benchmarking of networking devices. Descriptions of Test Topology,
Reporting Format are provided in addition to procedures for Benchmarks and Reporting Format are provided in addition to procedures
conducting various test cases. The methodology is to be used with for conducting various test cases. The methodology is to be used with
the companion terminology document [4]. the companion terminology document [4]. These guidelines can be used
as the basis for additional methodology documents that benchmark
specific network technologies under accelerated stress.
Table of Contents Table of Contents
1. Introduction ............................................... 2 1. Introduction ............................................... 2
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Test Setup.................................................. 3 3. Test Setup.................................................. 3
3.1 Test Topologies............................................ 3 3.1 Test Topologies............................................ 3
3.2 Test Considerations........................................ 3 3.2 Test Considerations........................................ 3
3.3 Reporting Format........................................... 4 3.3 Reporting Format........................................... 4
3.3.1 Configuration Sets....................................... 5 3.3.1 Configuration Sets....................................... 5
3.3.2 Instability Conditions................................... 6 3.3.2 Startup Conditions....................................... 6
3.3.3 Benchmarks............................................... 6 3.3.3 Instability Conditions................................... 6
4. Test Cases.................................................. 7 3.3.4 Benchmarks............................................... 7
4.1 Failed Primary EBGP Peer................................... 7 4. Example Test Case Procedure................................. 7
4.2 Establish New EBGP Peer.................................... 8 5. Security Considerations..................................... 9
4.3 BGP Route Explosion........................................ 8 6. Normative References........................................ 9
4.4 BGP Policy Configuration................................... 9 7. Informative References......................................10
4.5 Persistent BGP Flapping.................................... 9 8. Author's Address............................................10
4.6 BGP Route Flap Dampening...................................10
4.7 Nested Convergence Events..................................11
4.8 Restart Under Load.........................................12
4.9 Destination Control Processor..............................12
4.10 Destination Control Processor with Rate-Limiting..........13
4.11 Destination Interfaces....................................13
4.12 DoS Attack................................................14
5. Security Considerations.....................................14
6. Normative References........................................14
7. Normative References........................................15
8. Author's Address............................................15
1. Introduction 1. Introduction
Router testing benchmarks have consistently been made in a Router testing benchmarks have consistently been made in a monolithic
monolithic fashion wherein a single protocol or behavior is fashion wherein a single protocol or behavior is measured in an
measured in an isolated environment. It is important to know the isolated environment. It is important to know the limits for a
limits for a networking device's behavior for each protocol in isolation, networking device's behavior for each protocol in isolation, however
however this does not produce a reliable benchmark of the device's this does not produce a reliable benchmark of the device's behavior
behavior in an operational network. in an operational network.
Routers in an operational network are simultaneously configured with Routers in an operational network are simultaneously configured with
multiple protocols and security policies while forwarding traffic multiple protocols and security policies while forwarding traffic
and being managed. To accurately benchmark a router for deployment and being managed. To accurately benchmark a router for deployment
it is necessary to test that router in operational conditions by it is necessary to test that router in operational conditions by
simultaneously configuring and scaling network protocols and security simultaneously configuring and scaling network protocols and security
policies, forwarding traffic, and managing the device. It is helpful policies, forwarding traffic, and managing the device. It is helpful
to accelerate these network operational conditions with Instability to accelerate these network operational conditions with Instability
Conditions [4] so that the networking devices are stress tested. Conditions [4] so that the networking devices are stress tested.
skipping to change at page 3, line 26 skipping to change at page 3, line 14
These benefits produce significant advantages for network operations: These benefits produce significant advantages for network operations:
1. Increased stability of routers and protocols 1. Increased stability of routers and protocols
2. Hardened routers to DoS attacks 2. Hardened routers to DoS attacks
3. Verified manageability under stress 3. Verified manageability under stress
4. Planning router resources for growth and scale 4. Planning router resources for growth and scale
2. Existing definitions 2. Existing definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 document are to be interpreted as described in BCP 14, RFC 2119
[Br97]. RFC 2119 defines the use of these key words to help make the [6]. RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this intent of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track document uses these keywords, this document is not a standards track
document. document.
Terms related to Accelerated Stress Benchmarking are defined in [4]. Terms related to Accelerated Stress Benchmarking are defined in [4].
3. Test Setup 3. Test Setup
3.1 Test Topologies 3.1 Test Topologies
Figure 1 shows the physical configuration to be used for the Figure 1 shows the physical configuration to be used for the
methodologies provided in this document. The number of interfaces methodologies provided in this document. The number of interfaces
skipping to change at page 5, line 6 skipping to change at page 5, line 6
3.3 Reporting Format 3.3 Reporting Format
Each methodology requires reporting of information for test Each methodology requires reporting of information for test
repeatability when benchmarking the same or different devices. repeatability when benchmarking the same or different devices.
The information that are the Configuration Sets, Instability The information that are the Configuration Sets, Instability
Conditions, and Benchmarks, as defined in [4]. Example Conditions, and Benchmarks, as defined in [4]. Example
reporting formats for each are provided below. reporting formats for each are provided below.
3.3.1 Configuration Sets 3.3.1 Configuration Sets
Configuration Sets may include and are not limited to the following
examples.
Example Routing Protocol Configuration Set- Example Routing Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
BGP Enabled/Disabled BGP Enabled/Disabled
Number of EBGP Peers Peers Number of EBGP Peers Peers
Number of IBGP Peers Peers Number of IBGP Peers Peers
Number of BGP Route Instances Routes Number of BGP Route Instances Routes
Number of BGP Installed Routes Routes Number of BGP Installed Routes Routes
MBGP Enabled/Disabled MBGP Enabled/Disabled
Number of MBGP Route Instances Routes Number of MBGP Route Instances Routes
Number of MBGP Installed Routes Routes Number of MBGP Installed Routes Routes
IGP Enabled/Disabled IGP Enabled/Disabled
IGP-TE Enabled/Disabled IGP-TE Enabled/Disabled
Number of IGP Adjacencies Adjacencies Number of IGP Adjacencies Adjacencies
Number of IGP Routes Routes Number of IGP Routes Routes
Number of Nodes per Area Nodes Number of Nodes per Area Nodes
Example MPLS Protocol Configuration Set- Example MPLS Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
MPLS-TE MPLS-TE Enabled/Disabled
Number of Ingress Tunnels Tunnels Number of Ingress Tunnels Tunnels
Number of Mid-Point Tunnels Tunnels Number of Mid-Point Tunnels Tunnels
Number of Egress Tunnels Tunnels Number of Egress Tunnels Tunnels
LDP Enabled/Disabled
LDP
Number of Sessions Sessions Number of Sessions Sessions
Number of FECs FECs Number of FECs FECs
Example Multicast Protocol Configuration Set- Example Multicast Protocol Configuration Set-
PARAMETER UNITS PARAMETER UNITS
PIM-SM Enabled/Disabled PIM-SM Enabled/Disabled
RP Enabled/Disabled RP Enabled/Disabled
Number of Multicast Groups Groups Number of Multicast Groups Groups
MSDP Enabled/Disabled MSDP Enabled/Disabled
Example Data Plane Configuration Set- Example Data Plane Configuration Set-
PARAMETER UNITS PARAMETER UNITS
Traffic Forwarding Enabled/Disabled Traffic Forwarding Enabled/Disabled
Aggregate Offered Load bps (or pps) Aggregate Offered Load bps (or pps)
Number of Ingress Interfaces number Number of Ingress Interfaces number
Number of Egress Interfaces number Number of Egress Interfaces number
TRAFFIC PROFILE TRAFFIC PROFILE
Packet Size(s) bytes Packet Size(s) bytes
Packet Rate(interface) array of packets per second Offered Load (interface) array of bps
Number of Flows number Number of Flows number
Encapsulation(flow) array of encapsulation type Encapsulation(flow) array of encapsulation type
Management Configuration Set- Management Configuration Set-
PARAMETER UNITS PARAMETER UNITS
SNMP GET Rate SNMP Gets/minute SNMP GET Rate SNMP Gets/minute
Logging Enabled/Disabled Logging Enabled/Disabled
Protocol Debug Enabled/Disabled Protocol Debug Enabled/Disabled
Telnet Rate Sessions/Hour Telnet Rate Sessions/Hour
FTP Rate Sessions/Hour FTP Rate Sessions/Hour
Concurrent Telnet Sessions Sessions Concurrent Telnet Sessions Sessions
skipping to change at page 6, line 23 skipping to change at page 6, line 23
Packet Statistics Collector Enabled/Disabled Packet Statistics Collector Enabled/Disabled
Statistics Sampling Rate X:1 packets Statistics Sampling Rate X:1 packets
Security Configuration Set - Security Configuration Set -
PARAMETER UNITS PARAMETER UNITS
Packet Filters Enabled/Disabled Packet Filters Enabled/Disabled
Number of Filters For-Me number Number of Filters For-Me number
Number of Filter Rules For-Me number Number of Filter Rules For-Me number
Number of Traffic Filters number Number of Traffic Filters number
Number of Traffic Filter Rules number Number of Traffic Filter Rules number
IPsec tunnels number
SSH Enabled/Disabled SSH Enabled/Disabled
Number of simultaneous SSH sessions number Number of simultaneous SSH sessions number
RADIUS Enabled/Disabled RADIUS Enabled/Disabled
TACACS Enabled/Disabled TACACS Enabled/Disabled
3.3.2 Instability Conditions 3.3.2 Startup Conditions
Startup Conditions may include and are not limited to the following
examples:
PARAMETER UNITS
EBGP peering sessions negotiated Total EBGP Sessions
IBGP peering sessions negotiated Total IBGP Sessions
BGP routes learned rate BGP Routes per Second
ISIS adjacencies established Total ISIS Adjacencies
ISIS routes learned rate ISIS Routes per Second
IPsec tunnels negotiated Total IPsec Tunnels
IPsec tunnel establishment rate IPsec tunnels per second
3.3.3 Instability Conditions
Instability Conditions may include and are not limited to the
following examples:
PARAMETER UNITS PARAMETER UNITS
Interface Shutdown Cycling Rate interfaces per minute Interface Shutdown Cycling Rate interfaces per minute
BGP Session Flap Rate sessions per minute BGP Session Flap Rate sessions per minute
BGP Route Flap Rate routes per minutes BGP Route Flap Rate routes per minutes
IGP Route Flap Rate routes per minutes IGP Route Flap Rate routes per minutes
LSP Reroute Rate LSP per minute LSP Reroute Rate LSP per minute
Overloaded Links number Overloaded Links number
Amount Links Overloaded % of bandwidth Amount Links Overloaded % of bandwidth
FTP Rate Mb/minute FTP Rate Mb/minute
IPsec Session Loss sessions per minute IPsec Tunnel Flap Rate tunnels per minute
Filter Policy Changes policies per hour Filter Policy Changes policies per hour
SSH Session Restart SSH sessions per hour SSH Session Restart SSH sessions per hour
Telnet Session Restart Telnet session per hour Telnet Session Restart Telnet session per hour
3.3.4 Benchmarks
Benchmarks are as defined in [1] and listed as follow:
3.3.3 Benchmarks
PARAMETER UNITS PHASE PARAMETER UNITS PHASE
Stable Aggregate Forwarding Rate pps Startup Stable Aggregate Forwarding Rate pps Startup
Stable Latency seconds Startup Stable Latency seconds Startup
Stable Session Count sessions Startup Stable Session Count sessions Startup
Unstable Aggregate Forwarding Rate pps Instability Unstable Aggregate Forwarding Rate pps Instability
Degraded Aggregate Forwarding Rate pps Instability Degraded Aggregate Forwarding Rate pps Instability
Ave. Degraded Aggregate Forwarding Rate pps Instability Ave. Degraded Aggregate Forwarding Rate pps Instability
Unstable Latency seconds Instability Unstable Latency seconds Instability
Unstable Uncontrolled Sessions Lost sessions Instability Unstable Uncontrolled Sessions Lost sessions Instability
Recovered Aggregate Forwarding Rate pps Recovery Recovered Aggregate Forwarding Rate pps Recovery
Recovered Latency seconds Recovery Recovered Latency seconds Recovery
Recovery Time seconds Recovery Recovery Time seconds Recovery
Recovered Uncontrolled Sessions Lost sessions Recovery Recovered Uncontrolled Sessions Lost sessions Recovery
It is RECOMMENDED that Aggregate Forwarding Rates, Latencies, and
Session Losses be measured at one-second intervals. These same
benchmarks can also be used as Variability Benchmarks reported as
the differences between the Benchmarks for multiple iterations
with the same DUT. For the purpose of the Variability Benchmarks,
A more complete characterization of the DUT would be to apply
multiple test iterations for the same Configuration Sets and
Instability Conditions, measure the Variability Benchmarks, and
then vary the Configuration Set and/or Instability Conditions.
4. Test Cases
4.1 Failed Primary EBGP Peer
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when losing an EBGP
Peer from which most FIB routes have been learned.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Remove link to EBGP peer with most FIB routes. This SHOULD
be achieved by losing physical layer connectivity with
a local fiber pull. Loss of the peering session SHOULD
cause the DUT to withdraw 100,000 or greater routes.
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be significant packet loss
until the DUT converges from the lost EBGP link. Other DUT
operation should be stable without session loss or sustained
packet loss. Recovery time should not be infinite.
4.2 Establish New EBGP Peer 4. Example Test Case Procedure
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when establishing a
new EBGP Peer from which routes are learned.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Configure a new EBGP peering session at DUT and peering router.
Physical and Data Link Layer connectivity SHOULD already exist
to perform this step. Establishment of the peering session
MUST result in the DUT learning 100,000 or greater routes from
the BGP peer and advertising 100,000 or greater routes to
the BGP peer
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be zero packet loss as the DUT
learns the new routes. Other DUT operation should be stable
without session loss or sustained packet loss.
4.3 BGP Route Explosion
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when there is BGP Route
Explosion experienced in the network.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Advertise 1M BGP routes to the DUT from a single EBGP
neighbor.
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions, including BGP route
advertisement.
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be no additional packet loss from
the advertisement of routes from the BGP neighbor. Other
DUT operation should be stable without session loss.
4.4 BGP Policy Configuration
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when there is continuous
reconfiguration of BGP Policy at the DUT.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Configure BGP Policy on the DUT for each established neighbor.
The BGP Policy SHOULD filter 25% of the routes learned from
that neighbor. Note that the specific policy configuration
to achieve the filtering may be device specific.
7. Every 30 minutes remove the BGP Policy configuration and then
configure it gain so that it is reapplied.
8. Report benchmarks (for instability)
9. Stop applying all Instability Conditions, including Policy
changes
10. Report benchmarks (for recovery)
11. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be no packet loss resulting from
the continuous configuration and removal of BGP Policy for BGP
neighbors. Other DUT operation should be stable without session
loss.
4.5 Persistent BGP Flapping
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when flapping BGP Peering
sessions for an infinite period.
Procedure
1. Report Configuration Set 1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Repeatedly flap an IBGP and an EBGP peering session.
This SHOULD be achieved by losing physical layer connectivity
via a local interface shutdown/no shutdown every 180 seconds with
a delay of 10 seconds between the shut and no shut.
The loss of the EBGP peering session MUST cause the DUT to withdraw
100,000 or greater routes that are re-learned when the session
re-establishes. Route Flap Dampening SHOULD NOT be enabled.
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions, including flapping
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results BGP Enabled
It is expected that there will be significant packet loss 10 EBGP Peers
from repeated Convergence Events. Other DUT operation should be 30 IBGP Peers
stable without session loss. Recovery time should not be infinite. 500K BGP Route Instances
160K BGP FIB Routes
4.6 BGP Route Flap Dampening
Objective ISIS Enabled
The purpose of this test is to benchmark the performance ISIS-TE Disabled
of the DUT during stress conditions when flapping BGP Peering 30 ISIS Adjacencies
sessions for an infinite period and route flap dampening is 10K ISIS Level-1 Routes
enabled. 250 ISIS Nodes per Area
Procedure MPLS Disabled
1. Report Configuration Set IP Multicast Disabled
2. Configure Route Flap Dampening on the DUT with DEFAULT parameter
values.
3. Begin Startup Conditions with the DUT
4. Establish Configuration Sets with the DUT
5. Report benchmarks (for stability)
6. Apply Instability Conditions
7. Repeatedly flap an IBGP and an EBGP peering session.
This SHOULD be achieved by losing physical layer connectivity
via a local interface shutdown/no shutdown every 180 seconds with
a delay of 10 seconds between the shut and no shut.
The loss of the EBGP peering session MUST cause the DUT to withdraw
100,000 or greater routes that are re-learned when the session
re-establishes.
8. Report benchmarks (for instability)
9. Stop applying all Instability Conditions
10. Report benchmarks (for recovery)
11. Optional - Change Route Flap Dampening parameter values
12. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be significant packet loss
from repeated Convergence Events and flap dampening. Other DUT operation
should be stable without session loss. Recovery time should not be infinite.
4.7 Nested Convergence Events [5] IPsec Enabled
10K IPsec tunnels
640 Firewall Policies
100 Firewall Rules per Policy
Objective Traffic Forwarding Enabled
The purpose of this test is to benchmark the performance Aggregate Offered Load 10Gbps
of the DUT during stress conditions when flapping BGP Peering 30 Ingress Interfaces
sessions causes Nested Convergence Events. 30 Egress Interfaces
Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes
Forwarding Rate[1..30] = 1Gbps
10000 Flows
Encapsulation[1..5000] = IPv4
Encapsulation[5001.10000] = IPsec
Logging Enabled
Protocol Debug Disabled
SNMP Enabled
SSH Enabled
20 Concurrent SSH Sessions
FTP Enabled
RADIUS Enabled
TACACS Disabled
Packet Statistics Collector Enabled
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT 2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Repeatedly flap an IBGP and an EBGP peering session.
This SHOULD be achieved by losing physical layer connectivity
via a local interface shutdown/no shutdown every 10 seconds with
a delay of 1 second between the shut and no shut.
The loss of the EBGP peering session MUST cause the DUT to withdraw
100,000 or greater routes that are re-learned when the session
re-establishes. Route Flap Dampening SHOULD NOT be enabled.
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions, including flapping
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
It is expected that there will be significant packet loss
from Nested Convergence Events. New Other DUT operation should be
stable without session loss. Recovery time should not be infinite.
4.8 Restart Under Load 10 EBGP peering sessions negotiated
30 EBGP peering sessions negotiated
Objective 1K BGP routes learned per second
The purpose of this test is to benchmark the performance of the DUT 30 ISIS Adjacencies
during restart when stress conditions are applied. 1K ISIS routes learned per second
10K IPsec tunnels negotiated
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT 3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Restart DUT. This marks the beginning on the recovery period.
6. Report benchmarks (for recovery)
7. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
NOTE 1: Restart via the DUT's Command Line Interface rather than
power cycle is typically more stressful than power cycle
since hardware can maintain state.
NOTE 2: Instability Conditions are not applied for this test case.
Results 4. Report Stability Benchmarks as follow:
DUT should re-establish all control protocol sessions and have
a Recovery Time [4] that is not infinite.
4.9 Destination Control Processor Stable Aggregate Forwarding Rate
Stable Latency
Stable Session Count
Objective It is RECOMMENDED that the benchmarks be measured and
The purpose of this test is to benchmark the performance recorded at one-second intervals.
of the DUT during stress conditions when traffic is destined for
the Control Processor of the DUT.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Start Configuration Sets with the DUT, except Data Plane
Configuration Set
4. Report benchmarks (for stability)
5. Apply Instability Conditions 5. Apply Instability Conditions
6. Send offered load at maximum forwarding rate of DUT interfaces
to all DUT interfaces. Traffic MUST be configured so that the
offered load has a destination address that is the DUT's central
control processor
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions, including data traffic
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results Interface Shutdown Cycling Rate = 1 interface every 5 minutes
Results will vary with specific vendor implementations. BGP Session Flap Rate = 1 session every 10 minutes
It is possible that significant session loss is observed. BGP Route Flap Rate = 100 routes per minute
ISIS Route Flap Rate = 100 routes per minute
4.10 Destination Control Processor with Rate-Limiting IPsec Tunnel Flap Rate = 1 tunnel per minute
Overloaded Links = 5 of 30
Amount Links Overloaded = 20%
SNMP GETs = 1 per sec
SSH Restart Rate = 10 sessions per hour
FTP Restart Rate = 10 transfers per hour
FTP Transfer Rate = 100 Mbps
Statistics Sampling Rate = 1:1 packets
Objective 6. Apply Instability Condition specific to test case.
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when traffic is destined for
the Control processor of the DUT.
Procedure 7. Report Instability Benchmarks as follow:
1. Report Configuration Set Unstable Aggregate Forwarding Rate
2. Apply policy filter to rate-limit traffic arriving at the Central Degraded Aggregate Forwarding Rate
Processor to be only 1% of the offered load. Ave. Degraded Aggregate Forwarding Rate
3. Begin Startup Conditions with the DUT Unstable Latency
4. Start Configuration Sets with the DUT, except Data Plane Unstable Uncontrolled Sessions Lost
Configuration Set
5. Report benchmarks (for stability)
6. Apply Instability Conditions
7. Send offered load at maximum forwarding rate of DUT interfaces
to all DUT interfaces. Traffic MUST be configured so that the
offered load has a destination address that is the DUT's central
control processor
8. Report benchmarks (for instability)
9. Stop applying all Instability Conditions, including data traffic
10. Report benchmarks (for recovery)
11. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results It is RECOMMENDED that the benchmarks be measured and
Results will vary with specific vendor implementations. There should be recorded at one-second intervals.
no session loss observed.
4.11 Destination Interfaces 8. Stop applying all Instability Conditions
Objective
The purpose of this test is to benchmark the performance
of the DUT during stress conditions when traffic is destined for
the interfaces of the DUT.
Procedure 9. Report Recovery Benchmarks as follow:
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Start Configuration Sets with the DUT, except Data Plane
Configuration Set
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Send offered load at maximum forwarding rate of DUT interfaces
to all DUT interfaces. Traffic MUST be configured so that the
offered load has destination addresses of the interfaces receiving
traffic.
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions, including data traffic
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration
Results
Results will vary with specific vendor implementations.
There should be no session loss observed.
4.12 DoS Attack Recovered Aggregate Forwarding Rate
Recovered Latency
Recovery Time
Recovered Uncontrolled Sessions Lost
Objective It is RECOMMENDED that the benchmarks be measured and
The purpose of this test is to benchmark the performance of the recorded at one-second intervals.
DUT during stress conditions while experiencing a DoS attack.
Procedure
1. Report Configuration Set
2. Begin Startup Conditions with the DUT
3. Establish Configuration Sets with the DUT
4. Report benchmarks (for stability)
5. Apply Instability Conditions
6. Initiate DoS Attack against DUT. It is RECOMMENDED that
the SYN Flood attack be used for the DoS attack.
7. Report benchmarks (for instability)
8. Stop applying all Instability Conditions
9. Report benchmarks (for recovery)
10. Optional - Change Configuration Set and/or Instability 10. Optional - Change Configuration Set and/or Instability
Conditions for next iteration Conditions for next iteration
Results
DUT should be able to defend against DoS attack without additional
packet loss or session loss.
5. Security Considerations 5. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating is not performed on devices or systems connected to operating
networks. networks.
6. Normative References 6. Normative References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, February 1991. Interconnection Devices", RFC 1242, July 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Terminology for Accelerated [4] Poretsky, S. and Rao, S., "Terminology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-05, Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-05,
work in progress, February 2005. work in progress, July 2005.
[5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane [5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane
Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-05, Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-06,
work in progress, February 2005. work in progress, July 2005.
[6] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", RFC 2119, March 1997.
7. Informative References 7. Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", Scott [NANOG25] "Core Router Evaluation for Higher Availability", Scott
Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
[IEEECQR] "Router Stress Testing to Validate Readiness for Network [IEEECQR] "Router Stress Testing to Validate Readiness for Network
Deployment", Scott Poretsky, IEEE CQR 2003. Deployment", Scott Poretsky, IEEE CQR 2003.
[CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data Plane [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data Plane
Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05, Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05,
work in progress, February 2005. work in progress, July 2005.
8. Author's Address 8. Author's Address
Scott Poretsky Scott Poretsky
Quarry Technologies Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@quarrytech.com EMail: sporetsky@reefpoint.com
Shankar Rao Shankar Rao
950 17th Street 1801 California Street
Suite 1900 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80210 Denver, CO 80202
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Intellectual Property Statement Full Copyright Statement
The IETF takes no position regarding the validity or scope of any Intel- Copyright (C) The Internet Society (2005).
lectual Property Rights or other rights that might be claimed to pertain
to the implementation or use of the technology described in this docu- This document is subject to the rights, licenses and restrictions
ment or the extent to which any license under such rights might or might contained in BCP 78, and except as set forth therein, the authors
not be available; nor does it represent that it has made any independent retain all their rights.
effort to identify any such rights. Information on the procedures with
respect to rights in RFC documents can be found in BCP 78 and BCP 79. This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an attempt assurances of licenses to be made available, or the result of an
made to obtain a general license or permission for the use of such attempt made to obtain a general license or permission for the use of
proprietary rights by implementers or users of this specification can be such proprietary rights by implementers or users of this
obtained from the IETF on-line IPR repository at specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr. http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights copyrights, patents or patent applications, or other proprietary
that may cover technology that may be required to implement this stan- rights that may cover technology that may be required to implement
dard. Please address the information to the IETF at ietf-ipr@ietf.org. this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Disclaimer of Warranty
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR
IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMA-
TION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement Acknowledgement
Copyright (C) The Internet Society (2005). This document is subject to Funding for the RFC Editor function is currently provided by the
the rights, licenses and restrictions contained in BCP 78, and except as Internet Society.
set forth therein, the authors retain all their rights.
 End of changes. 

This html diff was produced by rfcdiff 1.25, available from http://www.levkowetz.com/ietf/tools/rfcdiff/