draft-ietf-bmwg-acc-bench-meth-04.txt   draft-ietf-bmwg-acc-bench-meth-05.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: December 2006
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
October 2005
Methodology Guidelines for Methodology Guidelines for
Accelerated Stress Benchmarking Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-meth-04.txt> <draft-ietf-bmwg-acc-bench-meth-05.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed,
and any of which I become aware will be disclosed, in accordance with
RFC 3668.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as other groups may also distribute working documents as
Internet-Drafts. Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
skipping to change at page 1, line 40 skipping to change at page 1, line 34
other groups may also distribute working documents as other groups may also distribute working documents as
Internet-Drafts. Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2005). All Rights Reserved. Copyright (C) The Internet Society (2006).
ABSTRACT ABSTRACT
Routers in an operational network are simultaneously configured Routers in an operational network are simultaneously configured
With multiple protocols and security policies while forwarding With multiple protocols and security policies while forwarding
traffic and being managed. To accurately benchmark a router for traffic and being managed. To accurately benchmark a router for
deployment it is necessary that the router be tested in these deployment it is necessary that the router be tested in these
simultaneous operational conditions, which is known as Stress simultaneous operational conditions, which is known as Stress
Testing. This document provides the Methodology Guidelines for Testing. This document provides the Methodology Guidelines for
performing Stress Benchmarking of networking devices. performing Acceleratd Stress Benchmarking of networking devices.
Descriptions of Test Topology, Benchmarks and Reporting Format Descriptions of Test Topology, Benchmarks and Reporting Format
are provided in addition to procedures for conducting various are provided in addition to procedures for conducting various
test cases. The methodology is to be used with the companion test cases. The methodology is to be used with the companion
terminology document [4]. These guidelines can be used as the terminology document [4]. These guidelines can be used as the
basis for additional methodology documents that benchmark specific basis for additional methodology documents that benchmark
network technologies under accelerated stress. stress conditions for specific network technologies.
Stress Benchmarking Stress Benchmarking
Table of Contents Table of Contents
1. Introduction ............................................... 2 1. Introduction ............................................... 2
2. Existing definitions ....................................... 3 2. Existing definitions ....................................... 3
3. Test Setup.................................................. 3 3. Test Setup.................................................. 3
3.1 Test Topologies............................................ 3 3.1 Test Topologies............................................ 3
3.2 Test Considerations........................................ 3 3.2 Test Considerations........................................ 3
3.3 Reporting Format........................................... 4 3.3 Reporting Format........................................... 4
3.3.1 Configuration Sets....................................... 5 3.3.1 Configuration Sets....................................... 5
skipping to change at page 6, line 37 skipping to change at page 6, line 37
Number of simultaneous SSH sessions number Number of simultaneous SSH sessions number
RADIUS Enabled/Disabled RADIUS Enabled/Disabled
TACACS Enabled/Disabled TACACS Enabled/Disabled
3.3.2 Startup Conditions 3.3.2 Startup Conditions
Startup Conditions may include and are not limited to the following Startup Conditions may include and are not limited to the following
examples: examples:
PARAMETER UNITS PARAMETER UNITS
EBGP peering sessions negotiated Total EBGP Sessions EBGP peering sessions negotiated Total EBGP Sessions
IBGP peering sessions negotiated Total IBGP Sessions IBGP peering sessions negotiated Total IBGP Sessions
BGP routes learned rate BGP Routes per Second
ISIS adjacencies established Total ISIS Adjacencies ISIS adjacencies established Total ISIS Adjacencies
ISIS routes learned rate ISIS Routes per Second ISIS routes learned rate ISIS Routes per Second
IPsec tunnels negotiated Total IPsec Tunnels IPsec tunnels negotiated Total IPsec Tunnels
IPsec tunnel establishment rate IPsec tunnels per second IPsec tunnel establishment rate IPsec tunnels per second
3.3.3 Instability Conditions 3.3.3 Instability Conditions
Instability Conditions may include and are not limited to the Instability Conditions may include and are not limited to the
following examples: following examples:
PARAMETER UNITS PARAMETER UNITS
Interface Shutdown Cycling Rate interfaces per minute Interface Shutdown Cycling Rate interfaces per minute
BGP Session Flap Rate sessions per minute ISIS Route Flap Rate routes per minutes
BGP Route Flap Rate routes per minutes
IGP Route Flap Rate routes per minutes
LSP Reroute Rate LSP per minute LSP Reroute Rate LSP per minute
Overloaded Links number Overloaded Links number
Amount Links Overloaded % of bandwidth Amount Links Overloaded % of bandwidth
FTP Rate Mb/minute FTP Rate Mb/minute
IPsec Tunnel Flap Rate tunnels per minute IPsec Tunnel Flap Rate tunnels per minute
Filter Policy Changes policies per hour Filter Policy Changes policies per hour
SSH Session Restart SSH sessions per hour SSH Session Rate SSH sessions per hour
Telnet Session Restart Telnet session per hour Telnet Session Rate Telnet session per hour
Command Entry Rate Commands per Hour
Message Flood Rate Messages per second
Stress Benchmarking Stress Benchmarking
3.3.4 Benchmarks 3.3.4 Benchmarks
Benchmarks are as defined in [1] and listed as follow: Benchmarks are as defined in [1] and listed as follow:
PARAMETER UNITS PHASE PARAMETER UNITS PHASE
Stable Aggregate Forwarding Rate pps Startup Stable Aggregate Forwarding Rate pps Startup
Stable Latency seconds Startup Stable Latency seconds Startup
Stable Session Count sessions Startup Stable Session Count sessions Startup
Unstable Aggregate Forwarding Rate pps Instability Unstable Aggregate Forwarding Rate pps Instability
Degraded Aggregate Forwarding Rate pps Instability Degraded Aggregate Forwarding Rate pps Instability
Ave. Degraded Aggregate Forwarding Rate pps Instability Ave. Degraded Aggregate Forwarding Rate pps Instability
Unstable Latency seconds Instability Unstable Latency seconds Instability
Unstable Uncontrolled Sessions Lost sessions Instability Unstable Uncontrolled Sessions Lost sessions Instability
Recovered Aggregate Forwarding Rate pps Recovery Recovered Aggregate Forwarding Rate pps Recovery
Recovered Latency seconds Recovery Recovered Latency seconds Recovery
Recovery Time seconds Recovery Recovery Time seconds Recovery
Recovered Uncontrolled Sessions Lost sessions Recovery Recovered Uncontrolled Sessions sessions Recovery
4. Example Test Case Procedure 4. Example Test Case Procedure
1. Report Configuration Set 1. Report Configuration Set
BGP Enabled BGP Enabled
10 EBGP Peers 10 EBGP Peers
30 IBGP Peers 30 IBGP Peers
500K BGP Route Instances 500K BGP Route Instances
160K BGP FIB Routes 160K BGP FIB Routes
skipping to change at page 8, line 10 skipping to change at page 8, line 10
Forwarding Rate[1..30] = 1Gbps Forwarding Rate[1..30] = 1Gbps
10000 Flows 10000 Flows
Encapsulation[1..5000] = IPv4 Encapsulation[1..5000] = IPv4
Encapsulation[5001.10000] = IPsec Encapsulation[5001.10000] = IPsec
Stress Benchmarking Stress Benchmarking
Logging Enabled Logging Enabled
Protocol Debug Disabled Protocol Debug Disabled
SNMP Enabled SNMP Enabled
SSH Enabled SSH Enabled
20 Concurrent SSH Sessions 10 Concurrent SSH Sessions
FTP Enabled FTP Enabled
RADIUS Enabled RADIUS Enabled
TACACS Disabled TACACS Disabled
Packet Statistics Collector Enabled Packet Statistics Collector Enabled
2. Begin Startup Conditions with the DUT 2. Begin Startup Conditions with the DUT
10 EBGP peering sessions negotiated 10 EBGP peering sessions negotiated
30 EBGP peering sessions negotiated 30 EBGP peering sessions negotiated
1K BGP routes learned per second 1K BGP routes learned per second
skipping to change at page 8, line 46 skipping to change at page 8, line 46
5. Apply Instability Conditions 5. Apply Instability Conditions
Interface Shutdown Cycling Rate = 1 interface every 5 minutes Interface Shutdown Cycling Rate = 1 interface every 5 minutes
BGP Session Flap Rate = 1 session every 10 minutes BGP Session Flap Rate = 1 session every 10 minutes
BGP Route Flap Rate = 100 routes per minute BGP Route Flap Rate = 100 routes per minute
ISIS Route Flap Rate = 100 routes per minute ISIS Route Flap Rate = 100 routes per minute
IPsec Tunnel Flap Rate = 1 tunnel per minute IPsec Tunnel Flap Rate = 1 tunnel per minute
Overloaded Links = 5 of 30 Overloaded Links = 5 of 30
Amount Links Overloaded = 20% Amount Links Overloaded = 20%
SNMP GETs = 1 per sec SNMP GETs = 1 per sec
SSH Restart Rate = 10 sessions per hour SSH Session Rate = 6 sessions per hour
FTP Restart Rate = 10 transfers per hour SSH Session Duration = 10 minutes
Command Rate via SSH = 20 commands per minute
FTP Restart Rate = 10 continuous transfers (Puts/Gets)
per hour
FTP Transfer Rate = 100 Mbps FTP Transfer Rate = 100 Mbps
Statistics Sampling Rate = 1:1 packets Statistics Sampling Rate = 1:1 packets
RADIUS Server Loss Rate = 1 per Hour
RADIUS Server Loss Duration = 3 seconds
6. Apply Instability Condition specific to test case. 6. Apply Instability Condition specific to test case.
Stress Benchmarking Stress Benchmarking
7. Report Instability Benchmarks as follow: 7. Report Instability Benchmarks as follow:
Unstable Aggregate Forwarding Rate Unstable Aggregate Forwarding Rate
Degraded Aggregate Forwarding Rate Degraded Aggregate Forwarding Rate
Ave. Degraded Aggregate Forwarding Rate Ave. Degraded Aggregate Forwarding Rate
Unstable Latency Unstable Latency
skipping to change at page 9, line 53 skipping to change at page 9, line 53
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, October 1991. Interconnection Devices", RFC 1242, October 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Terminology for Accelerated [4] Poretsky, S. and Rao, S., "Terminology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-07, Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-09,
work in progress, October 2005. work in progress, June 2006.
Stress Benchmarking Stress Benchmarking
[5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane [5] Poretsky, S., "Benchmarking Terminology for IGP Data Plane
Route Convergence", Route Convergence",
draft-ietf-bmwg-igp-dataplane-conv-term-08, work in draft-ietf-bmwg-igp-dataplane-conv-term-11, work in
progress, October 2005. progress, June 2006.
[6] Bradner, S., "Key words for use in RFCs to Indicate [6] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
8. Informative References 8. Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] "Core Router Evaluation for Higher Availability", [NANOG25] "Core Router Evaluation for Higher Availability",
Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA. Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
[IEEECQR] "Router Stress Testing to Validate Readiness for [IEEECQR] "Router Stress Testing to Validate Readiness for
Network Deployment", Scott Poretsky, IEEE CQR 2003. Network Deployment", Scott Poretsky, IEEE CQR 2003.
[CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data
Plane Route Convergence", Plane Route Convergence",
draft-ietf-bmwg-igp-dataplane-conv-meth-08, work in progress, draft-ietf-bmwg-igp-dataplane-conv-meth-11, work in
October 2005. progress, June 2006.
9. Author's Address 9. Author's Address
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
8 New England Executive Park 8 New England Executive Park
Burlington, MA 01803 Burlington, MA 01803
USA USA
Phone: + 1 781 395 5090 Phone: + 1 781 395 5090
EMail: sporetsky@reefpoint.com EMail: sporetsky@reefpoint.com
skipping to change at page 11, line 8 skipping to change at page 11, line 8
8th Floor 8th Floor
Qwest Communications Qwest Communications
Denver, CO 80202 Denver, CO 80202
USA USA
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Stress Benchmarking Stress Benchmarking
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2005). Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors contained in BCP 78, and except as set forth therein, the authors
retain all their rights. retain all their rights.
This document and the information contained herein are provided on an This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
 End of changes. 19 change blocks. 
29 lines changed or deleted 28 lines changed or added

This html diff was produced by rfcdiff 1.32. The latest version is available from http://www.levkowetz.com/ietf/tools/rfcdiff/