draft-ietf-bmwg-acc-bench-term-09.txt   draft-ietf-bmwg-acc-bench-term-10.txt 
Network Working Group Network Working Group
INTERNET-DRAFT INTERNET-DRAFT
Expires in: December 2006
Scott Poretsky Scott Poretsky
Reef Point Systems Reef Point Systems
Shankar Rao Shankar Rao
Qwest Communications Qwest Communications
October 2006
Terminology for Accelerated Stress Benchmarking Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-09.txt> <draft-ietf-bmwg-acc-bench-term-10.txt>
Intellectual Property Rights (IPR) statement: Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo Status of this Memo
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
skipping to change at page 3, line 13 skipping to change at page 3, line 13
4. IANA Considerations.........................................24 4. IANA Considerations.........................................24
Stress Benchmarking Stress Benchmarking
5. Security Considerations.....................................24 5. Security Considerations.....................................24
6. References..................................................24 6. References..................................................24
7. Author's Address............................................25 7. Author's Address............................................25
Appendix 1 - White Box Benchmarks..............................25 Appendix 1 - White Box Benchmarks..............................25
1. Introduction 1. Introduction
Routers in an operational network are simultaneously configured with Routers in an operational network are configured with multiple
multiple protocols and security policies while forwarding traffic and protocols and security policies while simultaneously forwarding
being managed. To accurately benchmark a router for deployment it is traffic and being managed. To accurately benchmark a router for
necessary to test that router in operational conditions by deployment, it is necessary to test that router under operational
simultaneously configuring and scaling network protocols and security conditions by simultaneously configuring and scaling network
policies, forwarding traffic, and managing the device. It is helpful protocols and security policies, forwarding traffic, and managing
to accelerate these network operational conditions so that the the device. It is useful to accelerate these network operational
router under test can be benchmarked with faster test duration. conditions so that the router under test can be benchmarked with
Testing a router in accelerated network conditions is known as a shorter test duration. Testing a router in accelerated network
Accelerated Stress Benchmarking. conditions is known as Accelerated Stress Benchmarking.
This document provides the Terminology for performing Stress This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase. benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase Forwarding Rate and control plane Session Count during each phase
of the test. For each plane, the Configuration Set, Startup of the test. For each plane, the Configuration Set, Startup
Conditions, and Instability Conditions are defined. Also defined are Conditions, and Instability Conditions are defined. Also defined are
the Benchmark Planes fundamental to stress testing configuration, the Benchmark Planes fundamental to stress testing configuration,
setup and measurement. These are the Control Plane, Data Plane, setup and measurement. These are the Control Plane, Data Plane,
Management Plane and Security Plane Multiple benchmarks are made Management Plane and Security Plane Multiple benchmarks are measured
for each Benchmark Plane during each Phase. Benchmarks can be for each Benchmark Plane during each Phase. Benchmarks can be
compared across multiple planes for the same DUT or at the same compared across multiple planes for the same DUT or at the same
plane for 2 or more DUTS. These benchmarks White Box benchmarks plane for 2 or more DUTS. Benchmarks of internal DUT characteristics
are provided in Appendix 1 for additional DUT behavior such as memory and CPU utilization (also known as White Box
measurements. The terminology is to be used with the companion benchmarks) are described in Appendix 1, to allow additional
methodology document [4]. The sequence of phases, actions, and characterization of DUT behavior. The terminology is to be used with
benchmarks are shown in Table 1. the companion methodology document [4]. The sequence of phases,
actions, and benchmarks are shown in Table 1.
2. Existing definitions 2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this Devices" should be consulted before attempting to make use of this
document. For the sake of clarity and continuity this RFC adopts document. For the sake of clarity and continuity this RFC adopts
the template for definitions set out in Section 2 of RFC 1242. the template for definitions set out in Section 2 of RFC 1242.
Definitions are indexed and grouped together in sections for ease Definitions are indexed and grouped together in sections for ease
of reference. of reference.
skipping to change at page 4, line 40 skipping to change at page 4, line 40
3. Term definitions 3. Term definitions
3.1 General Terms 3.1 General Terms
3.1.1 Benchmark Planes 3.1.1 Benchmark Planes
Definition: Definition:
The features, conditions, and behavior for the Accelerated Stress The features, conditions, and behavior for the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
There are four Benchmark Planes: Control Plane, Data Plane, There are four Benchmark Planes: Control Plane, Data Plane,
Management Plane, and Security Plane as shown in Figure 1. The Management Plane, and Security Plane as shown in Figure 1.
Benchmark Planes define the Configuration, Startup Conditions, Configuration, Startup Conditions, Instability Conditions, and
Instability Conditions, and Failure Conditions used for the test. Failure Conditions used for each test are defined for each of
these four Benchmark Planes.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Control Plane
Data Plane Data Plane
skipping to change at page 5, line 29 skipping to change at page 5, line 29
| | Data | | | | Data | |
|--->| Plane |<---| |--->| Plane |<---|
| | | |
----------- -----------
Figure 1. Router Accelerated Stress Benchmarking Planes Figure 1. Router Accelerated Stress Benchmarking Planes
3.1.2 Configuration Sets 3.1.2 Configuration Sets
Definition: Definition:
The features and scaling limits used during the Accelerated Stress The offered load, features, and scaling limits used during the
Benchmarking. Accelerated Stress Benchmarking.
Discussion: Discussion:
There are four Configuration Sets: Control Plane Configuration Set, There are four Configuration Sets: Control Plane Configuration
Data Plane Configuration Set, Management Plane Configuration Set, Set, Data Plane Configuration Set, Management Plane Configuration
and Security Plane Configuration Set. Set, and Security Plane Configuration Set. The minimum
Configuration Set that MUST be used is discussed in the
Methodology document [4].
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Configuration Set Control Plane Configuration Set
Data Plane Configuration Set Data Plane Configuration Set
skipping to change at page 7, line 23 skipping to change at page 7, line 23
See Also: See Also:
Startup Phase Startup Phase
3.1.6 Controlled Session Loss 3.1.6 Controlled Session Loss
Definition: Definition:
Control Plane sessions that are intentionally brought Control Plane sessions that are intentionally brought
down during the Stress test. down during the Stress test.
Discussion: Discussion:
The test equipment is able to control protocol Controlled Session Loss is performed during the test in
session state with the DUT. order to stress the DUT by forcing it to tear down Control
Plane sessions while handling traffic. It is assumed that
the test equipment is able to control protocol session
state with the DUT and is therefore able to introduce
Controlled Session Loss.
Measurement units: Measurement units:
None None
Issues: Issues:
None None
See Also: See Also:
Uncontrolled Session Loss Uncontrolled Session Loss
skipping to change at page 8, line 16 skipping to change at page 8, line 16
3.2 Benchmark Planes 3.2 Benchmark Planes
3.2.1 Control Plane 3.2.1 Control Plane
Definition: Definition:
The Description of the control protocols enabled for The Description of the control protocols enabled for
the Accelerated Stress Benchmarking. the Accelerated Stress Benchmarking.
Discussion: Discussion:
The Control Plane defines the Configuration, Startup The Control Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the control Conditions, and Instability Conditions of the control
protocols. Control Plane protocols may include routing protocols. Control Plane protocols MAY include routing
protocols, multicast protocols, and MPLS protocols. protocols, multicast protocols, and MPLS protocols.
These can be enabled or disabled for a benchmark test. These can be enabled or disabled for a benchmark test.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
skipping to change at page 9, line 15 skipping to change at page 9, line 15
Stress Benchmarking Stress Benchmarking
Discussion: Discussion:
A key component of the Accelerated Stress Benchmarking is the A key component of the Accelerated Stress Benchmarking is the
Management Plane to assess manageability of the router Management Plane to assess manageability of the router
under stress. The Management Plane defines the Configuration, under stress. The Management Plane defines the Configuration,
Startup Conditions, and Instability Conditions of the Startup Conditions, and Instability Conditions of the
management protocols and features. The Management Plane management protocols and features. The Management Plane
includes SNMP, Logging/Debug, Statistics Collection, and includes SNMP, Logging/Debug, Statistics Collection, and
management configuration sessions such as telnet, SSH, and management configuration sessions such as telnet, SSH, and
serial console. SNMP Gets SHOULD be performed continuously. serial console.
Management configuration sessions should be open
simultaneously and be repeatedly open and closed. Open
management sessions should have valid and invalid
configuration and show commands entered.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Benchmark Planes Benchmark Planes
Management Plane Configuration Set Management Plane Configuration Set
skipping to change at page 9, line 43 skipping to change at page 9, line 39
3.2.4 Security Plane 3.2.4 Security Plane
Definition: Definition:
The Security features used during the Accelerated Stress The Security features used during the Accelerated Stress
Benchmarking. Benchmarking.
Discussion: Discussion:
The Security Plane defines the Configuration, Startup The Security Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the security Conditions, and Instability Conditions of the security
features and protocols. The Security Plane includes the features and protocols. The Security Plane includes the
ACLs, Firewall, Secure Protocols, and User Login. Tunnels ACLs, Firewall, Secure Protocols, and User Login.
for those such as IPsec should be established and flapped.
Policies for Firewalls and ACLs should be repeatedly added
and removed from the configuration via telnet, SSH, or
serial management sessions.
Measurement units: Measurement units:
N/A N/A
Issues: None Issues: None
See Also: See Also:
Benchmark Planes Benchmark Planes
Security Plane Configuration Set Security Plane Configuration Set
Security Plane Startup Conditions Security Plane Startup Conditions
skipping to change at page 10, line 22 skipping to change at page 10, line 22
Startup Conditions are generated with the DUT. This Startup Conditions are generated with the DUT. This
begins with the attempt to establish the first session begins with the attempt to establish the first session
and ends when the last Control Plane session is and ends when the last Control Plane session is
established. established.
Discussion: Discussion:
The Startup Phase is the first Phase of the benchmarking The Startup Phase is the first Phase of the benchmarking
test preceding the Instability Phase and Recovery Phase. test preceding the Instability Phase and Recovery Phase.
It is specified by the Configuration Sets and Startup It is specified by the Configuration Sets and Startup
Conditions for each Benchmark Plane. The Startup Phase ends Conditions for each Benchmark Plane. The Startup Phase ends
and Instability Phase may begin when the Configuration Sets and Instability Phase MUST begin when the Configuration Sets
are achieved with the DUT. are achieved with the DUT.
Measurement Units: Measurement Units:
None None
Issues: Issues:
The 'last control plane session is established' may not The 'last control plane session is established' may not
be a sufficient indicator that steady-state is achieved be a sufficient indicator that steady-state is achieved
and Instability Conditions can be applied to begin the and Instability Conditions can be applied to begin the
Instability Phase. Instability Phase.
skipping to change at page 11, line 56 skipping to change at page 11, line 56
Stable Aggregate Forwarding Rate Stable Aggregate Forwarding Rate
3.3.2.3 Stable Session Count 3.3.2.3 Stable Session Count
Definition: Definition:
Total number of control plane sessions/adjacencies Total number of control plane sessions/adjacencies
established and maintained by the DUT during the Startup established and maintained by the DUT during the Startup
Phase and prior to Instability Conditions being initiated. Phase and prior to Instability Conditions being initiated.
Discussion: Discussion:
This measurement should be made after the Control This measurement SHOULD be made after the Control
Plane Startup Conditions are applied to the DUT. Plane Startup Conditions are applied to the DUT.
Stress Benchmarking Stress Benchmarking
Measurement units: Measurement units:
sessions sessions
Issues: Issues:
None None
skipping to change at page 14, line 37 skipping to change at page 14, line 37
The router management features enabled for the The router management features enabled for the
Accelerated Stress Benchmark. Accelerated Stress Benchmark.
Discussion: Discussion:
A key component of the Accelerated Stress Benchmark is the A key component of the Accelerated Stress Benchmark is the
Management Configuration Set to assess manageability of the Management Configuration Set to assess manageability of the
router under stress. The Management Configuration Set defines router under stress. The Management Configuration Set defines
the management configuration of the DUT. Features that are the management configuration of the DUT. Features that are
part of the Management Configuration Set include access, SNMP, part of the Management Configuration Set include access, SNMP,
Logging/Debug, and Statistics Collection, and services such as Logging/Debug, and Statistics Collection, and services such as
FTP, as shown in Figure 3. These features should be enabled FTP, as shown in Figure 3. These features SHOULD be enabled
throughout the Stress test. SNMP Gets should be made throughout the Stress test.
continuously with multiple FTP and Telnet sessions operating
simultaneously. FTP sessions should be opened and closed at
varying intervals and get and put files while open. Telnet
sessions should be opened and closed at varying intervals and
enter valid and invalid show and configuration commands while
open.
Measurement units: Measurement units:
N/A N/A
Issues: Issues:
None None
See Also: See Also:
Control Plane Configuration Set Control Plane Configuration Set
Data Plane Configuration Set Data Plane Configuration Set
skipping to change at page 15, line 40 skipping to change at page 15, line 40
3.3.6 Security Plane 3.3.6 Security Plane
3.3.6.1 Security Plane Configuration Set 3.3.6.1 Security Plane Configuration Set
Definition: Definition:
Security features and scaling enabled for the Accelerated Stress Security features and scaling enabled for the Accelerated Stress
Test. Test.
Discussion: Discussion:
The Security Plane Configuration Set includes the configuration The Security Plane Configuration Set includes the configuration
and scaling of ACLs, Firewall, IPsec, and User Access, as shown and scaling of ACLs, Firewall, IPsec, and User Access, as shown
in Figure 4. Tunnels should be established and policies in Figure 4. Tunnels SHOULD be established and policies
configured. Instability is introduced by flapping tunnels and configured. Instability is introduced by flapping tunnels and
configuring and removing policies. configuring and removing policies.
____________ ____________ ____________ ____________ ____________ ____________
| | | Secure | | User | | | | Secure | | User |
|ACL/Firewall| | Protocol | __| Access | |ACL/Firewall| | Protocol | __| Access |
| | | | | | | | | | | | | |
------------ ------------ | ------------ ------------ ------------ | ------------
| | | | | |
| | | | | |
skipping to change at page 21, line 38 skipping to change at page 21, line 38
Definition: Definition:
The portion of the benchmarking test in which the The portion of the benchmarking test in which the
Startup Conditions are generated with the DUT, but Startup Conditions are generated with the DUT, but
the Instability Conditions are no longer offered to the Instability Conditions are no longer offered to
the DUT. the DUT.
Discussion: Discussion:
The Recovery Phase is the final Phase of the The Recovery Phase is the final Phase of the
benchmarking test following the Startup Phase and benchmarking test following the Startup Phase and
Instability Phase. Startup Conditions must not be Instability Phase. Startup Conditions MUST NOT be
Restarted. Restarted.
Measurement Units: Measurement Units:
None None
Issues: Issues:
None None
See Also: See Also:
Startup Conditions Startup Conditions
skipping to change at page 23, line 56 skipping to change at page 23, line 56
3.5.2.5 Variability Benchmarks 3.5.2.5 Variability Benchmarks
Definition: Definition:
The difference between the measured Benchmarks of the The difference between the measured Benchmarks of the
same DUT over multiple iterations. same DUT over multiple iterations.
Discussion: Discussion:
Ideally, the measured benchmarks should be the same for multiple Ideally, the measured benchmarks should be the same for multiple
iterations with the same DUT. Configuration Sets and iterations with the same DUT. Configuration Sets and
Instability Conditions SHOULD be held constant for this Instability Conditions MUST be held constant for this
benchmark. Whether the DUT can exhibit such predictable and benchmark. Whether the DUT can exhibit such predictable and
repeatable behavior is an important benchmark in itself. repeatable behavior is an important benchmark in itself.
Stress Benchmarking Stress Benchmarking
Measurement units: Measurement units:
As applicable to each Benchmark. The results are to be As applicable to each Benchmark. The results are to be
presented in a table format for successive Iterations. presented in a table format for successive Iterations.
Ideally, the differences should be zero. Ideally, the differences should be zero.
skipping to change at page 24, line 40 skipping to change at page 24, line 40
6.1 Normative References 6.1 Normative References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network [1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, March 1991. Interconnection Devices", RFC 1242, March 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching [2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998. Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[4] Poretsky, S. and Rao, S., "Methodology for Accelerated [4] Poretsky, S. and Rao, S., "Methodology Guidelines for
Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-05, Accelerated Stress Benchmarking",
work in progress, June 2006. draft-ietf-bmwg-acc-bench-meth-06, work in progress,
October 2006.
[5] Bradner, S., "Key words for use in RFCs to Indicate [5] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
6.2 Informative References 6.2 Informative References
[RFC3871] RFC 3871 "Operational Security Requirements for Large [RFC3871] RFC 3871 "Operational Security Requirements for Large
Internet Service Provider (ISP) IP Network Infrastructure. Internet Service Provider (ISP) IP Network Infrastructure.
G. Jones, Ed.. IETF, September 2004. G. Jones, Ed.. IETF, September 2004.
[NANOG25] Poretsky, S., "Core Router Evaluation for Higher [NANOG25] Poretsky, S., "Core Router Evaluation for Higher
skipping to change at page 25, line 33 skipping to change at page 25, line 33
Phone: + 1 303 437 6643 Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com Email: shankar.rao@qwest.com
Appendix 1. White Box Benchmarking Terminology Appendix 1. White Box Benchmarking Terminology
Minimum Available Memory Minimum Available Memory
Definition: Definition:
Minimum DUT Available Memory during the duration of the Minimum DUT Available Memory during the duration of the
Accelerated Stress Benchmark. Accelerated Stress Benchmark.
Discussion: Discussion:
It is necessary to monitor DUT memory to measure this This benchmark enables the assessment of resources in the DUT.
benchmark. It is necessary to monitor DUT memory to measure this benchmark.
Measurement units: Measurement units:
bytes bytes
Issues: None Issues: None
See Also: See Also:
Maximum CPU Utilization Maximum CPU Utilization
Maximum CPU Utilization Maximum CPU Utilization
Definition: Definition:
Maximum DUT CPU utilization during the duration of the Maximum DUT CPU utilization during the duration of the
Accelerated Stress Benchmark. Accelerated Stress Benchmark.
Discussion: Discussion:
This benchmark enables the assessment of resources in the DUT.
It is necessary to monitor DUT CPU Utilization to measure It is necessary to monitor DUT CPU Utilization to measure
this benchmark. this benchmark.
Measurement units: % Measurement units: %
Issues: None Issues: None
See Also: See Also:
Minimum Available Memory Minimum Available Memory
Stress Benchmarking Stress Benchmarking
 End of changes. 22 change blocks. 
57 lines changed or deleted 55 lines changed or added

This html diff was produced by rfcdiff 1.33. The latest version is available from http://tools.ietf.org/tools/rfcdiff/