draft-ietf-bmwg-reset-06.txt   rfc6201.txt 
Benchmarking Methodology WG Rajiv Asati
Internet Draft Cisco
Updates: 1242, 2544 (if approved) Carlos Pignataro
Intended status: Informational Cisco
Expires: June 2011 Fernando Calabria
Cisco
Cesar Olvera
Consulintel
December 20, 2010 Internet Engineering Task Force (IETF) R. Asati
Request for Comments: 6201 C. Pignataro
Updates: 1242, 2544 F. Calabria
Category: Informational Cisco
ISSN: 2070-1721 C. Olvera
Consulintel
March 2011
Device Reset Characterization Device Reset Characterization
draft-ietf-bmwg-reset-06
Abstract Abstract
An operational forwarding device may need to be re-started An operational forwarding device may need to be restarted
(automatically or manually) for a variety of reasons, an event that (automatically or manually) for a variety of reasons, an event called
we call a "reset" in this document. Since there may be an a "reset" in this document. Since there may be an interruption in
interruption in the forwarding operation during a reset, it is the forwarding operation during a reset, it is useful to know how
useful to know how long a device takes to resume the forwarding long a device takes to resume the forwarding operation.
operation.
This document specifies a methodology for characterizing reset (and This document specifies a methodology for characterizing reset (and
reset time) during benchmarking of forwarding devices, and provides reset time) during benchmarking of forwarding devices and provides
clarity and consistency in reset test procedures beyond what's clarity and consistency in reset test procedures beyond what is
specified in RFC2544. It therefore updates RFC2544. This document specified in RFC 2544. Therefore, it updates RFC 2544. This
also defines the benchmarking term "Reset Time" and only in this document also defines the benchmarking term "reset time" and, only in
updates RFC1242. this, updates RFC 1242.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six Status of This Memo
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at This document is not an Internet Standards Track specification; it is
http://www.ietf.org/ietf/1id-abstracts.txt published for informational purposes.
The list of Internet-Draft Shadow Directories can be accessed at This document is a product of the Internet Engineering Task Force
http://www.ietf.org/shadow.html (IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
This Internet-Draft will expire on June 20, 2011. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc6201.
Copyright Notice Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with carefully, as they describe your rights and restrictions with respect
respect to this document. Code Components extracted from this to this document. Code Components extracted from this document must
document must include Simplified BSD License text as described in include Simplified BSD License text as described in Section 4.e of
Section 4.e of the Trust Legal Provisions and are provided without the Trust Legal Provisions and are provided without warranty as
warranty as described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Key Words to Reflect Requirements..............................4 1. Introduction ....................................................3
2. Introduction...................................................4 1.1. Scope ......................................................3
2.1. Scope.....................................................4 1.2. Reset Time .................................................4
2.2. Reset Time................................................5 1.3. Reset Time Measurement Methods .............................5
2.3. Reset Time Measurement Methods............................6 1.4. Reporting Format ...........................................6
2.4. Reporting Format..........................................7 2. Key Words to Reflect Requirements ...............................7
3. Test Requirements..............................................8 3. Test Requirements ...............................................7
4. Reset Test.....................................................9 4. Reset Tests .....................................................8
4.1. Hardware Reset Test......................................10 4.1. Hardware Reset Tests .......................................9
4.1.1. Routing Processor (RP) / Routing Engine Reset.......10 4.1.1. Routing Processor (RP) / Routing Engine Reset .......9
4.1.1.1. RP Reset for a single-RP device (REQUIRED).....11 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED) ....11
4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 4.2. Software Reset Tests ......................................12
........................................................11 4.2.1. Operating System (OS) Reset (REQUIRED) .............12
4.1.2. Line Card (LC) Removal and Insertion (REQUIRED).....13 4.2.2. Process Reset (OPTIONAL) ...........................13
4.2. Software Reset Test......................................14 4.3. Power Interruption Test ...................................14
4.2.1. Operating System (OS) Reset (REQUIRED)..............14 4.3.1. Power Interruption (REQUIRED) ......................14
4.2.2. Process Reset (OPTIONAL)............................15 5. Security Considerations ........................................15
4.3. Power Interruption Test..................................16 6. Acknowledgments ................................................16
4.3.1. Power Interruption (REQUIRED).......................16 7. References .....................................................16
5. Security Considerations.......................................17 7.1. Normative References ......................................16
6. IANA Considerations...........................................17 7.2. Informative References ....................................16
7. Acknowledgments...............................................17
8. References....................................................19
8.1. Normative References.....................................19
8.2. Informative References...................................19
Authors' Addresses...............................................20
1. Key Words to Reflect Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119
[RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards
track document.
2. Introduction 1. Introduction
An operational forwarding device (or one of its components) may need An operational forwarding device (or one of its components) may need
to be re-started for a variety of reasons, an event that we call a to be restarted for a variety of reasons, an event called a "reset"
"reset" in this document. Since there may be an interruption in the in this document. Since there may be an interruption in the
forwarding operation during a reset, it is useful to know how long a forwarding operation during a reset, it is useful to know how long a
device takes to resume the forwarding operation. In other words, it device takes to resume the forwarding operation. In other words, the
is desired to know the duration of the recovery time following the duration of the recovery time following the reset (see Section 1.2,
reset (reset time, see Section 2.2). "Reset Time") is what is in question.
However, the answer to this question is no longer simple and However, the answer to this question is no longer simple and
straight-forward as the modern forwarding devices employ many straightforward as the modern forwarding devices employ many hardware
hardware advancements (distributed forwarding, etc.) and software advancements (distributed forwarding, etc.) and software advancements
advancements (graceful restart, etc.) that influence the recovery (graceful restart, etc.) that influence the recovery time after the
time after the reset. reset.
2.1. Scope 1.1. Scope
This document specifies a methodology for characterizing reset (and This document specifies a methodology for characterizing reset (and
reset time) during benchmarking of forwarding devices, and provides reset time) during benchmarking of forwarding devices and provides
clarity and consistency in reset procedures beyond what is specified clarity and consistency in reset procedures beyond what is specified
in [RFC2544]. Software upgrades involve additional benchmarking in [RFC2544]. Software upgrades involve additional benchmarking
complexities and are outside the scope of this document. complexities and are outside the scope of this document.
These procedures may be used by other benchmarking documents such as These procedures may be used by other benchmarking documents such as
[RFC2544], [RFC5180], [RFC5695], etc., and is expected that other [RFC2544], [RFC5180], [RFC5695], etc., and it is expected that other
protocol-specific benchmarking documents would reference this protocol-specific benchmarking documents will reference this document
document for reset recovery time characterization. Specific Routing for reset recovery time characterization. Specific Routing
Information Base (RIB) and Forwarding Information Base (FIB) scaling Information Base (RIB) and Forwarding Information Base (FIB) scaling
considerations are outside the scope of this document and can be considerations are outside the scope of this document and can be
quite complex to characterize. However, other documents can quite complex to characterize. However, other documents can
characterize specific dynamic protocols scaling and interactions and characterize specific dynamic protocols' scaling and interactions as
leverage and augment the reset tests defined in this document. well as leverage and augment the reset tests defined in this
document.
This document updates Section 26.6 of [RFC2544], and defines the This document updates Section 26.6 of [RFC2544] and defines the
benchmarking term "Reset Time" updating [RFC1242]. benchmarking term "reset time", updating [RFC1242].
This document focuses only on the reset criterion of benchmarking, This document focuses only on the reset criterion of benchmarking and
and presumes that it would be beneficial to [RFC5180], [RFC5695], presumes that it would be beneficial to [RFC5180], [RFC5695], and
and other IETF Benchmarking Methodology Working Group (BMWG) other IETF Benchmarking Methodology Working Group (BMWG) efforts.
efforts.
2.2. Reset Time 1.2. Reset Time
Definition Definition
Reset Time is the total time when a device is determined to be out Reset time is the total time that a device is determined to be out
of operation, and includes the time to perform the reset and the of operation and includes the time to perform the reset and the
time to recover from it. time to recover from it.
Discussion Discussion
During a period of time after a reset or power up, network devices During a period of time after a reset or power up, network devices
may not accept and forward frames. The duration of this period of may not accept and forward frames. The duration of this period of
forwarding unavailability can be useful in evaluating devices. In forwarding unavailability can be useful in evaluating devices. In
addition, some network devices require some form of reset when addition, some network devices require some form of reset when
specific setup variables are modified. If the reset period were specific setup variables are modified. If the reset period were
long it might discourage network managers from modifying these long, it might discourage network managers from modifying these
variables on production networks. variables on production networks.
The events characterized in this document are entire reset events. The events characterized in this document are entire reset events.
That is, the recovery period measured includes the time to perform That is, the recovery period measured includes the time to perform
the reset and the time to recover from it. Some reset events will the reset and the time to recover from it. Some reset events will
be atomic (such as pressing a reset button) while others (such as be atomic (such as pressing a reset button) while others (such as
power cycling) may comprise multiple actions with a recognized power cycling) may comprise multiple actions with a recognized
interval between them. In both cases, the duration considered is interval between them. In both cases, the duration considered is
from the start of the event until full recovery of forwarding from the start of the event until full recovery of forwarding
after the completion of the reset events. after the completion of the reset events.
Measurement units Measurement Units
Time in milliseconds, providing sufficient resolution to Time, in milliseconds, providing sufficient resolution to
distinguish between different trials and different distinguish between different trials and different
implementations. See Section 2.4. implementations. See Section 1.4.
Issues Issues
There are various types of Reset: Hardware resets, software There are various types of resets: hardware resets, software
resets, and power interruption. See Section 4. resets, and power interruptions. See Section 4.
See Also See Also
This definition updates [RFC1242]. This definition updates [RFC1242].
2.3. Reset Time Measurement Methods 1.3. Reset Time Measurement Methods
The 'reset time' is the time during which the traffic forwarding is The reset time is the time during which traffic forwarding is
temporarily interrupted following a reset event. Strictly speaking, temporarily interrupted following a reset event. Strictly speaking,
this is the time over which one or more frames are lost. This this is the time over which one or more frames are lost. This
definition is similar to that of 'Loss of connectivity period' definition is similar to that of "Loss of Connectivity Period"
defined in [IGPConv] section 4. defined in [IGPConv], Section 4.
There are two accepted methods to measure the 'reset time': There are two accepted methods to measure the reset time:
1. Frame-Loss Method - This method requires test tool capability 1. Frame-Loss Method - This method requires test tool capability to
to monitor the number of lost frames. In this method, the monitor the number of lost frames. In this method, the offered
offered stream rate (frames per second) must be known. The stream rate (frames per second) must be known. The reset time is
reset time is calculated per the below equation: calculated per the equation below:
Frames_lost (packets) Frames_lost (packets)
Reset_time = ------------------------------------- Reset_time = -------------------------------------
Offered_rate (packets per second) Offered_rate (packets per second)
2. Time-Stamp Method - This method requires test tool capability 2. Timestamp Method - This method requires test tool capability to
to timestamp each frame. In this method, the test tool timestamp each frame. In this method, the test tool timestamps
timestamps each transmitted frame and monitors the received each transmitted frame and monitors the received frame's
frame's timestamp. During the test, the test tool would record timestamp. During the test, the test tool records the timestamp
the timestamp (Timestamp A) of the frame that was last (Timestamp A) of the frame that was last received prior to the
received prior to the reset interruption and the timestamp reset interruption and the timestamp of the first frame after the
(Timestamp B) of the first frame after the interruption interruption stopped (Timestamp B). The difference between
stopped. The difference between Timestamp B and Timestamp A is Timestamp B and Timestamp A is the reset time.
the reset time.
The tester / operator MAY use either method for reset time The tester/operator MAY use either method for reset time measurement
measurement depending on the test tool capability. However, the depending on the test tool capability. However, the Frame-Loss
Frame-loss method SHOULD be used if the test tool is capable of (a) method SHOULD be used if the test tool is capable of (a) counting the
counting the number of lost frames per stream, and (b) transmitting number of lost frames per stream and (b) transmitting test frame
test frame despite the physical link status, whereas Time-stamp despite the physical link status, whereas the Timestamp method SHOULD
method SHOULD be used if the test tool is capable of (a) time- be used if the test tool is capable of (a) timestamping each frame,
stamping each frame, (b) monitoring received frame's timestamp, and (b) monitoring received frame's timestamp, and (c) transmitting
(c) transmitting frames only if the physical link status is UP. That frames only if the physical link status is UP. That is, specific
is, specific test tool capabilities may dictate which method to use. test tool capabilities may dictate which method to use. If the test
If the test tool supports both methods based on its capabilities, tool supports both methods based on its capabilities, the
the tester / operator SHOULD use the one that provides more tester/operator SHOULD use the one that provides more accuracy.
accuracy.
2.4. Reporting Format 1.4. Reporting Format
All reset results are reported in a simple statement including the All reset results are reported in a simple statement including the
frame loss (if measured) and reset times. frame loss (if measured) and reset times.
For each test case, it is RECOMMENDED that the following parameters For each test case, it is RECOMMENDED that the following parameters
be reported in these units: be reported in these units:
Parameter Units or Examples Parameter Units or Examples
---------------------------------------------------------------
-----------------------------------------------------------------
Throughput Frames per second and bits per Throughput Frames per second and bits per
second second
Loss (average) Frames Loss (average) Frames
Reset Time (average) Milliseconds Reset Time (average) Milliseconds
Number of trials Integer count Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc. Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet), Port Media Ethernet, Gigabit Ethernet (GbE),
POS (Packet over SONET), etc. Packet over SONET (POS), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc. Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN, Interface Encap. Ethernet, Ethernet VLAN,
PPP, HDLC, etc. PPP, High-Level Data Link Control
(HDLC), etc.
For mixed protocol environments, frames SHOULD be distributed For mixed protocol environments, frames SHOULD be distributed between
between all the different protocols. The distribution MAY all the different protocols. The distribution MAY approximate the
approximate the network conditions of deployment. In all cases the network conditions of deployment. In all cases, the details of the
details of the mixed protocol distribution MUST be included in the mixed protocol distribution MUST be included in the reporting.
reporting.
Additionally, the DUT (Device Under Test) / SUT (System Under Test) Additionally, the DUT (Device Under Test) or SUT (System Under Test)
and test bed provisioning, port and Line Card arrangement, and test bed provisioning, port and line-card arrangement,
configuration, and deployed methodologies that may influence the configuration, and deployed methodologies that may influence the
overall reset time MUST be listed. (Refer to the additional factors overall reset time MUST be listed. (Refer to the additional factors
listed in Section 3). listed in Section 3).
The reporting of results MUST regard repeatability considerations The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results. trials and report average results.
3. Test Requirements 2. Key Words to Reflect Requirements
Tests SHOULD first be performed such that the forwarding state re- The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
establishment is independent from an external source (i.e., using "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119
[RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of Standards-Track documents as clear as possible. While
this document uses these keywords, this document is not a Standards-
Track document.
3. Test Requirements
Tests SHOULD first be performed such that the forwarding state
re-establishment is independent from an external source (i.e., using
static address resolution, routing and forwarding configuration, and static address resolution, routing and forwarding configuration, and
not dynamic protocols). However tests MAY subsequently be performed not dynamic protocols). However, tests MAY subsequently be performed
using dynamic protocols that the forwarding state depends on (e.g., using dynamic protocols that the forwarding state depends on (e.g.,
dynamic Interior Gateway Protocols (IGP), ARP, PPP Control dynamic Interior Gateway Protocols (IGP), Address Resolution Protocol
Protocols, etc.) The considerations in this Section apply. (ARP), PPP Control Protocols, etc.). The considerations in this
section apply.
In order to provide consistence and fairness while benchmarking a In order to provide consistence and fairness while benchmarking a set
set of different DUTs, the Network tester / operator MUST (a) use of different DUTs, the Network tester/operator MUST (a) use identical
identical control and data plane information during testing, (b) control and data plane information during testing and (b) document
document & report any factors that may influence the overall time and report any factors that may influence the overall time after
after reset / convergence. reset/convergence.
Some of these factors include: Some of these factors include the following:
1. Type of reset - Hardware (line-card crash, etc.) vs. Software 1. Type of reset - hardware (line-card crash, etc.) versus software
(protocol reset, process crash, etc.) or even complete power (protocol reset, process crash, etc.) or even complete power
failures failures
2. Manual vs. Automatic reset 2. Manual versus automatic reset
3. Scheduled vs. non-scheduled reset 3. Scheduled versus non-scheduled reset
4. Local vs. Remote reset 4. Local versus remote reset
5. Scale - Number of line cards present vs. in-use 5. Scale - Number of line cards present versus in use
6. Scale - Number of physical and logical interfaces 6. Scale - Number of physical and logical interfaces
7. Scale - Number of routing protocol instances 7. Scale - Number of routing protocol instances
8. Scale - Number of Routing Table entries
9. Scale - Number of Route Processors available 8. Scale - Number of routing table entries
10. Performance - Redundancy strategy deployed for route 9. Scale - Number of route processors available
processors and line cards 10. Performance - Redundancy strategy deployed for route processors
and line cards
11. Performance - Interface encapsulation as well as achievable 11. Performance - Interface encapsulation as well as achievable
Throughput [RFC2544] throughput [RFC2544]
12. Any other internal or external factor that may influence reset 12. Any other internal or external factor that may influence reset
time after a hardware or software reset time after a hardware or software reset
The reset time is one of the key characterization results reported The reset time is one of the key characterization results reported
after each test run. While the reset time during a reset test event after each test run. While the reset time during a reset test event
may be zero, there may still be effects on traffic, such as may be zero, there may still be effects on traffic, such as transient
transient delay variation or increased latency. However, that is not delay variation or increased latency. However, that is not covered
covered and deemed outside the scope of this document. In this case, and is deemed outside the scope of this document. In this case, only
only "no loss" is reported. "no loss" is reported.
4. Reset Test 4. Reset Tests
This section contains the description of the tests that are related This section contains descriptions of the tests that are related to
to the characterization of the time needed for DUTs (Device Under the characterization of the time needed for DUTs (Devices Under Test)
Test) / SUTs (System Under Test) to recover from a reset. There are or SUTs (Systems Under Test) to recover from a reset. There are
three types of reset considered in this document: three types of resets considered in this document:
1. Hardware resets 1. Hardware resets
2. Software resets 2. Software resets
3. Power interruption 3. Power interruption
Different types of reset have potentially different impact on the Different types of resets potentially have a different impact on the
forwarding behavior of the device. As an example, a software reset forwarding behavior of the device. As an example, a software reset
(of a routing process) might not result in forwarding interruption, (of a routing process) might not result in forwarding interruption,
whereas a hardware reset (of a line card) most likely will. whereas a hardware reset (of a line card) most likely will.
Section 4.1 describes various hardware resets, whereas Section 4.2 Section 4.1 describes various hardware resets, whereas Section 4.2
describes various software resets. Additionally, Section 4.3 describes various software resets. Additionally, Section 4.3
describes power interruption tests. These sections define and describes power interruption tests. These sections define and
characterize these resets. characterize these resets.
Additionally, since device specific implementations may vary for Additionally, since device-specific implementations may vary for
hardware and software type resets, it is desirable to classify each hardware and software type resets, it is desirable to classify each
test case as "REQUIRED" or "OPTIONAL". test case as "REQUIRED" or "OPTIONAL".
4.1. Hardware Reset Test 4.1. Hardware Reset Tests
A test designed to characterize the time it takes a DUT to recover A hardware reset test is a test designed to characterize the time it
from the hardware reset. takes a DUT to recover from a hardware reset.
A "hardware reset" generally involves the re-initialization of one A hardware reset generally involves the re-initialization of one or
or more physical components in the DUT, but not the entire DUT. more physical components in the DUT, but not the entire DUT.
A hardware reset is executed by the operator for example by physical A hardware reset is executed by the operator, for example, by
removal of a hardware component, by pressing on a "reset" button for physical removal of a hardware component, by pressing a reset button
the component, or could even be triggered from the command line for the component, or by being triggered from the command line
interface. interface (CLI).
Reset procedures that do not require the physical removal and Reset procedures that do not require the physical removal and
insertion of a hardware component are RECOMMENDED. These include insertion of a hardware component are RECOMMENDED. These include
using the Command Line Interface (CLI) or a physical switch or using the command line interface (CLI) or a physical switch or
button. If such procedures cannot be performed (e.g., for lack of button. If such procedures cannot be performed (e.g., because of a
platform support, or because the corresponding Test Case calls for lack of platform support or because the corresponding test case calls
them), human operation time SHOULD be minimized across different for them), human operation time SHOULD be minimized across different
platforms and Test Cases as much as possible, and variation in human platforms and test cases as much as possible, and variation in human
operator time SHOULD also be minimized across different vendors operator time SHOULD also be minimized across different vendors'
products as much as practical, by having the same person perform the products as much as practical by having the same person perform the
operation, and by practicing the operation. Additionally, the time operation and by practicing the operation. Additionally, the time
between removal and insertion SHOULD be recorded and reported. between removal and insertion SHOULD be recorded and reported.
For routers that do not contain separate Routing Processor and Line For routers that do not contain separate Routing Processor and Line
Card modules, the hardware reset tests are not performed since they Card modules, the hardware reset tests are not performed since they
are not relevant; instead, the power interruption tests MUST be are not relevant; instead, the power interruption tests MUST be
performed (see Section 4.3) in these cases. performed (see Section 4.3) in these cases.
4.1.1. Routing Processor (RP) / Routing Engine Reset 4.1.1. Routing Processor (RP) / Routing Engine Reset
The Routing Processor (RP) is the DUT module that is primarily The Routing Processor (RP) is the DUT module that is primarily
concerned with Control Plane functions. concerned with Control Plane functions.
4.1.1.1. RP Reset for a single-RP device (REQUIRED) 4.1.1.1. RP Reset for a Single-RP Device (REQUIRED)
Objective Objective
To characterize time needed for a DUT to recover from a Route To characterize the time needed for a DUT to recover from a Route
processor hardware reset in a single RP environment. Processor hardware reset in a single RP environment.
Procedure Procedure
First, ensure that the RP is in a permanent state to which it will First, ensure that the RP is in a permanent state to which it will
return to after the reset, by performing some or all of the return after the reset by performing some or all of the following
following operational tasks: save the current DUT configuration, operational tasks: save the current DUT configuration, specify
specify boot parameters, ensure the appropriate software files are boot parameters, ensure the appropriate software files are
available, or perform additional Operating System or hardware available, or perform additional operating system or hardware-
related task. related tasks.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing, and the rate should be sufficient for the DUT
attain the maximum forwarding throughput. This enables a finer to attain the maximum forwarding throughput. This enables a finer
granularity in the reset time measurement. granularity in the reset time measurement.
Third, perform the Route Processor (RP) hardware reset at this Third, perform the Route Processor (RP) hardware reset at this
point. This entails for example physically removing the RP to point. This entails, for example, physically removing the RP to
later re-insert it, or triggering a hardware reset by other means later re-insert it or triggering a hardware reset by other means
(e.g., command line interface, physical switch, etc.) (e.g., command line interface, physical switch, etc.).
Finally, the characterization is completed by recording the frame Finally, complete the characterization by recording the frame loss
loss or time stamps (as reported by the test tool) and calculating or timestamps (as reported by the test tool) and calculating the
the reset time (as defined in Section 2.3). reset time (as defined in Section 1.3).
Reporting format Reporting Format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 1.4.
4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 4.1.1.2. RP Switchover for a Multiple-RP Device (OPTIONAL)
Objective Objective
To characterize time needed for "secondary" Route Processor To characterize the time needed for the "secondary" Route
(sometimes referred to as "backup" RP) of a DUT to become active Processor (sometimes referred to as the "backup" RP) of a DUT to
after a "primary" (or "active") Route Processor hardware reset. become active after a "primary" (or "active") Route Processor
This process is often referred to as "RP Switchover". The hardware reset. This process is often referred to as "RP
characterization in this test should be done for the default DUT Switchover". The characterization in this test should be done for
behavior as well as a DUT's non-default configuration that the default DUT behavior and, if it exists, for the DUT's non-
minimizes frame loss, if exists. default configuration that minimizes frame loss.
Procedure Procedure
This test characterizes "RP Switchover". Many implementations This test characterizes RP Switchover. Many implementations allow
allow for optimized switchover capabilities that minimize the for optimized switchover capabilities that minimize the downtime
downtime during the actual switchover. This test consists of two during the actual switchover. This test consists of two sub-cases
sub-cases from a switchover characteristics standpoint: First, a from a switchover characteristic's standpoint: first, a default
default behavior (with no switchover-specific configurations); and behavior (with no switchover-specific configurations) and,
potentially second, a non-default behavior with switchover potentially second, a non-default behavior with switchover
configuration to minimize frame loss. Therefore, the procedures configuration to minimize frame loss. Therefore, the procedures
hereby described are executed twice, and reported separately. hereby described are executed twice and reported separately.
First, ensure that the RPs are in a permanent state such that the First, ensure that the RPs are in a permanent state such that the
secondary will be activated to the same state as the active is, by secondary RP will be activated to the same state as the active RP
performing some or all of the following operational tasks: save by performing some or all of the following operational tasks: save
the current DUT configuration, specify boot parameters, ensure the the current DUT configuration, specify boot parameters, ensure the
appropriate software files are available, or perform additional appropriate software files are available, or perform additional
Operating System or hardware related task. operating system or hardware-related tasks.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing, and the rate should be sufficient for the DUT
attain the maximum forwarding throughput. This enables a finer to attain the maximum forwarding throughput. This enables a finer
granularity in the reset time measurement. granularity in the reset time measurement.
Third, perform the primary Route Processor (RP) hardware reset at Third, perform the primary Route Processor (RP) hardware reset at
this point. This entails for example physically removing the RP, this point. This entails, for example, physically removing the RP
or triggering a hardware reset by other means (e.g., command line or triggering a hardware reset by other means (e.g., command line
interface, physical switch, etc.) It is up to the operator to interface, physical switch, etc.). It is up to the operator to
decide if the primary RP needs to be re-inserted after a grace decide whether or not the primary RP needs to be re-inserted after
period or not. a grace period.
Finally, the characterization is completed by recording the frame Finally, complete the characterization by recording the frame loss
loss or time stamps (as reported by the test tool) and calculating or timestamps (as reported by the test tool) and calculating the
the reset time (as defined in Section 2.3). reset time (as defined in Section 1.3).
Reporting format Reporting Format
The reset results are potentially reported twice, one for the The reset results are potentially reported twice, one for the
default switchover behavior (i.e., the DUT without any switchover- default switchover behavior (i.e., the DUT without any switchover-
specific enhanced configuration) and the other for the switchover- specific enhanced configuration) and the other for the switchover-
specific behavior if it exists (i.e., the DUT configured for specific behavior if it exists (i.e., the DUT configured for
optimized switchover capabilities that minimize the downtime optimized switchover capabilities that minimize the downtime
during the actual switchover). during the actual switchover).
The reporting format is defined in Section 2.4, and also includes The reporting format is defined in Section 1.4 and also includes
any specific redundancy scheme in place. any specific redundancy scheme in place.
4.1.2. Line Card (LC) Removal and Insertion (REQUIRED) 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED)
The Line Card (LC) is the DUT component that is responsible with The Line Card (LC) is the DUT component that is responsible for
packet forwarding. packet forwarding.
Objective Objective
To characterize time needed for a DUT to recover from a Line Card To characterize the time needed for a DUT to recover from a line-
removal and insertion event. card removal and insertion event.
Procedure Procedure
For this test, the Line Card that is being hardware-reset MUST be For this test, the line card that is being hardware-reset MUST be
on the forwarding path and all destinations MUST be directly on the forwarding path, and all destinations MUST be directly
connected. connected.
First, complete some or all of the following operational tasks: First, complete some or all of the following operational tasks:
save the current DUT configuration, specify boot parameters, save the current DUT configuration, specify boot parameters,
ensure the appropriate software files are available, or perform ensure the appropriate software files are available, or perform
additional Operating System or hardware related task. additional operating system or hardware-related tasks.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing, and the rate should be sufficient for the DUT
attain the maximum forwarding throughput. This enables a finer to attain the maximum forwarding throughput. This enables a finer
granularity in the reset time measurement. granularity in the reset time measurement.
Third, perform the Line Card (LC) hardware reset at this point. Third, perform the Line Card (LC) hardware reset at this point.
This entails for example physically removing the LC to later re- This entails, for example, physically removing the LC to later re-
insert it, or triggering a hardware reset by other means (e.g., insert it or triggering a hardware reset by other means (e.g.,
CLI, physical switch, etc.). CLI, physical switch, etc.).
Finally, the characterization is completed by recording the frame Finally, complete the characterization by recording the frame loss
loss or time stamps (as reported by the test tool) and calculating or timestamps (as reported by the test tool) and calculating the
the reset time (as defined in Section 2.3). reset time (as defined in Section 1.3).
Reporting Format Reporting Format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 1.4.
4.2. Software Reset Test 4.2. Software Reset Tests
To characterize time needed for a DUT to recover from the software A software reset test characterizes the time needed for a DUT to
reset. recover from a software reset.
In contrast to a "hardware reset", a "software reset" involves only In contrast to a hardware reset, a software reset involves only the
the re-initialization of the execution, data structures, and partial re-initialization of the execution, data structures, and partial
state within the software running on the DUT module(s). state within the software running on the DUT module(s).
A software reset is initiated for example from the DUT's CLI. A software reset is initiated, for example, from the DUT's CLI.
4.2.1. Operating System (OS) Reset (REQUIRED) 4.2.1. Operating System (OS) Reset (REQUIRED)
Objective Objective
To characterize time needed for a DUT to recover from an Operating To characterize the time needed for a DUT to recover from an
System (OS) software reset. operating system (OS) software reset.
Procedure Procedure
First, complete some or all of the following operational tasks: First, complete some or all of the following operational tasks:
save the current DUT configuration, specify software boot save the current DUT configuration, specify software boot
parameters, ensure the appropriate software files are available, parameters, ensure the appropriate software files are available,
or perform additional Operating System task. or perform additional operating system tasks.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing, and the rate should be sufficient for the DUT
attain the maximum forwarding throughput. This enables a finer to attain the maximum forwarding throughput. This enables a finer
granularity in the reset time measurement. granularity in the reset time measurement.
Third, trigger an Operating System re-initialization in the DUT, Third, trigger an operating system re-initialization in the DUT by
by operational means such as use of the DUT's CLI or other operational means such as use of the DUT's CLI or other management
management interface. interface.
Finally, the characterization is completed by recording the frame Finally, complete the characterization by recording the frame loss
loss or time stamps (as reported by the test tool) and calculating or timestamps (as reported by the test tool) and calculating the
the reset time (as defined in Section 2.3). reset time (as defined in Section 1.3).
Reporting format Reporting Format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 1.4.
4.2.2. Process Reset (OPTIONAL) 4.2.2. Process Reset (OPTIONAL)
Objective Objective
To characterize time needed for a DUT to recover from a software To characterize the time needed for a DUT to recover from a
process reset. software process reset.
Such time period may depend upon the number and types of process Such a time period may depend upon the number and types of
running in the DUT and which ones are tested. Different processes running in the DUT and which ones are tested. Different
implementations of forwarding devices include various common implementations of forwarding devices include various common
processes. A process reset should be performed only in the processes. A process reset should be performed only in the
processes most relevant to the tester and most impactful to processes most relevant to the tester and most impactful to
forwarding. forwarding.
Procedure Procedure
First, complete some or all of the following operational tasks: First, complete some or all of the following operational tasks:
save the current DUT configuration, specify software parameters or save the current DUT configuration, specify software parameters or
environmental variables, or perform additional Operating System environmental variables, or perform additional operating system
task. tasks.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing, and the rate should be sufficient for the DUT
attain the maximum forwarding throughput. This enables a finer to attain the maximum forwarding throughput. This enables a finer
granularity in the reset time measurement. granularity in the reset time measurement.
Third, trigger a process reset for each process running in the DUT Third, trigger a process reset for each process running in the DUT
and considered for testing from a management interface (e.g., by and considered for testing from a management interface (e.g., by
means of the CLI, etc.) means of the CLI, etc.).
Finally, the characterization is completed by recording the frame Finally, complete the characterization by recording the frame loss
loss or time stamps (as reported by the test tool) and calculating or timestamps (as reported by the test tool) and calculating the
the reset time (as defined in Section 2.3). reset time (as defined in Section 1.3).
Reporting format Reporting Format
The reporting format is defined in Section 2.4, and is used for The reporting format is defined in Section 1.4 and is used for
each process running in the DUT and tested. Given the each process running in the DUT and tested. Given the
implementation nature of this test, details of the actual process implementation nature of this test, details of the actual process
tested should be included along with the statement. tested should be included along with the statement.
4.3. Power Interruption Test 4.3. Power Interruption Test
"Power interruption" refers to the complete loss of power on the "Power interruption" refers to the complete loss of power on the DUT.
DUT. It can be viewed as a special case of a hardware reset, It can be viewed as a special case of a hardware reset, triggered by
triggered by the loss of the power supply to the DUT or its the loss of the power supply to the DUT or its components, and is
components, and is characterized by the re-initialization of all characterized by the re-initialization of all hardware and software
hardware and software in the DUT. in the DUT.
4.3.1. Power Interruption (REQUIRED) 4.3.1. Power Interruption (REQUIRED)
Objective Objective
To characterize time needed for a DUT to recover from a complete To characterize the time needed for a DUT to recover from a
loss of electric power or complete power interruption. This test complete loss of electric power or complete power interruption.
simulates a complete power failure or outage, and should be This test simulates a complete power failure or outage and should
indicative of the DUT/SUTs behavior during such event. be indicative of the DUT/SUT's behavior during such event.
Procedure Procedure
First, ensure that the entire DUT is at a permanent state to which First, ensure that the entire DUT is at a permanent state to which
it will return to after the power interruption, by performing some it will return after the power interruption by performing some or
or all of the following operational tasks: save the current DUT all of the following operational tasks: save the current DUT
configuration, specify boot parameters, ensure the appropriate configuration, specify boot parameters, ensure the appropriate
software files are available, or perform additional Operating software files are available, or perform additional operating
System or hardware related task. system or hardware-related tasks.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing, and the rate should be sufficient for the DUT
attain the maximum forwarding throughput. This enables a finer to attain the maximum forwarding throughput. This enables a finer
granularity in the reset time measurement. granularity in the reset time measurement.
Third, interrupt the power (AC or DC) that feeds the corresponding Third, interrupt the power (AC or DC) that feeds the corresponding
DUTs power supplies at this point. This entails for example DUT's power supplies at this point. This entails, for example,
physically removing the power supplies in the DUT to later re- physically removing the power supplies in the DUT to later re-
insert them, or simply disconnecting or switching off their power insert them or simply disconnecting or switching off their power
feeds (AC or DC as applicable). The actual power interruption feeds (AC or DC, as applicable). The actual power interruption
should last at least 15 seconds. should last at least 15 seconds.
Finally, the characterization is completed by recording the frame Finally, complete the characterization by recording the frame loss
loss or time stamps (as reported by the test tool) and calculating or timestamps (as reported by the test tool) and calculating the
the reset time (as defined in Section 2.3). reset time (as defined in Section 1.3).
For easier comparison with other testing, the 15 seconds are For easier comparison with other testing, 15 seconds are removed
removed from the reported reset time. from the reported reset time.
Reporting format Reporting Format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 1.4.
5. Security Considerations 5. Security Considerations
Benchmarking activities, as described in this memo, are limited to Benchmarking activities, as described in this document, are limited
technology characterization using controlled stimuli in a laboratory to technology characterization using controlled stimuli in a
environment, with dedicated address space and the constraints laboratory environment, with dedicated address space and the
specified in the sections above. constraints specified in the sections above.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Furthermore, benchmarking is performed on a "black-box" basis, Furthermore, benchmarking is performed on a "black-box" basis,
relying solely on measurements observable external to the DUT/SUT. relying solely on measurements observable externally to the DUT/SUT.
Special capabilities SHOULD NOT exist in the DUT/SUT specifically Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
for benchmarking purposes. Any implications for network security benchmarking purposes. Any implications for network security arising
arising from the DUT/SUT SHOULD be identical in the lab and in from the DUT/SUT SHOULD be identical in the lab and in production
production networks. networks.
There are no specific security considerations within the scope of There are no specific security considerations within the scope of
this document. this document.
6. IANA Considerations 6. Acknowledgments
There is no IANA consideration for this document.
7. Acknowledgments
The authors would like to thank Ron Bonica, who motivated us to The authors would like to thank Ron Bonica, who motivated us to write
write this document. The authors would also like to thank Al Morton, this document. The authors would also like to thank Al Morton,
Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player, Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player,
Jan Novak, Steve Maxwell, Ilya Varlashkin, and Sarah Banks for Jan Novak, Steve Maxwell, Ilya Varlashkin, and Sarah Banks for
providing thorough review, useful suggestions, and valuable input. providing thorough review, useful suggestions, and valuable input.
This document was prepared using 2-Word-v2.0.template.dot. 7. References
8. References
8.1. Normative References 7.1. Normative References
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking Terminology for Network
interconnection devices", RFC 1242, July 1991. Interconnection Devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
8.2. Informative References 7.2. Informative References
[IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking [IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking
Methodology for Link-State IGP Data Plane Route Methodology for Link-State IGP Data Plane Route
Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-21 Convergence", Work in Progress, February 2011.
(work in progress), May 2010.
[RFC5180] Popoviciu, C., et al, "IPv6 Benchmarking Methodology for [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D.
Network Interconnect Devices", RFC 5180, May 2008. Dugatkin, "IPv6 Benchmarking Methodology for Network
Interconnect Devices", RFC 5180, May 2008.
[RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding
Benchmarking Methodology for IP Flows", RFC 5695, November Benchmarking Methodology for IP Flows", RFC 5695, November
2009. 2009.
Authors' Addresses Authors' Addresses
Rajiv Asati Rajiv Asati
Cisco Systems Cisco Systems
7025-6 Kit Creek Road 7025-6 Kit Creek Road
RTP, NC 27709 Research Triangle Park, NC 27709
USA USA
Email: rajiva@cisco.com EMail: rajiva@cisco.com
Carlos Pignataro Carlos Pignataro
Cisco Systems Cisco Systems
7200-12 Kit Creek Road 7200-12 Kit Creek Road
RTP, NC 27709 Research Triangle Park, NC 27709
USA USA
Email: cpignata@cisco.com EMail: cpignata@cisco.com
Fernando Calabria Fernando Calabria
Cisco Systems Cisco Systems
7200-12 Kit Creek Road 7200-12 Kit Creek Road
RTP, NC 27709 Research Triangle Park, NC 27709
USA USA
Email: fcalabri@cisco.com EMail: fcalabri@cisco.com
Cesar Olvera Cesar Olvera Morales
Consulintel Consulintel
Joaquin Turina, 2 Joaquin Turina, 2
Pozuelo de Alarcon, Madrid, E-28224 Pozuelo de Alarcon, Madrid, E-28224
Spain Spain
Email: cesar.olvera@consulintel.es EMail: cesar.olvera@consulintel.es
 End of changes. 165 change blocks. 
484 lines changed or deleted 460 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/