draft-ietf-bmwg-reset-05.txt   draft-ietf-bmwg-reset-06.txt 
Benchmarking Methodology WG Rajiv Asati Benchmarking Methodology WG Rajiv Asati
Internet Draft Cisco Internet Draft Cisco
Updates: 2544 (if approved) Carlos Pignataro Updates: 1242, 2544 (if approved) Carlos Pignataro
Intended status: Informational Cisco Intended status: Informational Cisco
Expires: June 2011 Fernando Calabria Expires: June 2011 Fernando Calabria
Cisco Cisco
Cesar Olvera Cesar Olvera
Consulintel Consulintel
December 18, 2010 December 20, 2010
Device Reset Characterization Device Reset Characterization
draft-ietf-bmwg-reset-05 draft-ietf-bmwg-reset-06
Abstract Abstract
An operational forwarding device may need to be re-started An operational forwarding device may need to be re-started
(automatically or manually) for a variety of reasons, an event that (automatically or manually) for a variety of reasons, an event that
we call a "reset" in this document. Since there may be an we call a "reset" in this document. Since there may be an
interruption in the forwarding operation during a reset, it is interruption in the forwarding operation during a reset, it is
useful to know how long a device takes to resume the forwarding useful to know how long a device takes to resume the forwarding
operation. operation.
This document specifies a methodology for characterizing reset (and This document specifies a methodology for characterizing reset (and
recovery time) during benchmarking of forwarding devices, and reset time) during benchmarking of forwarding devices, and provides
provides clarity and consistency in reset test procedures beyond clarity and consistency in reset test procedures beyond what's
what's specified in RFC2544. It therefore updates RFC2544. specified in RFC2544. It therefore updates RFC2544. This document
also defines the benchmarking term "Reset Time" and only in this
updates RFC1242.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 2, line 4 skipping to change at page 2, line 6
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
This Internet-Draft will expire on June 18, 2011. This Internet-Draft will expire on June 20, 2011.
Copyright Notice Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 10 skipping to change at page 3, line 10
respect to this document. Code Components extracted from this respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License. warranty as described in the Simplified BSD License.
Table of Contents Table of Contents
1. Key Words to Reflect Requirements..............................4 1. Key Words to Reflect Requirements..............................4
2. Introduction...................................................4 2. Introduction...................................................4
2.1. Scope.....................................................4 2.1. Scope.....................................................4
2.2. Reset Characterization....................................5 2.2. Reset Time................................................5
2.3. Recovery Time Measurement Methods.........................6 2.3. Reset Time Measurement Methods............................6
2.4. Reporting Format..........................................7 2.4. Reporting Format..........................................7
3. Test Requirements..............................................8 3. Test Requirements..............................................8
4. Reset Test.....................................................9 4. Reset Test.....................................................9
4.1. Hardware Reset Test......................................10 4.1. Hardware Reset Test......................................10
4.1.1. Routing Processor (RP) / Routing Engine Reset.......10 4.1.1. Routing Processor (RP) / Routing Engine Reset.......10
4.1.1.1. RP Reset for a single-RP device (REQUIRED).....10 4.1.1.1. RP Reset for a single-RP device (REQUIRED).....11
4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL)
........................................................11 ........................................................11
4.1.2. Line Card (LC) Removal and Insertion (REQUIRED).....13 4.1.2. Line Card (LC) Removal and Insertion (REQUIRED).....13
4.2. Software Reset Test......................................13 4.2. Software Reset Test......................................14
4.2.1. Operating System (OS) Reset (REQUIRED)..............14 4.2.1. Operating System (OS) Reset (REQUIRED)..............14
4.2.2. Process Reset (OPTIONAL)............................14 4.2.2. Process Reset (OPTIONAL)............................15
4.3. Power Interruption Test..................................15 4.3. Power Interruption Test..................................16
4.3.1. Power Interruption (REQUIRED).......................16 4.3.1. Power Interruption (REQUIRED).......................16
5. Security Considerations.......................................17 5. Security Considerations.......................................17
6. IANA Considerations...........................................17 6. IANA Considerations...........................................17
7. Acknowledgments...............................................17 7. Acknowledgments...............................................17
8. References....................................................18 8. References....................................................19
8.1. Normative References.....................................18 8.1. Normative References.....................................19
8.2. Informative References...................................18 8.2. Informative References...................................19
Authors' Addresses...............................................19 Authors' Addresses...............................................20
1. Key Words to Reflect Requirements 1. Key Words to Reflect Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 document are to be interpreted as described in BCP 14, RFC 2119
[RFC2119]. RFC 2119 defines the use of these key words to help make [RFC2119]. RFC 2119 defines the use of these key words to help make
the intent of standards track documents as clear as possible. While the intent of standards track documents as clear as possible. While
this document uses these keywords, this document is not a standards this document uses these keywords, this document is not a standards
track document. track document.
2. Introduction 2. Introduction
An operational forwarding device (or one of its components) may need An operational forwarding device (or one of its components) may need
to be re-started for a variety of reasons, an event that we call a to be re-started for a variety of reasons, an event that we call a
"reset" in this document. Since there may be an interruption in the "reset" in this document. Since there may be an interruption in the
forwarding operation during a reset, it is useful to know how long a forwarding operation during a reset, it is useful to know how long a
device takes to resume the forwarding operation. In other words, it device takes to resume the forwarding operation. In other words, it
is desired to know the duration of the recovery time following the is desired to know the duration of the recovery time following the
reset. reset (reset time, see Section 2.2).
However, the answer to this question is no longer simple and However, the answer to this question is no longer simple and
straight-forward as the modern forwarding devices employ many straight-forward as the modern forwarding devices employ many
hardware advancements (distributed forwarding, etc.) and software hardware advancements (distributed forwarding, etc.) and software
advancements (graceful restart, etc.) that influence the recovery advancements (graceful restart, etc.) that influence the recovery
time after the reset. time after the reset.
2.1. Scope 2.1. Scope
This document specifies a methodology for characterizing reset (and This document specifies a methodology for characterizing reset (and
recovery time) during benchmarking of forwarding devices, and reset time) during benchmarking of forwarding devices, and provides
provides clarity and consistency in reset procedures beyond what is clarity and consistency in reset procedures beyond what is specified
specified in [RFC2544]. Software upgrades involve additional in [RFC2544]. Software upgrades involve additional benchmarking
benchmarking complexities and are outside the scope of this complexities and are outside the scope of this document.
document.
These procedures may be used by other benchmarking documents such as These procedures may be used by other benchmarking documents such as
[RFC2544], [RFC5180], [RFC5695], etc., and is expected that other [RFC2544], [RFC5180], [RFC5695], etc., and is expected that other
protocol-specific benchmarking documents would reference this protocol-specific benchmarking documents would reference this
document for reset recovery time characterization. Specific Routing document for reset recovery time characterization. Specific Routing
Information Base (RIB) and Forwarding Information Base (FIB) scaling Information Base (RIB) and Forwarding Information Base (FIB) scaling
considerations are outside the scope of this document and can be considerations are outside the scope of this document and can be
quite complex to characterize. However, other documents can quite complex to characterize. However, other documents can
characterize specific dynamic protocols scaling and interactions and characterize specific dynamic protocols scaling and interactions and
leverage and augment the reset tests defined in this document. leverage and augment the reset tests defined in this document.
This document updates Section 26.6 of [RFC2544]. This document updates Section 26.6 of [RFC2544], and defines the
benchmarking term "Reset Time" updating [RFC1242].
This document focuses only on the reset criterion of benchmarking, This document focuses only on the reset criterion of benchmarking,
and presumes that it would be beneficial to [RFC5180], [RFC5695], and presumes that it would be beneficial to [RFC5180], [RFC5695],
and other IETF Benchmarking Methodology Working Group (BMWG) and other IETF Benchmarking Methodology Working Group (BMWG)
efforts. efforts.
2.2. Reset Characterization 2.2. Reset Time
Definition Definition
Reset Time is the total time when a device is determined to be out Reset Time is the total time when a device is determined to be out
of operation, and includes the time to perform the reset and the of operation, and includes the time to perform the reset and the
time to recover from it. time to recover from it.
Discussion Discussion
During a period of time after a reset or power up, network devices During a period of time after a reset or power up, network devices
skipping to change at page 6, line 4 skipping to change at page 5, line 47
from the start of the event until full recovery of forwarding from the start of the event until full recovery of forwarding
after the completion of the reset events. after the completion of the reset events.
Measurement units Measurement units
Time in milliseconds, providing sufficient resolution to Time in milliseconds, providing sufficient resolution to
distinguish between different trials and different distinguish between different trials and different
implementations. See Section 2.4. implementations. See Section 2.4.
Issues Issues
There are various types of Reset: Hardware resets, software There are various types of Reset: Hardware resets, software
resets, and power interruption. See Section 4. resets, and power interruption. See Section 4.
2.3. Recovery Time Measurement Methods See Also
The 'recovery time' is the time during which the traffic forwarding This definition updates [RFC1242].
is temporarily interrupted following a reset event. Strictly
speaking, this is the time over which one or more frames are lost. 2.3. Reset Time Measurement Methods
This definition is similar to that of 'Loss of connectivity period'
The 'reset time' is the time during which the traffic forwarding is
temporarily interrupted following a reset event. Strictly speaking,
this is the time over which one or more frames are lost. This
definition is similar to that of 'Loss of connectivity period'
defined in [IGPConv] section 4. defined in [IGPConv] section 4.
There are two accepted methods to measure the 'recovery time': There are two accepted methods to measure the 'reset time':
1. Frame-Loss Method - This method requires test tool capability 1. Frame-Loss Method - This method requires test tool capability
to monitor the number of lost frames. In this method, the to monitor the number of lost frames. In this method, the
offered stream rate (frames per second) must be known. The offered stream rate (frames per second) must be known. The
recovery time is calculated per the below equation: reset time is calculated per the below equation:
Frames_lost (packets) Frames_lost (packets)
Recovery_time = ------------------------------------- Reset_time = -------------------------------------
Offered_rate (packets per second) Offered_rate (packets per second)
2. Time-Stamp Method - This method requires test tool capability 2. Time-Stamp Method - This method requires test tool capability
to timestamp each frame. In this method, the test tool to timestamp each frame. In this method, the test tool
timestamps each transmitted frame and monitors the received timestamps each transmitted frame and monitors the received
frame's timestamp. During the test, the test tool would record frame's timestamp. During the test, the test tool would record
the timestamp (Timestamp A) of the frame that was last the timestamp (Timestamp A) of the frame that was last
received prior to the reset interruption and the timestamp received prior to the reset interruption and the timestamp
(Timestamp B) of the first frame after the interruption (Timestamp B) of the first frame after the interruption
stopped. The difference between Timestamp B and Timestamp A is stopped. The difference between Timestamp B and Timestamp A is
the recovery time. the reset time.
The tester / operator MAY use either method for recovery time The tester / operator MAY use either method for reset time
measurement depending on the test tool capability. However, the measurement depending on the test tool capability. However, the
Frame-loss method SHOULD be used if the test tool is capable of (a) Frame-loss method SHOULD be used if the test tool is capable of (a)
counting the number of lost frames per stream, and (b) transmitting counting the number of lost frames per stream, and (b) transmitting
test frame despite the physical link status, whereas Time-stamp test frame despite the physical link status, whereas Time-stamp
method SHOULD be used if the test tool is capable of (a) time- method SHOULD be used if the test tool is capable of (a) time-
stamping each frame, (b) monitoring received frame's timestamp, and stamping each frame, (b) monitoring received frame's timestamp, and
(c) transmitting frames only if the physical link status is UP. That (c) transmitting frames only if the physical link status is UP. That
is, specific test tool capabilities may dictate which method to use. is, specific test tool capabilities may dictate which method to use.
If the test tool supports both methods based on its capabilities, If the test tool supports both methods based on its capabilities,
the tester / operator SHOULD use the one that provides more the tester / operator SHOULD use the one that provides more
accuracy. accuracy.
2.4. Reporting Format 2.4. Reporting Format
All reset results are reported in a simple statement including the All reset results are reported in a simple statement including the
frame loss (if measured) and recovery times. frame loss (if measured) and reset times.
For each test case, it is RECOMMENDED that the following parameters For each test case, it is RECOMMENDED that the following parameters
be reported in these units: be reported in these units:
Parameter Units or Examples Parameter Units or Examples
----------------------------------------------------------------- -----------------------------------------------------------------
Throughput Frames per second and bits per Throughput Frames per second and bits per
second second
Loss (average) Frames Loss (average) Frames
Recovery Time (average) Milliseconds Reset Time (average) Milliseconds
Number of trials Integer count Number of trials Integer count
Protocol IPv4, IPv6, MPLS, etc. Protocol IPv4, IPv6, MPLS, etc.
Frame Size Octets Frame Size Octets
Port Media Ethernet, GigE (Gigabit Ethernet), Port Media Ethernet, GigE (Gigabit Ethernet),
POS (Packet over SONET), etc. POS (Packet over SONET), etc.
skipping to change at page 8, line 5 skipping to change at page 8, line 8
For mixed protocol environments, frames SHOULD be distributed For mixed protocol environments, frames SHOULD be distributed
between all the different protocols. The distribution MAY between all the different protocols. The distribution MAY
approximate the network conditions of deployment. In all cases the approximate the network conditions of deployment. In all cases the
details of the mixed protocol distribution MUST be included in the details of the mixed protocol distribution MUST be included in the
reporting. reporting.
Additionally, the DUT (Device Under Test) / SUT (System Under Test) Additionally, the DUT (Device Under Test) / SUT (System Under Test)
and test bed provisioning, port and Line Card arrangement, and test bed provisioning, port and Line Card arrangement,
configuration, and deployed methodologies that may influence the configuration, and deployed methodologies that may influence the
overall recovery time MUST be listed. (Refer to the additional overall reset time MUST be listed. (Refer to the additional factors
factors listed in Section 3). listed in Section 3).
The reporting of results MUST regard repeatability considerations The reporting of results MUST regard repeatability considerations
from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple from Section 4 of [RFC2544]. It is RECOMMENDED to perform multiple
trials and report average results. trials and report average results.
3. Test Requirements 3. Test Requirements
Tests SHOULD first be performed such that the forwarding state re- Tests SHOULD first be performed such that the forwarding state re-
establishment is independent from an external source (i.e., using establishment is independent from an external source (i.e., using
static address resolution, routing and forwarding configuration, and static address resolution, routing and forwarding configuration, and
skipping to change at page 8, line 45 skipping to change at page 9, line 4
3. Scheduled vs. non-scheduled reset 3. Scheduled vs. non-scheduled reset
4. Local vs. Remote reset 4. Local vs. Remote reset
5. Scale - Number of line cards present vs. in-use 5. Scale - Number of line cards present vs. in-use
6. Scale - Number of physical and logical interfaces 6. Scale - Number of physical and logical interfaces
7. Scale - Number of routing protocol instances 7. Scale - Number of routing protocol instances
8. Scale - Number of Routing Table entries 8. Scale - Number of Routing Table entries
9. Scale - Number of Route Processors available 9. Scale - Number of Route Processors available
10. Performance - Redundancy strategy deployed for route 10. Performance - Redundancy strategy deployed for route
processors and line cards processors and line cards
11. Performance - Interface encapsulation as well as achievable 11. Performance - Interface encapsulation as well as achievable
Throughput [RFC2544] Throughput [RFC2544]
12. Any other internal or external factor that may influence 12. Any other internal or external factor that may influence reset
recovery time after a hardware or software reset time after a hardware or software reset
The recovery time is one of the key characterization results The reset time is one of the key characterization results reported
reported after each test run. While the recovery time during a reset after each test run. While the reset time during a reset test event
test event may be zero, there may still be effects on traffic, such may be zero, there may still be effects on traffic, such as
as transient delay variation or increased latency. However, that is transient delay variation or increased latency. However, that is not
not covered and deemed outside the scope of this document. In this covered and deemed outside the scope of this document. In this case,
case, only "no loss" is reported. only "no loss" is reported.
4. Reset Test 4. Reset Test
This section contains the description of the tests that are related This section contains the description of the tests that are related
to the characterization of the time needed for DUTs (Device Under to the characterization of the time needed for DUTs (Device Under
Test) / SUTs (System Under Test) to recover from a reset. There are Test) / SUTs (System Under Test) to recover from a reset. There are
three types of reset considered in this document: three types of reset considered in this document:
1. Hardware resets 1. Hardware resets
skipping to change at page 10, line 27 skipping to change at page 10, line 31
Reset procedures that do not require the physical removal and Reset procedures that do not require the physical removal and
insertion of a hardware component are RECOMMENDED. These include insertion of a hardware component are RECOMMENDED. These include
using the Command Line Interface (CLI) or a physical switch or using the Command Line Interface (CLI) or a physical switch or
button. If such procedures cannot be performed (e.g., for lack of button. If such procedures cannot be performed (e.g., for lack of
platform support, or because the corresponding Test Case calls for platform support, or because the corresponding Test Case calls for
them), human operation time SHOULD be minimized across different them), human operation time SHOULD be minimized across different
platforms and Test Cases as much as possible, and variation in human platforms and Test Cases as much as possible, and variation in human
operator time SHOULD also be minimized across different vendors operator time SHOULD also be minimized across different vendors
products as much as practical, by having the same person perform the products as much as practical, by having the same person perform the
operation, and by practicing the operation. operation, and by practicing the operation. Additionally, the time
between removal and insertion SHOULD be recorded and reported.
For routers that do not contain separate Routing Processor and Line For routers that do not contain separate Routing Processor and Line
Card modules, the hardware reset tests are not performed since they Card modules, the hardware reset tests are not performed since they
are not relevant; instead, the power interruption tests MUST be are not relevant; instead, the power interruption tests MUST be
performed (see Section 4.3) in these cases. performed (see Section 4.3) in these cases.
4.1.1. Routing Processor (RP) / Routing Engine Reset 4.1.1. Routing Processor (RP) / Routing Engine Reset
The Routing Processor (RP) is the DUT module that is primarily The Routing Processor (RP) is the DUT module that is primarily
concerned with Control Plane functions. concerned with Control Plane functions.
skipping to change at page 11, line 19 skipping to change at page 11, line 26
following operational tasks: save the current DUT configuration, following operational tasks: save the current DUT configuration,
specify boot parameters, ensure the appropriate software files are specify boot parameters, ensure the appropriate software files are
available, or perform additional Operating System or hardware available, or perform additional Operating System or hardware
related task. related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the reset time measurement.
Third, perform the Route Processor (RP) hardware reset at this Third, perform the Route Processor (RP) hardware reset at this
point. This entails for example physically removing the RP to point. This entails for example physically removing the RP to
later re-insert it, or triggering a hardware reset by other means later re-insert it, or triggering a hardware reset by other means
(e.g., command line interface, physical switch, etc.) (e.g., command line interface, physical switch, etc.)
Finally, the characterization is completed by recording the frame Finally, the characterization is completed by recording the frame
loss or time stamps (as reported by the test tool) and calculating loss or time stamps (as reported by the test tool) and calculating
the recovery time (as defined in Section 2.3). the reset time (as defined in Section 2.3).
Reporting format Reporting format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 2.4.
4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL) 4.1.1.2. RP Switchover for a multiple-RP device (OPTIONAL)
Objective Objective
To characterize time needed for "secondary" Route Processor To characterize time needed for "secondary" Route Processor
skipping to change at page 12, line 23 skipping to change at page 12, line 30
performing some or all of the following operational tasks: save performing some or all of the following operational tasks: save
the current DUT configuration, specify boot parameters, ensure the the current DUT configuration, specify boot parameters, ensure the
appropriate software files are available, or perform additional appropriate software files are available, or perform additional
Operating System or hardware related task. Operating System or hardware related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the reset time measurement.
Third, perform the primary Route Processor (RP) hardware reset at Third, perform the primary Route Processor (RP) hardware reset at
this point. This entails for example physically removing the RP, this point. This entails for example physically removing the RP,
or triggering a hardware reset by other means (e.g., command line or triggering a hardware reset by other means (e.g., command line
interface, physical switch, etc.) It is up to the operator to interface, physical switch, etc.) It is up to the operator to
decide if the primary RP needs to be re-inserted after a grace decide if the primary RP needs to be re-inserted after a grace
period or not. period or not.
Finally, the characterization is completed by recording the frame Finally, the characterization is completed by recording the frame
loss or time stamps (as reported by the test tool) and calculating loss or time stamps (as reported by the test tool) and calculating
the recovery time (as defined in Section 2.3). the reset time (as defined in Section 2.3).
Reporting format Reporting format
The reset results are potentially reported twice, one for the The reset results are potentially reported twice, one for the
default switchover behavior (i.e., the DUT without any switchover- default switchover behavior (i.e., the DUT without any switchover-
specific enhanced configuration) and the other for the switchover- specific enhanced configuration) and the other for the switchover-
specific behavior if it exists (i.e., the DUT configured for specific behavior if it exists (i.e., the DUT configured for
optimized switchover capabilities that minimize the downtime optimized switchover capabilities that minimize the downtime
during the actual switchover). during the actual switchover).
skipping to change at page 13, line 31 skipping to change at page 13, line 34
First, complete some or all of the following operational tasks: First, complete some or all of the following operational tasks:
save the current DUT configuration, specify boot parameters, save the current DUT configuration, specify boot parameters,
ensure the appropriate software files are available, or perform ensure the appropriate software files are available, or perform
additional Operating System or hardware related task. additional Operating System or hardware related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the reset time measurement.
Third, perform the Line Card (LC) hardware reset at this point. Third, perform the Line Card (LC) hardware reset at this point.
This entails for example physically removing the LC to later re- This entails for example physically removing the LC to later re-
insert it, or triggering a hardware reset by other means (e.g., insert it, or triggering a hardware reset by other means (e.g.,
CLI, physical switch, etc.). CLI, physical switch, etc.).
Finally, the characterization is completed by recording the frame Finally, the characterization is completed by recording the frame
loss or time stamps (as reported by the test tool) and calculating loss or time stamps (as reported by the test tool) and calculating
the recovery time (as defined in Section 2.3). the reset time (as defined in Section 2.3).
Reporting Format Reporting Format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 2.4.
4.2. Software Reset Test 4.2. Software Reset Test
To characterize time needed for a DUT to recover from the software To characterize time needed for a DUT to recover from the software
reset. reset.
skipping to change at page 14, line 30 skipping to change at page 14, line 35
First, complete some or all of the following operational tasks: First, complete some or all of the following operational tasks:
save the current DUT configuration, specify software boot save the current DUT configuration, specify software boot
parameters, ensure the appropriate software files are available, parameters, ensure the appropriate software files are available,
or perform additional Operating System task. or perform additional Operating System task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the reset time measurement.
Third, trigger an Operating System re-initialization in the DUT, Third, trigger an Operating System re-initialization in the DUT,
by operational means such as use of the DUT's CLI or other by operational means such as use of the DUT's CLI or other
management interface. management interface.
Finally, the characterization is completed by recording the frame Finally, the characterization is completed by recording the frame
loss or time stamps (as reported by the test tool) and calculating loss or time stamps (as reported by the test tool) and calculating
the recovery time (as defined in Section 2.3). the reset time (as defined in Section 2.3).
Reporting format Reporting format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 2.4.
4.2.2. Process Reset (OPTIONAL) 4.2.2. Process Reset (OPTIONAL)
Objective Objective
To characterize time needed for a DUT to recover from a software To characterize time needed for a DUT to recover from a software
process reset. process reset.
Such time period may depend upon the number and types of process Such time period may depend upon the number and types of process
running in the DUT and which ones are tested. Different running in the DUT and which ones are tested. Different
implementations of forwarding devices include various common implementations of forwarding devices include various common
processes. A process reset should be performed only in the processes. A process reset should be performed only in the
processes most relevant to the tester and most impactful to processes most relevant to the tester and most impactful to
forwarding. forwarding.
skipping to change at page 15, line 26 skipping to change at page 15, line 31
First, complete some or all of the following operational tasks: First, complete some or all of the following operational tasks:
save the current DUT configuration, specify software parameters or save the current DUT configuration, specify software parameters or
environmental variables, or perform additional Operating System environmental variables, or perform additional Operating System
task. task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the reset time measurement.
Third, trigger a process reset for each process running in the DUT Third, trigger a process reset for each process running in the DUT
and considered for testing from a management interface (e.g., by and considered for testing from a management interface (e.g., by
means of the CLI, etc.) means of the CLI, etc.)
Finally, the characterization is completed by recording the frame Finally, the characterization is completed by recording the frame
loss or time stamps (as reported by the test tool) and calculating loss or time stamps (as reported by the test tool) and calculating
the recovery time (as defined in Section 2.3). the reset time (as defined in Section 2.3).
Reporting format Reporting format
The reporting format is defined in Section 2.4, and is used for The reporting format is defined in Section 2.4, and is used for
each process running in the DUT and tested. Given the each process running in the DUT and tested. Given the
implementation nature of this test, details of the actual process implementation nature of this test, details of the actual process
tested should be included along with the statement. tested should be included along with the statement.
4.3. Power Interruption Test 4.3. Power Interruption Test
skipping to change at page 16, line 30 skipping to change at page 16, line 36
or all of the following operational tasks: save the current DUT or all of the following operational tasks: save the current DUT
configuration, specify boot parameters, ensure the appropriate configuration, specify boot parameters, ensure the appropriate
software files are available, or perform additional Operating software files are available, or perform additional Operating
System or hardware related task. System or hardware related task.
Second, ensure that the DUT is able to forward the traffic for at Second, ensure that the DUT is able to forward the traffic for at
least 15 seconds before any test activities are performed. The least 15 seconds before any test activities are performed. The
traffic should use the minimum frame size possible on the media traffic should use the minimum frame size possible on the media
used in the testing and rate should be sufficient for the DUT to used in the testing and rate should be sufficient for the DUT to
attain the maximum forwarding throughput. This enables a finer attain the maximum forwarding throughput. This enables a finer
granularity in the recovery time measurement. granularity in the reset time measurement.
Third, interrupt the power (AC or DC) that feeds the corresponding Third, interrupt the power (AC or DC) that feeds the corresponding
DUTs power supplies at this point. This entails for example DUTs power supplies at this point. This entails for example
physically removing the power supplies in the DUT to later re- physically removing the power supplies in the DUT to later re-
insert them, or simply disconnecting or switching off their power insert them, or simply disconnecting or switching off their power
feeds (AC or DC as applicable). The actual power interruption feeds (AC or DC as applicable). The actual power interruption
should last at least 15 seconds. should last at least 15 seconds.
Finally, the characterization is completed by recording the frame Finally, the characterization is completed by recording the frame
loss or time stamps (as reported by the test tool) and calculating loss or time stamps (as reported by the test tool) and calculating
the recovery time (as defined in Section 2.3). the reset time (as defined in Section 2.3).
For easier comparison with other testing, the 15 seconds are For easier comparison with other testing, the 15 seconds are
removed from the reported recovery time. removed from the reported reset time.
Reporting format Reporting format
The reporting format is defined in Section 2.4. The reporting format is defined in Section 2.4.
5. Security Considerations 5. Security Considerations
Benchmarking activities, as described in this memo, are limited to Benchmarking activities, as described in this memo, are limited to
technology characterization using controlled stimuli in a laboratory technology characterization using controlled stimuli in a laboratory
environment, with dedicated address space and the constraints environment, with dedicated address space and the constraints
skipping to change at page 18, line 9 skipping to change at page 19, line 9
Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player, Andrew Yourtchenko, David Newman, John E. Dawson, Timmons C. Player,
Jan Novak, Steve Maxwell, Ilya Varlashkin, and Sarah Banks for Jan Novak, Steve Maxwell, Ilya Varlashkin, and Sarah Banks for
providing thorough review, useful suggestions, and valuable input. providing thorough review, useful suggestions, and valuable input.
This document was prepared using 2-Word-v2.0.template.dot. This document was prepared using 2-Word-v2.0.template.dot.
8. References 8. References
8.1. Normative References 8.1. Normative References
[RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and McQuaid, J., "Benchmarking Methodology for [RFC2544] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
8.2. Informative References 8.2. Informative References
[IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking [IGPConv] Poretsky, S., Imhoff, B., and K. Michielsen, "Benchmarking
Methodology for Link-State IGP Data Plane Route Methodology for Link-State IGP Data Plane Route
 End of changes. 47 change blocks. 
65 lines changed or deleted 77 lines changed or added

This html diff was produced by rfcdiff 1.40. The latest version is available from http://tools.ietf.org/tools/rfcdiff/