draft-ietf-bmwg-issu-meth-00.txt   draft-ietf-bmwg-issu-meth-01.txt 
Benchmarking Working Group Sarah Banks Benchmarking Working Group Sarah Banks
Internet Draft VSS Monitoring Internet Draft VSS Monitoring
Intended status: Informational Fernando Calabria Intended status: Informational Fernando Calabria
Expires: September 9, 2015 Cisco Expires: November 30, 2015 Cisco Systems
Gery Czirjak Gery Czirjak
Ramdas Machat Ramdas Machat
Juniper Juniper Networks
March 9, 2015 May 30, 2015
ISSU Benchmarking Methodology ISSU Benchmarking Methodology
draft-ietf-bmwg-issu-meth-00 draft-ietf-bmwg-issu-meth-01
Abstract Abstract
Modern forwarding devices attempt to minimize any control and data Modern forwarding devices attempt to minimize any control and data
plane disruptions while performing planned software changes, by plane disruptions while performing planned software changes, by
implementing a technique commonly known as In Service Software implementing a technique commonly known as In Service Software
Upgrade (ISSU) This document specifies a set of common methodologies Upgrade (ISSU) This document specifies a set of common methodologies
and procedures designed to characterize the overall behavior of a and procedures designed to characterize the overall behavior of a
Device Under Test (DUT), subject to an ISSU event. Device Under Test (DUT), subject to an ISSU event.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six
and may be updated, replaced, or obsoleted by other documents at any months and may be updated, replaced, or obsoleted by other
time. It is inappropriate to use Internet-Drafts as reference documents at any time. It is inappropriate to use Internet-Drafts
material or to cite them other than as "work in progress." as reference material or to cite them other than as
"work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
This Internet-Draft will expire on September 6, 2015. This Internet-Draft will expire on September 6, 2015.
Copyright Notice Copyright Notice
skipping to change at page 3, line 21 skipping to change at page 3, line 21
9. IANA Considerations...........................................16 9. IANA Considerations...........................................16
10. References...................................................16 10. References...................................................16
10.1. Normative References....................................16 10.1. Normative References....................................16
10.2. Informative References..................................16 10.2. Informative References..................................16
11. Acknowledgments..............................................16 11. Acknowledgments..............................................16
1. Introduction 1. Introduction
As required by most Service Provider (SP) network operators, ISSU As required by most Service Provider (SP) network operators, ISSU
functionality has been implemented by modern forwarding devices to functionality has been implemented by modern forwarding devices to
upgrade or downgrade from one software version to another with a upgrade or downgrade from one software version to another with a
goal of eliminating the downtime of the router and/or the outage goal of eliminating the downtime of the router and/or the outage
of service. However, it is noted that while most operators desire of service. However, it is noted that while most operators desire
complete elimination of downtime, minimization of downtime and complete elimination of downtime, minimization of downtime and
service degradation is often the expectation. service degradation is often the expectation.
The ISSU operation may apply in terms of an atomic version change of The ISSU operation may apply in terms of an atomic version change of
the entire system software or it may be applied in a more modular the entire system software or it may be applied in a more modular
sense such as for a patch or maintenance upgrade. The procedure sense such as for a patch or maintenance upgrade. The procedure
described herein may be used to verify either approach, as may be described herein may be used to verify either approach, as may be
supported by the vendor hardware and software. supported by the vendor hardware and software.
skipping to change at page 4, line 40 skipping to change at page 4, line 40
control plane and data plane functionality, this document will focus control plane and data plane functionality, this document will focus
on methodologies which would be directly applicable to those on methodologies which would be directly applicable to those
platforms. It is anticipated that the concepts and approaches platforms. It is anticipated that the concepts and approaches
described herein may be readily extended to accommodate other device described herein may be readily extended to accommodate other device
architectures as well. architectures as well.
2. Conventions used in this document 2. Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119 [RFC2119]. document are to be interpreted as described in RFC-2119 [RFC 2119].
In this document, these words will appear with that interpretation In this document, these words will appear with that interpretation
only when in ALL CAPS. Lower case uses of these words are not to be only when in ALL CAPS. Lower case uses of these words are not to be
interpreted as carrying RFC-2119 significance. interpreted as carrying RFC-2119 significance.
In this document, the characters ">>" preceding an indented line(s)
indicates a compliance requirement statement using the key words
listed above. This convention aids reviewers in quickly identifying
or finding the explicit compliance requirements of this RFC.
3. Generic ISSU Process, phased approach 3. Generic ISSU Process, phased approach
ISSU may be viewed as the behavior of a device when exposed to a ISSU may be viewed as the behavior of a device when exposed to a
planned change in its software functionality. This may mean changes planned change in its software functionality. This may mean changes
to the core operating system, separate processes or daemons or even to the core operating system, separate processes or daemons or even
of firmware logic in programmable hardware devices (e.g. CPLD/FPGA). of firmware logic in programmable hardware devices (e.g. CPLD/FPGA).
The goal of an ISSU implementation is to permit such actions with The goal of an ISSU implementation is to permit such actions with
minimal or no disruption to the primary operation of the device in minimal or no disruption to the primary operation of the device in
question. question.
skipping to change at page 7, line 24 skipping to change at page 7, line 24
3.4. Upgrade Acceptance 3.4. Upgrade Acceptance
In this phase, the new version of software MUST be running in all In this phase, the new version of software MUST be running in all
the physical nodes of the logical forwarding device. (RP's and LC's the physical nodes of the logical forwarding device. (RP's and LC's
as applicable). At this point, configuration control is returned to as applicable). At this point, configuration control is returned to
the operator and normal device operation i.e. outside of ISSU- the operator and normal device operation i.e. outside of ISSU-
oriented operation, is resumed. oriented operation, is resumed.
4. Test Methodology 4. Test Methodology
As stated by https://tools.ietf.org/html/rfc6815, the Test Topology As stated by [RFC 6815], the Test Topology Setup must be part
Setup must be part of an ITE (Isolated Test Environment) of an ITE (Isolated Test Environment)
The reporting of results MUST take into account the repeatability The reporting of results MUST take into account the repeatability
considerations from Section 4 of [RFC2544]. It is RECOMMENDED to considerations from Section 4 of [RFC 2544]. It is RECOMMENDED to
perform multiple trials and report average results. The results are perform multiple trials and report average results. The results are
reported in a simple statement including the measured frame loss and reported in a simple statement including the measured frame loss and
ISSU impact times. ISSU impact times.
4.1. Test Topology 4.1. Test Topology
The hardware configuration of the DUT (Device Under test) SHOULD be The hardware configuration of the DUT (Device Under test) SHOULD be
identical to the one expected to be or currently deployed in identical to the one expected to be or currently deployed in
production in order for the benchmark to have relevance. This would production in order for the benchmark to have relevance. This would
include the number of RP's, hardware version, memory and initial include the number of RP's, hardware version, memory and initial
skipping to change at page 8, line 9 skipping to change at page 8, line 9
MAY be utilized. The recommended approach relies on "mimicking" the MAY be utilized. The recommended approach relies on "mimicking" the
existing production data and control plane information, in order to existing production data and control plane information, in order to
emulate all the necessary Layer1 through Layer3 communications and, emulate all the necessary Layer1 through Layer3 communications and,
if appropriate, the upper layer characteristics of the network, as if appropriate, the upper layer characteristics of the network, as
well as end to end traffic/communication pairs. In other words, well as end to end traffic/communication pairs. In other words,
design a representative load model of the production environment and design a representative load model of the production environment and
deploy a collapsed topology utilizing test tools and/or external deploy a collapsed topology utilizing test tools and/or external
devices, where the DUT will be tested. Note that, the negative devices, where the DUT will be tested. Note that, the negative
impact of ISSU operations is likely to impact scaled, dynamic impact of ISSU operations is likely to impact scaled, dynamic
topologies to a greater extent than simpler, static environments. As topologies to a greater extent than simpler, static environments. As
such, this methodology (based upon production environments) is such, this methodology (based upon production configuration) is
advised for most test scenarios. advised for most test scenarios.
The second, more simplistic approach is to deploy an ITE "Isolated The second, more simplistic approach is to deploy an ITE
Testing Environment" as described in some of the existing standards "Isolated test Environment" in which end-points are "directly"
for benchmarking methodologies (e.g. RFC2544/RFC6815) in which end- connected to the DUT. In this manner, control plane information is
points are "directly" connected to the DUT. In this manner control kept to a minimum (only connected interfaces) and only a basic data
plane information is kept to a minimum (only connected interfaces) plane only a basic data plane of sources and destinations is applied.
and only a basic data plane of sources and destinations is applied.
If this methodology is selected, care must be taken to understand If this methodology is selected, care must be taken to understand
that the systemic behavior of the ITE may not be identical to that that the systemic behavior of the ITE may not be identical to that
experienced by a device in a production network role. That is, experienced by a device in a production network role. That is,
control plane validation may be minimal to none if this methodology. control plane validation may be minimal to none with this
methodology. Consequently, if this approach is chosen, comparison
with at least one production configuration is recommended in order
to understand the direct relevance and limitations of the
test exercise.
4.2. Load Model 4.2. Load Model
In consideration of the defined test topology, a load model must be In consideration of the defined test topology, a load model must be
developed to exercise the DUT while the ISSU event is introduced. developed to exercise the DUT while the ISSU event is introduced.
This applied load should be defined in such a manner as to provide a This applied load should be defined in such a manner as to provide a
granular, repeatable verification of the ISSU impact on transit granular, repeatable verification of the ISSU impact on transit
traffic. Sufficient traffic load (rate) should be applied to permit traffic. Sufficient traffic load (rate) should be applied to permit
timing extrapolations at a minimum granularity of 100 milliseconds timing extrapolations at a minimum granularity of 100 milliseconds
e.g. 100Mbps for a 10Gbps interface. The use of steady traffic e.g. 100Mbps for a 10Gbps interface. The use of steady traffic
skipping to change at page 14, line 10 skipping to change at page 14, line 10
ISSU operation) ISSU operation)
Load Model description including protocol mixes and any divergence Load Model description including protocol mixes and any divergence
from the production environment from the production environment
Time Results as per above Time Results as per above
Anomalies Observed during ISSU Anomalies Observed during ISSU
Anomalies Observed in post-ISSU analysis Anomalies Observed in post-ISSU analysis
It is RECOMMENDED that the following parameters be reported in these It is RECOMMENDED that the following parameters be reported
units: as outlined below:
Parameter Units or Examples Parameter Units or Examples
--------------------------------------------------------------- ---------------------------------------------------------------
Traffic Load Frames per second and bits per Second Traffic Load Frames per second and bits per Second
Disruption (average) Frames Disruption (average) Frames
Impact Time (average) Milliseconds Impact Time (average) Milliseconds
skipping to change at page 14, line 34 skipping to change at page 14, line 34
Protocols IPv4, IPv6, MPLS, etc. Protocols IPv4, IPv6, MPLS, etc.
Frame Size Octets Frame Size Octets
Port Media Ethernet, Gigabit Ethernet (GbE), Port Media Ethernet, Gigabit Ethernet (GbE),
Packet over SONET (POS), etc. Packet over SONET (POS), etc.
Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc. Port Speed 10 Gbps, 1 Gbps, 100 Mbps, etc.
Interface Encap. Ethernet, Ethernet VLAN, Interface Encaps Ethernet, Ethernet VLAN,
PPP, High-Level Data Link PPP, High-Level Data Link
Control(HDLC),etc. Control(HDLC),etc.
Number of Prefixes Integer count Number of Prefixes Integer count
flapped (ON Interval) (Optional # of prefixes / Time flapped (ON Interval) (Optional # of prefixes / Time
(minutes) (minutes)
skipping to change at page 15, line 14 skipping to change at page 15, line 14
are aberrant changes due to software faults. In either of these are aberrant changes due to software faults. In either of these
cases, any unexpected behavioral changes should be analyzed and a cases, any unexpected behavioral changes should be analyzed and a
determination made as to the impact of the change (be it determination made as to the impact of the change (be it
functional variances or operational impacts to existing scripts or functional variances or operational impacts to existing scripts or
management mechanisms. management mechanisms.
7.1. Data collection considerations 7.1. Data collection considerations
When a DUT is undergoing an ISSU operation, it's worth noting that When a DUT is undergoing an ISSU operation, it's worth noting that
the DUT's data collection and reporting of data, such as counters, the DUT's data collection and reporting of data, such as counters,
interface statistics, log messages, etc., might not be accurate. As interface statistics, log messages, etc., may not be accurate. As
such, one SHOULD NOT rely on the DUTs data collection methods, but such, one SHOULD NOT rely on the DUTs data collection methods, but
rather, SHOULD use the test tools and equipment to collect data used rather, SHOULD use the test tools and equipment to collect data used
for reporting in Section 7. Care and consideration should be paid in for reporting in Section 7. Care and consideration should be paid in
testing or adding new test cases, such that the desired data can be testing or adding new test cases, such that the desired data can be
collected from the test tools themselves, or other external collected from the test tools themselves, or other external
equipment, outside of the DUT itself. equipment, outside of the DUT itself.
8. Security Considerations 8. Security Considerations
This Applicability Statement intends to help preserve the security All BMWG memos are limited to testing in a laboratory Isolated Test
of the Internet by clarifying that the scope of this document and Environment (ITE), thus avoiding accidental interruption to
other BMWG memos are all limited to testing in a laboratory ITE, production networks due to test activities.
thus avoiding accidental Denial-of-Service attacks or congestion due
to high traffic volume test streams. All benchmarking activities are All benchmarking activities are limited to technology
limited to technology characterization using controlled stimuli in a characterization using controlled stimuli in a laboratory
laboratory environment, with dedicated address space and the other environment with dedicated address space and the other constraints
constraints [RFC2544]. [RFC 2544]
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the device under test/ solely on measurements observable external to the device under test/
system under test (DUT/SUT). system under test (DUT/SUT).
skipping to change at page 16, line 13 skipping to change at page 16, line 13
production networks. production networks.
9. IANA Considerations 9. IANA Considerations
There are no IANA actions required by this memo. There are no IANA actions required by this memo.
10. References 10. References
10.1. Normative References 10.1. Normative References
[1] Bradner, S., "Key words for use in RFCs to Indicate [RFC 2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for
Syntax Specifications: ABNF", RFC 2234, Internet Mail
Consortium and Demon Internet Ltd., November 1997.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for [RFC 2544] S. Bradner , J. McQuaid "Benchmarking Methodology
Syntax Specifications: ABNF", RFC 2234, Internet Mail for Network Interconnect Devices" RFC 2544 ,
Consortium and Demon Internet Ltd., November 1997. March 1999
10.2. Informative References 10.2. Informative References
[3] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP [RFC 5880] D. Katz, D. Ward "Bidirectional Forwarding Detection
and Its Effect on Busy Servers", Proc. Infocom 1999 pp. 1573- (BFD)" RFC 5880 , June 2010
1583.
[Fab1999] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in [RFC 6815] S. Bradner, K. Dubray , J. McQuaid, A. Morton
TCP and Its Effect on Busy Servers", Proc. Infocom 1999 ?Applicability Statement for RFC 2544:
pp. 1573-1583. Use on Production Networks Considered Harmful?
RFC 6815 , November 2012
11. Acknowledgments 11. Acknowledgments
The authors wish to thanks Vibin Thomas for his valued review and The authors wish to thank Vibin Thomas for his valued review and
feedback. feedback.
Authors' Addresses Authors' Addresses
Sarah Banks Sarah Banks
VSS Monitoring VSS Monitoring
Email: sbanks@encrypted.net Email: sbanks@encrypted.net
Fernando Calabria Fernando Calabria
Cisco Systems Cisco Systems
 End of changes. 21 change blocks. 
56 lines changed or deleted 48 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/