draft-ietf-bmwg-2544-as-06.txt   draft-ietf-bmwg-2544-as-07.txt 
Network Working Group S. Bradner Network Working Group S. Bradner
Internet-Draft Harvard University Internet-Draft Harvard University
Updates: 2544 (if approved) K. Dubray Updates: 2544 (if approved) K. Dubray
Intended status: Informational Juniper Networks Intended status: Informational Juniper Networks
Expires: March 8, 2013 J. McQuaid Expires: March 22, 2013 J. McQuaid
Turnip Video Turnip Video
A. Morton A. Morton
AT&T Labs AT&T Labs
September 4, 2012 September 18, 2012
RFC 2544 Applicability Statement: RFC 2544 Applicability Statement:
Use on Production Networks Considered Harmful Use on Production Networks Considered Harmful
draft-ietf-bmwg-2544-as-06 draft-ietf-bmwg-2544-as-07
Abstract Abstract
Benchmarking Methodology Working Group (BMWG) has been developing key Benchmarking Methodology Working Group (BMWG) has been developing key
performance metrics and laboratory test methods since 1990, and performance metrics and laboratory test methods since 1990, and
continues this work at present. The methods described in RFC 2544 continues this work at present. The methods described in RFC 2544
are intended to generate traffic that overloads network device are intended to generate traffic that overloads network device
resources in order to assess their capacity. Overload of shared resources in order to assess their capacity. Overload of shared
resources would likely be harmful to user traffic performance on a resources would likely be harmful to user traffic performance on a
production network, and there are further negative consequences production network, and there are further negative consequences
skipping to change at page 1, line 46 skipping to change at page 1, line 46
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on March 8, 2013. This Internet-Draft will expire on March 22, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . 4
3. The Concept of an Isolated Test Environment . . . . . . . . . . 4 3. The Concept of an Isolated Test Environment . . . . . . . . . 4
4. Why RFC 2544 Methods are intended only for ITE . . . . . . . . 4 4. Why RFC 2544 Methods are intended only for ITE . . . . . . . . 4
4.1. Experimental Control and Accuracy . . . . . . . . . . . . . 4 4.1. Experimental Control and Accuracy . . . . . . . . . . . . 4
4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . . 5 4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . 5
5. Advisory on RFC 2544 Methods in Production Networks . . . . . . 5 5. Advisory on RFC 2544 Methods in Production Networks . . . . . 5
6. Considering Performance Testing in Production Networks . . . . 6 6. Considering Performance Testing in Production Networks . . . . 6
7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7 7. Security Considerations . . . . . . . . . . . . . . . . . . . 7
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8 10. Appendix - Example of RFC 2544 method failure in
10.1. Normative References . . . . . . . . . . . . . . . . . . . 8 production network measurement . . . . . . . . . . . . . . . . 8
10.2. Informative References . . . . . . . . . . . . . . . . . . 8 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8 11.1. Normative References . . . . . . . . . . . . . . . . . . . 9
11.2. Informative References . . . . . . . . . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10
1. Introduction 1. Introduction
This memo clarifies the scope and use of IETF Benchmarking This memo clarifies the scope and use of IETF Benchmarking
Methodology Working Group (BMWG) tests including [RFC2544], which Methodology Working Group (BMWG) tests including [RFC2544], which
discusses and defines several tests that may be used to characterize discusses and defines several tests that may be used to characterize
the performance of a network interconnecting device. the performance of a network interconnecting device. All readers of
this memo must read and fully understand [RFC2544].
Benchmarking methodologies (beginning with [RFC2544]) have always Benchmarking methodologies (beginning with [RFC2544]) have always
relied on test conditions that can only be produced and replicated relied on test conditions that can only be produced and replicated
reliably in the laboratory. These methodologies are not appropriate reliably in the laboratory. These methodologies are not appropriate
for inclusion in wider specifications such as: for inclusion in wider specifications such as:
1. Validation of telecommunication service configuration, such as 1. Validation of telecommunication service configuration, such as
the Committed Information Rate (CIR). the Committed Information Rate (CIR).
2. Validation of performance metrics in a telecommunication Service 2. Validation of performance metrics in a telecommunication Service
skipping to change at page 5, line 30 skipping to change at page 5, line 30
IPv4 address range for testing: IPv4 address range for testing:
"...The network addresses 198.18.0.0 through 198.19.255.255 have been "...The network addresses 198.18.0.0 through 198.19.255.255 have been
assigned to the BMWG by the IANA for this purpose. This assignment assigned to the BMWG by the IANA for this purpose. This assignment
was made to minimize the chance of conflict in case a testing device was made to minimize the chance of conflict in case a testing device
were to be accidentally connected to part of the Internet. The were to be accidentally connected to part of the Internet. The
specific use of the addresses is detailed below." specific use of the addresses is detailed below."
In other words, devices operating on the Internet may be configured In other words, devices operating on the Internet may be configured
to discard any traffic they observe in this address range, as it is to discard any traffic they observe in this address range, as it is
intended for laboratory ITE use only. Thus, testers using the intended for laboratory ITE use only. Thus, if testers using the
assigned testing address ranges are connected to the Internet and assigned testing address ranges are connected to the Internet and
test packets are forwarded across the Internet, it is likely that the test packets are forwarded across the Internet, it is likely that the
packets will be discarded and the test will not work. packets will be discarded and the test will not work.
We note that a range of IPv6 addresses has been assigned to BMWG for We note that a range of IPv6 addresses has been assigned to BMWG for
laboratory test purposes, in [RFC5180] (as amended by errata). laboratory test purposes, in [RFC5180] (as amended by errata).
See the Security Considerations Section below for further See the Security Considerations Section below for further
considerations on containing damage. considerations on containing damage.
skipping to change at page 6, line 13 skipping to change at page 6, line 13
indicator of resource exhaustion in [RFC2544] methods. In practice, indicator of resource exhaustion in [RFC2544] methods. In practice,
link-layer and physical-layer errors prevent production networks from link-layer and physical-layer errors prevent production networks from
operating loss-free. The [RFC2544] methods will not correctly assess operating loss-free. The [RFC2544] methods will not correctly assess
Throughput when loss from uncontrolled sources is present. Frame Throughput when loss from uncontrolled sources is present. Frame
loss occurring at the SLA levels of some networks could affect every loss occurring at the SLA levels of some networks could affect every
iteration of Throughput testing (when each step includes sufficient iteration of Throughput testing (when each step includes sufficient
packets to experience facility-related loss). Flawed results waste packets to experience facility-related loss). Flawed results waste
the time and resources of the testing service user and of the service the time and resources of the testing service user and of the service
provider when called to dispute the measurement. These are provider when called to dispute the measurement. These are
additional examples of harm that compliance with this advisory should additional examples of harm that compliance with this advisory should
help to avoid. help to avoid. See the Appendix for an example.
The methods described in [RFC2544] are intended to generate traffic The methods described in [RFC2544] are intended to generate traffic
that overloads network device resources in order to assess their that overloads network device resources in order to assess their
capacity. Overload of shared resources would likely be harmful to capacity. Overload of shared resources would likely be harmful to
user traffic performance on a production network. These tests MUST user traffic performance on a production network. These tests MUST
NOT be used on production networks and as discussed above. The tests NOT be used on production networks and as discussed above. The tests
will not produce a reliable or accurate benchmarking result on a will not produce a reliable or accurate benchmarking result on a
production network. production network.
[RFC2544] methods have never been validated on a network path, even [RFC2544] methods have never been validated on a network path, even
skipping to change at page 8, line 23 skipping to change at page 8, line 23
returning from other pursuits to develop this statement and see it returning from other pursuits to develop this statement and see it
through to approval. This has been a rare privilege; one that likely through to approval. This has been a rare privilege; one that likely
will not be matched in the IETF again: will not be matched in the IETF again:
Scott Bradner served as Chairman from 1990 to 1993 Scott Bradner served as Chairman from 1990 to 1993
Jim McQuaid served as Chairman from 1993 to 1995 Jim McQuaid served as Chairman from 1993 to 1995
Kevin Dubray served as Chairman from 1995 to 2006 Kevin Dubray served as Chairman from 1995 to 2006
It's all about the band. It's all about the band.
10. References 10. Appendix - Example of RFC 2544 method failure in production network
measurement
10.1. Normative References This Appendix provides an example illustrating how [RFC2544] methods
applied on production networks can easily produce a form of harm from
flawed and misleading results.
The [RFC2544] Throughput benchmarking method usually includes the
following steps:
a. Set the offered traffic level, less than max of the ingress
link(s).
b. Send the test traffic through the device under test (DUT) and
count all frames successfully transferred.
c. If all frames are received, increment traffic level and repeat
step b.
d. If one or more frames are lost, the level is in the DUT-overload
region (Step b may be repeated at a reduced traffic level to more
exactly determine the maximum rate at which none of the frames are
dropped by the DUT, defined as the Throughput [RFC1242]).
e. Report the Throughput values, the x-y of graph of frame size and
Throughput, and other information in accordance with [RFC2544].
In this method, frame loss is the sole indicator of overload and
therefore the determining factor in the measurement of Throughput
using the [RFC2544] methodology (even though the results may not
report frame loss per se).
Frame loss is subject to many factors in addition to operating above
the Throughput traffic level. These factors include optical
interference (which may be due to dirty interfaces, cross-over from
other signals, fiber bend and temperature, etc.) and electrical
interference (caused by local sources of radio signals, electrical
spikes, solar particles, etc.). In the laboratory environment many
of these issues can be carefully controlled through cleaning and
isolation. Since [RFC2544] methodologies are primarily intended to
test devices and not paths, the total length of path, the number of
interfaces, and compound risk of random frame loss can be kept to a
minimum.
In a production network, however, there will be many interfaces and
many kilometres of path under test. This considerably increases the
risk of random frame loss.
The risk of frame loss caused by outside effects is significantly
higher in production networks, and significantly higher with long
paths (both those with long physical path lengths, and those with
large numbers of interfaces in the path). Thus, the risk of falsely
low reported Throughput using an [RFC2544] methodology test is
considerably increased in a production network.
Therefore, to successfully conduct tests with similar objectives to
those in [RFC2544] in a production network, it will be necessary to
develop modifications to the methodologies defined in [RFC2544] and
standards to describe them.
11. References
11.1. Normative References
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991. interconnection devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D.
Dugatkin, "IPv6 Benchmarking Methodology for Network Dugatkin, "IPv6 Benchmarking Methodology for Network
Interconnect Devices", RFC 5180, May 2008. Interconnect Devices", RFC 5180, May 2008.
10.2. Informative References 11.2. Informative References
[Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation [Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation
Test Methodology", March 2011. Test Methodology", March 2011.
Authors' Addresses Authors' Addresses
Scott Bradner Scott Bradner
Harvard University Harvard University
29 Oxford St. 29 Oxford St.
Cambridge, MA 02138 Cambridge, MA 02138
 End of changes. 11 change blocks. 
27 lines changed or deleted 90 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/