draft-ietf-bmwg-2544-as-05.txt   draft-ietf-bmwg-2544-as-06.txt 
Network Working Group S. Bradner Network Working Group S. Bradner
Internet-Draft Harvard University Internet-Draft Harvard University
Intended status: Informational K. Dubray Updates: 2544 (if approved) K. Dubray
Expires: February 12, 2013 Juniper Networks Intended status: Informational Juniper Networks
J. McQuaid Expires: March 8, 2013 J. McQuaid
Turnip Video Turnip Video
A. Morton A. Morton
AT&T Labs AT&T Labs
August 11, 2012 September 4, 2012
RFC 2544 Applicability Statement: Use on Production Networks Considered RFC 2544 Applicability Statement:
Harmful Use on Production Networks Considered Harmful
draft-ietf-bmwg-2544-as-05 draft-ietf-bmwg-2544-as-06
Abstract Abstract
Benchmarking Methodology Working Group (BMWG) has been developing key Benchmarking Methodology Working Group (BMWG) has been developing key
performance metrics and laboratory test methods since 1990, and performance metrics and laboratory test methods since 1990, and
continues this work at present. Recent application of the methods continues this work at present. The methods described in RFC 2544
beyond their intended scope is cause for concern. The methods are intended to generate traffic that overloads network device
described in RFC 2544, where overload is a possible outcome, would no resources in order to assess their capacity. Overload of shared
doubt be harmful to user traffic performance on a production network. resources would likely be harmful to user traffic performance on a
This memo clarifies the scope of RFC 2544 and other benchmarking work production network, and there are further negative consequences
for the IETF community. identified with production application of the methods. This memo
clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work
for isolated test environments only, and encourages new standards
activity for measurement methods applicable outside that scope.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 12, 2013. This Internet-Draft will expire on March 8, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
skipping to change at page 2, line 21 skipping to change at page 2, line 23
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3
2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4
3. The Concept of an Isolated Test Environment . . . . . . . . . . 4 3. The Concept of an Isolated Test Environment . . . . . . . . . . 4
4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4 4. Why RFC 2544 Methods are intended only for ITE . . . . . . . . 4
4.1. Experimental Control and Accuracy . . . . . . . . . . . . . 4 4.1. Experimental Control and Accuracy . . . . . . . . . . . . . 4
4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . . 5 4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . . 5
5. Advisory on RFC 2544 Methods in Production Networks . . . . . . 5 5. Advisory on RFC 2544 Methods in Production Networks . . . . . . 5
6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6 6. Considering Performance Testing in Production Networks . . . . 6
7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7 7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8
10.1. Normative References . . . . . . . . . . . . . . . . . . . 8 10.1. Normative References . . . . . . . . . . . . . . . . . . . 8
10.2. Informative References . . . . . . . . . . . . . . . . . . 8 10.2. Informative References . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8
1. Introduction 1. Introduction
This memo clarifies the scope of RFC 2544 [RFC2544], which discusses This memo clarifies the scope and use of IETF Benchmarking
and defines several tests that may be used to characterize the Methodology Working Group (BMWG) tests including [RFC2544], which
performance of a network interconnecting device, and other discusses and defines several tests that may be used to characterize
benchmarking work for the IETF community. the performance of a network interconnecting device.
Benchmarking Methodologies (beginning with [RFC2544]) have always Benchmarking methodologies (beginning with [RFC2544]) have always
relied on test conditions that can only be produced and replicated relied on test conditions that can only be produced and replicated
reliably in the laboratory. Thus it was unfortunate to find that reliably in the laboratory. These methodologies are not appropriate
this foundation methodology was being cited in several unintended for inclusion in wider specifications such as:
specifications and products performing applications such as:
1. Validation of telecommunication service configuration, such as 1. Validation of telecommunication service configuration, such as
the Committed Information Rate (CIR). the Committed Information Rate (CIR).
2. Validation of performance metrics in a telecommunication Service 2. Validation of performance metrics in a telecommunication Service
Level Agreement (SLA), such as frame loss and latency. Level Agreement (SLA), such as frame loss and latency.
3. Telecommunication service activation testing, where traffic that 3. Telecommunication service activation testing, where traffic that
shares network resources with the test might be adversely shares network resources with the test might be adversely
affected. affected.
Above, we distinguish "telecommunication service" (where a network Above, we distinguish "telecommunication service" (where a network
service provider contracts with a customer to transfer information service provider contracts with a customer to transfer information
between specified interfaces at different geographic locations) from between specified interfaces at different geographic locations) from
the generic term "service". Below, we use the adjective "production" the generic term "service". Below, we use the adjective "production"
to refer to networks carrying live user traffic. [RFC2544] used the to refer to networks carrying live user traffic. [RFC2544] used the
term "real-world" to refer to production networks and to term "real-world" to refer to production networks and to
differentiate them from test networks. differentiate them from test networks.
Although RFC 2544 is held up as the standard reference for such Although RFC 2544 has been held up as the standard reference for such
testing, we believe that the actual methods used vary from RFC 2544 testing, we believe that the actual methods used vary from [RFC2544]
in significant ways. Since the only citation is to RFC 2544, the in significant ways. Since the only citation is to [RFC2544], the
modifications are opaque to the standards community and to users in modifications are opaque to the standards community and to users in
general (an undesirable situation). There is risk of harm to user general.
traffic from applying the test traffic and methods described in
[RFC2544] on a production network, because overload in shared
resources is a possible outcome.
To directly address this situation, the past and present Chairs of Since applying the test traffic and methods described in [RFC2544] on
the IETF Benchmarking Methodology Working Group (BMWG) have prepared a production network risks causing overload in shared resources there
this Applicability Statement for RFC 2544. is direct risk of harming user traffic if the methods are misused in
this way. Therefore, IETF BMWG developed this Applicability
Statement for [RFC2544] to directly address the situation.
1.1. Requirements Language 1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
2. Scope and Goals 2. Scope and Goals
This memo clarifies the scope of [RFC2544], with the goal to provide This memo clarifies the scope of [RFC2544] with the goal to provide
guidance to the community on its applicability, which is limited to guidance to the industry on its applicability, which is limited to
laboratory testing. laboratory testing.
3. The Concept of an Isolated Test Environment 3. The Concept of an Isolated Test Environment
An Isolated Test Environment (ITE) used with [RFC2544] methods (as An Isolated Test Environment (ITE) used with [RFC2544] methods (as
illustrated in Figures 1 through 3 of [RFC2544]) has the ability to: illustrated in Figures 1 through 3 of [RFC2544]) has the ability to:
o contain the test streams to paths within the desired set-up o contain the test streams to paths within the desired set-up
o prevent non-test traffic from traversing the test set-up o prevent non-test traffic from traversing the test set-up
These features allow unfettered experimentation, while at the same These features allow unfettered experimentation, while at the same
time protecting lab equipment management/control LANs and other time protecting lab equipment management/control LANs and other
production networks from the unwanted effects of the test traffic. production networks from the unwanted effects of the test traffic.
4. Why RFC 2544 Methods are intended for ITE 4. Why RFC 2544 Methods are intended only for ITE
The following sections discuss some of the reasons why RFC 2544 The following sections discuss some of the reasons why [RFC2544]
[RFC2544] methods were intended only for isolated laboratory use, and methods are applicable only for isolated laboratory use, and the
the difficulties of applying these methods outside the lab consequences of applying these methods outside the lab environment.
environment.
4.1. Experimental Control and Accuracy 4.1. Experimental Control and Accuracy
All of the tests described in RFC 2544 require that the tester and All of the tests described in RFC 2544 require that the tester and
device under test are the only devices on the networks that are device under test are the only devices on the networks that are
transmitting data. The presence of other unwanted traffic on the transmitting data. The presence of other traffic (unwanted on the
network would mean that the specified test conditions have not been ITE network) would mean that the specified test conditions have not
achieved. been achieved and flawed results are a likely consequence.
If any unwanted traffic appears and the amount varies over time, the If any other traffic appears and the amount varies over time, the
repeatability of any test result will likely depend to some degree on repeatability of any test result will likely depend to some degree on
the unwanted traffic. the amount and variation of the other traffic.
The presence of unwanted or unknown traffic makes accurate, The presence of other traffic makes accurate, repeatable, and
repeatable, and consistent measurements of the performance of the consistent measurements of the performance of the device under test
device under test very unlikely, since the complete details of test very unlikely, since the complete details of test conditions will not
conditions will not be reported. be reported.
For example, the RFC 2544 Throughput Test attempts to characterize a For example, the RFC 2544 Throughput Test attempts to characterize a
maximum reliable load, thus there will be testing above the maximum maximum reliable load, thus there will be testing above the maximum
that causes packet/frame loss. Any other sources of traffic on the that causes packet/frame loss. Any other sources of traffic on the
network will cause packet loss to occur at a tester data rate lower network will cause packet loss to occur at a tester data rate lower
than the rate that would be achieved without the extra traffic. than the rate that would be achieved without the extra traffic.
4.2. Containing Damage 4.2. Containing Damage
RFC 2544 methods, specifically to determine Throughput as defined in [RFC2544] methods, specifically to determine Throughput as defined in
[RFC1242] and other benchmarks, may overload the resources of the [RFC1242] and other benchmarks, may overload the resources of the
device under test, and may cause failure modes in the device under device under test, and may cause failure modes in the device under
test. Since failures can become the root cause of more wide-spread test. Since failures can become the root cause of more wide-spread
failure, it is clearly desirable to contain all test traffic within failure, it is clearly desirable to contain all test traffic within
the ITE. the ITE.
In addition, such testing can have a negative effect on any traffic In addition, such testing can have a negative effect on any traffic
that shares resources with the test stream(s) since, in most cases, that shares resources with the test stream(s) since, in most cases,
the traffic load will be close to the capacity of the network links. the traffic load will be close to the capacity of the network links.
skipping to change at page 5, line 36 skipping to change at page 5, line 31
"...The network addresses 198.18.0.0 through 198.19.255.255 have been "...The network addresses 198.18.0.0 through 198.19.255.255 have been
assigned to the BMWG by the IANA for this purpose. This assignment assigned to the BMWG by the IANA for this purpose. This assignment
was made to minimize the chance of conflict in case a testing device was made to minimize the chance of conflict in case a testing device
were to be accidentally connected to part of the Internet. The were to be accidentally connected to part of the Internet. The
specific use of the addresses is detailed below." specific use of the addresses is detailed below."
In other words, devices operating on the Internet may be configured In other words, devices operating on the Internet may be configured
to discard any traffic they observe in this address range, as it is to discard any traffic they observe in this address range, as it is
intended for laboratory ITE use only. Thus, testers using the intended for laboratory ITE use only. Thus, testers using the
assigned testing address ranges MUST NOT be connected to the assigned testing address ranges are connected to the Internet and
Internet. test packets are forwarded across the Internet, it is likely that the
packets will be discarded and the test will not work.
We note that a range of IPv6 addresses has been assigned to BMWG for We note that a range of IPv6 addresses has been assigned to BMWG for
laboratory test purposes, in [RFC5180] (as amended by errata). Also, laboratory test purposes, in [RFC5180] (as amended by errata).
the strong statements in the Security Considerations Section of this
memo make the scope even more clear; this is now a standard fixture See the Security Considerations Section below for further
of all BMWG memos. considerations on containing damage.
5. Advisory on RFC 2544 Methods in Production Networks 5. Advisory on RFC 2544 Methods in Production Networks
The tests in [RFC2544] were designed to measure the performance of The tests in [RFC2544] were designed to measure the performance of
network devices, not of networks, and certainly not production network devices, not of networks, and certainly not production
networks carrying user traffic on shared resources. There will be networks carrying user traffic on shared resources. There will be
unanticipated difficulties when applying these methods outside the undesirable consequences when applying these methods outside the
lab environment. isolated test environment.
One such difficulty stems from reliance on frame loss as an indicator One negative consequence stems from reliance on frame loss as an
of resource exhaust in RFC 2544 methods. In practice, link-layer and indicator of resource exhaustion in [RFC2544] methods. In practice,
physical-layer errors prevent production networks from operating link-layer and physical-layer errors prevent production networks from
loss-free. The RFC 2544 methods will not correctly assess Throughput operating loss-free. The [RFC2544] methods will not correctly assess
when loss from uncontrolled sources is present. Frame loss occurring Throughput when loss from uncontrolled sources is present. Frame
at the SLA levels of some networks could affect every iteration of loss occurring at the SLA levels of some networks could affect every
Throughput testing (when each step includes sufficient packets to iteration of Throughput testing (when each step includes sufficient
experience facility-related loss). Flawed results waste the time and packets to experience facility-related loss). Flawed results waste
resources of the testing service user, and of the service provider the time and resources of the testing service user and of the service
when called to dispute the measurement. These are additional provider when called to dispute the measurement. These are
examples of harm that compliance with this advisory should help to additional examples of harm that compliance with this advisory should
avoid. help to avoid.
Operating test equipment on production networks according to the The methods described in [RFC2544] are intended to generate traffic
methods described in [RFC2544], where overload is a possible outcome, that overloads network device resources in order to assess their
would no doubt be harmful to user traffic performance. These tests capacity. Overload of shared resources would likely be harmful to
MUST NOT be used on production networks and as discussed above, the user traffic performance on a production network. These tests MUST
tests will never produce a reliable or accurate benchmarking result NOT be used on production networks and as discussed above. The tests
on a production network. will not produce a reliable or accurate benchmarking result on a
production network.
[RFC2544] methods have never been validated on a network path, even [RFC2544] methods have never been validated on a network path, even
when that path is not part of a production network and carrying no when that path is not part of a production network and carrying no
other traffic. It is unknown whether the tests can be used to other traffic. It is unknown whether the tests can be used to
measure valid and reliable performance of a multi-device, multi- measure valid and reliable performance of a multi-device, multi-
network path. It is possible that some of the tests may prove valid network path. It is possible that some of the tests may prove valid
in some path scenarios, but that work has not been done or has not in some path scenarios, but that work has not been done or has not
been shared with the IETF community. Thus, such testing is contra- been shared with the IETF community. Thus, such testing is contra-
indicated by the BMWG. indicated by the BMWG.
6. What to do without RFC 2544? 6. Considering Performance Testing in Production Networks
The IETF has addressed the problem of production network performance The IETF has addressed the problem of production network performance
measurement by chartering a different working group: IP Performance measurement by chartering a different working group: IP Performance
Metrics (IPPM). This working group has developed a set of standard Metrics (IPPM). This working group has developed a set of standard
metrics to assess the quality, performance, and reliability of metrics to assess the quality, performance, and reliability of
Internet packet transfer services. These metrics can be measured by Internet packet transfer services. These metrics can be measured by
network operators, end users, or independent testing groups. We note network operators, end users, or independent testing groups. We note
that some IPPM metrics differ from RFC 2544 metrics with similar that some IPPM metrics differ from RFC 2544 metrics with similar
names, and there is likely to be confusion if the details are names, and there is likely to be confusion if the details are
ignored. ignored.
IPPM has not yet standardized methods for raw capacity measurement of IPPM has not yet standardized methods for raw capacity measurement of
Internet paths. Such testing needs to adequately consider the strong Internet paths. Such testing needs to adequately consider the strong
possibility for degradation to any other traffic that may be present possibility for degradation to any other traffic that may be present
due to congestion. There are no specific methods proposed for due to congestion. There are no specific methods proposed for
activation of a packet transfer service in IPPM. activation of a packet transfer service in IPPM at this time. Thus,
individuals who need to conduct capacity tests on production networks
should actively participate in standards development to ensure their
methods receive appropriate industry review and agreement, in the
IETF or in alternate standards development organizations.
Other standards may help to fill gaps in telecommunication service Other standards may help to fill gaps in telecommunication service
testing. For example, the IETF has many standards intended to assist testing. For example, the IETF has many standards intended to assist
with network operation, administration and maintenance (OAM), and with network operation, administration and maintenance (OAM), and
ITU-T Study Group 12 has a recommendation on service activation test ITU-T Study Group 12 has a Recommendation on service activation test
methodology [Y.1564]. methodology [Y.1564].
The world will not spin off axis while waiting for appropriate and The world will not spin off axis while waiting for appropriate and
standardized methods to emerge from the consensus process. standardized methods to emerge from the consensus process.
7. Security Considerations 7. Security Considerations
This Applicability Statement intends to help preserve the security of This Applicability Statement intends to help preserve the security of
the Internet by clarifying that the scope of [RFC2544] and other BMWG the Internet by clarifying that the scope of [RFC2544] and other BMWG
memos are all limited to testing in a laboratory ITE, thus avoiding memos are all limited to testing in a laboratory ITE, thus avoiding
skipping to change at page 7, line 48 skipping to change at page 8, line 8
from the DUT/SUT SHOULD be identical in the lab and in production from the DUT/SUT SHOULD be identical in the lab and in production
networks. networks.
8. IANA Considerations 8. IANA Considerations
This memo makes no requests of IANA. This memo makes no requests of IANA.
9. Acknowledgements 9. Acknowledgements
Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis
Villamizar, and David Newman for reading and suggesting improvements Villamizar, David Newman, and Adrian Farrel for suggesting
to this memo. improvements to this memo.
Specifically, Al Morton would like to thank his co-authors, who
constitute the complete set of Chairmen-Emeritus of the BMWG, for
returning from other pursuits to develop this statement and see it
through to approval. This has been a rare privilege; one that likely
will not be matched in the IETF again:
Scott Bradner served as Chairman from 1990 to 1993
Jim McQuaid served as Chairman from 1993 to 1995
Kevin Dubray served as Chairman from 1995 to 2006
It's all about the band.
10. References 10. References
10.1. Normative References 10.1. Normative References
[RFC1242] Bradner, S., "Benchmarking terminology for network [RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991. interconnection devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
 End of changes. 32 change blocks. 
84 lines changed or deleted 101 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/