draft-ietf-bmwg-2544-as-00.txt   draft-ietf-bmwg-2544-as-01.txt 
Network Working Group S. Bradner Network Working Group S. Bradner
Internet-Draft Harvard University Internet-Draft Harvard University
Intended status: Informational K. Dubray Intended status: Informational K. Dubray
Expires: February 7, 2012 Juniper Networks Expires: April 22, 2012 Juniper Networks
J. McQuaid J. McQuaid
Turnip Video Turnip Video
A. Morton A. Morton
AT&T Labs AT&T Labs
August 6, 2011 October 20, 2011
RFC 2544 Applicability Statement: Use on Real-World Networks Considered RFC 2544 Applicability Statement: Use on Production Networks Considered
Harmful Harmful
draft-ietf-bmwg-2544-as-00 draft-ietf-bmwg-2544-as-01
Abstract Abstract
Benchmarking Methodology Working Group (BMWG) has been developing key Benchmarking Methodology Working Group (BMWG) has been developing key
performance metrics and laboratory test methods since 1990, and performance metrics and laboratory test methods since 1990, and
continues this work at present. Recent application of the methods continues this work at present. Recent application of the methods
beyond their intended scope is cause for concern. This memo beyond their intended scope is cause for concern. This memo
clarifies the scope of RFC 2544 and other benchmarking work for the clarifies the scope of RFC 2544 and other benchmarking work for the
IETF community. IETF community.
skipping to change at page 1, line 41 skipping to change at page 1, line 41
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 7, 2012. This Internet-Draft will expire on April 22, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3
2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4
3. The Concept of an Isolated Test Environment . . . . . . . . . . 4 3. The Concept of an Isolated Test Environment . . . . . . . . . . 4
4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4 4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4
4.1. Experimental Control, Repeatability, and Accuracy . . . . . 4 4.1. Experimental Control, Repeatability, and Accuracy . . . . . 4
4.2. Containment of Implementation Failure Impact . . . . . . . 5 4.2. Containment of Implementation Failure Impact . . . . . . . 5
5. Advisory on RFC 2544 Methods in Real-world Networks . . . . . . 5 5. Advisory on RFC 2544 Methods in Real-world Networks . . . . . . 5
6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6 6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6
7. Security Considerations . . . . . . . . . . . . . . . . . . . . 6 7. Security Considerations . . . . . . . . . . . . . . . . . . . . 6
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 7 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 7
skipping to change at page 3, line 29 skipping to change at page 3, line 29
2. Validation of performance metrics in a telecommunication Service 2. Validation of performance metrics in a telecommunication Service
Level Agreement (SLA), such as frame loss and latency. Level Agreement (SLA), such as frame loss and latency.
3. As an integral part of telecommunication service activation 3. As an integral part of telecommunication service activation
testing, where traffic that shares network resources with the testing, where traffic that shares network resources with the
test might be adversely affected. test might be adversely affected.
Above, we distinguish "telecommunication service" (where a network Above, we distinguish "telecommunication service" (where a network
service provider contracts with a customer to transfer information service provider contracts with a customer to transfer information
between specified interfaces at different geographic locations in the between specified interfaces at different geographic locations in the
real world) from the generic term "service". Also, we use the term real world) from the generic term "service". Also, we use the
"real-world networks" to refer to production networks carrying live adjective "production" to refer to networks carrying live user
user traffic. traffic. [RFC2544] used the term "real-world" to refer production
networks and to differentiate them from test networks.
Although RFC 2544 is held up as the standard reference for such Although RFC 2544 is held up as the standard reference for such
testing, we believe that the actual methods used vary from RFC 2544 testing, we believe that the actual methods used vary from RFC 2544
in significant ways. Since the only citation is to RFC 2544, the in significant ways. Since the only citation is to RFC 2544, the
modifications are opaque to the standards community and to users in modifications are opaque to the standards community and to users in
general (an undesirable situation). general (an undesirable situation).
To directly address this situation, the past and present Chairs of To directly address this situation, the past and present Chairs of
the IETF Benchmarking Methodology Working Group (BMWG) have prepared the IETF Benchmarking Methodology Working Group (BMWG) have prepared
this Applicability Statement for RFC 2544. this Applicability Statement for RFC 2544.
skipping to change at page 5, line 8 skipping to change at page 5, line 10
For example, the RFC 2544 Throughput Test attempts to characterize a For example, the RFC 2544 Throughput Test attempts to characterize a
maximum reliable load, thus there will be testing above the maximum maximum reliable load, thus there will be testing above the maximum
that causes packet/frame loss. Any other sources of traffic on the that causes packet/frame loss. Any other sources of traffic on the
network will cause packet loss to occur at a tester data rate lower network will cause packet loss to occur at a tester data rate lower
than the rate that would be achieved without the extra traffic. than the rate that would be achieved without the extra traffic.
4.2. Containment of Implementation Failure Impact 4.2. Containment of Implementation Failure Impact
RFC 2544 methods, specifically to determine Throughput as defined in RFC 2544 methods, specifically to determine Throughput as defined in
[RFC1242] and other benchmarks, are designed to overload the [RFC1242] and other benchmarks, may overload the resources of the
resources of the device under test, and may cause failure modes in device under test, and may cause failure modes in the device under
the device under test. Since failures can become the root cause of test. Since failures can become the root cause of more wide-spread
more wide-spread failure, it is clearly desirable to contain all DUT failure, it is clearly desirable to contain all DUT traffic within
traffic within the ITE. the ITE.
In addition, such testing can have a negative affect on any traffic In addition, such testing can have a negative affect on any traffic
which shares resources with the test stream(s) since, in most cases, which shares resources with the test stream(s) since, in most cases,
the traffic load will be close to the capacity of the network links. the traffic load will be close to the capacity of the network links.
Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private
IPv4 address range for testing: IPv4 address range for testing:
"...The network addresses 198.18.0.0 through 198.19.255.255 have been "...The network addresses 198.18.0.0 through 198.19.255.255 have been
assigned to the BMWG by the IANA for this purpose. This assignment assigned to the BMWG by the IANA for this purpose. This assignment
skipping to change at page 5, line 46 skipping to change at page 5, line 48
even more clear; this is now a standard fixture of all BMWG memos. even more clear; this is now a standard fixture of all BMWG memos.
5. Advisory on RFC 2544 Methods in Real-world Networks 5. Advisory on RFC 2544 Methods in Real-world Networks
The tests in [RFC2544] were designed to measure the performance of The tests in [RFC2544] were designed to measure the performance of
network devices, not of networks, and certainly not production network devices, not of networks, and certainly not production
networks carrying user traffic on shared resources. There will be networks carrying user traffic on shared resources. There will be
unanticipated difficulties when applying these methods outside the unanticipated difficulties when applying these methods outside the
lab environment. lab environment.
Operating test equipment on real-world networks according to the Operating test equipment on production networks according to the
methods described in [RFC2544], where overload is a required outcome, methods described in [RFC2544], where overload is a possible outcome,
would no doubt be harmful to user traffic performance. These tests would no doubt be harmful to user traffic performance. These tests
MUST NOT be used on active networks. And as discussed above, the MUST NOT be used on active networks. And as discussed above, the
tests will never produce a reliable or accurate benchmarking result. tests will never produce a reliable or accurate benchmarking result.
[RFC2544] methods have never been validated on a network path, even [RFC2544] methods have never been validated on a network path, even
when that path is not part of a production network and carrying no when that path is not part of a production network and carrying no
other traffic. It is unknown whether the tests can be used to other traffic. It is unknown whether the tests can be used to
measure valid and reliable performance of a multi-device, multi- measure valid and reliable performance of a multi-device, multi-
network path. It is possible that some of the tests may prove to be network path. It is possible that some of the tests may prove to be
valid in some path scenarios, but that work has not been done or has valid in some path scenarios, but that work has not been done or has
not been shared with the IETF community. Thus, such testing is not been shared with the IETF community. Thus, such testing is
contra-indicated by the BMWG. contra-indicated by the BMWG.
6. What to do without RFC 2544? 6. What to do without RFC 2544?
The IETF has addressed the problem of real-world network performance The IETF has addressed the problem of production network performance
measurement by chartering a different working group: IP Performance measurement by chartering a different working group: IP Performance
Metrics (IPPM). This working group has developed a set of standard Metrics (IPPM). This working group has developed a set of standard
metrics to assess the quality, performance, and reliability of metrics to assess the quality, performance, and reliability of
Internet packet transfer services. These metrics can be measured by Internet packet transfer services. These metrics can be measured by
network operators, end users, or independent testing groups. We note network operators, end users, or independent testing groups. We note
that some IPPM metrics differ from RFC 2544 metrics with similar that some IPPM metrics differ from RFC 2544 metrics with similar
names, and there is likely to be confusion if the details are names, and there is likely to be confusion if the details are
ignored. ignored.
IPPM has not standardized methods for raw capacity measurement of IPPM has not standardized methods for raw capacity measurement of
 End of changes. 10 change blocks. 
17 lines changed or deleted 18 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/