draft-ietf-bmwg-2544-as-04.txt   draft-ietf-bmwg-2544-as-05.txt 
Network Working Group S. Bradner Network Working Group S. Bradner
Internet-Draft Harvard University Internet-Draft Harvard University
Intended status: Informational K. Dubray Intended status: Informational K. Dubray
Expires: December 14, 2012 Juniper Networks Expires: February 12, 2013 Juniper Networks
J. McQuaid J. McQuaid
Turnip Video Turnip Video
A. Morton A. Morton
AT&T Labs AT&T Labs
June 12, 2012 August 11, 2012
RFC 2544 Applicability Statement: Use on Production Networks Considered RFC 2544 Applicability Statement: Use on Production Networks Considered
Harmful Harmful
draft-ietf-bmwg-2544-as-04 draft-ietf-bmwg-2544-as-05
Abstract Abstract
Benchmarking Methodology Working Group (BMWG) has been developing key Benchmarking Methodology Working Group (BMWG) has been developing key
performance metrics and laboratory test methods since 1990, and performance metrics and laboratory test methods since 1990, and
continues this work at present. Recent application of the methods continues this work at present. Recent application of the methods
beyond their intended scope is cause for concern. The methods beyond their intended scope is cause for concern. The methods
described in RFC 2544, where overload is a possible outcome, would no described in RFC 2544, where overload is a possible outcome, would no
doubt be harmful to user traffic performance on a production network. doubt be harmful to user traffic performance on a production network.
This memo clarifies the scope of RFC 2544 and other benchmarking work This memo clarifies the scope of RFC 2544 and other benchmarking work
skipping to change at page 1, line 43 skipping to change at page 1, line 43
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 14, 2012. This Internet-Draft will expire on February 12, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 26 skipping to change at page 2, line 26
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . . 3
2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . . 4
3. The Concept of an Isolated Test Environment . . . . . . . . . . 4 3. The Concept of an Isolated Test Environment . . . . . . . . . . 4
4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4 4. Why RFC 2544 Methods are intended for ITE . . . . . . . . . . . 4
4.1. Experimental Control and Accuracy . . . . . . . . . . . . . 4 4.1. Experimental Control and Accuracy . . . . . . . . . . . . . 4
4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . . 5 4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . . 5
5. Advisory on RFC 2544 Methods in Production Networks . . . . . . 5 5. Advisory on RFC 2544 Methods in Production Networks . . . . . . 5
6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6 6. What to do without RFC 2544? . . . . . . . . . . . . . . . . . 6
7. Security Considerations . . . . . . . . . . . . . . . . . . . . 6 7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 7
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 7 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8
10.1. Normative References . . . . . . . . . . . . . . . . . . . 7 10.1. Normative References . . . . . . . . . . . . . . . . . . . 8
10.2. Informative References . . . . . . . . . . . . . . . . . . 8 10.2. Informative References . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 8
1. Introduction 1. Introduction
This memo clarifies the scope of RFC 2544 [RFC2544], which discusses This memo clarifies the scope of RFC 2544 [RFC2544], which discusses
and defines several tests that may be used to characterize the and defines several tests that may be used to characterize the
performance of a network interconnecting device, and other performance of a network interconnecting device, and other
benchmarking work for the IETF community. benchmarking work for the IETF community.
skipping to change at page 5, line 40 skipping to change at page 5, line 40
were to be accidentally connected to part of the Internet. The were to be accidentally connected to part of the Internet. The
specific use of the addresses is detailed below." specific use of the addresses is detailed below."
In other words, devices operating on the Internet may be configured In other words, devices operating on the Internet may be configured
to discard any traffic they observe in this address range, as it is to discard any traffic they observe in this address range, as it is
intended for laboratory ITE use only. Thus, testers using the intended for laboratory ITE use only. Thus, testers using the
assigned testing address ranges MUST NOT be connected to the assigned testing address ranges MUST NOT be connected to the
Internet. Internet.
We note that a range of IPv6 addresses has been assigned to BMWG for We note that a range of IPv6 addresses has been assigned to BMWG for
laboratory test purposes, in [RFC5180]. Also, the strong statements laboratory test purposes, in [RFC5180] (as amended by errata). Also,
in the Security Considerations Section of this memo make the scope the strong statements in the Security Considerations Section of this
even more clear; this is now a standard fixture of all BMWG memos. memo make the scope even more clear; this is now a standard fixture
of all BMWG memos.
5. Advisory on RFC 2544 Methods in Production Networks 5. Advisory on RFC 2544 Methods in Production Networks
The tests in [RFC2544] were designed to measure the performance of The tests in [RFC2544] were designed to measure the performance of
network devices, not of networks, and certainly not production network devices, not of networks, and certainly not production
networks carrying user traffic on shared resources. There will be networks carrying user traffic on shared resources. There will be
unanticipated difficulties when applying these methods outside the unanticipated difficulties when applying these methods outside the
lab environment. lab environment.
One such difficulty stems from reliance on frame loss as an indicator
of resource exhaust in RFC 2544 methods. In practice, link-layer and
physical-layer errors prevent production networks from operating
loss-free. The RFC 2544 methods will not correctly assess Throughput
when loss from uncontrolled sources is present. Frame loss occurring
at the SLA levels of some networks could affect every iteration of
Throughput testing (when each step includes sufficient packets to
experience facility-related loss). Flawed results waste the time and
resources of the testing service user, and of the service provider
when called to dispute the measurement. These are additional
examples of harm that compliance with this advisory should help to
avoid.
Operating test equipment on production networks according to the Operating test equipment on production networks according to the
methods described in [RFC2544], where overload is a possible outcome, methods described in [RFC2544], where overload is a possible outcome,
would no doubt be harmful to user traffic performance. These tests would no doubt be harmful to user traffic performance. These tests
MUST NOT be used on production networks and as discussed above, the MUST NOT be used on production networks and as discussed above, the
tests will never produce a reliable or accurate benchmarking result tests will never produce a reliable or accurate benchmarking result
on a production network. on a production network.
[RFC2544] methods have never been validated on a network path, even [RFC2544] methods have never been validated on a network path, even
when that path is not part of a production network and carrying no when that path is not part of a production network and carrying no
other traffic. It is unknown whether the tests can be used to other traffic. It is unknown whether the tests can be used to
 End of changes. 8 change blocks. 
10 lines changed or deleted 24 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/