draft-ietf-bmwg-2544-as-08.txt   rfc6815.txt 
Network Working Group S. Bradner Internet Engineering Task Force (IETF) S. Bradner
Internet-Draft Harvard University Request for Comments: 6815 Harvard University
Updates: 2544 (if approved) K. Dubray Updates: 2544 K. Dubray
Intended status: Informational Juniper Networks Category: Informational Juniper Networks
Expires: April 25, 2013 J. McQuaid ISSN: 2070-1721 J. McQuaid
Turnip Video Turnip Video
A. Morton A. Morton
AT&T Labs AT&T Labs
October 22, 2012 November 2012
RFC 2544 Applicability Statement: Applicability Statement for RFC 2544:
Use on Production Networks Considered Harmful Use on Production Networks Considered Harmful
draft-ietf-bmwg-2544-as-08
Abstract Abstract
Benchmarking Methodology Working Group (BMWG) has been developing key The Benchmarking Methodology Working Group (BMWG) has been developing
performance metrics and laboratory test methods since 1990, and key performance metrics and laboratory test methods since 1990, and
continues this work at present. The methods described in RFC 2544 continues this work at present. The methods described in RFC 2544
are intended to generate traffic that overloads network device are intended to generate traffic that overloads network device
resources in order to assess their capacity. Overload of shared resources in order to assess their capacity. Overload of shared
resources would likely be harmful to user traffic performance on a resources would likely be harmful to user traffic performance on a
production network, and there are further negative consequences production network, and there are further negative consequences
identified with production application of the methods. This memo identified with production application of the methods. This memo
clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work
for isolated test environments only, and encourages new standards for isolated test environments only, and it encourages new standards
activity for measurement methods applicable outside that scope. activity for measurement methods applicable outside that scope.
Status of this Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering This document is not an Internet Standards Track specification; it is
Task Force (IETF). Note that other groups may also distribute published for informational purposes.
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
This Internet-Draft will expire on April 25, 2013. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc6815.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
skipping to change at page 2, line 19 skipping to change at page 2, line 22
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction ....................................................3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language ......................................4
2. Scope and Goals . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope and Goals .................................................4
3. The Concept of an Isolated Test Environment . . . . . . . . . 4 3. The Concept of an Isolated Test Environment .....................4
4. Why RFC 2544 Methods are intended only for ITE . . . . . . . . 4 4. Why the Methods of RFC 2544 Are Intended Only for ITE ...........4
4.1. Experimental Control and Accuracy . . . . . . . . . . . . 4 4.1. Experimental Control and Accuracy ..........................4
4.2. Containing Damage . . . . . . . . . . . . . . . . . . . . 5 4.2. Containing Damage ..........................................5
5. Advisory on RFC 2544 Methods in Production Networks . . . . . 5 5. Advisory on RFC 2544 Methods in Production Networks .............5
6. Considering Performance Testing in Production Networks . . . . 6 6. Considering Performance Testing in Production Networks ..........6
7. Security Considerations . . . . . . . . . . . . . . . . . . . 7 7. Security Considerations .........................................7
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 8. Acknowledgements ................................................7
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8 9. References ......................................................8
10. Appendix - Example of RFC 2544 method failure in 9.1. Normative References .......................................8
production network measurement . . . . . . . . . . . . . . . . 8 9.2. Informative References .....................................8
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Appendix A. Example of RFC 2544 Method Failure in Production
11.1. Normative References . . . . . . . . . . . . . . . . . . . 9 Network Measurement ....................................9
11.2. Informative References . . . . . . . . . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10
1. Introduction 1. Introduction
This memo clarifies the scope and use of IETF Benchmarking This memo clarifies the scope and use of IETF Benchmarking
Methodology Working Group (BMWG) tests including [RFC2544], which Methodology Working Group (BMWG) tests including [RFC2544], which
discusses and defines several tests that may be used to characterize discusses and defines several tests that may be used to characterize
the performance of a network interconnecting device. All readers of the performance of a network interconnecting device. All readers of
this memo must read and fully understand [RFC2544]. this memo must read and fully understand [RFC2544].
Benchmarking methodologies (beginning with [RFC2544]) have always Benchmarking methodologies (beginning with [RFC2544]) have always
skipping to change at page 3, line 36 skipping to change at page 3, line 36
affected. affected.
Above, we distinguish "telecommunication service" (where a network Above, we distinguish "telecommunication service" (where a network
service provider contracts with a customer to transfer information service provider contracts with a customer to transfer information
between specified interfaces at different geographic locations) from between specified interfaces at different geographic locations) from
the generic term "service". Below, we use the adjective "production" the generic term "service". Below, we use the adjective "production"
to refer to networks carrying live user traffic. [RFC2544] used the to refer to networks carrying live user traffic. [RFC2544] used the
term "real-world" to refer to production networks and to term "real-world" to refer to production networks and to
differentiate them from test networks. differentiate them from test networks.
Although RFC 2544 has been held up as the standard reference for such Although [RFC2544] has been held up as the standard reference for the
testing, we believe that the actual methods used vary from [RFC2544] testing listed above, we believe that the actual methods used vary
in significant ways. Since the only citation is to [RFC2544], the from [RFC2544] in significant ways. Since the only citation is to
modifications are opaque to the standards community and to users in [RFC2544], the modifications are opaque to the standards community
general. and to users in general.
Since applying the test traffic and methods described in [RFC2544] on Since applying the test traffic and methods described in [RFC2544] on
a production network risks causing overload in shared resources there a production network risks causing overload in shared resources,
is direct risk of harming user traffic if the methods are misused in there is direct risk of harming user traffic if the methods are
this way. Therefore, IETF BMWG developed this Applicability misused in this way. Therefore, the IETF BMWG developed this
Statement for [RFC2544] to directly address the situation. Applicability Statement for [RFC2544] to directly address the
situation.
1.1. Requirements Language 1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
2. Scope and Goals 2. Scope and Goals
This memo clarifies the scope of [RFC2544] with the goal to provide This memo clarifies the scope of [RFC2544] with the goal of providing
guidance to the industry on its applicability, which is limited to guidance to the industry on its applicability, which is limited to
laboratory testing. laboratory testing.
3. The Concept of an Isolated Test Environment 3. The Concept of an Isolated Test Environment
An Isolated Test Environment (ITE) used with [RFC2544] methods (as An Isolated Test Environment (ITE) used with the methods of [RFC2544]
illustrated in Figures 1 through 3 of [RFC2544]) has the ability to: (as illustrated in Figures 1 through 3 of [RFC2544]) has the ability
to:
o contain the test streams to paths within the desired set-up o contain the test streams to paths within the desired setup
o prevent non-test traffic from traversing the test set-up o prevent non-test traffic from traversing the test setup
These features allow unfettered experimentation, while at the same These features allow unfettered experimentation, while at the same
time protecting lab equipment management/control LANs and other time protecting lab equipment management/control LANs and other
production networks from the unwanted effects of the test traffic. production networks from the unwanted effects of the test traffic.
4. Why RFC 2544 Methods are intended only for ITE 4. Why the Methods of RFC 2544 Are Intended Only for ITE
The following sections discuss some of the reasons why [RFC2544] The following sections discuss some of the reasons why [RFC2544]
methods are applicable only for isolated laboratory use, and the methods are applicable only for isolated laboratory use, and the
consequences of applying these methods outside the lab environment. consequences of applying these methods outside the lab environment.
4.1. Experimental Control and Accuracy 4.1. Experimental Control and Accuracy
All of the tests described in RFC 2544 require that the tester and All of the tests described in RFC 2544 require that the tester and
device under test are the only devices on the networks that are device under test are the only devices on the networks that are
transmitting data. The presence of other traffic (unwanted on the transmitting data. The presence of other traffic (unwanted on the
skipping to change at page 4, line 48 skipping to change at page 5, line 6
If any other traffic appears and the amount varies over time, the If any other traffic appears and the amount varies over time, the
repeatability of any test result will likely depend to some degree on repeatability of any test result will likely depend to some degree on
the amount and variation of the other traffic. the amount and variation of the other traffic.
The presence of other traffic makes accurate, repeatable, and The presence of other traffic makes accurate, repeatable, and
consistent measurements of the performance of the device under test consistent measurements of the performance of the device under test
very unlikely, since the complete details of test conditions will not very unlikely, since the complete details of test conditions will not
be reported. be reported.
For example, the RFC 2544 Throughput Test attempts to characterize a For example, the RFC 2544 Throughput Test attempts to characterize a
maximum reliable load, thus there will be testing above the maximum maximum reliable load; thus, there will be testing above the maximum
that causes packet/frame loss. Any other sources of traffic on the that causes packet/frame loss. Any other sources of traffic on the
network will cause packet loss to occur at a tester data rate lower network will cause packet loss to occur at a tester data rate lower
than the rate that would be achieved without the extra traffic. than the rate that would be achieved without the extra traffic.
4.2. Containing Damage 4.2. Containing Damage
[RFC2544] methods, specifically to determine Throughput as defined in [RFC2544] methods, specifically to determine Throughput as defined in
[RFC1242] and other benchmarks, may overload the resources of the [RFC1242] and other benchmarks, may overload the resources of the
device under test, and may cause failure modes in the device under device under test, and they may cause failure modes in the device
test. Since failures can become the root cause of more wide-spread under test. Since failures can become the root cause of more
failure, it is clearly desirable to contain all test traffic within widespread failure, it is clearly desirable to contain all test
the ITE. traffic within the ITE.
In addition, such testing can have a negative effect on any traffic In addition, such testing can have a negative effect on any traffic
that shares resources with the test stream(s) since, in most cases, that shares resources with the test stream(s) since, in most cases,
the traffic load will be close to the capacity of the network links. the traffic load will be close to the capacity of the network links.
Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private
IPv4 address range for testing: IPv4 address range for testing:
"...The network addresses 198.18.0.0 through 198.19.255.255 have been "...The network addresses 198.18.0.0 through 198.19.255.255 have been
assigned to the BMWG by the IANA for this purpose. This assignment assigned to the BMWG by the IANA for this purpose. This assignment
skipping to change at page 5, line 38 skipping to change at page 5, line 43
In other words, devices operating on the Internet may be configured In other words, devices operating on the Internet may be configured
to discard any traffic they observe in this address range, as it is to discard any traffic they observe in this address range, as it is
intended for laboratory ITE use only. Thus, if testers using the intended for laboratory ITE use only. Thus, if testers using the
assigned testing address ranges are connected to the Internet and assigned testing address ranges are connected to the Internet and
test packets are forwarded across the Internet, it is likely that the test packets are forwarded across the Internet, it is likely that the
packets will be discarded and the test will not work. packets will be discarded and the test will not work.
We note that a range of IPv6 addresses has been assigned to BMWG for We note that a range of IPv6 addresses has been assigned to BMWG for
laboratory test purposes, in [RFC5180] (as amended by errata). laboratory test purposes, in [RFC5180] (as amended by errata).
See the Security Considerations Section below for further See the Security Considerations section below for further
considerations on containing damage. considerations on containing damage.
5. Advisory on RFC 2544 Methods in Production Networks 5. Advisory on RFC 2544 Methods in Production Networks
The tests in [RFC2544] were designed to measure the performance of The tests in [RFC2544] were designed to measure the performance of
network devices, not of networks, and certainly not production network devices, not of networks, and certainly not production
networks carrying user traffic on shared resources. There will be networks carrying user traffic on shared resources. There will be
undesirable consequences when applying these methods outside the undesirable consequences when applying these methods outside the
isolated test environment. isolated test environment.
skipping to change at page 6, line 13 skipping to change at page 6, line 16
indicator of resource exhaustion in [RFC2544] methods. In practice, indicator of resource exhaustion in [RFC2544] methods. In practice,
link-layer and physical-layer errors prevent production networks from link-layer and physical-layer errors prevent production networks from
operating loss-free. The [RFC2544] methods will not correctly assess operating loss-free. The [RFC2544] methods will not correctly assess
Throughput when loss from uncontrolled sources is present. Frame Throughput when loss from uncontrolled sources is present. Frame
loss occurring at the SLA levels of some networks could affect every loss occurring at the SLA levels of some networks could affect every
iteration of Throughput testing (when each step includes sufficient iteration of Throughput testing (when each step includes sufficient
packets to experience facility-related loss). Flawed results waste packets to experience facility-related loss). Flawed results waste
the time and resources of the testing service user and of the service the time and resources of the testing service user and of the service
provider when called to dispute the measurement. These are provider when called to dispute the measurement. These are
additional examples of harm that compliance with this advisory should additional examples of harm that compliance with this advisory should
help to avoid. See the Appendix for an example. help to avoid. See Appendix A for an example.
The methods described in [RFC2544] are intended to generate traffic The methods described in [RFC2544] are intended to generate traffic
that overloads network device resources in order to assess their that overloads network device resources in order to assess their
capacity. Overload of shared resources would likely be harmful to capacity. Overload of shared resources would likely be harmful to
user traffic performance on a production network. These tests MUST user traffic performance on a production network. These tests MUST
NOT be used on production networks and as discussed above. The tests NOT be used on production networks and as discussed above. The tests
will not produce a reliable or accurate benchmarking result on a will not produce a reliable or accurate benchmarking result on a
production network. production network.
[RFC2544] methods have never been validated on a network path, even [RFC2544] methods have never been validated on a network path, even
when that path is not part of a production network and carrying no when that path is not part of a production network and carrying no
other traffic. It is unknown whether the tests can be used to other traffic. It is unknown whether the tests can be used to
measure valid and reliable performance of a multi-device, multi- measure valid and reliable performance of a multi-device, multi-
network path. It is possible that some of the tests may prove valid network path. It is possible that some of the tests may prove valid
in some path scenarios, but that work has not been done or has not in some path scenarios, but that work has not been done or has not
been shared with the IETF community. Thus, such testing is contra- been shared with the IETF community. Thus, such testing is
indicated by the BMWG. contraindicated by the BMWG.
6. Considering Performance Testing in Production Networks 6. Considering Performance Testing in Production Networks
The IETF has addressed the problem of production network performance The IETF has addressed the problem of production network performance
measurement by chartering a different working group: IP Performance measurement by chartering a different working group: IP Performance
Metrics (IPPM). This working group has developed a set of standard Metrics (IPPM). This working group has developed a set of standard
metrics to assess the quality, performance, and reliability of metrics to assess the quality, performance, and reliability of
Internet packet transfer services. These metrics can be measured by Internet packet transfer services. These metrics can be measured by
network operators, end users, or independent testing groups. We note network operators, end users, or independent testing groups. We note
that some IPPM metrics differ from RFC 2544 metrics with similar that some IPPM metrics differ from RFC 2544 metrics with similar
skipping to change at page 7, line 9 skipping to change at page 7, line 10
possibility for degradation to any other traffic that may be present possibility for degradation to any other traffic that may be present
due to congestion. There are no specific methods proposed for due to congestion. There are no specific methods proposed for
activation of a packet transfer service in IPPM at this time. Thus, activation of a packet transfer service in IPPM at this time. Thus,
individuals who need to conduct capacity tests on production networks individuals who need to conduct capacity tests on production networks
should actively participate in standards development to ensure their should actively participate in standards development to ensure their
methods receive appropriate industry review and agreement, in the methods receive appropriate industry review and agreement, in the
IETF or in alternate standards development organizations. IETF or in alternate standards development organizations.
Other standards may help to fill gaps in telecommunication service Other standards may help to fill gaps in telecommunication service
testing. For example, the IETF has many standards intended to assist testing. For example, the IETF has many standards intended to assist
with network operation, administration and maintenance (OAM), and with network Operations, Administration, and Maintenance (OAM).
ITU-T Study Group 12 has a Recommendation on service activation test ITU-T Study Group 12 has a Recommendation on service activation test
methodology [Y.1564]. methodology [Y.1564].
The world will not spin off axis while waiting for appropriate and The world will not spin off axis while waiting for appropriate and
standardized methods to emerge from the consensus process. standardized methods to emerge from the consensus process.
7. Security Considerations 7. Security Considerations
This Applicability Statement intends to help preserve the security of This Applicability Statement intends to help preserve the security of
the Internet by clarifying that the scope of [RFC2544] and other BMWG the Internet by clarifying that the scope of [RFC2544] and other BMWG
memos are all limited to testing in a laboratory ITE, thus avoiding memos are all limited to testing in a laboratory ITE, thus avoiding
accidental Denial of Service attacks or congestion due to high accidental Denial-of-Service attacks or congestion due to high
traffic volume test streams. traffic volume test streams.
All Benchmarking activities are limited to technology All benchmarking activities are limited to technology
characterization using controlled stimuli in a laboratory characterization using controlled stimuli in a laboratory
environment, with dedicated address space and the other constraints environment, with dedicated address space and the other constraints
[RFC2544]. [RFC2544].
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network, or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the device under test/ solely on measurements observable external to the device under test/
system under test (DUT/SUT). system under test (DUT/SUT).
Special capabilities SHOULD NOT exist in the DUT/SUT specifically for Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
benchmarking purposes. Any implications for network security arising benchmarking purposes. Any implications for network security arising
from the DUT/SUT SHOULD be identical in the lab and in production from the DUT/SUT SHOULD be identical in the lab and in production
networks. networks.
8. IANA Considerations 8. Acknowledgements
This memo makes no requests of IANA.
9. Acknowledgements
Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis
Villamizar, David Newman, and Adrian Farrel for suggesting Villamizar, David Newman, and Adrian Farrel for suggesting
improvements to this memo. improvements to this memo.
Specifically, Al Morton would like to thank his co-authors, who Specifically, Al Morton would like to thank his coauthors, who
constitute the complete set of Chairmen-Emeritus of the BMWG, for constitute the complete set of Chairmen-Emeritus of the BMWG, for
returning from other pursuits to develop this statement and see it returning from other pursuits to develop this statement and see it
through to approval. This has been a rare privilege; one that likely through to approval. This has been a rare privilege; one that likely
will not be matched in the IETF again: will not be matched in the IETF again:
Scott Bradner served as Chairman from 1990 to 1993 Scott Bradner served as Chairman from 1990 to 1993
Jim McQuaid served as Chairman from 1993 to 1995 Jim McQuaid served as Chairman from 1993 to 1995
Kevin Dubray served as Chairman from 1995 to 2006 Kevin Dubray served as Chairman from 1995 to 2006
It's all about the band. It's all about the band.
10. Appendix - Example of RFC 2544 method failure in production network 9. References
measurement
9.1. Normative References
[RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.
[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D.
Dugatkin, "IPv6 Benchmarking Methodology for Network
Interconnect Devices", RFC 5180, May 2008.
9.2. Informative References
[Bryant] Bonica, R. and S. Bryant, "RFC2544 Testing in Production
Network", Work in Progress, October 2012.
[Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation
Test Methodology", March 2011.
Appendix A. Example of RFC 2544 Method Failure in Production Network
Measurement
This Appendix provides an example illustrating how [RFC2544] methods This Appendix provides an example illustrating how [RFC2544] methods
applied on production networks can easily produce a form of harm from applied on production networks can easily produce a form of harm from
flawed and misleading results. flawed and misleading results.
The [RFC2544] Throughput benchmarking method usually includes the The [RFC2544] Throughput benchmarking method usually includes the
following steps: following steps:
a. Set the offered traffic level, less than max of the ingress a. Set the offered traffic level, less than max of the ingress
link(s). link(s).
b. Send the test traffic through the device under test (DUT) and b. Send the test traffic through the device under test (DUT) and
count all frames successfully transferred. count all frames successfully transferred.
c. If all frames are received, increment traffic level and repeat c. If all frames are received, increment traffic level and repeat
step b. step b.
d. If one or more frames are lost, the level is in the DUT-overload d. If one or more frames are lost, the level is in the DUT-overload
region (Step b may be repeated at a reduced traffic level to more region (step b may be repeated at a reduced traffic level to more
exactly determine the maximum rate at which none of the frames are exactly determine the maximum rate at which none of the frames
dropped by the DUT, defined as the Throughput [RFC1242]). are dropped by the DUT, defined as the Throughput [RFC1242]).
e. Report the Throughput values, the x-y of graph of frame size and e. Report the Throughput values, the x-y of graph of frame size and
Throughput, and other information in accordance with [RFC2544]. Throughput, and other information in accordance with [RFC2544].
In this method, frame loss is the sole indicator of overload and In this method, frame loss is the sole indicator of overload and
therefore the determining factor in the measurement of Throughput therefore the determining factor in the measurement of Throughput
using the [RFC2544] methodology (even though the results may not using the [RFC2544] methodology (even though the results may not
report frame loss per se). report frame loss per se).
Frame loss is subject to many factors in addition to operating above Frame loss is subject to many factors in addition to operating above
the Throughput traffic level. These factors include optical the Throughput traffic level. These factors include optical
interference (which may be due to dirty interfaces, cross-over from interference (which may be due to dirty interfaces, crossover from
other signals, fiber bend and temperature, etc.) and electrical other signals, fiber bend and temperature, etc.) and electrical
interference (caused by local sources of radio signals, electrical interference (caused by local sources of radio signals, electrical
spikes, solar particles, etc.). In the laboratory environment many spikes, solar particles, etc.). In the laboratory environment many
of these issues can be carefully controlled through cleaning and of these issues can be carefully controlled through cleaning and
isolation. Since [RFC2544] methodologies are primarily intended to isolation. Since [RFC2544] methodologies are primarily intended to
test devices and not paths, the total length of path, the number of test devices and not paths, the total length of path, the number of
interfaces, and compound risk of random frame loss can be kept to a interfaces, and compound risk of random frame loss can be kept to a
minimum. minimum.
In a production network, however, there will be many interfaces and In a production network, however, there will be many interfaces and
many kilometres of path under test. This considerably increases the many kilometers of path under test. This considerably increases the
risk of random frame loss. risk of random frame loss.
The risk of frame loss caused by outside effects is significantly The risk of frame loss caused by outside effects is significantly
higher in production networks, and significantly higher with long higher in production networks, and significantly higher with long
paths (both those with long physical path lengths, and those with paths (both those with long physical path lengths, and those with
large numbers of interfaces in the path). Thus, the risk of falsely large numbers of interfaces in the path). Thus, the risk of falsely
low reported Throughput using an [RFC2544] methodology test is low reported Throughput using an [RFC2544] methodology test is
considerably increased in a production network. considerably increased in a production network.
Therefore, to successfully conduct tests with similar objectives to Therefore, to successfully conduct tests with similar objectives to
those in [RFC2544] in a production network, it will be necessary to those in [RFC2544] in a production network, it will be necessary to
develop modifications to the methodologies defined in [RFC2544] and develop modifications to the methodologies defined in [RFC2544] and
standards to describe them. See [Bryant] for an in-progress effort standards to describe them. See [Bryant] for an in-progress effort
and [Y.1564] for an approved method adapted to production service and [Y.1564] for an approved method adapted to production service
activation. activation.
11. References
11.1. Normative References
[RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.
[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D.
Dugatkin, "IPv6 Benchmarking Methodology for Network
Interconnect Devices", RFC 5180, May 2008.
11.2. Informative References
[Bryant] Bonica, R. and S. Bryant, "Work in-progress, "RFC2544
Testing in Production Network",
(draft-bb-2544like-production-tests-00)", October 2012.
[Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation
Test Methodology", March 2011.
Authors' Addresses Authors' Addresses
Scott Bradner Scott Bradner
Harvard University Harvard University
29 Oxford St. 1350 Mass. Ave., Room 760
Cambridge, MA 02138 Cambridge, MA 02138
USA USA
Phone: +1 617 495 3864 Phone: +1 617 495 3864
Fax: EMail: sob@harvard.edu
Email: sob@harvard.edu
URI: http://www.sobco.com URI: http://www.sobco.com
Kevin Dubray Kevin Dubray
Juniper Networks Juniper Networks
Phone:
Fax:
Email: kdubray@juniper.net
URI:
Jim McQuaid Jim McQuaid
Turnip Video Turnip Video
6 Cobbleridge Court 6 Cobbleridge Court
Durham, North Carolina 27713 Durham, North Carolina 27713
USA USA
Phone: +1 919-619-3220 Phone: +1 919-619-3220
Fax: EMail: jim@turnipvideo.com
Email: jim@turnipvideo.com
URI: www.turnipvideo.com URI: www.turnipvideo.com
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown,, NJ 07748 Middletown,, NJ 07748
USA USA
Phone: +1 732 420 1571 Phone: +1 732 420 1571
Fax: +1 732 368 1192 Fax: +1 732 368 1192
Email: acmorton@att.com EMail: acmorton@att.com
URI: http://home.comcast.net/~acmacm/ URI: http://home.comcast.net/~acmacm/
 End of changes. 44 change blocks. 
126 lines changed or deleted 113 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/