draft-ietf-bmwg-ipv6-tran-tech-benchmarking-05.txt   draft-ietf-bmwg-ipv6-tran-tech-benchmarking-06.txt 
Benchmarking Working Group M. Georgescu Benchmarking Working Group M. Georgescu
Internet Draft L. Pislaru Internet Draft L. Pislaru
Intended status: Informational RCS&RDS Intended status: Informational RCS&RDS
Expires: September 2017 G. Lencse Expires: October 2017 G. Lencse
Szechenyi Istvan University Szechenyi Istvan University
March 29, 2017 April 18, 2017
Benchmarking Methodology for IPv6 Transition Technologies Benchmarking Methodology for IPv6 Transition Technologies
draft-ietf-bmwg-ipv6-tran-tech-benchmarking-05.txt draft-ietf-bmwg-ipv6-tran-tech-benchmarking-06.txt
Abstract Abstract
There are benchmarking methodologies addressing the performance of There are benchmarking methodologies addressing the performance of
network interconnect devices that are IPv4- or IPv6-capable, but the network interconnect devices that are IPv4- or IPv6-capable, but the
IPv6 transition technologies are outside of their scope. This IPv6 transition technologies are outside of their scope. This
document provides complementary guidelines for evaluating the document provides complementary guidelines for evaluating the
performance of IPv6 transition technologies. More specifically, performance of IPv6 transition technologies. More specifically,
this document targets IPv6 transition technologies that employ this document targets IPv6 transition technologies that employ
encapsulation or translation mechanisms, as dual-stack nodes can be encapsulation or translation mechanisms, as dual-stack nodes can be
very well tested using the recommendations of RFC2544 and RFC5180. very well tested using the recommendations of RFC2544 and RFC5180.
The methodology also includes a tentative metric for benchmarking The methodology also includes a metric for benchmarking load
load scalability. scalability.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 46 skipping to change at page 1, line 46
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
This Internet-Draft will expire on September 29, 2016. This Internet-Draft will expire on October 18, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 4, line 8 skipping to change at page 4, line 8
This document presents benchmarking guidelines dedicated to IPv6 This document presents benchmarking guidelines dedicated to IPv6
transition technologies. The benchmarking tests can provide insights transition technologies. The benchmarking tests can provide insights
about the performance of these technologies, which can act as useful about the performance of these technologies, which can act as useful
feedback for developers, as well as for network operators going feedback for developers, as well as for network operators going
through the IPv6 transition process. through the IPv6 transition process.
The document also includes an approach to quantify performance when The document also includes an approach to quantify performance when
operating in overload. Overload scalability can be defined as a operating in overload. Overload scalability can be defined as a
system's ability to gracefully accommodate greater numbers of flows system's ability to gracefully accommodate greater numbers of flows
than the maximum number of flows which the DUT can operate normally. than the maximum number of flows which the Device under test (DUT)
The approach taken here is to quantify the overload scalability by can operate normally. The approach taken here is to quantify the
measuring the performance created by an excessive number of network overload scalability by measuring the performance created by an
flows, and comparing performance to the non-overloaded case. excessive number of network flows, and comparing performance to the
non-overloaded case.
1.1. IPv6 Transition Technologies 1.1. IPv6 Transition Technologies
Two of the basic transition technologies, dual IP layer (also known Two of the basic transition technologies, dual IP layer (also known
as dual stack) and encapsulation are presented in [RFC4213]. as dual stack) and encapsulation are presented in [RFC4213].
IPv4/IPv6 Translation is presented in [RFC6144]. Most of the IPv4/IPv6 Translation is presented in [RFC6144]. Most of the
transition technologies employ at least one variation of these transition technologies employ at least one variation of these
mechanisms. Some of the more complex ones (e.g. DSLite [RFC6333]) mechanisms. In this context, a generic classification of the
are using all three. In this context, a generic classification of transition technologies can prove useful.
the transition technologies can prove useful.
Tentatively, we can consider a production network transitioning to We can consider a production network transitioning to IPv6 as being
IPv6 as being constructed using the following IP domains: constructed using the following IP domains:
o Domain A: IPvX specific domain o Domain A: IPvX specific domain
o Core domain: which may be IPvY specific or dual-stack(IPvX and o Core domain: which may be IPvY specific or dual-stack(IPvX and
IPvY) IPvY)
o Domain B: IPvX specific domain o Domain B: IPvX specific domain
Note: X,Y are part of the set {4,6}, and X NOT.EQUAL Y. Note: X,Y are part of the set {4,6}, and X NOT.EQUAL Y.
skipping to change at page 4, line 47 skipping to change at page 4, line 47
1. Dual-stack: the core domain devices implement both IP protocols. 1. Dual-stack: the core domain devices implement both IP protocols.
2. Single Translation: In this case, the production network is 2. Single Translation: In this case, the production network is
assumed to have only two domains, Domain A and the Core domain. assumed to have only two domains, Domain A and the Core domain.
The core domain is assumed to be IPvY specific. IPvX packets are The core domain is assumed to be IPvY specific. IPvX packets are
translated to IPvY at the edge between Domain A and the Core translated to IPvY at the edge between Domain A and the Core
domain. domain.
3. Double translation: The production network is assumed to have all 3. Double translation: The production network is assumed to have all
three domains, Domains A and B are IPvX specific, while the core three domains; Domains A and B are IPvX specific, while the core
domain is IPvY specific. A translation mechanism is employed for domain is IPvY specific. A translation mechanism is employed for
the traversal of the core network. The IPvX packets are the traversal of the core network. The IPvX packets are
translated to IPvY packets at the edge between Domain A and the translated to IPvY packets at the edge between Domain A and the
Core domain. Subsequently, the IPvY packets are translated back Core domain. Subsequently, the IPvY packets are translated back
to IPvX at the edge between the Core domain and Domain B. to IPvX at the edge between the Core domain and Domain B.
4. Encapsulation: The production network is assumed to have all 4. Encapsulation: The production network is assumed to have all
three domains, Domains A and B are IPvX specific, while the core three domains; Domains A and B are IPvX specific, while the core
domain is IPvY specific. An encapsulation mechanism is used to domain is IPvY specific. An encapsulation mechanism is used to
traverse the core domain. The IPvX packets are encapsulated to traverse the core domain. The IPvX packets are encapsulated to
IPvY packets at the edge between Domain A and the Core domain. IPvY packets at the edge between Domain A and the Core domain.
Subsequently, the IPvY packets are de-encapsulated at the edge Subsequently, the IPvY packets are de-encapsulated at the edge
between the Core domain and Domain B. between the Core domain and Domain B.
The performance of Dual-stack transition technologies can be fully The performance of Dual-stack transition technologies can be fully
evaluated using the benchmarking methodologies presented by evaluated using the benchmarking methodologies presented by
[RFC2544] and [RFC5180]. Consequently, this document focuses on the [RFC2544] and [RFC5180]. Consequently, this document focuses on the
other 3 categories: Single translation, Encapsulation and Double other 3 categories: Single translation, Encapsulation and Double
skipping to change at page 8, line 39 skipping to change at page 8, line 39
The test traffic represents the experimental workload and SHOULD The test traffic represents the experimental workload and SHOULD
meet the requirements specified in this section. The requirements meet the requirements specified in this section. The requirements
are dedicated to unicast IP traffic. Multicast IP traffic is outside are dedicated to unicast IP traffic. Multicast IP traffic is outside
of the scope of this document. of the scope of this document.
5.1. Frame Formats and Sizes 5.1. Frame Formats and Sizes
[RFC5180] describes the frame size requirements for two commonly [RFC5180] describes the frame size requirements for two commonly
used media types: Ethernet and SONET (Synchronous Optical Network). used media types: Ethernet and SONET (Synchronous Optical Network).
[RFC2544] covers also other media types, such as token ring and [RFC2544] covers also other media types, such as token ring and
FDDI. The two documents can be referred for the dual-stack FDDI. The recommendations of the two documents can be used for the
transition technologies. For the rest of the transition technologies dual-stack transition technologies. For the rest of the transition
the frame overhead introduced by translation or encapsulation MUST technologies, the frame overhead introduced by translation or
be considered. encapsulation MUST be considered.
The encapsulation/translation process generates different size The encapsulation/translation process generates different size
frames on different segments of the test setup. For instance, the frames on different segments of the test setup. For instance, the
single translation transition technologies will create different single translation transition technologies will create different
frame sizes on the receiving segment of the test setup, as IPvX frame sizes on the receiving segment of the test setup, as IPvX
packets are translated to IPvY. This is not a problem if the packets are translated to IPvY. This is not a problem if the
bandwidth of the employed media is not exceeded. To prevent bandwidth of the employed media is not exceeded. To prevent
exceeding the limitations imposed by the media, the frame size exceeding the limitations imposed by the media, the frame size
overhead needs to be taken into account when calculating the maximum overhead needs to be taken into account when calculating the maximum
theoretical frame rates. The calculation method for the Ethernet, as theoretical frame rates. The calculation method for the Ethernet, as
well as a calculation example are detailed in Appendix A. The well as a calculation example, are detailed in Appendix A. The
details of the media employed for the benchmarking tests MUST be details of the media employed for the benchmarking tests MUST be
noted in all test reports. noted in all test reports.
In the context of frame size overhead, MTU recommendations are In the context of frame size overhead, MTU recommendations are
needed in order to avoid frame loss due to MTU mismatch between the needed in order to avoid frame loss due to MTU mismatch between the
virtual encapsulation/translation interfaces and the physical virtual encapsulation/translation interfaces and the physical
network interface controllers (NICs). To avoid this situation, the network interface controllers (NICs). To avoid this situation, the
larger MTU between the physical NICs and virtual larger MTU between the physical NICs and virtual
encapsulation/translation interfaces SHOULD be set for all encapsulation/translation interfaces SHOULD be set for all
interfaces of the DUT and tester. To be more specific, the minimum interfaces of the DUT and tester. To be more specific, the minimum
skipping to change at page 10, line 32 skipping to change at page 10, line 32
For the benchmarks dedicated to stateful IPv6 transition For the benchmarks dedicated to stateful IPv6 transition
technologies, included in Section 8 of this memo (Concurrent TCP technologies, included in Section 8 of this memo (Concurrent TCP
Connection Capacity and Maximum TCP Connection Establishment Rate), Connection Capacity and Maximum TCP Connection Establishment Rate),
the traffic SHOULD follow the recommendations of [RFC3511], Sections the traffic SHOULD follow the recommendations of [RFC3511], Sections
5.2.2.2 and 5.3.2.2. 5.2.2.2 and 5.3.2.2.
6. Modifiers 6. Modifiers
The idea of testing under different operational conditions was first The idea of testing under different operational conditions was first
introduced in [RFC2544](Section 11) and represents an important introduced in [RFC2544](Section 11) and represents an important
aspect of benchmarking network elements, as it emulates to some aspect of benchmarking network elements, as it emulates, to some
extent the conditions of a production environment. Section 6 of extent, the conditions of a production environment. Section 6 of
[RFC5180] describes complementary testing conditions specific to [RFC5180] describes complementary testing conditions specific to
IPv6. Their recommendations can be referred for IPv6 transition IPv6. Their recommendations can be followed for IPv6 transition
technologies testing as well. technologies testing as well.
7. Benchmarking Tests 7. Benchmarking Tests
The following sub-sections contain the list of all recommended The following sub-sections contain the list of all recommended
benchmarking tests. benchmarking tests.
7.1. Throughput 7.1. Throughput
Use Section 26.1 of RFC2544 unmodified. Use Section 26.1 of RFC2544 unmodified.
skipping to change at page 12, line 16 skipping to change at page 12, line 16
Reporting Format: The report MUST state which definition of latency Reporting Format: The report MUST state which definition of latency
(from RFC 1242) was used for this test. The summarized latency (from RFC 1242) was used for this test. The summarized latency
results SHOULD be reported in the format of a table with a row for results SHOULD be reported in the format of a table with a row for
each of the tested frame sizes. There SHOULD be columns for the each of the tested frame sizes. There SHOULD be columns for the
frame size, the rate at which the latency test was run for that frame size, the rate at which the latency test was run for that
frame size, for the media types tested, and for the resultant frame size, for the media types tested, and for the resultant
typical latency and worst case latency values for each type of data typical latency and worst case latency values for each type of data
stream tested. To account for the variation, the 1st and 99th stream tested. To account for the variation, the 1st and 99th
percentiles of the 20 iterations MAY be reported in two separated percentiles of the 20 iterations MAY be reported in two separated
columns. For a fine grain analysis, the histogram (as exemplified in columns. For a fine grained analysis, the histogram (as exemplified
[RFC5481] Section 4.4) of one of the iterations MAY be displayed as in [RFC5481] Section 4.4) of one of the iterations MAY be displayed
well. as well.
7.3. Packet Delay Variation 7.3. Packet Delay Variation
Considering two of the metrics presented in [RFC5481], Packet Delay Considering two of the metrics presented in [RFC5481], Packet Delay
Variation (PDV) and Inter Packet Delay Variation (IPDV), it is Variation (PDV) and Inter Packet Delay Variation (IPDV), it is
RECOMMENDED to measure PDV. For a fine grain analysis of delay RECOMMENDED to measure PDV. For a fine grained analysis of delay
variation, IPDV measurements MAY be performed as well. variation, IPDV measurements MAY be performed as well.
7.3.1. PDV 7.3.1. PDV
Objective: To determine the Packet Delay Variation as defined in Objective: To determine the Packet Delay Variation as defined in
[RFC5481]. [RFC5481].
Procedure: As described by [RFC2544], first determine the throughput Procedure: As described by [RFC2544], first determine the throughput
for the DUT at each of the listed frame sizes. Send a stream of for the DUT at each of the listed frame sizes. Send a stream of
frames at a particular frame size through the DUT at the determined frames at a particular frame size through the DUT at the determined
skipping to change at page 14, line 26 skipping to change at page 14, line 26
7.7. Reset 7.7. Reset
Use Section 4 of [RFC6201] unmodified. Use Section 4 of [RFC6201] unmodified.
8. Additional Benchmarking Tests for Stateful IPv6 Transition 8. Additional Benchmarking Tests for Stateful IPv6 Transition
Technologies Technologies
This section describes additional tests dedicated to the stateful This section describes additional tests dedicated to the stateful
IPv6 transition technologies. For the tests described in this IPv6 transition technologies. For the tests described in this
section the DUT devices SHOULD follow the test setup and test section, the DUT devices SHOULD follow the test setup and test
parameters recommendations presented in [RFC3511] (Sections 4, 5). parameters recommendations presented in [RFC3511] (Sections 4, 5).
In addition to the IPv4/IPv6 transition function, a network node can In addition to the IPv4/IPv6 transition function, a network node can
have a firewall function. This document is targeting only the have a firewall function. This document is targeting only the
network devices that do not have a firewall function, as this network devices that do not have a firewall function, as this
function can be benchmarked using the recommendations of [RFC3511]. function can be benchmarked using the recommendations of [RFC3511].
Consequently, only the tests described in [RFC3511] (Sections 5.2, Consequently, only the tests described in [RFC3511] (Sections 5.2,
5.3) are RECOMMENDED. Namely, the following additional tests SHOULD 5.3) are RECOMMENDED. Namely, the following additional tests SHOULD
be performed: be performed:
skipping to change at page 15, line 48 skipping to change at page 15, line 48
server) server)
6. Synthesized AAAA record answer (from DNS64 server to client) 6. Synthesized AAAA record answer (from DNS64 server to client)
The Tester plays the role of DNS client as well as authoritative DNS The Tester plays the role of DNS client as well as authoritative DNS
server. It MAY be realized as a single physical device, or server. It MAY be realized as a single physical device, or
alternatively, two physical devices MAY be used. alternatively, two physical devices MAY be used.
Please note that: Please note that:
- If the DNS64 server implements caching and there is a cache hit - If the DNS64 server implements caching and there is a cache
then step 1 is followed by step 6 (and steps 2 through 5 are hit, then step 1 is followed by step 6 (and steps 2 through 5
omitted). are omitted).
- If the domain name has an AAAA record then it is returned in - If the domain name has an AAAA record, then it is returned in
step 3 by the authoritative DNS server, steps 4 and 5 are step 3 by the authoritative DNS server; steps 4 and 5 are
omitted, and the DNS64 server does not synthesizes an AAAA omitted, and the DNS64 server does not synthesizes an AAAA
record, but returns the received AAAA record to the client. record, but returns the received AAAA record to the client.
- As for the IP version used between the tester and the DUT, IPv6 - As for the IP version used between the tester and the DUT, IPv6
MUST be used between the client and the DNS64 server (as a MUST be used between the client and the DNS64 server (as a
DNS64 server provides service for an IPv6-only client), but DNS64 server provides service for an IPv6-only client), but
either IPv4 or IPv6 MAY be used between the DNS64 server and either IPv4 or IPv6 MAY be used between the DNS64 server and
the authoritative DNS server. the authoritative DNS server.
9.2. Benchmarking DNS Resolution Performance 9.2. Benchmarking DNS Resolution Performance
skipping to change at page 17, line 15 skipping to change at page 17, line 15
2. Existence of AAAA record 2. Existence of AAAA record
First, all the DNS queries MUST contain domain names which do not First, all the DNS queries MUST contain domain names which do not
have an AAAA record and have exactly one A record. have an AAAA record and have exactly one A record.
Then new tests MAY be executed with domain names, 20%, 40%, 60%, 80% Then new tests MAY be executed with domain names, 20%, 40%, 60%, 80%
and 100% of which have an AAAA record. and 100% of which have an AAAA record.
Please note that the two conditions above are orthogonal, thus all Please note that the two conditions above are orthogonal, thus all
their combinations are possible and MAY be tested. The testing with their combinations are possible and MAY be tested. The testing with
0% cached domain names and with 0% existing AAAA record is REQUIRED 0% cached domain names and with 0% existing AAAA record is REQUIRED
and the other combinations are OPTIONAL. (When all the domain names and the other combinations are OPTIONAL. (When all the domain names
are cached then the results do not depend on what percentage of the are cached, then the results do not depend on what percentage of the
domain names have AAAA records, thus these combinations are not domain names have AAAA records, thus these combinations are not
worth testing one by one.) worth testing one by one.)
Reporting format: The primary result of the DNS64 test is the median Reporting format: The primary result of the DNS64 test is the median
of the number of processed DNS queries per second measured with the of the number of processed DNS queries per second measured with the
above mentioned "0% + 0% combination". The median SHOULD be above mentioned "0% + 0% combination". The median SHOULD be
complemented with the 1st and 99th percentiles to show the stability complemented with the 1st and 99th percentiles to show the stability
of the result. If optional tests are done, the median and the 1st of the result. If optional tests are done, the median and the 1st
and 99th percentiles MAY be presented in a two dimensional table and 99th percentiles MAY be presented in a two dimensional table
where the dimensions are the proportion of the repeated domain names where the dimensions are the proportion of the repeated domain names
skipping to change at page 18, line 7 skipping to change at page 18, line 7
Explanation: When performing DNS64 testing, each AAAA record query Explanation: When performing DNS64 testing, each AAAA record query
may result in at most two queries sent by the DUT, the first one of may result in at most two queries sent by the DUT, the first one of
them is for an AAAA record and the second one is for an A record them is for an AAAA record and the second one is for an A record
(the are both sent when there is no cache hit and also no AAAA (the are both sent when there is no cache hit and also no AAAA
record exists). The parameters above guarantee that the record exists). The parameters above guarantee that the
authoritative DNS server subsystem of the DUT is able to answer the authoritative DNS server subsystem of the DUT is able to answer the
queries at the required frequency using up not more than the half of queries at the required frequency using up not more than the half of
the timeout time. the timeout time.
Remark: a sample open-source test program, dns64perf++ is available Remark: a sample open-source test program, dns64perf++, is available
from [Dns64perf] and it is documented in [Lencse1]. It implements from [Dns64perf] and it is documented in [Lencse1]. It implements
only the client part of the Tester and it should be used together only the client part of the Tester and it should be used together
with an authoritative DNS server implementation, e.g. BIND, NSD or with an authoritative DNS server implementation, e.g. BIND, NSD or
YADIFA. Its experimental extension for testing caching is available YADIFA. Its experimental extension for testing caching is available
from [Lencse2] and it is documented in [Lencse3]. from [Lencse2] and it is documented in [Lencse3].
10. Overload Scalability 10. Overload Scalability
Scalability has been often discussed; however, in the context of Scalability has been often discussed; however, in the context of
network devices, a formal definition or a measurement method has not network devices, a formal definition or a measurement method has not
skipping to change at page 20, line 5 skipping to change at page 20, line 5
| +-----------------+ | | | | +-----------------+ | | |
| +-----------------+ | | | | +-----------------+ | | |
+---------->| NF1 DUT 1 NF1 |--->|NF1 NF1|-----------+ +---------->| NF1 DUT 1 NF1 |--->|NF1 NF1|-----------+
+-----------------+ +---------------+ +-----------------+ +---------------+
Figure 5. Test setup 4 Figure 5. Test setup 4
This test setup can help to quantify the scalability of the server This test setup can help to quantify the scalability of the server
device. However, for testing the overload scalability of the client device. However, for testing the overload scalability of the client
DUTs additional recommendations are needed. DUTs additional recommendations are needed.
For encapsulation transition technologies a m:n setup can be For encapsulation transition technologies, a m:n setup can be
created, where m is the number of flows applied to the same client created, where m is the number of flows applied to the same client
device and n the number of client devices connected to the same device and n the number of client devices connected to the same
server device. server device.
For the translation based transition technologies the client devices For the translation based transition technologies, the client
can be separately tested with n network flows using the test setup devices can be separately tested with n network flows using the test
presented in Figure 4. setup presented in Figure 4.
10.2. Benchmarking Performance Degradation 10.2. Benchmarking Performance Degradation
10.2.1. Network performance degradation with simultaneous load 10.2.1. Network performance degradation with simultaneous load
Objective: To quantify the performance degradation introduced by n Objective: To quantify the performance degradation introduced by n
parallel and simultaneous network flows. parallel and simultaneous network flows.
Procedure: First, the benchmarking tests presented in Section 6 have Procedure: First, the benchmarking tests presented in Section 6 have
to be performed for one network flow. to be performed for one network flow.
skipping to change at page 21, line 7 skipping to change at page 21, line 7
10.2.2. Network performance degradation with incremental load 10.2.2. Network performance degradation with incremental load
Objective: To quantify the performance degradation introduced by n Objective: To quantify the performance degradation introduced by n
parallel and incrementally started network flows. parallel and incrementally started network flows.
Procedure: First, the benchmarking tests presented in Section 6 have Procedure: First, the benchmarking tests presented in Section 6 have
to be performed for one network flow. to be performed for one network flow.
The same tests have to be repeated for n network flows, where the The same tests have to be repeated for n network flows, where the
network flows are started incrementally in succession, each after network flows are started incrementally in succession, each after
time T. In other words, if flow I is started at time x, flow i+1 time t. In other words, if flow i is started at time x, flow i+1
will be started at time x+T. Considering the time T, the time will be started at time x+t. Considering the time t, the time
duration of each iteration must be extended with the time necessary duration of each iteration must be extended with the time necessary
to start all the flows, namely (n-1)xT. The measurement for the to start all the flows, namely (n-1)xt. The measurement for the
first flow SHOULD be at least 60 seconds in duration. first flow SHOULD be at least 60 seconds in duration.
The performance degradation of the X benchmarking dimension SHOULD The performance degradation of the x benchmarking dimension SHOULD
be calculated as relative performance change between the 1-flow be calculated as relative performance change between the 1-flow
results and the n-flow results, using the following formula results and the n-flow results, using the formula presented in
presented in Section 9.2.1. Intermediary degradation points for Section 10.2.1. Intermediary degradation points for 1/4*n, 1/2*n and
1/4*n, 1/2*n and 3/4*n MAY also be performed. 3/4*n MAY also be performed.
Reporting Format: The performance degradation SHOULD be expressed as Reporting Format: The performance degradation SHOULD be expressed as
a percentage. The number of tested parallel flows n MUST be clearly a percentage. The number of tested parallel flows n MUST be clearly
specified. For each of the performed benchmarking tests, there specified. For each of the performed benchmarking tests, there
SHOULD be a table containing a column for each frame size. The table SHOULD be a table containing a column for each frame size. The table
SHOULD also state the applied frame rate and time duration T, used SHOULD also state the applied frame rate and time duration T, used
as increment step between the network flows. The units of as increment step between the network flows. The units of
measurement for T SHOULD be seconds. A column for the intermediary measurement for T SHOULD be seconds. A column for the intermediary
degradation points MAY also be displayed. degradation points MAY also be displayed.
11. NAT44 and NAT66 11. NAT44 and NAT66
Although these technologies are not the primarily scope of this Although these technologies are not the primary scope of this
document, the benchmarking methodology associated with single document, the benchmarking methodology associated with single
translation technologies as defined in Section 4.1 can be employed translation technologies as defined in Section 4.1 can be employed
to benchmark NAT44 (as defined by [RFC2663] with the behavior to benchmark NAT44 (as defined by [RFC2663] with the behavior
described by [RFC7857]) implementations and NAT66 (as defined by described by [RFC7857]) implementations and NAT66 (as defined by
[RFC6296]) implementations. [RFC6296]) implementations.
12. Summarizing function and variation 12. Summarizing function and variation
To ensure the stability of the benchmarking scores obtained using To ensure the stability of the benchmarking scores obtained using
the tests presented in Sections 6 through 9, multiple test the tests presented in Sections 6 through 9, multiple test
skipping to change at page 22, line 11 skipping to change at page 22, line 11
the over-summarization effect. Empirical data obtained following the the over-summarization effect. Empirical data obtained following the
proposed methodology can also offer insights on which summarizing proposed methodology can also offer insights on which summarizing
function would fit better. function would fit better.
To that end, data presented in [ietf95pres] indicate the median as To that end, data presented in [ietf95pres] indicate the median as
suitable summarizing function and the 1st and 99th percentiles as suitable summarizing function and the 1st and 99th percentiles as
variation measures for DNS Resolution Performance and PDV. The variation measures for DNS Resolution Performance and PDV. The
median and percentile calculation functions SHOULD follow the median and percentile calculation functions SHOULD follow the
recommendations of [RFC2330] Section 11.3. recommendations of [RFC2330] Section 11.3.
For a fine grain analysis of the frequency distribution of the data, For a fine grained analysis of the frequency distribution of the
histograms or cumulative distribution function plots can be data, histograms or cumulative distribution function plots can be
employed. employed.
13. Security Considerations 13. Security Considerations
Benchmarking activities as described in this memo are limited to Benchmarking activities as described in this memo are limited to
technology characterization using controlled stimuli in a laboratory technology characterization using controlled stimuli in a laboratory
environment, with dedicated address space and the constraints environment, with dedicated address space and the constraints
specified in the sections above. specified in the sections above.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
skipping to change at page 23, line 15 skipping to change at page 23, line 15
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP performance metrics", RFC 2330, DOI "Framework for IP performance metrics", RFC 2330, DOI
10.17487/RFC2330, May 1998, <http://www.rfc- 10.17487/RFC2330, May 1998, <http://www.rfc-
editor.org/info/rfc2330>. editor.org/info/rfc2330>.
[RFC2544] Bradner, S., and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S., and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, DOI Network Interconnect Devices", RFC 2544, DOI
10.17487/RFC2544, March 1999, <http://www.rfc- 10.17487/RFC2544, March 1999, <http://www.rfc-
editor.org/info/rfc2544>. editor.org/info/rfc2544>.
[RFC2647] Newman, D., "Benchmarking Terminology for Firewall
Devices", RFC 2647, DOI 10.17487/RFC1242, August 1999,
<http://www.rfc-editor.org/info/rfc2647>.
[RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
Metric for IP Performance Metrics (IPPM)", RFC 3393, DOI Metric for IP Performance Metrics (IPPM)", RFC 3393, DOI
10.17487/RFC3393, November 2002, <http://www.rfc- 10.17487/RFC3393, November 2002, <http://www.rfc-
editor.org/info/rfc3393>. editor.org/info/rfc3393>.
[RFC3511] Hickman, B., Newman, D., Tadjudin, S. and T. Martin, [RFC3511] Hickman, B., Newman, D., Tadjudin, S. and T. Martin,
"Benchmarking Methodology for Firewall Performance", RFC "Benchmarking Methodology for Firewall Performance", RFC
3511, DOI 10.17487/RFC3511, April 2003, <http://www.rfc- 3511, DOI 10.17487/RFC3511, April 2003, <http://www.rfc-
editor.org/info/rfc3511>. editor.org/info/rfc3511>.
 End of changes. 30 change blocks. 
55 lines changed or deleted 51 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/