[Docs] [txt|pdf|xml|html] [Tracker] [Email] [Nits]



Benchmarking Methodology Working Group                         G. Lencse
Internet-Draft                                                      BUTE
Intended status: Informational                                  K. Shima
Expires: November 21, 2020                                        IIJ-II
                                                            May 20, 2020


An Upgrade to Benchmarking Methodology for Network Interconnect Devices
                    draft-lencse-bmwg-rfc2544-bis-00

Abstract

   RFC 2544 has defined a benchmarking methodology for network
   interconnect devices.  We recommend a few upgrades to it for
   producing more reasonable results.  The recommended upgrades can be
   classified into two categories: the application of the novelties of
   RFC 8219 for the legacy RFC 2544 use cases and the following new
   ones.  Checking a reasonably small timeout individually for every
   single frame in the throughput and frame loss rate benchmarking
   procedures.  Performing a statistically relevant number of tests for
   all benchmarking procedures.  Addition of an optional non-zero frame
   loss acceptance criterion for the throughput measurement procedure
   and defining its reporting format.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on November 21, 2020.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents



Lencse & Shima          Expires November 21, 2020               [Page 1]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Recommendation to Backport the Novelties of RFC8219 . . . . .   3
   3.  Improved Throughput and Frame Loss Rate Measurement
       Procedures using Individual Frame Timeout . . . . . . . . . .   3
   4.  Requirement of Statistically Relevant Number of Tests . . . .   4
   5.  An Optional Non-zero Frame Loss Acceptance Criterion for the
       Throughput Measurement  Procedure . . . . . . . . . . . . . .   5
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   6
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   6
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .   7
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   7
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .   7
     9.2.  Informative References  . . . . . . . . . . . . . . . . .   7
   Appendix A.  Change Log . . . . . . . . . . . . . . . . . . . . .   8
     A.1.  00  . . . . . . . . . . . . . . . . . . . . . . . . . . .   8
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   8

1.  Introduction

   [RFC2544] has defined a benchmarking methodology for network
   interconnect devices.  [RFC5180] addressed IPv6 specificities and
   also added technology updates, but declared IPv6 transition
   technologies out of its scope.  [RFC8219] addressed the IPv6
   transition technologies, and it added further measurement procedures
   (e.g. for packet delay variation (PDV) and inter packet delay
   variation (IPDV)).  It has also recommended to perform multiple tests
   (at least 20), and it proposed median as summarizing function and 1st
   and 99th percentiles as the measure of variation of the results of
   the multiple tests.  This is a significant change compared to
   [RFC2544], which always used only average as summarizing function.
   [RFC8219] also redefined the latency measurement procedure with the
   requirement of marking at least 500 frames with identifying tags for
   latency measurements, instead of using only a single one.  However,
   all these improvements apply only for the IPv6 transition
   technologies, and no update was made to [RFC2544] / [RFC5180], which
   we believe to be desirable.




Lencse & Shima          Expires November 21, 2020               [Page 2]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   Moreover, [RFC8219] has reused the throughput and frame loss rate
   benchmarking procedures from [RFC2544] with no changes.  When we
   tested their feasibility with a few SIIT [RFC7915] implementations,
   we have pointed out three possible improvements in [LEN2020A]:

   o  Checking a reasonably small timeout individually for every single
      frame with the throughput and frame loss rate benchmarking
      procedures.

   o  Performing a statistically relevant number of tests for these two
      benchmarking procedures.

   o  Addition of an optional non-zero frame loss acceptance criterion
      for the throughput benchmarking procedure and defining its
      reporting format.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  Recommendation to Backport the Novelties of RFC8219

   Besides addressing IPv6 transition technologies, [RFC8219] has also
   made several technological upgrades reflecting the current state of
   the art of networking technologies and benchmarking.  But all the
   novelties mentioned in Section 1 of this document currently apply
   only for the benchmarking of IPv6 transition technologies.  We
   contend that they could be simply backported to the benchmarking of
   network interconnect devices.  For example, siitperf [SIITPERF], our
   [RFC8219] compliant DPDK-based software Tester was designed for
   benchmarking different SIIT [RFC7915] (also called stateless NAT64)
   implementations, but if it is configured to have the same IP version
   on both sides, it can be used to test IPv4 or IPv6 (or dual stack)
   routers [LEN2020B].  We highly recommend the backporting of the
   latency, PDV and IPDV benchmarking measurement procedures of
   [RFC8219].

3.  Improved Throughput and Frame Loss Rate Measurement Procedures using
    Individual Frame Timeout

   The throughput measurement procedure defined in [RFC2544] only counts
   the number of the sent and received test frames, but it does not
   identify the test frames individually.  On the one hand, this
   approach allows the Tester to send always the very same test frame to



Lencse & Shima          Expires November 21, 2020               [Page 3]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   the DUT, which was very likely an important advantage in 1999.
   However, on the other hand, thus the Tester cannot check if the order
   of the frames is kept, or if the frames arrive back to the Tester
   within a given timeout time.  (Perhaps none of them was an issue of
   hardware based network interconnect devices in 1999.  But today
   network packet forwarding and manipulation is often implemented in
   software having larger buffers and producing potentially higher
   latencies.)

   Whereas real-time applications are obviously time sensitive, other
   applications like HTTP or FTP are often considered throughput hungry
   and time insensitive.  However, we have demonstrated that when we
   applied 100ms delay to 1% of the test frames, the throughput of HTTP
   download dropped by more that 50% [LEN2020C].  Therefore, an advanced
   throughput measurement procedure that checks the timeout time for
   every single test frame may produce more reasonable results.  We have
   shown that this measurement is now feasible [LEN2020B].  In this
   case, we used 64-bit integers to identify the test frames and
   measured the latency of the frames as required by the PDV measurement
   procedure in Section 7.3.1. of [RFC8219].  In our particular test, we
   used 10ms as frame timeout, which could be a suitable value, but we
   recommend further studies do determine the recommended timeout value.

   We recommend that the reported results of the improved throughput and
   frame loss rate measurements SHOULD include the applied timeout
   value.

4.  Requirement of Statistically Relevant Number of Tests

   Section 4 of [RFC2544] says that: "Furthermore, selection of the
   tests to be run and evaluation of the test data must be done with an
   understanding of generally accepted testing practices regarding
   repeatability, variance and statistical significance of small numbers
   of trials."  It is made a stronger requirement (by using a "MUST") in
   Section 3 of [RFC5180] stating that: "Test execution and results
   analysis MUST be performed while observing generally accepted testing
   practices regarding repeatability, variance, and statistical
   significance of small numbers of trials."  But no practical
   guidelines are provided concerning the minimally necessary number of
   tests.

   [RFC8219] mentions at four different places that the tests must be
   repeated at least 20 times.  These places are the benchmarking
   procedures for:

   o  latency (Section 7.2)

   o  packet delay variation (Section 7.3.1)



Lencse & Shima          Expires November 21, 2020               [Page 4]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   o  inter packet delay variation (Section 7.3.2)

   o  DNS64 performance (Section 9.2).

   We believe that a similar guideline for the minimal number of tests
   would be helpful for the throughput and frame loss rate benchmarking
   procedures.  We consider 20 as an affordable number of minimum
   repetitions of the frame loss rate measurements.  However, as for
   throughput measurements, we contend that the binary search may
   require rather high number of steps in certain situations (e.g. tens
   of millions of frames per second rate and high resolution) that the
   requirement of at least 20 repetitions of the binary search would
   result in unreasonably high measurement execution times.  Therefore,
   we recommend to use an algorithm that checks the statistical
   properties of the results of the tests and it may stop before 20
   repetitions, if the results are consistent, but it may require more
   than 20 repetitions, if the results are scattered.  (The algorithm is
   yet to be developed.)

5.  An Optional Non-zero Frame Loss Acceptance Criterion for the
    Throughput Measurement Procedure

   When we defined the measurement procedure for DNS64 performance in
   Section 9.2 of [RFC8219], we followed both spirit and wording of the
   [RFC2544] throughput measurement procedure including the requirement
   for absolutely zero packet loss.  We have elaborated our underlying
   considerations in our research paper [LEN2017] as follows:

   1.  Our goal is a well-defined performance metric, which can be
       measured simply and efficiently.  Allowing any packet loss would
       result in a need for scanning/trying a large range of rates to
       discover the highest rate of successfully processed DNS queries.

   2.  Even if users may tolerate a low loss rate (please note the DNS
       uses UDP with no guarantee for delivery), it cannot be
       arbitrarily high, thus, we could not avoid defining a limit.
       However, any other limits than zero percent would be hardly
       defensible.

   3.  Other benchmarking procedures use the same criteria of zero
       packet loss and this is the standard in IETF Benchmarking
       Methodology Working Group.

   On the one hand, we still consider our arguments valid, however, on
   the other hand, we are aware of different arguments for the
   justification of an optional non-zero frame loss acceptance
   criterion, too:




Lencse & Shima          Expires November 21, 2020               [Page 5]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   o  Frame loss is present in our networks from the very beginning and
      our applications are well prepared to handle frame loss.  They can
      definitely tolerate some low frame loss rates like 0.01% (1 frame
      from 10,000 frames).

   o  It is a wide-spread practice among benchmarking professionals to
      allow a certain low rate of frame loss for a long time [TOL2001]
      and commercially available network performance testers allow to
      specify a parameter usually called as "Loss Tolerance" to express
      a zero or non-zero acceptance criterion for throughput
      measurements.

   o  Today network packet forwarding and manipulation is often
      implemented in software.  They do not work the same as the
      hardware-based forwarding devices, and may be affected by other
      processes running in the same host hardware.  So it is not
      feasible to require 0% of frame loss in such forwarding devices.

   o  Forwarding devices (especially but not necessarily only the
      software-based ones) may today also have larger buffers and thus
      they may produce potentially higher latencies.  As we have shown
      in Section 3, late packets are not really useful for the
      applications, and thus they are to be considered as lost ones.
      For being strict with the latency during throughput measurements
      (e.g. 10ms timeout), we should make up with the loss tolerance to
      provide meaningful benchmarking results.

   o  Likely due to the high frame loss rates can be experienced in WiFi
      networks, the latest development direction of TCP congestion
      control algorithms considers loss no more a sign of congestion
      (e.g.  TCP BBR).

   So we felt the necessity of having options to allow frame loss.
   Therefore, we recommend that throughput measurement with some low
   tolerated frame loss rates like 0.001% or 0.01% be a recognized
   optional test for network interconnect devices.  To avoid the
   possibility of gaming, our recommendation is that the results of such
   tests MUST clearly state the applied loss tolerance rate.

6.  Acknowledgements

   The authors would like to thank ... (TBD)

7.  IANA Considerations

   This document does not make any request to IANA.





Lencse & Shima          Expires November 21, 2020               [Page 6]


Internet-Draft              RFC 2544 Upgrade                    May 2020


8.  Security Considerations

   We have no further security considerations beyond that of [RFC8219].
   Perhaps they should be cited here so that they be applied not only
   for the benchmarking of IPv6 transition technologies, but also for
   the benchmarking of all network interconnect devices.

9.  References

9.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544,
              DOI 10.17487/RFC2544, March 1999,
              <https://www.rfc-editor.org/info/rfc2544>.

   [RFC5180]  Popoviciu, C., Hamza, A., Van de Velde, G., and D.
              Dugatkin, "IPv6 Benchmarking Methodology for Network
              Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May
              2008, <https://www.rfc-editor.org/info/rfc5180>.

   [RFC7915]  Bao, C., Li, X., Baker, F., Anderson, T., and F. Gont,
              "IP/ICMP Translation Algorithm", RFC 7915,
              DOI 10.17487/RFC7915, June 2016,
              <https://www.rfc-editor.org/info/rfc7915>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8219]  Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking
              Methodology for IPv6 Transition Technologies", RFC 8219,
              DOI 10.17487/RFC8219, August 2017,
              <https://www.rfc-editor.org/info/rfc8219>.

9.2.  Informative References

   [LEN2017]  Lencse, G., Georgescu, M., and Y. Kadobayashi,
              "Benchmarking Methodology for DNS64 Servers",  Computer
              Communications, vol. 109, no. 1, pp. 162-175,  DOI:
              10.1016/j.comcom.2017.06.004, Sep 2017,
              <http://www.hit.bme.hu/~lencse/publications/ECC-
              2017-B-M-DNS64-revised.pdf>.



Lencse & Shima          Expires November 21, 2020               [Page 7]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   [LEN2020A]
              Lencse, G. and K. Shima, "Performance analysis of SIIT
              implementations: Testing and improving the
              methodology",  Computer Communications, vol. 156, no. 1,
              pp. 54-67,  DOI: 10.1016/j.comcom.2020.03.034, Apr 2020,
              <http://www.hit.bme.hu/~lencse/publications/ECC-2020-SIIT-
              Performance-published.pdf>.

   [LEN2020B]
              Lencse, G., "Design and Implementation of a Software
              Tester for Benchmarking Stateless NAT64 Gateways",  under
              second review in IEICE Transactions on Communications,
              <http://www.hit.bme.hu/~lencse/publications/IEICE-2020-
              siitperf-revised.pdf>.

   [LEN2020C]
              Lencse, G., Shima, K., and A. Kovacs, "Gaming with the
              Throughput and the Latency Benchmarking Measurement
              Procedures of RFC 2544",  under review in International
              Journal of Advances in Telecommunications,
              Electrotechnics, Signals and Systems,
              <http://www.hit.bme.hu/~lencse/publications/IJATES2-2020-
              Gaming-RFC2544-for-review.pdf>.

   [SIITPERF]
              Lencse, G. and Y. Kadobayashi, "Siitperf: An RFC 8219
              compliant SIIT (stateless NAT64) tester written in C++
              using DPDK",  source code,  available from GitHub, 2019,
              <https://github.com/lencsegabor/siitperf>.

   [TOL2001]  Tolly, K., "The real meaning of zero-loss testing",  IT
              World Canada, 2001,
              <https://www.itworldcanada.com/article/kevin-tolly-the-
              real-meaning-of-zero-loss-testing/33066>.

Appendix A.  Change Log

A.1.  00

   Initial version.

Authors' Addresses









Lencse & Shima          Expires November 21, 2020               [Page 8]


Internet-Draft              RFC 2544 Upgrade                    May 2020


   Gabor Lencse
   Budapest University of Technology and Economics
   Magyar Tudosok korutja 2.
   Budapest  H-1117
   Hungary

   Email: lencse@hit.bme.hu


   Keiichi Shima
   IIJ Innovation Institute
   Iidabashi Grand Bloom, 2-10-2 Fujimi
   Chiyoda-ku, Tokyo  102-0071
   Japan

   Email: keiichi@iijlab.net



































Lencse & Shima          Expires November 21, 2020               [Page 9]


Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/