--- 1/draft-ietf-ippm-metrictest-04.txt 2011-11-30 10:13:46.566754573 +0100 +++ 2/draft-ietf-ippm-metrictest-05.txt 2011-11-30 10:13:46.638755295 +0100 @@ -1,115 +1,124 @@ Internet Engineering Task Force R. Geib, Ed. Internet-Draft Deutsche Telekom -Intended status: Standards Track A. Morton -Expires: April 26, 2012 AT&T Labs +Intended status: BCP A. Morton +Expires: June 1, 2012 AT&T Labs R. Fardid Cariden Technologies A. Steinmitz Deutsche Telekom - October 24, 2011 + November 29, 2011 IPPM standard advancement testing - draft-ietf-ippm-metrictest-04 + draft-ietf-ippm-metrictest-05 Abstract This document specifies tests to determine if multiple independent instantiations of a performance metric RFC have implemented the specifications in the same way. This is the performance metric equivalent of interoperability, required to advance RFCs along the standards track. Results from different implementations of metric RFCs will be collected under the same underlying network conditions - and compared using state of the art statistical methods. The goal is - an evaluation of the metric RFC itself, whether its definitions are - clear and unambiguous to implementors and therefore a candidate for - advancement on the IETF standards track. + and compared using statistical methods. The goal is an evaluation of + the metric RFC itself; whether its definitions are clear and + unambiguous to implementors and therefore a candidate for advancement + on the IETF standards track. This document is an Internet Best + Current Practice. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on April 26, 2012. + This Internet-Draft will expire on June 1, 2012. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 - 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 7 - 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 7 - 3. Verification of conformance to a metric specification . . . . 8 + 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 + 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 3. Verification of conformance to a metric specification . . . . 7 3.1. Tests of an individual implementation against a metric - specification . . . . . . . . . . . . . . . . . . . . . . 9 + specification . . . . . . . . . . . . . . . . . . . . . . 8 3.2. Test setup resulting in identical live network testing - conditions . . . . . . . . . . . . . . . . . . . . . . . . 11 + conditions . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3. Tests of two or more different implementations against - a metric specification . . . . . . . . . . . . . . . . . . 16 - 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 17 - 3.5. Recommended Metric Verification Measurement Process . . . 18 + a metric specification . . . . . . . . . . . . . . . . . . 15 + 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 16 + 3.5. Recommended Metric Verification Measurement Process . . . 17 3.6. Proposal to determine an "equivalence" threshold for - each metric evaluated . . . . . . . . . . . . . . . . . . 21 - 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 - 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 22 - 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 - 7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 - 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 - 8.1. Normative References . . . . . . . . . . . . . . . . . . . 23 - 8.2. Informative References . . . . . . . . . . . . . . . . . . 24 - Appendix A. An example on a One-way Delay metric validation . . . 25 - A.1. Compliance to Metric specification requirements . . . . . 25 - A.2. Examples related to statistical tests for One-way Delay . 27 + each metric evaluated . . . . . . . . . . . . . . . . . . 20 + 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21 + 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 + 7. Security Considerations . . . . . . . . . . . . . . . . . . . 21 + 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 + 8.1. Normative References . . . . . . . . . . . . . . . . . . . 21 + 8.2. Informative References . . . . . . . . . . . . . . . . . . 22 + Appendix A. An example on a One-way Delay metric validation . . . 23 + A.1. Compliance to Metric specification requirements . . . . . 23 + A.2. Examples related to statistical tests for One-way Delay . 25 Appendix B. Anderson-Darling K-sample Reference and 2 sample - C++ code . . . . . . . . . . . . . . . . . . . . . . 29 - Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 37 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 + C++ code . . . . . . . . . . . . . . . . . . . . . . 27 + Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 36 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 1. Introduction - The Internet Standards Process RFC2026 [RFC2026] requires that for a - IETF specification to advance beyond the Proposed Standard level, at - least two genetically unrelated implementations must be shown to - interoperate correctly with all features and options. This - requirement can be met by supplying: - - o evidence that (at least a sub-set of) the specification has been - implemented by multiple parties, thus indicating adoption by the - IETF community and the extent of feature coverage. - - o evidence that each feature of the specification is sufficiently - well-described to support interoperability, as demonstrated - through testing and/or user experience with deployment. + The Internet Standards Process as updated by RFC6410 [RFC6410] + specifies that widespread deployment and use is sufficient to show + interoperability as a condition for advancement to Internet Standard. + The previous requirement of interoperability tests prior to advancing + an RFC to the Standard maturity level specified in RFC 2026 [RFC2026] + and RFC 5657 [RFC5657] has been removed. While the modified + requirement is applicable to protocols, wide deployment of different + measurement systems does not prove that the implementations measure + metrics in a standard way. Section 5.3 of RFC 5657 [RFC5657] + explicitly mentions the special case of Standards that are not "on- + the-wire" protocols. While this special case is not explicitly + mentioned by RFC6410 [RFC6410], the four criteria in Section 2.2 of + RFC6410 [RFC6410] are augmented by this document for RFCs that + specify performance metrics. This document takes the position that + flexible metric definitions can be proven to be clear and unambiguous + through tests that compare the results from independent + implementations. It describes tests which infer whether metric + specifications are sufficient using a definition of metric + "interoperability": measuring equivalent results (in a statistical + sense) under the same network conditions. The document expands on + this problem and its solution below. In the case of a protocol specification, the notion of "interoperability" is reasonably intuitive - the implementations must successfully "talk to each other", while exercising all features and options. To achieve interoperability, two implementors need to interpret the protocol specifications in equivalent ways. In the case of IP Performance Metrics (IPPM), this definition of interoperability is only useful for test and control protocols like the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the Two-Way Active Measurement Protocol, TWAMP [RFC5357]. @@ -132,21 +141,21 @@ Since many implementations of IP metrics are embedded in measurement systems that do not interact with one another (they were built before OWAMP and TWAMP), the interoperability evaluation called for in the IETF standards process cannot be determined by observing that independent implementations interact properly for various protocol exchanges. Instead, verifying that different implementations give statistically equivalent results under controlled measurement conditions takes the place of interoperability observations. Even when evaluating OWAMP and TWAMP RFCs for standards track advancement, the methods described here are useful to evaluate the measurement - results because their validity would not be ascertained in typical + results because their validity would not be ascertained in protocol interoperability testing. The standards advancement process aims at producing confidence that the metric definitions and supporting material are clearly worded and unambiguous, or reveals ways in which the metric definitions can be revised to achieve clarity. The process also permits identification of options that were not implemented, so that they can be removed from the advancing specification. Thus, the product of this process is information about the metric specification RFC itself: determination of the specifications or definitions that are clear and @@ -168,152 +177,63 @@ Conclusions on equivalence are reached by two measures. First, implementations are compared against individual metric specifications to make sure that differences in implementation are minimised or at least known. Second, a test setup is proposed ensuring identical networking conditions so that unknowns are minimized and comparisons are simplified. The resulting separate data sets may be seen as samples - taken from the same underlying distribution. Using state of the art - statistical methods, the equivalence of the results is verified. To - illustrate application of the process and methods defined here, - evaluation of the One-way Delay Metric [RFC2679] is provided in an - Appendix. While test setups will vary with the metrics to be - validated, the general methodology of determining equivalent results - will not. Documents defining test setups to evaluate other metrics - should be developed once the process proposed here has been agreed - and approved. + taken from the same underlying distribution. Using statistical + methods, the equivalence of the results is verified. To illustrate + application of the process and methods defined here, evaluation of + the One-way Delay Metric [RFC2679] is provided in an Appendix. While + test setups will vary with the metrics to be validated, the general + methodology of determining equivalent results will not. Documents + defining test setups to evaluate other metrics should be developed + once the process proposed here has been agreed and approved. The metric RFC advancement process begins with a request for protocol action accompanied by a memo that documents the supporting tests and results. The procedures of [RFC2026] are expanded in[RFC5657], including sample implementation and interoperability reports. - Section 3 of [morton-advance-metrics-01] can serve as a template for - a metric RFC report which accompanies the protocol action request to - the Area Director, including description of the test set-up, - procedures, results for each implementation and conclusions. - - Changes from WG-03 to WG-04: - - o Revisions to Appendix B code and add reference to "R" in the - Appendix and the text of section 3.6. - - Changes from WG-02 to WG-03: - - o Changes stemming from experiments that implemented this plan, in - general. - - o Adoption of the VLAN loopback figure in the main body of the memo - (section 3.2). - - Changes from WG-01 to WG-02: - - o Clarification of the number of test streams recommended in section - 3.2. - - o Clarifications on testing details in sections 3.3 and 3.4. - - o Spelling corrections throughout. - - Changes from WG -00 to WG -01 draft - - o Discussion on merits and requirements of a distributed lab test - using only local load generators. - - o Proposal of metrics suitable for tests using the proposed - measurement configuration. - - o Hint on delay caused by software based L2TPv3 implementation. - - o Added an appendix with a test configuration allowing remote tests - comparing different implementations across the network. - - o Proposal for maximum error of "equivalence", based on performance - comparison of identical implementations. This may be useful for - both ADK and non-ADK comparisons. - - Changes from prior ID -02 to WG -00 draft - - o Incorporation of aspects of reporting to support the protocol - action request in the Introduction and section 3.5 - - o Overhaul of section 3.2 regarding tunneling: Added generic - tunneling requirements and L2TPv3 as an example tunneling - mechanism fulfilling the tunneling requirements. Removed and - adapted some of the prior references to other tunneling protocols - - o Softened a requirement within section 3.4 (MUST to SHOULD on - precision) and removed some comments of the authors. - - o Updated contact information of one author and added a new author. - - o Added example C++ code of an Anderson-Darling two sample test - implementation. - - Changes from ID -01 to ID -02 version - - o Major editorial review, rewording and clarifications on all - contents. - - o Additional text on parallel testing using VLANs and GRE or - Pseudowire tunnels. - - o Additional examples and a glossary. - - Changes from ID -00 to ID -01 version - - o Addition of a comparison of individual metric implementations - against the metric specification (trying to pick up problems and - solutions for metric advancement [morton-advance-metrics]). - - o More emphasis on the requirement to carefully design and document - the measurement setup of the metric comparison. - - o Proposal of testing conditions under identical WAN network - conditions using IP in IP tunneling or Pseudo Wires and parallel - measurement streams. - - o Proposing the requirement to document the smallest resolution at - which an ADK test was passed by 95%. As no minimum resolution is - specified, IPPM metric compliance is not linked to a particular - performance of an implementation. - - o Reference to RFC 2330 and RFC 2679 for the 95% confidence interval - as preferred criterion to decide on statistical equivalence - - o Reducing the proposed statistical test to ADK with 95% confidence. + [morton-testplan-rfc2679] can serve as a template for a metric RFC + report which accompanies the protocol action request to the Area + Director, including description of the test set-up, procedures, + results for each implementation and conclusions. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 2. Basic idea The implementation of a standard compliant metric is expected to meet the requirements of the related metric specification. So before comparing two metric implementations, each metric implementation is individually compared against the metric specification. Most metric specifications leave freedom to implementors on non- fundamental aspects of an individual metric (or options). Comparing different measurement results using a statistical test with the assumption of identical test path and testing conditions requires knowledge of all differences in the overall test setup. Metric specification options chosen by implementors have to be documented. - It is REQUIRED to use identical implementation options wherever - possible for any test proposed here. Calibrations proposed by metric - standards should be performed to further identify (and possibly - reduce) potential sources of errors in the test setup. + It is RECOMMENDED to use identical metric options for any test + proposed here (an exception would be if a variable parameter of the + metric definition is not configurable in one or more + implementations). Calibrations specified by metric standards SHOULD + be performed to further identify (and possibly reduce) potential + sources of error in the test setup. The Framework for IP Performance Metrics [RFC2330] expects that a "methodology for a metric should have the property that it is repeatable: if the methodology is used multiple times under identical conditions, it should result in consistent measurements." This means an implementation is expected to repeatedly measure a metric with consistent results (repeatability with the same result). Small deviations in the test setup are expected to lead to small deviations in results only. To characterise statistical equivalence in the case of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% @@ -335,27 +255,28 @@ measurement setup on the result, network conditions and paths MUST be identical for the compared implementations to the largest possible degree. This includes both the stability and non- ambiguity of routes taken by the measurement packets. See RFC 2330 for a discussion on self-consistency. o To minimize the influence of implementation options on the result, metric implementations SHOULD use identical options and parameters for the metric under evaluation. - o The error induced by the sample size must be small enough to - minimize its influence on the test result. This may have to be - respected, especially if two implementations measure with - different average probing rates. + o The sample size must be large enough to minimize its influence on + the consistency of the test results. This consideration may be + especially important if two implementations measure with different + average packet transmission rates. - o The implementation with the lowest probing frequency determines - the smallest temporal interval for which samples can be compared. + o The implementation with the lowest average packet transmission + rate determines the smallest temporal interval for which samples + can be compared. o Repeat comparisons with several independent metric samples to avoid random indications of compatibility (or the lack of it). The metric specifications themselves are the primary focus of evaluation, rather than the implementations of metrics. The documentation produced by the advancement process should identify which metric definitions and supporting material were found to be clearly worded and unambiguous, OR, it should identify ways in which the metric specification text should be revised to achieve clarity @@ -396,73 +317,75 @@ impairment generators. o Use remotely separated test labs to compare the implementations and measure across the Internet. o Use remotely separated test labs to compare the implementations and measure across the Internet and include a single impairment generator to impact all measurement flows in non discriminatory way. - The first two approaches work, but cause higher expenses than the - other ones (due to travel and/or shipping+installation). For the + The first two approaches work, but involve higher expenses than the + others (due to travel and/or shipping plus installation). For the third option, ensuring two identically configured impairment generators requires well defined test cases and possibly identical - hard- and software. + hardware and software. As documented in a test report [morton-testplan-rfc2679], the last option was required to prove compatibility of two delay metric implementations. An impairment generator is probably required when - testing compatibility of most other metrics and it therefore - RECOMMENDED to include an impairment generator in metric test set - ups. + testing compatibility of most other metrics and it is therefore + RECOMMENDED to include an impairment generator in metric test setups. 3.1. Tests of an individual implementation against a metric specification - A metric implementation MUST support the requirements classified as - "MUST" and "REQUIRED" of the related metric specification to be - compliant to the latter. + A metric implementation is compliant with a metric specification if + it supports the requirements classified as "MUST" and "REQUIRED" of + the related metric specification. An implementation that implements + all requirements is fully compliant with the specification, and the + degree of compliance SHOULD be noted in the conclusions of the + report. Further, supported options of a metric implementation SHOULD be - documented in sufficient detail. The documentation of chosen options - is RECOMMENDED to minimise (and recognise) differences in the test - setup if two metric implementations are compared. Further, this - documentation is used to validate and improve the underlying metric - specification option, to remove options which saw no implementation - or which are badly specified from the metric specification to be - promoted to a standard. This documentation SHOULD be made for all - implementation-relevant specifications of a metric picked for a - comparison that are not explicitly marked as "MUST" or "REQUIRED" in - the RFC text. This applies for the following sections of all metric - specifications: + documented in sufficient detail to evaluate whether the specification + was correctly interpreted. The documentation of chosen options + should minimise (and recognise) differences in the test setup if two + metric implementations are compared. Further, this documentation is + used to validate or clarify the wording of the metric specification + option, to remove options which saw no implementation or which are + badly specified from the metric specification. This documentation + SHOULD be included for all implementation-relevant specifications of + a metric picked for a comparison, even those that are not explicitly + marked as "MUST" or "REQUIRED" in the RFC text. This applies for the + following sections of all metric specifications: o Singleton Definition of the Metric. o Sample Definition of the Metric. o Statistics Definition of the Metric. As statistics are compared by the test specified here, this documentation is required even in the case, that the metric specification does not contain a Statistics Definition. o Timing and Synchronisation related specification (if relevant for the Metric). o Any other technical part present or missing in the metric specification, which is relevant for the implementation of the Metric. RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric - implementations. A single IPPM conformant implementation MUST under - otherwise identical network conditions produce precise results for - repeated measurements of the same metric. + implementations. A single IPPM conforming implementation should + under otherwise identical network conditions produce precise results + for repeated measurements of the same metric. RFC 2330 prefers the "empirical distribution function" EDF to describe collections of measurements. RFC 2330 determines, that "unless otherwise stated, IPPM goodness-of-fit tests are done using 5% significance." The goodness of fit test determines by which precision two or more samples of a metric implementation belong to the same underlying distribution (of measured network performance events). The goodness of fit test suggested for the metric test is the Anderson-Darling K sample test (ADK sample test, K stands for the number of samples to be compared) [ADK]. Please note that RFC 2330 @@ -471,21 +394,21 @@ The results of a repeated test with a single implementation MUST pass an ADK sample test with confidence level of 95%. The conditions for which the ADK test has been passed with the specified confidence level MUST be documented. To formulate this differently: The requirement is to document the set of parameters with the smallest deviation, at which the results of the tested metric implementation pass an ADK test with a confidence level of 95%. The minimum resolution available in the reported results from each implementation MUST be taken into account in the ADK test. - The test conditions which MUST be documented for a passed metric test + The test conditions to be documented for a passed metric test include: o The metric resolution at which a test was passed (e.g. the resolution of timestamps) o The parameters modified by an impairment generator. o The impairment generator parameter settings. 3.2. Test setup resulting in identical live network testing conditions @@ -498,33 +421,33 @@ mechanisms like those that achieve load balancing (see [RFC4928]). This section proposes two measures to deal with both issues. Tunneling mechanisms can be used to avoid parallel processing of different flows in the network. Measuring by separate parallel probe flows results in repeated collection of data. If both measures are combined, WAN network conditions are identical for a number of independent measurement flows, no matter what the network conditions are in detail. - Any measurement setup MUST be made to avoid the probing traffic + Any measurement setup must be made to avoid the probing traffic itself to impede the metric measurement. The created measurement - load MUST NOT result in congestion at the access link connecting the + load must not result in congestion at the access link connecting the measurement implementation to the WAN. The created measurement load - MUST NOT overload the measurement implementation itself, e.g., by + must not overload the measurement implementation itself, e.g., by causing a high CPU load or by creating imprecisions due to internal transmit (receive respectively) probe packet collisions. Tunneling multiple flows reaching a network element on a single physical port may allow to transmit all packets of the tunnel via the same path. Applying tunnels to avoid undesired influence of standard routing for measurement purposes is a concept known from literature, - see e.g. GRE encapsulated multicast probing [GU+Duffield]. An + see e.g. GRE encapsulated multicast probing [GU-Duffield]. An existing IP in IP tunnel protocol can be applied to avoid Equal-Cost Multi-Path (ECMP) routing of different measurement streams if it meets the following criteria: o Inner IP packets from different measurement implementations are mapped into a single tunnel with single outer IP origin and destination address as well as origin and destination port numbers which are identical for all packets. o An easily accessible commodity tunneling protocol allows to carry @@ -612,37 +535,38 @@ to B and in the reverse direction. The remote site VLANs are U-bridged at the local site Ethernet switch. The measurement packets of site 1 travel tunnel A->B first, are U-bridged at site 2 and travel tunnel B->A second. Measurement packets of site 2 travel tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B second. So all measurement packets pass the same tunnel segments, but in different segment order. If tunneling is applied, two tunnels MUST carry all test traffic in between the test site and the remote site. For example, if 802.1Q - Ethernet Virtual LANs (VLAN) are applied and the measurement streams - are carried in different VLANs, the IP tunnel or Pseudo Wires - respectively MUST be set up in physical port mode to avoid set up of + Virtual LANs (VLAN) are applied and the measurement streams are + carried in different VLANs, the IP tunnel or Pseudo Wires + respectively are set up in physical port mode to avoid set up of Pseudo Wires per VLAN (which may see different paths due to ECMP routing), see RFC 4448. The remote router and the Ethernet switch shown in figure 3 has to support 802.1Q in this set up. The IP packet size of the metric implementation SHOULD be chosen small enough to avoid fragmentation due to the added Ethernet and tunnel headers. Otherwise, the impact of tunnel overhead on - fragmentation and interface MTU size MUST be understood and taken + fragmentation and interface MTU size must be understood and taken into account (see [RFC4459]). An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each containing measurement traffic of a single measurement system was successfully applied when testing compatibility of two metric - implementations [morton-testplan-rfc2679]. + implementations [morton-testplan-rfc2679]. Ethernet over L2TPv3 + [RFC4719] was picked for this test. The following headers may have to be accounted for when calculating total packet length, if VLANs and Ethernet over L2TPv3 tunnels are applied: o Ethernet 802.1Q: 22 Byte. o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 Byte for L2TPv3 data messages over UDP. @@ -701,35 +625,35 @@ with 4 samples even if a 2 sample test failed[morton-testplan-rfc2679]. Some additional guidelines to calculate and compare samples to perform a metric test are: o To compare different probes of a common underlying distribution in terms of metrics characterising a communication network requires to respect the temporal nature for which the assumption of common underlying distribution may hold. Any singletons or samples to be - compared MUST be captured within the same time interval. + compared must be captured within the same time interval. o If statistical events like rates are used to characterise measured - metrics of a time-interval, its RECOMMENDED to pick as a minimum 5 - singletons of a relevant metric to ensure a minimum confidence - into the reported value. The error margin of the determined rate - depends on the number singletons (refer to statistical textbooks - on Student's t-test). As an example, any packet loss measurement + metrics of a time-interval, a minimum 5 singletons of a relevant + metric should be picked to ensure a minimum confidence into the + reported value. The error margin of the determined rate depends + on the number singletons (refer to statistical textbooks on + Student's t-test). As an example, any packet loss measurement interval to be compared with the results of another implementation contains at least five lost packets to have some confidence that the observed loss rate wasn't caused by a small number of random packet drops. o The minimum number of singletons or samples to be compared by an - Anderson-Darling test SHOULD be 100 per tested metric + Anderson-Darling test should be 100 per tested metric implementation. Note that the Anderson-Darling test detects small differences in distributions fairly well and will fail for high number of compared results (RFC2330 mentions an example with 8192 measurements where an Anderson-Darling test always failed). o Generally, the Anderson-Darling test is sensitive to differences in the accuracy or bias associated with varying implementations or test conditions. These dissimilarities may result in differing averages of samples to be compared. An example may be different packet sizes, resulting in a constant delay difference between @@ -747,62 +671,64 @@ precisely, for every positive epsilon, there exists a positive delta, such that if two sets of conditions are within delta of each other, then the resulting measurements will be within epsilon of each other." A small variation in conditions in the context of the metric test proposed here can be seen as different implementations measuring the same metric along the same path. IPPM metric specifications however allow for implementor options to the largest possible degree. It cannot be expected that two implementors allow 100% identical options in their implementations. - Testers SHOULD to the highest degree possible pick the same - configurations for their systems when comparing their implementations - by a metric test. + Testers SHOULD pick the same metric measurement configurations for + their systems when comparing their implementations by a metric test. In some cases, a goodness of fit test may not be possible or show disappointing results. To clarify the difficulties arising from - different implementation options, the individual options picked for - every compared implementation SHOULD be documented in sufficient - detail. Based on this documentation, the underlying metric - specification should be improved before it is promoted to a standard. + different metric implementation options, the individual options + picked for every compared metric implementation should be documented + as specified in section 3.5. If the cause of the failure is a lack + of specification clarity or multiple legitimate interpretations of + the definition text, the text should be modified and the resulting + memo proposed for consensus and (possible) advancement to Internet + Standard. The same statistical test as applicable to quantify precision of a - single metric implementation MUST be used to compare metric result + single metric implementation must be used to compare metric result equivalence for different implementations. To document compatibility, the smallest measurement resolution at which the - compared implementations passed the ADK sample test MUST be + compared implementations passed the ADK sample test must be documented. For different implementations of the same metric, "variations in conditions" are reasonably expected. The ADK test comparing samples - of the different implementations MAY result in a lower precision than + of the different implementations may result in a lower precision than the test for precision in the same-implementation comparison. 3.4. Clock synchronisation Clock synchronization effects require special attention. Accuracy of one-way active delay measurements for any metrics implementation depends on clock synchronization between the source and destination of tests. Ideally, one-way active delay measurement (RFC 2679, [RFC2679]) test endpoints either have direct access to independent GPS or CDMA-based time sources or indirect access to nearby NTP primary (stratum 1) time sources, equipped with GPS receivers. Access to these time sources may not be available at all test locations associated with different Internet paths, for a variety of reasons out of scope of this document. When secondary (stratum 2 and above) time sources are used with NTP running across the same network, whose metrics are subject to comparative implementation tests, network impairments can affect clock synchronization, distort sample one-way values and their - interval statistics. It is RECOMMENDED to discard sample one-way - delay values for any implementation, when one of the following + interval statistics. It is recommended to discard sample one-way + delay values for any implementation when one of the following reliability conditions is met: o Delay is measured and is finite in one direction, but not the other. o Absolute value of the difference between the sum of one-way measurements in both directions and round-trip measurement is greater than X% of the latter value. Examination of the second condition requires RTT measurement for @@ -813,52 +739,53 @@ unreliable one-way delay samples and misidentification of reliable samples under a wide range of Internet path RTTs probably requires further study. An IPPM compliant metric implementation of an RFC that requires synchronized clocks is expected to provide precise measurement results. IF an implementation publishes a specification of its precision, such as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then - the specification SHOULD be met over a useful measurement duration. + the specification should be met over a useful measurement duration. For example, if the metric is measured along an Internet path which - is stable and not congested, then the precision specification SHOULD + is stable and not congested, then the precision specification should be met over durations of an hour or more. 3.5. Recommended Metric Verification Measurement Process In order to meet their obligations under the IETF Standards Process the IESG must be convinced that each metric specification advanced to - Draft Standard or Internet Standard status is clearly written, that - there are a sufficient number of verified equivalent implementations, - and that options that have been implemented are documented. + Internet Standard status is clearly written, that there are a + sufficient number of verified equivalent implementations, and that + options that have been implemented are documented. In the context of this document, metrics are designed to measure some characteristic of a data network. An aim of any metric definition should be that it should be specified in a way that can reliably measure the specific characteristic in a repeatable way across multiple independent implementations. Each metric, statistic or option of those to be validated MUST be - compared against a reference measurement or another implementation by - as specified by this document. + compared against a reference measurement or another implementation as + specified in this document. Finally, the metric definitions, embodied in the text of the RFCs, are the objects that require evaluation and possible revision in - order to advance to the next step on the standards track. + order to advance to Internet Standard. IF two (or more) implementations do not measure an equivalent metric as specified by this document, AND sources of measurement error do not adequately explain the lack of agreement, + THEN the details of each implementation should be audited along with the exact definition text, to determine if there is a lack of clarity that has caused the implementations to vary in a way that affects the correspondence of the results. IF there was a lack of clarity or multiple legitimate interpretations of the definition text, THEN the text should be modified and the resulting memo proposed for consensus and (possible) advancement along the standards track. @@ -857,64 +784,63 @@ that has caused the implementations to vary in a way that affects the correspondence of the results. IF there was a lack of clarity or multiple legitimate interpretations of the definition text, THEN the text should be modified and the resulting memo proposed for consensus and (possible) advancement along the standards track. Finally, all the findings MUST be documented in a report that can - support advancement on the standards track, similar to those - described in [RFC5657]. The list of measurement devices used in - testing satisfies the implementation requirement, while the test - results provide information on the quality of each specification in - the metric RFC (the surrogate for feature interoperability). + support advancement to Internet Standard, as described here (similar + to those described in [RFC5657]). The list of measurement devices + used in testing satisfies the implementation requirement, while the + test results provide information on the quality of each specification + in the metric RFC (the surrogate for feature interoperability). The complete process of advancing a metric specification to a standard as defined by this document is illustrated in Figure 4. ,---. / \ ( Start ) \ / Implementations `-+-' +-------+ | /| 1 `. +---+----+ / +-------+ `.-----------+ ,-------. | RFC | / |Check for | ,' was RFC `. YES | | / |Equivalence.... clause x ------+ | |/ +-------+ |under | `. clear? ,' | | Metric \.....| 2 ....relevant | `---+---' +----+-----+ | Metric |\ +-------+ |identical | No | |Report | | Metric | \ |network | +--+----+ |results + | | ... | \ |conditions | |Modify | |Advance | | | \ +-------+ | | |Spec +--+RFC | - +--------+ \| n |.'+-----------+ +-------+ |request(?)| + +--------+ \| n |.'+-----------+ +-------+ |request | +-------+ +----------+ Illustration of the metric standardisation process Figure 4 Any recommendation for the advancement of a metric specification MUST - be accompanied by an implementation report, as is the case with all - requests for the advancement of IETF specifications. The - implementation report needs to include the tests performed, the - applied test setup, the specific metrics in the RFC and reports of - the tests performed with two or more implementations. The test plan - needs to specify the precision reached for each measured metric and - thus define the meaning of "statistically equivalent" for the - specific metrics being tested. + be accompanied by an implementation report. The implementation + report needs to include the tests performed, the applied test setup, + the specific metrics in the RFC and reports of the tests performed + with two or more implementations. The test plan needs to specify the + precision reached for each measured metric and thus define the + meaning of "statistically equivalent" for the specific metrics being + tested. Ideally, the test plan would co-evolve with the development of the - metric, since that's when people have the most context in their - thinking regarding the different subtleties that can arise. + metric, since that's when participants have the clearest context in + their minds regarding the different subtleties that can arise. In particular, the implementation report MUST as a minimum document: o The metric compared and the RFC specifying it. This includes statements as required by the section "Tests of an individual implementation against a metric specification" of this document. o The measurement configuration and setup. o A complete specification of the measurement stream (mean rate, @@ -941,21 +867,21 @@ and network conditions allowing reproduction of these laboratory conditions for readers of the implementation report. o The documentation helping to improve metric specifications defined by this section. All of the tests for each set SHOULD be run in a test setup as specified in the section "Test setup resulting in identical live network testing conditions." - If a different test set up is chosen, it is RECOMMENDED to avoid + If a different test setup is chosen, it is recommended to avoid effects falsifying results of validation measurements caused by real data networks (like parallelism in devices and networks). Data networks may forward packets differently in the case of: o Different packet sizes chosen for different metric implementations. A proposed countermeasure is selecting the same packet size when validating results of two samples or a sample against an original distribution. o Selection of differing IP addresses and ports used by different @@ -1043,129 +968,113 @@ This memo does not raise any specific security issues. 8. References 8.1. Normative References [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996. - [RFC2026] Bradner, S., "The Internet Standards Process -- Revision - 3", BCP 9, RFC 2026, October 1996. - [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", RFC 2661, August 1999. [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way Delay Metric for IPPM", RFC 2679, September 1999. - [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way - Packet Loss Metric for IPPM", RFC 2680, September 1999. - - [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip - Delay Metric for IPPM", RFC 2681, September 1999. - [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, March 2000. [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, "Encapsulation Methods for Transport of Ethernet over MPLS Networks", RFC 4448, April 2006. - [RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- - Network Tunneling", RFC 4459, April 2006. - [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, September 2006. [RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport of Ethernet Frames over Layer 2 Tunneling Protocol Version 3 (L2TPv3)", RFC 4719, November 2006. [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal Cost Multipath Treatment in MPLS Networks", BCP 128, RFC 4928, June 2007. [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard", BCP 9, RFC 5657, September 2009. + [RFC6410] Housley, R., Crocker, D., and E. Burger, "Reducing the + Standards Track to Two Maturity Levels", BCP 9, RFC 6410, + October 2011. + 8.2. Informative References [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling Tests of fit, for continuous and discrete cases", University of Washington, Technical Report No. 81, May 1986. - [GU+Duffield] + [GU-Duffield] Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE Encapsulated Multicast Probing: A Scalable Technique for Measuring One-Way Loss", SIGMETRICS'07 San Diego, California, USA, June 2007. + [RFC2026] Bradner, S., "The Internet Standards Process -- Revision + 3", BCP 9, RFC 2026, October 1996. + + [RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- + Network Tunneling", RFC 4459, April 2006. + [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, October 2008. [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and Combinations of Such Tests. R package version 1.0.", , 2008. [Rtool] R Development Core Team, "R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org/", , 2011. - [Rule of thumb] - Hardy, M., "Confidence interval", March 2010. - [bradner-metrictest] Bradner, S., Mankin, A., and V. Paxson, "Advancement of metrics specifications on the IETF Standards Track", draft -bradner-metricstest-03, (work in progress), July 2007. - [morton-advance-metrics] - Morton, A., "Problems and Possible Solutions for Advancing - Metrics on the Standards Track", draft -morton-ippm- - advance-metrics-00, (work in progress), July 2009. - - [morton-advance-metrics-01] - Morton, A., "Lab Test Results for Advancing Metrics on the - Standards Track", draft -morton-ippm-advance-metrics-01, - (work in progress), June 2010. - [morton-testplan-rfc2679] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test Plan and Results for Advancing RFC 2679 on the Standards Track", draft -morton-ippm-testplan-rfc2679-01, (work in progress), June 2011. Appendix A. An example on a One-way Delay metric validation The text of this appendix is not binding. It is an example how parts of a One-way Delay metric test could look like. - http://xml.resource.org/public/rfc/bibxml/ A.1. Compliance to Metric specification requirements One-way Delay, Loss threshold, RFC 2679 This test determines if implementations use the same configured maximum waiting time delay from one measurement to another under different delay conditions, and correctly declare packets arriving in excess of the waiting time threshold as lost. See Section 3.5 of RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. @@ -1329,20 +1237,33 @@ but this is as it should be. The C++ code below will perform a 2-sample AD comparison when compiled and presented with two column vectors in a file (using white space as separation). This version contains modifications to use the vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The status of the comparison can be checked on the command line with "$ echo $?" or the last line can be replaced with a printf statement for adk_result instead. + /* + + Copyright (c) 2011 IETF Trust and the persons identified + as authors of the code. All rights reserved. + + Redistribution and use in source and binary forms, with + or without modification, is permitted pursuant to, and subject + to the license terms contained in, the Simplified BSD License + set forth in Section 4.c of the IETF Trust's Legal Provisions + Relating to IETF Documents (http://trustee.ietf.org/license-info). + + */ + /* Routines for computing the Anderson-Darling 2 sample * test statistic. * * Implemented based on the description in * "Anderson-Darling K Sample Test" Heckert, Alan and * Filliben, James, editors, Dataplot Reference Manual, * Chapter 15 Auxiliary, NIST, 2004. * Official Reference by 2010 * Heckert, N. A. (2001). Dataplot website at the * National Institute of Standards and Technology: