draft-ietf-ippm-metrictest-04.txt   draft-ietf-ippm-metrictest-05.txt 
Internet Engineering Task Force R. Geib, Ed. Internet Engineering Task Force R. Geib, Ed.
Internet-Draft Deutsche Telekom Internet-Draft Deutsche Telekom
Intended status: Standards Track A. Morton Intended status: BCP A. Morton
Expires: April 26, 2012 AT&T Labs Expires: June 1, 2012 AT&T Labs
R. Fardid R. Fardid
Cariden Technologies Cariden Technologies
A. Steinmitz A. Steinmitz
Deutsche Telekom Deutsche Telekom
October 24, 2011 November 29, 2011
IPPM standard advancement testing IPPM standard advancement testing
draft-ietf-ippm-metrictest-04 draft-ietf-ippm-metrictest-05
Abstract Abstract
This document specifies tests to determine if multiple independent This document specifies tests to determine if multiple independent
instantiations of a performance metric RFC have implemented the instantiations of a performance metric RFC have implemented the
specifications in the same way. This is the performance metric specifications in the same way. This is the performance metric
equivalent of interoperability, required to advance RFCs along the equivalent of interoperability, required to advance RFCs along the
standards track. Results from different implementations of metric standards track. Results from different implementations of metric
RFCs will be collected under the same underlying network conditions RFCs will be collected under the same underlying network conditions
and compared using state of the art statistical methods. The goal is and compared using statistical methods. The goal is an evaluation of
an evaluation of the metric RFC itself, whether its definitions are the metric RFC itself; whether its definitions are clear and
clear and unambiguous to implementors and therefore a candidate for unambiguous to implementors and therefore a candidate for advancement
advancement on the IETF standards track. on the IETF standards track. This document is an Internet Best
Current Practice.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 26, 2012. This Internet-Draft will expire on June 1, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 7 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5
2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Verification of conformance to a metric specification . . . . 8 3. Verification of conformance to a metric specification . . . . 7
3.1. Tests of an individual implementation against a metric 3.1. Tests of an individual implementation against a metric
specification . . . . . . . . . . . . . . . . . . . . . . 9 specification . . . . . . . . . . . . . . . . . . . . . . 8
3.2. Test setup resulting in identical live network testing 3.2. Test setup resulting in identical live network testing
conditions . . . . . . . . . . . . . . . . . . . . . . . . 11 conditions . . . . . . . . . . . . . . . . . . . . . . . . 9
3.3. Tests of two or more different implementations against 3.3. Tests of two or more different implementations against
a metric specification . . . . . . . . . . . . . . . . . . 16 a metric specification . . . . . . . . . . . . . . . . . . 15
3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 17 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 16
3.5. Recommended Metric Verification Measurement Process . . . 18 3.5. Recommended Metric Verification Measurement Process . . . 17
3.6. Proposal to determine an "equivalence" threshold for 3.6. Proposal to determine an "equivalence" threshold for
each metric evaluated . . . . . . . . . . . . . . . . . . 21 each metric evaluated . . . . . . . . . . . . . . . . . . 20
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21
5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 22 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 21
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 7. Security Considerations . . . . . . . . . . . . . . . . . . . 21
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21
8.1. Normative References . . . . . . . . . . . . . . . . . . . 23 8.1. Normative References . . . . . . . . . . . . . . . . . . . 21
8.2. Informative References . . . . . . . . . . . . . . . . . . 24 8.2. Informative References . . . . . . . . . . . . . . . . . . 22
Appendix A. An example on a One-way Delay metric validation . . . 25 Appendix A. An example on a One-way Delay metric validation . . . 23
A.1. Compliance to Metric specification requirements . . . . . 25 A.1. Compliance to Metric specification requirements . . . . . 23
A.2. Examples related to statistical tests for One-way Delay . 27 A.2. Examples related to statistical tests for One-way Delay . 25
Appendix B. Anderson-Darling K-sample Reference and 2 sample Appendix B. Anderson-Darling K-sample Reference and 2 sample
C++ code . . . . . . . . . . . . . . . . . . . . . . 29 C++ code . . . . . . . . . . . . . . . . . . . . . . 27
Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 37 Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 36
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36
1. Introduction 1. Introduction
The Internet Standards Process RFC2026 [RFC2026] requires that for a The Internet Standards Process as updated by RFC6410 [RFC6410]
IETF specification to advance beyond the Proposed Standard level, at specifies that widespread deployment and use is sufficient to show
least two genetically unrelated implementations must be shown to interoperability as a condition for advancement to Internet Standard.
interoperate correctly with all features and options. This The previous requirement of interoperability tests prior to advancing
requirement can be met by supplying: an RFC to the Standard maturity level specified in RFC 2026 [RFC2026]
and RFC 5657 [RFC5657] has been removed. While the modified
o evidence that (at least a sub-set of) the specification has been requirement is applicable to protocols, wide deployment of different
implemented by multiple parties, thus indicating adoption by the measurement systems does not prove that the implementations measure
IETF community and the extent of feature coverage. metrics in a standard way. Section 5.3 of RFC 5657 [RFC5657]
explicitly mentions the special case of Standards that are not "on-
o evidence that each feature of the specification is sufficiently the-wire" protocols. While this special case is not explicitly
well-described to support interoperability, as demonstrated mentioned by RFC6410 [RFC6410], the four criteria in Section 2.2 of
through testing and/or user experience with deployment. RFC6410 [RFC6410] are augmented by this document for RFCs that
specify performance metrics. This document takes the position that
flexible metric definitions can be proven to be clear and unambiguous
through tests that compare the results from independent
implementations. It describes tests which infer whether metric
specifications are sufficient using a definition of metric
"interoperability": measuring equivalent results (in a statistical
sense) under the same network conditions. The document expands on
this problem and its solution below.
In the case of a protocol specification, the notion of In the case of a protocol specification, the notion of
"interoperability" is reasonably intuitive - the implementations must "interoperability" is reasonably intuitive - the implementations must
successfully "talk to each other", while exercising all features and successfully "talk to each other", while exercising all features and
options. To achieve interoperability, two implementors need to options. To achieve interoperability, two implementors need to
interpret the protocol specifications in equivalent ways. In the interpret the protocol specifications in equivalent ways. In the
case of IP Performance Metrics (IPPM), this definition of case of IP Performance Metrics (IPPM), this definition of
interoperability is only useful for test and control protocols like interoperability is only useful for test and control protocols like
the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the
Two-Way Active Measurement Protocol, TWAMP [RFC5357]. Two-Way Active Measurement Protocol, TWAMP [RFC5357].
skipping to change at page 4, line 7 skipping to change at page 4, line 15
Since many implementations of IP metrics are embedded in measurement Since many implementations of IP metrics are embedded in measurement
systems that do not interact with one another (they were built before systems that do not interact with one another (they were built before
OWAMP and TWAMP), the interoperability evaluation called for in the OWAMP and TWAMP), the interoperability evaluation called for in the
IETF standards process cannot be determined by observing that IETF standards process cannot be determined by observing that
independent implementations interact properly for various protocol independent implementations interact properly for various protocol
exchanges. Instead, verifying that different implementations give exchanges. Instead, verifying that different implementations give
statistically equivalent results under controlled measurement statistically equivalent results under controlled measurement
conditions takes the place of interoperability observations. Even conditions takes the place of interoperability observations. Even
when evaluating OWAMP and TWAMP RFCs for standards track advancement, when evaluating OWAMP and TWAMP RFCs for standards track advancement,
the methods described here are useful to evaluate the measurement the methods described here are useful to evaluate the measurement
results because their validity would not be ascertained in typical results because their validity would not be ascertained in protocol
interoperability testing. interoperability testing.
The standards advancement process aims at producing confidence that The standards advancement process aims at producing confidence that
the metric definitions and supporting material are clearly worded and the metric definitions and supporting material are clearly worded and
unambiguous, or reveals ways in which the metric definitions can be unambiguous, or reveals ways in which the metric definitions can be
revised to achieve clarity. The process also permits identification revised to achieve clarity. The process also permits identification
of options that were not implemented, so that they can be removed of options that were not implemented, so that they can be removed
from the advancing specification. Thus, the product of this process from the advancing specification. Thus, the product of this process
is information about the metric specification RFC itself: is information about the metric specification RFC itself:
determination of the specifications or definitions that are clear and determination of the specifications or definitions that are clear and
skipping to change at page 4, line 43 skipping to change at page 4, line 51
Conclusions on equivalence are reached by two measures. Conclusions on equivalence are reached by two measures.
First, implementations are compared against individual metric First, implementations are compared against individual metric
specifications to make sure that differences in implementation are specifications to make sure that differences in implementation are
minimised or at least known. minimised or at least known.
Second, a test setup is proposed ensuring identical networking Second, a test setup is proposed ensuring identical networking
conditions so that unknowns are minimized and comparisons are conditions so that unknowns are minimized and comparisons are
simplified. The resulting separate data sets may be seen as samples simplified. The resulting separate data sets may be seen as samples
taken from the same underlying distribution. Using state of the art taken from the same underlying distribution. Using statistical
statistical methods, the equivalence of the results is verified. To methods, the equivalence of the results is verified. To illustrate
illustrate application of the process and methods defined here, application of the process and methods defined here, evaluation of
evaluation of the One-way Delay Metric [RFC2679] is provided in an the One-way Delay Metric [RFC2679] is provided in an Appendix. While
Appendix. While test setups will vary with the metrics to be test setups will vary with the metrics to be validated, the general
validated, the general methodology of determining equivalent results methodology of determining equivalent results will not. Documents
will not. Documents defining test setups to evaluate other metrics defining test setups to evaluate other metrics should be developed
should be developed once the process proposed here has been agreed once the process proposed here has been agreed and approved.
and approved.
The metric RFC advancement process begins with a request for protocol The metric RFC advancement process begins with a request for protocol
action accompanied by a memo that documents the supporting tests and action accompanied by a memo that documents the supporting tests and
results. The procedures of [RFC2026] are expanded in[RFC5657], results. The procedures of [RFC2026] are expanded in[RFC5657],
including sample implementation and interoperability reports. including sample implementation and interoperability reports.
Section 3 of [morton-advance-metrics-01] can serve as a template for [morton-testplan-rfc2679] can serve as a template for a metric RFC
a metric RFC report which accompanies the protocol action request to report which accompanies the protocol action request to the Area
the Area Director, including description of the test set-up, Director, including description of the test set-up, procedures,
procedures, results for each implementation and conclusions. results for each implementation and conclusions.
Changes from WG-03 to WG-04:
o Revisions to Appendix B code and add reference to "R" in the
Appendix and the text of section 3.6.
Changes from WG-02 to WG-03:
o Changes stemming from experiments that implemented this plan, in
general.
o Adoption of the VLAN loopback figure in the main body of the memo
(section 3.2).
Changes from WG-01 to WG-02:
o Clarification of the number of test streams recommended in section
3.2.
o Clarifications on testing details in sections 3.3 and 3.4.
o Spelling corrections throughout.
Changes from WG -00 to WG -01 draft
o Discussion on merits and requirements of a distributed lab test
using only local load generators.
o Proposal of metrics suitable for tests using the proposed
measurement configuration.
o Hint on delay caused by software based L2TPv3 implementation.
o Added an appendix with a test configuration allowing remote tests
comparing different implementations across the network.
o Proposal for maximum error of "equivalence", based on performance
comparison of identical implementations. This may be useful for
both ADK and non-ADK comparisons.
Changes from prior ID -02 to WG -00 draft
o Incorporation of aspects of reporting to support the protocol
action request in the Introduction and section 3.5
o Overhaul of section 3.2 regarding tunneling: Added generic
tunneling requirements and L2TPv3 as an example tunneling
mechanism fulfilling the tunneling requirements. Removed and
adapted some of the prior references to other tunneling protocols
o Softened a requirement within section 3.4 (MUST to SHOULD on
precision) and removed some comments of the authors.
o Updated contact information of one author and added a new author.
o Added example C++ code of an Anderson-Darling two sample test
implementation.
Changes from ID -01 to ID -02 version
o Major editorial review, rewording and clarifications on all
contents.
o Additional text on parallel testing using VLANs and GRE or
Pseudowire tunnels.
o Additional examples and a glossary.
Changes from ID -00 to ID -01 version
o Addition of a comparison of individual metric implementations
against the metric specification (trying to pick up problems and
solutions for metric advancement [morton-advance-metrics]).
o More emphasis on the requirement to carefully design and document
the measurement setup of the metric comparison.
o Proposal of testing conditions under identical WAN network
conditions using IP in IP tunneling or Pseudo Wires and parallel
measurement streams.
o Proposing the requirement to document the smallest resolution at
which an ADK test was passed by 95%. As no minimum resolution is
specified, IPPM metric compliance is not linked to a particular
performance of an implementation.
o Reference to RFC 2330 and RFC 2679 for the 95% confidence interval
as preferred criterion to decide on statistical equivalence
o Reducing the proposed statistical test to ADK with 95% confidence.
1.1. Requirements Language 1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
2. Basic idea 2. Basic idea
The implementation of a standard compliant metric is expected to meet The implementation of a standard compliant metric is expected to meet
the requirements of the related metric specification. So before the requirements of the related metric specification. So before
comparing two metric implementations, each metric implementation is comparing two metric implementations, each metric implementation is
individually compared against the metric specification. individually compared against the metric specification.
Most metric specifications leave freedom to implementors on non- Most metric specifications leave freedom to implementors on non-
fundamental aspects of an individual metric (or options). Comparing fundamental aspects of an individual metric (or options). Comparing
different measurement results using a statistical test with the different measurement results using a statistical test with the
assumption of identical test path and testing conditions requires assumption of identical test path and testing conditions requires
knowledge of all differences in the overall test setup. Metric knowledge of all differences in the overall test setup. Metric
specification options chosen by implementors have to be documented. specification options chosen by implementors have to be documented.
It is REQUIRED to use identical implementation options wherever It is RECOMMENDED to use identical metric options for any test
possible for any test proposed here. Calibrations proposed by metric proposed here (an exception would be if a variable parameter of the
standards should be performed to further identify (and possibly metric definition is not configurable in one or more
reduce) potential sources of errors in the test setup. implementations). Calibrations specified by metric standards SHOULD
be performed to further identify (and possibly reduce) potential
sources of error in the test setup.
The Framework for IP Performance Metrics [RFC2330] expects that a The Framework for IP Performance Metrics [RFC2330] expects that a
"methodology for a metric should have the property that it is "methodology for a metric should have the property that it is
repeatable: if the methodology is used multiple times under identical repeatable: if the methodology is used multiple times under identical
conditions, it should result in consistent measurements." This means conditions, it should result in consistent measurements." This means
an implementation is expected to repeatedly measure a metric with an implementation is expected to repeatedly measure a metric with
consistent results (repeatability with the same result). Small consistent results (repeatability with the same result). Small
deviations in the test setup are expected to lead to small deviations deviations in the test setup are expected to lead to small deviations
in results only. To characterise statistical equivalence in the case in results only. To characterise statistical equivalence in the case
of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95%
skipping to change at page 8, line 17 skipping to change at page 6, line 33
measurement setup on the result, network conditions and paths MUST measurement setup on the result, network conditions and paths MUST
be identical for the compared implementations to the largest be identical for the compared implementations to the largest
possible degree. This includes both the stability and non- possible degree. This includes both the stability and non-
ambiguity of routes taken by the measurement packets. See RFC ambiguity of routes taken by the measurement packets. See RFC
2330 for a discussion on self-consistency. 2330 for a discussion on self-consistency.
o To minimize the influence of implementation options on the result, o To minimize the influence of implementation options on the result,
metric implementations SHOULD use identical options and parameters metric implementations SHOULD use identical options and parameters
for the metric under evaluation. for the metric under evaluation.
o The error induced by the sample size must be small enough to o The sample size must be large enough to minimize its influence on
minimize its influence on the test result. This may have to be the consistency of the test results. This consideration may be
respected, especially if two implementations measure with especially important if two implementations measure with different
different average probing rates. average packet transmission rates.
o The implementation with the lowest probing frequency determines o The implementation with the lowest average packet transmission
the smallest temporal interval for which samples can be compared. rate determines the smallest temporal interval for which samples
can be compared.
o Repeat comparisons with several independent metric samples to o Repeat comparisons with several independent metric samples to
avoid random indications of compatibility (or the lack of it). avoid random indications of compatibility (or the lack of it).
The metric specifications themselves are the primary focus of The metric specifications themselves are the primary focus of
evaluation, rather than the implementations of metrics. The evaluation, rather than the implementations of metrics. The
documentation produced by the advancement process should identify documentation produced by the advancement process should identify
which metric definitions and supporting material were found to be which metric definitions and supporting material were found to be
clearly worded and unambiguous, OR, it should identify ways in which clearly worded and unambiguous, OR, it should identify ways in which
the metric specification text should be revised to achieve clarity the metric specification text should be revised to achieve clarity
skipping to change at page 9, line 30 skipping to change at page 7, line 47
impairment generators. impairment generators.
o Use remotely separated test labs to compare the implementations o Use remotely separated test labs to compare the implementations
and measure across the Internet. and measure across the Internet.
o Use remotely separated test labs to compare the implementations o Use remotely separated test labs to compare the implementations
and measure across the Internet and include a single impairment and measure across the Internet and include a single impairment
generator to impact all measurement flows in non discriminatory generator to impact all measurement flows in non discriminatory
way. way.
The first two approaches work, but cause higher expenses than the The first two approaches work, but involve higher expenses than the
other ones (due to travel and/or shipping+installation). For the others (due to travel and/or shipping plus installation). For the
third option, ensuring two identically configured impairment third option, ensuring two identically configured impairment
generators requires well defined test cases and possibly identical generators requires well defined test cases and possibly identical
hard- and software. hardware and software.
As documented in a test report [morton-testplan-rfc2679], the last As documented in a test report [morton-testplan-rfc2679], the last
option was required to prove compatibility of two delay metric option was required to prove compatibility of two delay metric
implementations. An impairment generator is probably required when implementations. An impairment generator is probably required when
testing compatibility of most other metrics and it therefore testing compatibility of most other metrics and it is therefore
RECOMMENDED to include an impairment generator in metric test set RECOMMENDED to include an impairment generator in metric test setups.
ups.
3.1. Tests of an individual implementation against a metric 3.1. Tests of an individual implementation against a metric
specification specification
A metric implementation MUST support the requirements classified as A metric implementation is compliant with a metric specification if
"MUST" and "REQUIRED" of the related metric specification to be it supports the requirements classified as "MUST" and "REQUIRED" of
compliant to the latter. the related metric specification. An implementation that implements
all requirements is fully compliant with the specification, and the
degree of compliance SHOULD be noted in the conclusions of the
report.
Further, supported options of a metric implementation SHOULD be Further, supported options of a metric implementation SHOULD be
documented in sufficient detail. The documentation of chosen options documented in sufficient detail to evaluate whether the specification
is RECOMMENDED to minimise (and recognise) differences in the test was correctly interpreted. The documentation of chosen options
setup if two metric implementations are compared. Further, this should minimise (and recognise) differences in the test setup if two
documentation is used to validate and improve the underlying metric metric implementations are compared. Further, this documentation is
specification option, to remove options which saw no implementation used to validate or clarify the wording of the metric specification
or which are badly specified from the metric specification to be option, to remove options which saw no implementation or which are
promoted to a standard. This documentation SHOULD be made for all badly specified from the metric specification. This documentation
implementation-relevant specifications of a metric picked for a SHOULD be included for all implementation-relevant specifications of
comparison that are not explicitly marked as "MUST" or "REQUIRED" in a metric picked for a comparison, even those that are not explicitly
the RFC text. This applies for the following sections of all metric marked as "MUST" or "REQUIRED" in the RFC text. This applies for the
specifications: following sections of all metric specifications:
o Singleton Definition of the Metric. o Singleton Definition of the Metric.
o Sample Definition of the Metric. o Sample Definition of the Metric.
o Statistics Definition of the Metric. As statistics are compared o Statistics Definition of the Metric. As statistics are compared
by the test specified here, this documentation is required even in by the test specified here, this documentation is required even in
the case, that the metric specification does not contain a the case, that the metric specification does not contain a
Statistics Definition. Statistics Definition.
o Timing and Synchronisation related specification (if relevant for o Timing and Synchronisation related specification (if relevant for
the Metric). the Metric).
o Any other technical part present or missing in the metric o Any other technical part present or missing in the metric
specification, which is relevant for the implementation of the specification, which is relevant for the implementation of the
Metric. Metric.
RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric
implementations. A single IPPM conformant implementation MUST under implementations. A single IPPM conforming implementation should
otherwise identical network conditions produce precise results for under otherwise identical network conditions produce precise results
repeated measurements of the same metric. for repeated measurements of the same metric.
RFC 2330 prefers the "empirical distribution function" EDF to RFC 2330 prefers the "empirical distribution function" EDF to
describe collections of measurements. RFC 2330 determines, that describe collections of measurements. RFC 2330 determines, that
"unless otherwise stated, IPPM goodness-of-fit tests are done using "unless otherwise stated, IPPM goodness-of-fit tests are done using
5% significance." The goodness of fit test determines by which 5% significance." The goodness of fit test determines by which
precision two or more samples of a metric implementation belong to precision two or more samples of a metric implementation belong to
the same underlying distribution (of measured network performance the same underlying distribution (of measured network performance
events). The goodness of fit test suggested for the metric test is events). The goodness of fit test suggested for the metric test is
the Anderson-Darling K sample test (ADK sample test, K stands for the the Anderson-Darling K sample test (ADK sample test, K stands for the
number of samples to be compared) [ADK]. Please note that RFC 2330 number of samples to be compared) [ADK]. Please note that RFC 2330
skipping to change at page 11, line 9 skipping to change at page 9, line 27
The results of a repeated test with a single implementation MUST pass The results of a repeated test with a single implementation MUST pass
an ADK sample test with confidence level of 95%. The conditions for an ADK sample test with confidence level of 95%. The conditions for
which the ADK test has been passed with the specified confidence which the ADK test has been passed with the specified confidence
level MUST be documented. To formulate this differently: The level MUST be documented. To formulate this differently: The
requirement is to document the set of parameters with the smallest requirement is to document the set of parameters with the smallest
deviation, at which the results of the tested metric implementation deviation, at which the results of the tested metric implementation
pass an ADK test with a confidence level of 95%. The minimum pass an ADK test with a confidence level of 95%. The minimum
resolution available in the reported results from each implementation resolution available in the reported results from each implementation
MUST be taken into account in the ADK test. MUST be taken into account in the ADK test.
The test conditions which MUST be documented for a passed metric test The test conditions to be documented for a passed metric test
include: include:
o The metric resolution at which a test was passed (e.g. the o The metric resolution at which a test was passed (e.g. the
resolution of timestamps) resolution of timestamps)
o The parameters modified by an impairment generator. o The parameters modified by an impairment generator.
o The impairment generator parameter settings. o The impairment generator parameter settings.
3.2. Test setup resulting in identical live network testing conditions 3.2. Test setup resulting in identical live network testing conditions
skipping to change at page 11, line 36 skipping to change at page 10, line 6
mechanisms like those that achieve load balancing (see [RFC4928]). mechanisms like those that achieve load balancing (see [RFC4928]).
This section proposes two measures to deal with both issues. This section proposes two measures to deal with both issues.
Tunneling mechanisms can be used to avoid parallel processing of Tunneling mechanisms can be used to avoid parallel processing of
different flows in the network. Measuring by separate parallel probe different flows in the network. Measuring by separate parallel probe
flows results in repeated collection of data. If both measures are flows results in repeated collection of data. If both measures are
combined, WAN network conditions are identical for a number of combined, WAN network conditions are identical for a number of
independent measurement flows, no matter what the network conditions independent measurement flows, no matter what the network conditions
are in detail. are in detail.
Any measurement setup MUST be made to avoid the probing traffic Any measurement setup must be made to avoid the probing traffic
itself to impede the metric measurement. The created measurement itself to impede the metric measurement. The created measurement
load MUST NOT result in congestion at the access link connecting the load must not result in congestion at the access link connecting the
measurement implementation to the WAN. The created measurement load measurement implementation to the WAN. The created measurement load
MUST NOT overload the measurement implementation itself, e.g., by must not overload the measurement implementation itself, e.g., by
causing a high CPU load or by creating imprecisions due to internal causing a high CPU load or by creating imprecisions due to internal
transmit (receive respectively) probe packet collisions. transmit (receive respectively) probe packet collisions.
Tunneling multiple flows reaching a network element on a single Tunneling multiple flows reaching a network element on a single
physical port may allow to transmit all packets of the tunnel via the physical port may allow to transmit all packets of the tunnel via the
same path. Applying tunnels to avoid undesired influence of standard same path. Applying tunnels to avoid undesired influence of standard
routing for measurement purposes is a concept known from literature, routing for measurement purposes is a concept known from literature,
see e.g. GRE encapsulated multicast probing [GU+Duffield]. An see e.g. GRE encapsulated multicast probing [GU-Duffield]. An
existing IP in IP tunnel protocol can be applied to avoid Equal-Cost existing IP in IP tunnel protocol can be applied to avoid Equal-Cost
Multi-Path (ECMP) routing of different measurement streams if it Multi-Path (ECMP) routing of different measurement streams if it
meets the following criteria: meets the following criteria:
o Inner IP packets from different measurement implementations are o Inner IP packets from different measurement implementations are
mapped into a single tunnel with single outer IP origin and mapped into a single tunnel with single outer IP origin and
destination address as well as origin and destination port numbers destination address as well as origin and destination port numbers
which are identical for all packets. which are identical for all packets.
o An easily accessible commodity tunneling protocol allows to carry o An easily accessible commodity tunneling protocol allows to carry
skipping to change at page 14, line 11 skipping to change at page 12, line 34
to B and in the reverse direction. The remote site VLANs are to B and in the reverse direction. The remote site VLANs are
U-bridged at the local site Ethernet switch. The measurement packets U-bridged at the local site Ethernet switch. The measurement packets
of site 1 travel tunnel A->B first, are U-bridged at site 2 and of site 1 travel tunnel A->B first, are U-bridged at site 2 and
travel tunnel B->A second. Measurement packets of site 2 travel travel tunnel B->A second. Measurement packets of site 2 travel
tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B
second. So all measurement packets pass the same tunnel segments, second. So all measurement packets pass the same tunnel segments,
but in different segment order. but in different segment order.
If tunneling is applied, two tunnels MUST carry all test traffic in If tunneling is applied, two tunnels MUST carry all test traffic in
between the test site and the remote site. For example, if 802.1Q between the test site and the remote site. For example, if 802.1Q
Ethernet Virtual LANs (VLAN) are applied and the measurement streams Virtual LANs (VLAN) are applied and the measurement streams are
are carried in different VLANs, the IP tunnel or Pseudo Wires carried in different VLANs, the IP tunnel or Pseudo Wires
respectively MUST be set up in physical port mode to avoid set up of respectively are set up in physical port mode to avoid set up of
Pseudo Wires per VLAN (which may see different paths due to ECMP Pseudo Wires per VLAN (which may see different paths due to ECMP
routing), see RFC 4448. The remote router and the Ethernet switch routing), see RFC 4448. The remote router and the Ethernet switch
shown in figure 3 has to support 802.1Q in this set up. shown in figure 3 has to support 802.1Q in this set up.
The IP packet size of the metric implementation SHOULD be chosen The IP packet size of the metric implementation SHOULD be chosen
small enough to avoid fragmentation due to the added Ethernet and small enough to avoid fragmentation due to the added Ethernet and
tunnel headers. Otherwise, the impact of tunnel overhead on tunnel headers. Otherwise, the impact of tunnel overhead on
fragmentation and interface MTU size MUST be understood and taken fragmentation and interface MTU size must be understood and taken
into account (see [RFC4459]). into account (see [RFC4459]).
An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each
containing measurement traffic of a single measurement system was containing measurement traffic of a single measurement system was
successfully applied when testing compatibility of two metric successfully applied when testing compatibility of two metric
implementations [morton-testplan-rfc2679]. implementations [morton-testplan-rfc2679]. Ethernet over L2TPv3
[RFC4719] was picked for this test.
The following headers may have to be accounted for when calculating The following headers may have to be accounted for when calculating
total packet length, if VLANs and Ethernet over L2TPv3 tunnels are total packet length, if VLANs and Ethernet over L2TPv3 tunnels are
applied: applied:
o Ethernet 802.1Q: 22 Byte. o Ethernet 802.1Q: 22 Byte.
o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28
Byte for L2TPv3 data messages over UDP. Byte for L2TPv3 data messages over UDP.
skipping to change at page 16, line 4 skipping to change at page 14, line 32
with 4 samples even if a 2 sample test with 4 samples even if a 2 sample test
failed[morton-testplan-rfc2679]. failed[morton-testplan-rfc2679].
Some additional guidelines to calculate and compare samples to Some additional guidelines to calculate and compare samples to
perform a metric test are: perform a metric test are:
o To compare different probes of a common underlying distribution in o To compare different probes of a common underlying distribution in
terms of metrics characterising a communication network requires terms of metrics characterising a communication network requires
to respect the temporal nature for which the assumption of common to respect the temporal nature for which the assumption of common
underlying distribution may hold. Any singletons or samples to be underlying distribution may hold. Any singletons or samples to be
compared MUST be captured within the same time interval. compared must be captured within the same time interval.
o If statistical events like rates are used to characterise measured o If statistical events like rates are used to characterise measured
metrics of a time-interval, its RECOMMENDED to pick as a minimum 5 metrics of a time-interval, a minimum 5 singletons of a relevant
singletons of a relevant metric to ensure a minimum confidence metric should be picked to ensure a minimum confidence into the
into the reported value. The error margin of the determined rate reported value. The error margin of the determined rate depends
depends on the number singletons (refer to statistical textbooks on the number singletons (refer to statistical textbooks on
on Student's t-test). As an example, any packet loss measurement Student's t-test). As an example, any packet loss measurement
interval to be compared with the results of another implementation interval to be compared with the results of another implementation
contains at least five lost packets to have some confidence that contains at least five lost packets to have some confidence that
the observed loss rate wasn't caused by a small number of random the observed loss rate wasn't caused by a small number of random
packet drops. packet drops.
o The minimum number of singletons or samples to be compared by an o The minimum number of singletons or samples to be compared by an
Anderson-Darling test SHOULD be 100 per tested metric Anderson-Darling test should be 100 per tested metric
implementation. Note that the Anderson-Darling test detects small implementation. Note that the Anderson-Darling test detects small
differences in distributions fairly well and will fail for high differences in distributions fairly well and will fail for high
number of compared results (RFC2330 mentions an example with 8192 number of compared results (RFC2330 mentions an example with 8192
measurements where an Anderson-Darling test always failed). measurements where an Anderson-Darling test always failed).
o Generally, the Anderson-Darling test is sensitive to differences o Generally, the Anderson-Darling test is sensitive to differences
in the accuracy or bias associated with varying implementations or in the accuracy or bias associated with varying implementations or
test conditions. These dissimilarities may result in differing test conditions. These dissimilarities may result in differing
averages of samples to be compared. An example may be different averages of samples to be compared. An example may be different
packet sizes, resulting in a constant delay difference between packet sizes, resulting in a constant delay difference between
skipping to change at page 16, line 50 skipping to change at page 15, line 31
precisely, for every positive epsilon, there exists a positive delta, precisely, for every positive epsilon, there exists a positive delta,
such that if two sets of conditions are within delta of each other, such that if two sets of conditions are within delta of each other,
then the resulting measurements will be within epsilon of each then the resulting measurements will be within epsilon of each
other." A small variation in conditions in the context of the metric other." A small variation in conditions in the context of the metric
test proposed here can be seen as different implementations measuring test proposed here can be seen as different implementations measuring
the same metric along the same path. the same metric along the same path.
IPPM metric specifications however allow for implementor options to IPPM metric specifications however allow for implementor options to
the largest possible degree. It cannot be expected that two the largest possible degree. It cannot be expected that two
implementors allow 100% identical options in their implementations. implementors allow 100% identical options in their implementations.
Testers SHOULD to the highest degree possible pick the same Testers SHOULD pick the same metric measurement configurations for
configurations for their systems when comparing their implementations their systems when comparing their implementations by a metric test.
by a metric test.
In some cases, a goodness of fit test may not be possible or show In some cases, a goodness of fit test may not be possible or show
disappointing results. To clarify the difficulties arising from disappointing results. To clarify the difficulties arising from
different implementation options, the individual options picked for different metric implementation options, the individual options
every compared implementation SHOULD be documented in sufficient picked for every compared metric implementation should be documented
detail. Based on this documentation, the underlying metric as specified in section 3.5. If the cause of the failure is a lack
specification should be improved before it is promoted to a standard. of specification clarity or multiple legitimate interpretations of
the definition text, the text should be modified and the resulting
memo proposed for consensus and (possible) advancement to Internet
Standard.
The same statistical test as applicable to quantify precision of a The same statistical test as applicable to quantify precision of a
single metric implementation MUST be used to compare metric result single metric implementation must be used to compare metric result
equivalence for different implementations. To document equivalence for different implementations. To document
compatibility, the smallest measurement resolution at which the compatibility, the smallest measurement resolution at which the
compared implementations passed the ADK sample test MUST be compared implementations passed the ADK sample test must be
documented. documented.
For different implementations of the same metric, "variations in For different implementations of the same metric, "variations in
conditions" are reasonably expected. The ADK test comparing samples conditions" are reasonably expected. The ADK test comparing samples
of the different implementations MAY result in a lower precision than of the different implementations may result in a lower precision than
the test for precision in the same-implementation comparison. the test for precision in the same-implementation comparison.
3.4. Clock synchronisation 3.4. Clock synchronisation
Clock synchronization effects require special attention. Accuracy of Clock synchronization effects require special attention. Accuracy of
one-way active delay measurements for any metrics implementation one-way active delay measurements for any metrics implementation
depends on clock synchronization between the source and destination depends on clock synchronization between the source and destination
of tests. Ideally, one-way active delay measurement (RFC 2679, of tests. Ideally, one-way active delay measurement (RFC 2679,
[RFC2679]) test endpoints either have direct access to independent [RFC2679]) test endpoints either have direct access to independent
GPS or CDMA-based time sources or indirect access to nearby NTP GPS or CDMA-based time sources or indirect access to nearby NTP
primary (stratum 1) time sources, equipped with GPS receivers. primary (stratum 1) time sources, equipped with GPS receivers.
Access to these time sources may not be available at all test Access to these time sources may not be available at all test
locations associated with different Internet paths, for a variety of locations associated with different Internet paths, for a variety of
reasons out of scope of this document. reasons out of scope of this document.
When secondary (stratum 2 and above) time sources are used with NTP When secondary (stratum 2 and above) time sources are used with NTP
running across the same network, whose metrics are subject to running across the same network, whose metrics are subject to
comparative implementation tests, network impairments can affect comparative implementation tests, network impairments can affect
clock synchronization, distort sample one-way values and their clock synchronization, distort sample one-way values and their
interval statistics. It is RECOMMENDED to discard sample one-way interval statistics. It is recommended to discard sample one-way
delay values for any implementation, when one of the following delay values for any implementation when one of the following
reliability conditions is met: reliability conditions is met:
o Delay is measured and is finite in one direction, but not the o Delay is measured and is finite in one direction, but not the
other. other.
o Absolute value of the difference between the sum of one-way o Absolute value of the difference between the sum of one-way
measurements in both directions and round-trip measurement is measurements in both directions and round-trip measurement is
greater than X% of the latter value. greater than X% of the latter value.
Examination of the second condition requires RTT measurement for Examination of the second condition requires RTT measurement for
skipping to change at page 18, line 20 skipping to change at page 16, line 50
unreliable one-way delay samples and misidentification of reliable unreliable one-way delay samples and misidentification of reliable
samples under a wide range of Internet path RTTs probably requires samples under a wide range of Internet path RTTs probably requires
further study. further study.
An IPPM compliant metric implementation of an RFC that requires An IPPM compliant metric implementation of an RFC that requires
synchronized clocks is expected to provide precise measurement synchronized clocks is expected to provide precise measurement
results. results.
IF an implementation publishes a specification of its precision, such IF an implementation publishes a specification of its precision, such
as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then
the specification SHOULD be met over a useful measurement duration. the specification should be met over a useful measurement duration.
For example, if the metric is measured along an Internet path which For example, if the metric is measured along an Internet path which
is stable and not congested, then the precision specification SHOULD is stable and not congested, then the precision specification should
be met over durations of an hour or more. be met over durations of an hour or more.
3.5. Recommended Metric Verification Measurement Process 3.5. Recommended Metric Verification Measurement Process
In order to meet their obligations under the IETF Standards Process In order to meet their obligations under the IETF Standards Process
the IESG must be convinced that each metric specification advanced to the IESG must be convinced that each metric specification advanced to
Draft Standard or Internet Standard status is clearly written, that Internet Standard status is clearly written, that there are a
there are a sufficient number of verified equivalent implementations, sufficient number of verified equivalent implementations, and that
and that options that have been implemented are documented. options that have been implemented are documented.
In the context of this document, metrics are designed to measure some In the context of this document, metrics are designed to measure some
characteristic of a data network. An aim of any metric definition characteristic of a data network. An aim of any metric definition
should be that it should be specified in a way that can reliably should be that it should be specified in a way that can reliably
measure the specific characteristic in a repeatable way across measure the specific characteristic in a repeatable way across
multiple independent implementations. multiple independent implementations.
Each metric, statistic or option of those to be validated MUST be Each metric, statistic or option of those to be validated MUST be
compared against a reference measurement or another implementation by compared against a reference measurement or another implementation as
as specified by this document. specified in this document.
Finally, the metric definitions, embodied in the text of the RFCs, Finally, the metric definitions, embodied in the text of the RFCs,
are the objects that require evaluation and possible revision in are the objects that require evaluation and possible revision in
order to advance to the next step on the standards track. order to advance to Internet Standard.
IF two (or more) implementations do not measure an equivalent metric IF two (or more) implementations do not measure an equivalent metric
as specified by this document, as specified by this document,
AND sources of measurement error do not adequately explain the lack AND sources of measurement error do not adequately explain the lack
of agreement, of agreement,
THEN the details of each implementation should be audited along with THEN the details of each implementation should be audited along with
the exact definition text, to determine if there is a lack of clarity the exact definition text, to determine if there is a lack of clarity
that has caused the implementations to vary in a way that affects the that has caused the implementations to vary in a way that affects the
correspondence of the results. correspondence of the results.
IF there was a lack of clarity or multiple legitimate interpretations IF there was a lack of clarity or multiple legitimate interpretations
of the definition text, of the definition text,
THEN the text should be modified and the resulting memo proposed for THEN the text should be modified and the resulting memo proposed for
consensus and (possible) advancement along the standards track. consensus and (possible) advancement along the standards track.
skipping to change at page 19, line 16 skipping to change at page 17, line 47
that has caused the implementations to vary in a way that affects the that has caused the implementations to vary in a way that affects the
correspondence of the results. correspondence of the results.
IF there was a lack of clarity or multiple legitimate interpretations IF there was a lack of clarity or multiple legitimate interpretations
of the definition text, of the definition text,
THEN the text should be modified and the resulting memo proposed for THEN the text should be modified and the resulting memo proposed for
consensus and (possible) advancement along the standards track. consensus and (possible) advancement along the standards track.
Finally, all the findings MUST be documented in a report that can Finally, all the findings MUST be documented in a report that can
support advancement on the standards track, similar to those support advancement to Internet Standard, as described here (similar
described in [RFC5657]. The list of measurement devices used in to those described in [RFC5657]). The list of measurement devices
testing satisfies the implementation requirement, while the test used in testing satisfies the implementation requirement, while the
results provide information on the quality of each specification in test results provide information on the quality of each specification
the metric RFC (the surrogate for feature interoperability). in the metric RFC (the surrogate for feature interoperability).
The complete process of advancing a metric specification to a The complete process of advancing a metric specification to a
standard as defined by this document is illustrated in Figure 4. standard as defined by this document is illustrated in Figure 4.
,---. ,---.
/ \ / \
( Start ) ( Start )
\ / Implementations \ / Implementations
`-+-' +-------+ `-+-' +-------+
| /| 1 `. | /| 1 `.
+---+----+ / +-------+ `.-----------+ ,-------. +---+----+ / +-------+ `.-----------+ ,-------.
| RFC | / |Check for | ,' was RFC `. YES | RFC | / |Check for | ,' was RFC `. YES
| | / |Equivalence.... clause x ------+ | | / |Equivalence.... clause x ------+
| |/ +-------+ |under | `. clear? ,' | | |/ +-------+ |under | `. clear? ,' |
| Metric \.....| 2 ....relevant | `---+---' +----+-----+ | Metric \.....| 2 ....relevant | `---+---' +----+-----+
| Metric |\ +-------+ |identical | No | |Report | | Metric |\ +-------+ |identical | No | |Report |
| Metric | \ |network | +--+----+ |results + | | Metric | \ |network | +--+----+ |results + |
| ... | \ |conditions | |Modify | |Advance | | ... | \ |conditions | |Modify | |Advance |
| | \ +-------+ | | |Spec +--+RFC | | | \ +-------+ | | |Spec +--+RFC |
+--------+ \| n |.'+-----------+ +-------+ |request(?)| +--------+ \| n |.'+-----------+ +-------+ |request |
+-------+ +----------+ +-------+ +----------+
Illustration of the metric standardisation process Illustration of the metric standardisation process
Figure 4 Figure 4
Any recommendation for the advancement of a metric specification MUST Any recommendation for the advancement of a metric specification MUST
be accompanied by an implementation report, as is the case with all be accompanied by an implementation report. The implementation
requests for the advancement of IETF specifications. The report needs to include the tests performed, the applied test setup,
implementation report needs to include the tests performed, the the specific metrics in the RFC and reports of the tests performed
applied test setup, the specific metrics in the RFC and reports of with two or more implementations. The test plan needs to specify the
the tests performed with two or more implementations. The test plan precision reached for each measured metric and thus define the
needs to specify the precision reached for each measured metric and meaning of "statistically equivalent" for the specific metrics being
thus define the meaning of "statistically equivalent" for the tested.
specific metrics being tested.
Ideally, the test plan would co-evolve with the development of the Ideally, the test plan would co-evolve with the development of the
metric, since that's when people have the most context in their metric, since that's when participants have the clearest context in
thinking regarding the different subtleties that can arise. their minds regarding the different subtleties that can arise.
In particular, the implementation report MUST as a minimum document: In particular, the implementation report MUST as a minimum document:
o The metric compared and the RFC specifying it. This includes o The metric compared and the RFC specifying it. This includes
statements as required by the section "Tests of an individual statements as required by the section "Tests of an individual
implementation against a metric specification" of this document. implementation against a metric specification" of this document.
o The measurement configuration and setup. o The measurement configuration and setup.
o A complete specification of the measurement stream (mean rate, o A complete specification of the measurement stream (mean rate,
skipping to change at page 21, line 9 skipping to change at page 19, line 38
and network conditions allowing reproduction of these laboratory and network conditions allowing reproduction of these laboratory
conditions for readers of the implementation report. conditions for readers of the implementation report.
o The documentation helping to improve metric specifications defined o The documentation helping to improve metric specifications defined
by this section. by this section.
All of the tests for each set SHOULD be run in a test setup as All of the tests for each set SHOULD be run in a test setup as
specified in the section "Test setup resulting in identical live specified in the section "Test setup resulting in identical live
network testing conditions." network testing conditions."
If a different test set up is chosen, it is RECOMMENDED to avoid If a different test setup is chosen, it is recommended to avoid
effects falsifying results of validation measurements caused by real effects falsifying results of validation measurements caused by real
data networks (like parallelism in devices and networks). Data data networks (like parallelism in devices and networks). Data
networks may forward packets differently in the case of: networks may forward packets differently in the case of:
o Different packet sizes chosen for different metric o Different packet sizes chosen for different metric
implementations. A proposed countermeasure is selecting the same implementations. A proposed countermeasure is selecting the same
packet size when validating results of two samples or a sample packet size when validating results of two samples or a sample
against an original distribution. against an original distribution.
o Selection of differing IP addresses and ports used by different o Selection of differing IP addresses and ports used by different
skipping to change at page 23, line 16 skipping to change at page 21, line 42
This memo does not raise any specific security issues. This memo does not raise any specific security issues.
8. References 8. References
8.1. Normative References 8.1. Normative References
[RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003,
October 1996. October 1996.
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision
3", BCP 9, RFC 2026, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330, "Framework for IP Performance Metrics", RFC 2330,
May 1998. May 1998.
[RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn,
G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"",
RFC 2661, August 1999. RFC 2661, August 1999.
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Delay Metric for IPPM", RFC 2679, September 1999. Delay Metric for IPPM", RFC 2679, September 1999.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
Delay Metric for IPPM", RFC 2681, September 1999.
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P.
Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, Traina, "Generic Routing Encapsulation (GRE)", RFC 2784,
March 2000. March 2000.
[RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling
Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005.
[RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron,
"Encapsulation Methods for Transport of Ethernet over MPLS "Encapsulation Methods for Transport of Ethernet over MPLS
Networks", RFC 4448, April 2006. Networks", RFC 4448, April 2006.
[RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the-
Network Tunneling", RFC 4459, April 2006.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, September 2006. (OWAMP)", RFC 4656, September 2006.
[RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport [RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport
of Ethernet Frames over Layer 2 Tunneling Protocol Version of Ethernet Frames over Layer 2 Tunneling Protocol Version
3 (L2TPv3)", RFC 4719, November 2006. 3 (L2TPv3)", RFC 4719, November 2006.
[RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal
Cost Multipath Treatment in MPLS Networks", BCP 128, Cost Multipath Treatment in MPLS Networks", BCP 128,
RFC 4928, June 2007. RFC 4928, June 2007.
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation
and Implementation Reports for Advancement to Draft and Implementation Reports for Advancement to Draft
Standard", BCP 9, RFC 5657, September 2009. Standard", BCP 9, RFC 5657, September 2009.
[RFC6410] Housley, R., Crocker, D., and E. Burger, "Reducing the
Standards Track to Two Maturity Levels", BCP 9, RFC 6410,
October 2011.
8.2. Informative References 8.2. Informative References
[ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling
Tests of fit, for continuous and discrete cases", Tests of fit, for continuous and discrete cases",
University of Washington, Technical Report No. 81, University of Washington, Technical Report No. 81,
May 1986. May 1986.
[GU+Duffield] [GU-Duffield]
Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE
Encapsulated Multicast Probing: A Scalable Technique for Encapsulated Multicast Probing: A Scalable Technique for
Measuring One-Way Loss", SIGMETRICS'07 San Diego, Measuring One-Way Loss", SIGMETRICS'07 San Diego,
California, USA, June 2007. California, USA, June 2007.
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision
3", BCP 9, RFC 2026, October 1996.
[RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the-
Network Tunneling", RFC 4459, April 2006.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, October 2008. RFC 5357, October 2008.
[Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and
Combinations of Such Tests. R package version 1.0.", , Combinations of Such Tests. R package version 1.0.", ,
2008. 2008.
[Rtool] R Development Core Team, "R: A language and environment [Rtool] R Development Core Team, "R: A language and environment
for statistical computing. R Foundation for Statistical for statistical computing. R Foundation for Statistical
Computing, Vienna, Austria. ISBN 3-900051-07-0, URL Computing, Vienna, Austria. ISBN 3-900051-07-0, URL
http://www.R-project.org/", , 2011. http://www.R-project.org/", , 2011.
[Rule of thumb]
Hardy, M., "Confidence interval", March 2010.
[bradner-metrictest] [bradner-metrictest]
Bradner, S., Mankin, A., and V. Paxson, "Advancement of Bradner, S., Mankin, A., and V. Paxson, "Advancement of
metrics specifications on the IETF Standards Track", metrics specifications on the IETF Standards Track",
draft -bradner-metricstest-03, (work in progress), draft -bradner-metricstest-03, (work in progress),
July 2007. July 2007.
[morton-advance-metrics]
Morton, A., "Problems and Possible Solutions for Advancing
Metrics on the Standards Track", draft -morton-ippm-
advance-metrics-00, (work in progress), July 2009.
[morton-advance-metrics-01]
Morton, A., "Lab Test Results for Advancing Metrics on the
Standards Track", draft -morton-ippm-advance-metrics-01,
(work in progress), June 2010.
[morton-testplan-rfc2679] [morton-testplan-rfc2679]
Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test
Plan and Results for Advancing RFC 2679 on the Standards Plan and Results for Advancing RFC 2679 on the Standards
Track", draft -morton-ippm-testplan-rfc2679-01, (work in Track", draft -morton-ippm-testplan-rfc2679-01, (work in
progress), June 2011. progress), June 2011.
Appendix A. An example on a One-way Delay metric validation Appendix A. An example on a One-way Delay metric validation
The text of this appendix is not binding. It is an example how parts The text of this appendix is not binding. It is an example how parts
of a One-way Delay metric test could look like. of a One-way Delay metric test could look like.
http://xml.resource.org/public/rfc/bibxml/
A.1. Compliance to Metric specification requirements A.1. Compliance to Metric specification requirements
One-way Delay, Loss threshold, RFC 2679 One-way Delay, Loss threshold, RFC 2679
This test determines if implementations use the same configured This test determines if implementations use the same configured
maximum waiting time delay from one measurement to another under maximum waiting time delay from one measurement to another under
different delay conditions, and correctly declare packets arriving in different delay conditions, and correctly declare packets arriving in
excess of the waiting time threshold as lost. See Section 3.5 of excess of the waiting time threshold as lost. See Section 3.5 of
RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679.
skipping to change at page 29, line 27 skipping to change at page 27, line 30
but this is as it should be. but this is as it should be.
The C++ code below will perform a 2-sample AD comparison when The C++ code below will perform a 2-sample AD comparison when
compiled and presented with two column vectors in a file (using white compiled and presented with two column vectors in a file (using white
space as separation). This version contains modifications to use the space as separation). This version contains modifications to use the
vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The
status of the comparison can be checked on the command line with "$ status of the comparison can be checked on the command line with "$
echo $?" or the last line can be replaced with a printf statement for echo $?" or the last line can be replaced with a printf statement for
adk_result instead. adk_result instead.
/* Routines for computing the Anderson-Darling 2 sample /*
* test statistic.
*
* Implemented based on the description in
* "Anderson-Darling K Sample Test" Heckert, Alan and
* Filliben, James, editors, Dataplot Reference Manual,
* Chapter 15 Auxiliary, NIST, 2004.
* Official Reference by 2010
* Heckert, N. A. (2001). Dataplot website at the
* National Institute of Standards and Technology:
* http://www.itl.nist.gov/div898/software/dataplot.html/
* June 2001.
*/
#include <iostream> Copyright (c) 2011 IETF Trust and the persons identified
#include <fstream> as authors of the code. All rights reserved.
#include <vector>
#include <sstream>
using namespace std; Redistribution and use in source and binary forms, with
or without modification, is permitted pursuant to, and subject
to the license terms contained in, the Simplified BSD License
set forth in Section 4.c of the IETF Trust's Legal Provisions
Relating to IETF Documents (http://trustee.ietf.org/license-info).
int main() { */
vector<double> vec1, vec2;
double adk_result;
static int k, val_st_z_samp1, val_st_z_samp2,
val_eq_z_samp1, val_eq_z_samp2,
j, n_total, n_sample1, n_sample2, L,
max_number_samples, line, maxnumber_z;
static int column_1, column_2;
static double adk, n_value, z, sum_adk_samp1,
sum_adk_samp2, z_aux;
static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux;
static bool next_z_sample2, equal_z_both_samples;
static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2,
old_eq_line1;
static double adk_criterium = 1.993; /* Routines for computing the Anderson-Darling 2 sample
* test statistic.
*
* Implemented based on the description in
* "Anderson-Darling K Sample Test" Heckert, Alan and
* Filliben, James, editors, Dataplot Reference Manual,
* Chapter 15 Auxiliary, NIST, 2004.
* Official Reference by 2010
* Heckert, N. A. (2001). Dataplot website at the
* National Institute of Standards and Technology:
* http://www.itl.nist.gov/div898/software/dataplot.html/
* June 2001.
*/
/* vec1 and vec2 to be initialised with sample 1 and #include <iostream>
* sample 2 values in ascending order */ #include <fstream>
while (!cin.eof()) { #include <vector>
double f1, f2; #include <sstream>
cin >> f1;
cin >> f2;
vec1.push_back(f1);
vec2.push_back(f2);
}
k = 2; using namespace std;
n_sample1 = vec1.size() - 1;
n_sample2 = vec2.size() - 1;
// -1 because vec[0] is a dummy value int main() {
n_total = n_sample1 + n_sample2; vector<double> vec1, vec2;
double adk_result;
static int k, val_st_z_samp1, val_st_z_samp2,
val_eq_z_samp1, val_eq_z_samp2,
j, n_total, n_sample1, n_sample2, L,
max_number_samples, line, maxnumber_z;
static int column_1, column_2;
static double adk, n_value, z, sum_adk_samp1,
sum_adk_samp2, z_aux;
static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux;
static bool next_z_sample2, equal_z_both_samples;
static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2,
old_eq_line1;
/* value equal to the line with a value = zj in sample 1. static double adk_criterium = 1.993;
* Here j=1, so the line is 1.
*/
val_eq_z_samp1 = 1;
/* value equal to the line with a value = zj in sample 2. /* vec1 and vec2 to be initialised with sample 1 and
* Here j=1, so the line is 1. * sample 2 values in ascending order */
*/ while (!cin.eof()) {
val_eq_z_samp2 = 1; double f1, f2;
cin >> f1;
cin >> f2;
vec1.push_back(f1);
vec2.push_back(f2);
}
/* value equal to the last line with a value < zj k = 2;
* in sample 1. Here j=1, so the line is 0. n_sample1 = vec1.size() - 1;
*/ n_sample2 = vec2.size() - 1;
val_st_z_samp1 = 0;
/* value equal to the last line with a value < zj // -1 because vec[0] is a dummy value
* in sample 1. Here j=1, so the line is 0. n_total = n_sample1 + n_sample2;
*/ /* value equal to the line with a value = zj in sample 1.
val_st_z_samp2 = 0; * Here j=1, so the line is 1.
*/
val_eq_z_samp1 = 1;
sum_adk_samp1 = 0; /* value equal to the line with a value = zj in sample 2.
sum_adk_samp2 = 0; * Here j=1, so the line is 1.
j = 1; */
val_eq_z_samp2 = 1;
// as mentioned above, j=1 /* value equal to the last line with a value < zj
equal_z_both_samples = false; * in sample 1. Here j=1, so the line is 0.
*/
val_st_z_samp1 = 0;
next_z_sample2 = false; /* value equal to the last line with a value < zj
* in sample 1. Here j=1, so the line is 0.
*/
val_st_z_samp2 = 0;
//assuming the next z to be of sample 1 sum_adk_samp1 = 0;
stop_loop1 = n_sample1 + 1; sum_adk_samp2 = 0;
j = 1;
// + 1 because vec[0] is a dummy, see n_sample1 declaration // as mentioned above, j=1
stop_loop2 = n_sample2 + 1; equal_z_both_samples = false;
stop_loop3 = n_total + 1;
/* The required z values are calculated until all values next_z_sample2 = false;
* of both samples have been taken into account. See the
* lines above for the stoploop values. Construct required
* to avoid a mathematical operation in the While condition
*/
while (((stop_loop1 > val_eq_z_samp1)
|| (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j)
{
if(val_eq_z_samp1 < n_sample1+1)
{
/* here, a preliminary zj value is set.
* See below how to calculate the actual zj.
*/
z = vec1[val_eq_z_samp1];
/* this while sequence calculates the number of values //assuming the next z to be of sample 1
* equal to z. stop_loop1 = n_sample1 + 1;
*/
while ((val_eq_z_samp1+1 < n_sample1)
&& z == vec1[val_eq_z_samp1+1] )
{
val_eq_z_samp1++;
}
}
else
{
val_eq_z_samp1 = 0;
val_st_z_samp1 = n_sample1;
// this should be val_eq_z_samp1 - 1 = n_sample1 // + 1 because vec[0] is a dummy, see n_sample1 declaration
} stop_loop2 = n_sample2 + 1;
stop_loop3 = n_total + 1;
if(val_eq_z_samp2 < n_sample2+1) /* The required z values are calculated until all values
{ * of both samples have been taken into account. See the
z_aux = vec2[val_eq_z_samp2];; * lines above for the stoploop values. Construct required
* to avoid a mathematical operation in the While condition
*/
while (((stop_loop1 > val_eq_z_samp1)
|| (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j)
{
if(val_eq_z_samp1 < n_sample1+1)
{
/* here, a preliminary zj value is set.
* See below how to calculate the actual zj.
*/
z = vec1[val_eq_z_samp1];
/* this while sequence calculates the number of values /* this while sequence calculates the number of values
* equal to z_aux * equal to z.
*/ */
while ((val_eq_z_samp1+1 < n_sample1)
&& z == vec1[val_eq_z_samp1+1] )
{
val_eq_z_samp1++;
}
}
else
{
val_eq_z_samp1 = 0;
val_st_z_samp1 = n_sample1;
while ((val_eq_z_samp2+1 < n_sample2) // this should be val_eq_z_samp1 - 1 = n_sample1
&& z_aux == vec2[val_eq_z_samp2+1] ) }
{
val_eq_z_samp2++;
}
/* the smaller of the two actual data values is picked if(val_eq_z_samp2 < n_sample2+1)
* as the next zj. {
*/ z_aux = vec2[val_eq_z_samp2];;
if(z > z_aux) /* this while sequence calculates the number of values
{ * equal to z_aux
z = z_aux; */
next_z_sample2 = true;
}
else
{
if (z == z_aux)
{
equal_z_both_samples = true;
}
/* This is the case, if the last value of column1 is while ((val_eq_z_samp2+1 < n_sample2)
* smaller than the remaining values of column2. && z_aux == vec2[val_eq_z_samp2+1] )
*/ {
if (val_eq_z_samp1 == 0) val_eq_z_samp2++;
{ }
z = z_aux;
next_z_sample2 = true;
}
}
}
else
{
val_eq_z_samp2 = 0;
val_st_z_samp2 = n_sample2;
// this should be val_eq_z_samp2 - 1 = n_sample2 /* the smaller of the two actual data values is picked
* as the next zj.
*/
} if(z > z_aux)
{
z = z_aux;
next_z_sample2 = true;
}
else
{
if (z == z_aux)
{
equal_z_both_samples = true;
}
/* in the following, sum j = 1 to L is calculated for /* This is the case, if the last value of column1 is
* sample 1 and sample 2. * smaller than the remaining values of column2.
*/ */
if (equal_z_both_samples) if (val_eq_z_samp1 == 0)
{ {
z = z_aux;
next_z_sample2 = true;
}
}
}
else
{
val_eq_z_samp2 = 0;
val_st_z_samp2 = n_sample2;
/* hj is the number of values in the combined sample // this should be val_eq_z_samp2 - 1 = n_sample2
* equal to zj
*/
hj = val_eq_z_samp1 - val_st_z_samp1
+ val_eq_z_samp2 - val_st_z_samp2;
/* H_j is the number of values in the combined sample }
* smaller than zj plus one half the the number of
* values in the combined sample equal to zj
* (that's hj/2).
*/
H_j = val_st_z_samp1 + val_st_z_samp2
+ hj / 2;
/* F1j is the number of values in the 1st sample /* in the following, sum j = 1 to L is calculated for
* which are less than zj plus one half the number * sample 1 and sample 2.
* of values in this sample which are equal to zj. */
*/ if (equal_z_both_samples)
{
F1j = val_st_z_samp1 + (double) /* hj is the number of values in the combined sample
(val_eq_z_samp1 - val_st_z_samp1) / 2; * equal to zj
*/
hj = val_eq_z_samp1 - val_st_z_samp1
+ val_eq_z_samp2 - val_st_z_samp2;
/* F2j is the number of values in the 1st sample /* H_j is the number of values in the combined sample
* which are less than zj plus one half the number * smaller than zj plus one half the the number of
* of values in this sample which are equal to zj. * values in the combined sample equal to zj
*/ * (that's hj/2).
F2j = val_st_z_samp2 + (double) */
(val_eq_z_samp2 - val_st_z_samp2) / 2; H_j = val_st_z_samp1 + val_st_z_samp2
+ hj / 2;
/* set the line of values equal to zj to the /* F1j is the number of values in the 1st sample
* actual line of the last value picked for zj. * which are less than zj plus one half the number
*/ * of values in this sample which are equal to zj.
val_st_z_samp1 = val_eq_z_samp1; */
/* Set the line of values equal to zj to the actual F1j = val_st_z_samp1 + (double)
* line of the last value picked for zjof each (val_eq_z_samp1 - val_st_z_samp1) / 2;
* sample. This is required as data smaller than zj
* is accounted differently than values equal to zj.
*/ /* F2j is the number of values in the 1st sample
val_st_z_samp2 = val_eq_z_samp2; * which are less than zj plus one half the number
* of values in this sample which are equal to zj.
*/
F2j = val_st_z_samp2 + (double)
(val_eq_z_samp2 - val_st_z_samp2) / 2;
/* next the lines of the next values z, ie. zj+1 /* set the line of values equal to zj to the
* are addressed. * actual line of the last value picked for zj.
*/ */
val_eq_z_samp1++; val_st_z_samp1 = val_eq_z_samp1;
/* next the lines of the next values z, ie. /* Set the line of values equal to zj to the actual
* zj+1 are addressed * line of the last value picked for zjof each
*/ * sample. This is required as data smaller than zj
val_eq_z_samp2++; * is accounted differently than values equal to zj.
} */
else val_st_z_samp2 = val_eq_z_samp2;
{
/* the smaller z value was contained in sample 2, /* next the lines of the next values z, ie. zj+1
* hence this value is the zj to base the following * are addressed.
* calculations on. */
*/ val_eq_z_samp1++;
if (next_z_sample2)
{
/* hj is the number of values in the combined /* next the lines of the next values z, ie.
* sample equal to zj, in this case these are * zj+1 are addressed
* within sample 2 only. */
*/ val_eq_z_samp2++;
hj = val_eq_z_samp2 - val_st_z_samp2; }
else
{
/* H_j is the number of values in the combined sample /* the smaller z value was contained in sample 2,
* smaller than zj plus one half the the number of * hence this value is the zj to base the following
* values in the combined sample equal to zj * calculations on.
* (that's hj/2). */
*/ if (next_z_sample2)
{
H_j = val_st_z_samp1 + val_st_z_samp2 /* hj is the number of values in the combined
+ hj / 2; * sample equal to zj, in this case these are
* within sample 2 only.
*/
hj = val_eq_z_samp2 - val_st_z_samp2;
/* F1j is the number of values in the 1st sample which /* H_j is the number of values in the combined sample
* are less than zj plus one half the number of values in * smaller than zj plus one half the the number of
* this sample which are equal to zj. * values in the combined sample equal to zj
* As val_eq_z_samp2 < val_eq_z_samp1, these are the * (that's hj/2).
* val_st_z_samp1 only. */
*/
F1j = val_st_z_samp1;
/* F2j is the number of values in the 1st sample which H_j = val_st_z_samp1 + val_st_z_samp2
* are less than zj plus one half the number of values in + hj / 2;
* this sample which are equal to zj. The latter are from
* sample 2 only in this case.
*/
F2j = val_st_z_samp2 + (double) /* F1j is the number of values in the 1st sample which
(val_eq_z_samp2 - val_st_z_samp2) / 2; * are less than zj plus one half the number of values in
* this sample which are equal to zj.
* As val_eq_z_samp2 < val_eq_z_samp1, these are the
* val_st_z_samp1 only.
*/
F1j = val_st_z_samp1;
/* Set the line of values equal to zj to the actual line /* F2j is the number of values in the 1st sample which
* of the last value picked for zj of sample 2 only in * are less than zj plus one half the number of values in
* this case. * this sample which are equal to zj. The latter are from
*/ * sample 2 only in this case.
val_st_z_samp2 = val_eq_z_samp2; */
/* next the line of the next value z, ie. zj+1 is F2j = val_st_z_samp2 + (double)
* addressed. Here, only sample 2 must be addressed. (val_eq_z_samp2 - val_st_z_samp2) / 2;
*/
val_eq_z_samp2++; /* Set the line of values equal to zj to the actual line
if (val_eq_z_samp1 == 0) * of the last value picked for zj of sample 2 only in
{ * this case.
val_eq_z_samp1 = stop_loop1; */
} val_st_z_samp2 = val_eq_z_samp2;
}
/* the smaller z value was contained in sample 2, /* next the line of the next value z, ie. zj+1 is
* hence this value is the zj to base the following * addressed. Here, only sample 2 must be addressed.
* calculations on. */
*/
else val_eq_z_samp2++;
{ if (val_eq_z_samp1 == 0)
{
val_eq_z_samp1 = stop_loop1;
}
}
/* hj is the number of values in the combined /* the smaller z value was contained in sample 2,
* sample equal to zj, in this case these are * hence this value is the zj to base the following
* within sample 1 only. * calculations on.
*/ */
hj = val_eq_z_samp1 - val_st_z_samp1;
/* H_j is the number of values in the combined else
* sample smaller than zj plus one half the the number {
* of values in the combined sample equal to zj
* (that's hj/2).
*/
H_j = val_st_z_samp1 + val_st_z_samp2 /* hj is the number of values in the combined
+ hj / 2; * sample equal to zj, in this case these are
* within sample 1 only.
*/
hj = val_eq_z_samp1 - val_st_z_samp1;
/* F1j is the number of values in the 1st sample which /* H_j is the number of values in the combined
* are less than zj plus, in this case these are within * sample smaller than zj plus one half the the number
* sample 1 only one half the number of values in this * of values in the combined sample equal to zj
* sample which are equal to zj. The latter are from * (that's hj/2).
* sample 1 only in this case. */
*/
F1j = val_st_z_samp1 + (double) H_j = val_st_z_samp1 + val_st_z_samp2
(val_eq_z_samp1 - val_st_z_samp1) / 2; + hj / 2;
/* F2j is the number of values in the 1st sample which /* F1j is the number of values in the 1st sample which
* are less than zj plus one half the number of values * are less than zj plus, in this case these are within
* in this sample which are equal to zj. As * sample 1 only one half the number of values in this
* val_eq_z_samp1 < val_eq_z_samp2, these are the * sample which are equal to zj. The latter are from
* val_st_z_samp2 only. * sample 1 only in this case.
*/ */
F2j = val_st_z_samp2; F1j = val_st_z_samp1 + (double)
(val_eq_z_samp1 - val_st_z_samp1) / 2;
/* Set the line of values equal to zj to the actual line /* F2j is the number of values in the 1st sample which
* of the last value picked for zj of sample 1 only in * are less than zj plus one half the number of values
* this case * in this sample which are equal to zj. As
*/ * val_eq_z_samp1 < val_eq_z_samp2, these are the
* val_st_z_samp2 only.
*/
val_st_z_samp1 = val_eq_z_samp1; F2j = val_st_z_samp2;
/* next the line of the next value z, ie. zj+1 is /* Set the line of values equal to zj to the actual line
* addressed. Here, only sample 1 must be addressed. * of the last value picked for zj of sample 1 only in
*/ * this case
val_eq_z_samp1++; */
if (val_eq_z_samp2 == 0) val_st_z_samp1 = val_eq_z_samp1;
{
val_eq_z_samp2 = stop_loop2;
}
}
}
denom_1_aux = n_total * F1j - n_sample1 * H_j; /* next the line of the next value z, ie. zj+1 is
denom_2_aux = n_total * F2j - n_sample2 * H_j; * addressed. Here, only sample 1 must be addressed.
*/
val_eq_z_samp1++;
sum_adk_samp1 = sum_adk_samp1 + hj if (val_eq_z_samp2 == 0)
* (denom_1_aux * denom_1_aux) / {
(H_j * (n_total - H_j) val_eq_z_samp2 = stop_loop2;
- n_total * hj / 4); }
sum_adk_samp2 = sum_adk_samp2 + hj }
* (denom_2_aux * denom_2_aux) / }
(H_j * (n_total - H_j)
- n_total * hj / 4);
next_z_sample2 = false;
equal_z_both_samples = false;
/* index to count the z. It is only required to prevent denom_1_aux = n_total * F1j - n_sample1 * H_j;
* the while slope to execute endless denom_2_aux = n_total * F2j - n_sample2 * H_j;
*/
j++;
}
// calculating the adk value is the final step. sum_adk_samp1 = sum_adk_samp1 + hj
adk_result = (double) (n_total - 1) / (n_total * (denom_1_aux * denom_1_aux) /
* n_total * (k - 1)) (H_j * (n_total - H_j)
* (sum_adk_samp1 / n_sample1 - n_total * hj / 4);
+ sum_adk_samp2 / n_sample2); sum_adk_samp2 = sum_adk_samp2 + hj
* (denom_2_aux * denom_2_aux) /
(H_j * (n_total - H_j)
- n_total * hj / 4);
/* if(adk_result <= adk_criterium) next_z_sample2 = false;
* adk_2_sample test is passed equal_z_both_samples = false;
*/
return adk_result <= adk_criterium; /* index to count the z. It is only required to prevent
} * the while slope to execute endless
*/
j++;
}
// calculating the adk value is the final step.
adk_result = (double) (n_total - 1) / (n_total
* n_total * (k - 1))
* (sum_adk_samp1 / n_sample1
+ sum_adk_samp2 / n_sample2);
/* if(adk_result <= adk_criterium)
* adk_2_sample test is passed
*/
return adk_result <= adk_criterium;
}
Figure 5 Figure 5
Appendix C. Glossary Appendix C. Glossary
+-------------+-----------------------------------------------------+ +-------------+-----------------------------------------------------+
| ADK | Anderson-Darling K-Sample test, a test used to | | ADK | Anderson-Darling K-Sample test, a test used to |
| | check whether two samples have the same statistical | | | check whether two samples have the same statistical |
| | distribution. | | | distribution. |
| ECMP | Equal Cost Multipath, a load balancing mechanism | | ECMP | Equal Cost Multipath, a load balancing mechanism |
 End of changes. 129 change blocks. 
575 lines changed or deleted 498 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/