draft-ietf-ippm-metrictest-04.txt | draft-ietf-ippm-metrictest-05.txt | |||
---|---|---|---|---|
Internet Engineering Task Force R. Geib, Ed. | Internet Engineering Task Force R. Geib, Ed. | |||
Internet-Draft Deutsche Telekom | Internet-Draft Deutsche Telekom | |||
Intended status: Standards Track A. Morton | Intended status: BCP A. Morton | |||
Expires: April 26, 2012 AT&T Labs | Expires: June 1, 2012 AT&T Labs | |||
R. Fardid | R. Fardid | |||
Cariden Technologies | Cariden Technologies | |||
A. Steinmitz | A. Steinmitz | |||
Deutsche Telekom | Deutsche Telekom | |||
October 24, 2011 | November 29, 2011 | |||
IPPM standard advancement testing | IPPM standard advancement testing | |||
draft-ietf-ippm-metrictest-04 | draft-ietf-ippm-metrictest-05 | |||
Abstract | Abstract | |||
This document specifies tests to determine if multiple independent | This document specifies tests to determine if multiple independent | |||
instantiations of a performance metric RFC have implemented the | instantiations of a performance metric RFC have implemented the | |||
specifications in the same way. This is the performance metric | specifications in the same way. This is the performance metric | |||
equivalent of interoperability, required to advance RFCs along the | equivalent of interoperability, required to advance RFCs along the | |||
standards track. Results from different implementations of metric | standards track. Results from different implementations of metric | |||
RFCs will be collected under the same underlying network conditions | RFCs will be collected under the same underlying network conditions | |||
and compared using state of the art statistical methods. The goal is | and compared using statistical methods. The goal is an evaluation of | |||
an evaluation of the metric RFC itself, whether its definitions are | the metric RFC itself; whether its definitions are clear and | |||
clear and unambiguous to implementors and therefore a candidate for | unambiguous to implementors and therefore a candidate for advancement | |||
advancement on the IETF standards track. | on the IETF standards track. This document is an Internet Best | |||
Current Practice. | ||||
Status of this Memo | Status of this Memo | |||
This Internet-Draft is submitted in full conformance with the | This Internet-Draft is submitted in full conformance with the | |||
provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
Drafts is at http://datatracker.ietf.org/drafts/current/. | Drafts is at http://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
This Internet-Draft will expire on April 26, 2012. | This Internet-Draft will expire on June 1, 2012. | |||
Copyright Notice | Copyright Notice | |||
Copyright (c) 2011 IETF Trust and the persons identified as the | Copyright (c) 2011 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(http://trustee.ietf.org/license-info) in effect on the date of | (http://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
described in the Simplified BSD License. | described in the Simplified BSD License. | |||
Table of Contents | Table of Contents | |||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 7 | 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 | |||
2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 7 | 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 5 | |||
3. Verification of conformance to a metric specification . . . . 8 | 3. Verification of conformance to a metric specification . . . . 7 | |||
3.1. Tests of an individual implementation against a metric | 3.1. Tests of an individual implementation against a metric | |||
specification . . . . . . . . . . . . . . . . . . . . . . 9 | specification . . . . . . . . . . . . . . . . . . . . . . 8 | |||
3.2. Test setup resulting in identical live network testing | 3.2. Test setup resulting in identical live network testing | |||
conditions . . . . . . . . . . . . . . . . . . . . . . . . 11 | conditions . . . . . . . . . . . . . . . . . . . . . . . . 9 | |||
3.3. Tests of two or more different implementations against | 3.3. Tests of two or more different implementations against | |||
a metric specification . . . . . . . . . . . . . . . . . . 16 | a metric specification . . . . . . . . . . . . . . . . . . 15 | |||
3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 17 | 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 16 | |||
3.5. Recommended Metric Verification Measurement Process . . . 18 | 3.5. Recommended Metric Verification Measurement Process . . . 17 | |||
3.6. Proposal to determine an "equivalence" threshold for | 3.6. Proposal to determine an "equivalence" threshold for | |||
each metric evaluated . . . . . . . . . . . . . . . . . . 21 | each metric evaluated . . . . . . . . . . . . . . . . . . 20 | |||
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 | 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 22 | 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 | 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 | |||
7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 | 7. Security Considerations . . . . . . . . . . . . . . . . . . . 21 | |||
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 | 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 | |||
8.1. Normative References . . . . . . . . . . . . . . . . . . . 23 | 8.1. Normative References . . . . . . . . . . . . . . . . . . . 21 | |||
8.2. Informative References . . . . . . . . . . . . . . . . . . 24 | 8.2. Informative References . . . . . . . . . . . . . . . . . . 22 | |||
Appendix A. An example on a One-way Delay metric validation . . . 25 | Appendix A. An example on a One-way Delay metric validation . . . 23 | |||
A.1. Compliance to Metric specification requirements . . . . . 25 | A.1. Compliance to Metric specification requirements . . . . . 23 | |||
A.2. Examples related to statistical tests for One-way Delay . 27 | A.2. Examples related to statistical tests for One-way Delay . 25 | |||
Appendix B. Anderson-Darling K-sample Reference and 2 sample | Appendix B. Anderson-Darling K-sample Reference and 2 sample | |||
C++ code . . . . . . . . . . . . . . . . . . . . . . 29 | C++ code . . . . . . . . . . . . . . . . . . . . . . 27 | |||
Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 37 | Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 36 | |||
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 | |||
1. Introduction | 1. Introduction | |||
The Internet Standards Process RFC2026 [RFC2026] requires that for a | The Internet Standards Process as updated by RFC6410 [RFC6410] | |||
IETF specification to advance beyond the Proposed Standard level, at | specifies that widespread deployment and use is sufficient to show | |||
least two genetically unrelated implementations must be shown to | interoperability as a condition for advancement to Internet Standard. | |||
interoperate correctly with all features and options. This | The previous requirement of interoperability tests prior to advancing | |||
requirement can be met by supplying: | an RFC to the Standard maturity level specified in RFC 2026 [RFC2026] | |||
and RFC 5657 [RFC5657] has been removed. While the modified | ||||
o evidence that (at least a sub-set of) the specification has been | requirement is applicable to protocols, wide deployment of different | |||
implemented by multiple parties, thus indicating adoption by the | measurement systems does not prove that the implementations measure | |||
IETF community and the extent of feature coverage. | metrics in a standard way. Section 5.3 of RFC 5657 [RFC5657] | |||
explicitly mentions the special case of Standards that are not "on- | ||||
o evidence that each feature of the specification is sufficiently | the-wire" protocols. While this special case is not explicitly | |||
well-described to support interoperability, as demonstrated | mentioned by RFC6410 [RFC6410], the four criteria in Section 2.2 of | |||
through testing and/or user experience with deployment. | RFC6410 [RFC6410] are augmented by this document for RFCs that | |||
specify performance metrics. This document takes the position that | ||||
flexible metric definitions can be proven to be clear and unambiguous | ||||
through tests that compare the results from independent | ||||
implementations. It describes tests which infer whether metric | ||||
specifications are sufficient using a definition of metric | ||||
"interoperability": measuring equivalent results (in a statistical | ||||
sense) under the same network conditions. The document expands on | ||||
this problem and its solution below. | ||||
In the case of a protocol specification, the notion of | In the case of a protocol specification, the notion of | |||
"interoperability" is reasonably intuitive - the implementations must | "interoperability" is reasonably intuitive - the implementations must | |||
successfully "talk to each other", while exercising all features and | successfully "talk to each other", while exercising all features and | |||
options. To achieve interoperability, two implementors need to | options. To achieve interoperability, two implementors need to | |||
interpret the protocol specifications in equivalent ways. In the | interpret the protocol specifications in equivalent ways. In the | |||
case of IP Performance Metrics (IPPM), this definition of | case of IP Performance Metrics (IPPM), this definition of | |||
interoperability is only useful for test and control protocols like | interoperability is only useful for test and control protocols like | |||
the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the | the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the | |||
Two-Way Active Measurement Protocol, TWAMP [RFC5357]. | Two-Way Active Measurement Protocol, TWAMP [RFC5357]. | |||
skipping to change at page 4, line 7 | skipping to change at page 4, line 15 | |||
Since many implementations of IP metrics are embedded in measurement | Since many implementations of IP metrics are embedded in measurement | |||
systems that do not interact with one another (they were built before | systems that do not interact with one another (they were built before | |||
OWAMP and TWAMP), the interoperability evaluation called for in the | OWAMP and TWAMP), the interoperability evaluation called for in the | |||
IETF standards process cannot be determined by observing that | IETF standards process cannot be determined by observing that | |||
independent implementations interact properly for various protocol | independent implementations interact properly for various protocol | |||
exchanges. Instead, verifying that different implementations give | exchanges. Instead, verifying that different implementations give | |||
statistically equivalent results under controlled measurement | statistically equivalent results under controlled measurement | |||
conditions takes the place of interoperability observations. Even | conditions takes the place of interoperability observations. Even | |||
when evaluating OWAMP and TWAMP RFCs for standards track advancement, | when evaluating OWAMP and TWAMP RFCs for standards track advancement, | |||
the methods described here are useful to evaluate the measurement | the methods described here are useful to evaluate the measurement | |||
results because their validity would not be ascertained in typical | results because their validity would not be ascertained in protocol | |||
interoperability testing. | interoperability testing. | |||
The standards advancement process aims at producing confidence that | The standards advancement process aims at producing confidence that | |||
the metric definitions and supporting material are clearly worded and | the metric definitions and supporting material are clearly worded and | |||
unambiguous, or reveals ways in which the metric definitions can be | unambiguous, or reveals ways in which the metric definitions can be | |||
revised to achieve clarity. The process also permits identification | revised to achieve clarity. The process also permits identification | |||
of options that were not implemented, so that they can be removed | of options that were not implemented, so that they can be removed | |||
from the advancing specification. Thus, the product of this process | from the advancing specification. Thus, the product of this process | |||
is information about the metric specification RFC itself: | is information about the metric specification RFC itself: | |||
determination of the specifications or definitions that are clear and | determination of the specifications or definitions that are clear and | |||
skipping to change at page 4, line 43 | skipping to change at page 4, line 51 | |||
Conclusions on equivalence are reached by two measures. | Conclusions on equivalence are reached by two measures. | |||
First, implementations are compared against individual metric | First, implementations are compared against individual metric | |||
specifications to make sure that differences in implementation are | specifications to make sure that differences in implementation are | |||
minimised or at least known. | minimised or at least known. | |||
Second, a test setup is proposed ensuring identical networking | Second, a test setup is proposed ensuring identical networking | |||
conditions so that unknowns are minimized and comparisons are | conditions so that unknowns are minimized and comparisons are | |||
simplified. The resulting separate data sets may be seen as samples | simplified. The resulting separate data sets may be seen as samples | |||
taken from the same underlying distribution. Using state of the art | taken from the same underlying distribution. Using statistical | |||
statistical methods, the equivalence of the results is verified. To | methods, the equivalence of the results is verified. To illustrate | |||
illustrate application of the process and methods defined here, | application of the process and methods defined here, evaluation of | |||
evaluation of the One-way Delay Metric [RFC2679] is provided in an | the One-way Delay Metric [RFC2679] is provided in an Appendix. While | |||
Appendix. While test setups will vary with the metrics to be | test setups will vary with the metrics to be validated, the general | |||
validated, the general methodology of determining equivalent results | methodology of determining equivalent results will not. Documents | |||
will not. Documents defining test setups to evaluate other metrics | defining test setups to evaluate other metrics should be developed | |||
should be developed once the process proposed here has been agreed | once the process proposed here has been agreed and approved. | |||
and approved. | ||||
The metric RFC advancement process begins with a request for protocol | The metric RFC advancement process begins with a request for protocol | |||
action accompanied by a memo that documents the supporting tests and | action accompanied by a memo that documents the supporting tests and | |||
results. The procedures of [RFC2026] are expanded in[RFC5657], | results. The procedures of [RFC2026] are expanded in[RFC5657], | |||
including sample implementation and interoperability reports. | including sample implementation and interoperability reports. | |||
Section 3 of [morton-advance-metrics-01] can serve as a template for | [morton-testplan-rfc2679] can serve as a template for a metric RFC | |||
a metric RFC report which accompanies the protocol action request to | report which accompanies the protocol action request to the Area | |||
the Area Director, including description of the test set-up, | Director, including description of the test set-up, procedures, | |||
procedures, results for each implementation and conclusions. | results for each implementation and conclusions. | |||
Changes from WG-03 to WG-04: | ||||
o Revisions to Appendix B code and add reference to "R" in the | ||||
Appendix and the text of section 3.6. | ||||
Changes from WG-02 to WG-03: | ||||
o Changes stemming from experiments that implemented this plan, in | ||||
general. | ||||
o Adoption of the VLAN loopback figure in the main body of the memo | ||||
(section 3.2). | ||||
Changes from WG-01 to WG-02: | ||||
o Clarification of the number of test streams recommended in section | ||||
3.2. | ||||
o Clarifications on testing details in sections 3.3 and 3.4. | ||||
o Spelling corrections throughout. | ||||
Changes from WG -00 to WG -01 draft | ||||
o Discussion on merits and requirements of a distributed lab test | ||||
using only local load generators. | ||||
o Proposal of metrics suitable for tests using the proposed | ||||
measurement configuration. | ||||
o Hint on delay caused by software based L2TPv3 implementation. | ||||
o Added an appendix with a test configuration allowing remote tests | ||||
comparing different implementations across the network. | ||||
o Proposal for maximum error of "equivalence", based on performance | ||||
comparison of identical implementations. This may be useful for | ||||
both ADK and non-ADK comparisons. | ||||
Changes from prior ID -02 to WG -00 draft | ||||
o Incorporation of aspects of reporting to support the protocol | ||||
action request in the Introduction and section 3.5 | ||||
o Overhaul of section 3.2 regarding tunneling: Added generic | ||||
tunneling requirements and L2TPv3 as an example tunneling | ||||
mechanism fulfilling the tunneling requirements. Removed and | ||||
adapted some of the prior references to other tunneling protocols | ||||
o Softened a requirement within section 3.4 (MUST to SHOULD on | ||||
precision) and removed some comments of the authors. | ||||
o Updated contact information of one author and added a new author. | ||||
o Added example C++ code of an Anderson-Darling two sample test | ||||
implementation. | ||||
Changes from ID -01 to ID -02 version | ||||
o Major editorial review, rewording and clarifications on all | ||||
contents. | ||||
o Additional text on parallel testing using VLANs and GRE or | ||||
Pseudowire tunnels. | ||||
o Additional examples and a glossary. | ||||
Changes from ID -00 to ID -01 version | ||||
o Addition of a comparison of individual metric implementations | ||||
against the metric specification (trying to pick up problems and | ||||
solutions for metric advancement [morton-advance-metrics]). | ||||
o More emphasis on the requirement to carefully design and document | ||||
the measurement setup of the metric comparison. | ||||
o Proposal of testing conditions under identical WAN network | ||||
conditions using IP in IP tunneling or Pseudo Wires and parallel | ||||
measurement streams. | ||||
o Proposing the requirement to document the smallest resolution at | ||||
which an ADK test was passed by 95%. As no minimum resolution is | ||||
specified, IPPM metric compliance is not linked to a particular | ||||
performance of an implementation. | ||||
o Reference to RFC 2330 and RFC 2679 for the 95% confidence interval | ||||
as preferred criterion to decide on statistical equivalence | ||||
o Reducing the proposed statistical test to ADK with 95% confidence. | ||||
1.1. Requirements Language | 1.1. Requirements Language | |||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", | |||
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this | |||
document are to be interpreted as described in RFC 2119 [RFC2119]. | document are to be interpreted as described in RFC 2119 [RFC2119]. | |||
2. Basic idea | 2. Basic idea | |||
The implementation of a standard compliant metric is expected to meet | The implementation of a standard compliant metric is expected to meet | |||
the requirements of the related metric specification. So before | the requirements of the related metric specification. So before | |||
comparing two metric implementations, each metric implementation is | comparing two metric implementations, each metric implementation is | |||
individually compared against the metric specification. | individually compared against the metric specification. | |||
Most metric specifications leave freedom to implementors on non- | Most metric specifications leave freedom to implementors on non- | |||
fundamental aspects of an individual metric (or options). Comparing | fundamental aspects of an individual metric (or options). Comparing | |||
different measurement results using a statistical test with the | different measurement results using a statistical test with the | |||
assumption of identical test path and testing conditions requires | assumption of identical test path and testing conditions requires | |||
knowledge of all differences in the overall test setup. Metric | knowledge of all differences in the overall test setup. Metric | |||
specification options chosen by implementors have to be documented. | specification options chosen by implementors have to be documented. | |||
It is REQUIRED to use identical implementation options wherever | It is RECOMMENDED to use identical metric options for any test | |||
possible for any test proposed here. Calibrations proposed by metric | proposed here (an exception would be if a variable parameter of the | |||
standards should be performed to further identify (and possibly | metric definition is not configurable in one or more | |||
reduce) potential sources of errors in the test setup. | implementations). Calibrations specified by metric standards SHOULD | |||
be performed to further identify (and possibly reduce) potential | ||||
sources of error in the test setup. | ||||
The Framework for IP Performance Metrics [RFC2330] expects that a | The Framework for IP Performance Metrics [RFC2330] expects that a | |||
"methodology for a metric should have the property that it is | "methodology for a metric should have the property that it is | |||
repeatable: if the methodology is used multiple times under identical | repeatable: if the methodology is used multiple times under identical | |||
conditions, it should result in consistent measurements." This means | conditions, it should result in consistent measurements." This means | |||
an implementation is expected to repeatedly measure a metric with | an implementation is expected to repeatedly measure a metric with | |||
consistent results (repeatability with the same result). Small | consistent results (repeatability with the same result). Small | |||
deviations in the test setup are expected to lead to small deviations | deviations in the test setup are expected to lead to small deviations | |||
in results only. To characterise statistical equivalence in the case | in results only. To characterise statistical equivalence in the case | |||
of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% | of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% | |||
skipping to change at page 8, line 17 | skipping to change at page 6, line 33 | |||
measurement setup on the result, network conditions and paths MUST | measurement setup on the result, network conditions and paths MUST | |||
be identical for the compared implementations to the largest | be identical for the compared implementations to the largest | |||
possible degree. This includes both the stability and non- | possible degree. This includes both the stability and non- | |||
ambiguity of routes taken by the measurement packets. See RFC | ambiguity of routes taken by the measurement packets. See RFC | |||
2330 for a discussion on self-consistency. | 2330 for a discussion on self-consistency. | |||
o To minimize the influence of implementation options on the result, | o To minimize the influence of implementation options on the result, | |||
metric implementations SHOULD use identical options and parameters | metric implementations SHOULD use identical options and parameters | |||
for the metric under evaluation. | for the metric under evaluation. | |||
o The error induced by the sample size must be small enough to | o The sample size must be large enough to minimize its influence on | |||
minimize its influence on the test result. This may have to be | the consistency of the test results. This consideration may be | |||
respected, especially if two implementations measure with | especially important if two implementations measure with different | |||
different average probing rates. | average packet transmission rates. | |||
o The implementation with the lowest probing frequency determines | o The implementation with the lowest average packet transmission | |||
the smallest temporal interval for which samples can be compared. | rate determines the smallest temporal interval for which samples | |||
can be compared. | ||||
o Repeat comparisons with several independent metric samples to | o Repeat comparisons with several independent metric samples to | |||
avoid random indications of compatibility (or the lack of it). | avoid random indications of compatibility (or the lack of it). | |||
The metric specifications themselves are the primary focus of | The metric specifications themselves are the primary focus of | |||
evaluation, rather than the implementations of metrics. The | evaluation, rather than the implementations of metrics. The | |||
documentation produced by the advancement process should identify | documentation produced by the advancement process should identify | |||
which metric definitions and supporting material were found to be | which metric definitions and supporting material were found to be | |||
clearly worded and unambiguous, OR, it should identify ways in which | clearly worded and unambiguous, OR, it should identify ways in which | |||
the metric specification text should be revised to achieve clarity | the metric specification text should be revised to achieve clarity | |||
skipping to change at page 9, line 30 | skipping to change at page 7, line 47 | |||
impairment generators. | impairment generators. | |||
o Use remotely separated test labs to compare the implementations | o Use remotely separated test labs to compare the implementations | |||
and measure across the Internet. | and measure across the Internet. | |||
o Use remotely separated test labs to compare the implementations | o Use remotely separated test labs to compare the implementations | |||
and measure across the Internet and include a single impairment | and measure across the Internet and include a single impairment | |||
generator to impact all measurement flows in non discriminatory | generator to impact all measurement flows in non discriminatory | |||
way. | way. | |||
The first two approaches work, but cause higher expenses than the | The first two approaches work, but involve higher expenses than the | |||
other ones (due to travel and/or shipping+installation). For the | others (due to travel and/or shipping plus installation). For the | |||
third option, ensuring two identically configured impairment | third option, ensuring two identically configured impairment | |||
generators requires well defined test cases and possibly identical | generators requires well defined test cases and possibly identical | |||
hard- and software. | hardware and software. | |||
As documented in a test report [morton-testplan-rfc2679], the last | As documented in a test report [morton-testplan-rfc2679], the last | |||
option was required to prove compatibility of two delay metric | option was required to prove compatibility of two delay metric | |||
implementations. An impairment generator is probably required when | implementations. An impairment generator is probably required when | |||
testing compatibility of most other metrics and it therefore | testing compatibility of most other metrics and it is therefore | |||
RECOMMENDED to include an impairment generator in metric test set | RECOMMENDED to include an impairment generator in metric test setups. | |||
ups. | ||||
3.1. Tests of an individual implementation against a metric | 3.1. Tests of an individual implementation against a metric | |||
specification | specification | |||
A metric implementation MUST support the requirements classified as | A metric implementation is compliant with a metric specification if | |||
"MUST" and "REQUIRED" of the related metric specification to be | it supports the requirements classified as "MUST" and "REQUIRED" of | |||
compliant to the latter. | the related metric specification. An implementation that implements | |||
all requirements is fully compliant with the specification, and the | ||||
degree of compliance SHOULD be noted in the conclusions of the | ||||
report. | ||||
Further, supported options of a metric implementation SHOULD be | Further, supported options of a metric implementation SHOULD be | |||
documented in sufficient detail. The documentation of chosen options | documented in sufficient detail to evaluate whether the specification | |||
is RECOMMENDED to minimise (and recognise) differences in the test | was correctly interpreted. The documentation of chosen options | |||
setup if two metric implementations are compared. Further, this | should minimise (and recognise) differences in the test setup if two | |||
documentation is used to validate and improve the underlying metric | metric implementations are compared. Further, this documentation is | |||
specification option, to remove options which saw no implementation | used to validate or clarify the wording of the metric specification | |||
or which are badly specified from the metric specification to be | option, to remove options which saw no implementation or which are | |||
promoted to a standard. This documentation SHOULD be made for all | badly specified from the metric specification. This documentation | |||
implementation-relevant specifications of a metric picked for a | SHOULD be included for all implementation-relevant specifications of | |||
comparison that are not explicitly marked as "MUST" or "REQUIRED" in | a metric picked for a comparison, even those that are not explicitly | |||
the RFC text. This applies for the following sections of all metric | marked as "MUST" or "REQUIRED" in the RFC text. This applies for the | |||
specifications: | following sections of all metric specifications: | |||
o Singleton Definition of the Metric. | o Singleton Definition of the Metric. | |||
o Sample Definition of the Metric. | o Sample Definition of the Metric. | |||
o Statistics Definition of the Metric. As statistics are compared | o Statistics Definition of the Metric. As statistics are compared | |||
by the test specified here, this documentation is required even in | by the test specified here, this documentation is required even in | |||
the case, that the metric specification does not contain a | the case, that the metric specification does not contain a | |||
Statistics Definition. | Statistics Definition. | |||
o Timing and Synchronisation related specification (if relevant for | o Timing and Synchronisation related specification (if relevant for | |||
the Metric). | the Metric). | |||
o Any other technical part present or missing in the metric | o Any other technical part present or missing in the metric | |||
specification, which is relevant for the implementation of the | specification, which is relevant for the implementation of the | |||
Metric. | Metric. | |||
RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric | RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric | |||
implementations. A single IPPM conformant implementation MUST under | implementations. A single IPPM conforming implementation should | |||
otherwise identical network conditions produce precise results for | under otherwise identical network conditions produce precise results | |||
repeated measurements of the same metric. | for repeated measurements of the same metric. | |||
RFC 2330 prefers the "empirical distribution function" EDF to | RFC 2330 prefers the "empirical distribution function" EDF to | |||
describe collections of measurements. RFC 2330 determines, that | describe collections of measurements. RFC 2330 determines, that | |||
"unless otherwise stated, IPPM goodness-of-fit tests are done using | "unless otherwise stated, IPPM goodness-of-fit tests are done using | |||
5% significance." The goodness of fit test determines by which | 5% significance." The goodness of fit test determines by which | |||
precision two or more samples of a metric implementation belong to | precision two or more samples of a metric implementation belong to | |||
the same underlying distribution (of measured network performance | the same underlying distribution (of measured network performance | |||
events). The goodness of fit test suggested for the metric test is | events). The goodness of fit test suggested for the metric test is | |||
the Anderson-Darling K sample test (ADK sample test, K stands for the | the Anderson-Darling K sample test (ADK sample test, K stands for the | |||
number of samples to be compared) [ADK]. Please note that RFC 2330 | number of samples to be compared) [ADK]. Please note that RFC 2330 | |||
skipping to change at page 11, line 9 | skipping to change at page 9, line 27 | |||
The results of a repeated test with a single implementation MUST pass | The results of a repeated test with a single implementation MUST pass | |||
an ADK sample test with confidence level of 95%. The conditions for | an ADK sample test with confidence level of 95%. The conditions for | |||
which the ADK test has been passed with the specified confidence | which the ADK test has been passed with the specified confidence | |||
level MUST be documented. To formulate this differently: The | level MUST be documented. To formulate this differently: The | |||
requirement is to document the set of parameters with the smallest | requirement is to document the set of parameters with the smallest | |||
deviation, at which the results of the tested metric implementation | deviation, at which the results of the tested metric implementation | |||
pass an ADK test with a confidence level of 95%. The minimum | pass an ADK test with a confidence level of 95%. The minimum | |||
resolution available in the reported results from each implementation | resolution available in the reported results from each implementation | |||
MUST be taken into account in the ADK test. | MUST be taken into account in the ADK test. | |||
The test conditions which MUST be documented for a passed metric test | The test conditions to be documented for a passed metric test | |||
include: | include: | |||
o The metric resolution at which a test was passed (e.g. the | o The metric resolution at which a test was passed (e.g. the | |||
resolution of timestamps) | resolution of timestamps) | |||
o The parameters modified by an impairment generator. | o The parameters modified by an impairment generator. | |||
o The impairment generator parameter settings. | o The impairment generator parameter settings. | |||
3.2. Test setup resulting in identical live network testing conditions | 3.2. Test setup resulting in identical live network testing conditions | |||
skipping to change at page 11, line 36 | skipping to change at page 10, line 6 | |||
mechanisms like those that achieve load balancing (see [RFC4928]). | mechanisms like those that achieve load balancing (see [RFC4928]). | |||
This section proposes two measures to deal with both issues. | This section proposes two measures to deal with both issues. | |||
Tunneling mechanisms can be used to avoid parallel processing of | Tunneling mechanisms can be used to avoid parallel processing of | |||
different flows in the network. Measuring by separate parallel probe | different flows in the network. Measuring by separate parallel probe | |||
flows results in repeated collection of data. If both measures are | flows results in repeated collection of data. If both measures are | |||
combined, WAN network conditions are identical for a number of | combined, WAN network conditions are identical for a number of | |||
independent measurement flows, no matter what the network conditions | independent measurement flows, no matter what the network conditions | |||
are in detail. | are in detail. | |||
Any measurement setup MUST be made to avoid the probing traffic | Any measurement setup must be made to avoid the probing traffic | |||
itself to impede the metric measurement. The created measurement | itself to impede the metric measurement. The created measurement | |||
load MUST NOT result in congestion at the access link connecting the | load must not result in congestion at the access link connecting the | |||
measurement implementation to the WAN. The created measurement load | measurement implementation to the WAN. The created measurement load | |||
MUST NOT overload the measurement implementation itself, e.g., by | must not overload the measurement implementation itself, e.g., by | |||
causing a high CPU load or by creating imprecisions due to internal | causing a high CPU load or by creating imprecisions due to internal | |||
transmit (receive respectively) probe packet collisions. | transmit (receive respectively) probe packet collisions. | |||
Tunneling multiple flows reaching a network element on a single | Tunneling multiple flows reaching a network element on a single | |||
physical port may allow to transmit all packets of the tunnel via the | physical port may allow to transmit all packets of the tunnel via the | |||
same path. Applying tunnels to avoid undesired influence of standard | same path. Applying tunnels to avoid undesired influence of standard | |||
routing for measurement purposes is a concept known from literature, | routing for measurement purposes is a concept known from literature, | |||
see e.g. GRE encapsulated multicast probing [GU+Duffield]. An | see e.g. GRE encapsulated multicast probing [GU-Duffield]. An | |||
existing IP in IP tunnel protocol can be applied to avoid Equal-Cost | existing IP in IP tunnel protocol can be applied to avoid Equal-Cost | |||
Multi-Path (ECMP) routing of different measurement streams if it | Multi-Path (ECMP) routing of different measurement streams if it | |||
meets the following criteria: | meets the following criteria: | |||
o Inner IP packets from different measurement implementations are | o Inner IP packets from different measurement implementations are | |||
mapped into a single tunnel with single outer IP origin and | mapped into a single tunnel with single outer IP origin and | |||
destination address as well as origin and destination port numbers | destination address as well as origin and destination port numbers | |||
which are identical for all packets. | which are identical for all packets. | |||
o An easily accessible commodity tunneling protocol allows to carry | o An easily accessible commodity tunneling protocol allows to carry | |||
skipping to change at page 14, line 11 | skipping to change at page 12, line 34 | |||
to B and in the reverse direction. The remote site VLANs are | to B and in the reverse direction. The remote site VLANs are | |||
U-bridged at the local site Ethernet switch. The measurement packets | U-bridged at the local site Ethernet switch. The measurement packets | |||
of site 1 travel tunnel A->B first, are U-bridged at site 2 and | of site 1 travel tunnel A->B first, are U-bridged at site 2 and | |||
travel tunnel B->A second. Measurement packets of site 2 travel | travel tunnel B->A second. Measurement packets of site 2 travel | |||
tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B | tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B | |||
second. So all measurement packets pass the same tunnel segments, | second. So all measurement packets pass the same tunnel segments, | |||
but in different segment order. | but in different segment order. | |||
If tunneling is applied, two tunnels MUST carry all test traffic in | If tunneling is applied, two tunnels MUST carry all test traffic in | |||
between the test site and the remote site. For example, if 802.1Q | between the test site and the remote site. For example, if 802.1Q | |||
Ethernet Virtual LANs (VLAN) are applied and the measurement streams | Virtual LANs (VLAN) are applied and the measurement streams are | |||
are carried in different VLANs, the IP tunnel or Pseudo Wires | carried in different VLANs, the IP tunnel or Pseudo Wires | |||
respectively MUST be set up in physical port mode to avoid set up of | respectively are set up in physical port mode to avoid set up of | |||
Pseudo Wires per VLAN (which may see different paths due to ECMP | Pseudo Wires per VLAN (which may see different paths due to ECMP | |||
routing), see RFC 4448. The remote router and the Ethernet switch | routing), see RFC 4448. The remote router and the Ethernet switch | |||
shown in figure 3 has to support 802.1Q in this set up. | shown in figure 3 has to support 802.1Q in this set up. | |||
The IP packet size of the metric implementation SHOULD be chosen | The IP packet size of the metric implementation SHOULD be chosen | |||
small enough to avoid fragmentation due to the added Ethernet and | small enough to avoid fragmentation due to the added Ethernet and | |||
tunnel headers. Otherwise, the impact of tunnel overhead on | tunnel headers. Otherwise, the impact of tunnel overhead on | |||
fragmentation and interface MTU size MUST be understood and taken | fragmentation and interface MTU size must be understood and taken | |||
into account (see [RFC4459]). | into account (see [RFC4459]). | |||
An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each | An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each | |||
containing measurement traffic of a single measurement system was | containing measurement traffic of a single measurement system was | |||
successfully applied when testing compatibility of two metric | successfully applied when testing compatibility of two metric | |||
implementations [morton-testplan-rfc2679]. | implementations [morton-testplan-rfc2679]. Ethernet over L2TPv3 | |||
[RFC4719] was picked for this test. | ||||
The following headers may have to be accounted for when calculating | The following headers may have to be accounted for when calculating | |||
total packet length, if VLANs and Ethernet over L2TPv3 tunnels are | total packet length, if VLANs and Ethernet over L2TPv3 tunnels are | |||
applied: | applied: | |||
o Ethernet 802.1Q: 22 Byte. | o Ethernet 802.1Q: 22 Byte. | |||
o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 | o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 | |||
Byte for L2TPv3 data messages over UDP. | Byte for L2TPv3 data messages over UDP. | |||
skipping to change at page 16, line 4 | skipping to change at page 14, line 32 | |||
with 4 samples even if a 2 sample test | with 4 samples even if a 2 sample test | |||
failed[morton-testplan-rfc2679]. | failed[morton-testplan-rfc2679]. | |||
Some additional guidelines to calculate and compare samples to | Some additional guidelines to calculate and compare samples to | |||
perform a metric test are: | perform a metric test are: | |||
o To compare different probes of a common underlying distribution in | o To compare different probes of a common underlying distribution in | |||
terms of metrics characterising a communication network requires | terms of metrics characterising a communication network requires | |||
to respect the temporal nature for which the assumption of common | to respect the temporal nature for which the assumption of common | |||
underlying distribution may hold. Any singletons or samples to be | underlying distribution may hold. Any singletons or samples to be | |||
compared MUST be captured within the same time interval. | compared must be captured within the same time interval. | |||
o If statistical events like rates are used to characterise measured | o If statistical events like rates are used to characterise measured | |||
metrics of a time-interval, its RECOMMENDED to pick as a minimum 5 | metrics of a time-interval, a minimum 5 singletons of a relevant | |||
singletons of a relevant metric to ensure a minimum confidence | metric should be picked to ensure a minimum confidence into the | |||
into the reported value. The error margin of the determined rate | reported value. The error margin of the determined rate depends | |||
depends on the number singletons (refer to statistical textbooks | on the number singletons (refer to statistical textbooks on | |||
on Student's t-test). As an example, any packet loss measurement | Student's t-test). As an example, any packet loss measurement | |||
interval to be compared with the results of another implementation | interval to be compared with the results of another implementation | |||
contains at least five lost packets to have some confidence that | contains at least five lost packets to have some confidence that | |||
the observed loss rate wasn't caused by a small number of random | the observed loss rate wasn't caused by a small number of random | |||
packet drops. | packet drops. | |||
o The minimum number of singletons or samples to be compared by an | o The minimum number of singletons or samples to be compared by an | |||
Anderson-Darling test SHOULD be 100 per tested metric | Anderson-Darling test should be 100 per tested metric | |||
implementation. Note that the Anderson-Darling test detects small | implementation. Note that the Anderson-Darling test detects small | |||
differences in distributions fairly well and will fail for high | differences in distributions fairly well and will fail for high | |||
number of compared results (RFC2330 mentions an example with 8192 | number of compared results (RFC2330 mentions an example with 8192 | |||
measurements where an Anderson-Darling test always failed). | measurements where an Anderson-Darling test always failed). | |||
o Generally, the Anderson-Darling test is sensitive to differences | o Generally, the Anderson-Darling test is sensitive to differences | |||
in the accuracy or bias associated with varying implementations or | in the accuracy or bias associated with varying implementations or | |||
test conditions. These dissimilarities may result in differing | test conditions. These dissimilarities may result in differing | |||
averages of samples to be compared. An example may be different | averages of samples to be compared. An example may be different | |||
packet sizes, resulting in a constant delay difference between | packet sizes, resulting in a constant delay difference between | |||
skipping to change at page 16, line 50 | skipping to change at page 15, line 31 | |||
precisely, for every positive epsilon, there exists a positive delta, | precisely, for every positive epsilon, there exists a positive delta, | |||
such that if two sets of conditions are within delta of each other, | such that if two sets of conditions are within delta of each other, | |||
then the resulting measurements will be within epsilon of each | then the resulting measurements will be within epsilon of each | |||
other." A small variation in conditions in the context of the metric | other." A small variation in conditions in the context of the metric | |||
test proposed here can be seen as different implementations measuring | test proposed here can be seen as different implementations measuring | |||
the same metric along the same path. | the same metric along the same path. | |||
IPPM metric specifications however allow for implementor options to | IPPM metric specifications however allow for implementor options to | |||
the largest possible degree. It cannot be expected that two | the largest possible degree. It cannot be expected that two | |||
implementors allow 100% identical options in their implementations. | implementors allow 100% identical options in their implementations. | |||
Testers SHOULD to the highest degree possible pick the same | Testers SHOULD pick the same metric measurement configurations for | |||
configurations for their systems when comparing their implementations | their systems when comparing their implementations by a metric test. | |||
by a metric test. | ||||
In some cases, a goodness of fit test may not be possible or show | In some cases, a goodness of fit test may not be possible or show | |||
disappointing results. To clarify the difficulties arising from | disappointing results. To clarify the difficulties arising from | |||
different implementation options, the individual options picked for | different metric implementation options, the individual options | |||
every compared implementation SHOULD be documented in sufficient | picked for every compared metric implementation should be documented | |||
detail. Based on this documentation, the underlying metric | as specified in section 3.5. If the cause of the failure is a lack | |||
specification should be improved before it is promoted to a standard. | of specification clarity or multiple legitimate interpretations of | |||
the definition text, the text should be modified and the resulting | ||||
memo proposed for consensus and (possible) advancement to Internet | ||||
Standard. | ||||
The same statistical test as applicable to quantify precision of a | The same statistical test as applicable to quantify precision of a | |||
single metric implementation MUST be used to compare metric result | single metric implementation must be used to compare metric result | |||
equivalence for different implementations. To document | equivalence for different implementations. To document | |||
compatibility, the smallest measurement resolution at which the | compatibility, the smallest measurement resolution at which the | |||
compared implementations passed the ADK sample test MUST be | compared implementations passed the ADK sample test must be | |||
documented. | documented. | |||
For different implementations of the same metric, "variations in | For different implementations of the same metric, "variations in | |||
conditions" are reasonably expected. The ADK test comparing samples | conditions" are reasonably expected. The ADK test comparing samples | |||
of the different implementations MAY result in a lower precision than | of the different implementations may result in a lower precision than | |||
the test for precision in the same-implementation comparison. | the test for precision in the same-implementation comparison. | |||
3.4. Clock synchronisation | 3.4. Clock synchronisation | |||
Clock synchronization effects require special attention. Accuracy of | Clock synchronization effects require special attention. Accuracy of | |||
one-way active delay measurements for any metrics implementation | one-way active delay measurements for any metrics implementation | |||
depends on clock synchronization between the source and destination | depends on clock synchronization between the source and destination | |||
of tests. Ideally, one-way active delay measurement (RFC 2679, | of tests. Ideally, one-way active delay measurement (RFC 2679, | |||
[RFC2679]) test endpoints either have direct access to independent | [RFC2679]) test endpoints either have direct access to independent | |||
GPS or CDMA-based time sources or indirect access to nearby NTP | GPS or CDMA-based time sources or indirect access to nearby NTP | |||
primary (stratum 1) time sources, equipped with GPS receivers. | primary (stratum 1) time sources, equipped with GPS receivers. | |||
Access to these time sources may not be available at all test | Access to these time sources may not be available at all test | |||
locations associated with different Internet paths, for a variety of | locations associated with different Internet paths, for a variety of | |||
reasons out of scope of this document. | reasons out of scope of this document. | |||
When secondary (stratum 2 and above) time sources are used with NTP | When secondary (stratum 2 and above) time sources are used with NTP | |||
running across the same network, whose metrics are subject to | running across the same network, whose metrics are subject to | |||
comparative implementation tests, network impairments can affect | comparative implementation tests, network impairments can affect | |||
clock synchronization, distort sample one-way values and their | clock synchronization, distort sample one-way values and their | |||
interval statistics. It is RECOMMENDED to discard sample one-way | interval statistics. It is recommended to discard sample one-way | |||
delay values for any implementation, when one of the following | delay values for any implementation when one of the following | |||
reliability conditions is met: | reliability conditions is met: | |||
o Delay is measured and is finite in one direction, but not the | o Delay is measured and is finite in one direction, but not the | |||
other. | other. | |||
o Absolute value of the difference between the sum of one-way | o Absolute value of the difference between the sum of one-way | |||
measurements in both directions and round-trip measurement is | measurements in both directions and round-trip measurement is | |||
greater than X% of the latter value. | greater than X% of the latter value. | |||
Examination of the second condition requires RTT measurement for | Examination of the second condition requires RTT measurement for | |||
skipping to change at page 18, line 20 | skipping to change at page 16, line 50 | |||
unreliable one-way delay samples and misidentification of reliable | unreliable one-way delay samples and misidentification of reliable | |||
samples under a wide range of Internet path RTTs probably requires | samples under a wide range of Internet path RTTs probably requires | |||
further study. | further study. | |||
An IPPM compliant metric implementation of an RFC that requires | An IPPM compliant metric implementation of an RFC that requires | |||
synchronized clocks is expected to provide precise measurement | synchronized clocks is expected to provide precise measurement | |||
results. | results. | |||
IF an implementation publishes a specification of its precision, such | IF an implementation publishes a specification of its precision, such | |||
as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then | as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then | |||
the specification SHOULD be met over a useful measurement duration. | the specification should be met over a useful measurement duration. | |||
For example, if the metric is measured along an Internet path which | For example, if the metric is measured along an Internet path which | |||
is stable and not congested, then the precision specification SHOULD | is stable and not congested, then the precision specification should | |||
be met over durations of an hour or more. | be met over durations of an hour or more. | |||
3.5. Recommended Metric Verification Measurement Process | 3.5. Recommended Metric Verification Measurement Process | |||
In order to meet their obligations under the IETF Standards Process | In order to meet their obligations under the IETF Standards Process | |||
the IESG must be convinced that each metric specification advanced to | the IESG must be convinced that each metric specification advanced to | |||
Draft Standard or Internet Standard status is clearly written, that | Internet Standard status is clearly written, that there are a | |||
there are a sufficient number of verified equivalent implementations, | sufficient number of verified equivalent implementations, and that | |||
and that options that have been implemented are documented. | options that have been implemented are documented. | |||
In the context of this document, metrics are designed to measure some | In the context of this document, metrics are designed to measure some | |||
characteristic of a data network. An aim of any metric definition | characteristic of a data network. An aim of any metric definition | |||
should be that it should be specified in a way that can reliably | should be that it should be specified in a way that can reliably | |||
measure the specific characteristic in a repeatable way across | measure the specific characteristic in a repeatable way across | |||
multiple independent implementations. | multiple independent implementations. | |||
Each metric, statistic or option of those to be validated MUST be | Each metric, statistic or option of those to be validated MUST be | |||
compared against a reference measurement or another implementation by | compared against a reference measurement or another implementation as | |||
as specified by this document. | specified in this document. | |||
Finally, the metric definitions, embodied in the text of the RFCs, | Finally, the metric definitions, embodied in the text of the RFCs, | |||
are the objects that require evaluation and possible revision in | are the objects that require evaluation and possible revision in | |||
order to advance to the next step on the standards track. | order to advance to Internet Standard. | |||
IF two (or more) implementations do not measure an equivalent metric | IF two (or more) implementations do not measure an equivalent metric | |||
as specified by this document, | as specified by this document, | |||
AND sources of measurement error do not adequately explain the lack | AND sources of measurement error do not adequately explain the lack | |||
of agreement, | of agreement, | |||
THEN the details of each implementation should be audited along with | THEN the details of each implementation should be audited along with | |||
the exact definition text, to determine if there is a lack of clarity | the exact definition text, to determine if there is a lack of clarity | |||
that has caused the implementations to vary in a way that affects the | that has caused the implementations to vary in a way that affects the | |||
correspondence of the results. | correspondence of the results. | |||
IF there was a lack of clarity or multiple legitimate interpretations | IF there was a lack of clarity or multiple legitimate interpretations | |||
of the definition text, | of the definition text, | |||
THEN the text should be modified and the resulting memo proposed for | THEN the text should be modified and the resulting memo proposed for | |||
consensus and (possible) advancement along the standards track. | consensus and (possible) advancement along the standards track. | |||
skipping to change at page 19, line 16 | skipping to change at page 17, line 47 | |||
that has caused the implementations to vary in a way that affects the | that has caused the implementations to vary in a way that affects the | |||
correspondence of the results. | correspondence of the results. | |||
IF there was a lack of clarity or multiple legitimate interpretations | IF there was a lack of clarity or multiple legitimate interpretations | |||
of the definition text, | of the definition text, | |||
THEN the text should be modified and the resulting memo proposed for | THEN the text should be modified and the resulting memo proposed for | |||
consensus and (possible) advancement along the standards track. | consensus and (possible) advancement along the standards track. | |||
Finally, all the findings MUST be documented in a report that can | Finally, all the findings MUST be documented in a report that can | |||
support advancement on the standards track, similar to those | support advancement to Internet Standard, as described here (similar | |||
described in [RFC5657]. The list of measurement devices used in | to those described in [RFC5657]). The list of measurement devices | |||
testing satisfies the implementation requirement, while the test | used in testing satisfies the implementation requirement, while the | |||
results provide information on the quality of each specification in | test results provide information on the quality of each specification | |||
the metric RFC (the surrogate for feature interoperability). | in the metric RFC (the surrogate for feature interoperability). | |||
The complete process of advancing a metric specification to a | The complete process of advancing a metric specification to a | |||
standard as defined by this document is illustrated in Figure 4. | standard as defined by this document is illustrated in Figure 4. | |||
,---. | ,---. | |||
/ \ | / \ | |||
( Start ) | ( Start ) | |||
\ / Implementations | \ / Implementations | |||
`-+-' +-------+ | `-+-' +-------+ | |||
| /| 1 `. | | /| 1 `. | |||
+---+----+ / +-------+ `.-----------+ ,-------. | +---+----+ / +-------+ `.-----------+ ,-------. | |||
| RFC | / |Check for | ,' was RFC `. YES | | RFC | / |Check for | ,' was RFC `. YES | |||
| | / |Equivalence.... clause x ------+ | | | / |Equivalence.... clause x ------+ | |||
| |/ +-------+ |under | `. clear? ,' | | | |/ +-------+ |under | `. clear? ,' | | |||
| Metric \.....| 2 ....relevant | `---+---' +----+-----+ | | Metric \.....| 2 ....relevant | `---+---' +----+-----+ | |||
| Metric |\ +-------+ |identical | No | |Report | | | Metric |\ +-------+ |identical | No | |Report | | |||
| Metric | \ |network | +--+----+ |results + | | | Metric | \ |network | +--+----+ |results + | | |||
| ... | \ |conditions | |Modify | |Advance | | | ... | \ |conditions | |Modify | |Advance | | |||
| | \ +-------+ | | |Spec +--+RFC | | | | \ +-------+ | | |Spec +--+RFC | | |||
+--------+ \| n |.'+-----------+ +-------+ |request(?)| | +--------+ \| n |.'+-----------+ +-------+ |request | | |||
+-------+ +----------+ | +-------+ +----------+ | |||
Illustration of the metric standardisation process | Illustration of the metric standardisation process | |||
Figure 4 | Figure 4 | |||
Any recommendation for the advancement of a metric specification MUST | Any recommendation for the advancement of a metric specification MUST | |||
be accompanied by an implementation report, as is the case with all | be accompanied by an implementation report. The implementation | |||
requests for the advancement of IETF specifications. The | report needs to include the tests performed, the applied test setup, | |||
implementation report needs to include the tests performed, the | the specific metrics in the RFC and reports of the tests performed | |||
applied test setup, the specific metrics in the RFC and reports of | with two or more implementations. The test plan needs to specify the | |||
the tests performed with two or more implementations. The test plan | precision reached for each measured metric and thus define the | |||
needs to specify the precision reached for each measured metric and | meaning of "statistically equivalent" for the specific metrics being | |||
thus define the meaning of "statistically equivalent" for the | tested. | |||
specific metrics being tested. | ||||
Ideally, the test plan would co-evolve with the development of the | Ideally, the test plan would co-evolve with the development of the | |||
metric, since that's when people have the most context in their | metric, since that's when participants have the clearest context in | |||
thinking regarding the different subtleties that can arise. | their minds regarding the different subtleties that can arise. | |||
In particular, the implementation report MUST as a minimum document: | In particular, the implementation report MUST as a minimum document: | |||
o The metric compared and the RFC specifying it. This includes | o The metric compared and the RFC specifying it. This includes | |||
statements as required by the section "Tests of an individual | statements as required by the section "Tests of an individual | |||
implementation against a metric specification" of this document. | implementation against a metric specification" of this document. | |||
o The measurement configuration and setup. | o The measurement configuration and setup. | |||
o A complete specification of the measurement stream (mean rate, | o A complete specification of the measurement stream (mean rate, | |||
skipping to change at page 21, line 9 | skipping to change at page 19, line 38 | |||
and network conditions allowing reproduction of these laboratory | and network conditions allowing reproduction of these laboratory | |||
conditions for readers of the implementation report. | conditions for readers of the implementation report. | |||
o The documentation helping to improve metric specifications defined | o The documentation helping to improve metric specifications defined | |||
by this section. | by this section. | |||
All of the tests for each set SHOULD be run in a test setup as | All of the tests for each set SHOULD be run in a test setup as | |||
specified in the section "Test setup resulting in identical live | specified in the section "Test setup resulting in identical live | |||
network testing conditions." | network testing conditions." | |||
If a different test set up is chosen, it is RECOMMENDED to avoid | If a different test setup is chosen, it is recommended to avoid | |||
effects falsifying results of validation measurements caused by real | effects falsifying results of validation measurements caused by real | |||
data networks (like parallelism in devices and networks). Data | data networks (like parallelism in devices and networks). Data | |||
networks may forward packets differently in the case of: | networks may forward packets differently in the case of: | |||
o Different packet sizes chosen for different metric | o Different packet sizes chosen for different metric | |||
implementations. A proposed countermeasure is selecting the same | implementations. A proposed countermeasure is selecting the same | |||
packet size when validating results of two samples or a sample | packet size when validating results of two samples or a sample | |||
against an original distribution. | against an original distribution. | |||
o Selection of differing IP addresses and ports used by different | o Selection of differing IP addresses and ports used by different | |||
skipping to change at page 23, line 16 | skipping to change at page 21, line 42 | |||
This memo does not raise any specific security issues. | This memo does not raise any specific security issues. | |||
8. References | 8. References | |||
8.1. Normative References | 8.1. Normative References | |||
[RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, | [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, | |||
October 1996. | October 1996. | |||
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision | ||||
3", BCP 9, RFC 2026, October 1996. | ||||
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | |||
Requirement Levels", BCP 14, RFC 2119, March 1997. | Requirement Levels", BCP 14, RFC 2119, March 1997. | |||
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, | [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, | |||
"Framework for IP Performance Metrics", RFC 2330, | "Framework for IP Performance Metrics", RFC 2330, | |||
May 1998. | May 1998. | |||
[RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, | [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, | |||
G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", | G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", | |||
RFC 2661, August 1999. | RFC 2661, August 1999. | |||
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way | [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way | |||
Delay Metric for IPPM", RFC 2679, September 1999. | Delay Metric for IPPM", RFC 2679, September 1999. | |||
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way | ||||
Packet Loss Metric for IPPM", RFC 2680, September 1999. | ||||
[RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip | ||||
Delay Metric for IPPM", RFC 2681, September 1999. | ||||
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. | [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. | |||
Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, | Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, | |||
March 2000. | March 2000. | |||
[RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling | [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling | |||
Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. | Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. | |||
[RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, | [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, | |||
"Encapsulation Methods for Transport of Ethernet over MPLS | "Encapsulation Methods for Transport of Ethernet over MPLS | |||
Networks", RFC 4448, April 2006. | Networks", RFC 4448, April 2006. | |||
[RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- | ||||
Network Tunneling", RFC 4459, April 2006. | ||||
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. | [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. | |||
Zekauskas, "A One-way Active Measurement Protocol | Zekauskas, "A One-way Active Measurement Protocol | |||
(OWAMP)", RFC 4656, September 2006. | (OWAMP)", RFC 4656, September 2006. | |||
[RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport | [RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport | |||
of Ethernet Frames over Layer 2 Tunneling Protocol Version | of Ethernet Frames over Layer 2 Tunneling Protocol Version | |||
3 (L2TPv3)", RFC 4719, November 2006. | 3 (L2TPv3)", RFC 4719, November 2006. | |||
[RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal | [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal | |||
Cost Multipath Treatment in MPLS Networks", BCP 128, | Cost Multipath Treatment in MPLS Networks", BCP 128, | |||
RFC 4928, June 2007. | RFC 4928, June 2007. | |||
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation | [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation | |||
and Implementation Reports for Advancement to Draft | and Implementation Reports for Advancement to Draft | |||
Standard", BCP 9, RFC 5657, September 2009. | Standard", BCP 9, RFC 5657, September 2009. | |||
[RFC6410] Housley, R., Crocker, D., and E. Burger, "Reducing the | ||||
Standards Track to Two Maturity Levels", BCP 9, RFC 6410, | ||||
October 2011. | ||||
8.2. Informative References | 8.2. Informative References | |||
[ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling | [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling | |||
Tests of fit, for continuous and discrete cases", | Tests of fit, for continuous and discrete cases", | |||
University of Washington, Technical Report No. 81, | University of Washington, Technical Report No. 81, | |||
May 1986. | May 1986. | |||
[GU+Duffield] | [GU-Duffield] | |||
Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE | Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE | |||
Encapsulated Multicast Probing: A Scalable Technique for | Encapsulated Multicast Probing: A Scalable Technique for | |||
Measuring One-Way Loss", SIGMETRICS'07 San Diego, | Measuring One-Way Loss", SIGMETRICS'07 San Diego, | |||
California, USA, June 2007. | California, USA, June 2007. | |||
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision | ||||
3", BCP 9, RFC 2026, October 1996. | ||||
[RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- | ||||
Network Tunneling", RFC 4459, April 2006. | ||||
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. | [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. | |||
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", | Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", | |||
RFC 5357, October 2008. | RFC 5357, October 2008. | |||
[Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and | [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and | |||
Combinations of Such Tests. R package version 1.0.", , | Combinations of Such Tests. R package version 1.0.", , | |||
2008. | 2008. | |||
[Rtool] R Development Core Team, "R: A language and environment | [Rtool] R Development Core Team, "R: A language and environment | |||
for statistical computing. R Foundation for Statistical | for statistical computing. R Foundation for Statistical | |||
Computing, Vienna, Austria. ISBN 3-900051-07-0, URL | Computing, Vienna, Austria. ISBN 3-900051-07-0, URL | |||
http://www.R-project.org/", , 2011. | http://www.R-project.org/", , 2011. | |||
[Rule of thumb] | ||||
Hardy, M., "Confidence interval", March 2010. | ||||
[bradner-metrictest] | [bradner-metrictest] | |||
Bradner, S., Mankin, A., and V. Paxson, "Advancement of | Bradner, S., Mankin, A., and V. Paxson, "Advancement of | |||
metrics specifications on the IETF Standards Track", | metrics specifications on the IETF Standards Track", | |||
draft -bradner-metricstest-03, (work in progress), | draft -bradner-metricstest-03, (work in progress), | |||
July 2007. | July 2007. | |||
[morton-advance-metrics] | ||||
Morton, A., "Problems and Possible Solutions for Advancing | ||||
Metrics on the Standards Track", draft -morton-ippm- | ||||
advance-metrics-00, (work in progress), July 2009. | ||||
[morton-advance-metrics-01] | ||||
Morton, A., "Lab Test Results for Advancing Metrics on the | ||||
Standards Track", draft -morton-ippm-advance-metrics-01, | ||||
(work in progress), June 2010. | ||||
[morton-testplan-rfc2679] | [morton-testplan-rfc2679] | |||
Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test | Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test | |||
Plan and Results for Advancing RFC 2679 on the Standards | Plan and Results for Advancing RFC 2679 on the Standards | |||
Track", draft -morton-ippm-testplan-rfc2679-01, (work in | Track", draft -morton-ippm-testplan-rfc2679-01, (work in | |||
progress), June 2011. | progress), June 2011. | |||
Appendix A. An example on a One-way Delay metric validation | Appendix A. An example on a One-way Delay metric validation | |||
The text of this appendix is not binding. It is an example how parts | The text of this appendix is not binding. It is an example how parts | |||
of a One-way Delay metric test could look like. | of a One-way Delay metric test could look like. | |||
http://xml.resource.org/public/rfc/bibxml/ | ||||
A.1. Compliance to Metric specification requirements | A.1. Compliance to Metric specification requirements | |||
One-way Delay, Loss threshold, RFC 2679 | One-way Delay, Loss threshold, RFC 2679 | |||
This test determines if implementations use the same configured | This test determines if implementations use the same configured | |||
maximum waiting time delay from one measurement to another under | maximum waiting time delay from one measurement to another under | |||
different delay conditions, and correctly declare packets arriving in | different delay conditions, and correctly declare packets arriving in | |||
excess of the waiting time threshold as lost. See Section 3.5 of | excess of the waiting time threshold as lost. See Section 3.5 of | |||
RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. | RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. | |||
skipping to change at page 29, line 27 | skipping to change at page 27, line 30 | |||
but this is as it should be. | but this is as it should be. | |||
The C++ code below will perform a 2-sample AD comparison when | The C++ code below will perform a 2-sample AD comparison when | |||
compiled and presented with two column vectors in a file (using white | compiled and presented with two column vectors in a file (using white | |||
space as separation). This version contains modifications to use the | space as separation). This version contains modifications to use the | |||
vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The | vectors and run as a stand-alone module by Wes Eddy, Sept 2011. The | |||
status of the comparison can be checked on the command line with "$ | status of the comparison can be checked on the command line with "$ | |||
echo $?" or the last line can be replaced with a printf statement for | echo $?" or the last line can be replaced with a printf statement for | |||
adk_result instead. | adk_result instead. | |||
/* Routines for computing the Anderson-Darling 2 sample | /* | |||
* test statistic. | ||||
* | ||||
* Implemented based on the description in | ||||
* "Anderson-Darling K Sample Test" Heckert, Alan and | ||||
* Filliben, James, editors, Dataplot Reference Manual, | ||||
* Chapter 15 Auxiliary, NIST, 2004. | ||||
* Official Reference by 2010 | ||||
* Heckert, N. A. (2001). Dataplot website at the | ||||
* National Institute of Standards and Technology: | ||||
* http://www.itl.nist.gov/div898/software/dataplot.html/ | ||||
* June 2001. | ||||
*/ | ||||
#include <iostream> | Copyright (c) 2011 IETF Trust and the persons identified | |||
#include <fstream> | as authors of the code. All rights reserved. | |||
#include <vector> | ||||
#include <sstream> | ||||
using namespace std; | Redistribution and use in source and binary forms, with | |||
or without modification, is permitted pursuant to, and subject | ||||
to the license terms contained in, the Simplified BSD License | ||||
set forth in Section 4.c of the IETF Trust's Legal Provisions | ||||
Relating to IETF Documents (http://trustee.ietf.org/license-info). | ||||
int main() { | */ | |||
vector<double> vec1, vec2; | ||||
double adk_result; | ||||
static int k, val_st_z_samp1, val_st_z_samp2, | ||||
val_eq_z_samp1, val_eq_z_samp2, | ||||
j, n_total, n_sample1, n_sample2, L, | ||||
max_number_samples, line, maxnumber_z; | ||||
static int column_1, column_2; | ||||
static double adk, n_value, z, sum_adk_samp1, | ||||
sum_adk_samp2, z_aux; | ||||
static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux; | ||||
static bool next_z_sample2, equal_z_both_samples; | ||||
static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2, | ||||
old_eq_line1; | ||||
static double adk_criterium = 1.993; | /* Routines for computing the Anderson-Darling 2 sample | |||
* test statistic. | ||||
* | ||||
* Implemented based on the description in | ||||
* "Anderson-Darling K Sample Test" Heckert, Alan and | ||||
* Filliben, James, editors, Dataplot Reference Manual, | ||||
* Chapter 15 Auxiliary, NIST, 2004. | ||||
* Official Reference by 2010 | ||||
* Heckert, N. A. (2001). Dataplot website at the | ||||
* National Institute of Standards and Technology: | ||||
* http://www.itl.nist.gov/div898/software/dataplot.html/ | ||||
* June 2001. | ||||
*/ | ||||
/* vec1 and vec2 to be initialised with sample 1 and | #include <iostream> | |||
* sample 2 values in ascending order */ | #include <fstream> | |||
while (!cin.eof()) { | #include <vector> | |||
double f1, f2; | #include <sstream> | |||
cin >> f1; | ||||
cin >> f2; | ||||
vec1.push_back(f1); | ||||
vec2.push_back(f2); | ||||
} | ||||
k = 2; | using namespace std; | |||
n_sample1 = vec1.size() - 1; | ||||
n_sample2 = vec2.size() - 1; | ||||
// -1 because vec[0] is a dummy value | int main() { | |||
n_total = n_sample1 + n_sample2; | vector<double> vec1, vec2; | |||
double adk_result; | ||||
static int k, val_st_z_samp1, val_st_z_samp2, | ||||
val_eq_z_samp1, val_eq_z_samp2, | ||||
j, n_total, n_sample1, n_sample2, L, | ||||
max_number_samples, line, maxnumber_z; | ||||
static int column_1, column_2; | ||||
static double adk, n_value, z, sum_adk_samp1, | ||||
sum_adk_samp2, z_aux; | ||||
static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux; | ||||
static bool next_z_sample2, equal_z_both_samples; | ||||
static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2, | ||||
old_eq_line1; | ||||
/* value equal to the line with a value = zj in sample 1. | static double adk_criterium = 1.993; | |||
* Here j=1, so the line is 1. | ||||
*/ | ||||
val_eq_z_samp1 = 1; | ||||
/* value equal to the line with a value = zj in sample 2. | /* vec1 and vec2 to be initialised with sample 1 and | |||
* Here j=1, so the line is 1. | * sample 2 values in ascending order */ | |||
*/ | while (!cin.eof()) { | |||
val_eq_z_samp2 = 1; | double f1, f2; | |||
cin >> f1; | ||||
cin >> f2; | ||||
vec1.push_back(f1); | ||||
vec2.push_back(f2); | ||||
} | ||||
/* value equal to the last line with a value < zj | k = 2; | |||
* in sample 1. Here j=1, so the line is 0. | n_sample1 = vec1.size() - 1; | |||
*/ | n_sample2 = vec2.size() - 1; | |||
val_st_z_samp1 = 0; | ||||
/* value equal to the last line with a value < zj | // -1 because vec[0] is a dummy value | |||
* in sample 1. Here j=1, so the line is 0. | n_total = n_sample1 + n_sample2; | |||
*/ | /* value equal to the line with a value = zj in sample 1. | |||
val_st_z_samp2 = 0; | * Here j=1, so the line is 1. | |||
*/ | ||||
val_eq_z_samp1 = 1; | ||||
sum_adk_samp1 = 0; | /* value equal to the line with a value = zj in sample 2. | |||
sum_adk_samp2 = 0; | * Here j=1, so the line is 1. | |||
j = 1; | */ | |||
val_eq_z_samp2 = 1; | ||||
// as mentioned above, j=1 | /* value equal to the last line with a value < zj | |||
equal_z_both_samples = false; | * in sample 1. Here j=1, so the line is 0. | |||
*/ | ||||
val_st_z_samp1 = 0; | ||||
next_z_sample2 = false; | /* value equal to the last line with a value < zj | |||
* in sample 1. Here j=1, so the line is 0. | ||||
*/ | ||||
val_st_z_samp2 = 0; | ||||
//assuming the next z to be of sample 1 | sum_adk_samp1 = 0; | |||
stop_loop1 = n_sample1 + 1; | sum_adk_samp2 = 0; | |||
j = 1; | ||||
// + 1 because vec[0] is a dummy, see n_sample1 declaration | // as mentioned above, j=1 | |||
stop_loop2 = n_sample2 + 1; | equal_z_both_samples = false; | |||
stop_loop3 = n_total + 1; | ||||
/* The required z values are calculated until all values | next_z_sample2 = false; | |||
* of both samples have been taken into account. See the | ||||
* lines above for the stoploop values. Construct required | ||||
* to avoid a mathematical operation in the While condition | ||||
*/ | ||||
while (((stop_loop1 > val_eq_z_samp1) | ||||
|| (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j) | ||||
{ | ||||
if(val_eq_z_samp1 < n_sample1+1) | ||||
{ | ||||
/* here, a preliminary zj value is set. | ||||
* See below how to calculate the actual zj. | ||||
*/ | ||||
z = vec1[val_eq_z_samp1]; | ||||
/* this while sequence calculates the number of values | //assuming the next z to be of sample 1 | |||
* equal to z. | stop_loop1 = n_sample1 + 1; | |||
*/ | ||||
while ((val_eq_z_samp1+1 < n_sample1) | ||||
&& z == vec1[val_eq_z_samp1+1] ) | ||||
{ | ||||
val_eq_z_samp1++; | ||||
} | ||||
} | ||||
else | ||||
{ | ||||
val_eq_z_samp1 = 0; | ||||
val_st_z_samp1 = n_sample1; | ||||
// this should be val_eq_z_samp1 - 1 = n_sample1 | // + 1 because vec[0] is a dummy, see n_sample1 declaration | |||
} | stop_loop2 = n_sample2 + 1; | |||
stop_loop3 = n_total + 1; | ||||
if(val_eq_z_samp2 < n_sample2+1) | /* The required z values are calculated until all values | |||
{ | * of both samples have been taken into account. See the | |||
z_aux = vec2[val_eq_z_samp2];; | * lines above for the stoploop values. Construct required | |||
* to avoid a mathematical operation in the While condition | ||||
*/ | ||||
while (((stop_loop1 > val_eq_z_samp1) | ||||
|| (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j) | ||||
{ | ||||
if(val_eq_z_samp1 < n_sample1+1) | ||||
{ | ||||
/* here, a preliminary zj value is set. | ||||
* See below how to calculate the actual zj. | ||||
*/ | ||||
z = vec1[val_eq_z_samp1]; | ||||
/* this while sequence calculates the number of values | /* this while sequence calculates the number of values | |||
* equal to z_aux | * equal to z. | |||
*/ | */ | |||
while ((val_eq_z_samp1+1 < n_sample1) | ||||
&& z == vec1[val_eq_z_samp1+1] ) | ||||
{ | ||||
val_eq_z_samp1++; | ||||
} | ||||
} | ||||
else | ||||
{ | ||||
val_eq_z_samp1 = 0; | ||||
val_st_z_samp1 = n_sample1; | ||||
while ((val_eq_z_samp2+1 < n_sample2) | // this should be val_eq_z_samp1 - 1 = n_sample1 | |||
&& z_aux == vec2[val_eq_z_samp2+1] ) | } | |||
{ | ||||
val_eq_z_samp2++; | ||||
} | ||||
/* the smaller of the two actual data values is picked | if(val_eq_z_samp2 < n_sample2+1) | |||
* as the next zj. | { | |||
*/ | z_aux = vec2[val_eq_z_samp2];; | |||
if(z > z_aux) | /* this while sequence calculates the number of values | |||
{ | * equal to z_aux | |||
z = z_aux; | */ | |||
next_z_sample2 = true; | ||||
} | ||||
else | ||||
{ | ||||
if (z == z_aux) | ||||
{ | ||||
equal_z_both_samples = true; | ||||
} | ||||
/* This is the case, if the last value of column1 is | while ((val_eq_z_samp2+1 < n_sample2) | |||
* smaller than the remaining values of column2. | && z_aux == vec2[val_eq_z_samp2+1] ) | |||
*/ | { | |||
if (val_eq_z_samp1 == 0) | val_eq_z_samp2++; | |||
{ | } | |||
z = z_aux; | ||||
next_z_sample2 = true; | ||||
} | ||||
} | ||||
} | ||||
else | ||||
{ | ||||
val_eq_z_samp2 = 0; | ||||
val_st_z_samp2 = n_sample2; | ||||
// this should be val_eq_z_samp2 - 1 = n_sample2 | /* the smaller of the two actual data values is picked | |||
* as the next zj. | ||||
*/ | ||||
} | if(z > z_aux) | |||
{ | ||||
z = z_aux; | ||||
next_z_sample2 = true; | ||||
} | ||||
else | ||||
{ | ||||
if (z == z_aux) | ||||
{ | ||||
equal_z_both_samples = true; | ||||
} | ||||
/* in the following, sum j = 1 to L is calculated for | /* This is the case, if the last value of column1 is | |||
* sample 1 and sample 2. | * smaller than the remaining values of column2. | |||
*/ | */ | |||
if (equal_z_both_samples) | if (val_eq_z_samp1 == 0) | |||
{ | { | |||
z = z_aux; | ||||
next_z_sample2 = true; | ||||
} | ||||
} | ||||
} | ||||
else | ||||
{ | ||||
val_eq_z_samp2 = 0; | ||||
val_st_z_samp2 = n_sample2; | ||||
/* hj is the number of values in the combined sample | // this should be val_eq_z_samp2 - 1 = n_sample2 | |||
* equal to zj | ||||
*/ | ||||
hj = val_eq_z_samp1 - val_st_z_samp1 | ||||
+ val_eq_z_samp2 - val_st_z_samp2; | ||||
/* H_j is the number of values in the combined sample | } | |||
* smaller than zj plus one half the the number of | ||||
* values in the combined sample equal to zj | ||||
* (that's hj/2). | ||||
*/ | ||||
H_j = val_st_z_samp1 + val_st_z_samp2 | ||||
+ hj / 2; | ||||
/* F1j is the number of values in the 1st sample | /* in the following, sum j = 1 to L is calculated for | |||
* which are less than zj plus one half the number | * sample 1 and sample 2. | |||
* of values in this sample which are equal to zj. | */ | |||
*/ | if (equal_z_both_samples) | |||
{ | ||||
F1j = val_st_z_samp1 + (double) | /* hj is the number of values in the combined sample | |||
(val_eq_z_samp1 - val_st_z_samp1) / 2; | * equal to zj | |||
*/ | ||||
hj = val_eq_z_samp1 - val_st_z_samp1 | ||||
+ val_eq_z_samp2 - val_st_z_samp2; | ||||
/* F2j is the number of values in the 1st sample | /* H_j is the number of values in the combined sample | |||
* which are less than zj plus one half the number | * smaller than zj plus one half the the number of | |||
* of values in this sample which are equal to zj. | * values in the combined sample equal to zj | |||
*/ | * (that's hj/2). | |||
F2j = val_st_z_samp2 + (double) | */ | |||
(val_eq_z_samp2 - val_st_z_samp2) / 2; | H_j = val_st_z_samp1 + val_st_z_samp2 | |||
+ hj / 2; | ||||
/* set the line of values equal to zj to the | /* F1j is the number of values in the 1st sample | |||
* actual line of the last value picked for zj. | * which are less than zj plus one half the number | |||
*/ | * of values in this sample which are equal to zj. | |||
val_st_z_samp1 = val_eq_z_samp1; | */ | |||
/* Set the line of values equal to zj to the actual | F1j = val_st_z_samp1 + (double) | |||
* line of the last value picked for zjof each | (val_eq_z_samp1 - val_st_z_samp1) / 2; | |||
* sample. This is required as data smaller than zj | ||||
* is accounted differently than values equal to zj. | ||||
*/ | /* F2j is the number of values in the 1st sample | |||
val_st_z_samp2 = val_eq_z_samp2; | * which are less than zj plus one half the number | |||
* of values in this sample which are equal to zj. | ||||
*/ | ||||
F2j = val_st_z_samp2 + (double) | ||||
(val_eq_z_samp2 - val_st_z_samp2) / 2; | ||||
/* next the lines of the next values z, ie. zj+1 | /* set the line of values equal to zj to the | |||
* are addressed. | * actual line of the last value picked for zj. | |||
*/ | */ | |||
val_eq_z_samp1++; | val_st_z_samp1 = val_eq_z_samp1; | |||
/* next the lines of the next values z, ie. | /* Set the line of values equal to zj to the actual | |||
* zj+1 are addressed | * line of the last value picked for zjof each | |||
*/ | * sample. This is required as data smaller than zj | |||
val_eq_z_samp2++; | * is accounted differently than values equal to zj. | |||
} | */ | |||
else | val_st_z_samp2 = val_eq_z_samp2; | |||
{ | ||||
/* the smaller z value was contained in sample 2, | /* next the lines of the next values z, ie. zj+1 | |||
* hence this value is the zj to base the following | * are addressed. | |||
* calculations on. | */ | |||
*/ | val_eq_z_samp1++; | |||
if (next_z_sample2) | ||||
{ | ||||
/* hj is the number of values in the combined | /* next the lines of the next values z, ie. | |||
* sample equal to zj, in this case these are | * zj+1 are addressed | |||
* within sample 2 only. | */ | |||
*/ | val_eq_z_samp2++; | |||
hj = val_eq_z_samp2 - val_st_z_samp2; | } | |||
else | ||||
{ | ||||
/* H_j is the number of values in the combined sample | /* the smaller z value was contained in sample 2, | |||
* smaller than zj plus one half the the number of | * hence this value is the zj to base the following | |||
* values in the combined sample equal to zj | * calculations on. | |||
* (that's hj/2). | */ | |||
*/ | if (next_z_sample2) | |||
{ | ||||
H_j = val_st_z_samp1 + val_st_z_samp2 | /* hj is the number of values in the combined | |||
+ hj / 2; | * sample equal to zj, in this case these are | |||
* within sample 2 only. | ||||
*/ | ||||
hj = val_eq_z_samp2 - val_st_z_samp2; | ||||
/* F1j is the number of values in the 1st sample which | /* H_j is the number of values in the combined sample | |||
* are less than zj plus one half the number of values in | * smaller than zj plus one half the the number of | |||
* this sample which are equal to zj. | * values in the combined sample equal to zj | |||
* As val_eq_z_samp2 < val_eq_z_samp1, these are the | * (that's hj/2). | |||
* val_st_z_samp1 only. | */ | |||
*/ | ||||
F1j = val_st_z_samp1; | ||||
/* F2j is the number of values in the 1st sample which | H_j = val_st_z_samp1 + val_st_z_samp2 | |||
* are less than zj plus one half the number of values in | + hj / 2; | |||
* this sample which are equal to zj. The latter are from | ||||
* sample 2 only in this case. | ||||
*/ | ||||
F2j = val_st_z_samp2 + (double) | /* F1j is the number of values in the 1st sample which | |||
(val_eq_z_samp2 - val_st_z_samp2) / 2; | * are less than zj plus one half the number of values in | |||
* this sample which are equal to zj. | ||||
* As val_eq_z_samp2 < val_eq_z_samp1, these are the | ||||
* val_st_z_samp1 only. | ||||
*/ | ||||
F1j = val_st_z_samp1; | ||||
/* Set the line of values equal to zj to the actual line | /* F2j is the number of values in the 1st sample which | |||
* of the last value picked for zj of sample 2 only in | * are less than zj plus one half the number of values in | |||
* this case. | * this sample which are equal to zj. The latter are from | |||
*/ | * sample 2 only in this case. | |||
val_st_z_samp2 = val_eq_z_samp2; | */ | |||
/* next the line of the next value z, ie. zj+1 is | F2j = val_st_z_samp2 + (double) | |||
* addressed. Here, only sample 2 must be addressed. | (val_eq_z_samp2 - val_st_z_samp2) / 2; | |||
*/ | ||||
val_eq_z_samp2++; | /* Set the line of values equal to zj to the actual line | |||
if (val_eq_z_samp1 == 0) | * of the last value picked for zj of sample 2 only in | |||
{ | * this case. | |||
val_eq_z_samp1 = stop_loop1; | */ | |||
} | val_st_z_samp2 = val_eq_z_samp2; | |||
} | ||||
/* the smaller z value was contained in sample 2, | /* next the line of the next value z, ie. zj+1 is | |||
* hence this value is the zj to base the following | * addressed. Here, only sample 2 must be addressed. | |||
* calculations on. | */ | |||
*/ | ||||
else | val_eq_z_samp2++; | |||
{ | if (val_eq_z_samp1 == 0) | |||
{ | ||||
val_eq_z_samp1 = stop_loop1; | ||||
} | ||||
} | ||||
/* hj is the number of values in the combined | /* the smaller z value was contained in sample 2, | |||
* sample equal to zj, in this case these are | * hence this value is the zj to base the following | |||
* within sample 1 only. | * calculations on. | |||
*/ | */ | |||
hj = val_eq_z_samp1 - val_st_z_samp1; | ||||
/* H_j is the number of values in the combined | else | |||
* sample smaller than zj plus one half the the number | { | |||
* of values in the combined sample equal to zj | ||||
* (that's hj/2). | ||||
*/ | ||||
H_j = val_st_z_samp1 + val_st_z_samp2 | /* hj is the number of values in the combined | |||
+ hj / 2; | * sample equal to zj, in this case these are | |||
* within sample 1 only. | ||||
*/ | ||||
hj = val_eq_z_samp1 - val_st_z_samp1; | ||||
/* F1j is the number of values in the 1st sample which | /* H_j is the number of values in the combined | |||
* are less than zj plus, in this case these are within | * sample smaller than zj plus one half the the number | |||
* sample 1 only one half the number of values in this | * of values in the combined sample equal to zj | |||
* sample which are equal to zj. The latter are from | * (that's hj/2). | |||
* sample 1 only in this case. | */ | |||
*/ | ||||
F1j = val_st_z_samp1 + (double) | H_j = val_st_z_samp1 + val_st_z_samp2 | |||
(val_eq_z_samp1 - val_st_z_samp1) / 2; | + hj / 2; | |||
/* F2j is the number of values in the 1st sample which | /* F1j is the number of values in the 1st sample which | |||
* are less than zj plus one half the number of values | * are less than zj plus, in this case these are within | |||
* in this sample which are equal to zj. As | * sample 1 only one half the number of values in this | |||
* val_eq_z_samp1 < val_eq_z_samp2, these are the | * sample which are equal to zj. The latter are from | |||
* val_st_z_samp2 only. | * sample 1 only in this case. | |||
*/ | */ | |||
F2j = val_st_z_samp2; | F1j = val_st_z_samp1 + (double) | |||
(val_eq_z_samp1 - val_st_z_samp1) / 2; | ||||
/* Set the line of values equal to zj to the actual line | /* F2j is the number of values in the 1st sample which | |||
* of the last value picked for zj of sample 1 only in | * are less than zj plus one half the number of values | |||
* this case | * in this sample which are equal to zj. As | |||
*/ | * val_eq_z_samp1 < val_eq_z_samp2, these are the | |||
* val_st_z_samp2 only. | ||||
*/ | ||||
val_st_z_samp1 = val_eq_z_samp1; | F2j = val_st_z_samp2; | |||
/* next the line of the next value z, ie. zj+1 is | /* Set the line of values equal to zj to the actual line | |||
* addressed. Here, only sample 1 must be addressed. | * of the last value picked for zj of sample 1 only in | |||
*/ | * this case | |||
val_eq_z_samp1++; | */ | |||
if (val_eq_z_samp2 == 0) | val_st_z_samp1 = val_eq_z_samp1; | |||
{ | ||||
val_eq_z_samp2 = stop_loop2; | ||||
} | ||||
} | ||||
} | ||||
denom_1_aux = n_total * F1j - n_sample1 * H_j; | /* next the line of the next value z, ie. zj+1 is | |||
denom_2_aux = n_total * F2j - n_sample2 * H_j; | * addressed. Here, only sample 1 must be addressed. | |||
*/ | ||||
val_eq_z_samp1++; | ||||
sum_adk_samp1 = sum_adk_samp1 + hj | if (val_eq_z_samp2 == 0) | |||
* (denom_1_aux * denom_1_aux) / | { | |||
(H_j * (n_total - H_j) | val_eq_z_samp2 = stop_loop2; | |||
- n_total * hj / 4); | } | |||
sum_adk_samp2 = sum_adk_samp2 + hj | } | |||
* (denom_2_aux * denom_2_aux) / | } | |||
(H_j * (n_total - H_j) | ||||
- n_total * hj / 4); | ||||
next_z_sample2 = false; | ||||
equal_z_both_samples = false; | ||||
/* index to count the z. It is only required to prevent | denom_1_aux = n_total * F1j - n_sample1 * H_j; | |||
* the while slope to execute endless | denom_2_aux = n_total * F2j - n_sample2 * H_j; | |||
*/ | ||||
j++; | ||||
} | ||||
// calculating the adk value is the final step. | sum_adk_samp1 = sum_adk_samp1 + hj | |||
adk_result = (double) (n_total - 1) / (n_total | * (denom_1_aux * denom_1_aux) / | |||
* n_total * (k - 1)) | (H_j * (n_total - H_j) | |||
* (sum_adk_samp1 / n_sample1 | - n_total * hj / 4); | |||
+ sum_adk_samp2 / n_sample2); | sum_adk_samp2 = sum_adk_samp2 + hj | |||
* (denom_2_aux * denom_2_aux) / | ||||
(H_j * (n_total - H_j) | ||||
- n_total * hj / 4); | ||||
/* if(adk_result <= adk_criterium) | next_z_sample2 = false; | |||
* adk_2_sample test is passed | equal_z_both_samples = false; | |||
*/ | ||||
return adk_result <= adk_criterium; | /* index to count the z. It is only required to prevent | |||
} | * the while slope to execute endless | |||
*/ | ||||
j++; | ||||
} | ||||
// calculating the adk value is the final step. | ||||
adk_result = (double) (n_total - 1) / (n_total | ||||
* n_total * (k - 1)) | ||||
* (sum_adk_samp1 / n_sample1 | ||||
+ sum_adk_samp2 / n_sample2); | ||||
/* if(adk_result <= adk_criterium) | ||||
* adk_2_sample test is passed | ||||
*/ | ||||
return adk_result <= adk_criterium; | ||||
} | ||||
Figure 5 | Figure 5 | |||
Appendix C. Glossary | Appendix C. Glossary | |||
+-------------+-----------------------------------------------------+ | +-------------+-----------------------------------------------------+ | |||
| ADK | Anderson-Darling K-Sample test, a test used to | | | ADK | Anderson-Darling K-Sample test, a test used to | | |||
| | check whether two samples have the same statistical | | | | check whether two samples have the same statistical | | |||
| | distribution. | | | | distribution. | | |||
| ECMP | Equal Cost Multipath, a load balancing mechanism | | | ECMP | Equal Cost Multipath, a load balancing mechanism | | |||
End of changes. 129 change blocks. | ||||
575 lines changed or deleted | 498 lines changed or added | |||
This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |