draft-ietf-ippm-metrictest-01.txt | draft-ietf-ippm-metrictest-02.txt | |||
---|---|---|---|---|
Internet Engineering Task Force R. Geib, Ed. | Internet Engineering Task Force R. Geib, Ed. | |||
Internet-Draft Deutsche Telekom | Internet-Draft Deutsche Telekom | |||
Intended status: Standards Track A. Morton | Intended status: Standards Track A. Morton | |||
Expires: April 27, 2011 AT&T Labs | Expires: September 15, 2011 AT&T Labs | |||
R. Fardid | R. Fardid | |||
Cariden Technologies | Cariden Technologies | |||
A. Steinmitz | A. Steinmitz | |||
HS Fulda | HS Fulda | |||
October 24, 2010 | March 14, 2011 | |||
IPPM standard advancement testing | IPPM standard advancement testing | |||
draft-ietf-ippm-metrictest-01 | draft-ietf-ippm-metrictest-02 | |||
Abstract | Abstract | |||
This document specifies tests to determine if multiple independent | This document specifies tests to determine if multiple independent | |||
instantiations of a performance metric RFC have implemented the | instantiations of a performance metric RFC have implemented the | |||
specifications in the same way. This is the performance metric | specifications in the same way. This is the performance metric | |||
equivalent of interoperability, required to advance RFCs along the | equivalent of interoperability, required to advance RFCs along the | |||
standards track. Results from different implementations of metric | standards track. Results from different implementations of metric | |||
RFCs will be collected under the same underlying network conditions | RFCs will be collected under the same underlying network conditions | |||
and compared using state of the art statistical methods. The goal is | and compared using state of the art statistical methods. The goal is | |||
skipping to change at page 1, line 44 | skipping to change at page 1, line 44 | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
Drafts is at http://datatracker.ietf.org/drafts/current/. | Drafts is at http://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
This Internet-Draft will expire on April 27, 2011. | This Internet-Draft will expire on September 15, 2011. | |||
Copyright Notice | Copyright Notice | |||
Copyright (c) 2010 IETF Trust and the persons identified as the | Copyright (c) 2011 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(http://trustee.ietf.org/license-info) in effect on the date of | (http://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
carefully, as they describe your rights and restrictions with respect | carefully, as they describe your rights and restrictions with respect | |||
to this document. Code Components extracted from this document must | to this document. Code Components extracted from this document must | |||
include Simplified BSD License text as described in Section 4.e of | include Simplified BSD License text as described in Section 4.e of | |||
the Trust Legal Provisions and are provided without warranty as | the Trust Legal Provisions and are provided without warranty as | |||
skipping to change at page 5, line 14 | skipping to change at page 5, line 14 | |||
The metric RFC advancement process begins with a request for protocol | The metric RFC advancement process begins with a request for protocol | |||
action accompanied by a memo that documents the supporting tests and | action accompanied by a memo that documents the supporting tests and | |||
results. The procedures of [RFC2026] are expanded in[RFC5657], | results. The procedures of [RFC2026] are expanded in[RFC5657], | |||
including sample implementation and interoperability reports. | including sample implementation and interoperability reports. | |||
Section 3 of [morton-advance-metrics-01] can serve as a template for | Section 3 of [morton-advance-metrics-01] can serve as a template for | |||
a metric RFC report which accompanies the protocol action request to | a metric RFC report which accompanies the protocol action request to | |||
the Area Director, including description of the test set-up, | the Area Director, including description of the test set-up, | |||
procedures, results for each implementation and conclusions. | procedures, results for each implementation and conclusions. | |||
Changes from WG-01 to WG-02: | ||||
o Clarification of the number of test streams recommended in section | ||||
3.2. | ||||
o Clarifications on testing details in sections 3.3 and 3.4. | ||||
o Spelling corrections throughout. | ||||
Changes from WG -00 to WG -01 draft | Changes from WG -00 to WG -01 draft | |||
o Discussion on merits and requirements of a distributed lab test | o Discussion on merits and requirements of a distributed lab test | |||
using only local load generators. | using only local load generators. | |||
o Proposal of metrics suitable for tests using the proposed | o Proposal of metrics suitable for tests using the proposed | |||
measurement configuration. | measurement configuration. | |||
o Hint on delay caused by software based L2TPv3 implementation. | o Hint on delay caused by software based L2TPv3 implementation. | |||
o Added an appendix with a test configuration allowing remote tests | o Added an appendix with a test configuration allowing remote tests | |||
comparing different implementations accross the network. | comparing different implementations across the network. | |||
o Proposal for maximum error of "equivalence", based on performance | o Proposal for maximum error of "equivalence", based on performance | |||
comparison of identical implementations. This may be useful for | comparison of identical implementations. This may be useful for | |||
both ADK and non-ADK comparisons. | both ADK and non-ADK comparisons. | |||
Changes from prior ID -02 to WG -00 draft | Changes from prior ID -02 to WG -00 draft | |||
o Incorporation of aspects of reporting to support the protocol | o Incorporation of aspects of reporting to support the protocol | |||
action request in the Introduction and section 3.5 | action request in the Introduction and section 3.5 | |||
o Overhaul of sectcion 3.2 regarding tunneling: Added generic | o Overhaul of section 3.2 regarding tunneling: Added generic | |||
tunneling requirements and L2TPv3 as an example tunneling | tunneling requirements and L2TPv3 as an example tunneling | |||
mechanism fulfilling the tunneling requirements. Removed and | mechanism fulfilling the tunneling requirements. Removed and | |||
adapted some of the prior references to other tunneling protocols | adapted some of the prior references to other tunneling protocols | |||
o Softened a requirement within section 3.4 (MUST to SHOULD on | o Softened a requirement within section 3.4 (MUST to SHOULD on | |||
precision) and removed some comments of the authors. | precision) and removed some comments of the authors. | |||
o Updated contact information of one author and added a new author. | o Updated contact information of one author and added a new author. | |||
o Added example C++ code of an Anderson-Darling two sample test | o Added example C++ code of an Anderson-Darling two sample test | |||
skipping to change at page 6, line 4 | skipping to change at page 6, line 11 | |||
o Softened a requirement within section 3.4 (MUST to SHOULD on | o Softened a requirement within section 3.4 (MUST to SHOULD on | |||
precision) and removed some comments of the authors. | precision) and removed some comments of the authors. | |||
o Updated contact information of one author and added a new author. | o Updated contact information of one author and added a new author. | |||
o Added example C++ code of an Anderson-Darling two sample test | o Added example C++ code of an Anderson-Darling two sample test | |||
implementation. | implementation. | |||
Changes from ID -01 to ID -02 version | Changes from ID -01 to ID -02 version | |||
o Major editorial review, rewording and clarifications on all | o Major editorial review, rewording and clarifications on all | |||
contents. | contents. | |||
o Additional text on parrallel testing using VLANs and GRE or | o Additional text on parallel testing using VLANs and GRE or | |||
Pseudowire tunnels. | Pseudowire tunnels. | |||
o Additional examples and a glossary. | o Additional examples and a glossary. | |||
Changes from ID -00 to ID -01 version | Changes from ID -00 to ID -01 version | |||
o Addition of a comparison of individual metric implementations | o Addition of a comparison of individual metric implementations | |||
against the metric specification (trying to pick up problems and | against the metric specification (trying to pick up problems and | |||
solutions for metric advancement [morton-advance-metrics]). | solutions for metric advancement [morton-advance-metrics]). | |||
skipping to change at page 9, line 25 | skipping to change at page 9, line 34 | |||
other ones (due to travel and/or shipping+installation). For the | other ones (due to travel and/or shipping+installation). For the | |||
third option, ensuring two identically configured impairment | third option, ensuring two identically configured impairment | |||
generators requires well defined test cases and possibly identical | generators requires well defined test cases and possibly identical | |||
hard- and software. >>>Comment: for some specific tests, impairment | hard- and software. >>>Comment: for some specific tests, impairment | |||
generator accuracy requirements are less-demanding than others, and | generator accuracy requirements are less-demanding than others, and | |||
in such cases there is more flexibility in impairment generator | in such cases there is more flexibility in impairment generator | |||
configuration. <<< | configuration. <<< | |||
It is a fair question, whether the last two options can result in any | It is a fair question, whether the last two options can result in any | |||
applicable test set up at all. While an experimental approach is | applicable test set up at all. While an experimental approach is | |||
given in Appendix C, the tradeoff that measurement packets of | given in Appendix C, the trade off that measurement packets of | |||
different sites pass the path segments but always in a different | different sites pass the path segments but always in a different | |||
order of segments probably can't be avoided. | order of segments probably can't be avoided. | |||
The question of which option above results in identical networking | The question of which option above results in identical networking | |||
conditions and is broadly accepted can't be answered without more | conditions and is broadly accepted can't be answered without more | |||
practical experience in comparing implementations. The last proposal | practical experience in comparing implementations. The last proposal | |||
has the advantage that, while the measurement equipment is remotely | has the advantage that, while the measurement equipment is remotely | |||
distributed, a single network impairment generator and the Internet | distributed, a single network impairment generator and the Internet | |||
can be used in combination to impact all measurement flows. | can be used in combination to impact all measurement flows. | |||
skipping to change at page 9, line 51 | skipping to change at page 10, line 11 | |||
compliant to the latter. | compliant to the latter. | |||
Further, supported options of a metric implementation SHOULD be | Further, supported options of a metric implementation SHOULD be | |||
documented in sufficient detail. The documentation of chosen options | documented in sufficient detail. The documentation of chosen options | |||
is RECOMMENDED to minimise (and recognise) differences in the test | is RECOMMENDED to minimise (and recognise) differences in the test | |||
setup if two metric implementations are compared. Further, this | setup if two metric implementations are compared. Further, this | |||
documentation is used to validate and improve the underlying metric | documentation is used to validate and improve the underlying metric | |||
specification option, to remove options which saw no implementation | specification option, to remove options which saw no implementation | |||
or which are badly specified from the metric specification to be | or which are badly specified from the metric specification to be | |||
promoted to a standard. This documentation SHOULD be made for all | promoted to a standard. This documentation SHOULD be made for all | |||
implementation relevant specifications of a metric picked for a | implementation-relevant specifications of a metric picked for a | |||
comparison, which aren't explicitly marked as "MUST" or "REQUIRED" in | comparison that are not explicitly marked as "MUST" or "REQUIRED" in | |||
the metric specification. This applies for the following sections of | the RFC text. This applies for the following sections of all metric | |||
all metric specifications: | specifications: | |||
o Singleton Definition of the Metric. | o Singleton Definition of the Metric. | |||
o Sample Definition of the Metric. | o Sample Definition of the Metric. | |||
o Statistics Definition of the Metric. As statistics are compared | o Statistics Definition of the Metric. As statistics are compared | |||
by the test specified here, this documentation is required even in | by the test specified here, this documentation is required even in | |||
the case, that the metric specification does not contain a | the case, that the metric specification does not contain a | |||
Statistics Definition. | Statistics Definition. | |||
skipping to change at page 11, line 26 | skipping to change at page 11, line 31 | |||
different flows in the network. Measuring by separate parallel probe | different flows in the network. Measuring by separate parallel probe | |||
flows results in repeated collection of data. If both measures are | flows results in repeated collection of data. If both measures are | |||
combined, WAN network conditions are identical for a number of | combined, WAN network conditions are identical for a number of | |||
independent measurement flows, no matter what the network conditions | independent measurement flows, no matter what the network conditions | |||
are in detail. | are in detail. | |||
Any measurement setup MUST be made to avoid the probing traffic | Any measurement setup MUST be made to avoid the probing traffic | |||
itself to impede the metric measurement. The created measurement | itself to impede the metric measurement. The created measurement | |||
load MUST NOT result in congestion at the access link connecting the | load MUST NOT result in congestion at the access link connecting the | |||
measurement implementation to the WAN. The created measurement load | measurement implementation to the WAN. The created measurement load | |||
MUST NOT overload the measurement implementation itself, eg. by | MUST NOT overload the measurement implementation itself, e.g., by | |||
causing a high CPU load or by creating imprecisions due to internal | causing a high CPU load or by creating imprecisions due to internal | |||
transmit (receive respectively) probe packet collisions. | transmit (receive respectively) probe packet collisions. | |||
Tunneling multiple flows reaching a network element on a single | Tunneling multiple flows reaching a network element on a single | |||
physical port may allow to transmit all packets of the tunnel via the | physical port may allow to transmit all packets of the tunnel via the | |||
same path. Applying tunnels to avoid undesired influence of standard | same path. Applying tunnels to avoid undesired influence of standard | |||
routing for measurement purposes is a concept known from literature, | routing for measurement purposes is a concept known from literature, | |||
see e.g. GRE encapsulated multicast probing [GU+Duffield]. An | see e.g. GRE encapsulated multicast probing [GU+Duffield]. An | |||
existing IP in IP tunnel protocol can be applied to avoid Equal-Cost | existing IP in IP tunnel protocol can be applied to avoid Equal-Cost | |||
Multi-Path (ECMP) routing of different measurement streams if it | Multi-Path (ECMP) routing of different measurement streams if it | |||
meets the following criteria: | meets the following criteria: | |||
o Inner IP packets from different measurement implementations are | o Inner IP packets from different measurement implementations are | |||
mapped into a single tunnel with single outer IP origin and | mapped into a single tunnel with single outer IP origin and | |||
destination address as well as origing and destination port | destination address as well as origin and destination port numbers | |||
numbers which are identical for all packets. | which are identical for all packets. | |||
o An easily accessible commodity tunneling protocol allows to carry | o An easily accessible commodity tunneling protocol allows to carry | |||
out a metric test from more test sites. | out a metric test from more test sites. | |||
o A low operational overhead may enable a broader audience to set up | o A low operational overhead may enable a broader audience to set up | |||
a metric test with the desired properties. | a metric test with the desired properties. | |||
o The tunneling protocol should be reliable and stable in set up and | o The tunneling protocol should be reliable and stable in set up and | |||
operation to avoid disturbances or influence on the test results. | operation to avoid disturbances or influence on the test results. | |||
o The tunneling protocol should not incurr any extra cost for those | o The tunneling protocol should not incur any extra cost for those | |||
interested in setting up a metric test. | interested in setting up a metric test. | |||
An illustration of a test setup with two tunnels and two flows | An illustration of a test setup with two tunnels and two flows | |||
between two linecards of one implementation is given in Figure 1. | between two linecards of one implementation is given in Figure 1. | |||
Implementation ,---. +--------+ | Implementation ,---. +--------+ | |||
+~~~~~~~~~~~/ \~~~~~~| Remote | | +~~~~~~~~~~~/ \~~~~~~| Remote | | |||
+------->-----F2->-| / \ |->---+ | | +------->-----F2->-| / \ |->---+ | | |||
| +---------+ | Tunnel 1( ) | | | | | +---------+ | Tunnel 1( ) | | | | |||
| | transmit|-F1->-| ( ) |->+ | | | | | transmit|-F1->-| ( ) |->+ | | | |||
skipping to change at page 14, line 21 | skipping to change at page 14, line 21 | |||
The applicability of one or more of the following tunneling protocols | The applicability of one or more of the following tunneling protocols | |||
may be investigated by interested parties if Ethernet over L2TPv3 is | may be investigated by interested parties if Ethernet over L2TPv3 is | |||
felt to be not suitable: IP in IP [RFC2003] or Generic Routing | felt to be not suitable: IP in IP [RFC2003] or Generic Routing | |||
Encapsulation (GRE) [RFC2784]. RFC 4928 [RFC4928] proposes measures | Encapsulation (GRE) [RFC2784]. RFC 4928 [RFC4928] proposes measures | |||
how to avoid ECMP treatment in MPLS networks. | how to avoid ECMP treatment in MPLS networks. | |||
L2TP is a commodity tunneling protocol [RFC2661]. By the time of | L2TP is a commodity tunneling protocol [RFC2661]. By the time of | |||
writing, L2TPv3 [RFC3931]is the latest version of L2TP. If L2TPv3 is | writing, L2TPv3 [RFC3931]is the latest version of L2TP. If L2TPv3 is | |||
applied, software based implementations of this protocol are not | applied, software based implementations of this protocol are not | |||
suitable for the test set up, as such implementations may cause | suitable for the test set up, as such implementations may cause | |||
uncalculable delay shifts. | incalculable delay shifts. | |||
Ethernet Pseudo Wires may also be set up on MPLS networks [RFC4448]. | Ethernet Pseudo Wires may also be set up on MPLS networks [RFC4448]. | |||
While there's no technical issue with this solution, MPLS interfaces | While there's no technical issue with this solution, MPLS interfaces | |||
are mostly found in the network provider domain. Hence not all of | are mostly found in the network provider domain. Hence not all of | |||
the above tunneling criteria are met. | the above tunneling criteria are met. | |||
Appendix C provides an experimental tunneling set up for metric | Appendix C provides an experimental tunneling set up for metric | |||
implementation testing between two (or more) remote sites. | implementation testing between two (or more) remote sites. | |||
Each test is repeated several times. WAN conditions may change over | Each test SHOULD be conducted multiple times. Sequential testing is | |||
time. Sequential testing is desirable, but may not be a useful | possible, but may not be a useful metric test option because WAN | |||
metric test option. It is RECOMMENDED that tests be carried out by | conditions are likely to change over time. It is RECOMMENDED that | |||
establishing N different parallel measurement flows. Two or three | tests be carried out by establishing at least 2 different parallel | |||
linecards per implementation serving to send or receive measurement | measurement flows. Two linecards per implementation that send and | |||
flows should be sufficient to create 5 or more parallel measurement | receive measurement flows should be sufficient to create 4 parallel | |||
flows. If three linecards are used, each card sends and receives 2 | measurement flows (when each card sends and receives 2 flows). Other | |||
flows. Other options are to separate flows by DiffServ marks | options are to separate flows by DiffServ marks (without deploying | |||
(without deploying any QoS in the inner or outer tunnel) or using a | any QoS in the inner or outer tunnel) or using a single CBR flow and | |||
single CBR flow and evaluating every n-th singleton to belong to a | evaluating every n-th singleton to belong to a specific measurement | |||
specific measurement flow. | flow. | |||
Some additional rules to calculate and compare samples have to be | Some additional rules to calculate and compare samples have to be | |||
respected to perform a metric test: | respected to perform a metric test: | |||
o To compare different probes of a common underlying distribution in | o To compare different probes of a common underlying distribution in | |||
terms of metrics characterising a communication network requires | terms of metrics characterising a communication network requires | |||
to respect the temporal nature for which the assumption of common | to respect the temporal nature for which the assumption of common | |||
underlying distribution may hold. Any singletons or samples to be | underlying distribution may hold. Any singletons or samples to be | |||
compared MUST be captured within the same time interval. | compared MUST be captured within the same time interval. | |||
skipping to change at page 15, line 28 | skipping to change at page 15, line 28 | |||
implementation. Note that the Anderson-Darling test detects small | implementation. Note that the Anderson-Darling test detects small | |||
differences in distributions fairly well and will fail for high | differences in distributions fairly well and will fail for high | |||
number of compared results (RFC2330 mentions an example with 8192 | number of compared results (RFC2330 mentions an example with 8192 | |||
measurements where an Anderson-Darling test always failed). | measurements where an Anderson-Darling test always failed). | |||
o Generally, the Anderson-Darling test is sensitive to differences | o Generally, the Anderson-Darling test is sensitive to differences | |||
in the accuracy or bias associated with varying implementations or | in the accuracy or bias associated with varying implementations or | |||
test conditions. These dissimilarities may result in differing | test conditions. These dissimilarities may result in differing | |||
averages of samples to be compared. An example may be different | averages of samples to be compared. An example may be different | |||
packet sizes, resulting in a constant delay difference between | packet sizes, resulting in a constant delay difference between | |||
compared samples. Therefore samples to be compared by an Anderson | compared samples. Therefore samples to be compared by an | |||
Darling test MAY be calibrated by the difference of the average | Anderson-Darling test MAY be calibrated by the difference of the | |||
values of the samples. Any calibration of this kind MUST be | average values of the samples. Any calibration of this kind MUST | |||
documented in the test result. | be documented in the test result. | |||
3.3. Tests of two or more different implementations against a metric | 3.3. Tests of two or more different implementations against a metric | |||
specification | specification | |||
RFC2330 expects "a methodology for a given metric [to] exhibit | RFC2330 expects "a methodology for a given metric [to] exhibit | |||
continuity if, for small variations in conditions, it results in | continuity if, for small variations in conditions, it results in | |||
small variations in the resulting measurements. Slightly more | small variations in the resulting measurements. Slightly more | |||
precisely, for every positive epsilon, there exists a positive delta, | precisely, for every positive epsilon, there exists a positive delta, | |||
such that if two sets of conditions are within delta of each other, | such that if two sets of conditions are within delta of each other, | |||
then the resulting measurements will be within epsilon of each | then the resulting measurements will be within epsilon of each | |||
other." A small variation in conditions in the context of the metric | other." A small variation in conditions in the context of the metric | |||
test proposed here can be seen as different implementations measuring | test proposed here can be seen as different implementations measuring | |||
the same metric along the same path. | the same metric along the same path. | |||
IPPM metric specification however allow for implementor options to | IPPM metric specifications however allow for implementor options to | |||
the largest possible degree. It can't be expected that two | the largest possible degree. It can not be expected that two | |||
implementors pick identical options for the implementations. | implementors pick identical value ranges in options for the | |||
Implementors SHOULD to the highest degree possible pick the same | implementations. Implementors SHOULD to the highest degree possible | |||
configurations for their systems when comparing their implementations | pick the same configurations for their systems when comparing their | |||
by a metric test. | implementations by a metric test. | |||
In some cases, a goodness of fit test may not be possible or show | In some cases, a goodness of fit test may not be possible or show | |||
disappointing results. To clarify the difficulties arising from | disappointing results. To clarify the difficulties arising from | |||
different implementation options, the individual options picked for | different implementation options, the individual options picked for | |||
every compared implementation SHOULD be documented in sufficient | every compared implementation SHOULD be documented in sufficient | |||
detail. Based on this documentation, the underlying metric | detail. Based on this documentation, the underlying metric | |||
specification should be improved before it is promoted to a standard. | specification should be improved before it is promoted to a standard. | |||
The same statistical test as applicable to quantify precision of a | The same statistical test as applicable to quantify precision of a | |||
single metric implementation MUST be passed to compare metric | single metric implementation MUST be used to compare metric result | |||
conformance of different implementations. To document compatibility, | equivalence for different implementations. To document | |||
the smallest measurement resolution at which the compared | compatibility, the smallest measurement resolution at which the | |||
implementations passed the ADK sample test MUST be documented. | compared implementations passed the ADK sample test MUST be | |||
documented. | ||||
For different implementations of the same metric, "variations in | For different implementations of the same metric, "variations in | |||
conditions" are reasonably expected. The ADK test comparing samples | conditions" are reasonably expected. The ADK test comparing samples | |||
of the different implementations may result in a lower precision than | of the different implementations MAY result in a lower precision than | |||
the test for precision of each implementation individually. | the test for precision in the same-implementation comparison. | |||
3.4. Clock synchronisation | 3.4. Clock synchronisation | |||
Clock synchronization effects require special attention. Accuracy of | Clock synchronization effects require special attention. Accuracy of | |||
one-way active delay measurements for any metrics implementation | one-way active delay measurements for any metrics implementation | |||
depends on clock synchronization between the source and destination | depends on clock synchronization between the source and destination | |||
of tests. Ideally, one-way active delay measurement (RFC 2679, | of tests. Ideally, one-way active delay measurement (RFC 2679, | |||
[RFC2679]) test endpoints either have direct access to independent | [RFC2679]) test endpoints either have direct access to independent | |||
GPS or CDMA-based time sources or indirect access to nearby NTP | GPS or CDMA-based time sources or indirect access to nearby NTP | |||
primary (stratum 1) time sources, equipped with GPS receivers. | primary (stratum 1) time sources, equipped with GPS receivers. | |||
skipping to change at page 17, line 11 | skipping to change at page 17, line 12 | |||
Examination of the second condition requires RTT measurement for | Examination of the second condition requires RTT measurement for | |||
reference, e.g., based on TWAMP (RFC5357, RFC 5357 [RFC5357]), in | reference, e.g., based on TWAMP (RFC5357, RFC 5357 [RFC5357]), in | |||
conjunction with one-way delay measurement. | conjunction with one-way delay measurement. | |||
Specification of X% to strike a balance between identification of | Specification of X% to strike a balance between identification of | |||
unreliable one-way delay samples and misidentification of reliable | unreliable one-way delay samples and misidentification of reliable | |||
samples under a wide range of Internet path RTTs probably requires | samples under a wide range of Internet path RTTs probably requires | |||
further study. | further study. | |||
An IPPM compliant metric implementation whose measurement requires | An implementation of an RFC that requires synchronized clocks is | |||
synchronized clocks is however expected to provide precise | expected to provide precise measurement results in order to claim | |||
measurement results. Any IPPM metric implementation SHOULD be of a | that the metric measured is compliant. | |||
precision of 1 ms (+/- 500 us) with a confidence of 95% if the metric | ||||
is captured along an Internet path which is stable and not congested | IF an implementation publishes a specification of its precision, such | |||
during a measurement duration of an hour or more. | as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then | |||
the specification SHOULD be met over a useful measurement duration. | ||||
For example, if the metric is measured along an Internet path which | ||||
is stable and not congested, then the precision specification SHOULD | ||||
be met over durations of an hour or more. | ||||
3.5. Recommended Metric Verification Measurement Process | 3.5. Recommended Metric Verification Measurement Process | |||
In order to meet their obligations under the IETF Standards Process | In order to meet their obligations under the IETF Standards Process | |||
the IESG must be convinced that each metric specification advanced to | the IESG must be convinced that each metric specification advanced to | |||
Draft Standard or Internet Standard status is clearly written, that | Draft Standard or Internet Standard status is clearly written, that | |||
there are the required multiple verifiably equivalent | there are the a sufficient number of verified equivalent | |||
implementations, and that all options have been implemented. | implementations, and that all options have been implemented. | |||
In the context of this document, metrics are designed to measure some | In the context of this document, metrics are designed to measure some | |||
characteristic of a data network. An aim of any metric definition | characteristic of a data network. An aim of any metric definition | |||
should be that it should be specified in a way that can reliably | should be that it should be specified in a way that can reliably | |||
measure the specific characteristic in a repeatable way across | measure the specific characteristic in a repeatable way across | |||
multiple independent implementations. | multiple independent implementations. | |||
Each metric, statistic or option of those to be validated MUST be | Each metric, statistic or option of those to be validated MUST be | |||
compared against a reference measurement or another implementation by | compared against a reference measurement or another implementation by | |||
skipping to change at page 21, line 19 | skipping to change at page 21, line 24 | |||
up than described here. Spatial and temporal effects combine in the | up than described here. Spatial and temporal effects combine in the | |||
case of packet re-ordering and measurements with different packet | case of packet re-ordering and measurements with different packet | |||
rates may always lead to different results. | rates may always lead to different results. | |||
As specified above, 5 singletons are the recommended basis to | As specified above, 5 singletons are the recommended basis to | |||
minimise interference of random events with the statistical test | minimise interference of random events with the statistical test | |||
proposed by this document. In the case of ratio measurements (like | proposed by this document. In the case of ratio measurements (like | |||
packet loss), the underlying sum of basic events, against the which | packet loss), the underlying sum of basic events, against the which | |||
the metric's monitored singletons are "rated", determines the | the metric's monitored singletons are "rated", determines the | |||
resolution of the test. A packet loss statistic with a resolution of | resolution of the test. A packet loss statistic with a resolution of | |||
1% requires one packet loss statistic-datapoint to consist of 500 | 1% requires one packet loss statistic-data point to consist of 500 | |||
delay singletons (of which at least 5 were lost). To compare EDFs on | delay singletons (of which at least 5 were lost). To compare EDFs on | |||
packet loss requires one hundred such statistics per flow. That | packet loss requires one hundred such statistics per flow. That | |||
means, all in all at least 50 000 delay singletons are required per | means, all in all at least 50 000 delay singletons are required per | |||
single measurement flow. Live network packet loss is assumed to be | single measurement flow. Live network packet loss is assumed to be | |||
present during main traffic hours only. Let this interval be 5 | present during main traffic hours only. Let this interval be 5 | |||
hours. The required minimum rate of a single measurement flow in | hours. The required minimum rate of a single measurement flow in | |||
that case is 2.8 packets/sec (assuming a loss of 1% during 5 hours). | that case is 2.8 packets/sec (assuming a loss of 1% during 5 hours). | |||
If this measurement is too demanding under live network conditions, | If this measurement is too demanding under live network conditions, | |||
an impairment generator should be used. | an impairment generator should be used. | |||
skipping to change at page 22, line 11 | skipping to change at page 22, line 16 | |||
singleton values, such as with a loss metric, or a duplication | singleton values, such as with a loss metric, or a duplication | |||
metric. Appendix A indicates how the ADK will work for 0ne-way | metric. Appendix A indicates how the ADK will work for 0ne-way | |||
delay, and should be likewise applicable to distributions of delay | delay, and should be likewise applicable to distributions of delay | |||
variation. | variation. | |||
Proposal: the implementation with the largest difference in | Proposal: the implementation with the largest difference in | |||
homogeneous comparison results is the lower bound on the equivalence | homogeneous comparison results is the lower bound on the equivalence | |||
threshold, noting that there may be other systematic errors to | threshold, noting that there may be other systematic errors to | |||
account for when comparing between implementations. | account for when comparing between implementations. | |||
Thus, when evaluationg equivalence in cross-implementation results: | Thus, when evaluating equivalence in cross-implementation results: | |||
Maximum_Error = Same_Implementation_Error + Systematic_Error | Maximum_Error = Same_Implementation_Error + Systematic_Error | |||
and only the systematic error need be decided beforehand. | and only the systematic error need be decided beforehand. | |||
In the case of ADK comparison, the largest same-implementation | In the case of ADK comparison, the largest same-implementation | |||
resolution of distribution equivalence can be used as a limit on | resolution of distribution equivalence can be used as a limit on | |||
cross-implementation resolutions (at the same confidence level). | cross-implementation resolutions (at the same confidence level). | |||
4. Acknowledgements | 4. Acknowledgements | |||
End of changes. 25 change blocks. | ||||
54 lines changed or deleted | 69 lines changed or added | |||
This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |