draft-ietf-tictoc-multi-path-synchronization-07.txt   rfc8039.txt 
Network Working Group A. Shpiner
Internet Draft Mellanox Internet Engineering Task Force (IETF) A. Shpiner
Intended status: Experimental R. Tse Request for Comments: 8039 Mellanox
Expires: April 2017 PMC-Sierra Category: Experimental R. Tse
ISSN: 2070-1721 Microsemi
C. Schelp C. Schelp
Oracle Oracle
T. Mizrahi T. Mizrahi
Marvell Marvell
October 27, 2016 December 2016
Multi-Path Time Synchronization Multipath Time Synchronization
draft-ietf-tictoc-multi-path-synchronization-07.txt
Abstract Abstract
Clock synchronization protocols are very widely used in IP-based Clock synchronization protocols are very widely used in IP-based
networks. The Network Time Protocol (NTP) has been commonly deployed networks. The Network Time Protocol (NTP) has been commonly deployed
for many years, and the last few years have seen an increasingly for many years, and the last few years have seen an increasingly
rapid deployment of the Precision Time Protocol (PTP). As time- rapid deployment of the Precision Time Protocol (PTP). As time-
sensitive applications evolve, clock accuracy requirements are sensitive applications evolve, clock accuracy requirements are
becoming increasingly stringent, requiring the time synchronization becoming increasingly stringent, requiring the time synchronization
protocols to provide high accuracy. This memo describes a multi-path protocols to provide high accuracy. This memo describes a multipath
approach to PTP and NTP over IP networks, allowing the protocols to approach to PTP and NTP over IP networks, allowing the protocols to
run concurrently over multiple communication paths between the master run concurrently over multiple communication paths between the master
and slave clocks, without modifying these protocols. The multi-path and slave clocks, without modifying these protocols. The multipath
approach can significantly contribute to clock accuracy, security and approach can significantly contribute to clock accuracy, security,
fault tolerance. The multi-path approach that is presented in this and fault tolerance. The multipath approach that is presented in
document enables backward compatibility with nodes that do not this document enables backward compatibility with nodes that do not
support the multi-path functionality. support the multipath functionality.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Status of This Memo
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at This document is not an Internet Standards Track specification; it is
http://www.ietf.org/ietf/1id-abstracts.txt. published for examination, experimental implementation, and
evaluation.
The list of Internet-Draft Shadow Directories can be accessed at This document defines an Experimental Protocol for the Internet
http://www.ietf.org/shadow.html. community. This document is a product of the Internet Engineering
Task Force (IETF). It represents the consensus of the IETF
community. It has received public review and has been approved for
publication by the Internet Engineering Steering Group (IESG). Not
all documents approved by the IESG are a candidate for any level of
Internet Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on April 27, 2017. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc8039.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction...................................................3 1. Introduction ....................................................3
2. Conventions Used in this Document..............................4 2. Conventions Used in This Document ...............................4
2.1. Abbreviations.............................................4 2.1. Abbreviations ..............................................4
2.2. Terminology...............................................5 2.2. Terminology ................................................4
3. Multiple Paths in IP Networks..................................5 3. Multiple Paths in IP Networks ...................................5
3.1. Load Balancing............................................5 3.1. Load Balancing .............................................5
3.2. Using Multiple Paths Concurrently.........................5 3.2. Using Multiple Paths Concurrently ..........................5
3.3. Two-Way Paths.............................................6 3.3. Two-Way Paths ..............................................5
4. Solution Overview..............................................6 4. Solution Overview ...............................................6
4.1. Path Configuration and Identification.....................6 4.1. Path Configuration and Identification ......................6
4.2. Combining.................................................7 4.2. Combining ..................................................6
5. Multi-Path Time Synchronization over IP Networks...............7 5. Multipath Time Synchronization over IP Networks .................7
5.1. Overview..................................................7 5.1. Overview ...................................................7
5.2. Single-Ended Multi-Path Synchronization...................8 5.2. Single-Ended Multipath Synchronization .....................8
5.2.1. Single-Ended MPPTP Synchronization Message Exchange..8 5.2.1. Single-Ended MPPTP Synchronization Message
5.2.2. Single-Ended MPNTP Synchronization Message Exchange..9 Exchange ............................................8
5.3. Dual-Ended Multi-Path Synchronization....................10 5.2.2. Single-Ended MPNTP Synchronization Message
5.3.1. Dual-Ended MPPTP Synchronization Message Exchange...10 Exchange ............................................9
5.3.2. Dual-Ended MPNTP Synchronization Message Exchange...11 5.3. Dual-Ended Multipath Synchronization ......................10
5.4. Using Traceroute for Path Discovery......................12 5.3.1. Dual-Ended MPPTP Synchronization Message Exchange ..10
5.5. Using Unicast Discovery for MPPTP........................13 5.3.2. Dual-Ended MPNTP Synchronization Message Exchange ..11
6. Combining Algorithm...........................................13 5.4. Using Traceroute for Path Discovery .......................12
7. Security Considerations.......................................14 5.5. Using Unicast Discovery for MPPTP .........................13
8. Scope of the Experiment.......................................14 6. Combining Algorithm ............................................13
9. IANA Considerations...........................................14 7. Security Considerations ........................................14
10. Acknowledgments..............................................15 8. Scope of the Experiment ........................................14
11. References...................................................15 9. References .....................................................15
11.1. Normative References....................................15 9.1. Normative References ......................................15
11.2. Informative References..................................15 9.2. Informative References ....................................15
Acknowledgments ...................................................17
Authors' Addresses ................................................17
1. Introduction 1. Introduction
The two most common time synchronization protocols in IP networks are The two most common time synchronization protocols in IP networks are
the Network Time Protocol [NTP], and the Precision Time Protocol (1) the Network Time Protocol [NTP] and (2) the Precision Time
(PTP), defined in the IEEE 1588 standard [IEEE1588]. Protocol (PTP) as defined in the IEEE 1588 standard [IEEE1588].
The accuracy of the time synchronization protocols directly depends The accuracy of the time synchronization protocols directly depends
on the stability and the symmetry of propagation delays on both on the stability and the symmetry of propagation delays in both
directions between the master and slave clocks. Depending on the directions between the master and slave clocks. Depending on the
nature of the underlying network, time synchronization protocol nature of the underlying network, time synchronization protocol
packets can be subject to variable network latency or path asymmetry packets can be subject to variable network latency or path asymmetry
(e.g. [ASSYMETRY], [ASSYMETRY2]). As time sensitive applications (e.g., [ASYMMETRY] [ASYMMETRY2]). As time-sensitive applications
evolve, accuracy requirements are becoming increasingly stringent. evolve, accuracy requirements are becoming increasingly stringent.
Using a single network path in a clock synchronization protocol Using a single network path in a clock synchronization protocol
closely ties the slave clock accuracy to the behavior of the specific closely ties the slave clock accuracy to the behavior of the specific
path, which may suffer from temporal congestion, faults or malicious path, which may suffer from temporal congestion, faults, or malicious
attacks. Relying on multiple clock servers as in NTP solves these attacks. Relying on multiple clock servers, as in NTP, solves these
problems, but requires active maintenance of multiple accurate problems but requires active maintenance of multiple accurate sources
sources in the network, which is not always possible. The usage of in the network, which is not always possible. The usage of
Transparent Clocks (TC) in PTP solves the congestion problem by Transparent Clocks (TCs) in PTP solves the congestion problem by
eliminating the queueing time from the delay calculations, but does eliminating the queuing time from the delay calculations but does not
not address security or fault-tolerance aspects. address security or fault-tolerance aspects.
____
______/ \_____
___/ \____
____/ \
____ / path 1 / ___
/ \ / ________________________ \ / \
/Master\____\___/ \____\________/Slave\
\Clock / / \________ _______________/ \ \Clock/
\____/ \ / \__ /
\____ path 2 __/
\_______ ______/
\_________/
Figure 1 Multi-Path Connection ____
______/ \_____
___/ \____
____/ \
____ / path 1 / ___
/ \ / ________________________ \ / \
/Master\____\___/ \____\________/Slave\
\Clock / / \________ _______________/ \ \Clock/
\____/ \ / \__ /
\____ path 2 __/
\_______ ______/
\_________/
Figure 1: Multipath Connection
Since master and slave clocks are often connected through more than Since master and slave clocks are often connected through more than
one path in the network, as shown in Figure 1, [SLAVEDIV] suggested one path in the network, as shown in Figure 1, [SLAVEDIV] suggested
that a time synchronization protocol can be run over multiple paths, that a time synchronization protocol can be run over multiple paths,
providing several advantages. First, it can significantly increase providing several advantages. First, it can significantly increase
the clock accuracy as shown in [SLAVEDIV]. Second, this approach the clock accuracy as shown in [SLAVEDIV]. Second, this approach
provides additional security, allowing to mitigate man-in-the-middle provides additional security, allowing the mitigation of
attacks against the time synchronization protocol [DELAY-ATT]. Third, man-in-the-middle attacks against the time synchronization protocol
using multiple paths concurrently provides an inherent failure [DELAY-ATT]. Third, using multiple paths concurrently provides an
protection mechanism. inherent failure protection mechanism.
This document introduces Multi-Path PTP (MPPTP) and Multi-Path NTP This document introduces Multipath PTP (MPPTP) and Multipath NTP
(MPNTP). The functionality of the multi-path approach is defined at (MPNTP). The functionality of the multipath approach is defined at
the network layer and does not require any changes in the PTP or in the network layer and does not require any changes in PTP or NTP.
the NTP protocols.
MPPTP and MPNTP are defined over IP networks. As IP networks MPPTP and MPNTP are defined over IP networks. As IP networks
typically combine ECMP routing, this property is leveraged for the typically combine ECMP routing, this property is leveraged for the
multiple paths used in MPPTP and MPNTP. The key property of the multiple paths used in MPPTP and MPNTP. The key property of the
multi-path approach is that clocks in the network can use more than multipath approach is that clocks in the network can use more than
one IP address. Each {master IP, slave IP} address pair defines a one IP address. Each {master IP, slave IP} address pair defines a
path. Depending on the network topology and configuration, the IP path. Depending on the network topology and configuration, the IP
combination pairs can form multiple diverse paths used by the multi- combination pairs can form multiple diverse paths used by the
path synchronization protocols. It has been shown [MULTI] that using multipath synchronization protocols. It has been shown [MULTI] that
multiple IP addresses over the wide Internet indeed allows two end- using multiple IP addresses over the wide Internet indeed allows two
points to attain multiple diverse paths. endpoints to attain multiple diverse paths.
This document introduces two variants of the multi-path approach; a This document introduces two variants of the multipath approach:
variant that requires both master and slave nodes to support the (1) a variant that requires both master and slave nodes to support
multi-path functionality, referred to as the dual-ended variant, and the multipath functionality, referred to as the dual-ended variant,
a backward compatible variant that allows a multi-path clock to and (2) a backward-compatible variant that allows a multipath clock
connect to a conventional single-path clock, referred to as the to connect to a conventional single-path clock, referred to as the
single-ended variant. single-ended variant.
2. Conventions Used in this Document 2. Conventions Used in This Document
2.1. Abbreviations 2.1. Abbreviations
BMC Best Master Clock [IEEE1588] BMC Best Master Clock [IEEE1588]
ECMP Equal Cost Multiple Path ECMP Equal-Cost Multipath
LAN Local Area Network LAN Local Area Network
MPNTP Multi-Path Network Time Protocol MPNTP Multipath Network Time Protocol
MPPTP Multipath Precision Time Protocol
MPPTP Multi-Path Precision Time Protocol
NTP Network Time Protocol [NTP] NTP Network Time Protocol [NTP]
PTP Precision Time Protocol [IEEE1588] PTP Precision Time Protocol [IEEE1588]
2.2. Terminology 2.2. Terminology
In the NTP terminology, a time synchronization protocol is run In the NTP terminology, a time synchronization protocol is run
between a client and a server, while PTP uses the terms master and between a client and a server, while PTP uses the terms 'master' and
slave. Throughout this document, the sections that refer to both PTP 'slave'. Throughout this document, the sections that refer to both
and NTP generically use the terms master and slave. PTP and NTP generically use the terms 'master' and 'slave'.
3. Multiple Paths in IP Networks 3. Multiple Paths in IP Networks
3.1. Load Balancing 3.1. Load Balancing
Traffic sent across IP networks is often load balanced across Traffic sent across IP networks is often load-balanced across
multiple paths. The load balancing decisions are typically based on multiple paths. The load-balancing decisions are typically based on
packet header fields: source and destination addresses, Layer 4 packet header fields: source and destination addresses, Layer 4
ports, the Flow Label field in IPv6, etc. ports, the Flow Label field in IPv6, etc.
Three common load balancing criteria are per-destination, per-flow
and per-packet. The per-destination load balancers take a load
balancing decision based on the destination IP address. Per-flow load
balancers use various fields in the packet header, e.g., IP addresses
and Layer 4 ports, for the load balancing decision. Per-packet load
balancers use flow-blind techniques such as round-robin without
basing the choice on the packet content.
3.2. Using Multiple Paths Concurrently Three common load-balancing criteria are per-destination, per-flow,
and per-packet. The per-destination load balancers take a
load-balancing decision based on the destination IP address.
Per-flow load balancers use various fields in the packet header,
e.g., IP addresses and Layer 4 ports, for the load-balancing
decision. Per-packet load balancers use flow-blind techniques such
as round-robin without basing the choice on the packet content.
To utilize the diverse paths that traverse per-destination load- 3.2. Using Multiple Paths Concurrently
balancers or per-flow load-balancers, the packet transmitter can vary
the IP addresses in the packet header. The analysis in [PARIS2] shows
that a significant majority of the flows on the internet traverse
per-destination or per-flow load-balancing. It presents statistics
that 72% of the flows traverse per-destination load balancing and 39%
of the flows traverse per-flow load-balancing, while only a
negligible part of the flows traverse per-packet load balancing.
These statistics show that the vast majority of the traffic on the
internet is load balanced based on packet header fields.
The approaches in this draft are based on varying the source and To utilize the diverse paths that traverse per-destination
destination IP addresses in the packet header. Possible extensions load balancers or per-flow load balancers, the packet transmitter can
have been considered that also vary the UDP ports. However some of vary the IP addresses in the packet header. The analysis in [PARIS2]
shows that a significant majority of the flows on the Internet
traverse per-destination or per-flow load balancing. It presents
statistics that 72% of the flows traverse per-destination
load balancing and 39% of the flows traverse per-flow load balancing,
while only a negligible part of the flows traverse per-packet
load balancing. These statistics show that the vast majority of the
traffic on the Internet is load-balanced based on packet header
fields.
The approaches in this document are based on varying the source and
destination IP addresses in the packet header. Possible extensions
have been considered that also vary the UDP ports. However, some of
the existing implementations of PTP and NTP use fixed UDP port values the existing implementations of PTP and NTP use fixed UDP port values
in both the source and destination UDP port fields, and thus do not in both the source and destination UDP port fields and thus do not
allow this approach. allow this approach.
3.3. Two-Way Paths 3.3. Two-Way Paths
A key property of IP networks is that packets forwarded from A to B A key property of IP networks is that packets forwarded from A to B
do not necessarily traverse the same path as packets from B to A. do not necessarily traverse the same path as packets from B to A.
Thus, we define a two-way path for a master-slave connection as a Thus, we define a two-way path for a master-slave connection as a
pair of one-way paths: the first from master to slave and the second pair of one-way paths: the first from master to slave and the second
from slave to master. from slave to master.
If possible, a traffic engineering approach can be used to verify If possible, a traffic engineering approach can be used to verify
that time synchronization traffic is always forwarded through that time synchronization traffic is always forwarded through
bidirectional two-way paths, i.e., that each two-way path uses the bidirectional two-way paths, i.e., that each two-way path uses the
same route on the forward and reverse directions, thus allowing same route in the forward and reverse directions, thus allowing
propagation time symmetry. However, in the general case two-way paths propagation time symmetry. However, in the general case, two-way
do not necessarily use the same path for the forward and reverse paths do not necessarily use the same path for the forward and
directions. reverse directions.
4. Solution Overview 4. Solution Overview
The multi-path time synchronization protocols we present are The multipath time synchronization protocols we present here are
comprised of two building blocks; one is the path configuration and comprised of two building blocks: one is the path configuration and
identification, and the other is the algorithm used by the slave to identification, and the other is the algorithm used by the slave to
combine the information received from the various paths. combine the information received from the various paths.
4.1. Path Configuration and Identification 4.1. Path Configuration and Identification
The master and slave clocks must be able to determine the path of The master and slave clocks must be able to determine the path of
transmitted protocol packets, and to identify the path of incoming transmitted protocol packets and to identify the path of incoming
protocol packets. A path is determined by a {master IP, slave IP} protocol packets. A path is determined by a {master IP, slave IP}
address pair. The synchronization protocol message exchange is run address pair. The synchronization protocol message exchange is run
independently through each path. independently through each path.
Each IP address pair defines a two-way path, and thus allows the Each IP address pair defines a two-way path and thus allows the
clocks to bind a transmitted packet to a specific path, or to clocks to bind a transmitted packet to a specific path or to identify
identify the path of an incoming packet. the path of an incoming packet.
If possible, the routing tables across the network should be If possible, the routing tables across the network should be
configured with multiple traffic engineered paths between the pair of configured with multiple traffic-engineered paths between the pair of
clocks. By carefully configuring the routers in such networks it is clocks. By carefully configuring the routers in such networks, it is
possible to create diverse paths for each of the IP address pairs possible to create diverse paths for each of the IP address pairs
between two clocks in the network. However, in public and provider between two clocks in the network. However, in public and provider
networks the load balancing behavior is hidden from the end users. In networks, the load-balancing behavior is hidden from the end users.
this case the actual number of paths may be less than the number of In this case, the actual number of paths may be less than the number
IP address pairs, since some of the address pairs may share common of IP address pairs, since some of the address pairs may share common
paths. paths.
4.2. Combining 4.2. Combining
Various methods can be used for combining the time information Various methods can be used for combining the time information
received from the different paths. The output of the combining received from the different paths. The output of the combining
algorithm is the accurate time offset. Combining methods are further algorithm is the accurate time offset. Combining methods are further
discussed in Section 6. discussed in Section 6.
5. Multi-Path Time Synchronization over IP Networks 5. Multipath Time Synchronization over IP Networks
5.1. Overview 5.1. Overview
This section presents two variants of MPPTP and MPNTP; single-ended This section presents two variants of MPPTP and MPNTP: single-ended
multi-path time synchronization and dual-ended multi-path time multipath time synchronization and dual-ended multipath time
synchronization. In the first variant, the multi-path approach is synchronization. In the first variant, the multipath approach is
only implemented by the slave and the master is not aware of its only implemented by the slave, and the master is not aware of its
usage. In the second variant, all clocks use multiple paths. usage. In the second variant, all clocks use multiple paths.
The dual-ended variant provides higher path diversity by using The dual-ended variant provides higher path diversity by using
multiple IP addresses at both ends, the master and slave, while the multiple IP addresses at both ends, the master and slave, while the
single-ended variant only uses multiple addresses at the slave. single-ended variant only uses multiple addresses at the slave.
Consequently, the single-ended approach is can interoperate with Consequently, the single-ended approach can interoperate with
existing implementations, which do not use multiple paths. The dual- existing implementations that do not use multiple paths. The
ended and single-ended approaches can co-exist in the same network; dual-ended and single-ended approaches can coexist in the same
each slave selects the connection(s) it wants to make with the network; each slave selects the connection(s) it wants to make with
available masters. A dual-ended slave could switch to single-ended the available masters. A dual-ended slave could switch to
mode if it does not see any dual-ended masters available. A single- single-ended mode if it does not see any dual-ended masters
ended slave could connect to a single IP address of a dual-ended available. A single-ended slave could connect to a single IP address
master. of a dual-ended master.
Multi-path time synchronization, in both variants, requires clocks to Multipath time synchronization, in both variants, requires clocks to
use multiple IP addresses. Using multiple IP addresses introduces a use multiple IP addresses. Using multiple IP addresses introduces a
tradeoff. A large number of IP addresses allows a large number of trade-off. A large number of IP addresses allows a large number of
diverse paths, providing the advantages of slave diversity discussed diverse paths, providing the advantages of slave diversity discussed
in Section 1. On the other hand, a large number of IP addresses is in Section 1. On the other hand, a large number of IP addresses is
more costly, requires the network topology to be more redundant, and more costly, requires the network topology to be more redundant, and
exacts extra management overhead. exacts extra management overhead.
If possible, the set of IP addresses for each clock should be chosen If possible, the set of IP addresses for each clock should be chosen
in a way that enables the establishment of paths that are the most in a way that enables the establishment of paths that are the most
different. If the load balancing rules in the network are known, it different. If the load-balancing rules in the network are known, it
is possible to choose the IP addresses in a way that enforces path is possible to choose the IP addresses in a way that enforces path
diversity. However, even if the load balancing scheme is not known, a diversity. However, even if the load-balancing scheme is not known,
careful choice of the IP addresses can increase the probability of a careful choice of the IP addresses can increase the probability of
path diversity. For example, choosing multiple addresses with path diversity. For example, choosing multiple addresses with
different prefixes is likely to produce higher path diversity, as BGP different prefixes is likely to produce higher path diversity, as BGP
routers are more likely to route these different prefixes through routers are more likely to route these different prefixes through
different routes. different routes.
The use of Network Address Translation (NAT) may significantly reduce The use of Network Address Translation (NAT) may significantly reduce
the effectiveness of multi-path synchronization in some cases. For the effectiveness of multipath synchronization in some cases. For
example, if a master uses multiple IP addresses that are translated example, if a master uses multiple IP addresses that are translated
to a single IP address, the path diversity can be dramatically to a single IP address, the path diversity can be dramatically
reduced compared to a network that does not use NAT. Thus, path reduced compared to a network that does not use NAT. Thus, path
discovery should be used to identify the possible paths between the discovery should be used to identify the possible paths between the
master and slave. Path discovery is further discussed in Section 5.4. master and slave. Path discovery is further discussed in
Section 5.4.
The concept of using multiple IP addresses or multiple interfaces is The concept of using multiple IP addresses or multiple interfaces is
a well-established concept that is being used today by various well established and is being used today by various applications and
applications and protocols, e.g., [MPTCP]. Using multiple interfaces protocols, e.g., [MPTCP]. Using multiple interfaces introduces some
introduces some challenges and issues, which were presented and challenges and issues, which were presented and discussed in [MIF].
discussed in [MIF].
The descriptions in this section refer to the end-to-end scheme of The descriptions in this section refer to the end-to-end scheme of
PTP, but are similarly applicable to the peer-to-peer scheme. MPNTP, PTP but are similarly applicable to the peer-to-peer scheme. MPNTP,
as described in this document, refers to the NTP client-server mode, as described in this document, refers to the NTP client-server mode,
although the concepts described here can be extended to include the although the concepts described here can be extended to include the
symmetric variant as well. symmetric variant as well.
Multi-path synchronization by nature requires protocol messages to be Multipath synchronization by nature requires protocol messages to be
sent as unicast. Specifically in PTP, the following messages must be sent as unicast. Specifically in PTP, the following messages must be
sent as unicast in MPPTP: Sync, Delay_Req, Delay_Resp, PDelay_Req, sent as unicast in MPPTP: Sync, Delay_Req, Delay_Resp, PDelay_Req,
PDelay_Resp, Follow_Up, and PDelay_Resp_Follow_Up. Note that PDelay_Resp, Follow_Up, and PDelay_Resp_Follow_Up. Note that
[IEEE1588] allows these messages to be sent either as multicast or as [IEEE1588] allows these messages to be sent either as multicast or as
unicast. unicast.
5.2. Single-Ended Multi-Path Synchronization 5.2. Single-Ended Multipath Synchronization
In the single-ended approach, only the slave is aware of the fact In the single-ended approach, only the slave is aware of the fact
that multiple paths are used, while the master is agnostic to the that multiple paths are used, while the master is agnostic to the
usage of multiple paths. This approach allows a hybrid network, where usage of multiple paths. This approach allows a hybrid network,
some of the clocks are multi-path clocks, and others are conventional where some of the clocks are multipath clocks and others are
one-path clocks. A single-ended multi-path clock presents itself to conventional one-path clocks. A single-ended multipath clock
the network as N independent clocks, using N IP addresses, as well as presents itself to the network as N independent clocks, using N IP
N clock identity values (in PTP). Thus, the usage of multiple slave addresses, as well as N clockIdentity [IEEE1588] values (in PTP).
identities by a slave clock is transparent from the master's point of Thus, the usage of multiple slave identities by a slave clock is
view, such that it treats each of the identities as a separate slave transparent from the master's point of view, such that it treats each
clock. of the identities as a separate slave clock.
5.2.1. Single-Ended MPPTP Synchronization Message Exchange 5.2.1. Single-Ended MPPTP Synchronization Message Exchange
The single-ended MPPTP message exchange procedure is as follows. The single-ended MPPTP message exchange procedure is as follows.
o Each single-ended MPPTP clock has a fixed set of N IP addresses o Each single-ended MPPTP clock has a fixed set of N IP addresses
and N corresponding clockIdentities. Each clock arbitrarily and N corresponding clockIdentities. Each clock arbitrarily
defines one of its IP addresses and clockIdentity values as the defines one of its IP addresses and clockIdentity values as the
clock primary identity. clock primary identity.
o A single-ended MPPTP port sends Announce messages only from its o A single-ended MPPTP port sends Announce messages only from its
primary identity, according to the BMC algorithm. primary identity, according to the BMC algorithm.
o The BMC algorithm at each clock determines the master, based on o The BMC algorithm at each clock determines the master, based on
the received Announce messages. the received Announce messages.
o A single-ended MPPTP port that is in the 'slave' state uses o A single-ended MPPTP port that is in the 'slave' state uses
unicast negotiation to request the master to transmit unicast unicast negotiation to request the master to transmit unicast
messages to each of the N slave clock identities. The slave port messages to each of the N slave clockIdentity values. The slave
periodically sends N Signaling messages to the master, using each port periodically sends N Signaling messages to the master, using
of its N identities. The Signaling message includes the each of its N identities. The Signaling message includes the
REQUEST_UNICAST_TRANSMISSION_TLV. REQUEST_UNICAST_TRANSMISSION TLV [IEEE1588].
o The master periodically sends unicast Sync messages from its o The master periodically sends unicast Sync messages from its
primary identity, identified by the sourcePortIdentity and IP primary identity, identified by the sourcePortIdentity [IEEE1588]
address, to each of the slave identities. and IP address, to each of the slave identities.
o The slave, upon receiving a Sync message, identifies its path o The slave, upon receiving a Sync message, identifies its path
according to the destination IP address. The slave sends a according to the destination IP address. The slave sends a
Delay_Req unicast message to the primary identity of the master. Delay_Req unicast message to the primary identity of the master.
The Delay_Req is sent using the slave identity corresponding to The Delay_Req is sent using the slave identity corresponding to
the path the Sync was received through. Note that the rate of the path through which the Sync was received. Note that the rate
Delay_Req messages may be lower than the Sync message rate, and of Delay_Req messages may be lower than the Sync message rate, and
thus a Sync message is not necessarily followed by a Delay_Req. thus a Sync message is not necessarily followed by a Delay_Req.
o The master, in response to a Delay_Req message from the slave, o The master, in response to a Delay_Req message from the slave,
responds with a Delay_Resp message using the IP address and responds with a Delay_Resp message using the IP address and
sourcePortIdentity from the Delay_Req message. sourcePortIdentity from the Delay_Req message.
o Upon receiving the Delay_Resp message, the slave identifies the o Upon receiving the Delay_Resp message, the slave identifies the
path using the destination IP address and the path using the destination IP address and the
requestingPortIdentity. The slave can then compute the requestingPortIdentity [IEEE1588]. The slave can then compute the
corresponding path delay and the offset from the master. corresponding path delay and the offset from the master.
o The slave combines the information from all negotiated paths. o The slave combines the information from all negotiated paths.
5.2.2. Single-Ended MPNTP Synchronization Message Exchange 5.2.2. Single-Ended MPNTP Synchronization Message Exchange
The single-ended MPNTP message exchange procedure is as follows. The single-ended MPNTP message exchange procedure is as follows.
o A single-ended MPNTP client has N separate identities, i.e., N IP o A single-ended MPNTP client has N separate identities, i.e., N IP
addresses. The assumption is that the server information, addresses. The assumption is that the server information,
including its IP address is known to the NTP clients. This is a including its IP address, is known to the NTP clients. This is a
fair assumption, as typically the address(es) of the NTP server(s) fair assumption, as typically the address(es) of the NTP server(s)
are provided to the NTP client by configuration. is provided to the NTP client by configuration.
o A single-ended MPNTP client initiates the NTP protocol with an NTP
server N times, using each of its N identities.
o The NTP protocol is maintained between the server and each of the o A single-ended MPNTP client initiates NTP with an NTP server
N client identities. N times, using each of its N identities.
o The client sends NTP messages to the master using each of its N o NTP is maintained between the server and each of the N client
identities. identities.
o The client sends NTP messages to the master using each of its
N identities.
o The server responds to the client's NTP messages using the IP o The server responds to the client's NTP messages using the IP
address from the received NTP packet. address from the received NTP packet.
o The client, upon receiving an NTP packet, uses the IP destination o The client, upon receiving an NTP packet, uses the IP destination
address to identify the path it came through, and uses the time address to identify the path through which it came, and it uses
information accordingly. the time information accordingly.
o The client combines the information from all paths. o The client combines the information from all paths.
5.3. Dual-Ended Multi-Path Synchronization 5.3. Dual-Ended Multipath Synchronization
In dual-ended multi-path synchronization each clock has N IP In dual-ended multipath synchronization, each clock has N IP
addresses. Time synchronization messages are exchanged between some addresses. Time synchronization messages are exchanged between some
of the combinations of {master IP, slave IP} addresses, allowing of the combinations of {master IP, slave IP} addresses, allowing
multiple paths between the master and slave. Note that the actual multiple paths between the master and slave. Note that the actual
number of paths between the master and slave may be less than the number of paths between the master and slave may be less than the
number of chosen {master, slave} IP address pairs. number of chosen {master IP, slave IP} address pairs.
Once the multiple two-way connections are established, a separate Once the multiple two-way connections are established, a separate
synchronization protocol exchange instance is run through each of synchronization protocol exchange instance is run through each
them. of them.
5.3.1. Dual-Ended MPPTP Synchronization Message Exchange 5.3.1. Dual-Ended MPPTP Synchronization Message Exchange
The dual-ended MPPTP message exchange procedure is as follows. The dual-ended MPPTP message exchange procedure is as follows.
o Every clock has N IP addresses, but uses a single clockIdentity. o Every clock has N IP addresses but uses a single clockIdentity.
o The BMC algorithm at each clock determines the master. The master o The BMC algorithm at each clock determines the master. The master
is identified by its clockIdentity, allowing other clocks to know is identified by its clockIdentity, allowing other clocks to know
the multiple IP addresses it uses. the multiple IP addresses it uses.
o When a clock sends an Announce message, it sends it from each of o When a clock sends an Announce message, it sends it from each of
its IP addresses with its clockIdentity. its IP addresses with its clockIdentity.
o A dual-ended MPPTP port that is in the 'slave' state uses unicast o A dual-ended MPPTP port that is in the 'slave' state uses unicast
negotiation to request the master to transmit unicast messages to negotiation to request the master to transmit unicast messages to
some or all of its N_s IP addresses. This negotiation is done some or all of its N_s IP addresses. This negotiation is done
individually between a slave IP address and the corresponding individually between a slave IP address and the corresponding
master IP address that the slave desires a connection with. The master IP address with which the slave desires a connection. The
slave port periodically sends Signaling messages to the master, slave port periodically sends Signaling messages to the master,
using some or all of its N_s IP addresses as source, to the using some or all of its N_s IP addresses as the source, to the
corresponding master's N_m IP addresses. The Signaling message corresponding master's N_m IP addresses. The Signaling message
includes the REQUEST_UNICAST_TRANSMISSION_TLV. includes the REQUEST_UNICAST_TRANSMISSION TLV [IEEE1588].
('N_s' and 'N_m' indicate the number of IP addresses of the slave
and master, respectively.)
o The master periodically sends unicast Sync messages from each of o The master periodically sends unicast Sync messages from each of
its IP addresses to the corresponding slave IP addresses for which its IP addresses to the corresponding slave IP addresses for which
a unicast connection was negotiated. a unicast connection was negotiated.
o The slave, upon receiving a Sync message, identifies its path o The slave, upon receiving a Sync message, identifies its path
according to the {source, destination} IP addresses. The slave according to the {source IP, destination IP} addresses. The slave
sends a Delay_Req unicast message, swapping the source and sends a Delay_Req unicast message, swapping the source and
destination IP addresses from the Sync message. Note that the rate destination IP addresses from the Sync message. Note that the
of Delay_Req messages may be lower than the Sync message rate, and rate of Delay_Req messages may be lower than the Sync message
thus a Sync message is not necessarily followed by a Delay_Req. rate, and thus a Sync message is not necessarily followed by a
Delay_Req.
o The master, in response to a Delay_Req message from the slave, o The master, in response to a Delay_Req message from the slave,
responds with a Delay_Resp message using the sourcePortIdentity responds with a Delay_Resp message using the sourcePortIdentity
from the Delay_Req message, and swapping the IP addresses from the from the Delay_Req message and swapping the IP addresses from the
Delay_Req. Delay_Req.
o Upon receiving the Delay_Resp message, the slave identifies the o Upon receiving the Delay_Resp message, the slave identifies the
path using the {source, destination} IP address pair. The slave path using the {source IP, destination IP} address pair. The
can then compute the corresponding path delay and the offset from slave can then compute the corresponding path delay and the offset
the master. from the master.
o The slave combines the information from all negotiated paths. o The slave combines the information from all negotiated paths.
5.3.2. Dual-Ended MPNTP Synchronization Message Exchange 5.3.2. Dual-Ended MPNTP Synchronization Message Exchange
The MPNTP message exchange procedure is as follows. The MPNTP message exchange procedure is as follows.
o Each NTP clock has a set of N IP addresses. The assumption is that o Each NTP clock has a set of N IP addresses. The assumption is
the server information, including its multiple IP addresses is that the server information, including its multiple IP addresses,
known to the NTP clients. is known to the NTP clients.
o The MPNTP client chooses N_svr of the N server IP addresses and o The MPNTP client chooses N_svr server IP addresses and N_c client
N_c of the N client IP addresses and initiates the N_svr*N_c IP addresses and initiates the N_svr*N_c instances of the
instances of the protocol, one for each {server IP, client IP} protocol, one for each {server IP, client IP} address pair,
pair, allowing the client to combine the information from the allowing the client to combine the information from the N_s*N_c
N_s*N_c paths. paths.
(N_svr and N_c indicate the number of IP addresses of the server
and client, respectively, which a client chooses to connect with) ('N_svr' and 'N_c' indicate the number of IP addresses of the
server and client, respectively, with which a client chooses to
connect.)
o The client sends NTP messages to the master using each of the o The client sends NTP messages to the master using each of the
source-destination address combinations. source-destination address combinations.
o The server responds to the client's NTP messages using the IP o The server responds to the client's NTP messages using the IP
address combination from the received NTP packet. address combination from the received NTP packet.
o Using the {source, destination} IP address pair in the received o Using the {source IP, destination IP} address pair in the received
packets, the client identifies the path, and performs its packets, the client identifies the path and performs its
computations for each of the paths accordingly. computations for each of the paths accordingly.
o The client combines the information from all paths. o The client combines the information from all paths.
5.4. Using Traceroute for Path Discovery 5.4. Using Traceroute for Path Discovery
The approach described thus far uses multiple IP addresses in a The approach described thus far uses multiple IP addresses in a
single clock to create multiple paths. However, although each two-way single clock to create multiple paths. However, although each
path is defined by a different {master, slave} address pair, some of two-way path is defined by a different {master IP, slave IP} address
the IP address pairs may traverse exactly the same network path, pair, some of the IP address pairs may traverse exactly the same
making them redundant. network path, making them redundant.
Traceroute-based path discovery can be used for filtering only the IP Traceroute-based path discovery can be used for filtering only the IP
addresses that obtain diverse paths. 'Paris Traceroute' [PARIS] and addresses that obtain diverse paths. 'Paris traceroute' [PARIS] and
'TraceFlow' [TRACEFLOW] are examples of tools that discover the paths 'TraceFlow' [TRACEFLOW] are examples of tools that discover the paths
between two points in the network. It should be noted that this between two points in the network. It should be noted that this
filtering approach is effective only if the Traceroute implementation filtering approach is effective only if the Traceroute implementation
uses the same IP addresses and UDP ports as the synchronization uses the same IP addresses and UDP ports as the synchronization
protocol packets. Since some Traceroute implementations vary the UDP protocol packets. Since some Traceroute implementations vary the UDP
ports, they may not be effective in identifying and filtering ports, they may not be effective in identifying and filtering
redundant paths in synchronization protocols. redundant paths in synchronization protocols.
The Traceroute-based filtering can be implemented by both master and Traceroute-based filtering can be implemented by both master and
slave nodes, or it can be restricted to run only on slave nodes to slave nodes, or it can be restricted to run only on slave nodes to
reduce the overhead on the master. For networks that guarantee that reduce the overhead on the master. For networks that guarantee that
the path of the timing packets in the forward and reverse direction the path of the timing packets in the forward and reverse directions
are the same, path discovery should only be performed at the slave. are the same, path discovery should only be performed at the slave.
Since network routes change over time, path discovery and redundant Since network routes change over time, path discovery and redundant
path filtering should be performed periodically. Two {master,slave} path filtering should be performed periodically. Two {master IP,
pairs that produce two diverse paths may be rerouted to use the same slave IP} address pairs that produce two diverse paths may be
paths. Thus, the set of addresses that are used by each clock should rerouted to use the same paths. Thus, the set of addresses that are
be reassessed regularly. used by each clock should be reassessed regularly.
5.5. Using Unicast Discovery for MPPTP 5.5. Using Unicast Discovery for MPPTP
As presented above, MPPTP uses Announce messages and the BMC As presented above, MPPTP uses Announce messages and the BMC
algorithm to discover the master. The unicast discovery option of PTP algorithm to discover the master. The unicast discovery option of
can be used as an alternative. PTP can be used as an alternative.
When using unicast discovery the MPPTP slave ports maintain a list of When using unicast discovery, the MPPTP slave ports maintain a list
the IP addresses of the master. The slave port uses unicast of the IP addresses of the master. The slave port uses unicast
negotiation to request unicast service from the master, as follows: negotiation to request unicast service from the master as follows:
o In single-ended MPPTP, the slave uses unicast negotiation from o In single-ended MPPTP, the slave uses unicast negotiation from
each of its identities to the master's (only) identity. each of its identities to the master's (only) identity.
o In dual-ended MPPTP, the slave uses unicast negotiation from its o In dual-ended MPPTP, the slave uses unicast negotiation from its
IP addresses, each to a corresponding master IP address to request IP addresses, each to a corresponding master IP address, to
unicast synchronization messages. request unicast synchronization messages.
Afterwards, the message exchange continues as described in sections Afterwards, the message exchange continues as described in
5.2.1. and 5.3.1. Sections 5.2.1 and 5.3.1.
The unicast discovery option can be used in networks that do not The unicast discovery option can be used in networks that do not
support multicast or in networks in which the master clocks are known support multicast or in networks in which the master clocks are known
in advance. In particular, unicast discovery avoids multicasting in advance. In particular, unicast discovery avoids multicasting
Announce messages. Announce messages.
6. Combining Algorithm 6. Combining Algorithm
Previous sections discussed the methods of creating the multiple Previous sections discussed the methods of creating the multiple
paths and obtaining the time information required by the slave paths and obtaining the time information required by the slave
algorithm. Once the time information is received through each of the algorithm. Once the time information is received through each of the
paths, the slave should use a combining algorithm, which consolidates paths, the slave should use a combining algorithm, which consolidates
the information from the different paths into a single clock. the information from the different paths into a single clock.
Various methods have been suggested for combining information from Various methods have been suggested for combining information from
different paths or from different clocks, e.g., [NTP], [SLAVEDIV], different paths or from different clocks, e.g., [NTP] [SLAVEDIV]
[HIGH-AVAI], [KALMAN]. The choice of the combining algorithm is local [HIGH-AVAI] [KALMAN]. The choice of the combining algorithm is local
to the slave, and does not affect interoperability. Hence, this to the slave and does not affect interoperability. Hence, this
document does not define a specific method to be used by the slave. document does not define a specific method to be used by the slave.
The combining algorithm should be chosen carefully based on the The combining algorithm should be chosen carefully based on the
system properties, as different combining algorithms provide system properties, as different combining algorithms provide
different advantages. For example, some combining algorithms (e.g., different advantages. For example, some combining algorithms (e.g.,
[NTP], [DELAY-ATT]) are intended to be robust in the face of security [NTP] [DELAY-ATT]) are intended to be robust in the face of security
attacks, while other combining algorithms (e.g., [KALMAN]) are more attacks, while other combining algorithms (e.g., [KALMAN]) are more
resilient to random delay variation. resilient to random delay variation.
7. Security Considerations 7. Security Considerations
The security aspects of time synchronization protocols are discussed The security aspects of time synchronization protocols are discussed
in detail in [RFC7384]. The methods described in this document in detail in [RFC7384]. The methods described in this document
propose to run a time synchronization protocol through redundant propose to run a time synchronization protocol through redundant
paths, and thus allow to detect and mitigate man-in-the-middle paths and thus allow the detection and mitigation of
attacks, as described in [DELAY-ATT]. Specifically, multi-path man-in-the-middle attacks, as described in [DELAY-ATT].
synchronization can mitigate the following threats (as per Specifically, multipath synchronization can mitigate the following
[RFC7384]): threats (as per [RFC7384]):
o Packet manipulation (Section 3.2.1 of [RFC7384]). o Packet manipulation (Section 3.2.1 of [RFC7384]).
o Packet Interception and Removal (Section 3.2.5 of [RFC7384]). o Packet interception and removal (Section 3.2.5 of [RFC7384]).
o Packet delay manipulation (Section 3.2.6 of [RFC7384]). o Packet delay manipulation (Section 3.2.6 of [RFC7384]).
It should be noted that when using multiple paths, these paths may It should be noted that when using multiple paths, these paths may
partially overlap, and thus an attack that takes place in a common partially overlap, and thus an attack that takes place in a common
segment of these paths is not mitigated by the redundancy. Moreover, segment of these paths is not mitigated by the redundancy. Moreover,
an on-path attacker may in some cases have access to more than one an on-path attacker may in some cases have access to more than one
router, or may be able to migrate from one router to another. router or may be able to migrate from one router to another.
Therefore, when using multiple paths it is important for the paths to Therefore, when using multiple paths, it is important for the paths
be as diverse and as independent as possible, making the redundancy to be as diverse and as independent as possible, making the
scheme more tolerant to on-path attacks. redundancy scheme more tolerant to on-path attacks.
It should be noted that the multi-path approach requires the master It should be noted that the multipath approach requires the master
(or NTP server) to dedicate more resources to each slave (client) (or NTP server) to dedicate more resources to each slave (client)
than the conventional single-path approach. Hence, well-known than the conventional single-path approach. Hence, well-known
Distributed Denial-of-Service (DDoS) attacks may porentially be Distributed Denial-of-Service (DDoS) attacks may potentially be
amplified when the multi-path approach is enabled. amplified when the multipath approach is enabled.
8. Scope of the Experiment 8. Scope of the Experiment
This memo is published as an experimental RFC. The purpose of the This memo is published as an Experimental RFC. The purpose of the
experimental period is to allow the community to analyze and to experimental period is to allow the community to analyze and to
verify the methods defined in this document. An experimental verify the methods defined in this document. An experimental
evaluation of some of these methods has been published in [MULTI]. It evaluation of some of these methods has been published in [MULTI].
is expected that the experimental period will allow the methods to be It is expected that the experimental period will allow the methods to
further investigated and verified by the community. The duration of be further investigated and verified by the community. The duration
the experiment is expected to be no less than two years from the of the experiment is expected to be no less than two years from the
publication of this document. publication of this document.
9. IANA Considerations 9. References
There are no IANA actions required by this document.
RFC Editor: please delete this section before publication.
10. Acknowledgments
The authors gratefully acknowledge the useful comments provided by
Peter Meyer, Doug Arnold, Joe Abley, Zhen Cao, Watson Ladd, and
Mirja Kuehlewind, as well as other comments received from the TICTOC
working group participants.
This document was prepared using 2-Word-v2.0.template.dot. 9.1. Normative References
11. References [IEEE1588] IEEE Instrumentation and Measurement Society, "IEEE
Standard for a Precision Clock Synchronization Protocol
for Networked Measurement and Control Systems", IEEE
Std 1588-2008, DOI 10.1109/IEEESTD.2008.4579760.
11.1. Normative References [NTP] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
"Network Time Protocol Version 4: Protocol and Algorithms
Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
<http://www.rfc-editor.org/info/rfc5905>.
[IEEE1588] IEEE Instrumentation and Measurement Society, "IEEE 9.2. Informative References
Standard for a Precision Clock Synchronization
Protocol for Networked Measurement and Control
Systems", IEEE Std 1588, 2008.
[NTP] D. Mills, J. Martin, J. Burbank, W. Kasch, "Network [ASYMMETRY]
Time Protocol Version 4: Protocol and Algorithms He, Y., Faloutsos, M., Krishnamurthy, S., and B. Huffaker,
Specification", IETF, RFC 5905, 2010. "On routing asymmetry in the Internet", IEEE GLOBECOM,
DOI 10.1109/GLOCOM.2005.1577769, December 2005.
11.2. Informative References [ASYMMETRY2]
Pathak, A., Pucha, H., Zhang, Y., Hu, C., and Z. Mao, "A
measurement study of internet delay asymmetry",
International Conference on Passive and Active Network
Measurement 2008, DOI 10.1007/978-3-540-79232-1_19,
April 2008.
[ASSYMETRY] Yihua He and Michalis Faloutsos and Srikanth [DELAY-ATT]
Krishnamurthy and Bradley Huffaker, "On routing Mizrahi, T., "A Game Theoretic Analysis of Delay Attacks
asymmetry in the internet", IEEE Globecom, 2005. against Time Synchronization Protocols", IEEE
International Symposium on Precision Clock Synchronization
for Measurement, Control and Communication (ISPCS),
DOI 10.1109/ISPCS.2012.6336612, September 2012.
[ASSYMETRY2] Abhinav Pathak, Himabindu Pucha, Ying Zhang, Y. [HIGH-AVAI]
Charlie Hu, and Z. Morley Mao, "A measurement study of Ferrari, P., Flammini, A., Rinaldi, S., and G. Prytz,
internet delay asymmetry", PAM'08, 2008. "High availability IEEE 1588 nodes over IEEE 802.1 aq
Shortest Path Bridging networks", IEEE International
Symposium on Precision Clock Synchronization for
Measurement, Control and Communication (ISPCS),
DOI 10.1109/ISPCS.2013.6644760, September 2013.
[DELAY-ATT] T. Mizrahi, "A Game Theoretic Analysis of Delay [KALMAN] Giorgi, G. and C. Narduzzi, "Kalman filtering for
Attacks against Time Synchronization Protocols", multi-path network synchronization", IEEE International
ISPCS, 2012. Symposium on Precision Clock Synchronization for
Measurement, Control and Communication (ISPCS),
DOI 10.1109/ISPCS.2014.6948693, September 2014.
[HIGH-AVAI] P. Ferrari, A. Flammini, S. Rinaldi, G. Prytz, "High [MIF] Blanchet, M. and P. Seite, "Multiple Interfaces and
availability IEEE 1588 nodes over IEEE 802.1 aq Provisioning Domains Problem Statement", RFC 6418,
Shortest Path Bridging networks" ISPCS, 2013. DOI 10.17487/RFC6418, November 2011,
<http://www.rfc-editor.org/info/rfc6418>.
[KALMAN] G. Giorgi, C. Narduzzi, "Kalman filtering for multi- [MPTCP] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
path network synchronization" ISPCS, 2014. "TCP Extensions for Multipath Operation with Multiple
Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013,
<http://www.rfc-editor.org/info/rfc6824>.
[MIF] Blanchet, M. and P. Seite, "Multiple Interfaces and [MULTI] Shpiner, A., Revah, Y., and T. Mizrahi, "Multi-path Time
Provisioning Domains Problem Statement", RFC 6418, DOI Protocols", IEEE International Symposium on Precision
10.17487/RFC6418, November 2011, <http://www.rfc- Clock Synchronization for Measurement, Control and
editor.org/info/rfc6418>. Communication (ISPCS), DOI 10.1109/ISPCS.2013.6644754,
September 2013.
[MPTCP] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, [PARIS] Augustin, B., Friedman, T., and R. Teixeira, "Measuring
"TCP Extensions for Multipath Operation with Multiple Load-balanced Paths in the Internet", 7th ACM SIGCOMM
Addresses", RFC 6824, DOI 10.17487/RFC6824, January conference on Internet measurement (IMC '07),
2013, <http://www.rfc-editor.org/info/rfc6824>. DOI 10.1145/1298306.1298329, October 2007.
[MULTI] A. Shpiner, Y. Revah, T. Mizrahi, "Multi-path Time [PARIS2] Augustin, B., Friedman, T., and R. Teixeira, "Measuring
Protocols" ISPCS, 2013. Multipath Routing in the Internet", IEEE/ACM Transactions
on Networking, 19(3), pp. 830-840,
DOI 10.1109/TNET.2010.2096232, June 2011.
[PARIS] Brice Augustin, Timur Friedman and Renata Teixeira, [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in
"Measuring Load-balanced Paths in the Internet", IMC, Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384,
2007. October 2014, <http://www.rfc-editor.org/info/rfc7384>.
[PARIS2] B. Augustin, T. Friedman, and R. Teixeira, "Measuring [SLAVEDIV] Mizrahi, T., "Slave Diversity: Using Multiple Paths to
Multipath Routing in the Internet", IEEE/ACM Improve the Accuracy of Clock Synchronization Protocols",
Transactions on Networking, 19(3), p. 830 - 840, June IEEE International Symposium on Precision Clock
2011. Synchronization for Measurement, Control and Communication
(ISPCS), DOI 10.1109/ISPCS.2012.6336621, September 2012.
[RFC7384] T. Mizrahi, "Security Requirements of Time Protocols [TRACEFLOW]
in Packet Switched Networks", IETF, RFC 7384, 2014. Narasimhan, J., Venkataswami, B., Groves, R., and P.
Hoose, "Traceflow", Work in Progress,
draft-janapath-intarea-traceflow-00, January 2012.
[SLAVEDIV] T. Mizrahi, "Slave Diversity: Using Multiple Paths to Acknowledgments
Improve the Accuracy of Clock Synchronization
Protocols", ISPCS, 2012.
[TRACEFLOW] J. Narasimhan, B. V. Venkataswami, R. Groves and P. The authors would like to thank Yoram Revah for his contribution to
Hoose, "Traceflow", IETF, draft-janapath-intarea- this work. The authors also gratefully acknowledge the useful
traceflow, work in progress, 2012. comments provided by Peter Meyer, Doug Arnold, Joe Abley, Zhen Cao,
Watson Ladd, and Mirja Kuehlewind, as well as other comments received
from the TICTOC working group participants.
Authors' Addresses Authors' Addresses
Alex Shpiner Alex Shpiner
Mellanox Technologies, Ltd. Mellanox Technologies, Ltd.
Hakidma 26 Hakidma 26
Ofer Industrial Park Ofer Industrial Park
Yokneam, 2069200, Israel Yokneam, 2069200
Israel
Email: alexshp@mellanox.com Email: alexshp@mellanox.com
Richard Tse Richard Tse
PMC-Sierra Microsemi
8555 Baxter Place 8555 Baxter Place
Burnaby, BC Burnaby, BC V5A 4V7
Canada Canada
V5A 4V7
Email: Richard.Tse@pmcs.com Email: Richard.Tse@microsemi.com
Craig Schelp Craig Schelp
Oracle Oracle
Email: craig.schelp@gmail.com Email: craig.schelp@oracle.com
Tal Mizrahi Tal Mizrahi
Marvell Marvell
6 Hamada St. 6 Hamada St.
Yokneam, 20692, Israel Yokneam, 2066721
Israel
Email: talmi@marvell.com Email: talmi@marvell.com
 End of changes. 158 change blocks. 
401 lines changed or deleted 425 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/