draft-bensley-tcpm-dctcp-00.txt   draft-bensley-tcpm-dctcp-01.txt 
Network Working Group S. Bensley Network Working Group S. Bensley
Internet-Draft Microsoft Internet-Draft Microsoft
Intended status: Standards Track L. Eggert Intended status: Informational L. Eggert
Expires: August 18, 2014 NetApp Expires: December 7, 2014 NetApp
D. Thaler D. Thaler
Microsoft Microsoft
February 14, 2014 June 5, 2014
Datacenter TCP (DCTCP): TCP Congestion Control for Datacenters Microsoft's Datacenter TCP (DCTCP):
draft-bensley-tcpm-dctcp-00 TCP Congestion Control for Datacenters
draft-bensley-tcpm-dctcp-01
Abstract Abstract
This memo describes Datacenter TCP (DCTCP), an improvement to TCP This memo describes Datacenter TCP (DCTCP), an improvement to TCP
congestion control for datacenter traffic. DCTCP enhances Explicit congestion control for datacenter traffic, as implemented in Windows
Congestion Notification (ECN) processing to estimate the fraction of Server 2012. DCTCP enhances Explicit Congestion Notification (ECN)
bytes that encounter congestion, rather than simply detecting that processing to estimate the fraction of bytes that encounter
some congestion has occurred. DCTCP then scales the TCP congestion congestion, rather than simply detecting that some congestion has
window based on this estimate. This method achieves high burst occurred. DCTCP then scales the TCP congestion window based on this
tolerance, low latency, and high throughput with shallow-buffered estimate. This method achieves high burst tolerance, low latency,
switches. and high throughput with shallow-buffered switches.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2014. This Internet-Draft will expire on December 7, 2014.
Copyright Notice Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. DCTCP Algorithm . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Marking Congestion on the Switch . . . . . . . . . . . . 3 3. DCTCP Algorithm . . . . . . . . . . . . . . . . . . . . . . . 3
2.2. Echoing Congestion Information on the Receiver . . . . . 4 3.1. Marking Congestion on the Switch . . . . . . . . . . . . 4
2.3. Processing Congestion Indications on the Sender . . . . . 4 3.2. Echoing Congestion Information on the Receiver . . . . . 4
3. Implementation Issues . . . . . . . . . . . . . . . . . . . . 6 3.3. Processing Congestion Indications on the Sender . . . . . 5
4. Deployment Issues . . . . . . . . . . . . . . . . . . . . . . 6 4. Implementation Issues . . . . . . . . . . . . . . . . . . . . 6
5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 5. Deployment Issues . . . . . . . . . . . . . . . . . . . . . . 7
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 6. Known Issues . . . . . . . . . . . . . . . . . . . . . . . . 7
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 7. Security Considerations . . . . . . . . . . . . . . . . . . . 8
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 7 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
8.1. Normative References . . . . . . . . . . . . . . . . . . 7 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 8
8.2. Informative References . . . . . . . . . . . . . . . . . 7 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 10.1. Normative References . . . . . . . . . . . . . . . . . . 8
10.2. Informative References . . . . . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9
1. Introduction 1. Introduction
Large datacenters necessarily need a large number of network switches Large datacenters necessarily need a large number of network switches
to interconnect the servers in the datacenter. Therefore, a to interconnect the servers in the datacenter. Therefore, a
datacenter can greatly reduce its capital expenditure by leveraging datacenter can greatly reduce its capital expenditure by leveraging
low cost switches. However, low cost switches tend to have limited low cost switches. However, low cost switches tend to have limited
queue capacities and thus are more susceptible to packet loss due to queue capacities and thus are more susceptible to packet loss due to
congestion. congestion.
Network traffic in the datacenter is often a mix of short and long Network traffic in the datacenter is often a mix of short and long
flows, where the short flows require low latency and the long flows flows, where the short flows require low latency and the long flows
require high throughput. Datacenters also experience incast bursts, require high throughput. Datacenters also experience incast bursts,
where many endpoints send traffic to a single server at the same where many endpoints send traffic to a single server at the same
time. For example, this is a natural consequence of MapReduce time. For example, this is a natural consequence of MapReduce
algorithms. The worker nodes complete at approximately the same algorithms. The worker nodes complete at approximately the same
time, and all reply to the master node concurrently. time, and all reply to the master node concurrently.
These factors place some conflicting demands on the switch's queue These factors place some conflicting demands on the queue occupancy
occupancy: of a switch:
o The queue must be short enough that it doesn't impose excessive o The queue must be short enough that it does not impose excessive
latency on short flows. latency on short flows.
o The queue must be long enough to buffer sufficient data for the o The queue must be long enough to buffer sufficient data for the
long flows to saturate the bandwidth. long flows to saturate the path bandwidth.
o The queue must be short enough to absorb incast bursts without o The queue must be short enough to absorb incast bursts without
excessive packet loss. excessive packet loss.
Standard TCP congestion control [RFC5681] relies on segment loss to Standard TCP congestion control [RFC5681] relies on segment loss to
detect congestion. This does not meet the demands described above. detect congestion. This does not meet the demands described above.
First, the short flows will start to experience unacceptable First, the short flows will start to experience unacceptable
latencies before packet loss occurs. Second, by the time TCP latencies before packet loss occurs. Second, by the time TCP
congestion control kicks in on the sender, most of the incast burst congestion control kicks in on the sender, most of the incast burst
has already been dropped. has already been dropped.
[RFC3168] describes a mechanism for using Explicit Congestion [RFC3168] describes a mechanism for using Explicit Congestion
Notification from the switch for early detection of congestion, Notification (ECN) from the switch for early detection of congestion,
rather than waiting for segment loss to occur. However, this method rather than waiting for segment loss to occur. However, this method
only detects the presence of congestion, not the extent. In the only detects the presence of congestion, not the extent. In the
presence of mild congestion, it reduces the TCP congestion window too presence of mild congestion, it reduces the TCP congestion window too
aggressively and unnecessarily affects the throughput of long flows. aggressively and unnecessarily affects the throughput of long flows.
Datacenter TCP (DCTCP) enhances Explicit Congestion Notification Datacenter TCP (DCTCP) enhances ECN processing to estimate the
(ECN) processing to estimate the fraction of bytes that encounter fraction of bytes that encounter congestion, rather than simply
congestion, rather than simply detecting that some congestion has detecting that some congestion has occurred. DCTCP then scales the
occurred. DCTCP then scales the TCP congestion window based on this TCP congestion window based on this estimate. This method achieves
estimate. This method achieves high burst tolerance, low latency, high burst tolerance, low latency, and high throughput with shallow-
and high throughput with shallow-buffered switches. buffered switches.
2. DCTCP Algorithm This document describes DCTCP as implemented in Microsoft Windows
Server 2012.
2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
3. DCTCP Algorithm
There are three components involved in the DCTCP algorithm: There are three components involved in the DCTCP algorithm:
o The switch (or other intermediate device on the network) detects o The switch (or other intermediate device on the network) detects
congestion and sets the Congestion Encountered (CE) codepoint in congestion and sets the Congestion Encountered (CE) codepoint in
the IP header. the IP header.
o The receiver echoes the congestion information back to the sender o The receiver echoes the congestion information back to the sender
using the ECN-Echo (ECE) flag in the TCP header. using the ECN-Echo (ECE) flag in the TCP header.
o The sender reacts to the congestion indication by reducing the TCP o The sender reacts to the congestion indication by reducing the TCP
congestion window (cwnd). congestion window (cwnd).
2.1. Marking Congestion on the Switch 3.1. Marking Congestion on the Switch
The switch indicates congestion to the end nodes by setting the CE The switch indicates congestion to the end nodes by setting the CE
codepoint in the IP header as specified in Section 5 of [RFC3168]. codepoint in the IP header as specified in Section 5 of [RFC3168].
For example, the switch may be configured with a congestion For example, the switch may be configured with a congestion
threshold. When a packet arrives at the switch and the switch's threshold. When a packet arrives at the switch and its queue length
queue length is greater than the congestion threshold, the switch is greater than the congestion threshold, the switch sets the CE
sets the CE codepoint in the packet. However, the actual algorithm codepoint in the packet. However, the actual algorithm for marking
for marking congestion is an implementation detail of the switch and congestion is an implementation detail of the switch and will
will generally not be known to the sender and receiver. generally not be known to the sender and receiver.
2.2. Echoing Congestion Information on the Receiver 3.2. Echoing Congestion Information on the Receiver
According to Section 6.1.3 of [RFC3168], the receiver sets the ECE According to Section 6.1.3 of [RFC3168], the receiver sets the ECE
flag if any of the packets being acknowledged had the CE code point flag if any of the packets being acknowledged had the CE code point
set. The receiver then continues to set the ECE flag until it set. The receiver then continues to set the ECE flag until it
receives a packet with the Congestion Window Reduced (CWR) flag set. receives a packet with the Congestion Window Reduced (CWR) flag set.
However, the DCTCP algorithm requires more detailed congestion However, the DCTCP algorithm requires more detailed congestion
information. In particular, the sender must be able to determine the information. In particular, the sender must be able to determine the
number of sent bytes that encountered congestion. Thus, the scheme number of sent bytes that encountered congestion. Thus, the scheme
described in [RFC3168] does not suffice. described in [RFC3168] does not suffice.
One possible solution is to ACK every packet and set the ECE flag in One possible solution is to ACK every packet and set the ECE flag in
the ACK if and only if the CE code point was set in the packet being the ACK if and only if the CE code point was set in the packet being
acknowledged. However, this prevents the use of delayed ACKs, which acknowledged. However, this prevents the use of delayed ACKs, which
are an important performance optimization in datacenters. Instead, are an important performance optimization in datacenters.
we introduce a new Boolean TCP state variable, DCTCP Congestion
Encountered (DCTCP.CE), which is initialized to false and stored in Instead, DCTCP introduces a new Boolean TCP state variable, DCTCP
the Transmission Control Block (TCB). When sending an ACK, the ECE Congestion Encountered (DCTCP.CE), which is initialized to false and
flag is set if and only if DCTCP.CE is true. When receiving packets, stored in the Transmission Control Block (TCB). When sending an ACK,
the CE codepoint is processed as follows: the ECE flag MUST be set if and only if DCTCP.CE is true. When
receiving packets, the CE codepoint MUST be processed as follows:
1. If the CE codepoint is set and DCTCP.CE is false, send an ACK for 1. If the CE codepoint is set and DCTCP.CE is false, send an ACK for
any previously unacknowledged packets and set DCTCP.CE to true. any previously unacknowledged packets and set DCTCP.CE to true.
2. If the CE codepoint is not set and DCTCP.CE is true, send an ACK 2. If the CE codepoint is not set and DCTCP.CE is true, send an ACK
for any previously unacknowledged packets and set DCTCP.CE to for any previously unacknowledged packets and set DCTCP.CE to
false. false.
3. Otherwise, the CE codepoint is ignored. 3. Otherwise, ignore the CE codepoint.
2.3. Processing Congestion Indications on the Sender 3.3. Processing Congestion Indications on the Sender
The sender estimates the fraction of sent bytes that encountered The sender estimates the fraction of sent bytes that encountered
congestion. The current estimate is stored in a new TCP state congestion. The current estimate is stored in a new TCP state
variable, DCTCP.Alpha, which is initialized to 1 and updated as variable, DCTCP.Alpha, which is initialized to 1 and MUST be updated
follows: as follows:
DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M
where where
o g is the estimation gain, a real number between 0 and 1. The o g is the estimation gain, a real number between 0 and 1. The
selection of g is left to the implementation. selection of g is left to the implementation. See Section 4 for
further considerations.
o M is the fraction of sent bytes that encountered congestion during o M is the fraction of sent bytes that encountered congestion during
the previous observation window, where the observation window is the previous observation window, where the observation window is
chosen to be approximately the Round Trip Time (RTT). chosen to be approximately the Round Trip Time (RTT). In
particular, an observation window ends when all the sent bytes in
Whenever the TCP congestion estimate is updated, the sender also flight at the beginning of the window have been acknowledged.
updates the TCP congestion window as follows:
cwnd = cwnd * (1 - DCTCP.Alpha / 2)
Thus, when there is no congestion at all, Alpha equals zero, and the
congestion window is left unchanged. When there is total congestion,
Alpha equals one, and the congestion window is reduced by half.
Lower levels of congestion will result in correspondingly lesser
reductions to the congestion window.
In order to update DCTCP.Alpha, we make use of the TCP state In order to update DCTCP.Alpha, the TCP state variables defined in
variables defined in [RFC0793], and introduce three additional TCP [RFC0793] are used, and three additional TCP state variables are
state variables: introduced:
o DCTCP.WindowEnd - The TCP sequence number threshold for beginning o DCTCP.WindowEnd: The TCP sequence number threshold for beginning a
a new observation window -- initialized to SND.UNA. new observation window; initialized to SND.UNA.
o DCTCP.BytesSent - The number of bytes sent during the current o DCTCP.BytesSent: The number of bytes sent during the current
window -- initialized to zero. observation window; initialized to zero.
o DCTCP.BytesMarked - The number of bytes sent during the current o DCTCP.BytesMarked: The number of bytes sent during the current
window that encountered congestion -- initialized to zero. observation window that encountered congestion; initialized to
zero.
The congestion estimator on the sender processes acceptable ACKs as The congestion estimator on the sender MUST process acceptable ACKs
follows: as follows:
1. Compute the bytes acknowledged: 1. Compute the bytes acknowledged (TCP SACK options [RFC2018] are
ignored):
BytesAcked = SEG.ACK - SND.UNA BytesAcked = SEG.ACK - SND.UNA
2. Update the bytes sent: 2. Update the bytes sent:
DCTCP.BytesSent += BytesAcked DCTCP.BytesSent += BytesAcked
3. If the ECE flag is set, update the bytes marked: 3. If the ECE flag is set, update the bytes marked:
DCTCP.BytesMarked += BytesAcked DCTCP.BytesMarked += BytesAcked
4. If the sequence number is less than or equal to DCTCP.WindowEnd, 4. If the sequence number is less than or equal to DCTCP.WindowEnd,
then stop processing. Otherwise, we've reached the end of the then stop processing. Otherwise, the end of the observation
observation window, so proceed to update the congestion estimate. window was reached, so proceed to update the congestion estimate
as follows:
5. Compute the congestion for the current window: 5. Compute the congestion level for the current observation window:
M = DCTCP.BytesMarked / DCTCP.BytesSent M = DCTCP.BytesMarked / DCTCP.BytesSent
6. Update the congestion estimate: 6. Update the congestion estimate:
DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M
7. Set the end of the new window: 7. Determine the end of the next observation window:
DCTCP.WindowEnd = SND.NXT DCTCP.WindowEnd = SND.NXT
8. Reset the byte counters: 8. Reset the byte counters:
DCTCP.BytesSent = DCTCP.BytesMarked = 0 DCTCP.BytesSent = DCTCP.BytesMarked = 0
3. Implementation Issues Rather than always halving the congestion window as described in
[RFC3168], when the sender receives an indication of congestion, the
sender MUST update cwnd as follows:
As noted in Section 2.3, the implementation must choose a suitable cwnd = cwnd * (1 - DCTCP.Alpha / 2)
Thus, when no sent byte experienced congestion, DCTCP.Alpha equals
zero, and cwnd is left unchanged. When all sent bytes experienced
congestion, DCTCP.Alpha equals one, and cwnd is reduced by half.
Lower levels of congestion will result in correspondingly smaller
reductions to cwnd.
Just as specified in [RFC3168], TCP should not react to congestion
indications more than once every window of data. The setting of the
"Congestion Window Reduced" (CWR) bit is also exactly as per
[RFC3168].
4. Implementation Issues
As noted in Section 3.3, the implementation must choose a suitable
estimation gain. [DCTCP10] provides a theoretical basis for estimation gain. [DCTCP10] provides a theoretical basis for
selecting the gain. However, it may be more practical to use selecting the gain. However, it may be more practical to use
experimentation to select a suitable gain for a particular network experimentation to select a suitable gain for a particular network
and workload. The Microsoft implementation of DCTCP in Windows and workload. The Microsoft implementation of DCTCP in Windows
Server 2012 uses a fixed estimation gain of 1/16. Server 2012 uses a fixed estimation gain of 1/16.
The implementation must also decide when to use DCTCP. Datacenter The implementation must also decide when to use DCTCP. Datacenter
servers may need to communicate with endpoints outside the servers may need to communicate with endpoints outside the
datacenter, where DCTCP is unsuitable or unsupported. Thus, a global datacenter, where DCTCP is unsuitable or unsupported. Thus, a global
configuration setting to enable DCTCP will generally not suffice. configuration setting to enable DCTCP will generally not suffice.
DCTCP may be configured based on the IP address of the remote DCTCP may be configured based on the IP address of the remote
endpoint. Microsoft Windows Server 2012 also supports automatic endpoint. Microsoft Windows Server 2012 also supports automatic
selection of DCTCP if the estimated RTT is less than 10 msec, under selection of DCTCP if the estimated RTT is less than 10 msec, under
the assumption that if the RTT is low, then the two endpoints are the assumption that if the RTT is low, then the two endpoints are
likely on the same datacenter network. likely on the same datacenter network.
4. Deployment Issues 5. Deployment Issues
Since DCTCP relies on congestion marking by the switch, DCTCP can Since DCTCP relies on congestion marking by the switch, DCTCP can
only be deployed in datacenters where the network infrastructure only be deployed in datacenters where the network infrastructure
supports ECN. The switches may also support configuration of the supports ECN. The switches may also support configuration of the
congestion threshold used for marking. [DCTCP10] provides a congestion threshold used for marking. [DCTCP10] provides a
theoretical basis for selecting the congestion threshold, but as with theoretical basis for selecting the congestion threshold, but as with
estimation gain, it may be more practical to rely on experimentation estimation gain, it may be more practical to rely on experimentation
or simply to use the device's default configuration. or simply to use the default configuration of the device.
DCTCP requires changes on both the sender and the receiver, so in a DCTCP requires changes on both the sender and the receiver, so both
heterogeneous datacenter, all the endpoints should support DCTCP and endpoints must support DCTCP. Furthermore, DCTCP provides no
should be configured to use it. mechanism for negotiating its use, so both endpoints must be
configured through some out-of-band mechanism to use DCTCP. A
variant of DCTCP that can be deployed unilaterally and only requires
standard ECN behavior has been described in [ODCTCP], but requires
additional experimental evaluation.
5. Security Considerations 6. Known Issues
DCTCP relies on the sender's ability to reconstruct the stream of CE
codepoints received by the remote endpoint. To accomplish this,
DCTCP avoids using a single ACK packet to acknowledge segments
received both with and without the CE codepoint set. However, if an
ACK packet is dropped, it's possible that a subsequent ACK will
indeed acknowledge a mix of CE and non-CE segments. This will, of
course, result in a less accurate congestion estimate. There are
some potential mitigations:
o Even with a degraded congestion estimate, DCTCP may still perform
better than [RFC3168].
o If the estimation gain is small relative to the packet loss rate,
the estimate may not be degraded much.
o If packet losses mostly occur under heavy congestion, most drops
will occur during an unbroken string of CE packets, and the
estimate will be unaffected.
However, the affect of packet drops on DCTCP under real world
conditions has not been analyzed.
DCTCP provides no mechanism for negotiating its use. Thus, there is
additional management and configuration overhead required to ensure
that DCTCP is not used with non-DCTCP endpoints. The affect of using
DCTCP with a standard ECN endpoint has been analyzed in [ODCTCP].
Furthermore, it's possible that other implementations may also modify
[RFC3168] behavior without negotiation, causing further
interoperability issues.
Much like standard TCP, DCTCP is biased against flows with longer
RTTs. A method for improving the fairness of DCTCP has been proposed
in [ADCTCP], but requires additional experimental evaluation.
7. Security Considerations
DCTCP enhances ECN and thus inherits the security considerations DCTCP enhances ECN and thus inherits the security considerations
discussed in [RFC3168]. The processing changes introduced by DCTCP discussed in [RFC3168]. The processing changes introduced by DCTCP
do not exacerbate these considerations or introduce new ones. In do not exacerbate these considerations or introduce new ones. In
particular, with either algorithm, the network infrastructure or the particular, with either algorithm, the network infrastructure or the
remote endpoint can falsely report congestion and thus cause the remote endpoint can falsely report congestion and thus cause the
sender to reduce its congestion window. However, this is no worse sender to reduce cwnd. However, this is no worse than what can be
than what can be achieved by simply dropping packets. achieved by simply dropping packets.
6. IANA Considerations 8. IANA Considerations
This document has no actions for IANA. This document has no actions for IANA.
7. Acknowledgements 9. Acknowledgements
The DCTCP algorithm was originally proposed and analyzed in [DCTCP10] The DCTCP algorithm was originally proposed and analyzed in [DCTCP10]
by Mohammad Alizadeh, Albert Greenberg, Dave Maltz, Jitu Padhye, by Mohammad Alizadeh, Albert Greenberg, Dave Maltz, Jitu Padhye,
Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari
Sridharan. Sridharan.
8. References 10. References
8.1. Normative References 10.1. Normative References
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981. 793, September 1981.
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
Selective Acknowledgment Options", RFC 2018, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP", RFC of Explicit Congestion Notification (ECN) to IP", RFC
3168, September 2001. 3168, September 2001.
8.2. Informative References 10.2. Informative References
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, September 2009. Control", RFC 5681, September 2009.
[DCTCP10] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, [DCTCP10] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel,
P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data
Center TCP (DCTCP)", December 2010, Center TCP (DCTCP)", December 2010,
<http://www.sigcomm.org/ccr/papers/2010/October/ <http://www.sigcomm.org/ccr/papers/2010/
1851275.1851192/>. October/1851275.1851192/>.
[ODCTCP] Kato, M., "Improving Transmission Performance with One-
Sided Datacenter TCP", M.S. Thesis, Keio University, 2014,
<http://eggert.org/students/kato-thesis.pdf>.
[ADCTCP] Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis
of DCTCP: Stability, Convergence, and Fairness", June
2011,
<http://simula.stanford.edu/~alizade/Site/DCTCP_files/
dctcp_analysis-full.pdf>.
Authors' Addresses Authors' Addresses
Stephen Bensley Stephen Bensley
Microsoft Microsoft
One Microsoft Way One Microsoft Way
Redmond, WA 98052 Redmond, WA 98052
USA USA
Phone: +1 425 703 5570 Phone: +1 425 703 5570
Email: sbens@microsoft.com Email: sbens@microsoft.com
Lars Eggert Lars Eggert
skipping to change at page 8, line 21 skipping to change at page 9, line 49
Email: sbens@microsoft.com Email: sbens@microsoft.com
Lars Eggert Lars Eggert
NetApp NetApp
Sonnenallee 1 Sonnenallee 1
Kirchheim 85551 Kirchheim 85551
Germany Germany
Phone: +49 151 120 55791 Phone: +49 151 120 55791
Email: lars@netapp.com Email: lars@netapp.com
URI: http://eggert.org/
Dave Thaler Dave Thaler
Microsoft Microsoft
Phone: +1 425 703 8835 Phone: +1 425 703 8835
Email: dthaler@microsoft.com Email: dthaler@microsoft.com
 End of changes. 46 change blocks. 
102 lines changed or deleted 184 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/