draft-ietf-aqm-fq-codel-02.txt   draft-ietf-aqm-fq-codel-03.txt 
AQM working group T. Hoeiland-Joergensen AQM working group T. Hoeiland-Joergensen
Internet-Draft Karlstad University Internet-Draft Karlstad University
Intended status: Informational P. McKenney Intended status: Informational P. McKenney
Expires: April 21, 2016 IBM Linux Technology Center Expires: May 23, 2016 IBM Linux Technology Center
D. Taht D. Taht
Teklibre Teklibre
J. Gettys J. Gettys
E. Dumazet E. Dumazet
Google, Inc. Google, Inc.
October 19, 2015 November 20, 2015
FlowQueue-Codel FlowQueue-Codel
draft-ietf-aqm-fq-codel-02 draft-ietf-aqm-fq-codel-03
Abstract Abstract
This memo presents the FQ-CoDel hybrid packet scheduler/AQM This memo presents the FQ-CoDel hybrid packet scheduler/AQM
algorithm, a powerful tool for fighting bufferbloat and reducing algorithm, a powerful tool for fighting bufferbloat and reducing
latency. latency.
FQ-CoDel mixes packets from multiple flows and reduces the impact of FQ-CoDel mixes packets from multiple flows and reduces the impact of
head of line blocking from bursty traffic. It provides isolation for head of line blocking from bursty traffic. It provides isolation for
low-rate traffic such as DNS, web, and videoconferencing traffic. It low-rate traffic such as DNS, web, and videoconferencing traffic. It
skipping to change at page 1, line 47 skipping to change at page 1, line 47
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 21, 2016. This Internet-Draft will expire on May 23, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 27 skipping to change at page 2, line 27
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Terminology and concepts . . . . . . . . . . . . . . . . 3 1.1. Terminology and concepts . . . . . . . . . . . . . . . . 3
1.2. Informal summary of FQ-CoDel . . . . . . . . . . . . . . 4 1.2. Informal summary of FQ-CoDel . . . . . . . . . . . . . . 4
2. CoDel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. CoDel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Flow Queueing . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Flow Queueing . . . . . . . . . . . . . . . . . . . . . . . . 5
4. FQ-CoDel Parameters and Data Structures . . . . . . . . . . . 6 4. The FQ-CoDel scheduler . . . . . . . . . . . . . . . . . . . 6
4.1. Parameters . . . . . . . . . . . . . . . . . . . . . . . 6 4.1. Enqueue . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1.1. Interval . . . . . . . . . . . . . . . . . . . . . . 6 4.1.1. Alternative classification schemes . . . . . . . . . 7
4.1.2. Target . . . . . . . . . . . . . . . . . . . . . . . 7 4.2. Dequeue . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.1.3. Packet limit . . . . . . . . . . . . . . . . . . . . 7 5. Implementation considerations . . . . . . . . . . . . . . . . 9
4.1.4. Quantum . . . . . . . . . . . . . . . . . . . . . . . 7 5.1. Data structures . . . . . . . . . . . . . . . . . . . . . 9
4.1.5. Flows . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2. Parameters . . . . . . . . . . . . . . . . . . . . . . . 10
4.1.6. ECN . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.2.1. Interval . . . . . . . . . . . . . . . . . . . . . . 10
4.1.7. CE threshold . . . . . . . . . . . . . . . . . . . . 8 5.2.2. Target . . . . . . . . . . . . . . . . . . . . . . . 10
4.2. Data structures . . . . . . . . . . . . . . . . . . . . . 8 5.2.3. Packet limit . . . . . . . . . . . . . . . . . . . . 10
4.2.1. Internal queues . . . . . . . . . . . . . . . . . . . 8 5.2.4. Quantum . . . . . . . . . . . . . . . . . . . . . . . 11
4.2.2. New and old queues lists . . . . . . . . . . . . . . 9 5.2.5. Flows . . . . . . . . . . . . . . . . . . . . . . . . 11
5. The FQ-CoDel scheduler . . . . . . . . . . . . . . . . . . . 9 5.2.6. ECN . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.1. Enqueue . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.2.7. CE threshold . . . . . . . . . . . . . . . . . . . . 11
5.1.1. Alternative classification schemes . . . . . . . . . 10 5.3. Probability of hash collisions . . . . . . . . . . . . . 12
5.2. Dequeue . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.4. Memory Overhead . . . . . . . . . . . . . . . . . . . . . 12
6. Implementation considerations . . . . . . . . . . . . . . . . 11 5.5. Per-Packet Timestamping . . . . . . . . . . . . . . . . . 13
6.1. Probability of hash collisions . . . . . . . . . . . . . 11 5.6. Other forms of "Fair Queueing" . . . . . . . . . . . . . 13
6.2. Memory Overhead . . . . . . . . . . . . . . . . . . . . . 12 5.7. Differences between CoDel and FQ-CoDel behaviour . . . . 13
6.3. Per-Packet Timestamping . . . . . . . . . . . . . . . . . 13 6. Limitations of flow queueing . . . . . . . . . . . . . . . . 14
6.4. Other forms of "Fair Queueing" . . . . . . . . . . . . . 13 6.1. Fairness between things other than flows . . . . . . . . 14
6.5. Differences between CoDel and FQ-CoDel behaviour . . . . 13 6.2. Flow bunching by opaque encapsulation . . . . . . . . . . 15
7. Limitations of flow queueing . . . . . . . . . . . . . . . . 14 6.3. Low-priority congestion control algorithms . . . . . . . 15
7.1. Fairness between things other than flows . . . . . . . . 14 7. Deployment status and future work . . . . . . . . . . . . . . 16
7.2. Flow bunching by opaque encapsulation . . . . . . . . . . 15 8. Security Considerations . . . . . . . . . . . . . . . . . . . 16
7.3. Low-priority congestion control algorithms . . . . . . . 15 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17
8. Deployment status and future work . . . . . . . . . . . . . . 16 11. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 17
9. Security Considerations . . . . . . . . . . . . . . . . . . . 16 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 17
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 12.1. Normative References . . . . . . . . . . . . . . . . . . 17
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 12.2. Informative References . . . . . . . . . . . . . . . . . 18
12. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 17 12.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 18
13. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19
13.1. Normative References . . . . . . . . . . . . . . . . . . 17
13.2. Informative References . . . . . . . . . . . . . . . . . 18
13.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18
1. Introduction 1. Introduction
The FQ-CoDel algorithm is a combined packet scheduler and AQM The FQ-CoDel algorithm is a combined packet scheduler and AQM
developed as part of the bufferbloat-fighting community effort. It developed as part of the bufferbloat-fighting community effort. It
is based on a modified Deficit Round Robin (DRR) queue scheduler, is based on a modified Deficit Round Robin (DRR) queue scheduler,
with the CoDel AQM algorithm operating on each queue. This document with the CoDel AQM algorithm operating on each queue. This document
describes the combined algorithm; reference implementations are describes the combined algorithm; reference implementations are
available for ns2 and ns3 and it is included in the mainline Linux available for ns2 and ns3 and it is included in the mainline Linux
kernel as the fq_codel queueing discipline. kernel as the fq_codel queueing discipline.
Since the FQ-CoDel algorithm was originally developed in the Linux
kernel, that implementation is still considered canonical. This
document strives to describe the algorithm in the abstract in the
first sections and separate out most implementation details in
subsequent sections, but does use the Linux implementation as
reference for default behaviour in the algorithm description itself.
The rest of this document is structured as follows: This section The rest of this document is structured as follows: This section
gives some concepts and terminology used in the rest of the document, gives some concepts and terminology used in the rest of the document,
and gives a short informal summary of the FQ-CoDel algorithm. and gives a short informal summary of the FQ-CoDel algorithm.
Section 2 gives an overview of the CoDel algorithm. Section 3 covers Section 2 gives an overview of the CoDel algorithm. Section 3 covers
the flow hashing and DRR portion. Section 4 defines the parameters the flow hashing and DRR portion. Section 4 then describes the
and data structures employed by FQ-CoDel. Section 5 describes the working of the algorithm in detail, while Section 5 describes
working of the algorithm in detail. Section 6 describes implementation details and considerations. Section 6 lists some of
implementation considerations, and Section 7 lists some of the the limitations of using flow queueing. Section 7 outlines the
limitations of using flow queueing. Section 8 outlines the current current status of FQ-CoDel deployment and lists some possible future
status of FQ-CoDel deployment and lists some possible future areas of areas of inquiry, and Section 8 reiterates some important security
inquiry, and finally, Section 12 concludes. points that must be observed in the implementation. Finally,
Section 11 concludes.
1.1. Terminology and concepts 1.1. Terminology and concepts
Flow: A flow is typically identified by a 5-tuple of source IP, Flow: A flow is typically identified by a 5-tuple of source IP,
destination IP, source port, destination port, and protocol. It can destination IP, source port, destination port, and protocol. It can
also be identified by a superset or subset of those parameters, or by also be identified by a superset or subset of those parameters, or by
mac address, or other means. mac address, or other means.
Queue: A queue of packets represented internally in FQ-CoDel. In Queue: A queue of packets represented internally in FQ-CoDel. In
most instances each flow gets its own queue; however because of the most instances each flow gets its own queue; however because of the
skipping to change at page 6, line 13 skipping to change at page 6, line 17
Depending on the characteristics of the link, this can be tuned to Depending on the characteristics of the link, this can be tuned to
trade off memory for a lower probability of hash collisions. See trade off memory for a lower probability of hash collisions. See
Section 6 for a more in-depth discussion of this. Section 6 for a more in-depth discussion of this.
By default, the flow hashing is performed on the 5-tuple of source By default, the flow hashing is performed on the 5-tuple of source
and destination IP and port numbers and IP protocol number. While and destination IP and port numbers and IP protocol number. While
the hashing can be customised to match on arbitrary packet bytes, the hashing can be customised to match on arbitrary packet bytes,
care should be taken when doing so: Much of the benefit of the FQ- care should be taken when doing so: Much of the benefit of the FQ-
CoDel scheduler comes from this per-flow distinction. However, the CoDel scheduler comes from this per-flow distinction. However, the
default hashing does have some limitations, as discussed in default hashing does have some limitations, as discussed in
Section 7. Section 6.
FQ-CoDel's DRR scheduler is byte-based, employing a deficit round- FQ-CoDel's DRR scheduler is byte-based, employing a deficit round-
robin mechanism between queues. This works by keeping track of the robin mechanism between queues. This works by keeping track of the
current byte _deficit_ of each queue. This deficit is initialised to current byte _deficit_ of each queue. This deficit is initialised to
the configurable quantum; each time a queue gets a dequeue the configurable quantum; each time a queue gets a dequeue
opportunity, it gets to dequeue packets, decreasing the deficit by opportunity, it gets to dequeue packets, decreasing the deficit by
the packet size for each packet, until the deficit runs into the the packet size for each packet, until the deficit runs into the
negative, at which point it is increased by one quantum, and the negative, at which point it is increased by one quantum, and the
dequeue opportunity ends. dequeue opportunity ends.
skipping to change at page 6, line 38 skipping to change at page 6, line 42
packets are not penalised by the difference in packet sizes; rather, packets are not penalised by the difference in packet sizes; rather,
the DRR scheme approximates a (single-)byte-based fairness queueing. the DRR scheme approximates a (single-)byte-based fairness queueing.
The size of the quantum determines the scheduling granularity, with The size of the quantum determines the scheduling granularity, with
the tradeoff from too small a quantum being scheduling overhead. For the tradeoff from too small a quantum being scheduling overhead. For
small bandwidths, lowering the quantum from the default MTU size can small bandwidths, lowering the quantum from the default MTU size can
be advantageous. be advantageous.
Unlike plain DRR there are two sets of flows - a "new" list for flows Unlike plain DRR there are two sets of flows - a "new" list for flows
that have not built a queue recently, and an "old" list for flow- that have not built a queue recently, and an "old" list for flow-
building queues. This distinction is an integral part of the FQ- building queues. This distinction is an integral part of the FQ-
CoDel scheduler and is described in more detail in Section 5. CoDel scheduler and is described in more detail in Section 4.
4. FQ-CoDel Parameters and Data Structures
This section goes into the parameters and data structures in FQ-
CoDel.
4.1. Parameters
4.1.1. Interval
The _interval_ parameter has the same semantics as CoDel and is used
to ensure that the measured minimum delay does not become too stale.
The minimum delay MUST be experienced in the last epoch of length
interval. It SHOULD be set on the order of the worst-case RTT
through the bottleneck to give end-points sufficient time to react.
The default interval value is 100 ms.
4.1.2. Target
The _target_ parameter has the same semantics as CoDel. It is the
acceptable minimum standing/persistent queue delay for each FQ-CoDel
Queue. This minimum delay is identified by tracking the local
minimum queue delay that packets experience.
The default target value is 5 ms, but this value should be tuned to
be at least the transmission time of a single MTU-sized packet at the
prevalent egress link speed (which for e.g. 1Mbps and MTU 1500 is
~15ms), to prevent CoDel from being too aggressive at low bandwidths.
It should otherwise be set to on the order of 5-10% of the configured
interval.
4.1.3. Packet limit
Routers do not have infinite memory, so some packet limit MUST be
enforced.
The _limit_ parameter is the hard limit on the real queue size,
measured in number of packets. This limit is a global limit on the
number of packets in all queues; each individual queue does not have
an upper limit. When the limit is reached and a new packet arrives
for enqueue, a packet is dropped from the head of the largest queue
(measured in bytes) to make room for the new packet.
In Linux, the default packet limit is 10240 packets, which is
suitable for up to 10GigE speeds. In practice, the hard limit is
rarely, if ever, hit, as drops are performed by the CoDel algorithm
long before the limit is hit. For platforms that are severely memory
constrained, a lower limit can be used.
4.1.4. Quantum
The _quantum_ parameter is the number of bytes each queue gets to
dequeue on each round of the scheduling algorithm. The default is
set to 1514 bytes which corresponds to the Ethernet MTU plus the
hardware header length of 14 bytes.
In TSO-enabled systems, where a "packet" consists of an offloaded
packet train, it can presently be as large as 64K bytes. In GRO-
enabled systems, up to 17 times the TCP max segment size (or 25K
bytes). These mega-packets severely impact FQ-CoDel's ability to
schedule traffic, and hurt latency needlessly. There is ongoing work
in Linux to make smarter use of offload engines.
4.1.5. Flows
The _flows_ parameter sets the number of queues into which the
incoming packets are classified. Due to the stochastic nature of
hashing, multiple flows may end up being hashed into the same slot.
This parameter can be set only at load time since memory has to be
allocated for the hash table in the current implementation.
The default value is 1024 in the current Linux implementation.
4.1.6. ECN
ECN is _enabled_ by default. Rather than do anything special with
misbehaved ECN flows, FQ-CoDel relies on the packet scheduling system
to minimise their impact, thus unresponsive packets in a flow being
marked with ECN can grow to the overall packet limit, but will not
otherwise affect the performance of the system.
It can be disabled by specifying the _noecn_ parameter.
4.1.7. CE threshold
This parameter enables DCTCP-like processing to enable CE marking ECT
packets at a lower setpoint than the default codel target.
The parameter, _ce_threshold_, is disabled by default and can be set
to a number of usec to enable.
4.2. Data structures
4.2.1. Internal queues
The main data structure of FQ-CoDel is the array of queues, which is
instantiated to the number of queues specified by the _flows_
parameter at instantiation time. Each queue consists simply of an
ordered list of packets with FIFO semantics, two state variables
tracking the queue deficit and total number of bytes enqueued, and
the set of CoDel state variables. Other state variables to track
queue statistics can also be included: for instance, the Linux
implementation keeps a count of dropped packets.
Queue space is shared: there's a global limit on the number of
packets the queues can hold, but not one per queue.
4.2.2. New and old queues lists
FQ-CoDel maintains two lists of active queues, called "new" and "old"
queues. Each list is an ordered list containing references to the
array of queues. When a packet is added to a queue that is not
currently active, that queue becomes active by being added to the
list of new queues. Later on, it is moved to the list of old queues,
from which it is removed when it is no longer active. This behaviour
is the source of some subtlety in the packet scheduling at dequeue
time, explained below.
5. The FQ-CoDel scheduler 4. The FQ-CoDel scheduler
This section describes the operation of the FQ-CoDel scheduler and To make its scheduling decisions, FQ-CoDel maintains two ordered
AQM. It is split into two parts explaining the enqueue and dequeue lists of active queues, called "new" and "old" queues. When a packet
operations. is added to a queue that is not currently active, that queue becomes
active by being added to the list of new queues. Later on, it is
moved to the list of old queues, from which it is removed when it is
no longer active. This behaviour is the source of some subtlety in
the packet scheduling at dequeue time, explained below.
5.1. Enqueue 4.1. Enqueue
The packet enqueue mechanism consists of three stages: classification The packet enqueue mechanism consists of three stages: classification
into a queue, timestamping and bookkeeping, and optionally dropping a into a queue, timestamping and bookkeeping, and optionally dropping a
packet when the total number of enqueued packets goes over the packet when the total number of enqueued packets goes over the
maximum. maximum.
When a packet is enqueued, it is first classified into the When a packet is enqueued, it is first classified into the
appropriate queue. By default, this is done by hashing (using a appropriate queue. By default, this is done by hashing (using a
Jenkins hash function) on the 5-tuple of IP protocol, and source and Jenkins hash function) on the 5-tuple of IP protocol, and source and
destination IP and port numbers, permuted by a random value selected destination IP and port numbers, permuted by a random value selected
at initialisation time, and taking the hash value modulo the number at initialisation time, and taking the hash value modulo the number
of queues. of queues.
Once the packet has been successfully classified into a queue, it is Once the packet has been successfully classified into a queue, it is
handed over to the CoDel algorithm for timestamping. It is then handed over to the CoDel algorithm for timestamping. It is then
added to the tail of the selected queue, and the queue's byte count added to the tail of the selected queue, and the queue's byte count
is updated by the packet size. Then, if the queue is not currently is updated by the packet size. Then, if the queue is not currently
active (i.e. if it is not in either the list of new or the list of active (i.e. if it is not in either the list of new or the list of
old queues), it is added to the end of the list of new queues, and old queues), it is added to the end of the list of new queues, and
its deficit is initiated to the configured quantum. Otherwise it is its deficit is initiated to the configured quantum. Otherwise, the
added to the old queue list. queue is left in its current queue list.
Finally, the total number of enqueued packets is compared with the Finally, the total number of enqueued packets is compared with the
configured limit, and if it is _above_ this value (which can happen configured limit, and if it is _above_ this value (which can happen
since a packet was just enqueued), a packet is dropped from the head since a packet was just enqueued), a packet is dropped from the head
of the queue with the largest current byte count. Note that this in of the queue with the largest current byte count. Note that this in
most cases means that the packet that gets dropped is different from most cases means that the packet that gets dropped is different from
the one that was just enqueued, and may even be from a different the one that was just enqueued, and may even be from a different
queue. queue.
5.1.1. Alternative classification schemes 4.1.1. Alternative classification schemes
As mentioned previously, it is possible to modify the classification As mentioned previously, it is possible to modify the classification
scheme to provide a different notion of a 'flow'. The Linux scheme to provide a different notion of a 'flow'. The Linux
implementation provides this option in the form of the "tc filter" implementation provides this option in the form of the "tc filter"
command. While this can add capabilities (for instance, matching on command. While this can add capabilities (for instance, matching on
other possible parameters such as mac address, diffserv, firewall and other possible parameters such as mac address, diffserv, firewall and
flow specific markings, etc.), care should be taken to preserve the flow specific markings, etc.), care should be taken to preserve the
notion of 'flow' as much of the benefit of the FQ-CoDel scheduler notion of 'flow' as much of the benefit of the FQ-CoDel scheduler
comes from keeping flows in separate queues. We are not aware of any comes from keeping flows in separate queues. We are not aware of any
deployments utilising the custom classification feature. deployments utilising the custom classification feature.
An alternative to changing the classification scheme is to perform An alternative to changing the classification scheme is to perform
decapsulation prior to hashing. The Linux implementation does this decapsulation prior to hashing. The Linux implementation does this
for common encapsulations known to the kernel, such as 6in4, IPIP and for common encapsulations known to the kernel, such as 6in4, IPIP and
GRE tunnels. This helps to distinguish between flows that share the GRE tunnels. This helps to distinguish between flows that share the
same (outer) 5-tuple, but of course is limited to unencrypted tunnels same (outer) 5-tuple, but of course is limited to unencrypted tunnels
(see Section 7.2). (see Section 6.2).
5.2. Dequeue 4.2. Dequeue
Most of FQ-CoDel's work is done at packet dequeue time. It consists Most of FQ-CoDel's work is done at packet dequeue time. It consists
of three parts: selecting a queue from which to dequeue a packet, of three parts: selecting a queue from which to dequeue a packet,
actually dequeuing it (employing the CoDel algorithm in the process), actually dequeuing it (employing the CoDel algorithm in the process),
and some final bookkeeping. and some final bookkeeping.
For the first part, the scheduler first looks at the list of new For the first part, the scheduler first looks at the list of new
queues; for each queue in that list, if that queue has a negative queues; for the queue at the head of that list, if that queue has a
deficit (i.e. it has already dequeued at least a quantum of bytes), negative deficit (i.e. it has already dequeued at least a quantum of
its deficit is increased by one quantum, and the queue is put onto bytes), its deficit is increased by one quantum, and the queue is put
_the end of_ the list of old queues, and the routine selects the next onto _the end of_ the list of old queues, and the routine selects the
queue and starts again. next queue and starts again.
Otherwise, that queue is selected for dequeue. If the list of new Otherwise, that queue is selected for dequeue. If the list of new
queues is empty, the scheduler proceeds down the list of old queues queues is empty, the scheduler proceeds down the list of old queues
in the same fashion (checking the deficit, and either selecting the in the same fashion (checking the deficit, and either selecting the
queue for dequeuing, or increasing the deficit and putting the queue queue for dequeuing, or increasing the deficit and putting the queue
back at the end of the list). back at the end of the list).
After having selected a queue from which to dequeue a packet, the After having selected a queue from which to dequeue a packet, the
CoDel algorithm is invoked on that queue. This applies the CoDel CoDel algorithm is invoked on that queue. This applies the CoDel
control law, and may discard one or more packets from the head of control law, and may discard one or more packets from the head of
that queue, before returning the packet that should be dequeued (or that queue, before returning the packet that should be dequeued (or
nothing if the queue is or becomes empty while being handled by the nothing if the queue is or becomes empty while being handled by the
CoDel algorithm). CoDel algorithm).
Finally, if the CoDel algorithm did not return a packet, the queue is Finally, if the CoDel algorithm does not return a packet, then the
empty, and the scheduler does one of two things: if the queue queue must be empty, and the scheduler does one of two things: if the
selected for dequeue came from the list of new queues, it is moved to queue selected for dequeue came from the list of new queues, it is
_the end of_ the list of old queues. If instead it came from the moved to _the end of_ the list of old queues. If instead it came
list of old queues, that queue is removed from the list, to be added from the list of old queues, that queue is removed from the list, to
back (as a new queue) the next time a packet arrives that hashes to be added back (as a new queue) the next time a packet arrives that
that queue. Then (since no packet was available for dequeue), the hashes to that queue. Then (since no packet was available for
whole dequeue process is restarted from the beginning. dequeue), the whole dequeue process is restarted from the beginning.
If, instead, the scheduler _did_ get a packet back from the CoDel If, instead, the scheduler _did_ get a packet back from the CoDel
algorithm, it updates the byte deficit for the selected queue before algorithm, it updates the byte deficit for the selected queue and
returning the packet as the result of the dequeue operation. moves it to the end of the list of old queues, before returning the
packet as the result of the dequeue operation.
The step that moves an empty queue from the list of new queues to The step that moves an empty queue from the list of new queues to
_the end of_ the list of old queues before it is removed is crucial _the end of_ the list of old queues before it is removed is crucial
to prevent starvation. Otherwise the queue could reappear (the next to prevent starvation. Otherwise the queue could reappear (the next
time a packet arrives for it) before the list of old queues is time a packet arrives for it) before the list of old queues is
visited; this can go on indefinitely even with a small number of visited; this can go on indefinitely even with a small number of
active flows, if the flow providing packets to the queue in question active flows, if the flow providing packets to the queue in question
transmits at just the right rate. This is prevented by first moving transmits at just the right rate. This is prevented by first moving
the queue to _the end of_ the list of old queues, forcing a pass the queue to _the end of_ the list of old queues, forcing a pass
through that, and thus preventing starvation. Moving it to the end through that, and thus preventing starvation. Moving it to the end
skipping to change at page 11, line 43 skipping to change at page 9, line 30
+-------+---------+ +------------------+ | +-------+---------+ +------------------+ |
| ^ ^ |Quantum | ^ ^ |Quantum
|Arrival | | |Exceeded |Arrival | | |Exceeded
v | | | v | | |
+-----------------+ | | | +-----------------+ | | |
| | Empty or | | | | | Empty or | | |
| New +-------------------+ +--------+ | New +-------------------+ +--------+
| | Quantum exceeded | | Quantum exceeded
+-----------------+ +-----------------+
6. Implementation considerations 5. Implementation considerations
6.1. Probability of hash collisions This section contains implementation details for the FQ-CoDel
algorithm. This includes the data structures and parameters used in
the Linux implementation, as well as discussion of some required
features of the target platform and other considerations.
5.1. Data structures
The main data structure of FQ-CoDel is the array of queues, which is
instantiated with the number of queues specified by the _flows_
parameter at instantiation time. Each queue consists simply of an
ordered list of packets with FIFO semantics, two state variables
tracking the queue deficit and total number of bytes enqueued, and
the set of CoDel state variables. Other state variables to track
queue statistics can also be included: for instance, the Linux
implementation keeps a count of dropped packets.
In addition to the queue structures themselves, FQ-CoDel maintains
two ordered lists containing references to the subset of queues that
are currently active. These are the list of 'new' queues and the
list of 'old' queues, as explained in Section 4 above.
In the Linux implementation, queue space is shared: there's a global
limit on the number of packets the queues can hold, but not one per
queue.
5.2. Parameters
The following are the user configuration parameters exposed by the
Linux implementation of FQ-CoDel.
5.2.1. Interval
The _interval_ parameter has the same semantics as CoDel and is used
to ensure that the measured minimum delay does not become too stale.
The minimum delay MUST be experienced in the last epoch of length
interval. It SHOULD be set on the order of the worst-case RTT
through the bottleneck to give end-points sufficient time to react.
The default interval value is 100 ms.
5.2.2. Target
The _target_ parameter has the same semantics as CoDel. It is the
acceptable minimum standing/persistent queue delay for each FQ-CoDel
Queue. This minimum delay is identified by tracking the local
minimum queue delay that packets experience.
The default target value is 5 ms, but this value should be tuned to
be at least the transmission time of a single MTU-sized packet at the
prevalent egress link speed (which for e.g. 1Mbps and MTU 1500 is
~15ms), to prevent CoDel from being too aggressive at low bandwidths.
It should otherwise be set to on the order of 5-10% of the configured
interval.
5.2.3. Packet limit
Routers do not have infinite memory, so some packet limit MUST be
enforced.
The _limit_ parameter is the hard limit on the real queue size,
measured in number of packets. This limit is a global limit on the
number of packets in all queues; each individual queue does not have
an upper limit. When the limit is reached and a new packet arrives
for enqueue, a packet is dropped from the head of the largest queue
(measured in bytes) to make room for the new packet.
In Linux, the default packet limit is 10240 packets, which is
suitable for up to 10GigE speeds. In practice, the hard limit is
rarely, if ever, hit, as drops are performed by the CoDel algorithm
long before the limit is hit. For platforms that are severely memory
constrained, a lower limit can be used.
5.2.4. Quantum
The _quantum_ parameter is the number of bytes each queue gets to
dequeue on each round of the scheduling algorithm. The default is
set to 1514 bytes which corresponds to the Ethernet MTU plus the
hardware header length of 14 bytes.
In TSO-enabled systems, where a "packet" consists of an offloaded
packet train, it can presently be as large as 64K bytes. In GRO-
enabled systems, up to 17 times the TCP max segment size (or 25K
bytes). These mega-packets severely impact FQ-CoDel's ability to
schedule traffic, and hurt latency needlessly. There is ongoing work
in Linux to make smarter use of offload engines.
5.2.5. Flows
The _flows_ parameter sets the number of queues into which the
incoming packets are classified. Due to the stochastic nature of
hashing, multiple flows may end up being hashed into the same slot.
This parameter can be set only at load time since memory has to be
allocated for the hash table in the current implementation.
The default value is 1024 in the current Linux implementation.
5.2.6. ECN
ECN is _enabled_ by default. Rather than do anything special with
misbehaved ECN flows, FQ-CoDel relies on the packet scheduling system
to minimise their impact, thus unresponsive packets in a flow being
marked with ECN can grow to the overall packet limit, but will not
otherwise affect the performance of the system.
It can be disabled by specifying the _noecn_ parameter.
5.2.7. CE threshold
This parameter enables DCTCP-like processing to enable CE marking ECT
packets at a lower setpoint than the default codel target.
The parameter, _ce_threshold_, is disabled by default and can be set
to a number of usec to enable.
5.3. Probability of hash collisions
Since the Linux FQ-CoDel implementation by default uses 1024 hash Since the Linux FQ-CoDel implementation by default uses 1024 hash
buckets, the probability that (say) 100 VoIP sessions will all hash buckets, the probability that (say) 100 flows will all hash to the
to the same bucket is something like ten to the power of minus 300. same bucket is something like ten to the power of minus 300. Thus,
Thus, the probability that at least one of the VoIP sessions will the probability that at least one of the flows will hash to some
hash to some other queue is very high indeed. other queue is very high indeed.
Expanding on this, based on analytical equations for hash collision Expanding on this, based on analytical equations for hash collision
probabilities, for 100 VoIP sessions, the probability of no collision probabilities, for 100 flows, the probability of no collision is
is 90.78%; the probability that no more than two of the 100 VoIP 90.78%; the probability that no more than two of the 100 flows will
sessions will be involved in any given collision = 99.57%; and the be involved in any given collision = 99.57%; and the probability that
probability that no more than three of the 100 VoIP sessions will be no more than three of the 100 flows will be involved in any given
involved in any given collision = 99.99%. collision = 99.99%.
These probabilities can be improved upon by using set-associative These probabilities can be improved upon by using set-associative
hashing, a technique used in the Cake algorithm currently being hashing, a technique used in the Cake algorithm currently being
developed as a further development upon the FQ-CoDel principles. For developed as a further development upon the FQ-CoDel principles. For
a 4-way associative hash with the same number of total queues, the a 4-way associative hash with the same number of total queues, the
probability of no collisions for 100 VoIP sessions is 99.93%, while probability of no collisions for 100 flows is 99.93%, while for an
for an 8-way associative hash it is ~100%. 8-way associative hash it is ~100%.
6.2. Memory Overhead 5.4. Memory Overhead
FQ-CoDel can be implemented with a very low memory footprint (less FQ-CoDel can be implemented with a very low memory footprint (less
than 64 bytes per queue on 64 bit systems). These are the data than 64 bytes per queue on 64 bit systems). These are the data
structures used in the Linux implementation: structures used in the Linux implementation:
struct codel_vars { struct codel_vars {
u32 count; u32 count;
u32 lastcount; u32 lastcount;
bool dropping; bool dropping;
u16 rec_inv_sqrt; u16 rec_inv_sqrt;
skipping to change at page 13, line 21 skipping to change at page 13, line 21
u32 quantum; /* psched_mtu(qdisc_dev(sch)); */ u32 quantum; /* psched_mtu(qdisc_dev(sch)); */
struct codel_params cparams; struct codel_params cparams;
struct codel_stats cstats; struct codel_stats cstats;
u32 drop_overlimit; u32 drop_overlimit;
u32 new_flow_count; u32 new_flow_count;
struct list_head new_flows; /* list of new flows */ struct list_head new_flows; /* list of new flows */
struct list_head old_flows; /* list of old flows */ struct list_head old_flows; /* list of old flows */
}; };
6.3. Per-Packet Timestamping 5.5. Per-Packet Timestamping
The CoDel portion of the algorithm requires per-packet timestamps be The CoDel portion of the algorithm requires per-packet timestamps be
stored along with the packet. While this approach works well for stored along with the packet. While this approach works well for
software-based routers, it may be impossible to retrofit devices that software-based routers, it may be impossible to retrofit devices that
do most of their processing in silicon and lack space or mechanism do most of their processing in silicon and lack space or mechanism
for timestamping. for timestamping.
Also, while perfect resolution is not needed, timestamp resolution Also, while perfect resolution is not needed, timestamp resolution
below the target is necessary. Furthermore, timestamping functions below the target is necessary. Furthermore, timestamping functions
in the core OS need to be efficient as they are called at least once in the core OS need to be efficient as they are called at least once
on each packet enqueue and dequeue. on each packet enqueue and dequeue.
6.4. Other forms of "Fair Queueing" 5.6. Other forms of "Fair Queueing"
Much of the scheduling portion of FQ-CoDel is derived from DRR and is Much of the scheduling portion of FQ-CoDel is derived from DRR and is
substantially similar to DRR++. SFQ-based versions have also been substantially similar to DRR++. SFQ-based versions have also been
produced and tested in ns2. Other forms of Fair Queueing, such as produced and tested in ns2. Other forms of Fair Queueing, such as
WFQ or QFQ, have not been thoroughly explored. WFQ or QFQ, have not been thoroughly explored, but there's no a
priori reason why the round-robin scheduling of FQ-CoDel couldn't be
replaced with something else.
6.5. Differences between CoDel and FQ-CoDel behaviour 5.7. Differences between CoDel and FQ-CoDel behaviour
CoDel can be applied to a single queue system as a straight AQM, CoDel can be applied to a single queue system as a straight AQM,
where it converges towards an "ideal" drop rate (i.e. one that where it converges towards an "ideal" drop rate (i.e. one that
minimises delay while keeping a high link utilisation), and then minimises delay while keeping a high link utilisation), and then
optimises around that control point. optimises around that control point.
The scheduling of FQ-CoDel mixes packets of competing flows, which The scheduling of FQ-CoDel mixes packets of competing flows, which
acts to pace bursty flows to better fill the pipe. Additionally, a acts to pace bursty flows to better fill the pipe. Additionally, a
new flow gets substantial "credit" over other flows until CoDel finds new flow gets substantial "credit" over other flows until CoDel finds
an ideal drop rate for it. However, for a new flow that exceeds the an ideal drop rate for it. However, for a new flow that exceeds the
skipping to change at page 14, line 19 skipping to change at page 14, line 21
Finally, the flow isolation FQ-CoDel provides means that the CoDel Finally, the flow isolation FQ-CoDel provides means that the CoDel
drop mechanism operates on the flows actually building queues, which drop mechanism operates on the flows actually building queues, which
results in packets being dropped more accurately from the largest results in packets being dropped more accurately from the largest
flows than CoDel alone manages. Additionally, flow isolation flows than CoDel alone manages. Additionally, flow isolation
radically improves the transient behaviour of the network when radically improves the transient behaviour of the network when
traffic or link characteristics change (e.g. when new flows start up traffic or link characteristics change (e.g. when new flows start up
or the link bandwidth changes); while CoDel itself can take a while or the link bandwidth changes); while CoDel itself can take a while
to respond, fq_codel doesn't miss a beat. to respond, fq_codel doesn't miss a beat.
7. Limitations of flow queueing 6. Limitations of flow queueing
While FQ-CoDel has been shown in many scenarios to offer significant While FQ-CoDel has been shown in many scenarios to offer significant
performance gains, there are some scenarios where the scheduling performance gains, there are some scenarios where the scheduling
algorithm in particular is not a good fit. This section documents algorithm in particular is not a good fit. This section documents
some of the known cases which either may require tweaking the default some of the known cases which either may require tweaking the default
behaviour, or where alternatives to flow queueing should be behaviour, or where alternatives to flow queueing should be
considered. considered.
7.1. Fairness between things other than flows 6.1. Fairness between things other than flows
In some parts of the network, enforcing flow-level fairness may not In some parts of the network, enforcing flow-level fairness may not
be desirable, or some other form of fairness may be more important. be desirable, or some other form of fairness may be more important.
An example of this can be an Internet Service Provider that may be An example of this can be an Internet Service Provider that may be
more interested in ensuring fairness between customers than between more interested in ensuring fairness between customers than between
flows. Or a hosting or transit provider that wishes to ensure flows. Or a hosting or transit provider that wishes to ensure
fairness between connecting Autonomous Systems or networks. Another fairness between connecting Autonomous Systems or networks. Another
issue can be that the number of simultaneous flows experienced at a issue can be that the number of simultaneous flows experienced at a
particular link can be too high for flow-based fairness queueing to particular link can be too high for flow-based fairness queueing to
be effective. be effective.
skipping to change at page 15, line 7 skipping to change at page 15, line 8
any available packet field to partition packets into virtual queues, any available packet field to partition packets into virtual queues,
to for instance match on address or subnet source/destination pairs, to for instance match on address or subnet source/destination pairs,
application layer characteristics, etc. application layer characteristics, etc.
Furthermore, as commonly deployed today, FQ-CoDel is used with three Furthermore, as commonly deployed today, FQ-CoDel is used with three
or more tiers of classification: priority, best effort and or more tiers of classification: priority, best effort and
background, based on diffserv markings. Some products do more background, based on diffserv markings. Some products do more
detailed classification, including deep packet inspection and detailed classification, including deep packet inspection and
destination-specific filters to achieve their desired result. destination-specific filters to achieve their desired result.
7.2. Flow bunching by opaque encapsulation 6.2. Flow bunching by opaque encapsulation
Where possible, FQ-CoDel will attempt to decapsulate packets before Where possible, FQ-CoDel will attempt to decapsulate packets before
matching on the header fields for the flow hashing. However, for matching on the header fields for the flow hashing. However, for
some encapsulation techniques, most notably encrypted VPNs, this is some encapsulation techniques, most notably encrypted VPNs, this is
not possible. If several flows are bunched into one such not possible. If several flows are bunched into one such
encapsulated tunnel, they will be seen as one flow by the FQ-CoDel encapsulated tunnel, they will be seen as one flow by the FQ-CoDel
algorithm. This means that they will share a queue, and drop algorithm. This means that they will share a queue, and drop
behaviour, and so flows inside the encapsulation will not benefit behaviour, and so flows inside the encapsulation will not benefit
from the implicit prioritisation of FQ-CoDel, but will continue to from the implicit prioritisation of FQ-CoDel, but will continue to
benefit from the reduced overall queue length from the CoDel benefit from the reduced overall queue length from the CoDel
skipping to change at page 15, line 30 skipping to change at page 15, line 31
flow, and not assigned a share of the bandwidth based on how many flow, and not assigned a share of the bandwidth based on how many
flows are inside the encapsulation. flows are inside the encapsulation.
Depending on the application, this may or may not be desirable Depending on the application, this may or may not be desirable
behaviour. In cases where it is not, changing FQ-CoDel's matching to behaviour. In cases where it is not, changing FQ-CoDel's matching to
not be flow-based (as detailed in the previous subsection above) can not be flow-based (as detailed in the previous subsection above) can
be a mitigation. Going forward, having some mechanism for opaque be a mitigation. Going forward, having some mechanism for opaque
encapsulations to express to the outer layer which flow a packet encapsulations to express to the outer layer which flow a packet
belongs to, could be a way to mitigate this. belongs to, could be a way to mitigate this.
7.3. Low-priority congestion control algorithms 6.3. Low-priority congestion control algorithms
In the presence of queue management schemes that contain latency In the presence of queue management schemes that contain latency
under load, low-priority congestion control algorithms such as LEDBAT under load, low-priority congestion control algorithms such as LEDBAT
[RFC6817] (or, in general, algorithms that try to voluntarily use up [RFC6817] (or, in general, algorithms that try to voluntarily use up
less than their fair share of bandwidth) experiences very little less than their fair share of bandwidth) experiences very little
added latency when the link is congested. Thus, they lack the signal added latency when the link is congested. Thus, they lack the signal
to back off that added latency previously afforded them. This effect to back off that added latency previously afforded them. This effect
is seen with FQ-CoDel as well as with any effective AQM [GONG2014]. is seen with FQ-CoDel as well as with any effective AQM [GONG2014].
As such, these delay-based algorithms tend to revert to loss-based As such, these delay-based algorithms tend to revert to loss-based
congestion control, and will consume the fair share of bandwidth congestion control, and will consume the fair share of bandwidth
afforded to them by the FQ-CoDel scheduler. However, low-priority afforded to them by the FQ-CoDel scheduler. However, low-priority
congestion control mechanisms may be able to take steps to continue congestion control mechanisms may be able to take steps to continue
to be low priority, for instance by taking into account the vastly to be low priority, for instance by taking into account the vastly
reduced level of delay afforded by an AQM, or by using a coupled reduced level of delay afforded by an AQM, or by using a coupled
approach to observing the behaviour of multiple flows. approach to observing the behaviour of multiple flows.
8. Deployment status and future work 7. Deployment status and future work
The FQ-CoDel algorithm as described in this document has been shipped The FQ-CoDel algorithm as described in this document has been shipped
as part of the Linux kernel since version 3.5, released on the 21st as part of the Linux kernel since version 3.5, released on the 21st
of July, 2012, with the ce_threshold being added in version 4.2. The of July, 2012, with the ce_threshold being added in version 4.2. The
algorithm has seen widespread testing in a variety of contexts and is algorithm has seen widespread testing in a variety of contexts and is
configured as the default queueing discipline in a number of mainline configured as the default queueing discipline in a number of mainline
Linux distributions (as of this writing at least OpenWRT, Arch Linux Linux distributions (as of this writing at least OpenWRT, Arch Linux
and Fedora). We believe it to be a safe default and encourage people and Fedora). We believe it to be a safe default and encourage people
running Linux to turn it on: It is a massive improvement over the running Linux to turn it on: It is a massive improvement over the
previous default FIFO queue. previous default FIFO queue.
skipping to change at page 16, line 40 skipping to change at page 16, line 40
o Interactions between flow queueing and delay-based congestion o Interactions between flow queueing and delay-based congestion
control algorithms and scavenger protocols. control algorithms and scavenger protocols.
o Other scheduling mechanisms to replace the DRR portion of the o Other scheduling mechanisms to replace the DRR portion of the
algorithm, e.g. QFQ or WFQ. algorithm, e.g. QFQ or WFQ.
o Sensitivity of parameters, most notably the number of queues and o Sensitivity of parameters, most notably the number of queues and
the CoDel parameters. the CoDel parameters.
9. Security Considerations 8. Security Considerations
There are no specific security exposures associated with FQ-CoDel There are no specific security exposures associated with FQ-CoDel
that are not also present in current FIFO systems. And some are in that are not also present in current FIFO systems. And some are in
fact reduced (e.g. simple minded packet floods). However, some care fact reduced (e.g. simple minded packet floods). However, some care
is needed in the implementation to ensure this is the case. These is needed in the implementation to ensure this is the case. These
are included in the description above, however we reiterate them are included in the description above, however we reiterate them
here: here:
o To prevent packets in the new queues from starving old queues, it o To prevent packets in the new queues from starving old queues, it
is important that when a queue on the list of new queues empties, is important that when a queue on the list of new queues empties,
it is moved to _the end of_ the list of old queues. This is it is moved to _the end of_ the list of old queues. This is
described at the end of Section 5.2. described at the end of Section 4.2.
o To prevent an attacker targeting a specific flow for a denial of o To prevent an attacker targeting a specific flow for a denial of
service attack, the hash that maps packets to queues should not be service attack, the hash that maps packets to queues should not be
predictable. To achieve this, FQ-CoDel salts the hash, as predictable. To achieve this, FQ-CoDel salts the hash, as
described in the beginning of Section 5.1. The size of the salt described in the beginning of Section 4.1. The size of the salt
and the strength of the hash function is obviously a tradeoff and the strength of the hash function is obviously a tradeoff
between performance and security. The Linux implementation uses a between performance and security. The Linux implementation uses a
32 bit random value as the salt and a Jenkins hash function. This 32 bit random value as the salt and a Jenkins hash function. This
makes it possible to achieve very high throughput, and we consider makes it possible to achieve very high throughput, and we consider
it sufficient to ward off the most obvious attacks. it sufficient to ward off the most obvious attacks.
10. IANA Considerations o Packet fragments without a layer 4 header can be hashed into
different bins than the first fragment with the header intact.
This can cause reordering and/or adversely affect the performance
of the flow. Keeping state to match the fragments to the
beginning of the packet, or simply putting all fragmented packets
into the same queue, are two ways to alleviate this.
9. IANA Considerations
This document has no actions for IANA. This document has no actions for IANA.
11. Acknowledgements 10. Acknowledgements
Our deepest thanks to Kathie Nichols, Van Jacobson, and all the Our deepest thanks to Kathie Nichols, Van Jacobson, and all the
members of the bufferbloat.net effort for all the help on developing members of the bufferbloat.net effort for all the help on developing
and testing the algorithm. In addition, our thanks to Anil Agarwal and testing the algorithm. In addition, our thanks to Anil Agarwal
for his help with getting the hash collision probabilities in this for his help with getting the hash collision probabilities in this
document right. document right.
12. Conclusions 11. Conclusions
FQ-CoDel is a very general, efficient, nearly parameterless queue FQ-CoDel is a very general, efficient, nearly parameterless queue
management approach combining flow queueing with CoDel. It is a very management approach combining flow queueing with CoDel. It is a very
powerful tool for solving bufferbloat, and we believe it to be safe powerful tool for solving bufferbloat, and we believe it to be safe
to turn on by default, as has already happened in a number of Linux to turn on by default, as has already happened in a number of Linux
distributions. In this document we have documented the Linux distributions. In this document we have documented the Linux
implementation in sufficient detail for an independent implementation in sufficient detail for an independent
implementation, and we encourage such implementations be widely implementation, and we encourage such implementations be widely
deployed. deployed.
13. References 12. References
13.1. Normative References 12.1. Normative References
[CODELDRAFT] [CODELDRAFT]
Nichols, K., Jacobson, V., McGregor, A., and J. Iyengar, Nichols, K., Jacobson, V., McGregor, A., and J. Iyengar,
"Controlled Delay Active Queue Management", October 2014, "Controlled Delay Active Queue Management", October 2014,
<https://datatracker.ietf.org/doc/draft-ietf-aqm-codel/>. <https://datatracker.ietf.org/doc/draft-ietf-aqm-codel/>.
[RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, [RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind,
"Low Extra Delay Background Transport (LEDBAT)", RFC 6817, "Low Extra Delay Background Transport (LEDBAT)", RFC 6817,
DOI 10.17487/RFC6817, December 2012, DOI 10.17487/RFC6817, December 2012,
<http://www.rfc-editor.org/info/rfc6817>. <http://www.rfc-editor.org/info/rfc6817>.
[RTCWEB-QOS] [RTCWEB-QOS]
Dhesikan, S., Jennings, C., Druta, D., Jones, P., and J. Dhesikan, S., Jennings, C., Druta, D., Jones, P., and J.
Polk, "DSCP and other packet markings for RTCWeb QoS", Polk, "DSCP and other packet markings for RTCWeb QoS",
December 2014, <https://datatracker.ietf.org/doc/draft- December 2014, <https://datatracker.ietf.org/doc/draft-
dhesikan-tsvwg-rtcweb-qos/>. dhesikan-tsvwg-rtcweb-qos/>.
13.2. Informative References 12.2. Informative References
[CODEL2012] [CODEL2012]
Nichols, K. and V. Jacobson, "Controlling Queue Delay", Nichols, K. and V. Jacobson, "Controlling Queue Delay",
July 2012, <http://queue.acm.org/detail.cfm?id=2209336>. July 2012, <http://queue.acm.org/detail.cfm?id=2209336>.
[DRR] Shreedhar, M. and G. Varghese, "Efficient Fair Queueing [DRR] Shreedhar, M. and G. Varghese, "Efficient Fair Queueing
Using Deficit Round Robin", June 1996, Using Deficit Round Robin", June 1996,
<http://users.ece.gatech.edu/~siva/ECE4607/presentations/ <http://users.ece.gatech.edu/~siva/ECE4607/presentations/
DRR.pdf>. DRR.pdf>.
skipping to change at page 18, line 40 skipping to change at page 18, line 45
low priority congestion control", July 2014, low priority congestion control", July 2014,
<http://perso.telecom-paristech.fr/~drossi/paper/ <http://perso.telecom-paristech.fr/~drossi/paper/
rossi14comnet-b.pdf>. rossi14comnet-b.pdf>.
[SQF2012] Bonald, T., Muscariello, L., and N. Ostallo, "On the [SQF2012] Bonald, T., Muscariello, L., and N. Ostallo, "On the
impact of TCP and per-flow scheduling on Internet impact of TCP and per-flow scheduling on Internet
Performance - IEEE/ACM transactions on Networking", April Performance - IEEE/ACM transactions on Networking", April
2012, <http://perso.telecom- 2012, <http://perso.telecom-
paristech.fr/~bonald/Publications_files/BMO2011.pdf>. paristech.fr/~bonald/Publications_files/BMO2011.pdf>.
13.3. URIs 12.3. URIs
[1] http://www.bufferbloat.net/projects/codel/wiki/Cake [1] http://www.bufferbloat.net/projects/codel/wiki/Cake
Authors' Addresses Authors' Addresses
Toke Hoeiland-Joergensen Toke Hoeiland-Joergensen
Karlstad University Karlstad University
Dept. of Computer Science Dept. of Computer Science
Karlstad 65188 Karlstad 65188
Sweden Sweden
 End of changes. 44 change blocks. 
227 lines changed or deleted 241 lines changed or added

This html diff was produced by rfcdiff 1.42. The latest version is available from http://tools.ietf.org/tools/rfcdiff/