[Docs] [txt|pdf|xml|html] [Tracker] [Email] [Nits]



Internet Engineering Task Force                             M. Sivaraman
Internet-Draft                             Akira Systems Private Limited
Intended status: Experimental                                        Liu
Expires: December 27, 2020                                      Infoblox
                                                           June 25, 2020


                    The DNS thundering herd problem
                draft-muks-dnsop-dns-thundering-herd-00

Abstract

   This document describes an observed regular pattern of spikes in
   queries that affects caching resolvers, and recommends software
   mitigations for it.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on December 27, 2020.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.




Sivaraman & Liu         Expires December 27, 2020               [Page 1]


Internet-Draft       The DNS thundering herd problem           June 2020


Table of Contents

   1.  Problem Description . . . . . . . . . . . . . . . . . . . . .   2
   2.  Requirements Notation . . . . . . . . . . . . . . . . . . . .   4
   3.  Mitigations . . . . . . . . . . . . . . . . . . . . . . . . .   4
     3.1.  Combine identical queries to upstream nameservers . . . .   4
     3.2.  Include noise in response TTLs from caching resolvers . .   4
     3.3.  Other mitigations . . . . . . . . . . . . . . . . . . . .   4
   4.  Security Considerations . . . . . . . . . . . . . . . . . . .   5
   5.  IANA considerations . . . . . . . . . . . . . . . . . . . . .   5
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   5
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   5
     7.1.  Normative references  . . . . . . . . . . . . . . . . . .   5
     7.2.  Informative references  . . . . . . . . . . . . . . . . .   5
   Appendix A.  Change history (to be removed before publication)  .   6
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   6

1.  Problem Description

   Typically, DNS caching resolvers prepare answers for multiple clients
   from a single cached RRset [RFC1034].  Depending on when in time the
   clients make their queries, caching resolvers reply with lower and
   lower valued TTLs, before the cached RRset from which answers are
   prepared expires.  Clients themselves may cache and use their copies
   of RRsets until the TTL that the resolver replied with expires.  A
   key property is that all these copies of answers, and the cached
   answer from which they are prepared, expire at the same absolute
   time.

   As an example, consider the following query sequence received by a
   resolver from 10 clients all querying for a popular
   www.example.org./A RRset.  We use this example to illustrate two
   kinds of spikes in queries.


















Sivaraman & Liu         Expires December 27, 2020               [Page 2]


Internet-Draft       The DNS thundering herd problem           June 2020


   +--------+------------+--------+------------------------------------+
   | Client | Query time | Answer | Notes                              |
   |        | (seconds   | RRset  |                                    |
   |        | since      | TTL    |                                    |
   |        | epoch)     |        |                                    |
   +--------+------------+--------+------------------------------------+
   | C1     | 1591441620 | 600    | Answer was not found in cache.     |
   |        |            |        | Resolver performs a resolution     |
   |        |            |        | and caches authoritative answer    |
   |        |            |        | with TTL=600.                      |
   | C2     | 1591441626 | 594    | Answered from cache.               |
   | C3     | 1591441713 | 507    | Answered from cache.               |
   | C4     | 1591441780 | 440    | Answered from cache.               |
   | C5     | 1591441866 | 354    | Answered from cache.               |
   | C6     | 1591442006 | 214    | Answered from cache.               |
   | C7     | 1591442070 | 150    | Answered from cache.               |
   | C8     | 1591442070 | 150    | Answered from cache.               |
   | C9     | 1591442213 | 7      | Answered from cache.               |
   | C3     | 1591442220 | 600    | Previously cached answer had       |
   |        |            |        | expired in the resolver's          |
   |        |            |        | cache. So the resolver performs a  |
   |        |            |        | fresh resolution and caches        |
   |        |            |        | authoritative answer with TTL=600. |
   | C5     | 1591442220 | 600    | Ditto if not joined with previous. |
   | C2     | 1591442220 | 600    | Ditto if not joined with previous. |
   | C6     | 1591442220 | 600    | Ditto if not joined with previous. |
   | C1     | 1591442221 | 599    | Answered from cache.               |
   | C9     | 1591442221 | 599    | Answered from cache.               |
   | C4     | 1591442221 | 599    | Answered from cache.               |
   | C8     | 1591442221 | 599    | Answered from cache.               |
   | C7     | 1591442221 | 599    | Answered from cache.               |
   | C10    | 1591442227 | 593    | Answered from cache.               |
   | C7     | 1591442820 | 600    | Previously cached answer had       |
   |        |            |        | expired in the resolver's          |
   |        |            |        | cache. So the resolver performs a  |
   |        |            |        | fresh resolution and caches        |
   |        |            |        | authoritative answer with TTL=600. |
   | C4     | 1591442820 | 600    | Ditto if not joined with previous. |
   | C1     | 1591442820 | 600    | Ditto if not joined with previous. |
   | C2     | 1591442820 | 600    | Ditto if not joined with previous. |
   | C10    | 1591442820 | 600    | Ditto if not joined with previous. |
   | C8     | 1591442820 | 600    | Ditto if not joined with previous. |
   | C3     | 1591442821 | 599    | Answered from cache.               |
   | C9     | 1591442821 | 599    | Answered from cache.               |
   | C5     | 1591442821 | 599    | Answered from cache.               |
   | C6     | 1591442821 | 599    | Answered from cache.               |
   +--------+------------+--------+------------------------------------+




Sivaraman & Liu         Expires December 27, 2020               [Page 3]


Internet-Draft       The DNS thundering herd problem           June 2020


2.  Requirements Notation

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   [RFC2119] [RFC8174] when, and only when, they appear in all capitals,
   as shown here.

3.  Mitigations

3.1.  Combine identical queries to upstream nameservers

   At a resolver, when multiple queries have arrived together asking the
   same question and there is no existing unexpired cached answer, DNS
   resolutions have to be performed to answer these queries.  De-
   duplication of these multiple resolutions into a single DNS
   resolution by the resolver is RECOMMENDED where possible.

   If such de-duplication is not performed, the client queries will
   effectively be forwarded 1:1 by the resolver to upstream nameservers,
   and they will significantly increase the upstream nameservers' query
   rate in spikes.  Some nameserver operators may have deployed measures
   such as response rate limiting [RRL] and other IP-address based rate
   limiting, which may cause them to deny service to the resolver due to
   the query spikes of identical queries.

3.2.  Include noise in response TTLs from caching resolvers

   Caching resolvers are permitted to lower the TTLs of RRsets in their
   answers as they please [RFC2181].  This can be used to distribute the
   time at which RRset copies received by clients expire from a single
   absolute time to a time interval.  However, this has to be done with
   some consideration such that the thundering herd doesn't re-converge
   at the expiry time of the cached RRset that is used to generate
   answers to the clients.

   TBD.

3.3.  Other mitigations

   With very low authoritative RRset TTLs (such as under 60s) for
   popular questions, the frequency of the thundering herd increases and
   including noise in response TTLs is less effective because the
   maximum TTL to work with is low.  In other words, there is a shorter
   interval over which the thundering herd can be distributed by adding
   noise.  Some implementations permit an operator to set a minimum TTL
   value such that authoritative RRset TTLs with lower values are
   increased and clamped to the minimum TTL value.  This breaks



Sivaraman & Liu         Expires December 27, 2020               [Page 4]


Internet-Draft       The DNS thundering herd problem           June 2020


   currently accepted DNS protocol, and hence this document does not
   make any recommendation about it.

4.  Security Considerations

   There are no security considerations.

5.  IANA considerations

   There are no IANA considerations.

6.  Acknowledgements

   This document was prepared from thundering herd client query patterns
   noticed at resolvers of ISPs and large institutions, which resulted
   in traffic spikes that caused performance issues and lookup failures.
   The authors acknowledge the contribution of Ramesh Damodaran who
   participated in analysis of these patterns.

7.  References

7.1.  Normative references

   [RFC1034]  Mockapetris, P., "Domain names - concepts and facilities",
              STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987,
              <https://www.rfc-editor.org/info/rfc1034>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2181]  Elz, R. and R. Bush, "Clarifications to the DNS
              Specification", RFC 2181, DOI 10.17487/RFC2181, July 1997,
              <https://www.rfc-editor.org/info/rfc2181>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

7.2.  Informative references

   [RRL]      Vixie, P. and V. Schryver, "DNS Response Rate Limiting
              (DNS RRL)", 2012,
              <https://ftp.isc.org/isc/pubs/tn/isc-tn-2012-1.txt>.






Sivaraman & Liu         Expires December 27, 2020               [Page 5]


Internet-Draft       The DNS thundering herd problem           June 2020


Appendix A.  Change history (to be removed before publication)

   o  draft-muks-dnsop-dns-thundering-herd-00
      * Initial draft.

Authors' Addresses

   Mukund Sivaraman
   Akira Systems Private Limited
   1 Coleman Street, #05-05 The Adelphi
   Singapore  179803
   SG

   Email: muks@akira.org
   URI:   https://akira.org/


   Cricket Liu
   Infoblox
   3111 Coronado Drive
   Santa Clara  95054
   US

   Email: cricket@infoblox.com
   URI:   http://www.infoblox.com/


























Sivaraman & Liu         Expires December 27, 2020               [Page 6]


Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/