--- 1/draft-ietf-trans-gossip-04.txt 2018-01-14 14:13:13.085483333 -0800 +++ 2/draft-ietf-trans-gossip-05.txt 2018-01-14 14:13:13.185485706 -0800 @@ -1,53 +1,53 @@ TRANS L. Nordberg Internet-Draft NORDUnet Intended status: Experimental D. Gillmor -Expires: July 14, 2017 ACLU +Expires: July 18, 2018 ACLU T. Ritter - - January 10, 2017 + January 14, 2018 Gossiping in CT - draft-ietf-trans-gossip-04 + draft-ietf-trans-gossip-05 Abstract The logs in Certificate Transparency are untrusted in the sense that the users of the system don't have to trust that they behave correctly since the behavior of a log can be verified to be correct. This document tries to solve the problem with logs presenting a - "split view" of their operations. It describes three gossiping - mechanisms for Certificate Transparency: SCT Feedback, STH - Pollination and Trusted Auditor Relationship. + "split view" of their operations or failing to incorporate a + submission within MMD. It describes three gossiping mechanisms for + Certificate Transparency: SCT Feedback, STH Pollination and Trusted + Auditor Relationship. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on July 14, 2017. + This Internet-Draft will expire on July 18, 2018. Copyright Notice - Copyright (c) 2017 IETF Trust and the persons identified as the + Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as @@ -59,77 +59,78 @@ 2. Defining the problem . . . . . . . . . . . . . . . . . . . . 4 3. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.1. Pre-Loaded vs Locally Added Anchors . . . . . . . . . . . 5 5. Who gossips with whom . . . . . . . . . . . . . . . . . . . . 5 6. What to gossip about and how . . . . . . . . . . . . . . . . 6 7. Data flow . . . . . . . . . . . . . . . . . . . . . . . . . . 6 8. Gossip Mechanisms . . . . . . . . . . . . . . . . . . . . . . 7 8.1. SCT Feedback . . . . . . . . . . . . . . . . . . . . . . 7 8.1.1. SCT Feedback data format . . . . . . . . . . . . . . 8 - 8.1.2. HTTPS client to server . . . . . . . . . . . . . . . 8 + 8.1.2. HTTPS client to server . . . . . . . . . . . . . . . 9 8.1.3. HTTPS server operation . . . . . . . . . . . . . . . 11 8.1.4. HTTPS server to auditors . . . . . . . . . . . . . . 13 8.2. STH pollination . . . . . . . . . . . . . . . . . . . . . 14 - 8.2.1. HTTPS Clients and Proof Fetching . . . . . . . . . . 15 + 8.2.1. HTTPS Clients and Proof Fetching . . . . . . . . . . 16 8.2.2. STH Pollination without Proof Fetching . . . . . . . 17 8.2.3. Auditor Action . . . . . . . . . . . . . . . . . . . 17 - 8.2.4. STH Pollination data format . . . . . . . . . . . . . 17 - 8.3. Trusted Auditor Stream . . . . . . . . . . . . . . . . . 17 - 8.3.1. Trusted Auditor data format . . . . . . . . . . . . . 18 - 9. 3-Method Ecosystem . . . . . . . . . . . . . . . . . . . . . 19 - 9.1. SCT Feedback . . . . . . . . . . . . . . . . . . . . . . 19 + 8.2.4. STH Pollination data format . . . . . . . . . . . . . 18 + 8.3. Trusted Auditor Stream . . . . . . . . . . . . . . . . . 18 + 8.3.1. Trusted Auditor data format . . . . . . . . . . . . . 19 + 9. 3-Method Ecosystem . . . . . . . . . . . . . . . . . . . . . 20 + 9.1. SCT Feedback . . . . . . . . . . . . . . . . . . . . . . 20 9.2. STH Pollination . . . . . . . . . . . . . . . . . . . . . 20 9.3. Trusted Auditor Relationship . . . . . . . . . . . . . . 21 9.4. Interaction . . . . . . . . . . . . . . . . . . . . . . . 22 - 10. Security considerations . . . . . . . . . . . . . . . . . . . 22 - 10.1. Attacks by actively malicious logs . . . . . . . . . . . 22 + 10. Security considerations . . . . . . . . . . . . . . . . . . . 23 + 10.1. Attacks by actively malicious logs . . . . . . . . . . . 23 10.2. Dual-CA Compromise . . . . . . . . . . . . . . . . . . . 23 10.3. Censorship/Blocking considerations . . . . . . . . . . . 24 10.4. Flushing Attacks . . . . . . . . . . . . . . . . . . . . 25 10.4.1. STHs . . . . . . . . . . . . . . . . . . . . . . . . 25 10.4.2. SCTs & Certificate Chains on HTTPS Servers . . . . . 26 - 10.4.3. SCTs & Certificate Chains on HTTPS Clients . . . . . 26 + 10.4.3. SCTs & Certificate Chains on HTTPS Clients . . . . . 27 10.5. Privacy considerations . . . . . . . . . . . . . . . . . 27 10.5.1. Privacy and SCTs . . . . . . . . . . . . . . . . . . 27 10.5.2. Privacy in SCT Feedback . . . . . . . . . . . . . . 27 10.5.3. Privacy for HTTPS clients performing STH Proof Fetching . . . . . . . . . . . . . . . . . . . . . . 28 - 10.5.4. Privacy in STH Pollination . . . . . . . . . . . . . 28 + 10.5.4. Privacy in STH Pollination . . . . . . . . . . . . . 29 10.5.5. Privacy in STH Interaction . . . . . . . . . . . . . 29 - 10.5.6. Trusted Auditors for HTTPS Clients . . . . . . . . . 29 + 10.5.6. Trusted Auditors for HTTPS Clients . . . . . . . . . 30 10.5.7. HTTPS Clients as Auditors . . . . . . . . . . . . . 30 - 11. Policy Recommendations . . . . . . . . . . . . . . . . . . . 30 + 11. Policy Recommendations . . . . . . . . . . . . . . . . . . . 31 11.1. Blocking Recommendations . . . . . . . . . . . . . . . . 31 11.1.1. Frustrating blocking . . . . . . . . . . . . . . . . 31 11.1.2. Responding to possible blocking . . . . . . . . . . 31 - 11.2. Proof Fetching Recommendations . . . . . . . . . . . . . 32 + 11.2. Proof Fetching Recommendations . . . . . . . . . . . . . 33 11.3. Record Distribution Recommendations . . . . . . . . . . 33 11.3.1. Mixing Algorithm . . . . . . . . . . . . . . . . . . 34 11.3.2. The Deletion Algorithm . . . . . . . . . . . . . . . 35 11.4. Concrete Recommendations . . . . . . . . . . . . . . . . 36 11.4.1. STH Pollination . . . . . . . . . . . . . . . . . . 36 - 11.4.2. SCT Feedback . . . . . . . . . . . . . . . . . . . . 39 + 11.4.2. SCT Feedback . . . . . . . . . . . . . . . . . . . . 40 12. IANA considerations . . . . . . . . . . . . . . . . . . . . . 53 13. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 53 14. ChangeLog . . . . . . . . . . . . . . . . . . . . . . . . . . 53 - 14.1. Changes between ietf-03 and ietf-04 . . . . . . . . . . 53 - 14.2. Changes between ietf-02 and ietf-03 . . . . . . . . . . 54 - 14.3. Changes between ietf-01 and ietf-02 . . . . . . . . . . 54 - 14.4. Changes between ietf-00 and ietf-01 . . . . . . . . . . 54 - 14.5. Changes between -01 and -02 . . . . . . . . . . . . . . 55 - 14.6. Changes between -00 and -01 . . . . . . . . . . . . . . 55 + 14.1. Changes between ietf-04 and ietf-05 . . . . . . . . . . 54 + 14.2. Changes between ietf-03 and ietf-04 . . . . . . . . . . 54 + 14.3. Changes between ietf-02 and ietf-03 . . . . . . . . . . 54 + 14.4. Changes between ietf-01 and ietf-02 . . . . . . . . . . 54 + 14.5. Changes between ietf-00 and ietf-01 . . . . . . . . . . 54 + 14.6. Changes between -01 and -02 . . . . . . . . . . . . . . 55 + 14.7. Changes between -00 and -01 . . . . . . . . . . . . . . 55 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 55 15.1. Normative References . . . . . . . . . . . . . . . . . . 55 15.2. Informative References . . . . . . . . . . . . . . . . . 56 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 56 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 57 1. Introduction The purpose of the protocols in this document, collectively referred to as CT Gossip, is to detect certain misbehavior by CT logs. In particular, CT Gossip aims to detect logs that are providing inconsistent views to different log clients, and logs failing to include submitted certificates within the time period stipulated by MMD. @@ -155,79 +156,80 @@ case, each client might see a unique view of the log. The CT logs are public, append-only and untrusted and thus have to be audited for consistency, i.e., they should never rewrite history. Additionally, auditors and other log clients need to exchange information about logs in order to be able to detect a partitioning attack (as described above). Gossiping about log behavior helps address the problem of detecting malicious or compromised logs with respect to a partitioning attack. - We want some side of the partitioned tree, and ideally both sides, to - see the other side. + We want some side of the partitioned tree, and ideally all sides, to + see at least one other side. Disseminating information about a log poses a potential threat to the - privacy of end users. Some data of interest (e.g. SCTs) is linkable + privacy of end users. Some data of interest (e.g., SCTs) is linkable to specific log entries and thereby to specific websites, which makes sharing them with others a privacy concern. Gossiping about this data has to take privacy considerations into account in order not to expose associations between users of the log (e.g., web browsers) and certificate holders (e.g., web sites). Even sharing STHs (which do not link to specific log entries) can be problematic - user tracking by fingerprinting through rare STHs is one potential attack (see Section 8.2). 3. Overview This document presents three gossiping mechanisms: SCT Feedback, STH Pollination, and a Trusted Auditor Relationship. SCT Feedback enables HTTPS clients to share Signed Certificate - Timestamps (SCTs) (Section 3.3 of [RFC-6962-BIS-09]) with CT auditors + Timestamps (SCTs) (Section 4.8 of [RFC-6962-BIS-27]) with CT auditors in a privacy-preserving manner by sending SCTs to originating HTTPS - servers, who in turn share them with CT auditors. + servers, which in turn share them with CT auditors. In STH Pollination, HTTPS clients use HTTPS servers as pools to share - Signed Tree Heads (STHs) (Section 3.6 of [RFC-6962-BIS-09]) with + Signed Tree Heads (STHs) (Section 4.10 of [RFC-6962-BIS-27]) with other connecting clients in the hope that STHs will find their way to CT auditors. HTTPS clients in a Trusted Auditor Relationship share SCTs and STHs with trusted CT auditors directly, with expectations of privacy sensitive data being handled according to whatever privacy policy is agreed on between client and trusted party. Despite the privacy risks with sharing SCTs there is no loss in privacy if a client sends SCTs for a given site to the site - corresponding to the SCT. This is because the site's logs would - already indicate that the client is accessing that site. In this way + corresponding to the SCT. This is because the site's cookies could + already indicate that the client had accessed that site. In this way a site can accumulate records of SCTs that have been issued by various logs for that site, providing a consolidated repository of SCTs that could be shared with auditors. Auditors can use this information to detect a misbehaving log that fails to include a - certificate within the time period stipulated by its MMD metadata. + certificate within the time period stipulated by its MMD log + parameter. Sharing an STH is considered reasonably safe from a privacy perspective as long as the same STH is shared by a large number of other log clients. This safety in numbers can be achieved by only allowing gossiping of STHs issued in a certain window of time, while also refusing to gossip about STHs from logs with too high an STH issuance frequency (see Section 8.2). 4. Terminology This document relies on terminology and data structures defined in - [RFC-6962-BIS-09], including MMD, STH, SCT, Version, LogID, SCT + [RFC-6962-BIS-27], including MMD, STH, SCT, Version, LogID, SCT timestamp, CtExtensions, SCT signature, Merkle Tree Hash. This document relies on terminology defined in - [draft-ietf-trans-threat-analysis-03], including Auditing. + [draft-ietf-trans-threat-analysis-12], including Auditing. 4.1. Pre-Loaded vs Locally Added Anchors Through the document, we refer to both Trust Anchors (Certificate Authorities) and Logs. Both Logs and Trust Anchors may be locally added by an administrator. Unless otherwise clarified, in both cases we refer to the set of Trust Anchors and Logs that come pre-loaded and pre-trusted in a piece of client software. 5. Who gossips with whom @@ -228,24 +230,24 @@ Authorities) and Logs. Both Logs and Trust Anchors may be locally added by an administrator. Unless otherwise clarified, in both cases we refer to the set of Trust Anchors and Logs that come pre-loaded and pre-trusted in a piece of client software. 5. Who gossips with whom o HTTPS clients and servers (SCT Feedback and STH Pollination) o HTTPS servers and CT auditors (SCT Feedback and STH Pollination) - o CT auditors (Trusted Auditor Relationship) - Additionally, some HTTPS clients may engage with an auditor who they - trust with their privacy: + + Additionally, some HTTPS clients may engage with an auditor which + they trust with their privacy: o HTTPS clients and CT auditors (Trusted Auditor Relationship) 6. What to gossip about and how There are three separate gossip streams: o SCT Feedback - transporting SCTs and certificate chains from HTTPS clients to CT auditors via HTTPS servers. @@ -269,29 +271,29 @@ | | CA | ----------+ | + SCT -> +----------+ | v | Cert [& SCT] +----------+ | | Log | ---------- SCT -----------+ +----------+ v | ^ +----------+ | | SCTs & Certs --- | Website | | |[1] | +----------+ | |[2] STHs ^ | - | |[3] v | | + | |[3] v | HTTPS traffic | | +----------+ | | - | +--------> | Auditor | | HTTPS traffic + | +--------> | Auditor | | SCT & Cert | +----------+ | | - STH | SCT & Cert - | SCTs & Certs | - Log entries | | - | STHs STHs - v | | + STH | STH & Inclusion proof + | | | + Log entries SCTs & Certs | + | | | + v STHs | +----------+ | v | Monitor | +----------+ +----------+ | Browser | +----------+ # Auditor Log [1] |--- get-sth ------------------->| |<-- STH ------------------------| [2] |--- leaf hash + tree size ----->| |<-- index + inclusion proof --->| @@ -327,59 +329,70 @@ HTTPS servers store SCTs and certificate chains received from clients, as described in Section 8.1.3. They later share them with CT auditors by either posting them to auditors or making them available via a well-known URL. This is described in Section 8.1.4. 8.1.1. SCT Feedback data format The data shared between HTTPS clients and servers, as well as between HTTPS servers and CT auditors, is a JSON array [RFC7159]. Each item - in the array is a JSON object with the following content: + in the array is a JSON object containing at least the first of the + following members: - o x509_chain: An array of PEM-encoded X.509 certificates. The first - element is the end-entity certificate, the second certifies the - first and so on. + o "x509_chain" : An array of PEM-encoded X.509 certificates. The + first element is the end-entity certificate, the second certifies + the first and so on. The "x509_chain" member is mandatory to + include. - o sct_data: An array of objects consisting of the base64 - representation of the binary SCT data as defined in - [RFC-6962-BIS-09] Section 3.3. + o "sct_data_v1" : An array of base64 encoded + "SignedCertificateTimestampList"s as defined in [RFC6962] section + 3.3. The "sct_data_v1" member is optional. + + o "sct_data_v2" : An array of base64 encoded "TransItem" structures + of type "x509_sct_v2" or "precert_sct_v2" as defined in + [RFC-6962-BIS-27] section 4.8. The "sct_data_v2" member is + optional. We will refer to this object as 'sct_feedback'. The x509_chain element always contains a full chain from a leaf certificate to a self-signed trust anchor. See Section 8.1.2 for details on what the sct_data element contains as well as more details about the x509_chain element. 8.1.2. HTTPS client to server When an HTTPS client connects to an HTTPS server, the client receives a set of SCTs as part of the TLS handshake. SCTs are included in the TLS handshake using one or more of the three mechanisms described in - [RFC-6962-BIS-09] section 3.4 - in the server certificate, in a TLS + [RFC-6962-BIS-27] section 6 - in the server certificate, in a TLS extension, or in an OCSP extension. The client MUST discard SCTs that are not signed by a log known to the client and SHOULD store the remaining SCTs together with a locally constructed certificate chain - which is trusted (i.e. terminated in a pre-loaded or locally + which is trusted (i.e., terminated in a pre-loaded or locally installed Trust Anchor) in an sct_feedback object or equivalent data structure for later use in SCT Feedback. The SCTs stored on the client MUST be keyed by the exact domain name - the client contacted. They MUST NOT be sent to any domain not - matching the original domain (e.g. if the original domain is - sub.example.com they must not be sent to sub.sub.example.com or to - example.com.) They MUST NOT be sent to any Subject Alternate Names + the client contacted. They MUST NOT be sent to the well-known URI of + any domain not matching the original domain (e.g., if the original + domain is sub.example.com they must not be sent to + sub.sub.example.com or to example.com.) In particular, they MUST NOT + be sent to the well-known URI of any Subject Alternate Names specified in the certificate. In the case of certificates that - validate multiple domain names, the same SCT is expected to be stored - multiple times. + validate multiple domain names, after visiting a second domain name + specified in the certificate, the same SCT is expected to be stored + once under each domain name's key. If Connection Reuse as defined in + [RFC7540] is available, reusing an existing connection to + sub.example.com to send data to sub.sub.example.com is permitted. Not following these constraints would increase the risk for two types of privacy breaches. First, the HTTPS server receiving the SCT would learn about other sites visited by the HTTPS client. Second, auditors receiving SCTs from the HTTPS server would learn information about other HTTPS servers visited by its clients. If the client later again connects to the same HTTPS server, it again receives a set of SCTs and calculates a certificate chain, and again creates an sct_feedback or similar object. If this object does not @@ -402,52 +415,56 @@ Refer to Section 11.3 for recommendations for implementation. Because SCTs can be used as a tracking mechanism (see Section 10.5.2), they deserve special treatment when they are received from (and provided to) domains that are loaded as subresources from an origin domain. Such domains are commonly called 'third party domains'. An HTTPS client SHOULD store SCT Feedback using a 'double-keying' approach, which isolates third party domains by the first party domain. This is described in [double-keying]. - Gossip would be performed normally for third party domains only when the user revisits the first party domain. In lieu of 'double- keying', an HTTPS client MAY treat SCT Feedback in the same manner it treats other security mechanisms that can enable tracking (such as HSTS and HPKP.) - If the HTTPS client has configuration options for not sending cookies - to third parties, SCTs of third parties MUST be treated as cookies - with respect to this setting. This prevents third party tracking - through the use of SCTs/certificates, which would bypass the cookie - policy. For domains that are only loaded as third party domains, the - client may never perform SCT Feedback; however the client may perform - STH Pollination after fetching an inclusion proof, as specified in - Section 8.2. + SCT Feedback is only performed when a user connects to a site via + intentional web browsing or normal third party resource inclusion. + It MUST NOT be performed automatically as part of some sort of + background process. + + Finally, if the HTTPS client has configuration options for not + sending cookies to third parties, SCTs of third parties MUST be + treated as cookies with respect to this setting. This prevents third + party tracking through the use of SCTs/certificates, which would + bypass the cookie policy. For domains that are only loaded as third + party domains, the client may never perform SCT Feedback; however the + client may perform STH Pollination after fetching an inclusion proof, + as specified in Section 8.2. SCTs and corresponding certificates are POSTed to the originating HTTPS server at the well-known URL: https:///.well-known/ct-gossip/v1/sct-feedback The data sent in the POST is defined in Section 8.1.1. This data SHOULD be sent in an already-established TLS session. This makes it hard for an attacker to disrupt SCT Feedback without also disturbing ordinary secure browsing (https://). This is discussed more in Section 11.1.1. The HTTPS server SHOULD respond with an HTTP 200 response code and an empty body if it was able to process the request. An HTTPS client - who receives any other response SHOULD consider it an error. + which receives any other response SHOULD consider it an error. - Some clients have trust anchors or logs that are locally added (e.g. + Some clients have trust anchors or logs that are locally added (e.g., by an administrator or by the user themselves). These additions are potentially privacy-sensitive because they can carry information about the specific configuration, computer, or user. Certificates validated by locally added trust anchors will commonly have no SCTs associated with them, so in this case no action is needed with respect to CT Gossip. SCTs issued by locally added logs MUST NOT be reported via SCT Feedback. If a certificate is validated by SCTs that are issued by publicly @@ -478,34 +495,33 @@ will only track leaf certificates and SCTs applicable to those leaf certificates. In the more complex mode, the server will confirm the client's chain validation and store the certificate chain. The latter mode requires more configuration, but is necessary to prevent denial of service (DoS) attacks on the server's storage space. In the simple mode of operation, upon receiving a submission at the sct-feedback well-known URL, an HTTPS server will perform a set of operations, checking on each sct_feedback object before storing it: - 1. the HTTPS server MAY modify the sct_feedback object, and discard - all items in the x509_chain array except the first item (which is - the end-entity certificate) + o (1) the HTTPS server MAY modify the sct_feedback object, and + discard all items in the x509_chain array except the first item + (which is the end-entity certificate) - 2. if a bit-wise compare of the sct_feedback object matches one - already in the store, this sct_feedback object SHOULD be - discarded + o (2) if a bit-wise compare of the sct_feedback object matches one + already in the store, this sct_feedback object SHOULD be discarded - 3. if the leaf cert is not for a domain for which the server is + o (3) if the leaf cert is not for a domain for which the server is authoritative, the SCT MUST be discarded - 4. if an SCT in the sct_data array can't be verified to be a valid - SCT for the accompanying leaf cert, and issued by a known log, - the individual SCT SHOULD be discarded + o (4) if an SCT in the sct_data array can't be verified to be a + valid SCT for the accompanying leaf cert, and issued by a known + log, the individual SCT SHOULD be discarded The modification in step number 1 is necessary to prevent a malicious client from exhausting the server's storage space. A client can generate their own issuing certificate authorities, and create an arbitrary number of chains that terminate in an end-entity certificate with an existing SCT. By discarding all but the end- entity certificate, we prevent a simple HTTPS server from storing this data. Note that operation in this mode will not prevent the attack described in [dual-ca-compromise-attack]. Skipping this step requires additional configuration as described below. @@ -517,41 +533,35 @@ MAY be performed. If the certificate chain exactly matches an existing certificate chain, the server MAY store the union of the SCTs from the two objects in the first (existing) object. If the validity check on any of the SCTs fails, the server SHOULD NOT store the union of the SCTs. The check in step 3 is to help malfunctioning clients from exposing which sites they visit. It additionally helps prevent DoS attacks on the server. - [ Note: Thinking about building this, how does the SCT Feedback app - know which sites it's authoritative for? It will need that amount of - configuration at least. ] - The check in step 4 is to prevent DoS attacks where an adversary fills up the store prior to attacking a client (thus preventing the client's feedback from being recorded), or an attack where an adversary simply attempts to fill up server's storage space. The above describes the simpler mode of operation. In the more advanced server mode, the server will detect the attack described in [dual-ca-compromise-attack]. In this configuration the server will not modify the sct_feedback object prior to performing checks 2, 3, - and 4. - - To prevent a malicious client from filling the server's data store, - the HTTPS server SHOULD perform an additional check in the more - advanced mode: + and 4. Instead, to prevent a malicious client from filling the + server's data store, the HTTPS server SHOULD perform an additional + check in the more advanced mode: - o if the x509_chain consists of an invalid certificate chain, or the - culminating trust anchor is not recognized by the server, the + o (5) if the x509_chain consists of an invalid certificate chain, or + the culminating trust anchor is not recognized by the server, the server SHOULD modify the sct_feedback object, discarding all items in the x509_chain array except the first item The HTTPS server MAY choose to omit checks 4 or 5. This will place the server at risk of having its data store filled up by invalid data, but can also allow a server to identify interesting certificate or certificate chains that omit valid SCTs, or do not chain to a trusted root. This information may enable an HTTPS server operator to detect attacks or unusual behavior of Certificate Authorities even outside the Certificate Transparency ecosystem. @@ -653,22 +664,22 @@ happen is regarded as undefined policy with the exception of privacy concerns explained below. Suggestions for the policy can be found in Section 11.3. An HTTPS client could be tracked by giving it a unique or rare STH. To address this concern, we place restrictions on different components of the system to ensure an STH will not be rare. o HTTPS clients silently ignore STHs from logs with an STH issuance frequency of more than one STH per hour. Logs use the STH - Frequency Count metadata to express this ([RFC-6962-BIS-09] - sections 3.6 and 5.1). + Frequency Count log parameter to express this ([RFC-6962-BIS-27] + section 4.1). o HTTPS clients silently ignore STHs which are not fresh. An STH is considered fresh iff its timestamp is less than 14 days in the past. Given a maximum STH issuance rate of one per hour, an attacker has 336 unique STHs per log for tracking. Clients MUST ignore STHs older than 14 days. We consider STHs within this validity window not to be personally identifiable data, and STHs outside this window to be personally identifiable. @@ -676,27 +687,27 @@ STHs, this number of unique STHs grow and the negative privacy implications grow with it. It's important that this is taken into account when logs are chosen for default settings in HTTPS clients. This concern is discussed upon in Section 10.5.5. A log may cease operation, in which case there will soon be no STH within the validity window. Clients SHOULD perform all three methods of gossip about a log that has ceased operation since it is possible the log was still compromised and gossip can detect that. STH Pollination is the one mechanism where a client must know about a log - shutdown. A client who does not know about a log shutdown MUST NOT + shutdown. A client which does not know about a log shutdown MUST NOT attempt any heuristic to detect a shutdown. Instead the client MUST - be informed about the shutdown from a verifiable source (e.g. a - software update). The client SHOULD be provided the final STH issued - by the log and SHOULD resolve SCTs and STHs to this final STH. If an - SCT or STH cannot be resolved to the final STH, clients SHOULD follow - the requirements and recommendations set forth in Section 11.1.2. + be informed about the shutdown from a verifiable source (e.g., a + software update), and be provided the final STH issued by the log. + The client SHOULD resolve SCTs and STHs to this final STH. If an SCT + or STH cannot be resolved to the final STH, clients SHOULD follow the + requirements and recommendations set forth in Section 11.1.2. 8.2.1. HTTPS Clients and Proof Fetching There are two types of proofs a client may retrieve; inclusion proofs and consistency proofs. An HTTPS client will retrieve SCTs together with certificate chains from an HTTPS server. Using the timestamp in the SCT together with the end-entity certificate and the issuer key hash, it can obtain an inclusion proof to an STH in order to verify the promise made by the @@ -704,25 +715,30 @@ An HTTPS client will have STHs from performing STH Pollination, and may obtain a consistency proof to a more recent STH. An HTTPS client may also receive an SCT bundled with an inclusion proof to a historical STH via an unspecified future mechanism. Because this historical STH is considered personally identifiable information per above, the client needs to obtain a consistency proof to a more recent STH. - A client SHOULD perform proof fetching. A client MUST NOT perform - proof fetching for any SCTs or STHs issued by a locally added log. A - client MAY fetch an inclusion proof for an SCT (issued by a pre- - loaded log) that validates a certificate chaining to a locally added - trust anchor. + A client SHOULD attempt proof fetching. A client MAY do network + probing to determine if proof fetching may succeed, and if it learns + that it does not, SHOULD periodically re-probe (especially after + network change, if it is aware of these events.) If it does succeed, + queued events can be processed. + + A client MUST NOT perform proof fetching for any SCTs or STHs issued + by a locally added log. A client MAY fetch an inclusion proof for an + SCT (issued by a pre-loaded log) that validates a certificate + chaining to a locally added trust anchor. If a client requested either proof directly from a log or auditor, it would reveal the client's browsing habits to a third party. To mitigate this risk, an HTTPS client MUST retrieve the proof in a manner that disguises the client. Depending on the client's DNS provider, DNS may provide an appropriate intermediate layer that obfuscates the linkability between the user of the client and the request for inclusion (while at the same time providing a caching layer for oft-requested @@ -754,63 +770,69 @@ 8.2.2. STH Pollination without Proof Fetching An HTTPS client MAY participate in STH Pollination without fetching proofs. In this situation, the client receives STHs from a server, applies the same validation logic to them (signed by a known log, within the validity window) and will later pass them to another HTTPS server. When operating in this fashion, the HTTPS client is promoting gossip for Certificate Transparency, but derives no direct benefit itself. - In comparison, a client who resolves SCTs or historical STHs to + In comparison, a client which resolves SCTs or historical STHs to recent STHs and pollinates them is assured that if it was attacked, there is a probability that the ecosystem will detect and respond to the attack (by distrusting the log). 8.2.3. Auditor Action CT auditors participate in STH pollination by retrieving STHs from HTTPS servers. They verify that the STH is valid by checking the signature, and requesting a consistency proof from the STH to the most recent STH. After retrieving the consistency proof to the most recent STH, they SHOULD pollinate this new STH among participating HTTPS servers. In this way, as STHs "age out" and are no longer fresh, their "lineage" continues to be tracked in the system. 8.2.4. STH Pollination data format The data sent from HTTPS clients and CT auditors to HTTPS servers is - a JSON object [RFC7159] with the following content: + a JSON object [RFC7159] with one or both of the following two + members: - o sths - an array of 0 or more fresh SignedTreeHeads as defined in - [RFC-6962-BIS-09] Section 3.6.1. + o "v1" : array of 0 or more objects each containing an STH as + returned from ct/v1/get-sth, see [RFC6962] section 4.3 + + o "v2" : array of 0 or more objects each containing an STH as + returned from ct/v2/get-sth, see [RFC-6962-BIS-27] section 5.2 + + Note that all STHs MUST be fresh as defined in Section 8.2. 8.3. Trusted Auditor Stream HTTPS clients MAY send SCTs and cert chains, as well as STHs, directly to auditors. If sent, this data MAY include data that reflects locally added logs or trust anchors. Note that there are privacy implications in doing so, these are outlined in Section 10.5.1 and Section 10.5.6. The most natural trusted auditor arrangement arguably is a web browser that is "logged in to" a provider of various internet services. Another equivalent arrangement is a trusted party like a corporation to which an employee is connected through a VPN or by other similar means. A third might be individuals or smaller groups of people running their own services. In such a setting, retrieving proofs from that third party could be considered reasonable from a privacy perspective. The HTTPS client may also do its own auditing and might additionally share SCTs and STHs with the trusted party to - contribute to herd immunity. Here, the ordinary [RFC-6962-BIS-09] + contribute to herd immunity. Here, the ordinary [RFC-6962-BIS-27] protocol is sufficient for the client to do the auditing while SCT Feedback and STH Pollination can be used in whole or in parts for the gossip part. Another well established trusted party arrangement on the internet today is the relation between internet users and their providers of DNS resolver services. DNS resolvers are typically provided by the internet service provider (ISP) used, which by the nature of name resolving already know a great deal about which sites their users visit. As mentioned in Section 8.2.1, in order for HTTPS clients to @@ -887,34 +909,32 @@ o Maximize its chance of detecting a misissued certificate by configuring a trust store of CAs o Establish a "push" mechanism for POSTing SCTs to CT auditors These configuration needs, and the simple fact that it would require some deployment of software, means that some percentage of HTTPS servers will not deploy SCT Feedback. - It is worthwhile to note that an attacker may be able to prevent - detection of an attack on a webserver (in all cases) if SCT Feedback - is not implemented. This attack is detailed in Section 10.1). - If SCT Feedback was the only mechanism in the ecosystem, any server that did not implement the feature would open itself and its users to attack without any possibility of detection. - If SCT Feedback is not deployed by a webserver, malicious logs will - be able to attack all users of the webserver (who do not have a - Trusted Auditor relationship) with impunity. Additionally, users who - wish to have the strongest measure of privacy protection (by - disabling STH Pollination Proof Fetching and forgoing a Trusted - Auditor) could be attacked without risk of detection. + A webserver not deploying SCT Feedback (or an alternative method + providing equivalent functionality) may never learn that it was a + target of an attack by a malicious log, as described in Section 10.1, + although the presence of an attack by the log could be learned + through STH Pollination. Additionally, users who wish to have the + strongest measure of privacy protection (by disabling STH Pollination + Proof Fetching and forgoing a Trusted Auditor) could be attacked + without risk of detection. 9.2. STH Pollination STH Pollination requires the cooperation of HTTPS clients, HTTPS servers, and logs. For a client to fully participate in STH Pollination, and have this mechanism detect attacks against it, the client must have a way to safely perform Proof Fetching in a privacy preserving manner. (The client may pollinate STHs it receives without performing Proof @@ -925,37 +945,38 @@ amount of disk space to the endeavor. Logs (or a third party mirroring the logs) must provide access to clients to query proofs in a privacy preserving manner, most likely through DNS. Unlike SCT Feedback, the STH Pollination mechanism is not hampered if only a minority of HTTPS servers deploy it. However, it makes an assumption that an HTTPS client performs Proof Fetching (such as the DNS mechanism discussed). Unfortunately, any manner that is - anonymous for some (such as clients who use shared DNS services such - as a large ISP), may not be anonymous for others. + anonymous for some (such as clients which use shared DNS services + such as a large ISP), may not be anonymous for others. For instance, DNS requests expose a considerable amount of sensitive information (including what data is already present in the cache) in plaintext over the network. For this reason, some percentage of HTTPS clients may choose to not enable the Proof Fetching component of STH Pollination. (Although they can still request and send STHs among participating HTTPS servers, even when this affords them no direct benefit.) If STH Pollination was the only mechanism deployed, users that disable it would be able to be attacked without risk of detection. - If STH Pollination was not deployed, HTTPS clients visiting HTTPS - Servers who did not deploy SCT Feedback could be attacked without - risk of detection. + If STH Pollination (or an alternative method providing equivalent + functionality) was not deployed, HTTPS clients visiting HTTPS Servers + which did not deploy SCT Feedback could be attacked without risk of + detection. 9.3. Trusted Auditor Relationship The Trusted Auditor Relationship is expected to be the rarest gossip mechanism, as an HTTPS client is providing an unadulterated report of its browsing history to a third party. While there are valid and common reasons for doing so, there is no appropriate way to enter into this relationship without retrieving informed consent from the user. @@ -981,76 +1002,90 @@ If the Trusted Auditor relationship was not deployed, crawlers and organizations would build it themselves for their own needs. By standardizing it, users who wish to opt-in (for instance those unwilling to participate fully in STH Pollination) can have an interoperable standard they can use to choose and change their trusted auditor. 9.4. Interaction - The interactions of the mechanisms is thus outlined: + Assuming no other log consistency measures exist, clients who perform + only a subset of the mechanisms described in this document are + exposed to the following vulnerabilities: HTTPS clients can be attacked without risk of detection if they do not participate in any of the three mechanisms. HTTPS clients are afforded the greatest chance of detecting an attack when they either participate in both SCT Feedback and STH Pollination with Proof Fetching or if they have a Trusted Auditor relationship. - (Participating in SCT Feedback is required to prevent a malicious log - from refusing to ever resolve an SCT to an STH, as put forward in - Section 10.1). Additionally, participating in SCT Feedback enables - an HTTPS client to assist in detecting the exact target of an attack. + (Participating in SCT Feedback is the only way specified in this + document to prevent a malicious log from refusing to ever resolve an + SCT to an STH, as put forward in Section 10.1). Additionally, + participating in SCT Feedback enables an HTTPS client to assist in + detecting the exact target of an attack. HTTPS servers that omit SCT Feedback enable malicious logs to carry out attacks without risk of detection. If these servers are targeted specifically, even if the attack is detected, without SCT Feedback they may never learn that they were specifically targeted. HTTPS servers without SCT Feedback do gain some measure of herd immunity, but only because their clients participate in STH Pollination (with Proof Fetching) or have a Trusted Auditor Relationship. When HTTPS servers omit SCT feedback, it allows their users to be attacked without detection by a malicious log; the vulnerable users are those who do not have a Trusted Auditor relationship. 10. Security considerations 10.1. Attacks by actively malicious logs One of the most powerful attacks possible in the CT ecosystem is a trusted log that has actively decided to be malicious. It can carry - out an attack in two ways: + out an attack in at least two ways: In the first attack, the log can present a split view of the log for - all time. The only way to detect this attack is to resolve each view - of the log to the two most recent STHs and then force the log to - present a consistency proof. (Which it cannot.) This attack can be - detected by CT auditors participating in STH Pollination, as long as - they are explicitly built to handle the situation of a log - continuously presenting a split view. + all time. This attack can be detected by CT auditors, but a naive + auditor implementation may fail to do so. The simplest, least + efficient way to detect the attack is to mirror the entire log and + assert inclusion of every peice of data. If an auditor does not + mirror the log, one way to detect this attack is to resolve each view + of the log to the most recent STHs available and then force the log + to present a consistency proof. (Which it cannot.) We highly + recommend auditors plan for this attack scenario and ensure it will + be detected. In the second attack, the log can sign an SCT, and refuse to ever include the certificate that the SCT refers to in the tree. (Alternately, it can include it in a branch of the tree and issue an STH, but then abandon that branch.) Whenever someone requests an inclusion proof for that SCT (or a consistency proof from that STH), the log would respond with an error, and a client may simply regard the response as a transient error. This attack can be detected using SCT Feedback, or an Auditor of Last Resort, as presented in Section 11.1.2. + Both of these attack variants can be detected by CT auditors who have + obtained an STH of an 'abnormal' view of the log. However, they may + not be able to link the STH to any particular SCT or Certificate. + This means that while the log misbehavior was successfully detected, + the target of the attack was not identified. To assertively identify + the target(s) of the attack, SCT Feedback is necessary. + 10.2. Dual-CA Compromise [dual-ca-compromise-attack] describes an attack possible by an adversary who compromises two Certificate Authorities and a Log. This attack is difficult to defend against in the CT ecosystem, and + [dual-ca-compromise-attack] describes a few approaches to doing so. We note that Gossip is not intended to defend against this attack, but can in certain modes. Defending against the Dual-CA Compromise attack requires SCT Feedback, and explicitly requires the server to save full certificate chains (described in Section 8.1.3 as the 'complex' configuration.) After CT auditors receive the full certificate chains from servers, they MAY compare the chain built by clients to the chain supplied by the log. If the chains differ significantly, the auditor SHOULD @@ -1051,34 +1086,20 @@ Defending against the Dual-CA Compromise attack requires SCT Feedback, and explicitly requires the server to save full certificate chains (described in Section 8.1.3 as the 'complex' configuration.) After CT auditors receive the full certificate chains from servers, they MAY compare the chain built by clients to the chain supplied by the log. If the chains differ significantly, the auditor SHOULD raise a concern. A method of determining if chains differ significantly is by asserting that one chain is not a subset of the other and that the roots of the chains are different. - [Note: Justification for this algorithm: - - Cross-Signatures could result in a different org being treated as the - 'root', but in this case, one chain would be a subset of the other. - - Intermediate swapping (e.g. different signature algorithms) could - result in different chains, but the root would be the same. - - (Hitting both those cases at once would cause a false positive - though, but this would likely be rare.) - - Are there other cases that could occur? (Left for the purposes of - reading during pre-Last Call, to be removed by Editor)] - 10.3. Censorship/Blocking considerations We assume a network attacker who is able to fully control the client's internet connection for some period of time, including selectively blocking requests to certain hosts and truncating TLS connections based on information observed or guessed about client behavior. In order to successfully detect log misbehavior, the gossip mechanisms must still work even in these conditions. There are several gossip connections that can be blocked: @@ -1148,29 +1169,30 @@ window SHOULD NOT be deleted. An attacker cannot flush an item from the cache if it is never removed so flushing attacks are completely mitigated. The required disk space for all STHs within the validity window is 336 STHs per log that is trusted. If 20 logs are trusted, and each STH takes 1 Kilobytes, this is 6.56 Megabytes. Note that it is important that implementors do not calculate the exact size of cache expected - if an attack does occur, a small - number of additional STHs will enter into the cache. These STHs will - be in addition to the expected set, and will be evidence of the - attack. + number of additional, fraudulent STHs will enter into the cache. + These STHs will be in addition to the expected set, and will be + evidence of the attack. Flooding the cache will not work, as an + attacker would have to include fraudulent STHs in the flood. If an HTTPS client or HTTPS server is operating in a constrained environment and cannot devote enough storage space to hold all STHs within the validity window it is recommended to use the below - Deletion Algorithm Section 11.3.2 to make it more difficult for the - attacker to perform a flushing attack. + Deletion Algorithm in section Section 11.3.2 to make it more + difficult for the attacker to perform a flushing attack. 10.4.2. SCTs & Certificate Chains on HTTPS Servers An HTTPS server will only accept SCTs and Certificate Chains for domains it is authoritative for. Therefore the storage space needed is bound by the number of logs it accepts, multiplied by the number of domains it is authoritative for, multiplied by the number of certificates issued for those domains. Imagine a server authoritative for 10,000 domains, and each domain @@ -1188,21 +1210,21 @@ SCT(s) and Certificate Chain(s) will enter into the cache. This data will be in addition to the expected set, and will be evidence of the attack. If an HTTPS server is operating in a constrained environment and cannot devote enough storage space to hold all SCTs and Certificate Chains it is authoritative for it is recommended to configure the SCT Feedback mechanism to allow only certain certificates that are known to be valid. These chains and SCTs can then be discarded without being stored or subsequently provided to any clients or auditors. If - the allowlist is not sufficient, the below Deletion Algorithm + the allowlist is not sufficient, the below Deletion Algorithm in Section 11.3.2 is recommended to make it more difficult for the attacker to perform a flushing attack. 10.4.3. SCTs & Certificate Chains on HTTPS Clients HTTPS clients will accumulate SCTs and Certificate Chains without bound. It is expected they will choose a particular cache size and delete entries when the cache size meets its limit. This does not mitigate flushing attacks, and such an attack is documented in [gossip-mixing]. @@ -1227,22 +1249,24 @@ is better to err on the side of safety, and not assume a client is okay with giving up its privacy. 10.5.1. Privacy and SCTs An SCT contains information that links it to a particular web site. Because the client-server relationship is sensitive, gossip between clients and servers about unrelated SCTs is risky. Therefore, a client with an SCT for a given server SHOULD NOT transmit that information in any other than the following two channels: to the - server associated with the SCT itself; or to a Trusted Auditor, if - one exists. + server associated with the SCT itself (via a TLS connection with a + certificate identifying the Domain Name of the web site with a Host + header specifying the domain name); or to a Trusted Auditor, if one + exists. 10.5.2. Privacy in SCT Feedback SCTs introduce yet another mechanism for HTTPS servers to store state on an HTTPS client, and potentially track users. HTTPS clients which allow users to clear history or cookies associated with an origin MUST clear stored SCTs and certificate chains associated with the origin as well. Auditors should treat all SCTs as sensitive data. SCTs received @@ -1256,21 +1280,21 @@ There is a possible fingerprinting attack where a log issues a unique SCT for targeted log client(s). A colluding log and HTTPS server operator could therefore be a threat to the privacy of an HTTPS client. Given all the other opportunities for HTTPS servers to fingerprint clients - TLS session tickets, HPKP and HSTS headers, HTTP Cookies, etc. - this is considered acceptable. The fingerprinting attack described above would be mitigated by a requirement that logs must use a deterministic signature scheme when - signing SCTs ([RFC-6962-BIS-09] Section 2.1.4). A log signing using + signing SCTs ([RFC-6962-BIS-27] section 2.2). A log signing using RSA is not required to use a deterministic signature scheme. Since logs are allowed to issue a new SCT for a certificate already present in the log, mandating deterministic signatures does not stop this fingerprinting attack altogether. It does make the attack harder to pull off without being detected though. There is another similar fingerprinting attack where an HTTPS server tracks a client by using a unique certificate or a variation of cert chains. The risk for this attack is accepted on the same grounds as @@ -1278,21 +1302,21 @@ 10.5.3. Privacy for HTTPS clients performing STH Proof Fetching An HTTPS client performing Proof Fetching SHOULD NOT request proofs from a CT log that it doesn't accept SCTs from. An HTTPS client SHOULD regularly request an STH from all logs it is willing to accept, even if it has seen no SCTs from that log. The time between two polls for new STH's SHOULD NOT be significantly shorter than the MMD of the polled log divided by its STH Frequency - Count ([RFC-6962-BIS-09] section 5.1). + Count ([RFC-6962-BIS-27] section 4.1). The actual mechanism by which Proof Fetching is done carries considerable privacy concerns. Although out of scope for the document, DNS is a mechanism currently discussed. DNS exposes data in plaintext over the network (including what sites the user is visiting and what sites they have previously visited) and may not be suitable for some. 10.5.4. Privacy in STH Pollination @@ -1311,21 +1335,24 @@ There is a possible fingerprinting attack where a log issues a unique STH for a targeted HTTPS client. This is similar to the fingerprinting attack described in Section 10.5.2, but can operate cross-origin. If a log (or HTTPS server cooperating with a log) provides a unique STH to a client, the targeted client will be the only client pollinating that STH cross-origin. It is mitigated partially because the log is limited in the number of STHs it can issue. It must 'save' one of its STHs each MMD to - perform the attack. + perform the attack. A log violating its STH Frequency Count + ([RFC-6962-BIS-27] section 4.1) can be identified as non-compliant by + CT auditors following the procedure described in [RFC-6962-BIS-27] + section 8.3. 10.5.5. Privacy in STH Interaction An HTTPS client may pollinate any STH within the last 14 days. An HTTPS client may also pollinate an STH for any log that it knows about. When a client pollinates STHs to a server, it will release more than one STH at a time. It is unclear if a server may 'prime' a client and be able to reliably detect the client at a later time. It's clear that a single site can track a user any way they wish, but @@ -1355,21 +1382,21 @@ to the auditor. In particular, it will identify the web sites that the client has visited to the auditor. Some clients may already share this information to a third party, for example, when using a server to synchronize browser history across devices in a server- visible way, or when doing DNS lookups through a trusted DNS resolver. For clients with such a relationship already established, sending SCTs to a trusted auditor run by the same organization does not appear to expose any additional information to the trusted third party. - Clients who wish to contact a CT auditor without associating their + Clients which wish to contact a CT auditor without associating their identities with their SCTs may wish to use an anonymizing network like Tor to submit SCT Feedback to the auditor. Auditors SHOULD accept SCT Feedback that arrives over such anonymizing networks. Clients sending feedback to an auditor may prefer to reduce the temporal granularity of the history exposure to the auditor by caching and delaying their SCT Feedback reports. This is elaborated upon in Section 11.3. This strategy is only as effective as the granularity of the timestamps embedded in the SCTs and STHs. @@ -1390,21 +1417,21 @@ consistency proof between two valid STHs or (b) an inclusion proof for a certificate to an STH any time after the log's MMD has elapsed from the issuance of the SCT. The log's inability to provide either proof will not be externally cryptographically- verifiable, as it may be indistinguishable from a network error. 11. Policy Recommendations This section is intended as suggestions to implementors of HTTPS Clients, HTTPS servers, and CT auditors. It is not a requirement for - technique of implementation, so long as privacy considerations + technique of implementation, so long as the privacy considerations established above are obeyed. 11.1. Blocking Recommendations 11.1.1. Frustrating blocking When making gossip connections to HTTPS servers or Trusted Auditors, it is desirable to minimize the plaintext metadata in the connection that can be used to identify the connection as a gossip connection and therefore be of interest to block. Additionally, introducing @@ -1647,20 +1674,29 @@ 11.4. Concrete Recommendations We present the following pseudocode as a concrete outline of our policy recommendations. Both suggestions presented are applicable to both clients and servers. Servers may not perform proof fetching, in which case large portions of the pseudocode are not applicable. But it should work in either case. + Note that we use a function 'rand()' in the pseudocode, this function + is assumed to be a cryptographically secure pseudorandom number + generator. Additionally, when N unique items are needed, they are + chosen at random by drawing a random index repeatedly until the N + unique items from an array have been chosen. Although simple, when + the array is N or near-N items in length this is inefficient. A + secure shuffle algorithm followed by selecting the first N items may + be more efficient, especially when N is large. + 11.4.1. STH Pollination The STH class contains data pertaining specifically to the STH itself. class STH { uint16 proof_attempts uint16 proof_failure_count uint32 num_reports_to_thirdparty @@ -1699,26 +1735,29 @@ // When it is time to perform STH Pollination, the HTTPS client // calls this function to get a selection of STHs to send as // feedback def get_pollination_selection() { if(len(this.sth_list) < MAX_STH_TO_GOSSIP) return this.sth_list else { indexes = set() modulus = len(this.sth_list) - while(len(indexes) < MAX_STH_TO_GOSSIP) { + outdated_sths = 0 + while(len(indexes) + outdated_sths < MAX_STH_TO_GOSSIP) { r = randomInt() % modulus + if(r not in indexes) // Ignore STHs that are past the validity window but not // yet removed. - if(r not in indexes - && now() - this.sth_list[i].timestamp < TWO_WEEKS) + if(now() - this.sth_list[i].timestamp < TWO_WEEKS) + outdated_sths++; + else indexes.insert(r) } return_selection = [] foreach(i in indexes) { return_selection.insert(this.sth_list[i]) } return return_selection } } @@ -1877,87 +1917,20 @@ counters are reset 4. If a domain succeeds, but then begins failing, it must fail more than 90% of the time (configurable) and then the process begins at (2). If a domain is visited infrequently (say, once every 7 months) then it will be evicted from the cache and start all over again (according to the suggestion values in the below pseudocode). - [ Note: To be certain the logic is correct I give the following test - cases which illustrate the intended behavior. Hopefully the code - matches! - - Succeed 1 Time num_submissions_attempted=1 num_submissions_succeeded=1 num_feedback_loop_failures=0 - Fail 10 Times num_submissions_attempted=11 num_submissions_succeeded=1 num_feedback_loop_failures=0 - ... wait a month ... - Fail 1 month later num_submissions_attempted=12 num_submissions_succeeded=1 num_feedback_loop_failures=1 - ... wait a month ... - Succeed 1 month later num_submissions_attempted=13 num_submissions_succeeded=2 num_feedback_loop_failures=0(r) indicates (Reset) - -> Feedback is attempted regularly. - - Succeed 1 Time num_submissions_attempted=1 num_submissions_succeeded=1 num_feedback_loop_failures=0 - Fail 10 Times num_submissions_attempted=11 num_submissions_succeeded=1 num_feedback_loop_failures=0 - ... wait a month ... - Fail 1 month later num_submissions_attempted=12 num_submissions_succeeded=1 num_feedback_loop_failures=1 - ... wait a month ... - Fail 1 month later num_submissions_attempted=13 num_submissions_succeeded=1 num_feedback_loop_failures=2 - ... wait a month ... - Succeed 1 month later num_submissions_attempted=14 num_submissions_succeeded=2 num_feedback_loop_failures=0(r) - -> Feedback is attempted regularly. - - Succeed 1 Time num_submissions_attempted=1 num_submissions_succeeded=1 num_feedback_loop_failures=0 - Fail 10 Times num_submissions_attempted=11 num_submissions_succeeded=1 num_feedback_loop_failures=0 - ... wait a month ... - Fail 1 month later num_submissions_attempted=12 num_submissions_succeeded=1 num_feedback_loop_failures=1 - ... wait a month ... - Fail 1 month later num_submissions_attempted=13 num_submissions_succeeded=1 num_feedback_loop_failures=2 - ... wait a month ... - Fail 1 month later num_submissions_attempted=14 num_submissions_succeeded=2 num_feedback_loop_failures=3 - ... clear_old_data() is run every hour ... - num_submissions_attempted=0 num_submissions_succeeded=0 num_feedback_loop_failures=3 - sct_feedback_failing_longterm=True - Fail 1 month later num_submissions_attempted=1 num_submissions_succeeded=0 num_feedback_loop_failures=4 - sct_feedback_failing_longterm=True - ... clear_old_data() is run every hour ... - num_submissions_attempted=0(r) num_submissions_succeeded=0 num_feedback_loop_failures=3 - sct_feedback_failing_longterm=True - Succeed 1 month later num_submissions_attempted=2 num_submissions_succeeded=1 num_feedback_loop_failures=0(r) - sct_feedback_failing_longterm=False - -> Feedback is attempted regularly. - - Note above that the second run of clear_old_data() will reset num_submissions_attempted from 1 to 0. This is - CRITICAL. Otherwise, we would have the below bug (where after 10 months of failures, a success would not hit - the required ratio to keep going) - - //The below represents a bug. - Succeed 1 Time num_submissions_attempted=1 num_submissions_succeeded=1 num_feedback_loop_failures=0 - Fail 10 Times num_submissions_attempted=11 num_submissions_succeeded=1 num_feedback_loop_failures=0 - ... wait a month ... - Fail 1 month later num_submissions_attempted=12 num_submissions_succeeded=1 num_feedback_loop_failures=1 - ... wait a month ... - Fail 1 month later num_submissions_attempted=13 num_submissions_succeeded=1 num_feedback_loop_failures=2 - ... wait a month ... - Fail 1 month later num_submissions_attempted=14 num_submissions_succeeded=2 num_feedback_loop_failures=3 - ... clear_old_data() is run every hour ... - num_submissions_attempted=0 num_submissions_succeeded=0 num_feedback_loop_failures=3 - sct_feedback_failing_longterm=True - Fail 1 month later num_submissions_attempted=1 num_submissions_succeeded=0 num_feedback_loop_failures=4 - sct_feedback_failing_longterm=True - Fail 9 times for 9 months - num_submissions_attempted=10 num_submissions_succeeded=0 num_feedback_loop_failures=13 - sct_feedback_failing_longterm=True - Succeed 1 month later num_submissions_attempted=11 num_submissions_succeeded=1 num_feedback_loop_failures=0(r) - sct_feedback_failing_longterm=False - -> Feedback is NOT attempted regularly. \] - //Suggestions: // After concluding a domain doesn't support feedback, we try again // after WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS amount of time to see if // they added support #define WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS 1 month // If we've waited MIN_SCT_FEEDBACK_ATTEMPTS_BEFORE_OMITTING_STORAGE // multiplied by WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS amount of time, we // still attempt SCT Feedback, but no longer bother storing any data // until the domain supports SCT Feedback @@ -1982,33 +1955,33 @@ public datetime last_contact_for_domain // This is the last time SCT Feedback was attempted for the domain. // It is updated whenever feedback is attempted - responsibility for // updating lies OUTSIDE of the class // This is not used when this algorithm runs on servers public datetime last_sct_feedback_attempt // This is the number of times we have waited an // WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS amount of time, and still failed - // e.g. 10 months of failures + // e.g., 10 months of failures // This is not used when this algorithm runs on servers private uint16 num_feedback_loop_failures // This is whether or not SCT Feedback has failed enough times that we - // should not bother storing data for it anymore. It is a small function - // used for illustrative purposes + // should not bother storing data for it anymore. It is a small + // function used for illustrative purposes. // This is not used when this algorithm runs on servers private bool sct_feedback_failing_longterm() - { num_feedback_loop_failures >= MIN_SCT_FEEDBACK_ATTEMPTS_BEFORE_OMITTING_STORAGE } + { num_feedback_loop_failures >= + MIN_SCT_FEEDBACK_ATTEMPTS_BEFORE_OMITTING_STORAGE } // This is the number of SCT Feedback submissions attempted. - // Responsibility for incrementing lies OUTSIDE of the class // (And watch for integer overflows) // This is not used when this algorithm runs on servers public uint16 num_submissions_attempted // This is the number of successful SCT Feedback submissions. This // variable is updated by the class. // This is not used when this algorithm runs on servers private uint16 num_submissions_succeeded @@ -2017,21 +1990,22 @@ SCTBundle[] observed_records // This function can be called to determine if we should attempt // SCT Feedback for this domain. def should_attempt_feedback() { // Servers always perform feedback! if(operator_is_server) return true // If we have not tried in a month, try again - if(now() - last_sct_feedback_attempt > WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS) + if(now() - last_sct_feedback_attempt > + WAIT_BETWEEN_SCT_FEEDBACK_ATTEMPTS) return true // If we have tried recently, and it seems to be working, go for it! if((num_submissions_succeeded / num_submissions_attempted) > MIN_RATIO_FOR_SCT_FEEDBACK_TO_BE_WORKING) return true // Otherwise don't try return false } @@ -2292,21 +2270,21 @@ } } If the cache is getting full, these three member functions of the SCTStoreManager class will be used. // ----------------------------------------------------------------- // This function is called when the cache is not yet full, but is // nearing it. It prioritizes deleting data that should be safe // to delete (because it has been shared with the site or resolved -// to a STH) + // to an STH) def cache_pressure_is_imminent() { bundlesToDelete = [] foreach(domainEntry in all_sct_stores) { foreach(sctBundle in domainEntry.observed_records) { if(proof_fetching_enabled) { // First, queue proofs for anything not already queued. if(!sctBundle.has_been_fully_resolved_to_sths()) { foreach(sct in bundle.sct_list) { @@ -2334,21 +2312,22 @@ } // Third, delete the eligible entries at random until the cache is // at a safe level uint recalculateIndex = 0 #define RECALCULATE_EVERY_N_OPERATIONS 50 while(bundlesToDelete.length > 0 && current_cache_percentage() > CACHE_PRESSURE_SAFE) { uint rndIndex = rand() % bundlesToDelete.length - bundlesToDelete[rndIndex].domainEntry.observed_records.remove(bundlesToDelete[rndIndex].sctBundle) + bundlesToDelete[rndIndex].domainEntry.observed_records.remove( + bundlesToDelete[rndIndex].sctBundle) bundlesToDelete.removeAt(rndIndex) recalculateIndex++ if(recalculateIndex % RECALCULATE_EVERY_N_OPERATIONS == 0) { update_cache_percentage() } } // Finally, tell the proof fetching engine to go faster if(proof_fetching_enabled) { @@ -2367,24 +2346,26 @@ // appear to have proof fetching failures def cache_pressure_almost_full() { uint recalculateIndex = 0 uint savedRecords = 0 #define RECALCULATE_EVERY_N_OPERATIONS 50 while(all_sct_stores.length > savedRecords && current_cache_percentage() > CACHE_PRESSURE_SAFE) { uint rndIndex1 = rand() % all_sct_stores.length - uint rndIndex2 = rand() % all_sct_stores[rndIndex1].observed_records.length + uint rndIndex2 = rand() % + all_sct_stores[rndIndex1].observed_records.length if(proof_fetching_enabled) { - if(all_sct_stores[rndIndex1].observed_records[rndIndex2].max_proof_failures() > + if(all_sct_stores[rndIndex1].observed_records[ + rndIndex2].max_proof_failures() > MIN_PROOF_FAILURES_CONSIDERED_SUSPICIOUS) { savedRecords++ continue } } // If proof fetching is not enabled we need some other logic else { if(sctBundle.num_reports_to_thirdparty == 0) { savedRecords++ @@ -2409,111 +2392,118 @@ // This function is called when the cache is full, and will evict // cache entries at random def cache_is_full() { uint recalculateIndex = 0 #define RECALCULATE_EVERY_N_OPERATIONS 50 while(all_sct_stores.length > 0 && current_cache_percentage() > CACHE_PRESSURE_SAFE) { uint rndIndex1 = rand() % all_sct_stores.length - uint rndIndex2 = rand() % all_sct_stores[rndIndex1].observed_records.length + uint rndIndex2 = rand() % + all_sct_stores[rndIndex1].observed_records.length all_sct_stores[rndIndex1].observed_records.removeAt(rndIndex2) if(all_sct_stores[rndIndex1].observed_records.length == 0) { all_sct_stores.removeAt(rndIndex1) } recalculateIndex++ if(recalculateIndex % RECALCULATE_EVERY_N_OPERATIONS == 0) { update_cache_percentage() } } update_cache_percentage(); } 12. IANA considerations - [ TBD ] + There are no IANA considerations. 13. Contributors The authors would like to thank the following contributors for - valuable suggestions: Al Cutter, Ben Laurie, Benjamin Kaduk, Josef - Gustafsson, Karen Seo, Magnus Ahltorp, Steven Kent, Yan Zhu. + valuable suggestions: Al Cutter, Andrew Ayer, Ben Laurie, Benjamin + Kaduk, Graham Edgecombe, Josef Gustafsson, Karen Seo, Magnus Ahltorp, + Steven Kent, Yan Zhu. 14. ChangeLog +14.1. Changes between ietf-04 and ietf-05 -14.1. Changes between ietf-03 and ietf-04 + o STH and SCT data formats changed to support CT v1 and v2. + + o Address ED review comments. + +14.2. Changes between ietf-03 and ietf-04 o No changes. -14.2. Changes between ietf-02 and ietf-03 +14.3. Changes between ietf-02 and ietf-03 o TBD's resolved. o References added. o Pseduocode changed to work for both clients and servers. -14.3. Changes between ietf-01 and ietf-02 +14.4. Changes between ietf-01 and ietf-02 o Requiring full certificate chain in SCT Feedback. o Clarifications on what clients store for and send in SCT Feedback added. o SCT Feedback server operation updated to protect against DoS attacks on servers. o Pre-Loaded vs Locally Added Anchors explained. o Base for well-known URL's changed. o Remove all mentions of monitors - gossip deals with auditors. o New sections added: Trusted Auditor protocol, attacks by actively malicious log, the Dual-CA compromise attack, policy recommendations, -14.4. Changes between ietf-00 and ietf-01 +14.5. Changes between ietf-00 and ietf-01 o Improve language and readability based on feedback from Stephen Kent. o STH Pollination Proof Fetching defined and indicated as optional. o 3-Method Ecosystem section added. o Cases with Logs ceasing operation handled. o Text on tracking via STH Interaction added. o Section with some early recommendations for mixing added. o Section detailing blocking connections, frustrating it, and the implications added. -14.5. Changes between -01 and -02 +14.6. Changes between -01 and -02 o STH Pollination defined. o Trusted Auditor Relationship defined. o Overview section rewritten. o Data flow picture added. o Section on privacy considerations expanded. -14.6. Changes between -00 and -01 +14.7. Changes between -00 and -01 o Add the SCT feedback mechanism: Clients send SCTs to originating web server which shares them with auditors. o Stop assuming that clients see STHs. o Don't use HTTP headers but instead .well-known URL's - avoid that battle. o Stop referring to trans-gossip and trans-gossip-transport-https - @@ -2523,46 +2513,56 @@ back and add more later. o Add more reasoning about privacy. o Do specify data formats. 15. References 15.1. Normative References - [RFC-6962-BIS-09] + [RFC-6962-BIS-27] Laurie, B., Langley, A., Kasper, E., Messeri, E., and R. - Stradling, "Certificate Transparency", October 2015, + Stradling, "Certificate Transparency", October 2017, . - [RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data - Interchange Format", RFC 7159, March 2014. + [RFC6962] Laurie, B., Langley, A., and E. Kasper, "Certificate + Transparency", RFC 6962, DOI 10.17487/RFC6962, June 2013, + . + + [RFC7159] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data + Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March + 2014, . + + [RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext + Transfer Protocol Version 2 (HTTP/2)", RFC 7540, + DOI 10.17487/RFC7540, May 2015, . 15.2. Informative References [double-keying] Perry, M., Clark, E., and S. Murdoch, "Cross-Origin Identifier Unlinkability", May 2015, . [draft-ct-over-dns] Laurie, B., Phaneuf, P., and A. Eijdenberg, "Certificate Transparency over DNS", February 2016, . - [draft-ietf-trans-threat-analysis-03] - Kent, S., "Attack Model and Threat for Certificate - Transparency", October 2015, + [draft-ietf-trans-threat-analysis-12] + Kent, S., "Attack and Threat Model for Certificate + Transparency", October 2017, . [dual-ca-compromise-attack] Gillmor, D., "can CT defend against dual CA compromise?", n.d., . [gossip-mixing] Ritter, T., "A Bit on Certificate Transparency Gossip",