draft-ietf-cdni-logging-01.txt | draft-ietf-cdni-logging-02.txt | |||
---|---|---|---|---|
Internet Engineering Task Force G. Bertrand, Ed. | Internet Engineering Task Force G. Bertrand, Ed. | |||
Internet-Draft I. Oprescu, Ed. | Internet-Draft I. Oprescu, Ed. | |||
Intended status: Informational E. Stephan | Intended status: Informational France Telecom - Orange | |||
Expires: August 26, 2013 France Telecom - Orange | Expires: November 28, 2013 F. Le Faucheur, Ed. | |||
Cisco Systems | ||||
R. Peterkofsky | R. Peterkofsky | |||
Skytide, Inc. | Skytide, Inc. | |||
F. Le Faucheur, Ed. | May 27, 2013 | |||
Cisco Systems | ||||
P. Grochocki | ||||
Orange Polska | ||||
February 22, 2013 | ||||
CDNI Logging Interface | CDNI Logging Interface | |||
draft-ietf-cdni-logging-01 | draft-ietf-cdni-logging-02 | |||
Abstract | Abstract | |||
This memo specifies the Logging interface between a downstream CDN | This memo specifies the Logging interface between a downstream CDN | |||
(dCDN) and an upstream CDN (uCDN) that are interconnected as per the | (dCDN) and an upstream CDN (uCDN) that are interconnected as per the | |||
CDN Interconnection (CDNI) framework. First, it describes a | CDN Interconnection (CDNI) framework. First, it describes a | |||
reference model for CDNI logging. Then, it specifies the actual | reference model for CDNI logging. Then, it specifies the CDNI | |||
protocol for CDNI logging information exchange covering the | Logging File format and the actual protocol for exchange of CDNI | |||
information elements as well as the transport of those elements. | Logging Files. | |||
Status of this Memo | Status of This Memo | |||
This Internet-Draft is submitted in full conformance with the | This Internet-Draft is submitted in full conformance with the | |||
provisions of BCP 78 and BCP 79. | provisions of BCP 78 and BCP 79. | |||
Internet-Drafts are working documents of the Internet Engineering | Internet-Drafts are working documents of the Internet Engineering | |||
Task Force (IETF). Note that other groups may also distribute | Task Force (IETF). Note that other groups may also distribute | |||
working documents as Internet-Drafts. The list of current Internet- | working documents as Internet-Drafts. The list of current Internet- | |||
Drafts is at http://datatracker.ietf.org/drafts/current/. | Drafts is at http://datatracker.ietf.org/drafts/current/. | |||
Internet-Drafts are draft documents valid for a maximum of six months | Internet-Drafts are draft documents valid for a maximum of six months | |||
and may be updated, replaced, or obsoleted by other documents at any | and may be updated, replaced, or obsoleted by other documents at any | |||
time. It is inappropriate to use Internet-Drafts as reference | time. It is inappropriate to use Internet-Drafts as reference | |||
material or to cite them other than as "work in progress." | material or to cite them other than as "work in progress." | |||
This Internet-Draft will expire on August 26, 2013. | This Internet-Draft will expire on November 28, 2013. | |||
Copyright Notice | Copyright Notice | |||
Copyright (c) 2013 IETF Trust and the persons identified as the | Copyright (c) 2013 IETF Trust and the persons identified as the | |||
document authors. All rights reserved. | document authors. All rights reserved. | |||
This document is subject to BCP 78 and the IETF Trust's Legal | This document is subject to BCP 78 and the IETF Trust's Legal | |||
Provisions Relating to IETF Documents | Provisions Relating to IETF Documents | |||
(http://trustee.ietf.org/license-info) in effect on the date of | (http://trustee.ietf.org/license-info) in effect on the date of | |||
publication of this document. Please review these documents | publication of this document. Please review these documents | |||
skipping to change at page 3, line 7 | skipping to change at page 2, line 24 | |||
modifications of such material outside the IETF Standards Process. | modifications of such material outside the IETF Standards Process. | |||
Without obtaining an adequate license from the person(s) controlling | Without obtaining an adequate license from the person(s) controlling | |||
the copyright in such materials, this document may not be modified | the copyright in such materials, this document may not be modified | |||
outside the IETF Standards Process, and derivative works of it may | outside the IETF Standards Process, and derivative works of it may | |||
not be created outside the IETF Standards Process, except to format | not be created outside the IETF Standards Process, except to format | |||
it for publication as an RFC or to translate it into languages other | it for publication as an RFC or to translate it into languages other | |||
than English. | than English. | |||
Table of Contents | Table of Contents | |||
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 | 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 | |||
1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 | 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 | |||
1.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 8 | 2. CDNI Logging Reference Model . . . . . . . . . . . . . . . . 5 | |||
2. CDNI Logging Reference Model . . . . . . . . . . . . . . . . . 8 | 2.1. CDNI Logging interactions . . . . . . . . . . . . . . . . 5 | |||
2.1. CDNI Logging interactions . . . . . . . . . . . . . . . . 8 | 2.2. Overall Logging Chain . . . . . . . . . . . . . . . . . . 8 | |||
2.2. Overall Logging Chain . . . . . . . . . . . . . . . . . . 12 | 2.2.1. Logging Generation and During-Generation Aggregation 9 | |||
2.2.1. Logging Generation and During-Generation | 2.2.2. Logging Collection . . . . . . . . . . . . . . . . . 10 | |||
Aggregation . . . . . . . . . . . . . . . . . . . . . 13 | 2.2.3. Logging Filtering . . . . . . . . . . . . . . . . . . 10 | |||
2.2.2. Logging Collection . . . . . . . . . . . . . . . . . . 14 | 2.2.4. Logging Rectification and Post-Generation Aggregation 11 | |||
2.2.3. Logging Filtering . . . . . . . . . . . . . . . . . . 14 | 2.2.5. Log-Consuming Applications . . . . . . . . . . . . . 12 | |||
2.2.4. Logging Rectification and Post-Generation | 2.2.5.1. Maintenance/Debugging . . . . . . . . . . . . . . 12 | |||
Aggregation . . . . . . . . . . . . . . . . . . . . . 15 | 2.2.5.2. Accounting . . . . . . . . . . . . . . . . . . . 12 | |||
2.2.5. Log-Consuming Applications . . . . . . . . . . . . . . 15 | 2.2.5.3. Analytics and Reporting . . . . . . . . . . . . . 13 | |||
2.2.5.1. Maintenance/Debugging . . . . . . . . . . . . . . 15 | 2.2.5.4. Security . . . . . . . . . . . . . . . . . . . . 13 | |||
2.2.5.2. Accounting . . . . . . . . . . . . . . . . . . . . 16 | 2.2.5.5. Legal Logging Duties . . . . . . . . . . . . . . 13 | |||
2.2.5.3. Analytics and Reporting . . . . . . . . . . . . . 16 | ||||
2.2.5.4. Security . . . . . . . . . . . . . . . . . . . . . 16 | ||||
2.2.5.5. Legal Logging Duties . . . . . . . . . . . . . . . 16 | ||||
2.2.5.6. Notions common to multiple Log Consuming | 2.2.5.6. Notions common to multiple Log Consuming | |||
Applications . . . . . . . . . . . . . . . . . . . 16 | Applications . . . . . . . . . . . . . . . . . . 13 | |||
3. CDNI Logging Transport Requirements . . . . . . . . . . . . . 18 | 3. CDNI Logging File Format . . . . . . . . . . . . . . . . . . 15 | |||
3.1. Timeliness . . . . . . . . . . . . . . . . . . . . . . . . 19 | 3.1. CDNI Logging File Directives . . . . . . . . . . . . . . 16 | |||
3.2. Reliability . . . . . . . . . . . . . . . . . . . . . . . 19 | 3.2. Logging Records . . . . . . . . . . . . . . . . . . . . . 19 | |||
3.3. Security . . . . . . . . . . . . . . . . . . . . . . . . . 19 | 3.2.1. HTTP Request Logging Record . . . . . . . . . . . . . 20 | |||
3.4. Scalability . . . . . . . . . . . . . . . . . . . . . . . 19 | 3.2.2. CDNI Logging File Example . . . . . . . . . . . . . . 26 | |||
3.5. Consistency between CDNI Logging and CDN Logging . . . . . 20 | 3.3. Fields and Directives Formats . . . . . . . . . . . . . . 27 | |||
3.6. Dispatching/Filtering . . . . . . . . . . . . . . . . . . 20 | 4. CDNI Logging File Exchange Protocol . . . . . . . . . . . . . 27 | |||
4. CDNI Logging Information Structure and Transport . . . . . . . 20 | 4.1. CDNI Logging Feed . . . . . . . . . . . . . . . . . . . . 28 | |||
5. CDNI Logging Fields . . . . . . . . . . . . . . . . . . . . . 22 | 4.2. CDNI Logging File Pull . . . . . . . . . . . . . . . . . 28 | |||
5.1. Semantics of CDNI Logging Fields . . . . . . . . . . . . . 22 | 5. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 29 | |||
5.2. Syntax of CDNI Logging Fields . . . . . . . . . . . . . . 26 | 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 | |||
6. CDNI Logging Records . . . . . . . . . . . . . . . . . . . . . 27 | 7. Security Considerations . . . . . . . . . . . . . . . . . . . 31 | |||
6.1. Content Delivery . . . . . . . . . . . . . . . . . . . . . 27 | 7.1. Authentication, Confidentiality, Integrity Protection . . 31 | |||
6.2. Content Invalidation and Purging . . . . . . . . . . . . . 29 | 7.2. Non Repudiation . . . . . . . . . . . . . . . . . . . . . 32 | |||
6.3. Request Routing . . . . . . . . . . . . . . . . . . . . . 29 | 7.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 32 | |||
6.4. Logging Extensibility . . . . . . . . . . . . . . . . . . 29 | 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 | |||
7. CDNI Logging File Format . . . . . . . . . . . . . . . . . . . 29 | 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 33 | |||
7.1. Logging Files . . . . . . . . . . . . . . . . . . . . . . 29 | 9.1. Normative References . . . . . . . . . . . . . . . . . . 33 | |||
7.2. File Format . . . . . . . . . . . . . . . . . . . . . . . 29 | 9.2. Informative References . . . . . . . . . . . . . . . . . 33 | |||
7.2.1. Headers . . . . . . . . . . . . . . . . . . . . . . . 30 | Appendix A. Requirements . . . . . . . . . . . . . . . . . . . . 34 | |||
7.2.2. Body (Logging Records) Format . . . . . . . . . . . . 31 | A.1. Compliance with cdni-requirements . . . . . . . . . . . . 34 | |||
7.2.3. Footer Format . . . . . . . . . . . . . . . . . . . . 31 | A.2. Additional Requirements . . . . . . . . . . . . . . . . . 34 | |||
8. CDNI Logging File Transport Protocol . . . . . . . . . . . . . 31 | A.2.1. Timeliness . . . . . . . . . . . . . . . . . . . . . 34 | |||
9. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 32 | A.2.2. Reliability . . . . . . . . . . . . . . . . . . . . . 35 | |||
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 | A.2.3. Security . . . . . . . . . . . . . . . . . . . . . . 35 | |||
11. Security Considerations . . . . . . . . . . . . . . . . . . . 32 | A.2.4. Scalability . . . . . . . . . . . . . . . . . . . . . 35 | |||
11.1. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 33 | A.2.5. Consistency between CDNI Logging and CDN Logging . . 35 | |||
11.2. Non Repudiation . . . . . . . . . . . . . . . . . . . . . 33 | A.2.6. Dispatching/Filtering . . . . . . . . . . . . . . . . 35 | |||
12. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33 | Appendix B. Analysis of candidate protocols for Logging | |||
13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 33 | Transport . . . . . . . . . . . . . . . . . . . . . 36 | |||
13.1. Normative References . . . . . . . . . . . . . . . . . . . 33 | B.1. Syslog . . . . . . . . . . . . . . . . . . . . . . . . . 36 | |||
13.2. Informative References . . . . . . . . . . . . . . . . . . 33 | B.2. XMPP . . . . . . . . . . . . . . . . . . . . . . . . . . 36 | |||
Appendix A. Examples Log Format . . . . . . . . . . . . . . . . . 34 | B.3. SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . 36 | |||
A.1. W3C Common Log File (CLF) Format . . . . . . . . . . . . . 35 | Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 36 | |||
A.2. W3C Extended Log File (ELF) Format . . . . . . . . . . . . 35 | ||||
A.3. National Center for Supercomputing Applications (NCSA) | ||||
Common Log Format . . . . . . . . . . . . . . . . . . . . 37 | ||||
A.4. NCSA Combined Log Format . . . . . . . . . . . . . . . . . 37 | ||||
A.5. NCSA Separate Log Format . . . . . . . . . . . . . . . . . 37 | ||||
A.6. Squid 2.0 Native Log Format for Access Logs . . . . . . . 37 | ||||
Appendix B. Requirements . . . . . . . . . . . . . . . . . . . . 38 | ||||
B.1. Additional Requirements . . . . . . . . . . . . . . . . . 38 | ||||
B.2. Compliancy with Requirements draft . . . . . . . . . . . . 39 | ||||
Appendix C. Analysis of candidate protocols for Logging | ||||
Transport . . . . . . . . . . . . . . . . . . . . . . 39 | ||||
C.1. Syslog . . . . . . . . . . . . . . . . . . . . . . . . . . 40 | ||||
C.2. XMPP . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 | ||||
C.3. SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 | ||||
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 40 | ||||
1. Introduction | 1. Introduction | |||
This memo specifies the Logging interface between a downstream CDN | This memo specifies the Logging interface between a downstream CDN | |||
(dCDN) and an upstream CDN (uCDN). First, it describes a reference | (dCDN) and an upstream CDN (uCDN). First, it describes a reference | |||
model for CDNI logging. Then, it specifies the actual protocol for | model for CDNI logging. Then, it specifies the CDNI Logging File | |||
CDNI logging information exchange covering the information elements | format and the actual protocol for exchange of CDNI Logging Files. | |||
as well as the transport of those elements. | ||||
The reader should be familiar with the work of the CDNI WG: | The reader should be familiar with the following documents: | |||
o CDNI problem statement [RFC6707] and framework | o CDNI problem statement [RFC6707] and framework | |||
[I-D.ietf-cdni-framework] identify a Logging interface, | [I-D.ietf-cdni-framework] identify a Logging interface, | |||
o Section 7 of [I-D.ietf-cdni-requirements] specifies a set of | o Section 8 of [I-D.ietf-cdni-requirements] specifies a set of | |||
requirements for Logging, | requirements for Logging, | |||
o [RFC6770] outlines real world use-cases for interconnecting CDNs. | o [RFC6770] outlines real world use-cases for interconnecting CDNs. | |||
These use cases require the exchange of Logging information | These use cases require the exchange of Logging information | |||
between the dCDN and the uCDN. | between the dCDN and the uCDN. | |||
As stated in [RFC6707], "the CDNI Logging interface enables details | As stated in [RFC6707], "the CDNI Logging interface enables details | |||
of logs or events to be exchanged between interconnected CDNs". | of logs or events to be exchanged between interconnected CDNs". | |||
The present document describes: | The present document describes: | |||
o The CDNI Logging reference model (Section 2), | o The CDNI Logging reference model (Section 2), | |||
o The CDNI Logging information structure and Transport (Section 4), | o The CDNI Logging File format (Section 3), | |||
o The CDNI Logging Fields (Section 5), | ||||
o The CDNI Logging Records (Section 6), | ||||
o The CDNI Logging File format (Section 7), | ||||
o The CDNI Logging File Transport Protocol (Section 8), | ||||
In the Appendices, the document provides: | ||||
o A list of identified requirements (Appendix B.1), which should be | o The CDNI Logging File Exchange protocol (Section 4). | |||
considered for inclusion in [I-D.ietf-cdni-requirements], | ||||
1.1. Terminology | 1.1. Terminology | |||
In this document, the first letter of each CDNI-specific term is | In this document, the first letter of each CDNI-specific term is | |||
capitalized. We adopt the terminology described in [RFC6707] and | capitalized. We adopt the terminology described in [RFC6707] and | |||
[I-D.ietf-cdni-framework], and extend it with the additional terms | [I-D.ietf-cdni-framework], and extend it with the additional terms | |||
defined below. | defined below. | |||
For clarity, we use the word "Log" only for referring to internal CDN | For clarity, we use the word "Log" only for referring to internal CDN | |||
logs and we use the word "Logging" for any inter-CDN information | logs and we use the word "Logging" for any inter-CDN information | |||
skipping to change at page 6, line 29 | skipping to change at page 4, line 42 | |||
CDNI Logging Field: an atomic element of information that can be | CDNI Logging Field: an atomic element of information that can be | |||
included in a CDNI Logging Record. The time an event/task started, | included in a CDNI Logging Record. The time an event/task started, | |||
the IP address of an End user to whom content was delivered, and the | the IP address of an End user to whom content was delivered, and the | |||
URI of the content delivered are examples of CDNI Logging Fields. | URI of the content delivered are examples of CDNI Logging Fields. | |||
CDNI Logging Record: an information record providing information | CDNI Logging Record: an information record providing information | |||
about a specific event. This comprises a collection of CDNI Logging | about a specific event. This comprises a collection of CDNI Logging | |||
Fields. | Fields. | |||
Separator Character: a specific character used to enable the parsing | ||||
of Logging Records. This character separates the Logging Fields that | ||||
compose a Logging Record. | ||||
CDNI Logging File: a file containing CDNI Logging Records, as well as | CDNI Logging File: a file containing CDNI Logging Records, as well as | |||
additional information facilitating the processing of the CDNI | additional information facilitating the processing of the CDNI | |||
Logging Records. | Logging Records. | |||
CDN Reporting: the process of providing the relevant information that | CDN Reporting: the process of providing the relevant information that | |||
will be used to create a formatted content delivery report provided | will be used to create a formatted content delivery report provided | |||
to the CSP in deferred time. Such information typically includes | to the CSP in deferred time. Such information typically includes | |||
aggregated data that can cover a large period of time (e.g., from | aggregated data that can cover a large period of time (e.g., from | |||
hours to several months). Uses of Reporting include the collection | hours to several months). Uses of Reporting include the collection | |||
of charging data related to CDN services and the computation of Key | of charging data related to CDN services and the computation of Key | |||
Performance Indicators (KPIs). | Performance Indicators (KPIs). | |||
CDN Monitoring: the process of providing content delivery information | CDN Monitoring: the process of providing content delivery information | |||
in real-time. Monitoring typically includes data in real time to | in real-time. Monitoring typically includes data in real time to | |||
provide visibility of the deliveries in progress, for service | provide visibility of the deliveries in progress, for service | |||
operation purposes. It presents a view of the global health of the | operation purposes. It presents a view of the global health of the | |||
services as well as information on usage and performance, for network | services as well as information on usage and performance, for network | |||
services supervision and operation management. In particular, | services supervision and operation management. In particular, | |||
monitoring data can be used to generate alarms. | monitoring data can be used to generate alarms. | |||
End-User experience management: study of Logging data using | ||||
statistical analysis to discover, understand, and predict user | ||||
behavior patterns. | ||||
Class-of-requests: A Class-of-requests identifies a set of content | ||||
Requests, related to a specific CSP, received from clients in a given | ||||
footprint and sharing common properties. These properties include: | ||||
o Any header, URL parameter, query parameter of an HTTP (or RTMP) | ||||
content request | ||||
o Any header, or sub-domain of the FQDN of a DNS lookup request | ||||
Examples: | ||||
o Class-of-Requests = all the requests that include the HTTP header | ||||
"User-Agent: Mozilla/5.0" related to CSP | ||||
"http://*.cdn.example.com" from AS3215 | ||||
o Class-of-Requests = all the DNS requests from anywhere and related | ||||
to CSP "cdn*.example.com" | ||||
Delivery Service: A Delivery Service is defined by a set of Class-of- | ||||
Requests and a list of parameters that apply to all these Class-of- | ||||
Requests (logging format, delivery quality/capabilities | ||||
requirements...) | ||||
Service Agreement: A service agreement is defined by a uCDN | ||||
identifier, a dCDN identifier, a set of Delivery Services and a list | ||||
of parameters that apply to the Service Agreement. | ||||
Once a Service Agreement is agreed between the administrative | ||||
entities managing the CDNs to be interconnected, the upstream CDN and | ||||
the downstream CDN of the CDNI interconnection must be configured | ||||
according to this agreed Service Agreement. For instance, a given | ||||
uCDN (uCDN1) may request a given dCDN (dCDN1) to configure one | ||||
Delivery Service for handling requests for HTTP Adaptive streaming | ||||
videos delegated by uCDN1 and related to a specific CSP (CSP1) and | ||||
another one for handling requests for static pictures delegated by | ||||
uCDN1 and related to CSP1. These Delivery services would belong to | ||||
the Service Agreement between uCDN1 and dCDN1 for CSP1. In this | ||||
simple example, uCDN1 may request dCDN1 to include Delivery Service | ||||
information in its CDNI Logging, to help uCDN1 to provide relevant | ||||
reports to CSP1. | ||||
1.2. Abbreviations | ||||
o API: Application Programming Interface | ||||
o CCID: Content Collection Identifier | ||||
o CDN: Content Delivery Network | ||||
o CDNP: Content Delivery Network Provider | ||||
o CoDR: Content Delivery Record | ||||
o CSP: Content Service Provider | ||||
o DASH: Dynamic Adaptive Streaming over HTTP | ||||
o dCDN: downstream CDN | ||||
o FTP: File Transfer Protocol | ||||
o HAS: HTTP Adaptive Streaming | ||||
o KPI: Key Performance Indicator | ||||
o PVR: Personal Video Recorder | ||||
o SID: Session Identifier | ||||
o SFTP: SSH File Transfer Protocol | ||||
o SNMP: Simple Network Management Protocol | ||||
o uCDN: upstream CDN | ||||
2. CDNI Logging Reference Model | 2. CDNI Logging Reference Model | |||
2.1. CDNI Logging interactions | 2.1. CDNI Logging interactions | |||
The CDNI logging reference model between a given uCDN and a given | The CDNI logging reference model between a given uCDN and a given | |||
dCDN involves the following interactions: | dCDN involves the following interactions: | |||
o customization by the uCDN of the CDNI logging information to be | o customization by the uCDN of the CDNI logging information to be | |||
provided by the dCDN to the uCDN (e.g. control of which logging | provided by the dCDN to the uCDN (e.g. control of which logging | |||
fields are to be communicated to the uCDN for a given task | fields are to be communicated to the uCDN for a given task | |||
performed by the dCDN, control of which types of events are to be | performed by the dCDN, control of which types of events are to be | |||
logged). The dCDN takes into account this CDNI logging | logged). The dCDN takes into account this CDNI logging | |||
customization information to determine what logging information to | customization information to determine what logging information to | |||
provide to the uCDN, but it may, or may not, take into account | provide to the uCDN, but it may, or may not, take into account | |||
this CDNI logging customization information to influence what CDN | this CDNI logging customization information to influence what CDN | |||
logging information is to be generated and collected within the | logging information is to be generated and collected within the | |||
dCDN (e.g. even if the uCDN requests a restricted subset of the | dCDN (e.g. even if the uCDN requests a restricted subset of the | |||
logging information, the dCDN may elect to generate a broader set | logging information, the dCDN may elect to generate a broader set | |||
of logging information). The mechanism to support the | of logging information). The mechanism to support the | |||
customisation by the uCDN of CDNI Logging information is outside | customisation by the uCDN of CDNI Logging information is outside | |||
the scope of this document and left for further study. We note | the scope of this document and left for further study. We note | |||
that the CDNI Control interface ore the CDNI Metadata interfaces | that the CDNI Control interface or the CDNI Metadata interface | |||
appear as candidate interfaces on which to potentially build such | appear as candidate interfaces on which to potentially build such | |||
a customisation mechanism. Before such a mechanism is available, | a customisation mechanism in the future. Before such a mechanism | |||
the uCDN and dCDN are expected to agree off-line on what CDNI | is available, the uCDN and dCDN are expected to agree off-line on | |||
logging information is to be provide by dCDN to UCDN and rely on | what CDNI logging information is to be provide by dCDN to UCDN and | |||
management plane actions to configure the CDNI Logging functions | rely on management plane actions to configure the CDNI Logging | |||
to generate (respectively, expect) in dCDN (respectively, in | functions to generate (respectively, expect) in dCDN | |||
uCDN). | (respectively, in uCDN). | |||
o generation and collection by the dCDN of logging information | o generation and collection by the dCDN of logging information | |||
related to the completion of any task performed by the dCDN on | related to the completion of any task performed by the dCDN on | |||
behalf of the uCDN (e.g., delivery of the content to an end user) | behalf of the uCDN (e.g., delivery of the content to an end user) | |||
or related to events happening in the dCDN that are relevant to | or related to events happening in the dCDN that are relevant to | |||
the uCDN (e.g., failures or unavailability in dCDN). This takes | the uCDN (e.g., failures or unavailability in dCDN). This takes | |||
place within the dCDN and does not directly involve CDNI | place within the dCDN and does not directly involve CDNI | |||
interfaces. | interfaces. | |||
o communication by the dCDN to the uCDN of the logging information | o communication by the dCDN to the uCDN of the logging information | |||
skipping to change at page 10, line 27 | skipping to change at page 6, line 48 | |||
o to analyze the performance of the delivery operated by the dCDNs | o to analyze the performance of the delivery operated by the dCDNs | |||
and to adjust its operations (e.g., request routing) as | and to adjust its operations (e.g., request routing) as | |||
appropriate, | appropriate, | |||
o to provide reporting (non real-time) and monitoring (real-time) | o to provide reporting (non real-time) and monitoring (real-time) | |||
information to CSP. | information to CSP. | |||
For instance, uCDN merges Logging data, extracts relevant KPIs, and | For instance, uCDN merges Logging data, extracts relevant KPIs, and | |||
presents a formatted report to the CSP, in addition to a bill for the | presents a formatted report to the CSP, in addition to a bill for the | |||
content delivered by uCDN itself or by its dCDNs on his behalf. uCDN | content delivered by uCDN itself or by its dCDNs on his behalf. uCDN | |||
may also provide Logging data as raw log files to the CSP, so that | may also provide Logging data as raw log files to the CSP, so that | |||
the CSP can use its own logging analysis tools. | the CSP can use its own logging analysis tools. | |||
+-----+ | +-----+ | |||
| CSP | | | CSP | | |||
+-----+ | +-----+ | |||
^ Reporting and monitoring data | ^ Reporting and monitoring data | |||
* Billing | * Billing | |||
,--*--. | ,--*--. | |||
Logging ,-' `-. | Logging ,-' `-. | |||
skipping to change at page 13, line 35 | skipping to change at page 10, line 21 | |||
(e.g., debugging). | (e.g., debugging). | |||
[I-D.brandenburg-cdni-has] discusses logging for HTTP Adaptive | [I-D.brandenburg-cdni-has] discusses logging for HTTP Adaptive | |||
Streaming (HAS). In accordance with the recommendations articulated | Streaming (HAS). In accordance with the recommendations articulated | |||
there, it is expected that a surrogate will generate separate logging | there, it is expected that a surrogate will generate separate logging | |||
information for delivery of each chunk of HAS content. This ensures | information for delivery of each chunk of HAS content. This ensures | |||
that separate logging information can then be provided to | that separate logging information can then be provided to | |||
interconnected CDNs over the CDNI Logging interface. Still in line | interconnected CDNs over the CDNI Logging interface. Still in line | |||
with the recommendations of [I-D.brandenburg-cdni-has], the logging | with the recommendations of [I-D.brandenburg-cdni-has], the logging | |||
information for per-chunck delivery may include some information (a | information for per-chunck delivery may include some information (a | |||
Content Collection IDentifier and a Session IDentifier as discussed | Content Collection IDentifier and a Session IDentifier) intended to | |||
in Section 5) intended to facilitate subsequent post-generation | facilitate subsequent post-generation aggregation of per-chunk logs | |||
aggregation of per-chunk logs into per-session logs. Note that a CDN | into per-session logs. Note that a CDN may also elect to generate | |||
may also elect to generate aggregate per-session logs when performing | aggregate per-session logs when performing HAS delivery, but this | |||
HAS delivery, but this needs to be in addition to, and not instead | needs to be in addition to, and not instead of, the per-chunk | |||
of, the per-chunk delivery logs. We note that this may be revisited | delivery logs. We note that this may be revisited in future versions | |||
in future versions of this document. | of this document. | |||
Note that in the case of non real-time logging, the trigger of the | Note that in the case of non real-time logging, the trigger of the | |||
transmission or generation of the logging file appears to be a | transmission or generation of the logging file appears to be a | |||
synchronous process from a protocol standpoint. The implementation | synchronous process from a protocol standpoint. The implementation | |||
algorithm can choose to enforce a maximum size for the logging file | algorithm can choose to enforce a maximum size for the logging file | |||
beyound which the transmission is automatically triggered (and thus | beyond which the transmission is automatically triggered (and thus | |||
allow for an asynchrounous transmission process). | allow for an asynchronous transmission process). | |||
2.2.2. Logging Collection | 2.2.2. Logging Collection | |||
This is the process that continuously collects logs generated by the | This is the process that continuously collects logs generated by the | |||
log-generating entities within a CDN. | log-generating entities within a CDN. | |||
In a CDNI environment, in addition to collecting logging information | In a CDNI environment, in addition to collecting logging information | |||
from log-generating entities within the local CDN, the Collection | from log-generating entities within the local CDN, the Collection | |||
process also collects logging information provided by another CDN, or | process also collects logging information provided by another CDN, or | |||
other CDNs, through the CDNI Logging interface. This is illustrated | other CDNs, through the CDNI Logging interface. This is illustrated | |||
skipping to change at page 16, line 12 | skipping to change at page 12, line 46 | |||
Logging enables the CDN providers to identify and troubleshoot | Logging enables the CDN providers to identify and troubleshoot | |||
performance degradations. In particular, Logging enables the | performance degradations. In particular, Logging enables the | |||
communication of traffic data (e.g., the amount of traffic that has | communication of traffic data (e.g., the amount of traffic that has | |||
been forwarded by a dCDN on behalf of an uCDN over a given period of | been forwarded by a dCDN on behalf of an uCDN over a given period of | |||
time), which is particularly useful for CDN and network planning | time), which is particularly useful for CDN and network planning | |||
operations. | operations. | |||
2.2.5.2. Accounting | 2.2.5.2. Accounting | |||
Logging is essential for accounting, to permit inter-CDN billing and | Logging is essential for accounting, to permit inter-CDN billing and | |||
CSP billing by uCDNs. For instance, Logging enables the uCDN to | CSP billing by uCDNs. For instance, Logging information provided by | |||
check the total amount of traffic delivered by every dCDN and for | dCDNs enables the uCDN to compute the total amount of traffic | |||
every Delivery Service, as well as, the associated bandwidth usage | delivered by every dCDN for a particular Content Provider, as well | |||
(e.g., peak, 95th percentile), and the maximum number of simultaneous | as, the associated bandwidth usage (e.g., peak, 95th percentile), and | |||
sessions over a given period of time. | the maximum number of simultaneous sessions over a given period of | |||
time. | ||||
2.2.5.3. Analytics and Reporting | 2.2.5.3. Analytics and Reporting | |||
The goal of analytics is to gather any relevant information to track | The goal of analytics is to gather any relevant information to track | |||
audience, analyze user behavior, and monitor the performance and | audience, analyze user behavior, and monitor the performance and | |||
quality of content delivery. For instance, Logging enables the CDN | quality of content delivery. For instance, Logging enables the CDN | |||
providers to report on content consumption (e.g., delivered sessions | providers to report on content consumption (e.g., delivered sessions | |||
per content) in a specific geographic area. | per content) in a specific geographic area. | |||
The goal of reporting is to gather any relevant information to | The goal of reporting is to gather any relevant information to | |||
skipping to change at page 18, line 6 | skipping to change at page 14, line 39 | |||
o Number of failures listed by failure type (e.g., HTTP error code) | o Number of failures listed by failure type (e.g., HTTP error code) | |||
for requests received from End Users in a given region and for | for requests received from End Users in a given region and for | |||
each piece of content, during a given period of time (e.g., hour/ | each piece of content, during a given period of time (e.g., hour/ | |||
day/week/month) | day/week/month) | |||
o Number and cause of premature delivery termination for End Users | o Number and cause of premature delivery termination for End Users | |||
in a given region and for each piece of content, during a given | in a given region and for each piece of content, during a given | |||
period of time (e.g., hour/day/week/month) | period of time (e.g., hour/day/week/month) | |||
o Maximum and mean number of simultaneous sessions established by | o Maximum and mean number of simultaneous sessions established by | |||
End Users in a given region, for a given Delivery Service, and | End Users in a given region, for a given Content Provider, and | |||
during a given period of time (e.g., hour/day/week/month) | during a given period of time (e.g., hour/day/week/month) | |||
o Volume of traffic delivered for sessions established by End Users | o Volume of traffic delivered for sessions established by End Users | |||
in a given region, for a given Delivery Service, and during a | in a given region, for a given Content Provider, and during a | |||
given period of time (e.g., hour/day/week/month) | given period of time (e.g., hour/day/week/month) | |||
o Maximum, mean, and minimum delivery throughput for sessions | o Maximum, mean, and minimum delivery throughput for sessions | |||
established by End Users in a given region, for a given Delivery | established by End Users in a given region, for a given Content | |||
Service, and during a given period of time (e.g., hour/day/week/ | Provider, and during a given period of time (e.g., hour/day/week/ | |||
month) | month) | |||
o Cache-hit and byte-hit ratios for requests received from End Users | o Cache-hit and byte-hit ratios for requests received from End Users | |||
in a given region for each piece of content, during a given period | in a given region for each piece of content, during a given period | |||
of time (e.g., hour/day/week/month) | of time (e.g., hour/day/week/month) | |||
o Top 10 of the most popularly requested content (during a given | o Top 10 of the most popularly requested content (during a given day | |||
day/week/month), | /week/month), | |||
o Terminal type (mobile, PC, STB, if this information can be | o Terminal type (mobile, PC, STB, if this information can be | |||
acquired from the browser type header, for example). | acquired from the browser type header, for example). | |||
Additional KPIs can be computed from other sources of information | Additional KPIs can be computed from other sources of information | |||
than the Logging, for instance, data collected by a content portal or | than the Logging, for instance, data collected by a content portal or | |||
by specific client-side APIs. Such KPIs are out of scope for the | by specific client-side application programming interfaces. Such | |||
present memo. | KPIs are out of scope for the present memo. | |||
The KPIs used depend strongly on the considered log-consuming | The KPIs used depend strongly on the considered log-consuming | |||
application -- the CDN operator may be interested in different | application -- the CDN operator may be interested in different | |||
metrics than the CSP is. In particular, CDN operators are often | metrics than the CSP is. In particular, CDN operators are often | |||
interested in delivery and acquisition performance KPIs, information | interested in delivery and acquisition performance KPIs, information | |||
related to Surrogates' performance, caching information to evaluate | related to Surrogates' performance, caching information to evaluate | |||
the cache-hit ratio, information about the delivered file size to | the cache-hit ratio, information about the delivered file size to | |||
compute the volume of content delivered during peak hour, etc. | compute the volume of content delivered during peak hour, etc. | |||
Some of the KPIs, for instance those providing an instantaneous | Some of the KPIs, for instance those providing an instantaneous | |||
vision of the active sessions for a given CSP's content, are useful | vision of the active sessions for a given CSP's content, are useful | |||
essentially if they are provided in real-time. By contrast, some | essentially if they are provided in real-time. By contrast, some | |||
other KPIs, such as the one averaged on a long period of time, can be | other KPIs, such as the one averaged on a long period of time, can be | |||
provided in non-real time. | provided in non-real time. | |||
3. CDNI Logging Transport Requirements | 3. CDNI Logging File Format | |||
3.1. Timeliness | ||||
Some applications consuming CDNI Logging information, such as | ||||
accounting or trend analytics, only require logging information to be | ||||
available with a timeliness of the order of a day or the hour. This | ||||
document focuses on addressing this requirement. | ||||
Some applications consuming CDNI Logging information, such as real- | ||||
time analytics, require logging information to be available in real- | ||||
time (i.e. of the order of a second after the corresponding event). | ||||
This document leaves this requirement out of scope. | ||||
3.2. Reliability | ||||
CDNI logging information must be transmitted reliably. The transport | ||||
protocol should contain an anti-replay mechanism. | ||||
3.3. Security | ||||
CDNI logging information exchange must allow authentication, | ||||
integrity protection, and confidentiality protection. Also, a non- | ||||
repudiation mechanism is mandatory, the transport protocol should | ||||
support it. | ||||
3.4. Scalability | ||||
CDNI logging information exchange must support large scale | ||||
information exchange, particularly so in the presence of HTTP | ||||
Adaptive Streaming. | ||||
For example, if we consider a client pulling HTTP Progressive | ||||
Download content with an average duration of 10 minutes, this | ||||
represents 1/600 CDNI delivery Logging Records per second. If we | ||||
assume the dCDN is simultaneously serving 100,000 such clients on | ||||
behalf of the uCDN, the dCDN will be generating 167 Logging Records | ||||
per second to be communicated to the uCDN over the CDNI Logging | ||||
interface. Or equivalently, if we assume an average delivery rate of | ||||
2Mb/s, the dCDN generates 0.83 CDNI Logging Records per second for | ||||
every Gb/s of streaming on behalf of the uCDN. | ||||
For example, if we consider a client pulling HAS content and | ||||
receiving a video chunk every 2 seconds, a separate audio chunck | ||||
every 2 seconds and a refreshed manifest every 10 seconds, this | ||||
represents 1.1 delivery Logging Record per second. If we assume the | ||||
dCDN is simultaneously serving 100,000 such clients on behalf of the | ||||
uCDN, the dCDN will be generating 110,000 Logging Records per second | ||||
to be communicated to the uCDN over the CDNI Logging interface. Or | ||||
equivalently, if we assume an average delivery rate of 2Mb/s, the | ||||
dCDN generates 550 CDNI Logging Records per second for every Gb/s of | ||||
streaming on behalf of the uCDN. | ||||
3.5. Consistency between CDNI Logging and CDN Logging | ||||
There are benefits in using a CDNI logging format as close as | ||||
possible to intra-CDN logging format commonly used in CDNs tody in | ||||
order to minimize systematic translation at CDN/CDNI boundary. | ||||
3.6. Dispatching/Filtering | ||||
When a CDN is acting as a dCDN for multiple uCDNs, the dCDN needs to | ||||
dispatch each CDNI Logging Record to the uCDN that redirected the | ||||
corresponding request. The CDNI Logging format need to allow, and | ||||
possibly facilitate, such a dispatching. | ||||
4. CDNI Logging Information Structure and Transport | ||||
As defined in Section 1.1 a CDNI logging field is as an atomic | As defined in Section 1.1 a CDNI logging field is as an atomic | |||
logging information element and a CDNI Logging Record is a collection | logging information element and a CDNI Logging Record is a collection | |||
of CDNI Logging Fields containing all logging information | of CDNI Logging Fields containing all logging information | |||
corresponding to a single logging event. | corresponding to a single logging event. This document defines a | |||
third level of structure, the CDNI Logging File, that is a collection | ||||
This document defines non-real-time transport of CDNI Logging | of CDNI Logging Records. This structure is illustrated in Figure 3. | |||
information over the CDNI interface. For such non-real-time | The CDNI Logging File structure and encoding is specified in the | |||
transport, this documents defines a third level of structure, the | present section. | |||
CDNI Logging File, that is a collection of CDNI Logging Records. | ||||
This structure is described in Figure 3. This document then | ||||
specifies how to transport such CDNI Logging Files across | ||||
interconnected CDNs. We observe that this approach can be tuned in a | ||||
real deployment to achieve near-real time exchange of CDNI Logging | ||||
information, e.g., by increasing the frequency of logging file | ||||
creation and distribution throughout the Logging chain, but it is not | ||||
expected that this approach can support real time transport (e.g., | ||||
sub-second) of CDNI logging information. | ||||
+------------------------------------------------------+ | +------------------------------------------------------+ | |||
|CDNI Logging File | | |CDNI Logging File | | |||
| | | | | | |||
| +--------------------------------------------------+ | | | +--------------------------------------------------+ | | |||
| |CDNI Logging Record | | | | |CDNI Logging Record | | | |||
| | +-------------+ +-------------+ +-------------+ | | | | | +-------------+ +-------------+ +-------------+ | | | |||
| | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | |||
| | | Field | | Field | | Field | | | | | | | Field | | Field | | Field | | | | |||
| | +-------------+ +-------------+ +-------------+ | | | | | +-------------+ +-------------+ +-------------+ | | | |||
skipping to change at page 21, line 35 | skipping to change at page 16, line 26 | |||
| |CDNI Logging Record | | | | |CDNI Logging Record | | | |||
| | +-------------+ +-------------+ +-------------+ | | | | | +-------------+ +-------------+ +-------------+ | | | |||
| | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | |||
| | | Field | | Field | | Field | | | | | | | Field | | Field | | Field | | | | |||
| | +-------------+ +-------------+ +-------------+ | | | | | +-------------+ +-------------+ +-------------+ | | | |||
| +--------------------------------------------------+ | | | +--------------------------------------------------+ | | |||
+------------------------------------------------------+ | +------------------------------------------------------+ | |||
Figure 3: Structure of Logging Files | Figure 3: Structure of Logging Files | |||
It is expected that future version of this document will also specify | The CDNI Logging File format is inspired from the W3C Extended Log | |||
real time transport of CDNI Logging information over the CDNI | File Format [ELF]. However, it is fully specified by the present | |||
interface. We note that this might involve direct transport of CDNI | document. Where the present document differs from the W3C Extended | |||
Logging Records without prior grouping into a file structure to avoid | Log File Format, an implementation of CDNI Logging MUST comply with | |||
the latency associated with creating and transporting such a file | the present document. | |||
structure throughout the logging chain. | ||||
The semantics and encoding of the CDNI Logging fields are specified | A CDNI Logging File MUST contain a sequence of lines containing US- | |||
in Section 5. The semantics and encoding of CDNI Records are | ASCII characters [CHAR_SET] terminated by either the sequence LF or | |||
specified in Section 6. The CDNI Logging File format is specified in | CRLF. A CDNI Logging implementation consuming CDNI Logging Files | |||
Section 7. The protocol for transport of CDNI Logging File is | MUST accept lines terminated by either LF or CRLF. | |||
specified in Section 8. | ||||
5. CDNI Logging Fields | Each line of a CDNI Logging File MUST contain either a directive or a | |||
CDNI Logging Record. | ||||
Existing CDNs Logging functions collect and consolidate logs | Directives record information about the CDNI Logging process itself. | |||
performed by their Surrogates. Surrogates usually store the logs | Lines containing directives MUST begin with the "#" character. | |||
using a format derived from Web servers' and caching proxies' log | Directives are specified in Section 3.1. | |||
standards such as W3C, NCSA [ELF] [CLF], or Squid format [squid]. In | ||||
practice, these formats are adapted to cope with CDN specifics. | ||||
Appendix A presents examples of commonly used log formats. | ||||
5.1. Semantics of CDNI Logging Fields | Logging Records provide actual details of the logged event. Logging | |||
Records are specified in Section 3.2. | ||||
This section specifies the semantics of the CDNI Logging Fields. The | 3.1. CDNI Logging File Directives | |||
specific subset of CDNI Logging fields that can be found in each type | An implementation of the CDNI Logging interface MUST support the | |||
of Logging Record is specified in Section 6. | following directives (formats specified in the form <...> are | |||
specified in Section 3.3): | ||||
The semantics of the CDNI Logging Fields are specified in Table 1. | o Version: | |||
+--------------+----------------------------------------------------+ | * format: <digit>.<digit> | |||
| Name | Description | | ||||
+--------------+----------------------------------------------------+ | ||||
| Start-time | A start date and time associated with a logged | | ||||
| | event; for instance, the time at which a Surrogate | | ||||
| | received a content delivery request or the time at | | ||||
| | which an origin server received a content | | ||||
| | acquisition request. | | ||||
| End-time | An end date and time associated with a logged | | ||||
| | event. For instance, the time at which a | | ||||
| | Surrogate completed the handling of a content | | ||||
| | delivery request (e.g., end of delivery or error). | | ||||
| Duration | The duration of an operation in milliseconds. For | | ||||
| | instance, this field could be used to provide the | | ||||
| | time it took the Surrogate to send the requested | | ||||
| | file to the End-User or the time it took the | | ||||
| | Surrogate to acquire the file on a cache-miss | | ||||
| | event. In the case where Start-time, End-time, | | ||||
| | and Duration appear in a Logging Record, the | | ||||
| | Duration is to be interpreted as a total activity | | ||||
| | time related to the logged operation. | | ||||
| Client-IP | The IP address of the User Agent that issued the | | ||||
| | logged request or of a proxy, for instance | | ||||
| | "203.0.113.1". | | ||||
| Client-port | The source port of the logged request (e.g., 9542) | | ||||
| Destination- | The IP address of the host that received the | | ||||
| IP | logged request (e.g., 192.0.2.2). | | ||||
| Destination- | The hostname of the host that received the logged | | ||||
| hostname | request (e.g., Surrogate1.cdna.com). | | ||||
| Destination- | The destination port of the logged request (e.g., | | ||||
| port | 80). | | ||||
| Operation | The kind of operation that is logged; for instance | | ||||
| | Delivery or Purging. | | ||||
| URI_full | The full requested URL (e.g., | | ||||
| | "http://node1.peer-a.op-b.net/cdn.csp.com/movies/p | | ||||
| | otter.avi?param=11&user=toto"). When HTTP request | | ||||
| | redirection is used, this URI includes the | | ||||
| | Surrogate FQDN. If the association of requests t | | ||||
| | oSurrogates is confidential, the dCDN can present | | ||||
| | only URI_part to uCDN. | | ||||
| URI_part | The requested URL path (e.g., | | ||||
| | /cdn.csp.com/movies/potter.avi?param=11&user=toto | | ||||
| | if the full request URL was | | ||||
| | "http://node1.peer-a.op-b.net/cdn.csp.com/movies/p | | ||||
| | otter.avi?param=11&user=toto"). The URI without | | ||||
| | host-name typically includes the "CDN domain" | | ||||
| | (ex.cdn.csp.com) - cf. [I-D.ietf-cdni-framework]: | | ||||
| | it enables the identification of the CSP service | | ||||
| | agreed between the CSP and the CDNP operating the | | ||||
| | uCDN. | | ||||
| Protocol | The protocol and protocol version of the message | | ||||
| | that triggered the Logging entry (e.g., HTTP/1.1). | | ||||
| Request-meth | The protocol method of the request message that | | ||||
| od | triggered the Logging entry. | | ||||
| Status | The protocol status of the reply message related | | ||||
| | to the Logging entry | | ||||
| Bytes-Sent | The number of bytes at application-layer | | ||||
| | protocol-level (e.g., HTTP) of the reply message | | ||||
| | related to the Logging entry. It includes the | | ||||
| | size of the response headers. | | ||||
| Headers-Sent | The number of bytes corresponding to response | | ||||
| | headers at application-layer protocol-level (e.g., | | ||||
| | HTTP) of the reply message related to the Logging | | ||||
| | entry. | | ||||
| Bytes-receiv | The number of bytes (headers + body) of the | | ||||
| ed | message that triggered the Logging entry. | | ||||
| Referrer | The value of the Referrer header in an HTTP | | ||||
| | request. | | ||||
| User-Agent | The value of the User Agent header in an HTTP | | ||||
| | request. | | ||||
| Cookie | The value of the Cookie header in an HTTP request. | | ||||
| Byte-Range | [Ed. note: to be defined] | | ||||
| Cache-contro | The value of the cache-control header in an HTTP | | ||||
| l | answer. This header is particularly important for | | ||||
| | content acquisition logs. | | ||||
| Record-diges | A digest of the Logging Record; it enables | | ||||
| t | detecting corrupted Logging Records. | | ||||
| CCID | A Content Collection IDentifier (CCID) eases the | | ||||
| | correlation of several Logging Records related to | | ||||
| | a Content Collection (e.g., a movie split in | | ||||
| | chunks). | | ||||
| SID | A Session Identifier (SID) eases the correlation | | ||||
| | (and aggregation) of several Logging Records | | ||||
| | related to a session. The SID is especially | | ||||
| | relevant for summarizing HAS Logging information | | ||||
| | [I-D.brandenburg-cdni-has]. | | ||||
| uCDN-ID | An element authenticating the operator of the uCDN | | ||||
| | as the authority having delegated the request to | | ||||
| | the dCDN. | | ||||
| Delivering-C | An identifier (e.g., an aggregation of an IP | | ||||
| DN-ID | address and a FQDN) of the Delivering CDN. The | | ||||
| | Delivering-CDN-ID might be considered as | | ||||
| | confidential by the dCDN. In such case, the dCDN | | ||||
| | could either not provide this field to the uCDN or | | ||||
| | overwrite the Delivering-CDN-ID with its on | | ||||
| | identifier. | | ||||
| Cache-bytes | The number of body bytes served from caches. This | | ||||
| | quantity permits the computation of the byte hit | | ||||
| | ratio. | | ||||
| Action | The Action describes how a given request was | | ||||
| | treated locally: through which transport protocol, | | ||||
| | with or without content revalidation, with a cache | | ||||
| | hit or cache miss, with fresh or stale content, | | ||||
| | and (if relevant) with which error. Example with | | ||||
| | Squid format [squid]: "TCP_REFRESH_FAIL_HIT" means | | ||||
| | that an expired copy of an object requested | | ||||
| | through TCP was in the cache. Squid attempted to | | ||||
| | make an If-Modified-Since request, but it failed. | | ||||
| | The old (stale) object was delivered to the | | ||||
| | client. | | ||||
| MIME-Type | The MIME-Type of the requested content | | ||||
| dCDN | An element authenticating the operator of the dCDN | | ||||
| identifier | as the authority requesting the content to the | | ||||
| | uCDN | | ||||
| Caching_date | Date at which the delivered content was stored in | | ||||
| | cache | | ||||
| Validity_hea | A copy of all headers related to content validity: | | ||||
| ders | Pragma or Cache-Control (no-cache), ETag, Vary, | | ||||
| | last-modified... | | ||||
| Lookup_durat | Duration of the DNS resolution for resolving the | | ||||
| ion | FQDN of (uCDN's or CSP's) origin server. | | ||||
| Delay_to_fir | Duration of the operations from the sending of the | | ||||
| st_bit | content acquisition request to the reception of | | ||||
| | the first bit of the requested content. | | ||||
| Delay_to_las | Duration of the operations from the sending of the | | ||||
| t_bit | content acquisition request to the reception of | | ||||
| | the last bit of the requested content. | | ||||
+--------------+----------------------------------------------------+ | ||||
Table 1: Semantics of CDNI Logging Fields | * semantic: indicates the version of the CDNI Logging File | |||
format. The value MUST be "1.0" for the version specified in | ||||
the present document. | ||||
NB: we define three fields related to the timing of logged | * occurrence: there MUST be one and only one instance of this | |||
operations: Start-time, End-time, and Duration. Start-time is | directive. It MUST be the first line of the CDNI Logging file. | |||
typically useful for human readers (e.g., while debugging), however, | ||||
some servers log the operation's End-time which corresponds to the | ||||
time of log record generation. In absence of Logging summarization, | ||||
only two of these three fields are required to obtain relevant timing | ||||
information on the operation. However, when some kind of Logging | ||||
aggregation/summarization is used, it can be advantageous to keep the | ||||
three fields: for instance, in the case of HAS, keeping the three | ||||
fields permits computing an average delivery bitrate from a single | ||||
Logging Record aggregating information on the delivery of multiple | ||||
consecutive video chunks. | ||||
Multiple header fields, in addition to the ones explicitly listed in | o UUID: | |||
the table could be reproduced in the Logging records. | ||||
Note that uCDN may want to filter Logging data by user (and not by IP | * format: <string> | |||
address) to provide more relevant information to the CSP. In such | ||||
case, a user may be identified as a combination of several pieces of | ||||
information such as the client IP and User Agent or through the SID. | ||||
The URI_full provides information on the Surrogate that provided the | * semantic: this is Universally Unique IDentifier for the CDNI | |||
content. This information can be relevant, for instance, for the | Logging File as specified in [RFC4122]. | |||
Inter-Affiliates use case described in [RFC6770]. However, in some | ||||
cases it may be considered as confidential and the dCDN may provide | ||||
URI_part instead. | ||||
Other information that could be logged include operations that refer | * occurrence: there MUST be one and only one instance of this | |||
to the general state of the request, before it gets processed | directive. | |||
locally. Such information is related to the authorization of the | ||||
requests, URL rewriting rules enforced, the X-FORWARDED-FOR non | ||||
standard HTTP header... | ||||
[Editor's Note: CDNI Logging information may be used for debugging. | o Origin: | |||
Therefore, various CDN operations might be logged, depending on the | ||||
agreement between the dCDN and the uCDN, such as operations related | ||||
to Request Routing and Metadata. These may call for a few additional | ||||
Fields to be defined]. | ||||
5.2. Syntax of CDNI Logging Fields | * format: <host> | |||
This section is intended to contain the specification for the syntax | * semantic: this identifies the entity transmitting the CDNI | |||
and encoding of the CDNI Logging fields. For now, Table 2 | Logging File (e.g. the host in a dCDN supporting the CDNI | |||
illustrates the definition of some information elements. It provides | Logging interface) or the entity responsible for transmitting | |||
examples using Apache log format strings [apache] when they exist. | the CDNI Logging File (e.g. the dCDN). | |||
[Ed. note: specify for all Logging Fields the type (e.g., varchar, | * occurrence: there MUST be zero or one instance of this | |||
int, float, ...) and the maximum size (e.g., varchar(200))] | directive. This directive MAY be included by the | |||
+----------+-------------------+------------------------------------+ | implementation transmitting the CDNI Logging file. When | |||
| Name | String | Example | | included by the transmitting side, it MUST be validated or | |||
+----------+-------------------+------------------------------------+ | over-written by the receiving side. When, it is not included | |||
| Time | %t | [10/Oct/2000:13:55:36-0700] | | by the transmitting side, it MAY be added locally by the | |||
| Duration | %D | - | | receiving side. [Editor's Note if we include a non-repudiation | |||
| Client-I | %a | 203.0.113.45 | | mechanism: discuss the fact that this will provide incentive to | |||
| P | | | | dCDN to not cheat , as it can be detected] | |||
| Operatio | - | - | | ||||
| n | | | | ||||
| URI_full | %U | - | | ||||
| Protocol | %H | HTTP/1.0 | | ||||
| Request | %m | GET | | ||||
| method | | | | ||||
| Status | %>s | 200 | | ||||
| Bytes | %O | 2326 | | ||||
| Sent | | | | ||||
| Bytes | %I | 432 | | ||||
| received | | | | ||||
| Header | \"%{Referrer}i\" | "http://www.example.com/start.html | | ||||
| | \"%{User-agent}i\ | ""Mozilla/4.08 [en] (Win98; I | | ||||
| | " | ;Nav)" | | ||||
+----------+-------------------+------------------------------------+ | ||||
Table 2: Examples using Apache format | o Record-Type: | |||
6. CDNI Logging Records | * format: <string> | |||
* semantic: indicates the type of the CDNI Logging Records that | ||||
follow this directive, until another Record-Type directive (or | ||||
the end of the CDNI Logging File). "cdni_http_request_v1" MUST | ||||
be indicated in the Record-Type directive for CDNI Logging | ||||
records corresponding to HTTP request (e.g. a HTTP delivery | ||||
request) as specified in Section 3.2.1. | ||||
[Ed. note: we need to specify the encoding of the file, the | * occurrence: there MUST be at least one instance of this | |||
separation character, etc...] | directive. The first instance of this directive MUST precede a | |||
Fields directive and precede any CDNI Logging Record. | ||||
This section defines the events for which a CDNI Logging record can | o Fields: | |||
be exchanged over the CDNI Logging interafce and for each type of | ||||
Logging Record indicates the allowed set of CDNI Information | ||||
Elements. | ||||
We classify the logged events depending on the CDN operation to which | * format: <field-name>[ <field-name>], where the allowed list of | |||
they relate: Content Delivery, Content Acquisition, Content | <field-name> are specified for each Record-Type in Section 3.2. | |||
Invalidation/Purging, etc. | ||||
6.1. Content Delivery | * semantic: this lists the names of all the fields for which a | |||
value is to appear in the CDNI Logging Records that are after | ||||
this directive. The names of the fields, as well as their | ||||
possible occurrences, are specified for each type of CDNI | ||||
Logging Records in Section 3.2. The field names listed in this | ||||
directive MUST be separated by a whitespace (" "). | ||||
The content delivery event triggering the generation of a Logging | * occurrence: there MUST be at least one instance of this | |||
Record include: | directive per Record-Type directive. The first instance of | |||
this directive for a given Record-Type MUST precede any CDNI | ||||
Logging Record for this Record-Type. | ||||
o Reception by a dCDN Surrogate of a content request | o Integrity-Hash: | |||
The Logging Record for Content Delivery contains the following set of | * format: <string> | |||
CDNI Logging Elements: | ||||
+----------------------+--------------------------------------------+ | * semantic: This directive permits the detection of a corrupted | |||
| Name | Mandatory/Optional | | CDNI Logging File. This can be useful, for instance, if a | |||
+----------------------+--------------------------------------------+ | problem occurs on the filesystem of the dCDN Logging system and | |||
| Start-time | Mandatory | | leads to a truncation of a logging file. The Integrity-Hash | |||
| Duration | Mandatory | | value is computed, and included in this directive by the entity | |||
| Client-IP | Mandatory | | that transmits the CDNI Logging File, by applying the MD5 | |||
| Client-port | Optional | | ([RFC1321]) cryptographic hash function on the CDNI Logging | |||
| Destination-IP | Mandatory if Destination-Hostname is | | File, including all the directives and logging records, up to | |||
| | absent | | the Intergrity-Hash directive itself, excluding the Integrity- | |||
| Destination-Hostname | Mandatory if Destination-IP is absent | | Hash directive itself and, when present, also excluding the | |||
| Destination-port | Optional | | Non-Repudiation-Hash directive. The Integrity-Hash value is | |||
| Operation | Optional | | represented as a US-ASCII encoded hexadecimal number, 32 digits | |||
| URI_full | Mandatory if URI_part is absent | | long (representing a 128 bit hash value). The entity receiving | |||
| URI_part | Mandatory if URI_full is absent | | the CDNI Logging File also computes in a similar way the MD5 | |||
| Protocol | Mandatory if protocol is different to | | hash on the received CDNI Logging File and compares this hash | |||
| | HTTP/1.1 | | to the value of the Integrity-Hash directive. If the two | |||
| Request-method | Mandatory | | values are equal, then the received CDNI Logging File MUST be | |||
| Status | Mandatory | | considered non-corrupted. If the two values are different, the | |||
| Bytes-Sent | Mandatory | | received CDNI Logging File MUST be considered corrupted. The | |||
| Headers-Sent | Optional | | behavior of the entity that received a corrupted CDNI Logging | |||
| Bytes-received | Optional | | File is outside the scope of this specification; we note that | |||
| Referrer | Optional | | the entity MAY attempt to pull again the same CDNI Logging file | |||
| User-Agent | Optional | | from the transmitting entity. | |||
| Cookie | Optional | | ||||
| Byte-Range | ? | | ||||
| Cache-control | Optional | | ||||
| Record-digest | ? | | ||||
| CCID | Optional. Only applicable to HTTP | | ||||
| | Adaptive Streaming delivery. | | ||||
| SID | Optional. Only applicable to HTTP | | ||||
| | Adaptive Streaming delivery. | | ||||
| Cache-bytes | Optional | | ||||
| Action | Mandatory (in particulat re cache | | ||||
| | Hit/Miss) | | ||||
| MIME-Type | Mandatory | | ||||
+----------------------+--------------------------------------------+ | ||||
Table 3: CDNI Logging Fields in Delivery Logging Record | * occurrence: there MUST be one and only one instance of this | |||
directive. This field MUST be the last line of the CDNI | ||||
Logging File when the Non-Repudiation-Hash is absent, and MUST | ||||
be the one before last line of the CDNI Logging File when the | ||||
Non-Repudiation-Hash is present. | ||||
In Table 3, "Mandatory" means that this field MUST be included in | o Non-Repudiation-Hash: | |||
each Delivery Record and "Optional" means that it can be included | ||||
based on the agreement between the dCDN and the uCDN as established | ||||
via mechanism outside the scope of this document (e.g., by human | ||||
agreement). | ||||
6.2. Content Invalidation and Purging | * format: <string> | |||
Given that the Purge interface is expected to contain a mechanism to | * semantic: This hash field permits the non-repudiation of the | |||
report on completion of the Invalidation/purge request, there is no | CDNI Logging File by the entity that transmitted the CDNI | |||
need to specify separate Log Records for these events. | Logging File. [Editor's Note: I need help for specifying the | |||
appropriate hash - ie hash must be signed with private-key of | ||||
entity transmitting the CDNI Logging File] | ||||
6.3. Request Routing | * occurrence: there MAY be one and only one instance of this | |||
directive. When present, this directive MUST be the last line | ||||
of the CDNI Logging File. | ||||
[Editor's Note: Is there a requirement for the dCDN to provide logs | 3.2. Logging Records | |||
for request routing events?] | ||||
6.4. Logging Extensibility | A CDNI Logging Record consists of a sequence of CDNI Logging Fields | |||
relating to that single CDNI Logging Record. | ||||
Future usages might introduce the need for additional Logging fields. | CDNI Logging Fields MUST be separated by the "horizontal tabulation | |||
In addition, some use-cases such as an Inter-Affiliate | (TAB)" character. | |||
Interconnection [RFC6770], might take advantage of extended Logging | ||||
exchanges. Therefore, it is important to permit CDNs to use | ||||
additional Logging fields besides the standard ones, if they want. | ||||
For instance, an "Account-name" identifying the contract enforced by | ||||
the dCDN for a given request could be provided in extended fields. | ||||
The required Logging Records may depend on the considered services. | Some CDNI Logging field names use a prefix scheme similar to the one | |||
For instance, static file delivery (e.g., pictures) typically does | used in W3C Extended Log File Format [ELF] to facilitate readability. | |||
not include any delivery restrictions. By contrast, video delivery | The semantics of the prefix in the present document is: | |||
typically implies strong content delivery restrictions, as explained | ||||
in [RFC6770], and Logging could include information about the | ||||
enforcement of these restrictions. Therefore, to ease the support of | ||||
varied services as well as of future services, the Logging interface | ||||
should support optional Logging Records. | ||||
7. CDNI Logging File Format | o c: refers to the User Agent that issues the request (corresponds | |||
to the "client" of W3C Extended Log Format) | ||||
Interconnected CDNs may support various Logging formats. However, | o s: refers to the dCDN Surrogate that serves the request | |||
they must support at least the default Logging File format described | (corresponds to the "server" of W3C Extended Log Format) | |||
here. | ||||
7.1. Logging Files | o cs: refers to communication from the dCDN Surrogate towards the | |||
User-Agent | ||||
[Ed. Note: How many files (one per type of Delivery Service (e.g., | o sc: refers to communication from the User-Agent towards the dCDN | |||
HTTP, WMP) and per type of Event (e.g., Errors, Delivery, | Surrogate | |||
Acquisition,...?)and what would be inside... These aspects needs to | ||||
be detailed...] | ||||
7.2. File Format | [Editor's Note: see discussion with Rob about adding definition for | |||
"r"] | ||||
The Logging file format should be independent from the selected | An implementation of the CDNI Logging interface as per the present | |||
transport protocol, to guarantee a flexible choice of transport | specification MUST support the CDNI HTTP Delivery Records as | |||
protocols. [Ed. note: for the real time Logging exchanges, this | specified in Section 3.2.1. [Editor's Note": other types of delivery | |||
might be hard] | records will be listed here if we specify other types for this | |||
version eg Request Routing]. | ||||
All Logging Records in a Logging File must share the same format | The formats listed in this section in the form <...> are specified in | |||
(same set of Logging Fields, in the same order, with the same | Section 3.3). | |||
semantics, separated by the same Separator Character), to ease the | ||||
parsing of the Logging data by the CDN that receives the Logging | ||||
File. The CDN that provides the Logging data is responsible for | ||||
guaranteeing the consistency of the Logging records' formats, | ||||
typically via its log filtering and aggregation processes (see | ||||
Section 2.2.3). | ||||
7.2.1. Headers | 3.2.1. HTTP Request Logging Record | |||
Logging files must include a header with the information described in | The HTTP Request Logging Record contains the following CDNI Logging | |||
Figure 4. | Fields, listed by their field name: | |||
+----------------+-------------------+------------------------------+ | o date: | |||
| Field | Description | Examples | | ||||
+----------------+-------------------+------------------------------+ | ||||
| Format | Identification of | standard_cdni_errors_http_v1 | | ||||
| | CDNI Log format. | | | ||||
| Fields | A description of | | | ||||
| | the record format | | | ||||
| | (list of fields). | | | ||||
| Log-ID | Identifier | abcdef1234 | | ||||
| | for the CDNI Log | | | ||||
| | file (facilitates | | | ||||
| | detection of | | | ||||
| | duplicate Logs | | | ||||
| | and tracking in | | | ||||
| | case of | | | ||||
| | aggregation). | | | ||||
| Log-Timestamp | Time, in | [20/Feb/2012:00:29.510+0200] | | ||||
| | milliseconds, the | | | ||||
| | CDNI Log was | | | ||||
| | generated. | | | ||||
| Log-Origin | Identifier of the | cdn1.cdni.example.com | | ||||
| | authority (e.g., | | | ||||
| | dCDN or uCDN) | | | ||||
| | providing the Log-| | | ||||
| | -ging | | | ||||
+----------------+-------------------+------------------------------+ | ||||
Figure 4: Logging Headers | * format: <date> | |||
All time-related Logging Fields and data in the Logging File headers/ | * semantic: the date at which the processing of request started | |||
footers must provide a time zone and be at least at millisecond (ms) | on the Surrogate. | |||
accuracy. The accuracy must be consistent to permit the computation | ||||
of KPIs involving operations realized on several CDNs. | ||||
[Ed. note: would it make sense to add a kind of "example Logging | * occurrence: there MUST be one and only one instance of this | |||
Record" in the Logging file and associated semantic (e.g., in a | field. | |||
structure data format) ?] | ||||
7.2.2. Body (Logging Records) Format | o time: | |||
[Ed. note: the W3C extended log format is a good base candidate to | * format: <time> | |||
look at. ] | ||||
Since records for real time information and non-real time information | * semantic: the time at which the processing of request started | |||
could use different formats, we do not yet solve the problem of real | on the Surrogate. | |||
time logging exchanges in this version. | ||||
7.2.3. Footer Format | * occurrence: there MUST be one and only one instance of this | |||
field. | ||||
Logging files must include a footer with the information described in | o time-taken: | |||
Figure 5. | ||||
+---------+----------------------------------------------+----------+ | * format: <fixed> | |||
| Field | Description | Examples | | ||||
+---------+----------------------------------------------+----------+ | ||||
| Log | Digest of the complete Log (facilitates | | | ||||
| Digest | detection of Log corruption) | | | ||||
+---------+----------------------------------------------+----------+ | ||||
Figure 5: Logging footers | * semantic: duration, in seconds, between the start of the | |||
processing of the request and the completion of the delivery by | ||||
the Surrogate. | ||||
This digest field permits the detection of corrupted Logging files. | * occurrence: there MUST be one and only one instance of this | |||
This can be useful, for instance, if a problem occurs on the | field. | |||
filesystem of the dCDN Logging system and leads to a truncation of a | ||||
logging file. Additional mechanisms to avoid corrupted Logging files | ||||
are expected to be provided by the Logging transport protocol, cf. | ||||
Section 8. | ||||
8. CDNI Logging File Transport Protocol | o c-ip: | |||
As presented in [RFC6707], several protocols already exist that could | * format: <address> | |||
potentially be used to exchange CDNI Logging between interconnected | ||||
CDNs. | ||||
The offline exchange of non real-time Logging could rely on several | * semantic: the source IPv4 or IPv6 address (i.e. the "client" | |||
protocols. In particular, the dCDN could publish the Logging on a | address) in the request received by the Surrogate. | |||
server where the uCDN would retrieve them using a secure protocol. | ||||
For managed file transfer, the recommended protocol is SSH File | * occurrence: there MUST be one and only one instance of this | |||
Transfer Protocol (SFTP) [I-D.ietf-secsh-filexfer]. SFTP is widely | field. | |||
deployed and it guarantees the respect of the criteria expressed by | ||||
the CDNI Logging Transport Requirements: timeliness, reliability, | ||||
security and scalability. | ||||
[Ed note: include options for lossless compression] | o c-port: | |||
9. Open Issues | * format: <integer> | |||
The main remaining tasks on this ID are the following: | * semantic: the source TCP port (i.e. the "client" port) in the | |||
request received by the Surrogate. | ||||
o Finalise the list of CDNI Logging Fields | * occurrence: there MUST be zero or exactly one instance of this | |||
field. | ||||
o Finalise the encoding of CDNI Logging Fields, Records and File. | o s-ip: | |||
o Identify what can be done (if anything) to maximise reuse of | * format: <address> | |||
Logging Fields and Logging Records encoding for future support of | ||||
real-time CDNI Logging exchange | ||||
[Ed. Note: The format for Time is still to be agreed on. RFC 5322 | * semantic: the IPv4 or IPv6 address of the Surrogate that served | |||
(Section 3.3) format could be used or ISO 8601 formatted date and | the request (i.e. the "server" address). | |||
time in UTC (same format as proposed in | ||||
[draft-caulfield-cdni-metadata-core-00]). Also see RFC5424 Section | ||||
6.2.3.] | ||||
[Ed. note: (comment from Kevin) how are errors handled ? If the | * occurrence: there MUST be zero or exactly one instance of this | |||
client gets handed a bunch of 403s and 404s, but still gets the | field. | |||
content eventually, without triggering an event, are those still | ||||
logged? For Bytes-Sent, if there were aborted requests, do those get | ||||
counted as well? Not all client behavior can be correlated with the | ||||
simplified log] | ||||
10. IANA Considerations | o s-hostname: | |||
TBD | * format: <host> | |||
11. Security Considerations | * semantic: the hostname of the Surrogate that served the request | |||
11.1. Privacy | (i.e. the "server" hostname). | |||
CDNs have the opportunity to collect detailed information about the | * occurrence: there MUST be zero or exactly one instance of this | |||
downloads performed by End-Users. The provision of this information | field. | |||
to another CDN introduces End-Users privacy protection concerns. | ||||
11.2. Non Repudiation | o s-port: | |||
Logging provides the raw material for charging. It permits the dCDN | * format: <integer> | |||
to bill the uCDN for the content deliveries that the dCDN makes on | * semantic: the destination TCP port (i.e. the "server" port) in | |||
behalf of the uCDN. It also permits the uCDN to bill the CSP for the | the request received by the Surrogate. | |||
content Delivery Service. Therefore, non-repudiation of Logging data | ||||
is essential. | ||||
12. Acknowledgments | * occurrence: there MUST be zero or exactly one instance of this | |||
field. | ||||
The authors would like to thank Sebastien Cubaud, Anne Marrec, | o cs-method: | |||
Yannick Le Louedec, and Christian Jacquenet for detailed feedback on | ||||
early versions of this document and for their input on existing Log | ||||
formats. | ||||
The authors would like also to thank Fabio Costa, Sara Oueslati, Yvan | * format: <string> | |||
Massot, Renaud Edel, and Joel Favier for their input and comments. | ||||
Finally, they thank the contributors of the EU FP7 OCEAN project for | * semantic: this is the HTTP method of the HTTP request received | |||
valuable inputs. | by the Surrogate. | |||
13. References | * occurrence: There MUST be one and only one instance of this | |||
field. | ||||
13.1. Normative References | o cs-uri: [Editor's note: rename "sr-uri" ?] | |||
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | * format: <uri> | |||
Requirement Levels", BCP 14, RFC 2119, March 1997. | ||||
[RFC5424] Gerhards, R., "The Syslog Protocol", RFC 5424, March 2009. | * semantic: this is the absolute-URI of the request received by | |||
the Surrogate. [Editor's Note: do we agree this should be an | ||||
absolute-URI even if teh request uses a relative-URI?] | ||||
13.2. Informative References | * occurrence: there MUST be zero or exactly one instance of this | |||
field. | ||||
[CLF] A. Luotonen, "The Common Log-file Format, W3C (work in | o ucdn-centric-uri: | |||
progress)", 1995, <http://www.w3.org/pub/WWW/Daemon/User/ | ||||
Config/Logging.html>. | ||||
[ELF] Phillip M. Hallam-Baker and Brian Behlendorf, "Extended | * format: <uri> | |||
Log File Format, W3C (work in progress), WD-logfile- | ||||
960323", <http://www.w3.org/TR/WD-logfile.html>. | ||||
[I-D.brandenburg-cdni-has] | * semantic: this is an absolute URI derived from the absolute-URI | |||
Brandenburg, R., Deventer, O., Faucheur, F., and K. Leung, | of the request received by the Surrogate but modified by the | |||
"Models for adaptive-streaming-aware CDN Interconnection", | entity generating or transmitting the CDNI Logging Record, in a | |||
draft-brandenburg-cdni-has-04 (work in progress), | way that is agreed upon between the two ends of the CDNI | |||
January 2013. | Logging interface. For example, the two ends of the CDNI | |||
Logging interface could agree that the ucdn-centric-uri strips | ||||
the part of the delivery-uri that exposes which individual | ||||
Surrogate actually performed the delivery. The details of | ||||
modification performed to generate the ucdn-centric-uri, as | ||||
well as the mechanism to agree on these modifications between | ||||
the two sides of the CDNI Logging interface are outside the | ||||
scope of the present document. [Editor's Note: do we agree | ||||
this should be an absolute-URI even if the request uses a | ||||
relative-URI?] | ||||
[I-D.ietf-cdni-framework] | * occurrence: there MUST be one and only one instance of this | |||
Peterson, L. and B. Davie, "Framework for CDN | field. | |||
Interconnection", draft-ietf-cdni-framework-03 (work in | ||||
progress), February 2013. | ||||
[I-D.ietf-cdni-requirements] | o protocol: | |||
Leung, K. and Y. Lee, "Content Distribution Network | ||||
Interconnection (CDNI) Requirements", | ||||
draft-ietf-cdni-requirements-04 (work in progress), | ||||
December 2012. | ||||
[I-D.ietf-secsh-filexfer] | * format: <string> | |||
Galbraith, J. and O. Saarenmaa, "SSH File Transfer | ||||
Protocol", draft-ietf-secsh-filexfer-13 (work in | ||||
progress), July 2006. | ||||
[RFC6707] Niven-Jenkins, B., Le Faucheur, F., and N. Bitar, "Content | * semantic: this is value of the HTTP-Version field as specified | |||
Distribution Network Interconnection (CDNI) Problem | in [RFC2616] of the Request-Line of the request received by the | |||
Statement", RFC 6707, September 2012. | Surrogate (e.g. "HTTP/1.1"). | |||
[RFC6770] Bertrand, G., Stephan, E., Burbridge, T., Eardley, P., Ma, | * occurrence: there MUST be one and only one instance of this | |||
K., and G. Watson, "Use Cases for Content Delivery Network | field. | |||
Interconnection", RFC 6770, November 2012. | ||||
[apache] "Apache 2.2 log files documentation", Feb. 2012, | o sc-status: | |||
<http://httpd.apache.org/docs/current/logs.html>. | ||||
[squid] "Squid Log-Format documentation", Feb. 2012, | * format: <digit><digit><digit> | |||
<http://wiki.squid-cache.org/SquidFaq/SquidLogs>. | ||||
Appendix A. Examples Log Format | * semantic: this is the HTTP Status-Code in the HTTP response | |||
from the Surrogate. | ||||
This section provides example of log formats implemented in existing | * occurrence: There MUST be one and only one instance of this | |||
CDNs, web servers, and caching proxies. | field. | |||
Web servers (e.g., Apache) maintain at least one log file for logging | o sc-total-bytes: | |||
accesses to content (the Access Log). They can typically be | ||||
configured to log errors in a separate log file (the Error Log). The | ||||
log formats can be specified in the server's configuration files. | ||||
However, webmasters often use standard log formats to ease the log | ||||
processing with available log analysis tools. | ||||
A.1. W3C Common Log File (CLF) Format | * format: <integer> | |||
The Common Log File (CLF) format defined by the World Wide Web | * semantic: this is the total number of bytes of the HTTP | |||
Consortium (W3C) working group is compatible with many log analysis | response sent by the Surrogate in response to the request. | |||
tools and is supported by the main web servers (e.g., Apache) Access | This includes the bytes of the Status-Line (including HTTP | |||
Logs. | headers) and of the message-body. | |||
According to [CLF], the common log-file format is as follows: | * occurrence: There MUST be one and only one instance of this | |||
remotehost rfc931 authuser [date] "request" status bytes. | field. | |||
Example (from [apache]): 127.0.0.1 - frank [10/Oct/2000:13:55:36 | o sc-entity-bytes: | |||
-0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 | ||||
The fields are defined as follows [CLF]: | * format: <integer> | |||
+------------+------------------------------------------------------+ | * semantic: this is the number of bytes of the message-body in | |||
| Element | Definition | | the HTTP response sent by the Surrogate in response to the | |||
+------------+------------------------------------------------------+ | request. This does not include the bytes of the Status-Line | |||
| remotehost | Remote hostname (or IP number if DNS hostname is not | | (and therefore does not include the bytes of the HTTP headers). | |||
| | available, or if DNSLookup is Off. | | ||||
| rfc931 | The remote logname of the user. | | ||||
| authuser | The username that the user employed to authenticate | | ||||
| | himself. | | ||||
| [date] | Date and time of the request. | | ||||
| "request" | An exact copy of the request line that came from the | | ||||
| | client. | | ||||
| status | The status code of the HTTP reply returned to the | | ||||
| | client. | | ||||
| bytes | The content-length of the document transferred. | | ||||
+------------+------------------------------------------------------+ | ||||
Table 4: Information elements in CLF format | * occurrence: there MUST be zero or exactly one instance of this | |||
field. | ||||
A.2. W3C Extended Log File (ELF) Format | o cs(<HTTP-header>): | |||
The Extended Log File (ELF) format defined by W3C extends the CLF | * format: <string> | |||
with new fields. This format is supported by Microsoft IIS 4.0 and | * semantic: the value of the HTTP header identified in the field | |||
5.0. | name as it appears in the request processed by the Surrogate. | |||
The supported fields are listed below [ELF]. | * occurrence: there MUST be zero, one or any number of instance | |||
of this field. | ||||
+------------+---------------------------------------------------+ | o sc(<HTTP-header>): | |||
| Element | Definition | | ||||
+------------+---------------------------------------------------+ | ||||
| date | Date at which transaction completed | | ||||
| time | Time at which transaction completed | | ||||
| time-taken | Time taken for transaction to complete in seconds | | ||||
| bytes | bytes transferred | | ||||
| cached | Records whether a cache hit occurred | | ||||
| ip | IP address and port | | ||||
| dns | DNS name | | ||||
| status | Status code | | ||||
| comment | Comment returned with status code | | ||||
| method | Method | | ||||
| uri | URI | | ||||
| uri-stem | Stem portion alone of URI (omitting query) | | ||||
| uri-query | Query portion alone of URI | | ||||
+------------+---------------------------------------------------+ | ||||
Table 5: Information elements in ELF format | * format: <string> | |||
Some fields start with a prefix (e.g., "c-", "s-"), which explains | * semantic: the value of the HTTP header identified in the field | |||
which host (client/server/proxy) the field refers to. | name as it appears in the response issued by the Surrogate to | |||
serve the request. | ||||
o Prefix Description | * occurrence: there MUST be zero, one or any number of instance | |||
of this field. | ||||
o c- Client | o s-ccid: | |||
o s- Server | * format: [Editor's Note: to be based on cdni-metadata or | |||
relevant companion I-D] | ||||
o r- Remote | * semantic: this contains the value of the Content Collection | |||
IDentifier specified in [I-D.ietf-cdni-metadata] and associated | ||||
to the content served by the Surrogate through the CDNI | ||||
Metadata interface. | ||||
o cs- Client to Server. | * occurrence: there MUST be zero or exactly one instance of this | |||
field. | ||||
o sc- Server to Client. | o s-sid: | |||
o sr- Server to Remote Server (used by proxies) | * format: [Editor's Note: add reference to the I-D defining the | |||
format of Session ID>?] | ||||
o rs- Remote Server to Server (used by proxies) | * semantic: this contains the value of the Session IDentifier | |||
specified in ??? and associated to the served request by the | ||||
Surrogate. | ||||
Example: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs- | * occurrence: there MUST be zero or exactly one instance of this | |||
username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status | field. | |||
time-taken | ||||
2011-11-23 15:22:01 x.x.x.x GET /file 80 y.y.y.y Mozilla/ | o s-cached: [Editor's Note: W3C uses "cached" . is "s-cached" | |||
5.0+(Windows;+U;+Windows+NT+6.1;+en-US;+rv:1.9.1.6)+Gecko/ | better?] | |||
20091201+Firefox/3.5.6+GTB6 200 0 0 2137 | ||||
A.3. National Center for Supercomputing Applications (NCSA) Common Log | * format: <string> | |||
Format | * semantic: this characterises whether the Surrogate could serve | |||
the request using content already stored on its local cache. | ||||
The allowed values are "0" (for miss) and "1" for hit). "1" | ||||
MUST be used when the Surrogate could serve the request using | ||||
exclusively content already stored on its local cache. "0" | ||||
MUST be used otherwise (including cases where the Surrogate | ||||
served the request using some, but not all, content already | ||||
stored on its local cache). Note that a "0" only means a cache | ||||
miss in the Surrogate and does not provide any information on | ||||
whether the content was already stored, or not, in another | ||||
device of the dCDN i.e. whether this was a "dCDN hit" or "dCDN | ||||
miss". | ||||
This format for Access Logs offers the following fields: | * occurrence: there MUST be zero or exactly one instance of this | |||
field. | ||||
o host rfc931 date:time "request" statuscode bytes | o s-uri-signing: | |||
o x.x.x.x userfoo [10/Jan/2010:21:15:05 +0500] "GET /index.html | * format: <string> | |||
HTTP/1.0" 200 1043 | ||||
A.4. NCSA Combined Log Format | * semantic: this characterises the uri signing validation | |||
performed by the Surrogate on the request. The allowed values | ||||
are: | ||||
The NCSA Combined log format is an extension of the NCSA Common log | * | |||
format with three (optional) additional fields: the referral field, | ||||
the user_agent field, and the cookie field. | ||||
o host rfc931 username date:time request statuscode bytes referrer | + "0" : no uri signature validation performed | |||
user_agent cookie | ||||
o Example: x.x.x.x - userfoo [21/Jan/2012:12:13:56 +0500] "GET | + "1" : uri signature validation performed and validated | |||
/index.html HTTP/1.0" 200 1043 "http://www.example.com/" "Mozilla/ | ||||
4.05 [en] (WinNT; I)" "USERID=CustomerA;IMPID=01234" | ||||
A.5. NCSA Separate Log Format | + "2" : uri signature validation performed and rejected | |||
The NCSA Separate log format refers to a log format in which the | * occurrence: there MUST be zero or exactly one instance of this | |||
information gathered is separated into three separate files. This | field. | |||
way, every entry in the Access Log (in the NCSA Common log format) is | ||||
complemented with an entry in a Referral log and another one in an | ||||
Agent log. These three records can be correlated easily thanks to | ||||
the date:time value. The format of the Referral log is as follows: | ||||
o date:time referrer | The "Fields" directive corresponding to a HTTP Request Logging Record | |||
MUST list all the fields whose occurrence is specified above as | ||||
"There MUST be one and only one instance of this field". These | ||||
fields MUST be present in every HTTP Request Logging Record. | ||||
o Example: [21/Jan/2012:12:13:56 +0500] | The "Fields" directive corresponding to a HTTP Request Logging Record | |||
"http://www.example.com/index.html" | MAY list all the fields whose occurrence is specified above as "there | |||
MUST be zero or exactly one instance of this field" or "there MUST be | ||||
zero, one or any number of instance of this field". The set of such | ||||
fields actually listed in the "Fields" directive is selected by the | ||||
implementation generating the CDNI Logging File based on agreements | ||||
between the interconnected CDNs established through mechanisms | ||||
outside the scope of this specification (e.g. contractual | ||||
agreements) . When such a field is not listed in the "Fields" | ||||
directive, it MUST NOT be included in the Logging Record. When such | ||||
a field is listed in the "Fields" directive, it MUST be included in | ||||
the Logging Record; in that case, if the value for the field is not | ||||
available, this MUST be conveyed via a dash character ("-"). | ||||
The format of the Agent log is as follows: | The fields listed in the "Fields" directive can be listed in the | |||
order in which they are listed in Section 3.2.1 or in any other | ||||
order. | ||||
o date:time agent | [Editor's Note: discuss private fields ] | |||
o [21/Jan/2012:12:13:56 +0500] "Microsoft Internet Explorer - 5.0" | 3.2.2. CDNI Logging File Example | |||
A.6. Squid 2.0 Native Log Format for Access Logs | #Version: 1.0 | |||
Squid [squid] is a popular piece of open-source software for | #UUID: urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6??? | |||
transforming a Linux host into a caching proxy. Variations of Squid | ||||
log format are supported by some CDNs. | ||||
Squid common access log format is as follow: time elapsed remotehost | #Origin: cdni-logging-entity.dcdn.example.com | |||
code/status bytes method URL rfc931 peerstatus/peerhost type. | ||||
Squid also supports a more detailed native access log format: | #Record-Type: cdni_http_request_v1 | |||
Timestamp Elapsed Client Action/Code Size Method URI Ident Hierarchy/ | ||||
From Content | ||||
According to Squid 2.0 documentation [squid], these fields are | #Fields: date time time-taken c-ip cs-method ucdn-centric-uri | |||
defined as follows: | protocol sc-status sc-total-bytes cs(User-Agent) cs(Referer) s-cached | |||
+-----------+-------------------------------------------------------+ | 2013-05-17 00:38:06.825 88.958 10.5.7.1 GET http://cdni- | |||
| Element | Definition | | ucdn.dcdn.example.com/video/movie100.mp4 HTTP/1.1 200 672989 Mozilla/ | |||
+-----------+-------------------------------------------------------+ | 5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, | |||
| time | Unix timestamp as UTC seconds with a millisecond | | like Gecko) Chrome/5.0.375.127 Safari /533.4 host1.example.com 1 | |||
| | resolution. | | ||||
| duration | The elapsed time in milliseconds the transaction | | ||||
| | busied the cache. | | ||||
| client | The client IP address. | | ||||
| address | | | ||||
| bytes | The size is the amount of data delivered to the | | ||||
| | client, including headers. | | ||||
| request | The request method to obtain an object. | | ||||
| method | | | ||||
| URL | The requested URL. | | ||||
| rfc931 | may contain the ident lookups for the requesting | | ||||
| | client (turned off by default) | | ||||
| hierarchy | The hierarchy information provides information on how | | ||||
| code | the request was handled (forwarding it to another | | ||||
| | cache, or requesting the content to the Origin | | ||||
| | Server). | | ||||
| type | The content type of the object as seen in the HTTP | | ||||
| | reply header. | | ||||
+-----------+-------------------------------------------------------+ | ||||
Table 6: Information elements in Squid format | 2013-05-17 00:39:09.145 169.790 10.5.10.5 GET http://cdni- | |||
ucdn.dcdn.example.com/video/movie118.mp4 HTTP/1.1200 1579920 Mozilla/ | ||||
5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, | ||||
like Gecko) Chrome/5.0.375.127 Safari /533.4 host1.example.com 1 | ||||
Squid also uses a "store log", which covers the objects currently | 2013-05-17 00:42:53.437 2.879 10.5.10.5 GET http://cdni- | |||
kept on disk or removed ones, for debugging purposes typically. | ucdn.dcdn.example.com/video/picture11.mp4 HTTP/1.0 200 17724 Mozilla/ | |||
5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, | ||||
like Gecko) Chrome/5.0.375.127 Safari /533.4 host5.example.com 0 | ||||
Appendix B. Requirements | #Integrity-Hash: 9e107d9d372bb6826bd81d3542a419d6 [Editor's Note: | |||
include the correct MD5-hash value for the actual example] | ||||
B.1. Additional Requirements | 3.3. Fields and Directives Formats | |||
Section 7 of [I-D.ietf-cdni-requirements], already specifies a set of | [Editor's Note: still needs work to minimise the number of types | |||
requirements for Logging (LOG-1 to LOG-16). Some security | defined across this section and specific types defined inside the | |||
requirements also affect Logging (e.g., SEC-4). | field definitions themselves] | |||
This section is a placeholder for requirements identified in the work | o <digit> = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | | |||
on logging, before they are proposed to the requirements draft | "9" | |||
authors. | ||||
Logging data is sensitive as it provides the raw material for | o <integer> = 1*<digit> | |||
producing bills etc. Therefore, the protocol delivering the Logging | ||||
data must be reliable to avoid information loss. In addition, the | ||||
protocol must scale to support the transport of large amounts of | ||||
Logging data. | ||||
CDNs need to trust Logging information, thus, they want to know: | o <address> = <integer> [ "." *<integer> ] [ ":" <integer> ] | |||
o who issued the Logging (authentication), and | o <host> = as specified in [RFC3986]. | |||
o if the Logging has been modified by a third party (integrity). | o <date> = 4<digit> "-" 2<digit> "-" 2<digit> | |||
Logging also contains confidential data, and therefore, it should be | * Dates are recorded in the format YYYY-MM-DD where YYYY, MM and | |||
protected from eavesdropping. | DD stand for the numeric year, month and day respectively. All | |||
dates are specified in Universal Time Coordinated (UTC). | ||||
All these needs translate into security requirements on both the | o <time> = 2<digit> ":" 2<digit> ":" 2<digit> ["." *<digit>] | |||
Logging data format and on the Logging protocol. | ||||
Finally, this protocol must comply with the requirements identified | * Times are recorded in the form HH:MM:SS or HH:MM:SS.S where HH | |||
in [I-D.ietf-cdni-requirements]. | is the hour in 24 hour format, MM is minutes and SS is seconds. | |||
All times are specified in Universal Time Coordinated (UTC). | ||||
[Ed. note: cf. requirements draft: "SEC-4 [MED] The CDNI solution | o <uri> = <string> containing a URI as specified in [RFC3986]. | |||
should be able to ensure that the Downstream CDN cannot spoof a | ||||
transaction log attempting to appear as if it corresponds to a | ||||
request redirected by a given Upstream CDN when that request has not | ||||
been redirected by this Upstream CDN. This ensures non-repudiation | ||||
by the Upstream CDN of transaction logs generated by the Downstream | ||||
CDN for deliveries performed by the Downstream CDN on behalf of the | ||||
Upstream CDN."] | ||||
B.2. Compliancy with Requirements draft | o <fixed> = Fixed Format Float = 1*<digit> [. *<digit>] | |||
o <HTTP-header> = <string> containing a HTTP header field name (e.g. | ||||
"User-Agent", "Referer") as specified in [RFC2616]. | ||||
4. CDNI Logging File Exchange Protocol | ||||
This document specifies a protocol for the exchange of CDNI Logging | ||||
Files as specified in Section 3. | ||||
This protocol comprises: | ||||
o a CDNI Logging feed, allowing the dCDN to notify the uCDN about | ||||
the CDNI Logging files that can be retrieved by that uCDN from the | ||||
dCDN, as well as all the information necessary for retrieving each | ||||
of these CDNI Logging File. The CDNI Logging feed is specified in | ||||
Section 4.1. | ||||
o a CDNI Logging File pull mechanism, allowing the uCDN to obtain | ||||
from the dCDN a given CDNI Logging File at the uCDN convenience. | ||||
The CDNI Logging File pull mechanisms is specified in Section 4.2. | ||||
An implementation of the CDNI Logging interface as per the present | ||||
document generating CDNI Logging file (i.e. on the dCDN side) MUST | ||||
support the server side of the CDNI Logging feed and the server side | ||||
of the CDNI Logging pull mechanism. | ||||
An implementation of the CDNI Logging interface as per the present | ||||
document consuming CDNI Logging file (i.e. on the uCDN side) MUST | ||||
support the client side of the CDNI Logging feed and the client side | ||||
of the CDNI Logging pull mechanism. | ||||
[Editor's note: verify that the client side and server side are well | ||||
defined in the respective sections] | ||||
We note that implementations of the CDNI Logging interface MAY also | ||||
support other mechanisms to exchange CDNI Logging Files, for example | ||||
in view of exchanging logging information with minimum time-lag (e.g. | ||||
sub-minute or sub-second) between when the event occurred in the dCDN | ||||
and when the corresponding Logging Record is made available to the | ||||
uCDN (e.g. for log-consuming applications requiring extremely fresh | ||||
logging information such as near-real-time content delivery | ||||
monitoring). Such mechanism might be defined in future version of | ||||
the present document. | ||||
4.1. CDNI Logging Feed | ||||
[Editor's Note: text to be added. Feed is based on ATOM and contains | ||||
a UUID + URI for each CDNI Logging File in "window" - if appropriate | ||||
the text should refer to the side generating the CDNI Logging Feed | ||||
"as server-side", and the side consuming the Feed as the client- | ||||
side]. | ||||
4.2. CDNI Logging File Pull | ||||
A client-side implementation of the CDNI Logging interface MAY pull | ||||
at its convenience any CDNI Logging File that is advertised by the | ||||
server-side in the CDNI Logging Feed. To do so, the client-side: | ||||
o MUST use HTTP v1.1 | ||||
o SHOULD use TLS (i.e. use what is loosely referred to as "HTTPS") | ||||
o MUST use the URI associated to the CDNI Logging File in the CDNI | ||||
Logging Feed | ||||
o SHOULD indicate the compression schemes it supports | ||||
Note that a client-side implementation of the CDNI Logging interface | ||||
MAY pull a CDNI Logging File that it has already pulled, as long as | ||||
the file is still advertised by the server-side in the CDNI Logging | ||||
Feed. | ||||
The server-side implementation MUST respond to any valid pull request | ||||
by a client-side implementation for a CDNI Logging File advertised by | ||||
the server-side in the CDNI Logging Feed. The server-side | ||||
implementation: | ||||
o MUST handle the client-side request as per HTTP v1.1 | ||||
o MUST include the CDNI Logging File identified by the request URI | ||||
inside the body of the HTTP response | ||||
o MUST support the gzip and deflate compression schemes | ||||
o MAY support other compression schemes | ||||
o when the client-side request indicates client-supported | ||||
compression schemes, SHOULD use a compression scheme that it | ||||
supports and is supported by the client-side | ||||
[Editor's Note: discuss Non-Repudiation : it is a nice to have and | ||||
how it could be supported, via a different digest than the one for | ||||
integrity] | ||||
5. Open Issues | ||||
o The proposed format for Date and Time is based on W3C and is only | ||||
in UTC. Is this all OK? RFC 5322 (Section 3.3) format could be | ||||
used or ISO 8601 formatted date and time in UTC (same format as | ||||
proposed in [draft-caulfield-cdni-metadata-core-00]). Also see | ||||
RFC5424 Section 6.2.3. We currently use same field names as W3C | ||||
since we have same definition. | ||||
o (comment from Kevin) how are errors handled ? If the client gets | ||||
handed a bunch of 403s and 404s, but still gets the content | ||||
eventually, without triggering an event, are those still logged? | ||||
For Bytes-Sent, if there were aborted requests, do those get | ||||
counted as well? Not all client behavior can be correlated with | ||||
the simplified log | ||||
o Do we need to specify Logs for Request Routing performed by dCDN? | ||||
Observation: Probably can be generalized to the requirement for | ||||
"event" logging (e.g. dCDN request Router not able to redirect, | ||||
dCDN cannot acquire metadata, dCDN cannot aquire content, "dCDN | ||||
Busy Tone" ) Recommendation: Try first specify what events and | ||||
what information needs to be exchanged. Depending on progress | ||||
include in initial logging spec or not i.e. handle as a [MED] | ||||
requirement. | ||||
o Privacy: do we need some explicit support of IP address masking by | ||||
dCDN to uCDN, or is it OK to assume that uCDN is to keep this info | ||||
confidential (like dCDN is assumed to do already)? | ||||
o definition of field prefixes: add "r" is uCDN. This one is less | ||||
clear to me. I need to see how you propose to use "r" below, | ||||
before I can agree. (Just for my own notes, I thought "r" could | ||||
be used if the dCDN Surrogate was going to Log something related | ||||
to acquisition of content by the dCDN Surrogate from some content | ||||
source. Also, in a delivery log generated by a dCDN Surrogate , | ||||
how can it know about acquisition from uCDN that can be done by | ||||
other devices than the dCDN Surrogate). "ucdn-centric-uri": ROB> | ||||
going back to the definitions of s/c/r suggested above, for a CDNI | ||||
logfile field would then just be "sr-uri". So we don't need to | ||||
invent a new prefix for CDNI, we can use the basic w3c naming? | ||||
FRANCOIS: I am OK to use "sr-uri" as long as we feel confident | ||||
that we will never need Surrogate to log information about how it | ||||
acquires from within the dCDN (ie regular use of "r" prefix). Are | ||||
we confident? | ||||
o Do we need Record-Type as File Directive?: ROB> Is this needed - | ||||
would a record type per file do the job? ... if we don't allow | ||||
mixed record types, we can include the record type in the ATOM | ||||
feed (to allow the reader to decide whether there might be records | ||||
it's interested in without getting the logfile). I can't think of | ||||
a reason to mix, (for example) http/rtmp records, or delivery/req- | ||||
routing. Different things are likely to be generating those | ||||
records anyway. A version change can always be done by starting a | ||||
new file. <Francois> Here are a couple potential use cases for | ||||
mixing record types in a single file: * we later define | ||||
"cdni_has_delivery_v1" record types for HTTP Adaptive BitRate | ||||
sessions. Then a dCDN Surrogate will be generating a continuous | ||||
mixture of "cdni_http_request_v1" records for PDL requests and | ||||
"cdni_has_request_v1" records for HAS sessions. Why should we be | ||||
forced to break those? * we later define some record types for | ||||
events taking place on Surrogates , which can happen any time in | ||||
the middle of sessions. Why shoudl we be forced to break those | ||||
into separate files. It seems wise to keep the flexibility in the | ||||
File structure to allow the mix in the future. And the overhead | ||||
is very small since it is encoded in a Directive. | ||||
o Integrity-Hash:ROB> draft-snell-atompub-link-extensions adds a | ||||
hash of the resource to the ATOM feed (not sure about the status | ||||
of that doc, looks like it's stalled a bit). But if we include | ||||
that in the ATOM feed, the value in the feed would need to include | ||||
this Integrity-Hash in the log file itself, which might mean re- | ||||
calculating the hash (especially if the feed is not generated in | ||||
the same place as the logfile). So we probably only want one of | ||||
the two? I think my preference would be to keep it in the feed, | ||||
saves any complications about what to hash (just running "md5sum" | ||||
on a downloaded logfile would work, rather than needing to remove | ||||
the last line). The draft-snell also allows other hashes, "sha1" | ||||
and so on - for cdni interoperability, we could limit it to md5 or | ||||
stick with draft-snell's base set. <Francois> Very good point. I | ||||
agree we should probably want one of the two in a typical | ||||
deployment. Leveraging draft-snell-atompub-link-extensions is | ||||
attractive because it leverages generic ATOM features and | ||||
expertise. It has the potential drawback of introducing a | ||||
dependency on a document that may be published later (or | ||||
potentially never since it is not even a WG doc). Defining our | ||||
own hash in the file is attractive because we can be done right | ||||
away, and there could be simple short term implementation that | ||||
start using the CDNI Logging File without relying on the ATOM | ||||
Feed. At the same time we don't want to end up with two redundant | ||||
hashes eventually. How about an approach where : * we define a | ||||
simple MD5 has only, and make it optional * when there is no other | ||||
mechanism to get the hash, it can be included in the file * when | ||||
there are other mechanism (e.g. draft-snell-atompub-link- | ||||
extensions), it is not included in the file. | ||||
o Compression: <Ben>When we say the server MUST support gzip & | ||||
deflate we probably need to think through whether we mean content- | ||||
encoding, transfer-encoding or both. The semantics get a little | ||||
confusing so we probably just need to think them through to ensure | ||||
we allow a server to store compressed logs as transmit them | ||||
compressed. | ||||
6. IANA Considerations | ||||
TBD | ||||
7. Security Considerations | ||||
7.1. Authentication, Confidentiality, Integrity Protection | ||||
The use of TLS for transport of the CDNI Logging feed mechanism | ||||
(Section 4.1) and CDNI Logging File pull mechanism (Section 4.2) | ||||
allows: | ||||
o the dCDN and uCDN to authenticate each other (to ensure they are | ||||
transmitting/receiving CDNI Logging File from an authenticated | ||||
CDN) | ||||
o the CDNI Logging information to be transmitted with | ||||
confidentiality | ||||
o the integrity of the CDNI Logging information to be protected | ||||
during the exchange. | ||||
The Integrity-Hash directive inside the CDNI Logging File provides | ||||
additional integrity protection, this time targeting potential | ||||
corruption of the CDNI logging information during the CDNI Logging | ||||
File generation. This mechanism does not allow restoration of the | ||||
corrupted CDNI Logging information, but it allows detection of such | ||||
corruption and therefore triggering of appropraite correcting actions | ||||
(e.g. discard of corrupted information, attempt to re-obtain the | ||||
CDNI Logging information). | ||||
7.2. Non Repudiation | ||||
The Non-Repudiation-Hash directive in the CDNI Logging File allows | ||||
support of non-repudiation of the CDNI Logging File by the dCDN. The | ||||
optional Non-Repudiation-Hash can be used on the CDNI Logging | ||||
interface where needed. | ||||
7.3. Privacy | ||||
CDNs have the opportunity to collect detailed information about the | ||||
downloads performed by End-Users. The provision of this information | ||||
to another CDN introduces End-Users privacy protection concerns. | ||||
[Editor's Note: see list of open questions] | ||||
8. Acknowledgments | ||||
This document borrows from the W3C Extended Log Format [ELF]. | ||||
The authors would like to thank Sebastien Cubaud, Pawel Grochocki, | ||||
Christian Jacquenet, Yannick Le Louedec, Anne Marrec and Emile | ||||
Stephan for their contributions on early versions of this document. | ||||
The authors would like also to thank Rob Murray, Fabio Costa, Sara | ||||
Oueslati, Yvan Massot, Renaud Edel, and Joel Favier for their input | ||||
and comments. | ||||
Finally, they thank the contributors of the EU FP7 OCEAN project for | ||||
valuable inputs. | ||||
9. References | ||||
9.1. Normative References | ||||
[I-D.ietf-cdni-metadata] | ||||
Niven-Jenkins, B., Murray, R., Watson, G., Caulfield, M., | ||||
Leung, K., and K. Ma, "CDN Interconnect Metadata", draft- | ||||
ietf-cdni-metadata-01 (work in progress), February 2013. | ||||
[RFC1321] Rivest, R., "The MD5 Message-Digest Algorithm", RFC 1321, | ||||
April 1992. | ||||
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate | ||||
Requirement Levels", BCP 14, RFC 2119, March 1997. | ||||
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., | ||||
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext | ||||
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. | ||||
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform | ||||
Resource Identifier (URI): Generic Syntax", STD 66, RFC | ||||
3986, January 2005. | ||||
[RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally | ||||
Unique IDentifier (UUID) URN Namespace", RFC 4122, July | ||||
2005. | ||||
[RFC5424] Gerhards, R., "The Syslog Protocol", RFC 5424, March 2009. | ||||
9.2. Informative References | ||||
[CHAR_SET] | ||||
, "IANA Character Sets registry", , <http://www.iana.org/ | ||||
assignments/character-sets/character-sets.xml>. | ||||
[ELF] Phillip M. Hallam-Baker, and Brian Behlendorf, "Extended | ||||
Log File Format, W3C (work in progress), WD- | ||||
logfile-960323", , <http://www.w3.org/TR/WD-logfile.html>. | ||||
[I-D.brandenburg-cdni-has] | ||||
Brandenburg, R., Deventer, O., Faucheur, F., and K. Leung, | ||||
"Models for adaptive-streaming-aware CDN Interconnection", | ||||
draft-brandenburg-cdni-has-05 (work in progress), April | ||||
2013. | ||||
[I-D.ietf-cdni-framework] | ||||
Peterson, L. and B. Davie, "Framework for CDN | ||||
Interconnection", draft-ietf-cdni-framework-03 (work in | ||||
progress), February 2013. | ||||
[I-D.ietf-cdni-requirements] | ||||
Leung, K. and Y. Lee, "Content Distribution Network | ||||
Interconnection (CDNI) Requirements", draft-ietf-cdni- | ||||
requirements-06 (work in progress), April 2013. | ||||
[RFC6707] Niven-Jenkins, B., Le Faucheur, F., and N. Bitar, "Content | ||||
Distribution Network Interconnection (CDNI) Problem | ||||
Statement", RFC 6707, September 2012. | ||||
[RFC6770] Bertrand, G., Stephan, E., Burbridge, T., Eardley, P., Ma, | ||||
K., and G. Watson, "Use Cases for Content Delivery Network | ||||
Interconnection", RFC 6770, November 2012. | ||||
Appendix A. Requirements | ||||
A.1. Compliance with cdni-requirements | ||||
This section checks that all the identified requirements in the | This section checks that all the identified requirements in the | |||
Requirements draft are fulfilled by this document. | Section 7 of [I-D.ietf-cdni-requirements] are fulfilled by this | |||
document. | ||||
[Ed. node: to be written later] | [Editor's node: to be written later] | |||
Appendix C. Analysis of candidate protocols for Logging Transport | A.2. Additional Requirements | |||
This section identies additional requirements that must also be met. | ||||
[Editor's node: How do we incorporate this info into the I-D: in | ||||
appendix? in main body? does it remain after publication or is | ||||
temporary?] | ||||
A.2.1. Timeliness | ||||
Some applications consuming CDNI Logging information, such as | ||||
accounting or trend analytics, only require logging information to be | ||||
available with a timeliness of the order of a day or the hour. This | ||||
document focuses on addressing this requirement. | ||||
Some applications consuming CDNI Logging information, such as real- | ||||
time analytics, require logging information to be available in real- | ||||
time (i.e. of the order of a second after the corresponding event). | ||||
This document leaves this requirement out of scope. | ||||
A.2.2. Reliability | ||||
CDNI logging information must be transmitted reliably. The transport | ||||
protocol should contain an anti-replay mechanism. | ||||
A.2.3. Security | ||||
CDNI logging information exchange must allow authentication, | ||||
integrity protection, and confidentiality protection. Also, a non- | ||||
repudiation mechanism is mandatory, the transport protocol should | ||||
support it. | ||||
A.2.4. Scalability | ||||
CDNI logging information exchange must support large scale | ||||
information exchange, particularly so in the presence of HTTP | ||||
Adaptive Streaming. | ||||
For example, if we consider a client pulling HTTP Progressive | ||||
Download content with an average duration of 10 minutes, this | ||||
represents 1/600 CDNI delivery Logging Records per second. If we | ||||
assume the dCDN is simultaneously serving 100,000 such clients on | ||||
behalf of the uCDN, the dCDN will be generating 167 Logging Records | ||||
per second to be communicated to the uCDN over the CDNI Logging | ||||
interface. Or equivalently, if we assume an average delivery rate of | ||||
2Mb/s, the dCDN generates 0.83 CDNI Logging Records per second for | ||||
every Gb/s of streaming on behalf of the uCDN. | ||||
For example, if we consider a client pulling HAS content and | ||||
receiving a video chunk every 2 seconds, a separate audio chunck | ||||
every 2 seconds and a refreshed manifest every 10 seconds, this | ||||
represents 1.1 delivery Logging Record per second. If we assume the | ||||
dCDN is simultaneously serving 100,000 such clients on behalf of the | ||||
uCDN, the dCDN will be generating 110,000 Logging Records per second | ||||
to be communicated to the uCDN over the CDNI Logging interface. Or | ||||
equivalently, if we assume an average delivery rate of 2Mb/s, the | ||||
dCDN generates 550 CDNI Logging Records per second for every Gb/s of | ||||
streaming on behalf of the uCDN. | ||||
A.2.5. Consistency between CDNI Logging and CDN Logging | ||||
There are benefits in using a CDNI logging format as close as | ||||
possible to intra-CDN logging format commonly used in CDNs today in | ||||
order to minimize systematic translation at CDN/CDNI boundary. | ||||
A.2.6. Dispatching/Filtering | ||||
When a CDN is acting as a dCDN for multiple uCDNs, the dCDN needs to | ||||
dispatch each CDNI Logging Record to the uCDN that redirected the | ||||
corresponding request. The CDNI Logging format need to allow, and | ||||
possibly facilitate, such a dispatching. | ||||
Appendix B. Analysis of candidate protocols for Logging Transport | ||||
This section will be expanded later with an analysis of alternative | This section will be expanded later with an analysis of alternative | |||
candidate protocols for transport of CDNI Logging in non-real-time as | candidate protocols for transport of CDNI Logging in non-real-time as | |||
well as real-time. | well as real-time. | |||
C.1. Syslog | B.1. Syslog | |||
[Ed. node: to be written later] | [Ed. node: to be written later] | |||
C.2. XMPP | B.2. XMPP | |||
[Ed. node: to be written later] | [Ed. node: to be written later] | |||
C.3. SNMP | B.3. SNMP | |||
Authors' Addresses | Authors' Addresses | |||
Gilles Bertrand (editor) | Gilles Bertrand (editor) | |||
France Telecom - Orange | France Telecom - Orange | |||
38-40 rue du General Leclerc | 38-40 rue du General Leclerc | |||
Issy les Moulineaux, 92130 | Issy les Moulineaux 92130 | |||
FR | FR | |||
Phone: +33 1 45 29 89 46 | Phone: +33 1 45 29 89 46 | |||
Email: gilles.bertrand@orange.com | Email: gilles.bertrand@orange.com | |||
Iuniana Oprescu (editor) | Iuniana Oprescu (editor) | |||
France Telecom - Orange | France Telecom - Orange | |||
38-40 rue du General Leclerc | 38-40 rue du General Leclerc | |||
Issy les Moulineaux, 92130 | Issy les Moulineaux 92130 | |||
FR | FR | |||
Phone: +33 6 89 06 92 72 | Phone: +33 6 89 06 92 72 | |||
Email: iuniana.oprescu@orange.com | Email: iuniana.oprescu@orange.com | |||
Francois Le Faucheur (editor) | ||||
Cisco Systems | ||||
E.Space Park - Batiment D | ||||
6254 Allee des Ormes - BP 1200 | ||||
Mougins cedex 06254 | ||||
FR | ||||
Stephan Emile | Phone: +33 4 97 23 26 19 | |||
France Telecom - Orange | Email: flefauch@cisco.com | |||
2 avenue Pierre Marzin | ||||
Lannion F-22307 | ||||
France | ||||
Email: emile.stephan@orange.com | ||||
Roy Peterkofsky | Roy Peterkofsky | |||
Skytide, Inc. | Skytide, Inc. | |||
One Kaiser Plaza, Suite 785 | One Kaiser Plaza, Suite 785 | |||
Oakland CA 94612 | Oakland CA 94612 | |||
USA | USA | |||
Phone: +01 510 250 4284 | Phone: +01 510 250 4284 | |||
Email: roy@skytide.com | Email: roy@skytide.com | |||
Francois Le Faucheur (editor) | ||||
Cisco Systems | ||||
Greenside, 400 Avenue de Roumanille | ||||
Sophia Antipolis 06410 | ||||
FR | ||||
Phone: +33 4 97 23 26 19 | ||||
Email: flefauch@cisco.com | ||||
Pawel Grochocki | ||||
Orange Polska | ||||
ul. Obrzezna 7 | ||||
Warsaw 02-691 | ||||
Poland | ||||
Email: pawel.grochocki@orange.com | ||||
End of changes. 201 change blocks. | ||||
950 lines changed or deleted | 885 lines changed or added | |||
This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/ |