[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]

Versions: (draft-pentikousis-icn-scenarios) 00 01 02 03 04 05 RFC 7945

ICNRG                                                K. Pentikousis, Ed.
Internet-Draft                                                      EICT
Intended Status: Informational                                 B. Ohlman
Expires: April 24, 2014                                         Ericsson
                                                               E. Davies
                                                  Trinity College Dublin
                                                               S. Spirou
                                                        Intracom Telecom
                                                               G. Boggia
                                                     Politecnico di Bari
                                                            P. Mahadevan
                                                                    PARC
                                                        October 21, 2013


         Information-centric Networking: Evaluation Methodology
               draft-irtf-icnrg-evaluation-methodology-00


Abstract

   This document surveys the evaluation tools currently available to
   researchers in the information-centric networking (ICN) area and
   provides suggestions regarding methodology and metrics.  Finally,
   this document sheds some light on the impact of ICN on network
   security.


Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html



Pentikousis, et al.      Expires April 24, 2014                 [Page 1]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


Copyright and License Notice

   Copyright (c) 2013 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  2
   2.  Evaluation Methodology . . . . . . . . . . . . . . . . . . . .  4
     2.1.  ICN Simulators and Testbeds  . . . . . . . . . . . . . . .  4
       2.1.1.  CCN and NDN  . . . . . . . . . . . . . . . . . . . . .  4
       2.1.2.  Publish/Subscribe Internet Architecture  . . . . . . .  6
       2.1.3.  NetInf . . . . . . . . . . . . . . . . . . . . . . . .  6
       2.1.4.  COMET  . . . . . . . . . . . . . . . . . . . . . . . .  7
       2.1.5.  Large-scale Testing  . . . . . . . . . . . . . . . . .  7
     2.2.  Topology Selection . . . . . . . . . . . . . . . . . . . .  8
     2.3.  Traffic Load . . . . . . . . . . . . . . . . . . . . . . .  9
     2.4.  Choosing Relevant Metrics  . . . . . . . . . . . . . . . . 11
       2.4.1.  Traffic Metrics  . . . . . . . . . . . . . . . . . . . 13
       2.4.2.  System Metrics . . . . . . . . . . . . . . . . . . . . 15
     2.5.  Resource Equivalence and Tradeoffs . . . . . . . . . . . . 16
   3.  ICN Security Aspects . . . . . . . . . . . . . . . . . . . . . 16
     3.1. Authentication  . . . . . . . . . . . . . . . . . . . . . . 17
     3.2. Authorization, Access Control and Statistics  . . . . . . . 19
     3.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . 19
     3.4. Changes to the Network Security Threat Model  . . . . . . . 20
   4.  Security Considerations  . . . . . . . . . . . . . . . . . . . 21
   5.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 21
   6.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 21
   7.  Informative References . . . . . . . . . . . . . . . . . . . . 21
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26



1.  Introduction

   Information-centric networking (ICN) marks a fundamental shift in
   communications and networking.  As discussed in [draft-irtf-icnrg-
   scenarios], the development phase that ICN is going through, and the
   plethora of approaches to tackle the hardest problems, make this a



Pentikousis, et al.      Expires April 24, 2014                 [Page 2]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   very active and growing research area but, on the downside, it also
   makes it more difficult to compare different proposals on an equal
   footing.  Different ICN approaches have been evaluated in the peer-
   reviewed literature using a mixture of theoretical analysis,
   simulation and emulation techniques, and empirical (testbed)
   measurements.  These are all popular methods for evaluating network
   protocols, architectures, and services in the networking community.
   Typically, researchers follow a specific methodology based on the
   goal of their experiment, e.g., whether they want to evaluate
   scalability, quantify resource utilization, analyze economic
   incentives, and so on, as we have discussed earlier.  In addition,
   though, we observe that ease and convenience of setting up and
   running experiments can sometimes be a factor in published
   evaluations.

   It is worth pointing out that for well-established protocols, such as
   TCP, performance evaluation using actual network deployments has the
   benefit of realistic workloads and reflects the environment where the
   service or protocol will be deployed.  However, results obtained in
   this environment are often difficult to replicate independently.
   Beyond this, the difficulty of deploying future Internet
   architectures and then engaging sufficient users to make such
   evaluation realistic is often prohibitive.

   Moreover, for ICN in particular, it is not yet clear what qualifies
   as a "realistic workload".  As such, trace-based analysis of ICN is
   in its infancy, and more work is needed towards defining
   characteristic workloads for ICN evaluation studies.  Accordingly,
   the experimental process itself as well as the evaluation methodology
   are being actively researched for ICN architectures.   Numerous
   factors affect the experimental results, including the topology
   selected, the background traffic that an application is being
   subjected to, network conditions such as available link capacities,
   link delays, and loss-rate characteristics throughout the selected
   topology; failure and disruption patterns; node mobility; as well as
   other aspects such as the diversity of devices used, and so on, as we
   explain in the remainder of this section.

   Apart from the technical evaluation of the functionality of an ICN
   architecture, its future success will be largely driven by its
   deployability and economic viability.  Thus any evaluation will also
   have to include an assessment of its incremental deployability in the
   existing network environment together with a view of how the
   technical functions will incentivize deployers to invest in the
   capabilities that allow the architecture to spread across the
   network.

   This document incorporates input from ICNRG participants and their



Pentikousis, et al.      Expires April 24, 2014                 [Page 3]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   corresponding text contributions, has been reviewed by several ICNRG
   active participants (see section 6), and represents the consensus of
   the research group.  That said, note that this document does not
   constitute an IETF standard; see also [RFC5743].

   The remainder of this document is organized as follows. Section 2
   presents various techniques and considerations for evaluating
   different ICN architectures.  Then, Section 3 discusses the impact of
   ICN on network security.


2.  Evaluation Methodology

   At this stage, we do not intend to develop a complete methodology or
   a benchmarking tool.  Instead, this document proposes key guidelines
   alongside suggested data sets and high-level approaches that we
   expect to be of interest to the ICN community as a whole.  Through
   this, researchers and practitioners alike would be able to compare
   and contrast different ICN designs against each other, as well as
   against the state of the art in host-centric solutions, and identify
   the respective strengths and weaknesses.


2.1.  ICN Simulators and Testbeds

   Since ICN is still an emerging area, the community is still in the
   process of developing effective evaluation environments, including
   simulators, emulators, and testbeds.  To date, none of the available
   evaluation methodologies can be seen as the one and only community
   reference evaluation tool.  Furthermore, no single environment
   supports all well-known ICN approaches.  Simulators and emulators
   should be able to capture, faithfully, all features and operations of
   the respective ICN architecture(s).  It is also essential that these
   tools and environments come with adequate logging facilities so that
   one can use them for in-depth analysis as well as debugging.
   Additional requirements include the ability to support mid- to large-
   scale experiments, the ability to quickly and correctly set various
   configurations and parameters, as well as to support the playback of
   traffic traces captured on a real testbed or network.  Obviously,
   this does not even begin to touch upon the need for strong validation
   of any evaluated implementations.

   The rest of this subsection summarizes the ICN simulators and
   testbeds currently available to the community.


2.1.1.  CCN and NDN




Pentikousis, et al.      Expires April 24, 2014                 [Page 4]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   The CCN project has open-sourced a software reference implementation
   of the architecture and protocol called CCNx (www.ccnx.org). CCNx is
   available for deployment on various operating systems and includes C
   and Java libraries that can be used to build CCN applications. CCN-
   lite (www.ccn-lite.net) is a lightweight implementation of the CCN
   protocol, supports most of the key features of CCNx, and is
   interoperable with CCNx. The core CCNx logic has been implemented in
   about 1000 lines of code and is ideal for classroom work and course
   projects as well as for quickly experimenting with CCNx extensions.

   ndnSIM [ndnSIM] is a module that can be plugged into the ns-3
   simulator and supports the core features of CCN.  One can use ndnSIM
   to experiment with various CCN applications and services as well as
   components developed for CCN such as routing protocols, caching and
   forwarding strategies.  The code for ns-3 and ndnSIM is openly
   available to the community and can be used as the basis for
   implementing ICN protocols or applications.  For more details see
   http://www.nsnam.org and http://www.ndnsim.net.

   ccnSim [ccnSim] is another CCN-specific simulator that was specially
   designed to handle forwarding of a large number of CCN-chunks.
   ccnSim is written in C++ for the OMNeT++ simulation framework
   (www.omnetpp.org).  Interested readers could consider also the
   Content Centric Networking Packet Level Simulator [CCNPL].  Finally,
   CCN-Joker [CCNj] is an application-layer platform that can be used to
   build a CCN overlay.  CCN-Joker emulates in user-space all basic
   aspects of a CCN node (e.g., handling of Interest and Data packets,
   cache sizing, replacement policies), including both flow and
   congestion control.  The code is open source and is suitable for both
   emulation-based analyses and real experiments.

   An example of a testbed that supports CCN is the Open Network Lab
   (see https://onl.wustl.edu/).  The ONL testbed currently comprises 18
   extensible gigabit routers and over a 100 computers representing
   clients and is freely available to the public for running CCN
   experiments.  Nodes in ONL are preloaded with CCNx software.  ONL
   provides a graphical user interface for easy configuration and
   testbed set up as per the experiment requirements, and also serves as
   a control mechanism, allowing access to various control variables and
   traffic counters.  It is also possible to run and evaluate CCN over
   popular testbeds such as PlanetLab (www.planet-lab.org), Emulab
   (www.emulab.net), and Deter (www.isi.deterlab.net) by directly
   running the CCNx open-source code on PlanetLab and Deter nodes,
   respectively.

   NEPI, the Network Experimentation Programming Interface,
   (http://nepi.inria.fr) is a tool developed for controlling and
   managing large-scale network experiments. NEPI provides an experiment



Pentikousis, et al.      Expires April 24, 2014                 [Page 5]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   description language to design network experiments, describing
   topology, applications, and a controller to automatically deploy
   those experiments on target experimentation environments, such as
   PlanetLab. The controller is also capable of collecting result and
   log files during the experiment execution. NEPI also allows to
   specify node selection filters while designing the experiment,
   thereby supporting automatic discovery and provisioning of testbed
   nodes during experiment deployment, without the user having to hand-
   pick them. It is simple and efficient to use NEPI to evaluate CCNx on
   large-scale testbeds such as PlanetLab.


2.1.2.  Publish/Subscribe Internet Architecture

   The PSIRP project has open-sourced its Blackhawk publish-subscribe
   (Pub/Sub) implementation for FreeBSD; more details are available
   online at http://www.psirp.org/downloads.html.  Despite being limited
   to one operating system, the code base also provides a virtual image
   to allow its deployment on other environments through virtualization.
    The code distribution features a kernel module, a file system and
   scope daemon, as well as a set of tools, test applications and
   scripts.  This work was extended as part of the PURSUIT project,
   resulting in the development of the Blackadder prototype for Linux
   and FreeBSD.  It currently runs on a testbed across Europe, America
   (MIT) and Japan (NICT).  All sites are connected via OpenVPN, which
   exports a virtual Ethernet device to all machines in the testbed.  In
   total, 40 machines in a graph topology containing one Topology
   Manager and one Rendezvous node that handle all publish/subscribe and
   topology formation requests are interconnected [IEICE].

   Moreover, the ICN simulation environment [ICN-Sim] allows the
   simulation of new techniques for topology management following the
   Publish-Subscribe paradigm and the PSIRP approach.  The simulator is
   based on the OMNET++ simulator and the INET/MANET frameworks.  It is
   currently publicly available at
   http://sourceforge.net/projects/icnsim.  A design characteristic of
   this platform is the separation between the network and topology
   management policies.  An interface is used to provide this
   functionality and policies can be imported and applied in the network
   as topology manager applications running on top of this interface.


2.1.3.  NetInf

   The EU FP7 4WARD and SAIL projects have made a set of open-source
   implementations available; see http://www.netinf.org/open-source for
   more details.  Of note, two software packages are available.  The
   first one is a set of tools for NetInf implementing different aspects



Pentikousis, et al.      Expires April 24, 2014                 [Page 6]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   of the protocol (e.g., NetInf URI format, HTTP and UDP convergence
   layer) using different programming languages.  The Java
   implementation provides a local caching proxy and client.  The second
   one, is a OpenNetInf prototype from the 4WARD project.  Besides a
   rich set of NetInf mechanisms implemented, it also provides a browser
   plug-in and video streaming software. The SAIL project developed a
   hybrid host-centric and information-centric network architecture
   called the Global Information Network (GIN). The prototype for this
   can be downloaded from http://gin.ngnet.it.


2.1.4.  COMET

   The EU FP7 COMET project developed a simulator, called Icarus, which
   implements ProbCache [PROBCACHE], centrality-based in-network caching
   [CL4M] and the hash-route-based algorithms detailed in [HASHROUTE].
   The simulator is built in Python and makes use of the Fast Network
   Simulator Setup tool [FNSS] to configure the related parameters of
   the simulation. The simulator is available from:
   https://github.com/lorenzosaino/icarus/


2.1.5.  Large-scale Testing

   An important consideration in the evaluation of any kind of future
   Internet mechanism, lies in the characteristics of that evaluation
   itself.  Often, central to the assessment of the features provided by
   a novel mechanism, lies the consideration of how it improves over
   already existing technologies, and by "how much."  With the
   disruptive nature of clean-slate approaches generating new and
   different technological requirements, it is complex to provide
   meaningful results for a network layer framework, in comparison with
   what is deployed in the current Internet.  Thus, despite the
   availability of ICN implementations and simulators, the need for
   large-scale environments supporting experimental evaluation of novel
   research is of prime importance to the advancement of ICN deployment.

   In this regard, initiatives such as the Future Internet Research and
   Experimentation Initiative (www.ict-fire.eu), enable researchers to
   test new protocols and architectures in real conditions over
   production networks (e.g., through virtualization and software-
   defined networking mechanisms), simplifying the validation of future
   evolutions and reducing the gap between research and deployment.
   Similarly, Future Internet Design (www.nets-find.net) is a long-term
   initiative along the same direction in the US. GENI (www.geni.net)
   also offers experimentation infrastructure as does PlanetLab
   (www.planet-lab.org), which likely offers the largest testbed
   available today. Those wishing to perform smaller, more controlled



Pentikousis, et al.      Expires April 24, 2014                 [Page 7]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   experiments can also consider the Emulab testbed (www.emulab.net),
   which allows various topologies to be configured.

   The Asia Future Internet Forum (www.asiafi.net) has also designed a
   testbed mainly used for ICN experiments.  This testbed consists of
   multiple servers located in Asia and will be (presumably) expanded
   with servers in other locations.  Each testbed server includes
   multiple Linux kernel-based LXCs.  One container is called "bridge
   container" and the other container is called "user container".  A
   bridge container has a global IP address used to connect to the
   physical network through bridge mode, and a local (private) IP
   address.  A user container, which is assigned to each researcher,
   connects to the bridge container in the same server using their local
   IP addresses.  A user container connects to the researcher's remote
   containers located in different servers via tunnels established
   between its local bridge container and the remote bridge containers.

   Finally, the National Institute of Information and Communications
   Technology (NICT) builds and operates the high-performance testbed
   JGN-X (see http://www.jgn.nict.go.jp/english/index.html), which has
   cutting-edge network functions and technologies including those
   currently in development.  JGN-X aims to establish new-generation
   network technology and accelerate the R&D in areas such as network
   virtualization and advanced operations of virtualized layers.  JGN-X
   is used for collaboration among developers in order to foster the
   establishment and expansion of new-generation network technology.


2.2.  Topology Selection

   [draft-irtf-icnrg-scenarios] introduced several topologies that have
   been used in ICN studies so far but, to date and to the best of our
   understanding, there is no single topology that can be used to easily
   evaluate all aspects of the ICN paradigm.  There is rough consensus
   that the classic dumbbell topology cannot serve well future
   evaluations of ICN approaches.  Therefore, one should consider a
   range of topologies, each of which would stress different aspects, as
   outlined earlier in this document. Current Internet traces are also
   available to assist in this, e.g. see
   http://www.caida.org/data/active/internet-topology-data-kit and
   http://www.cs.washington.edu/research/networking/rocketfuel.

   Depending on what is the focus of the evaluation, intra-domain
   topologies alone may be appropriate.  However, those interested, for
   example, in quantifying transit costs will require inter-domain
   traces (note that the above CAIDA traces offer this).  Scalability is
   an important consideration in this choice of this with CAIDA's ITDK
   traces recording millions of routers across thousands of domains.



Pentikousis, et al.      Expires April 24, 2014                 [Page 8]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   Beyond these traces there is a wide range of synthetic topologies,
   such as the Barabasi-Albert model [BA] and the Watts-Strogatz small-
   world topology [WATTS].  These synthetic traces allow experiments to
   be performed whilst controlling various key parameters (e.g. degree).
    Through this, different aspects can be investigated, such as
   inspecting resilience properties.  For some research, this may be
   more appropriate as, practically speaking, there are no assurances
   that a future ICN will share the same topology with today's networks.

   Besides defining the evaluation topology as a graph G = (V,E), where
   V is the set of vertices (nodes) and E is the set of edges (links),
   one should also clearly define and list the respective matrices that
   correspond to the network, storage and computation capacities
   available at each node as well as the delay characteristics of each
   link, so that the results obtained can be easily replicated in other
   studies.  Recent work by Hussain and Chen [Montage], although
   currently addressing host-centric networks, could also be leveraged
   and be extended by the ICN community. Measurement information can
   also be taken from existing platforms such as iPlane
   (http://iplane.cs.washington.edu), which can be used to provide
   configuration parameters such as access link capacity and delay.
   Alternatively, synthetic models such as [DELAY] can be used to
   configure such topologies.

   Finally, the dynamic aspects of a topology, such as node and content
   mobility, disruption patterns, packet loss rates as well as link and
   node failure rates, to name a few, should also be carefully
   considered.  As mentioned in [draft-irtf-icnrg-scenarios], for
   example, contact traces from the DTN community could also be used in
   ICN evaluations.


2.3.  Traffic Load

   As we are still lacking ICN-specific traffic workloads we can
   currently only extrapolate from today's workloads.  In this
   subsection we provide a first draft of a set of common guidelines, in
   the form of what we will refer to as a content catalog for different
   scenarios.  This catalog, which is based on previously published
   work, could be used to evaluate different ICN proposals, for example,
   on routing, congestion control, and performance, and can be
   considered as other kinds of ICN contributions emerge.

   We take scenarios from today's Web, file sharing (BitTorrent-like)
   and User Generated Content (UGC) platforms (e.g., YouTube), as well
   as Video on Demand (VoD) services.  Publicly available traces for
   these include those available from web sites such as
   http://mikel.tlm.unavarra.es/~mikel/bt_pam2004,



Pentikousis, et al.      Expires April 24, 2014                 [Page 9]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   http://multiprobe.ewi.tudelft.nl/multiprobe.html,
   http://an.kaist.ac.kr/traces/IMC2007.html, and
   http://traces.cs.umass.edu/index.php/Network/Network.

   The content catalog for each type of traffic can be characterized by
   a specific set of parameters: the cardinality of the estimated
   content catalog, the average size of the exchanged contents (either
   chunks or entire named information objects), and the statistical
   distribution that best reflect the popularity of objects and their
   request frequency.  Table I summarizes the content catalog.  With
   this shared point of reference, the use of the same set of parameters
   (depending on the scenario of interest) among researchers will be
   eased, and different proposals could be compared on a common base.

   Table I. Content Catalog

   Traffic | Catalog |  Mean Object Size  |  Popularity Distribution
    Load   |  Size   |  [L4][L5][L7][L8]  |  [L3][L5][L6][L11][L12]
           | [L1][L2]|  [L9][L10]         |
           | [L3][L5]|                    |
   ==================================================================
   Web     |  10^12  | Chunk: 1-10 kB     | Zipf with
           |         |                    | 0.64 <= alpha <= 0.83
   ------------------------------------------------------------------
   File    | 5x10^6  | Chunk: 250-4096 kB | Zipf with
   sharing |         | Object: ~800 MB    | 0.75 <= alpha<= 0.82
   ------------------------------------------------------------------
   UGC     |  10^8   | Object: ~10 MB     | Zipf, alpha >= 2
   ------------------------------------------------------------------
   VoD     |  10^4   | Object: ~100 MB    | Zipf, 0.65 <= alpha <= 1
   ==================================================================

   * UGC = User Generated Content ** VoD = Video on Demand

   Several studies in the past years have stated that Zipf's law is the
   discrete distribution that best represents the request frequency in a
   number of application scenarios, ranging from the Web to VoD
   services.  The key aspect of this distribution is that the frequency
   of a content request is inversely proportional to the rank of the
   content itself, i.e., the smaller the rank, the higher the request
   frequency.  If we denote with M the content catalog cardinality and
   with 1 <= i <= M the rank of the i-th most popular content, we can
   express the probability of requesting the content with rank "i" as:

   P(X=i) = ( 1/i^(alpha) ) / C, with C = SUM(1 / j^(alpha)), alpha > 0

   where the sum is obtained considering all values of j, 1 <= j <= M.




Pentikousis, et al.      Expires April 24, 2014                [Page 10]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   Further, a variation of the Zipf distribution, termed the Mandelbrot-
   Zipf distribution, has been suggested by [P2PMod] to better model
   environments where nodes can locally store previously requested
   content.  For example, it was observed that peer-to-peer file sharing
   applications typically exhibited a 'fetch-at-most-once' style of
   behavior.  This is because peers tend to persistently store the files
   they download, a behavior that may also be prevalent in ICN.


2.4.  Choosing Relevant Metrics

   ICN is a networking concept that spun out of the desire to align the
   operation model of a network with the model of its typical use.  For
   TCP/IP networks, this means to change the mechanisms of data access
   and transport from a host-to-host model to a user-to-information
   model.  The premise is that the effort invested in changing models
   will be offset, or even surpassed, by the potential of a "better"
   network.  However, such a claim can be validated only if it is
   quantified.

   Quantification of network performance requires a set of standard
   metrics. These metrics should be broad enough so they can be applied
   equally to host-centric and information-centric (or other) networks.
   This will allow reasoning about a certain ICN approach in relation to
   an earlier version of the same approach, to another ICN approach or
   to the incumbent host-centric approach. It will therefore be less
   difficult to gauge optimization and research direction.  On the other
   hand, the metrics should be targeted to network performance only and
   should avoid unnecessary expansion into the physical and application
   layers.  Similarly, at this point, it is more important to capture as
   metrics only the main figures of merit and to leave more esoteric and
   less frequent cases for the future.

   To arrive at a set of relevant metrics, it would be beneficial to
   look at the metrics used in existing ICN approaches, such as CCN
   [CCN] [VoCCN] [NDNP], NetInf [4WARD6.1] [4WARD6.3] [SAIL-B2] [SAIL-
   B3], PURSUIT [PRST4.5], COMET [CMT-D5.2] [CMT-D6.2], Connect [SHARE]
   [RealCCN], and CONVERGENCE [ICN-Web] [ICN-Scal] [ICN-Tran].  The
   metrics used in these approaches fall into two categories: metrics
   for the approach as a whole, and metrics for individual components
   (resolution, routing, etc.).  Metrics for the entire approach are
   further subdivided into traffic and system metrics. It is important
   to note that the various approaches do not name or define metrics
   consistently. This is a major problem when trying to find metrics
   that allow comparison between approaches.  For the purposes of
   exposition, in what follows we have tried to smooth differences by
   pitting similarly defined metrics under the same name.  Also, due to
   space constraints, we have chosen to report here only the most common



Pentikousis, et al.      Expires April 24, 2014                [Page 11]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   metrics between approaches.  For more details the reader should
   consult the references for each approach.

   Traffic metrics in existing ICN approaches are summarized in Table
   II.  These are metrics for evaluating an approach mainly from the
   perspective of the end user, i.e., the consumer, provider, or owner
   of the content or service.  Depending on the level where these
   metrics are measured, we have made the distinction into user,
   application and network-level traffic metrics.  So for example,
   network-level metrics are mostly focused on packet characteristics,
   whereas user-level metrics can cover elements of human perception.
   The approaches don't make this distinction explicitly, but we can see
   from the table that CCN and NetInf have used metrics from all levels,
   PURSUIT and COMET have focused on lower-level metrics, and Connect
   and CONVERGENCE prefer higher-level metrics.  Throughput and download
   time seem to be the most popular metrics altogether.

   Table II. Traffic metrics used in ICN evaluations

                   User   |    Application    |        Network
               ======================================================
                 Download | Goodput | Startup | Throughput |  Packet
                   time   |         | latency |            |  delay
   ==================================================================
   CCN         |    x     |    x    |         |      x     |    x
   ------------------------------------------------------------------
   NetInf      |    x     |         |    x    |      x     |    x
   ------------------------------------------------------------------
   PURSUIT     |          |         |    x    |      x     |    x
   ------------------------------------------------------------------
   COMET       |          |         |    x    |      x     |
   ------------------------------------------------------------------
   Connect     |    x     |         |         |            |
   ------------------------------------------------------------------
   CONVERGENCE |    x     |    x    |         |            |
   ==================================================================


   While traffic metrics are more important for the end user, the owner
   or operator of the networking infrastructure is normally more
   interested in system metrics, which can reveal the efficiency of an
   approach.  The various ICN approaches have used system metrics, but
   unfortunately the situation is not as coherent as with the traffic
   metrics.  The most common system metrics used are: protocol overhead,
   total traffic, transit traffic, cost savings, router cost, and router
   energy consumption.

   Besides the traffic and systems metrics that aim to evaluate an



Pentikousis, et al.      Expires April 24, 2014                [Page 12]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   approach as a whole, all of the surveyed approaches also evaluate the
   performance of individual components. The name resolution,
   request/data routing, and data caching are the most typical
   components, so Table III presents the popular metrics for each of
   those components.  FIB size and path length, i.e., the routing
   component metrics, are almost ubiquitous among approaches, perhaps
   due to the networking background of the involved researchers.  That
   might be also the reason for the sometimes decreased focus on traffic
   and system metrics, in favor of component metrics.  It can certainly
   be argued that traffic and system metrics are affected by component
   metrics, however no approach has made the relationship clear.  With
   this in mind, and also taking into account that traffic and system
   metrics are readily useful to end users and network operators, we
   will restrict ourselves to those in the following sections.

   Table III. Component metrics in existing ICN approaches

                      Resolution      |    Routing    |    Cache
               ======================================================
                 Resolution | Request | FIB  |  Path  | Size |  Hit
                    time    |  rate   | size | length |      | ratio
   ==================================================================
   CCN         |     x      |         |  x   |   x    |   x  |   x
   ------------------------------------------------------------------
   NetInf      |     x      |    x    |      |   x    |      |   x
   ------------------------------------------------------------------
   PURSUIT     |            |         |  x   |   x    |      |
   ------------------------------------------------------------------
   COMET       |     x      |    x    |  x   |   x    |      |   x
   ------------------------------------------------------------------
   CONVERGENCE |            |    x    |  x   |        |   x  |
   ==================================================================


   Before proceeding, we should note that we'd like our metrics to be
   applicable to host-centric networks as well.  Standard metrics
   already exist for IP networks and it would certainly be beneficial to
   take them into account.  It is encouraging that many of the metrics
   used by existing ICN approaches can also be used on IP networks and
   that all of the approaches have tried on occasion to draw the
   parallels.


2.4.1.  Traffic Metrics

   At their core, host-centric and information-centric networking
   function as data transport services.  Information of interest to a
   user resides in one or more storage points connected to the network



Pentikousis, et al.      Expires April 24, 2014                [Page 13]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   and, on the user's request, the network transports this information
   to the user for consumption. We could therefore do worse than to
   quantify the data transport performance of the network in terms of
   Quality of Service (QoS) metrics.

   The IETF has been working for more than a decade on devising metrics
   and methods for measuring the performance of IP networks. The work
   has been carried out largely within the IPPM WG, guided by a relevant
   framework [RFC2330].  IPPM metrics include delay, delay variation,
   loss, reordering, and duplication.  While the IPPM work is certainly
   based on packet-switched IP networks, it is conceivable that it can
   be modified and extended to cover ICN networks as well. However, more
   study is necessary to turn this claim into a certainty.  Many experts
   have toiled for a long time on devising and refining the IPPM metrics
   and methods, so it would be an advantage to use IPPM on measuring ICN
   performance.  In addition, IPPM works already for host-centric
   networks, so comparison with information-centric networks would
   entail only the ICN extension of the IPPM framework.  Finally, an
   important benefit of measuring the transport performance of a network
   at it's output, using QoS metrics such as IPPM, is that it can be
   done mostly without any dependence to applications.

   Another option for measuring transport performance would be to use
   Quality of Service metrics, not at the output of the network like
   with IPPM, but at the input to the application.  So for an
   application like live video streaming the relevant metrics would be
   startup latency, playout lag and playout continuity.  The benefit of
   this approach is that it abstracts away all details of the underlying
   transport network, so it can be readily applied to compare between
   networks of different concepts (host-centric, information-centric, or
   other).  As implied earlier, the drawback of the approach is its
   dependence on the application, so it is likely that different (types
   of) applications will require different metrics.  It might be
   possible to identify standard metrics for each type of application,
   but the situation is not as clear as with IPPM metrics and further
   investigation is necessary.

   At a higher level of abstraction, we could measure the network's
   transport performance at the application output.  This entails
   measuring the quality of the transported and reconstructed
   information as perceived by the user during consumption.  In such an
   instance we would use Quality of Experience (QoE) metrics, which are
   by definition dependent on the application. For example, the
   standardized methods for obtaining a Mean Opinion Score (MOS) for
   VoIP (e.g., ITU-T P.800) is quite different from those for IPTV
   (e.g., PEVQ).  These methods are notoriously hard to implement, as
   they involve real users in a controlled environment.  Such
   constraints can be relaxed or dropped by using methods that model



Pentikousis, et al.      Expires April 24, 2014                [Page 14]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   human perception under certain environments, but these methods are
   typically intrusive.  The most important drawback of measuring
   network performance at the output of the application is that only one
   part of each measurement is related to network performance.  The rest
   is related to application performance, e.g., video coding, or even
   device capabilities, both of which are irrelevant to our purposes
   here and are generally hard to separate.  We therefore see the use of
   QoE metrics in measuring ICN performance as a poor choice.


2.4.2.  System Metrics

   Overall system metrics that need to be considered include
   reliability, scalability, energy efficiency, and delay/disconnection
   tolerance.  In deployments where ICN is addressing specific
   scenarios, relevant system metrics could be derived from current
   experience.  For example, in IoT scenarios, which were discussed
   earlier in [draft-irtf-icnrg-scenarios], it is reasonable to consider
   the current generation of sensor nodes, sources of information, and
   even measurement gateways (e.g., for smart metering at homes) or
   smartphones.  In this case, ICN operation ought to be evaluated with
   respect not only to overall scalability and network efficiency, but
   also the impact on the nodes themselves.  Karnouskos et al.
   [SensReqs] provide a comprehensive set of sensor and IoT-related
   requirements, for example, which include aspects such as resource
   utilization, service life-cycle management and device management.

   Additionally, various specific metrics are also critical in
   constrained environments, such as CPU processing requirements,
   signaling overhead, and memory allocation for caching procedures in
   addition to power consumption and battery lifetime. Also, in nodes
   acting as gateways, which typically not only act as a point of
   service to a large number of nodes, but also have to satisfy the
   information requests from remote entities; they need to consider
   scalability-related metrics, such as frequency and processing of
   successfully satisfied information requests.

   Finally, given the in-network caching functionality of Information-
   Centric Networks, metrics for the efficiency and performance of in-
   network caching have to be defined.  Such metrics will need to guide
   researchers and operators regarding the performance of in-network
   caching algorithms.  A first step on this direction has been made in
   [L9].  The paper proposes a formula that approximates the proportion
   of time that a content stays in a network cache.  The model takes as
   input the rate of requests for a given content (the Content of
   Interest) and the rate of requests for all other contents that go
   through the given network element (router) and move the CoI down in
   the (LRU) cache.  The formula takes also into account the size of the



Pentikousis, et al.      Expires April 24, 2014                [Page 15]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   cache of this router.

   The output of the model essentially reflects the probability that the
   CoI will be found in a given cache.  The initial study [L9] is
   applied to the CCN/NDN framework, where contents get cached at every
   node they traverse, while efforts are underway to assess the accuracy
   of the model for other caching strategies.  The formula according to
   which the probability or proportion is calculated is given by:

   pi = [mu/(mu+lambda)]^N,

   where lambda is the request rate for CoI, mu is the request rate for
   contents that move CoI down the cache and N is the size of the cache
   (in slots).

   The formula can be used to assess the caching performance of the
   system and can also potentially be used to identify the gain of the
   system due to caching. This can then be used to compare against gains
   by other factors, e.g., addition of extra bandwidth in the network.


2.5.  Resource Equivalence and Tradeoffs

   As we have seen above, every ICN network is built from a set of
   resources, which include link capacities, different types of memory
   structures and repositories used for storing named information
   objects and chunks temporarily (i.e. caching) or persistently, as
   well as name resolution and other lookup services.  Complexity and
   processing needs in terms of forwarding decisions, management (e.g.
   need for manual configuration, explicit garbage collection, and so
   on), and routing (i.e. amount of state needed, need for manual
   configuration of routing tables, support for mobility, etc.) set the
   stage for a range of engineering tradeoffs.

   In order to be able to compare different ICN approaches it would be
   beneficial to be able to define equivalence in terms of different
   resources which today are considered incomparable.  For example,
   would provisioning an additional 5 Mb/s link capacity lead to better
   performance than adding 100 GB of in-network storage?  Within this
   context one would consider resource equivalence (and the associated
   tradeoffs) for example for cache hit ratios per GB of cache,
   forwarding decision times, CPU cycles per forwarding decision, and so
   on.


3.  ICN Security Aspects

   The introduction of an information-centric networking architecture



Pentikousis, et al.      Expires April 24, 2014                [Page 16]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   and the corresponding communication paradigm changes many aspects of
   network security.  These will affect all the scenarios described in
   [draft-irtf-icnrg-scenarios].  Additional evaluation will be required
   to ensure relevant security requirements are appropriately met by the
   implementation of the chosen architecture in the various scenarios.

   The various ICN architectures that are currently proposed have
   concentrated on authentication of delivered content to ensure the
   integrity of the content.  However the approaches are primarily
   applicable to freely accessible content that does not require access
   authorization, although they will generally support delivery of
   encrypted content.

   The introduction of widespread caching mechanisms may also provide
   additional attack surfaces.  The caching architecture to be used also
   needs to be evaluated to ensure that it meets the requirements of the
   usage scenarios.

   In practice, the work on security in the various ICN research
   projects has been heavily concentrated on authentication of content.
   Work on authorization, access control, privacy and security threats
   due to the expanded role of in-network caches has been quite limited.
    A roadmap for improving the security model in NetInf can be found in
   [NETINFSC].  In the rest of this section we briefly consider the
   issues and provide pointers to the work that has been done on the
   security aspects of the architectures proposed.


3.1. Authentication

   For fully secure content distribution, content access requires that
   the receiver needs to be able to reliably assess:

      validity:   is it a complete, uncorrupted copy of what was
                  originally published;

      provenance: can the receiver identify the publisher, and, if so,
                  whether it and the source of any cached version of the
                  document can be adequately trusted; and

      relevance:  is the content an answer to the question that the
                  receiver asked.

   All the ICN architectures considered in this document primarily
   target the validity requirement using strong cryptographic means to
   tie the content request name to the content.  Provenance and
   relevance are directly targeted to varying extents:  There is a
   tussle or trade-off between simplicity and efficiency of access and



Pentikousis, et al.      Expires April 24, 2014                [Page 17]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   level of assurance of all these traits.   For example, maintaining
   provenance information can become extremely costly, particularly when
   considering (historic) relationships between multiple objects.
   Architectural decisions have therefore been taken in each case as to
   whether the assessment is carried out by the ICN or left to the
   application.

   An additional consideration for authentication is whether a name
   should be irrevocably and immutably tied to a static piece of
   preexisting content or whether the name can be used to refer to
   dynamically or subsequently generated content.  Schemes that only
   target immutable content can be less resource hungry as they can use
   digest functions rather than public key cryptography for generating
   and checking signatures.  However, this can increase the load on
   applications. This is because they are required to manage many names,
   rather than using a single name for an item of evolving content that
   changes over time (e.g. a piece of data containing an age reference).

   NetInf uses the Named Information (ni) URI scheme [RFC6920] to
   identify content.  This allows NetInf to assure validity without any
   additional information but gives no assurance on provenance or
   relevance.  A "search" request allows an application to identify
   relevant content and applications may choose to structure content to
   allow provenance assurance but this will typically require additional
   network access.  NetInf validity authentication is consequently
   efficient in a network environment with intermittent connectivity as
   it does not force additional network accesses and allows the
   application to decide on provenance validation if required.  NetInf
   primarily targets static content, but an extension would allow
   dynamic content to be handled.  The immutable case only uses digest
   functions.

   DONA [DONA] and CCN [CCN], [SECCONT] integrate most of the data
   needed to verify provenance into all content retrievals but need to
   be able to retrieve additional information (typically a security
   certificate) in order to complete the provenance authentication.
   Whether the application has any control of this extra retrieval will
   depend on the implementation.  CCN is explicitly designed to handle
   dynamic content allowing names to be pre-allocated and attached to
   subsequently generated content.  DONA offers variants for dynamic and
   immutable content.

   PURSUIT [PSTSEC] appears to allow implementers to choose the
   authentication mechanism so that it can, in theory, emulate the
   authentication strategy of any of the other architectures. It is not
   clear whether different choices would lead to lack of
   interoperability.




Pentikousis, et al.      Expires April 24, 2014                [Page 18]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


3.2. Authorization, Access Control and Statistics

   A potentially major concern for all the ICN architectures considered
   here is that they do not provide any inbuilt support for an
   authorization framework or for statistics monitoring.  Once content
   has been published and cached in servers, routers or end points not
   controlled by the publisher, the publisher has no way to enforce
   access control, determine which users have accessed the content or
   revoke its publication.  In fact, in some cases, it is even difficult
   for the publishers themselves to perform access control, where
   requests do not necessarily contain host/user identifier information.

   Access could be limited by encrypting the content but the necessity
   of distributing keys out-of-band appears to negate the advantages of
   in-network caching.  This also creates significant challenges when
   attempting to manage and restrict key access. An authorization
   delegation scheme has been proposed [ACDICN] but this requires access
   to a server controlled by the publisher to obtain an access token
   making it essentially just an out-of-band key distribution system.

   Evaluating the impact of the absence of these features will be
   essential for any scenario where an ICN architecture might be
   deployed.  It may have a seriously negative impact on the
   applicability of ICN in commercial environments unless a solution can
   be found.


3.3. Privacy

   Another area where the architectures have not been significantly
   analyzed is privacy.  Caching implies a trade-off between network
   efficiency and privacy.  The activity of users is significantly more
   exposed to the scrutiny of cache owners with whom they may not have
   any relationship.

   Although in many ICN architectures, the source of a request is not
   explicitly identified, an attacker may be able to obtain considerable
   information if s/he can monitor transactions on the cache and obtain
   details of the objects accessed, the topological direction of
   requests and information about the timing of transactions.   The
   persistence of data in the cache can make life easier for an attacker
   by giving a longer timescale for analysis.

   The impact of CCN on privacy has been investigated in a useful
   master's thesis [CCNSEC].  The analysis in this thesis is mostly
   applicable to all of the ICN architectures because it is mostly
   focused on the common caching aspect.  The privacy risks of named
   data networking are also highlighted in [CCNPRIV].  Further work on



Pentikousis, et al.      Expires April 24, 2014                [Page 19]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   privacy in ICNs can be found in [CONPRV].

3.4. Changes to the Network Security Threat Model

   The architectural differences of the various ICN models as compared
   to TCP/IP have consequences for network security.  There is limited
   consideration of the threat models and potential mitigation in the
   various documents describing the architectures.  The references
   [CCNSEC] and [CONPRV] also consider the changed threat model.  Some
   of the key aspects are:

    o Caching implies a tradeoff between network efficiency and user
      privacy as discussed in Section 3.3.

    o More powerful routers upgraded to handle persistent caching
      increase the network's attack surface.  This is particularly the
      case in systems (e.g., CCN) that may need to perform cryptographic
      checks on content that is being cached. For example, not doing
      this could lead routers to disseminate invalid content.

    o ICNs makes it difficult to identify the origin of a request as
      mentioned in Section 4.3 slowing down the process of blocking
      requests and requiring alternative mechanisms to differentiate
      legitimate requests from inappropriate ones as access control
      lists (ACLs) will probably be of little value for ICN requests.

    o Denial-of-service (DoS) attacks may require more effort on ICN
      than on TCP/IP but they are still feasible.  One reason for this
      is that it is difficult for the attacker to force repeated
      requests for the same content onto a single node; ICNs naturally
      spread content so that after the initial few requests, subsequent
      requests will generally be satisfied by alternative sources,
      blunting the impact of a DoS attack. That said, there are many
      ways around this, e.g., generating random suffix identifiers that
      always result in cache misses.

    o Per-request state in routers can be abused for DoS attacks.

    o Caches can be misused in the following ways:

       + Attackers can use caches as storage to make their own content
         available.

       + The efficiency of caches can be decreased by attackers with the
         goal of DoS attacks.

       + Content can be extracted by any attacker connected to the
         cache, putting users' privacy at risk.



Pentikousis, et al.      Expires April 24, 2014                [Page 20]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   Appropriate mitigation of these threats will need to be considered in
   each scenario.


4.  Security Considerations

   This document does not impact the security of the Internet.


5.  IANA Considerations

   This document presents no IANA considerations.


6.  Acknowledgments

   Daniel Corujo and Gareth Tyson contributed to an earlier version of
   this document.

   This document has benefited from reviews, pointers to the growing ICN
   literature, suggestions, comments and proposed text provided by the
   following members of the IRTF Information-Centric Networking Research
   Group (ICNRG), listed in alphabetical order: Marica Amadeo, Hitoshi
   Asaeda, Claudia Campolo, Suyong Eum, Dorothy Gellert, Luigi Alfredo
   Grieco, Myeong-Wuk Jang, Ren Jing, Will Liu, Antonella Molinaro,
   Ioannis Psaras, Dirk Trossen, Jianping Wang, Yuanzhe Xuan, and Xinwen
   Zhang.


7.  Informative References

   [RFC5743]  Falk, A., "Definition of an Internet Research Task Force
              (IRTF) Document Stream", RFC 5743, December 2009.

   [RFC2330]  Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
              "Framework for IP Performance Metrics", RFC 2330, May
              1998.

   [RFC6920]  Farrell, S., Kutscher, D., Dannewitz, C., Ohlman, B.,
              Keranen, A., and P. Hallam-Baker, "Naming Things with
              Hashes", RFC 6920, April 2013.

   [ndnSIM]   Afanasyev, A. et al., ndnSIM: NDN simulator for NS-3 NDN
              Technical Report NDN-0005, Revision 2, October 2012.

   [ccnSim]   Rossini, G. and D. Rossi, "Large scale simulation of CCN
              networks", Proc. Algotel 2012 , La Grande Motte, France,
              May 2012.



Pentikousis, et al.      Expires April 24, 2014                [Page 21]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   [CCNPL]    Muscariello, L., "Content centric networking packet level
              simulator", available online at
              http://perso.rd.francetelecom.fr/muscariello/sim.html

   [CCNj]     Cianci, I. et al. "CCN - Java Opensource Kit EmulatoR for
              Wireless Ad Hoc Networks", Proc. 7th ACM Int. Conf. on
              Future Internet Technologies, Seoul, Korea, Sept., 2012.

   [IEICE]    G. Parisis, D. Trossen, and H. Asaeda, "A Node Design and
              a Framework for Development and Experimentation for an
              Information-Centric Network", IEICE Trans. Commun., vol.
              E96-B, no. 7, pp.1650-1660, July 2013.

   [ICN-Sim]  N. Vastardis et al., "Simulation Tools Enabling Research
              on Information-centric Networks", Proc. ICC FutureNet
              Workshop. IEEE, 2012.

   [PROBCACHE] I. Psaras, W. Chai, G. Pavlou, "Probabilistic In-Network
              Caching for Information-Centric Networks", Proc. SIGCOMM
              ICN Workshop. ACM, 2012.

   [CL4M]     Chai, W. K. et al., "Cache 'Less for More' in Information-
              centric Networks", Proc. Networking.  IFIP, 2012.

   [HASHROUTE] L. Saino, I. Psaras, G. Pavlou, "Hash-routing Schemes for
              Information-Centric Networking", Proc. SIGCOMM ICN
              Workshop. ACM, 2013.

   [FNSS]     L. Saino, C. Cocora and G. Pavlou, "A Toolchain for
              Simplifying Network Simulation Setup", Proc. SIMUTOOLS.
              ACM, 2013.

   [BA]       Barabasi, A. and R. Albert, "Emergence of scaling in
              random networks", Science, vol. 286, no. 5439, pp. 509-
              512, 1999.

   [WATTS]    Watts, D. J. and S. H. Strogatz, "Collective dynamics of
              small-world networks", Nature, vol. 393, no. 6684, pp. 40-
              "10, 1998.

   [Montage]  Hussain, A. and J. Chen, "Montage Topology Manager: Tools
              for Constructing and Sharing Representative Internet
              Topologies", DETER Technical Report, ISI-TR-684, Aug.
              2012.

   [DELAY]    Kaune, S. et al., "Modelling the Internet Delay Space
              Based on Geographical Locations", Proc. Euromicro, Weimar,
              Germany, 2009.



Pentikousis, et al.      Expires April 24, 2014                [Page 22]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   [L1]       http://googleblog.blogspot.it/2008/07/we-knew-web-was-
              big.html

   [L2]       Zhang, C., Dhungel, P., and K. Di Wu., "Unraveling the
              BitTorrent ecosystem", IEEE Transactions on Parallel and
              Distributed Systems, pp. 1164-1177, 2010.

   [L3]       Cha, M., Kwak, H., Rodriguez, P., Ahn, Y.-Y., and S. Moon,
              "I tube, you tube, everybody tubes: analyzing the world's
              largest user generated content video system", Proc. ACM
              SIGCOMM conference on Internet measurement (IMC), San
              Diego (CA), USA, Oct. 2007.

   [L4]       Zhou, J., Li,  Y., Adhikari, K., and Z.-L. Zhang,
              "Counting YouTube videos via random prefix sampling", In
              Proc. of IMC'11, Berlin, Germany, Nov. 2011.

   [L5]       Fricker, C., Robert,  P., Roberts, J. and N. Sbihim,
              "Impact of traffic mix on caching performance in a
              content-centric network", In Proc. of IEEE NOMEN 2012,
              Workshop on Emerging Design Choices in Name-Oriented
              Networking, Orlando, USA, Mar. 2012.

   [L6]       Yu, H., Zheng,  D., Zhao, B. Y. and W. Zheng,
              "Understanding user behavior in large-scale video-on-
              demand systems", In SIGOPS Oper. Syst. Rev., Vol. 40, pp.
              333-344, April 2006.

   [L7]       Marciniak, P., Liogkas, N., Legout, A. and E. Kohler,
              "Small is not always beautiful",  In Proc. of IPTPS,
              International Workshop of Peer-to-Peer Systems, Tampa Bay,
              Florida (FL), USA, Feb. 2008.

   [L8]       Bellissimo, A., Levine, B. and P. Shenoy, "Exploring the
              use of BitTorrent as the basis for a large trace
              repository", University of Massachusetts, Tech. Rep.,
              2004.

   [L9]       Psaras, I. et al., "Modelling and Evaluation of CCN-
              Caching Trees", In Proc. of the 10th international IFIP
              conference on Networking, Valencia, Spain, May 2011.

   [L10]      Carofiglio, G., Gallo, M., Muscariello, L., and D. Perino,
              "Modeling Data Transfer in Content-Centric Networking", In
              Proc. of  ITC, San Francisco, USA, Sep. 2011.

   [L11]      Breslau, L., Cao,  P., Fan, L., Phillips, G. and S.
              Shenker, "Web caching and zipf-like distributions:



Pentikousis, et al.      Expires April 24, 2014                [Page 23]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


              evidence and implications", In Proc. of INFOCOM '99, New
              York (NY), USA,  Mar. 1999.

   [L12]      Mahanti, A., Williamson, C., and D. Eager., "Traffic
              analysis of a web proxy caching hierarchy", IEEE Network,
              Vol.14, No.3, pp.16-23, May/June 2000.

   [P2PMod]   Saleh, O., and M. Hefeeda, "Modeling and caching of peer-
              to-peer traffic", Proc. ICNP. IEEE, 2006.

   [CCN]      Jacobson, V. et al., "Networking Named Content", Proc.
              CoNEXT.  ACM, 2009.

   [VoCCN]    Jacobson, V. et al., "VoCCN: Voice-over Content-Centric
              Networks", Proc. CoNEXT Re-Arch Workshop.  ACM, 2009.

   [NDNP]     Zhang, L. et al., "Named Data Networking (NDN) Project",
              NDN Technical Report NDN-0001, Oct. 2010.  Available:
              http://named-data.net/publications/techreports/

   [4WARD6.1] Ohlman, B. et al., "First NetInf Architecture
              Description", 4WARD Project Deliverable D-6.1, Apr. 2009.

   [4WARD6.3] Ahlgren, B. et al., "NetInf Evaluation", 4WARD Project
              Deliverable D-6.3, June 2010.

   [SAIL-B2]  SAIL, "NetInf Content Delivery and Operations", SAIL
              Project Deliverable D-B.2, May. 2012.

   [SAIL-B3]  Kutscher, D. (ed.) et al., "Final NetInf Architecture",
              SAIL Project Deliverable D-B.3 , Jan. 2013.  Available:
              http://www.sail-project.eu/deliverables/

   [PRST4.5]  Riihijarvi, J. et al., "Final Architecture Validation and
              Performance Evaluation Report", PURSUIT Project
              Deliverable D4.5, Jan. 2013.

   [CMT-D5.2] B ben, A. et al, "Scalability of COMET System", COMET
              Project Deliverable D5.2, Feb. 2013.

   [CMT-D6.2] Georgiades, M. et al., "Prototype Experimentation and
              Demonstration", COMET Project Deliverable D6.2, Feb. 2013.

   [SHARE]    Muscariello, L. et al., "Bandwidth and storage sharing
              performance in information centric networking", Proc.
              SIGCOMM ICN Workshop.  ACM, 2011.

   [RealCCN]  Perino, D. et al., "A Reality Check for Content Centric



Pentikousis, et al.      Expires April 24, 2014                [Page 24]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


              Networking", Proc. SIGCOMM ICN Workshop. ACM, 2011.

   [ICN-Web]  Detti, A. et al., "Supporting the Web with an Information
              Centric Network that Routes by Name", Elsevier Computer
              Networks, vol. 56, no. 17, Nov. 2012.

   [ICN-Scal] Blefari Melazzi, N. et al., "Scalability Measurements in
              an Information-Centric Network", Springer Lecture Notes in
              Computer Science (LNCS), vol. 7586, 2012.

   [ICN-Tran] Salsano, S. et al., "Transport-Layer Issues in Information
              Centric Networks ", Proc. SIGCOMM ICN Workshop. ACM, 2012.

   [SensReqs] Karnouskos, S.  et al., "Requirement considerations for
              ubiquitous integration of cooperating objects", Proc.
              NTMS. IFIP, 2011.

   [NETINFSC] Renault, E, Ahmad, A., and M. Abid, "Towards a Security
              Model for the Future Network of Information", Proc. Conf.
              Ubiquitous Information Technologies and Applications,
              IEEE, 2009.

   [DONA]     Koponen, T. et al., "A Data-Oriented (and Beyond) Network
              Architecture", Proc. SIGCOMM.  ACM, 2007.

   [SECCONT]  Smetters, D., and V. Jacobson, "Securing network content",
              Technical Report TR-2009-01, PARC, 2009.

   [PSTSEC]   Tagger, B., et al, "Update on the Architecture and Report
              on Security Analysis", Deliverable 2.4, PURSUIT EU FP7
              project, April 2012.

   [ACDICN]   Fotiou, N. et al., "Access control enforcement delegation
              for information-centric networking architectures", Proc.
              SIGCOMM ICN Workshop. ACM, 2012.

   [CCNSEC]   Lauinger, T., "Security and Scalability of Content-Centric
              Networking", Masters Thesis, Technische Universitaet
              Darmstadt and Eurecom, Sep. 2010.

   [CCNPRIV]  Lauinger, Y., et al, "Privacy Risks in Named Data
              Networking: What is the Cost of Performance?", ACM SIGCOMM
              Computer Communication Review Editorial Note, vol. 42,
              iss. 5, 2012

   [CONPRV]   Chaabane, A et al, "Privacy in Content-Oriented
              Networking: Threats and Countermeasures", arXiv:1211.5183,
              2012.



Pentikousis, et al.      Expires April 24, 2014                [Page 25]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


Authors' Addresses


   Kostas Pentikousis (editor)
   EICT GmbH
   Torgauer Strasse 12-15
   10829 Berlin
   Germany

   Email: k.pentikousis@eict.de


   Borje Ohlman
   Ericsson Research
   S-16480 Stockholm
   Sweden

   Email: Borje.Ohlman@ericsson.com


   Elwyn Davies
   Trinity College Dublin/Folly Consulting Ltd
   Dublin, 2
   Ireland

   Email: davieseb@scss.tcd.ie


   Spiros Spirou
   Intracom Telecom
   19.7 km Markopoulou Avenue
   19002 Peania, Athens
   Greece

   Email: spis@intracom.com


   Gennaro Boggia
   Dep. of Electrical and Information Engineering
   Politecnico di Bari
   Via Orabona 4
   70125 Bari
   Italy

   Email: g.boggia@poliba.it


   Priya Mahadevan



Pentikousis, et al.      Expires April 24, 2014                [Page 26]


INTERNET DRAFT         ICN Evaluation Methodology       October 21, 2013


   Palo Alto Research Center
   3333 Coyote Hill Rd
   Palo Alto, CA 94304
   USA

   Email: Priya.Mahadevan@parc.com













































Pentikousis, et al.      Expires April 24, 2014                [Page 27]


Html markup produced by rfcmarkup 1.129c, available from https://tools.ietf.org/tools/rfcmarkup/