INTERNET-DRAFT                                              Marc Linsner
Intended Status: Informational                             Cisco Systems
Expires: October 4, 2014 March 27, 2015                                   Philip Eardley
                                                        Trevor Burbridge
                                                                      BT
                                                          Frode Sorensen
                                                                     NPT
                                                           April 2,
                                                      September 23, 2014

              Large-Scale Broadband Measurement Use Cases
                      draft-ietf-lmap-use-cases-03
                      draft-ietf-lmap-use-cases-04

Abstract

   Measuring broadband performance on a large scale is important for
   network diagnostics by providers and users, as well as for public
   policy.  Understanding the various scenarios and users of measuring
   broadband performance is essential to development of the Large-scale
   Measurement of Broadband Performance (LMAP) framework, information
   model and protocol. This document details two use cases that can
   assist to developing that framework.  The details of the measurement
   metrics themselves are beyond the scope of this document.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

Copyright and License Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2  Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . .  3
     2.1 Internet Service Provider (ISP) Use Case . . . . . . . . . .  3
     2.2 Regulators . . . . Regulator Use Case . . . . . . . . . . . . . . . . . . . . .  4
     2.3 Implementation options . . . . . . . . . . . . . . . . . . .  5
   3  Details of ISP Use Case . . . . . . . . . . . . . . . . . . . .  6  5
     3.1 Understanding the quality experienced by customers . . . . .  6  5
     3.2 Understanding the impact and operation of new devices and
         technology . . . . . . . . . . . . . . . . . . . . . . . . .  7  6
     3.3 Design and planning  . . . . . . . . . . . . . . . . . . . .  7  6
     3.4 Monitoring Service Level Agreements  . . . . . . . . . . . .  8  7
     3.5 Identifying, isolating and fixing network problems . . . . .  8  7
   4  Details of Regulator Use Case . . . . . . . . . . . . . . . . .  9  8
     4.1 Promoting competition through transparency . . . . . . . . .  9  8
     4.2 Promoting broadband deployment . . . . . . . . . . . . . . . 10  9
     4.3 Monitoring "net neutrality"  . . . . . . . . . . . . . . . . 11 10
   5  Implementation Options  . . . . . . . . . . . . . . . . . . . . 11
   6  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 12
   6
   7  Security Considerations . . . . . . . . . . . . . . . . . . . . 13
   7
   8  IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 14
   Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
   Informative References . . . . . . . . . . . . . . . . . . . . . . 14
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 16

1  Introduction

   This document describes two use cases for the Large-scale Measurement
   of Broadband Performance (LMAP), in particular use cases for ISPs (LMAP). Firstly, to enable network operators
   to understand the performance of the network and
   regulators.  Although there the quality
   experienced by customers. Secondly, to enable regulators to provide
   information on the performance of the ISPs in their jurisdiction.
   There are many other use cases for large-scale
   measurements systems, the two described here that are not the consensus
   starting point focus of the initial LMAP
   work, for defining example end users would like to use measurements to help
   identify problems in their home network and to monitor the system.
   performance of their broadband provider; it is expected that the same
   mechanisms are applicable.

2  Use Cases

   From the LMAP perspective, there is no difference between fixed
   service and mobile (cellular) service used for Internet access.
   Hence, like measurements will take place on both fixed and mobile
   networks.  Fixed services, commonly known as "Last Mile" include
   technologies like DSL, Cable, and Carrier Ethernet.  Mobile services
   include all those advertised as 2G, 3G, 4G, and LTE.  A metric
   defined to measure end-to-end services will execute similarly on all
   access technologies. Other metrics may be access technology specific.
   The LMAP architecture also covers both IPv4 and IPv6 networks.

2.1 Internet Service Provider (ISP) Use Case

   An ISP, or indeed another

   A network operator, operator needs to understand the performance of their
   networks, the performance of the suppliers (downstream and upstream
   networks), the performance of services, and the impact that such
   performance has on the experience of their customers. Largely, the
   processes that ISPs operate (which are based on network measurement)
   include:

      o Identifying, isolating and fixing problems problems, which may be in the
      network,
      services or with CPE and the service provider, or in the end user equipment.
      Such problems may be common to a point in the network topology
      (e.g. a single exchange), common to a vendor or equipment type
      (e.g. line card or home gateway) or unique to a single user line
      (e.g. copper access). Part of this process may also be helping
      users understand whether the problem exists in their home network
      or with an over-
      the-top over-the-top service instead of with their broadband
      (BB) product.

      o Design and planning. Through identifying the end user experience
      the ISP can design and plan their network to ensure specified
      levels of user experience. Services may be moved closer to end
      users, services upgraded, the impact of QoS assessed or more
      capacity deployed at certain locations. Service Level Agreements
      (SLAs) may be defined at network or product boundaries.

      o Understanding the quality experienced by customers. Alongside
      benchmarking competitors, gaining better insight into the user's
      service through a sample panel of the operator's own customers.
      The end-to-end perspective matters, across home/enterprise
      networks, peering points, Content Delivery Networks (CDNs), etc.

      o Understanding the impact and operation of new devices and
      technology. As a new product is deployed, or a new technology
      introduced into the network, it is essential that its operation
      and its impact on other services is measured. This also helps to quantify the
      advantage that the new technology is bringing and support the
      business case for larger roll-out.

2.2 Regulators Regulator Use Case

   Regulators in jurisdictions around the world are responding to
   consumers' adoption of Internet access services for traditional
   telecommunications and media services by promoting competition among
   providers of electronic communications, to ensure that users derive
   maximum benefit in terms of choice, price, and quality.

   Some jurisdictions have responded to a need for greater information
   about Internet access service performance in the development of
   regulatory policies and approaches for broadband technologies by
   developing large-scale measurement programs. Programs such as the
   U.S. Federal Communications Commission's (FCC) Measuring Broadband
   America (MBA), European Commission's Quality of Broadband Services in
   the EU reports and a growing list of other programs employ a diverse
   set of operational and technical approaches to gathering data to
   perform analysis and reporting on diverse aspects of broadband
   performance.

   While each jurisdiction responds to distinct consumer, industry, and
   regulatory concerns, much commonality exists in the need to produce
   datasets that are able to compare multiple Internet access service
   providers, diverse technical solutions, geographic and regional
   distributions, and marketed and provisioned levels and combinations
   of broadband Internet access services. In some jurisdictions, the
   role of measuring is provided by a measurement provider.

   Measurement providers measure network performance from users towards
   multiple content and application providers, including dedicated test
   measurement servers, to show a performance of the actual Internet
   access service provided by different ISPs. Users need to know the
   performance that they are achieving from their own ISP. In addition,
   they need to know the performance of other ISPs of same location as
   background information for selecting their ISP. Measurement providers
   will provide measurement results with associated measurement methods
   and measurement metrics.

   From a consumer perspective, the differentiation between fixed and
   mobile (cellular) Internet access services is blurring as the
   applications used are very similar. Hence, regulators are measuring
   both fixed and mobile Internet access services.

   A regulator's role in the development and enforcement of broadband
   Internet access service policies also requires that the measurement
   approaches meet a high level of verifiability, accuracy and provider-
   independence to support valid and meaningful comparisons of Internet
   access service performance performance.

   LMAP standards could answer regulators shared needs by providing
   scalable, cost-effective, scientifically robust solutions to the
   measurement and collection of broadband Internet access service
   performance information.

2.3 Implementation options

   There are several ways

3  Details of implementing a measurement system. The
   choice may be influenced ISP Use Case

3.1 Understanding the quality experienced by customers

   Operators want to understand the details quality of the particular use case
   and what the most important criteria are for the regulator, ISP or
   third party operating the measurement system.

   One way involves experience (QoE) of their
   broadband customers. The understanding can be gained through a special hardware device that
   "panel", i.e., a measurement probe is connected directly
   to the home gateway. The devices are deployed to a carefully selected
   panel few 100 or 1000
   of end users and they perform measurements according to a
   defined schedule. its customers. The schedule can run throughout the day, panel needs to allow
   continuous assessment include a representative sample
   for each of the network. Careful design ensures that
   measurements do not detrimentally impact the home user experience or
   corrupt the results by testing when the user is also using the operator's technologies (Fiber To The Premise (FTTP),
   Fiber To The Curb (FTTC), DSL...) and broadband line. options (80Mb/s,
   20Mb/s, basic...), ~100 probes for each. The system is therefore tightly controlled by the operator of would like the measurement system. One advantage
   end-to-end view of this approach is
   that it is possible to get reliable benchmarks for the performance of
   a service, rather than (say) just the access
   portion. So as well as simple network with only a few devices. One disadvantage is that it would
   be expensive to deploy hardware devices on a mass scale sufficient statistics like speed and loss
   rates they want to understand what the performance of the network at service feels like to the granularity of a
   single broadband user.

   Another approach
   customer. This involves implementing relating the measurement capability as
   a webpage or an "app" that end users are encouraged pure network parameters to download onto
   their mobile phone or computing device. Measurements are triggered by
   the end user, for example the user interface may have
   something like a button to
   "test my broadband now". Compared with the previous approach, the
   system 'mean opinion score' which will be service dependent
   (for instance web browsing QoE is much more loosely controlled, as the panel of end users and
   the schedule of tests are largely determined by latency above
   a few Mb/s).

   An operator will also want compound metrics such as "reliability",
   which might involve packet loss, DNS failures, re-training of the end users themselves
   rather than the measurement system. It would be easier to get large-
   scale, however it is harder
   line, video streaming under-runs etc.

   The operator really wants to get comparable benchmarks as understand the
   measurements are affected by end-to-end service
   experience. However, the home network and also the population (Ethernet, WiFi, powerline) is self-selecting
   highly variable and so potentially biased towards those who think
   they outside its control. To date, operators (and
   regulators) have a problem. This could be alleviated by stimulating
   widespread downloading of instead measured performance from the app and careful post-processing of home gateway.
   However, mobile operators clearly must include the
   results to reduce biases.

   There wireless link in
   the measurement.

   Active measurements are several other possibilities. For example, as a variant on the first most obvious approach, i.e., special
   measurement traffic is sent by - and to - the probe. In order not to
   degrade the service of the customer, the measurement capability could data should only
   be implemented
   as software embedded in sent when the home gateway, which would make user is silent, and it more
   viable to have shouldn't reduce the capability on every user line. As a variant
   customer's data allowance. The other approach is passive measurements
   on the
   second approach, customer's ordinary traffic; the end user could initiate measurements in response
   to a request from advantage is that it measures
   what the customer actually does, but it creates extra variability
   (different traffic mixes give different results) and especially it
   raises privacy concerns. [RFC6973] discusses privacy considerations
   for Internet protocols in general, whilst [framework] discusses them
   specifically for large-scale measurement system.

3  Details of ISP Use Case

3.1 Understanding the quality experienced by systems.

   From an operator's viewpoint, understanding customers

   Operators want better enables
   it to understand offer better services. Also, simple metrics can be more easily
   understood by senior managers who make investment decisions and by
   sales and marketing.

3.2 Understanding the quality impact and operation of experience (QoE) new devices and technology

   Another type of their
   broadband customers. The understanding can be gained through a
   "panel", i.e., a measurement probe is deployed to test new capabilities before they
   are rolled out. For example, the operator may want to: check whether
   a few 100 or 1000
   of its customers. The panel needs customer can be upgraded to include a representative sample
   for each of the operator's technologies (Fiber To The Premise (FTTP),
   Fiber To The Curb (FTTC), DSL...) and new broadband options (80Mb/s,
   20Mb/s, basic...), ~100 probes for each. The operator would like option; understand the
   end-to-end view
   impact of the service, rather than (say) just the access
   portion. So as well as simple network statistics like speed and loss
   rates they want IPv6 before it makes it available to understand its customers (will v6
   packets get through, what the service feels like to the
   customer. This involves relating the pure network parameters to
   something like a 'mean opinion score' which will the latency be service dependent
   (for instance web browsing QoE is largely determined by latency above to major websites, what
   transition mechanisms will be most appropriate?); check whether a few Mb/s).

   An operator new
   capability can be signaled using TCP options (how often it will also want compound metrics such as "reliability",
   which might involve packet loss, DNS failures, re-training of be
   blocked by a middlebox? - along the
   line, video streaming under-runs etc.

   The operator really wants lines of some existing
   experiments) [Extend TCP]; investigate a quality of service mechanism
   (e.g. checking whether Diffserv markings are respected on some path);
   and so on.

3.3 Design and planning

   Operators can use large scale measurements to understand help with their network
   planning - proactive activities to improve the end-to-end service
   experience. However, network.

   For example, by probing from several different vantage points the home
   operator can see that a particular group of customers has performance
   below that expected during peak hours, which should help capacity
   planning. Naturally operators already have tools to help this - a
   network (Ethernet, WiFi, powerline) is
   highly variable and outside element reports its control. To date, operators individual utilization (and
   regulators) have instead measured performance from the home gateway. perhaps other
   parameters). However, mobile operators clearly must include the wireless link in
   the measurement.

   Active making measurements are across a path rather than
   at a point may make it easier to understand the most obvious approach, i.e., special
   measurement traffic is sent network. There may
   also be parameters like bufferbloat that aren't currently reported by -
   equipment and/or that are intrinsically path metrics.

   With information gained from measurement results, capacity planning
   and network design can be more effective. Such planning typically
   uses simulations to - the probe. In order not to
   degrade emulate the service measured performance of the customer, current
   network and understand the measurement likely impact of new capacity and
   potential changes to the topology. Simulations - informed by data should only
   be sent when
   from a limited panel of probes - can help quantify the user is silent, advantage that
   a new technology brings and it shouldn't reduce support the
   customer's data allowance. The other approach business case for larger
   roll-out.

   It may also be possible to use probes to run stress tests for risk
   analysis. For example, an operator could run a carefully controlled
   and limited experiment in which probing is passive measurements
   on the customer's ordinary traffic; used to assess the advantage
   potential impact if some new application becomes popular.

3.4 Monitoring Service Level Agreements

   Another example is that it measures
   what the customer actually does, but it creates extra variability
   (different traffic mixes give different results) and operator may want to monitor performance
   where there is a service level agreement. This could be with its own
   customers, especially it
   raises privacy concerns.

   From enterprises may have an operator's viewpoint, understanding customers better enables
   it to offer better services. Also, simple metrics SLA. The operator can be more easily
   understood by senior managers who make investment decisions and by
   sales and marketing.

3.2 Understanding
   proactively spot when the impact and operation of new devices and technology

   Another type of measurement service is degrading near to test new capabilities the SLA limit,
   and services
   before they are rolled out. For example, get information that will enable more informed conversations with
   the customer at contract renewal.

   An operator may also want to:
   check whether a customer can be upgraded to a new broadband option;
   understand monitor the impact performance of IPv6 before it makes it available to its
   customers (will v6 packets get through, what will the latency be
   suppliers, to
   major websites, what transition mechanisms will be most
   appropriate?); check whether they meet their SLA or to compare two
   suppliers if it is dual-sourcing. This could include its transit
   operator, CDNs, peering, video source, local network provider (for a new capability can be signaled using
   TCP options (how often
   global operator in countries where it will be blocked by doesn't have its own network),
   even the whole network for a middlebox? virtual operator.

   Through a better understanding of its own network and its suppliers,
   the operator should be able to focus investment more effectively - along in
   the
   lines of some existing experiments) [Extend TCP]; investigate right place at the right time with the right technology.

3.5 Identifying, isolating and fixing network problems

   Operators can use large scale measurements to help identify a
   quality of service mechanism (e.g. checking whether Diffserv markings
   are respected on fault
   more rapidly and decide how to solve it.

   Operators already have Test and Diagnostic tools, where a network
   element reports some path); problem or failure to a management system.
   However, many issues are not caused by a point failure but something
   wider and so on.

3.3 Design will trigger too many alarms, whilst other issues will
   cause degradation rather than failure and planning

   Operators can use large scale so not trigger any alarm.
   Large-scale measurements to can help with their provide a more nuanced view that
   helps network
   planning - proactive activities management to improve the network.

   For example, by probing from several different vantage points identify and fix problems more rapidly
   and accurately. The network management tools may use simulations to
   emulate the network and so help identify a fault and assess possible
   solutions.

   An operator can see obtain useful information without measuring the
   performance on every broadband line. By measuring a subset, the
   operator identify problems that affect a particular group of customers has performance
   below that expected during peak hours, which should help capacity
   planning. Naturally operators already have tools to help this - a
   network element reports its individual utilization (and perhaps other
   parameters). However, making measurements across a path rather than customers. For
   example, the issue could be at a shared point may make it easier to understand in the network. There may
   also be parameters like bufferbloat that aren't currently reported by network topology
   (such as an exchange) or common to a vendor or equipment and/or that are intrinsically path metrics.

   With information gained from type
   [IETF85-Plenary] describes a case where a particular home gateway
   upgrade had caused a (mistaken!) drop in line rate. A more extensive
   deployment of the measurement results, capacity planning
   and network design capability to every broadband line
   would enable an operator to identify issues unique to a single
   customer. Overall, large-scale measurements can be help an operator help
   an operator fix the fault more effective. Such planning typically
   uses simulations to emulate rapidly and/or allow the measured performance of affected
   customers to be informed what's happening. More accurate information
   enables the current
   network operator to reassure customers and understand the likely impact of new capacity take more rapid and
   potential changes
   effective action to cure the topology. Simulations problem.

   Often customers experience poor broadband due to problems in the home
   network - informed by data the ISP's network is fine. For example they may have moved
   too far away from a limited panel their wireless access point. Perhaps 80% of probes - can help quantify the advantage that
   a new technology brings
   customer calls about fixed BB problems are due to in-home wireless
   issues. These issues are expensive and support the business case frustrating for larger
   roll-out.

   It may also be possible an operator,
   as they are extremely hard to use probes diagnose and solve. The operator would
   like to run stress tests for risk
   analysis. For example, narrow down whether the problem is in the home (with the home
   network or edge device or home gateway), in the operator's network,
   or with an over-the-top service. The operator could run a carefully controlled
   and limited experiment in which probing is used would like two
   capabilities. Firstly, self-help tools that customers use to assess the
   potential impact if some new application becomes popular.

3.4 Monitoring Service Level Agreements

   Another improve
   their own service or understand its performance better, for example is
   to re-position their devices for better WiFi coverage. Secondly, on-
   demand tests that can the operator may want to monitor performance
   where there is a service level agreement. This could be with its own
   customers, especially enterprises may have an SLA. The operator can
   proactively spot when run instantly - so the service is degrading near to call
   center person answering the SLA limit, phone (or e-chat) could trigger a test
   and get information that will enable more informed conversations with the result whilst the customer at contract renewal.

   An operator may also want is still in an on-line
   session.

4  Details of Regulator Use Case

4.1 Promoting competition through transparency

   Competition plays a vital role in regulation of the electronic
   communications markets. For competition to monitor successfully discipline
   operators' behavior in the performance interests of its
   suppliers, their customers, end users
   must be fully aware of the characteristics of the ISPs' access
   offers. In some jurisdictions regulators mandate transparent
   information made available about service offers.

   End users need effective transparency to check whether they meet be able to make informed
   choices throughout the different stages of their SLA relationship with
   ISPs, when selecting Internet access service offers, and when
   considering switching service offer within an ISP or to compare two
   suppliers if it is dual-sourcing. This an
   alternative ISP. Quality information about service offers could
   include its transit
   operator, CDNs, peering, video source, local network provider (for a
   global operator in countries where it doesn't have its own network),
   even the whole network for a virtual operator.

   Through a better understanding speed, delay, and jitter. Regulators can publish such
   information to facilitate end users' choice of its own network service provider and its suppliers,
   offer. It may also encourage ISPs to use the operator should be able same metrics in their
   service level contracts, which would further help end users to focus investment more effectively choose
   an ISP. Finally, transparency may help content, application, service
   and device providers develop their Internet offerings.

   The published information needs to be:

      o  Accurate - in
   the right place at the right time with the right technology.

3.5 Identifying, isolating measurement results must be correct and fixing network problems

   Operators not
      influenced by errors or side effects. The results should be
      reproducible and consistent over time.

      o  Comparable - common metrics should be used across different
      ISPs and service offerings so that measurement results can use large scale be
      compared.

      o  Meaningful - the metrics used for measurements need to help identify a fault
   more rapidly reflect
      what end users value about their broadband Internet access service

      o  Reliable - the number and decide how to solve it.

   Operators already have Test distribution of measurement agents,
      and Diagnostic tools, where a network
   element reports some problem or failure the statistical processing of the raw measurement data, needs
      to a management system.
   However, many issues be appropriate

   A set of measurement parameters and associated measurement methods
   are not caused by a point failure but something
   wider used over time, e.g. speed, delay, and so will trigger too many alarms, whilst other issues will
   cause degradation rather than failure jitter. Then the
   measurement raw data are collected and so not trigger any alarm.
   Large scale measurements can help provide a more nuance view that
   helps network management go through statistical post-
   processing before the results can be published in an Internet access
   service quality index to identify and fix problems more rapidly facilitate end users' choice of service
   provider and accurately. offer.

   The network management tools may use simulations regulator can also promote competition through transparency by
   encouraging end users to
   emulate monitor the network and so help identify a fault and assess possible
   solutions.

   One example was described in [IETF85-Plenary]. The operator was
   running a measurement panel for reasons discussed in sub performance of their own
   broadband Internet access service. They might use case #1.
   It was noticed this information to
   check that the performance of some lines had unexpectedly
   degraded. This led to a detailed (off-line) investigation which
   discovered meets that a particular home gateway upgrade had caused a
   (mistaken!) drop specified in line rate.

   Another example is that occasionally some internal network management
   event (like re-routing) can be customer-affecting (of course this their contract or
   to understand whether their current subscription is
   unusual). This affects a whole group of customers, for instance those
   on the same DSLAM. Understanding this will help most
   appropriate.

4.2 Promoting broadband deployment

   Governments sometimes set strategic goals for high-speed broadband
   penetration as an operator fix important component of the
   fault more rapidly and/or allow economic, cultural and
   social development of the affected customers to be informed
   what's happening and/or request them to re-set their home hub
   (required to cure some conditions). More accurate information enables society. To evaluate the operator to reassure customers and take more rapid and effective
   action to cure effect of the problem.

   There may also
   stimulated growth over time, broadband Internet access take-up and
   penetration of high-speed access can be problems unique to a single user line (e.g. copper
   access) that need monitored through measurement
   campaigns.

   An example of such an initiative is the "Digital Agenda for Europe"
   which was adopted in 2010, to be identified.

   Often customers experience poor achieve universal broadband due to problems in the home
   network - the ISP's network access. The
   goal is fine. For example they may have moved
   too far away from their wireless to achieve by 2020, access point. Perhaps 80% for all Europeans to Internet
   access speeds of
   customer calls about fixed BB problems are due 30 Mbps or above, and 50% or more of European
   households subscribing to in-home wireless
   issues. These issues Internet connections above 100 Mbps.

   To monitor actual broadband Internet access performance in a specific
   country or a region, extensive measurement campaigns are expensive needed. A
   panel can be built based on operators and frustrating for an operator,
   as they are extremely hard to diagnose packages in the market,
   spread over urban, suburban and solve. The operator would
   like rural areas. Probes can then be
   distributed to narrow down whether the problem is in participants of the home (with campaign.

   Periodic tests running on the home
   network or edge device or home gateway), in probes can for example measure actual
   speed at peak and off-peak hours, but also other detailed quality
   metrics like delay and jitter. Collected data goes afterwards through
   statistical analysis, deriving estimates for the operator's network, whole population
   which can then be presented and published regularly.

   Using a harmonized or with an over-the-top service. The operator would like two
   capabilities. Firstly, self-help tools that customers use standardized measurement methodology, or even a
   common quality measurement platform, measurement results could also
   be used for benchmarking of providers and/or countries.

4.3 Monitoring "net neutrality"

   Regulatory approaches related to improve
   their own service or understand its performance better, net neutrality and the open Internet
   has been introduced in some jurisdictions. Examples of such efforts
   are the Internet policy as outlined by the Body of European
   Regulators for example
   to re-position their devices Electronic Communications Guidelines for better WiFi coverage. Secondly, on-
   demand tests that can quality of
   service [BEREC Guidelines] and US FCC Preserving the operator Open Internet
   Report and Order [FCC R&O].  Although legal challenges can run instantly - so change the call
   center person answering
   status of policy such as the phone (or e-chat) could trigger a test
   and get court action negating the result whilst FCC R&O, the customer
   take-away for LMAP purposes is still in an on-line
   session.

4  Details of Regulator Use Case

4.1 Promoting competition through transparency

   Competition plays a vital role in regulation of the electronic
   communications markets. For competition that policy-makers are looking for
   measurement solutions to successfully discipline
   operators' behavior assist them in the interests of their customers, end users
   must be fully aware of the characteristics discovering biased treatment
   of traffic flows.  The exact definitions and requirements vary from
   one jurisdiction to another; the ISPs' access
   offers. In comments below provide some jurisdictions regulators mandate transparent
   information made available hints
   about service offers.

   End users need effective transparency to be able to make informed
   choices throughout the different stages potential role of their relationship with
   ISPs, when selecting measurements.

   Net neutrality regulations do not necessarily require every packet to
   be treated equally. Typically they allow "reasonable" traffic
   management (for example if there is exceptional congestion) and allow
   "specialized services" in parallel to, but separate from, ordinary
   Internet access service offers, and when
   considering switching service offer within an ISP or (for example for facilities-based IPTV). A regulator
   may want to an
   alternative ISP. Quality information about service offers could
   include speed, delay, and jitter. Regulators can publish monitor such
   information to facilitate end users' choice of service provider and
   offer. It may also help content, application, service and device
   providers develop their Internet offerings.

   The published information needs practices as input to be:

      o  Accurate - the measurement results must be correct and not
      influenced by errors or side effects. The results should be
      reproducible regulatory
   evaluation. However, these concepts are evolving and consistent over time.

      o  Comparable - common metrics should be used differ across different
      ISPs and service offerings
   jurisdictions, so that measurement results can should be
      compared.

      o  Meaningful - the metrics used for measurements need to reflect
      what end users value about their broadband Internet access service

      o  Reliable - the number and distribution assessed with
   caution.

   A regulator could monitor departures from application agnosticism
   such as blocking or throttling of measurement agents, traffic from specific applications,
   and the statistical processing preferential treatment of the raw measurement raw data,
      needs to be appropriate specific applications. A set of measurement parameters and associated measurement methods
   are used over time, e.g. speed, delay,
   system could send, or passively monitor, application-specific traffic
   and jitter. Then then measure in detail the
   measurement raw data are collected and go through statistical post-
   processing before transfer of the results can be published in an Internet access
   service quality index different packets.
   Whilst it is relatively easy to measure port blocking, it is a
   research topic how to facilitate end users' choice detect other types of service
   provider and offer. differentiated treatment.
    The paper, "Glasnost: Enabling End Users to Detect Traffic
   Differentiation" [M-Labs NSDI 2010] and follow-on tool "Glasnost"
   [Glasnost] are examples of work in this area.

   A regulator can could also promote competition through transparency by
   encouraging end users to monitor the performance of their own
   broadband Internet access service. They might use this information to
   check that the performance meets that specified in their contract or
   to understand whether their current subscription is the most
   appropriate.

4.2 Promoting broadband deployment

   Governments sometimes set strategic goals for high-speed broadband
   penetration as an important component of the economic, cultural
   service over time, to try and
   social development of detect if the society. To evaluate specialized service is
   provided at the effect expense of the
   stimulated growth over time, broadband Internet access take-up and
   penetration of high-speed access can service. Comparison
   between ISPs or between different countries may also be monitored through measurement
   campaigns.

   An example of such an initiative is the "Digital Agenda for Europe"
   which was adopted in 2010, to achieve universal broadband access. The
   goal is to achieve by 2020, access relevant for all Europeans to Internet
   access speeds of 30 Mbps or above, and 50% or more
   this kind of European
   households subscribing to Internet connections above 100 Mbps.

   To monitor actual broadband Internet access performance in a specific
   country or a region, extensive measurement campaigns evaluation.

5  Implementation Options

   There are needed. A
   panel can be built based on operators and packages in the market,
   spread over urban, suburban and rural areas. Probes can then several ways of implementing a measurement system. The
   choice may be
   distributed to influenced by the participants details of the campaign.

   Periodic tests running on the probes can for example measure actual
   speed at peak and off-peak hours, but also other detailed quality
   metrics like delay particular use case
   and jitter. Collected data goes afterwards through
   statistical analysis, deriving estimates what the most important criteria are for the whole population
   which can then be presented and published regularly.

   Using a harmonized regulator, ISP or standardized
   third party operating the measurement methodology, or even system.

   One way involves a
   common quality measurement platform, measurement results could also
   be used for benchmarking of providers and/or countries.

4.3 Monitoring "net neutrality"

   Regulatory approaches related special hardware device that is connected directly
   to net neutrality the home gateway. The devices are deployed to a carefully selected
   panel of end users and they perform measurements according to a
   defined schedule. The schedule can run throughout the open Internet
   has been introduced in some jurisdictions. Examples day, to allow
   continuous assessment of such efforts
   are the Internet policy as outlined network. Careful design ensures that
   measurements do not detrimentally impact the home user experience or
   corrupt the results by testing when the Body user is also using the
   broadband line. The system is therefore tightly controlled by the
   operator of European
   Regulators for Electronic Communications Guidelines for quality the measurement system. One advantage of
   service [BEREC Guidelines] and US FCC Preserving this approach is
   that it is possible to get reliable benchmarks for the Open Internet
   Report and Order [FCC R&O].  Although legal challenges can change performance of
   a network with only a few devices. One disadvantage is that it would
   be expensive to deploy hardware devices on a mass scale sufficient to
   understand the
   status performance of policy such as the court action negating network at the FCC R&O, granularity of a
   single broadband user.

   Another approach involves implementing the
   take away for LMAP purposes are measurement capability as
   a webpage or an "app" that policy-makers end users are looking for
   measurement solutions to assist them in discovering biased treatment
   of traffic flows.  The exact definitions and requirements vary from
   one jurisdiction encouraged to another; download onto
   their mobile phone or computing device. Measurements are triggered by
   the comments below provide some hints
   about end user, for example the potential role of measurements.

   Net neutrality regulations do not necessarily require every packet user interface may have a button to
   be treated equally. Typically they allow "reasonable" traffic
   management (for example if there
   "test my broadband now". One advantage of this approach is exceptional congestion) and allow
   "specialized services" in parallel to, but separate from, ordinary
   Internet access (for example for facilities-based IPTV). A regulator
   may want that the
   performance is measured to monitor such practices as input the end user, rather than to the regulatory
   evaluation. However, these concepts are evolving home
   gateway, and differ across
   jurisdictions, so measurement results should be assessed with
   caution.

   A regulator could monitor departures from application agnosticism
   such includes the home network. Another difference is that
   the system is much more loosely controlled, as blocking or throttling of traffic from specific applications,
   and preferential treatment the panel of specific applications. A measurement
   system could send, or passively monitor, application-specific traffic end users
   and then measure in detail the transfer schedule of tests are determined by the different packets.
   Whilst it is relatively easy end users themselves
   rather than the measurement system. It would be easier to measure port blocking, get large-
   scale, however it is a
   research topic how to detect other types of differentiated treatment.
    The paper, "Glasnost: Enabling End Users harder to Detect Traffic
   Differentiation" [M-Labs NSDI 2010] and follow-on tool "Glasnost"

   [Glasnost] get comparable benchmarks as the
   measurements are examples of work in this area.

   A regulator could affected by the home network and also monitor the performance population
   is self-selecting and so potentially biased towards those who think
   they have a problem. This could be alleviated by stimulating
   widespread downloading of the broadband
   service over time, to try app and detect if careful post-processing of the specialized service is
   provided at
   results to reduce biases.

   There are several other possibilities. For example, as a variant on
   the expense of first approach, the Internet access service. Comparison
   between ISPs or between different countries may also measurement capability could be relevant for
   this kind of evaluation.

5 implemented
   as software embedded in the home gateway, which would make it more
   viable to have the capability on every user line. As a variant on the
   second approach, the end user could initiate measurements in response
   to a request from the measurement system.

6  Conclusions

   Large-scale measurements of broadband performance are useful for both
   network operators and regulators. Network operators would like to use
   measurements to help them better understand the quality experienced
   by their customers, identify problems in the network and design
   network improvements. Regulators would like to use measurements to
   help promote competition between network operators, stimulate the
   growth of broadband access and monitor 'net neutrality'. There are
   other use cases that are not the focus of the initial LMAP charter
   (although it is expected that the mechanisms developed would be
   readily applied), for example end users would like to use
   measurements to help identify problems in their home network and to
   monitor the performance of their broadband provider.

   From consideration of the various use cases, several common themes
   emerge whilst there are also some detailed differences. These
   characteristics guide the development of LMAP's framework,
   information model and protocol.

   A measurement capability is needed across a wide number of
   heterogeneous environments. Tests may be needed in the home network,
   in the ISP's network or beyond; they may be measuring a fixed or
   wireless network; they may measure just the access network or across
   several networks; at least some of which are not operated by the
   measurement provider.

   There is a role for both standardized and non-standardized
   measurements. For example, a regulator would like to publish
   standardized performance metrics for all network operators, whilst an
   ISP may need their own tests to understand some feature special to
   their network. Most use cases need active measurements, which create
   and measure specific test traffic, but some need passive measurements
   of the end user's traffic.

   Regardless of the tests being operated, there needs to be a way to
   demand or schedule the tests. Most use cases need a regular schedule
   of measurements, but sometimes ad hoc testing is needed, for example
   for troubleshooting. It needs to be ensured that measurements do not
   affect the user experience and are not affected by user traffic
   (unless desired). In addition there needs to be a common way to
   collect the results. Standardization of this control and reporting
   functionality allows the operator of a measurement system to buy the
   various components from different vendors.

   After the measurement results are collected, they need to be
   understood and analyzed. Often it is sufficient to measure only a
   small subset of end users, but per-line fault diagnosis requires the
   ability to test every individual line. Analysis requires accurate
   definition and understanding of where the test points are, as well as
   contextual information about the topology, line, product and the
   subscriber's contract. The actual analysis of results is beyond the
   scope of LMAP, as is the key challenge of how to integrate the
   measurement system into a network operator's existing tools for
   diagnostics and network planning.

   Finally the test data, along with any associated network, product or
   subscriber contract data is commercial or private information and
   needs to be protected.

6

7  Security Considerations

   This informational document provides an overview of the use cases for
   LMAP and so does not, in itself, raise any security issues.

   The framework document [framework] discusses the potential security,
   privacy (data protection) and business sensitivity issues that LMAP
   raises. The main threats are:

      1. a malicious party that gains control of Measurement Agents to
      launch DoS attacks at a target, or to alter (perhaps subtly)
      Measurement Tasks in order to compromise the end user's privacy,
      the business confidentiality of the network, or the accuracy of
      the measurement system.

      2. a malicious party that gains control of Measurement Agents to
      create a platform for pervasive monitoring [RFC7258], in order to
      attack the privacy of Internet users and organisations.

      3. a malicious party that intercepts or corrupts the Measurement
      Results &/or other information about the Subscriber, for similar
      nefarious purposes.

      3.

      4. a malicious party that uses fingerprinting techniques to
      identify individual end users, even from anonymized data

      4.

      5. a measurement system that does not obtain the end user's
      informed consent, or fails to specify a specific purpose in the
      consent, or uses the collected information for secondary uses
      beyond those specified.

      5.

      6. a measurement system that is vague about who is responsible for
      privacy (data protection); this role is often termed the "data
      controller".

   The [framework] also considers some potential mitigations of these
   issues. They will need to be considered by an LMAP protocol and more
   generally by any measurement system.

7

8  IANA Considerations

   None

Contributors

   The information in this document is partially derived from text
   written by the following contributors:

   James Miller		jamesmilleresquire@gmail.com

   Rachel Huang		rachel.huang@huawei.com

Informative References

   [IETF85-Plenary] Crawford, S., "Large-Scale Active Measurement of
              Broadband Networks",
              http://www.ietf.org/proceedings/85/slides/slides-85-iesg-
              opsandtech-7.pdf 'example' from slide 18

   [Extend TCP] Michio Honda, Yoshifumi Nishida, Costin Raiciu, Adam
              Greenhalgh, Mark Handley and Hideyuki Tokuda. "Is it Still
              Possible to Extend TCP?" Proc. ACM Internet Measurement
              Conference (IMC), November 2011, Berlin, Germany.
              http://www.ietf.org/proceedings/82/slides/IRTF-1.pdf

   [framework] Eardley, P., Morton, A., Bagnulo, M., Burbridge, T.,
              Aitken, P., Akhter, A.  "A framework for large-scale
              measurement platforms (LMAP)",
              http://datatracker.ietf.org/doc/draft-ietf-lmap-framework/

   [RFC6973]  Cooper, A., Tschofenig, H.z., Aboba, B., Peterson, J.,
              Morris, J., Hansen, M., and R. Smith, "Privacy
              Considerations for Internet Protocols", RFC 6973, July
              2013.

   [RFC7258]  Farrell, S., Tschofenig, H., "PPervasive Monitoring Is an
              Attack", RFC 7258, May 2014.

   [FCC R&O]  United States Federal Communications Commission, 10-201,
              "Preserving the Open Internet, Broadband Industries
              Practices, Report and Order",
              http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-10-
              201A1.pdf

   [BEREC Guidelines] Body of European Regulators for Electronic
              Communications, "BEREC Guidelines for quality of service
              in the scope of net neutrality",
              http://berec.europa.eu/eng/document_register/
              subject_matter/berec/download/0/1101-berec-guidelines-for-
              quality-of-service-_0.pdf

   [M-Labs NSDI 2010] M-Lab, "Glasnost: Enabling End Users to Detect
              Traffic Differentiation",
              http://www.measurementlab.net/download/AMIfv945ljiJXzG-
              fgUrZSTu2hs1xRl5Oh-rpGQMWL305BNQh-
              BSq5oBoYU4a7zqXOvrztpJhK9gwk5unOe-fOzj4X-vOQz_HRrnYU-
              aFd0rv332RDReRfOYkJuagysstN3GZ__lQHTS8_UHJTWkrwyqIUjffVeDxQ/

   [Glasnost] M-Lab tool "Glasnost", http://mlab-live.appspot.com/tools/
              glasnost

Authors' Addresses

              Marc Linsner
              Cisco Systems, Inc.
              Marco Island, FL
              USA

              EMail: mlinsner@cisco.com

              Philip Eardley
              BT
              B54 Room 77, Adastral Park, Martlesham
              Ipswich, IP5 3RE
              UK

              Email: philip.eardley@bt.com

              Trevor Burbridge
              BT
              B54 Room 77, Adastral Park, Martlesham
              Ipswich, IP5 3RE
              UK

              Email: trevor.burbridge@bt.com

              Frode Sorensen
              Norwegian Post and Telecommunications Authority (NPT)
              Lillesand
              Norway

              Email: frode.sorensen@npt.no