[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00 01

IPFIX Working Group                                         A. Kobayashi
Internet-Draft                                                H. Nishida
Intended status: Informational                               NTT PF Lab.
Expires: April 18, 2008                                        C. Sommer
                                                             F. Dressler
                                                          Univ. Erlangen
                                                              E. Stephan
                                                          France Telecom
                                                        October 16, 2007

         Problems with Flow Collection in Large-Scale Networks

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on April 18, 2008.

Copyright Notice

   Copyright (C) The IETF Trust (2007).

Kobayashi, et al.        Expires April 18, 2008                 [Page 1]

Internet-Draft        Problem Statement in large NW         October 2007


   Flow measurement techniques have been developed in order to cope with
   increasing bandwiths.  Flow measurement is currently being used as a
   popular method for monitoring large networks, e.g. at Internet
   Service Providers or multinational corporations, for accounting,
   management, and security purposes.  To construct the measurement
   system for such networks, the biggest challenge is scalability.
   Whereas packet sampling functions in current flow measurement
   solutions such as NetFlow/sFlow/IPFIX help to approach this issue,
   some problems still remain.  This document describes such open
   issues, which a flow-based measurement system might encounter in
   large-scale networks.  The results are intended to be used to define
   and to develope an IPFIX mediator device.

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Problem Statement  . . . . . . . . . . . . . . . . . . . . . .  4
     2.1.  Traffic Measurement in Service Provider Networks . . . . .  4
     2.2.  Flow-Based Measurement in Integrated Networks  . . . . . .  4
     2.3.  Approaches to Scalability  . . . . . . . . . . . . . . . .  5
       2.3.1.  Allocating Multiple Collectors in Parallel . . . . . .  5
       2.3.2.  Adjusting Sampling Rates . . . . . . . . . . . . . . .  6
       2.3.3.  Exporting Aggregated Flows from Original Exporters . .  6
       2.3.4.  Using a Hierarchy of IPFIX Concentrators . . . . . . .  6
   3.  Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . .  8
   4.  Security Considerations  . . . . . . . . . . . . . . . . . . .  9
   5.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 10
   6.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
     6.1.  Normative References . . . . . . . . . . . . . . . . . . . 11
     6.2.  Informative References . . . . . . . . . . . . . . . . . . 11
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12
   Intellectual Property and Copyright Statements . . . . . . . . . . 14

Kobayashi, et al.        Expires April 18, 2008                 [Page 2]

Internet-Draft        Problem Statement in large NW         October 2007

1.  Introduction

   Flow-based measurement has dramatically improved the quality of
   traffic measurement methods available for service provider networks.
   It is quickly becoming one of the essential operation tools in many
   networks.  Especially monitoring operations in large-scale networks
   need flow measurement techniques to ensure stable and efficient
   operations.  However, network providers might soon encounter
   problematic situations because of scalability issues.  With traffic
   volume ever increasing in large networks, the number of exported Flow
   Records becomes huge.  Also, Flow information is required for
   multiple purposes and it is difficult to maintain the scalability of
   a flow-based measurement system and at the same time offer Flows in
   the way required for purposes such as Traffic Engineering,
   Troubleshooting, Accounting, and Security.  This document provides
   the problem description and, thus, the requirements for an extension
   to IPFIX, the IPFIX mediator.

   The IPFIX applicability draft [I-D.ietf-ipfix-as] describes a variety
   of applications for the IPFIX protocol [I-D.ietf-ipfix-protocol].  In
   most networks, the flexibility of the IPFIX protocol could meet the
   requirements of a flow-based measurement system.  But, from the
   viewpoint of large-scale and fast-growing networks, there are still
   some unresolved problems.  This document describes such problematic
   cases in large-scale networks.

Kobayashi, et al.        Expires April 18, 2008                 [Page 3]

Internet-Draft        Problem Statement in large NW         October 2007

2.  Problem Statement

2.1.  Traffic Measurement in Service Provider Networks

   Flow measurement methods are one of the most popular measurement
   methods of network service providers.  Usually, an operator measures
   traffic on several observation points for a specific purpose, using a
   certain level of sampling rate such as from 1/10,000 to 1/1,000.
   This value depends on the several factors, such as the capacity of
   the management network, the available storage and speed of the
   Collector, or the load of the routers/switches.  This section
   describes some of use cases of flow-based measurement on the service
   provider networks.

   o  In gateway routers, exchange traffic between neighboring AS is
      monitored using flow-based measurement.  The result helps with the
      planing of peering policies and with traffic engineering to
      optimize the network resources.

   o  In server side routers, the traffic toward a specific customer's
      server is measured.  The collected data is used to detect traffic
      anomalies, such as distributed denial-of-service attacks and
      packet scans.  Furthermore, this data is valuable for network
      forensics, when customers ask to investigate their server traffic.

   o  In access routers connected toward the broadband customers, the
      traffic of each customer is measured to profile them.  Operators
      pay special attention to heavy users that abuse network bandwidth
      by using P2P file exchange applications.  Results give good
      feedback about the implemented billing policy.

   As described above, the network providers are making heavy use of
   Flow measurements.  The number of Observation Points in the networks
   can even be increased to improve on the effectiveness of these
   methods.  In the near future, we anticipate that the advanced
   features of IPFIX, such as monitoring wide-area traffic exchange
   behavior and QoS performance will accelerate the utilization.  The
   increasing amount of the traffic that is brought about by broadband
   users might have a impact on the present measurement parameters, such
   as the sampling rate or the granularity of Flows.

2.2.  Flow-Based Measurement in Integrated Networks

   Recently several networks seem to shift towards Integrated Networks,
   such as the Internet and MPLS, which include IPv4, IPv6 and VPN
   traffic.  These sorts of Flow Records need to be analyzed separately
   and from different perspectives.  However, handling them separately
   without improving the capability of the Collector is difficult.  If

Kobayashi, et al.        Expires April 18, 2008                 [Page 4]

Internet-Draft        Problem Statement in large NW         October 2007

   the original Exporter can not distribute specific Flow Records based
   on the Flow contents, a single Collector needs to be able to handle
   several kinds of Flow Records.  Except for the original Exporter,
   other devices assisting in this function would thus be necessary.

2.3.  Approaches to Scalability

   The volume of traffic on ordinary service provider networks has been
   increasing year by year.  As the sizes of networks become larger, the
   amount of Flow Records becomes greater.  The huge amount of Flow
   Records has been burdening management networks and Collecting
   Processes.  Maintaining scalability is difficult as a particular
   network grows.

   Generally, large-scale networks already have multiple 10Gb/s links,
   their total traffic exceeding 100Gb/s.  In the near future, broadband
   users' traffic will increase by approximately 50% per year according
   to [TRAFGRW].  When operators monitor traffic of 500Gb/s with a
   sampling rate of 1/1000, the amount of exported Flow Records from
   Exporters could exceed 50k f/s.  This value reaches beyond the
   ability of a single Collector.  It should be noted that the current
   sampling rate might not be able to be applied to the Exporter within
   the next five years.  We consider the problem to be expected to
   encounter within the next five years in large-scale networks.

   In addition, network operators need to monitor the overall wide-scale
   traffic exchange, and investigate detailed traffic behavior when
   traffic incidents happen.  Achieving this seems to be difficult for a
   single Collector because of the huge amount of Flow Records.

   To that end, the network operators can consider several solutions.
   The next sections describe how operators can cope with such a huge
   amount of Flow Records using available solutions of NetFlow/sFlow/
   IPFIX.  However, each solution also presents new problems.

2.3.1.  Allocating Multiple Collectors in Parallel

   Generally speaking, allocating multiple Collectors to cope with the
   amount of Flow Records can resolve many scalability problems.  But,
   in that case, network operators will have difficulties to get an
   overview of traffic exchange behavior in whole network.  Most of
   Flow-based applications require the observation of flows from the
   whole network under observation.  For example, they correspond to the
   wide-area traffic matrices that are the router-by-router or PoP-by-
   PoP traffic matrix.  To get the wide-area traffic matrices, single
   Collector device needs to gather the Flow Records from the whole

Kobayashi, et al.        Expires April 18, 2008                 [Page 5]

Internet-Draft        Problem Statement in large NW         October 2007

2.3.2.  Adjusting Sampling Rates

   Adjusting the sampling rate definitely reduces the amount of Flow
   Records, and a flow-based measurement system can thus easily adapt to
   the ability of the Collecting and Exporting Processes.  But, in that
   case, Flows with small traffic volume could easily get lost.  When
   the traffic incidents happen, network operators can no longer
   investigate traffic behavior.  While traffic volume on networks
   continues to increase, network operators will not be able to maintain
   the sampling rates currently used.  In the near future, it is
   possible that flow-based measurement systems will not be able to
   detect traffic anomalies which can currently be detected.

2.3.3.  Exporting Aggregated Flows from Original Exporters

   When using Flow aggregation, the Flow Records are generally
   aggregated based on network prefix, peering AS (autonomous system)
   number, or BGP Next-Hop.  This solution is especially useful for
   measurements of wide-area traffic exchange.  However, network
   operators can no longer monitor traffic behavior in-depth, the
   simplest type of Flows being those comprised of all packets having
   the same 5-tuple of protocol, source and destination IP addresses and
   port numbers.

   As one of solutions that resolve this problem, if the Exporter allows
   aggregation with multiple granularities, network operators can
   flexibly adapt the granularity of aggregation according to their
   requirements.  An Exporter might, for example, by default export
   aggregated Flow Records.  If the traffic incident happens, network
   operators could then change the Flow Key to measure fine-grained
   Flows.  To implement this solution, a system that dynamically changes
   the Flow Key in cooperation with Exporters and Collectors is
   required.  It is, however, difficult to replace all Exporters in the
   whole network, as all Exporters in the network will need to implement
   this solution.

2.3.4.  Using a Hierarchy of IPFIX Concentrators

   In large-scale networks, creating a hierarchical measurement system
   by using IPFIX Concentrators, as described in [RFC3917], can prove to
   be very useful.  If IPFIX Concentrators store the received Flow
   Records, and then the stored Flow Records are allowed to be retrieved
   by other devices, this architecture might actually become a most
   useful distributed collection system.  As described in
   [I-D.dressler-ipfix-aggregation], in the case of a measurement system
   consisting of both aggregating Exporters and non-aggregating
   Exporters, an IPFIX Concentrator that can assist the latter by
   aggregating received Flow Records to any granularity.

Kobayashi, et al.        Expires April 18, 2008                 [Page 6]

Internet-Draft        Problem Statement in large NW         October 2007

   However, IPFIX header information, such as Exporter IP address,
   Observation Domain ID and Export Time would likely be lost in the
   process of mediation that is performed by the Concentrator.  Exporter
   IP address and Observation Domain ID indicate the observation point
   from the viewpoint of entire network domain.  Export Time indicates
   the reference of time for flow information.  Such information is
   necessary for guaranteeing the continuity of the work of the top
   level Collector.  Additionally in some cases the information that is
   reported by Option Templates, such as the sampling rate, could be
   lost.  It depends on the implementation of the IPFIX Concentrator.
   It should be noted that the information communicated by IPFIX
   Concentrator needs to be defined.

Kobayashi, et al.        Expires April 18, 2008                 [Page 7]

Internet-Draft        Problem Statement in large NW         October 2007

3.  Conclusion

   This document has covered the following problems related to flow-
   based measurement systems in large-scale networks.

   o  Adjusting the sampling rate and granularity of Flow causes small
      Flows to become invisible.  Large-scale networks need another
      solution that can handle the huge amounts of Flow Records when
      using current sampling rates, which range from 1/10,000 to 1/1000.

   o  Using IPFIX Concentrators might lose data that should be
      communicated by the original Exporter.

   o  In Integrated Networks, which mix MPLS VPN with other tunnel
      traffic and IPv4/IPv6, flow-based measurement might not work well.

   These problems have their roots in constructing scalable flow-based
   measurement systems.  Although network providers can resolve these
   problems by utilizing advanced features of Exporters and Collectors,
   it is difficult to perform an atomic migration of the whole
   measurement system while the amount of traffic grows.  To assist the
   ability of the Exporters and the Collectors, it should be noted that
   there are some IPFIX devices for the network providers to select.
   They correspond to Flow mediation devices.  Especially standardized
   IPFIX Concentrator could have been useful devices.

Kobayashi, et al.        Expires April 18, 2008                 [Page 8]

Internet-Draft        Problem Statement in large NW         October 2007

4.  Security Considerations

   A flow-based measurement system might lead to privacy violation
   problems, such as the export of flows to an unexpected address, if
   the system is not confined to the large-scale network under
   observation.  General security issues of the IPFIX protocol are
   covered by the security considerations section in
   [I-D.ietf-ipfix-protocol].  Security MUST be considered, if different
   networks exchange the flow information.  As the security of the
   exchange relies mostly on the protocol used, UDP does not look
   appropriate to exchange information between networks.

Kobayashi, et al.        Expires April 18, 2008                 [Page 9]

Internet-Draft        Problem Statement in large NW         October 2007

5.  IANA Considerations

   This document has no actions for IANA.

Kobayashi, et al.        Expires April 18, 2008                [Page 10]

Internet-Draft        Problem Statement in large NW         October 2007

6.  References

6.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

6.2.  Informative References

              Dressler, F., Sommer, C., and G. Munz, "IPFIX
              Aggregation", draft-dressler-ipfix-aggregation-03.txt
              (work in progress) , June 2006.

              Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IPFIX
              Applicability", draft-dressler-ipfix-as-10.txt (work in
              progress) , June 2006.

              Claise, B., "IPFIX Protocol Specification",
              draft-ietf-ipfix-protocol-23 (work in progress) ,
              October 2006.

   [RFC3917]  Quittek, J., Zseby, T., Claise, B., and S. Zander,
              "Requirements for IP Flow Information Export(IPFIX)",
              October 2004.

   [TRAFGRW]  Cho, K., Fukuda, K., Esaki, H., and A. Kato, "The Impact
              and Implications of the Growth in Residential User-to-User
              Traffic", SIGCOMM2006, pp. 207-218, Pisa, Italy, September
              2006. , October 2006.

Kobayashi, et al.        Expires April 18, 2008                [Page 11]

Internet-Draft        Problem Statement in large NW         October 2007

Authors' Addresses

   Atsushi Kobayashi
   NTT Information Sharing Platform Laboratories
   3-9-11 Midori-cho
   Musashino-shi, Tokyo  180-8585

   Phone: +81-422-59-3978
   Email: akoba@nttv6.net

   Haruhiko Nishida
   NTT Information Sharing Platform Laboratories
   3-9-11 Midori-cho
   Musashino-shi, Tokyo  180-8585

   Phone: +81-422-59-3978
   Email: nishida.haruhiko@lab.ntt.co.jp

   Christoph Sommer
   University of Erlangen-Nuremberg
   Department of Computer Science 7
   Martensstr. 3
   Erlangen  91058

   Phone: +49 9131 85-27993
   Email: christoph.sommer@informatik.uni-erlangen.de
   URI:   http://www7.informatik.uni-erlangen.de/~sommer/

   Falko Dressler
   University of Erlangen-Nuremberg
   Department of Computer Science 7
   Martensstr. 3
   Erlangen  91058

   Phone: +49 9131 85-27914
   Email: dressler@informatik.uni-erlangen.de
   URI:   http://www7.informatik.uni-erlangen.de/~dressler/

Kobayashi, et al.        Expires April 18, 2008                [Page 12]

Internet-Draft        Problem Statement in large NW         October 2007

   Emile Stephan
   France Telecom Division R&D
   2 avenue Pierre Marzin
   Lannion  F-22307

   Phone: +33 2 96 05 18 52
   Email: emile.stephan@orange-ftgroup.com

Kobayashi, et al.        Expires April 18, 2008                [Page 13]

Internet-Draft        Problem Statement in large NW         October 2007

Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at


   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).

Kobayashi, et al.        Expires April 18, 2008                [Page 14]

Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/