[Docs] [txt|pdf|xml|html] [Tracker] [Email] [Nits]

Versions: 00

IPFIX Working Group                                          B. Trammell
Internet-Draft                                                 E. Boschi
Intended status: Experimental                                 ETH Zurich
Expires: January 6, 2011                                       A. Wagner
                                                             Consecom AG
                                                            July 5, 2010


  Exporting Aggregated Flow Data using the IP Flow Information Export
                            (IPFIX) Protocol
                    draft-trammell-ipfix-a8n-00.txt

Abstract

   This document describes the export of aggregated Flow information
   using IPFIX.  An aggregated Flow is essentially an IPFIX Flow with an
   externally imposed time interval, generally representing packets from
   one or more original Flows.  The document describes aggregated Flow
   export within the framework of IPFIX Mediators, defines an
   interoperable, implementation-independent method for aggregated Flow
   export, covering time interval export, counter distribution and
   export, and counting of original Flows.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 6, 2011.

Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of



Trammell, et al.         Expires January 6, 2011                [Page 1]


Internet-Draft              IPFIX Aggregation                  July 2010


   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  3
   3.  Requirements for Aggregation Support in IPFIX  . . . . . . . .  4
   4.  Aggregation of IP Flows  . . . . . . . . . . . . . . . . . . .  5
     4.1.  A general model for IP Flow Aggregation  . . . . . . . . .  5
     4.2.  Aggregating and Distributing Counters  . . . . . . . . . .  5
     4.3.  Counting Original Flows  . . . . . . . . . . . . . . . . .  6
   5.  Aggregation in the IPFIX Architecture  . . . . . . . . . . . .  7
   6.  Export of Aggregated IP Flows using IPFIX  . . . . . . . . . .  8
     6.1.  Flow Count Export  . . . . . . . . . . . . . . . . . . . .  8
       6.1.1.  originalFlowsPresent Information Element . . . . . . .  9
       6.1.2.  originalFlowsInitiated InformationElement  . . . . . .  9
       6.1.3.  originalFlowsCompleted InformationElement  . . . . . .  9
       6.1.4.  originalFlows InformationElement . . . . . . . . . . .  9
     6.2.  Aggregate Counter Distibution Export . . . . . . . . . . . 10
     6.3.  Time Interval Export . . . . . . . . . . . . . . . . . . . 10
   7.  Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
   8.  Security Considerations  . . . . . . . . . . . . . . . . . . . 10
   9.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 10
   10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
     10.1. Normative References . . . . . . . . . . . . . . . . . . . 11
     10.2. Informative References . . . . . . . . . . . . . . . . . . 11
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12


















Trammell, et al.         Expires January 6, 2011                [Page 2]


Internet-Draft              IPFIX Aggregation                  July 2010


1.  Introduction

   The aggregation of packet data into flows serves a variety of
   different purposes, as noted in [RFC3917] and [RFC5472].  Aggregation
   beyond the flow level, into records representing multiple flows, is a
   common analysis and data reduction technique as well.  Often these
   aggregation operations result in a time series of keys and counters,
   which is useful for gaining insight into trends and anomalies.
   Aggregation may also has added benefits for privacy: when data about
   specific transactions and hosts are aggregated together, it has an
   anonymising effect on the data.

   Aggregation can be applied in parallel with storage of original
   packet and flow data.  In certain situations, aggregation can be
   performed at mediator, with aggregated data can be used for initial
   analysis (e.g., to provide a reduced data stream for analysis at a
   central point, with original flow data stored in a higher-access-
   latency manner (e.g. tape storage at the mediator at a remote site).
   Other applications may benefit from a reverse application: online
   analysis of original flows, with long-term storage of aggregated
   data.  Aggregation can also be used to provide a lower-volume, lower-
   privacy-risk source of data for reporting or data exchange across
   organizational boundaries.

   While IPFIX can be used to export [RFC5101] and store [RFC5655]
   aggregated data, there exists as yet no common terminology or
   aggregation metadata representation for this purpose, no set of
   requirements to define what is meant by aggregation, and no facility
   for counting original Flows from which Aggregated Flows are derived.
   This document seeks to remedy this situation, defining a common basis
   for the application of IPFIX to the handling of aggregated data, with
   applicability to large-scale network data processing, archiving, and
   inter-organization exchange.


2.  Terminology

   Terms used in this document that are defined in the Terminology
   section of the IPFIX Protocol [RFC5101] document are to be
   interpreted as defined there.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

   In addition, this document defines the following terms





Trammell, et al.         Expires January 6, 2011                [Page 3]


Internet-Draft              IPFIX Aggregation                  July 2010


   Aggregated Flow:   A Flow, as defined by [RFC5101], derived from a
      set of zero or more original Flows within a defined time interval.
      The two primary differences between a Flow and an Aggregated Flow
      are (1) that the time interval of a Flow is generally derived from
      information about the timing of the packets comprising the Flow,
      while the time interval of an Aggregated Flow are generally
      externally imposed; and (2) that an Aggregated Flow may represent
      zero packets (i.e., an assertion that no packets were seen for a
      given Flow Key in a given time interval).

   (Intermediate) Aggregation Function:   A mapping from a set of zero
      or more original Flows into a set of aggregated Flow, that
      separates the original Flows into a set of one or more given time
      intervals.

   (Intermediate) Aggregation Process:   An Intermediate Process, as in
      [I-D.ietf-ipfix-mediators-framework], hosting an Intermediate
      Aggregation Function.

   [TODO: may want to define a more precise terminology for aggregated
   flow time interval, "contribution"/"presence" of an original Flow
   within an aggregated Flow...]


3.  Requirements for Aggregation Support in IPFIX

   In defining a terminology, additional information elements, and
   metadata export methods for Aggregated Flow export using IPFIX, we
   have sought to meet the following requirements.

   First, a specification of Aggregated Flow export must seek to be as
   interoperable as possible.  Export of Aggregated Flows using the
   techniques described in this document will result in Flow data which
   can be collected by Collecting Processes and read by File Readers
   which do not provide any special support for Aggregated Flow export.

   Second, a specification of Aggregated Flow export must seek to be as
   implementation-independent as the IPFIX protocol itself.  In
   Section 5, we specify the flow aggregation process as an intermediate
   process within the IPFIX Mediator framework
   [I-D.ietf-ipfix-mediators-framework], and specify a variety of
   different architectural arrangements for flow aggregation; these are
   meant to be descriptive as opposed to proscriptive.  In metadata
   export, we seek to define properties of the set of exported
   Aggregated Flows, as opposed to the properties of the specific
   algorithms used to aggregate these Flows.  Specifically out of scope
   for this effort are any definition of a language for proscribing
   aggregation records, or the configuration parameters of Aggregation



Trammell, et al.         Expires January 6, 2011                [Page 4]


Internet-Draft              IPFIX Aggregation                  July 2010


   Processes.

   From the definition of presented in Section 2, an Aggregated Flow is
   a Flow as in [RFC5101], with a restricted defintion as to the packets
   making up the Flow.  Practically speaking, Aggregated Flows are
   derived from original Flows, as opposed to a raw packet stream.  Key
   to this definition of Aggregated Flow is how timing affects the
   process of aggregation, as for the most part flow aggregation takes
   place within some set of (usually regular) time intervals.  Any
   specification for Aggregated Flow export must account for the special
   role time intervals play in aggregation, and the many-to-many
   relationship between Aggregated Flows and original Flows which this
   implies.


4.  Aggregation of IP Flows

   [TODO: frontmatter: restate definition: a flow is just a flow.  Here
   we talk about Aggregation Process actions.]

4.1.  A general model for IP Flow Aggregation

   [TODO: introduce as little of an Aggregation Process as necessary to
   be able to talk about things practically.  Define in simple terms the
   role of externally imposed time intervals on aggregation. ]

4.2.  Aggregating and Distributing Counters

   In general, counters in Aggregated Flows are treated the same as in
   any Flow: on a per-Information Element basis, the counters are
   calculated as if they were derived from the set of packets in the
   original flow.  For the most part, when aggregating original Flows
   into Aggregated Flows, this is simply done by summation.

   However, this raises a complication when aggregating original Flows
   for which original packet data is not available: often, when imposing
   an external time interval on an original Flow, the original Flow will
   incompletely cover one or more time intervals, and apply to one or
   more Aggregated Flows; in this case, the Aggregation Process must
   distribute the counters in the original Flows across the multiple
   Aggregated Flows.  There are several methods for doing this, listed
   here in increasing order of complexity and accuracy:

   End Interval:   The counters for an original Flow are added to the
      counters of the appropriate Aggregated Flow containing the end
      time of the original Flow.





Trammell, et al.         Expires January 6, 2011                [Page 5]


Internet-Draft              IPFIX Aggregation                  July 2010


   Start Interval:   The counters for an original Flow are added to the
      counters of the appropriate Aggregated Flow containing the start
      time of the original Flow.

   Mid Interval:   The counters for an original Flow are added to the
      counters of a single appropriate Aggregated Flow containing some
      timestamp between start and end time of the original Flow.

   Simple Uniform Distribution:   Each counter for an original Flow is
      divided by the number of time intervals the original Flow covers
      (i.e., of appropriate Aggregated Flows sharing the same Flow Key),
      and this number is added to each corresponding counter in each
      Aggregated Flow.

   Proportional Uniform Distribution:   Each counter for an original
      Flow is divided by the number of time _units_ the original Flow
      covers, to derive a mean count rate.  This mean count rate is then
      multiplied by the number of time units in the intersection of the
      duration of the original Flow and the time interval of each
      Aggregated Flow.  This is like simple uniform distribtion, but
      accounts for the fractional portions of a time interval covered by
      an original Flow in the first and last time interval.

   Simulated Process:   Each counter of the original Flow is distributed
      among the intervals of the Aggregated Flows according to some
      function the Aggregation Process uses based upon properties of
      Flows presumed to be like the original Flow.  For example, bulk
      transfer flows might follow a more or less proportional uniform
      distribtion, while interactive processes are far more bursty.

   Direct:   The Aggregating Process has access to the original packet
      timings from the packets making up the original Flow, and uses
      these to distribute or recalculate the counters.

   A method for exporting the distribution of counters across multiple
   Aggregated Flows is detailed in Section 6.2.  In any case, counters
   MUST be distributed across the multiple Aggregated Flows in such a
   way that the total count is preserved; this property allows data to
   be aggregated and re-aggregated without any loss of original count
   information.  To avoid confusion in interpretation of the aggregated
   data, all the counters for a set of given original Flows SHOULD be
   distributed via the same method.

4.3.  Counting Original Flows

   When aggregating multiple original Flows into an Aggregated Flow, it
   is often useful to know how many original Flows are present in the
   Aggregated Flow.  This document introduces four new information



Trammell, et al.         Expires January 6, 2011                [Page 6]


Internet-Draft              IPFIX Aggregation                  July 2010


   elements in Section 6.1 to export these counters.

   There are two possible ways to count original Flows, which we call
   here conservative and non-conservative.  Conservative flow counting
   has the property that each original Flow contributes exactly one to
   the total flow count within a set of aggregated Flows.  In other
   words, conservative flow counters are distributed just as any other
   counter, except each original Flow is assumed to have a flow count of
   one.  When a count for an original Flow must be distributed across a
   set of Aggregated Flows, and a distribution method is used which does
   not account for that original Flow completely within a single
   Aggregated Flow, conservative flow counting requires a fractional
   representation.

   By contrast, non-conservative flow counting is used to count how many
   flows are represented in an Aggregated Flow.  Flow counters are not
   distributed in this case.  An original Flow which is present within N
   Aggregated Flows would add N to the total non-conservative flow
   count, one to each Aggregated Flow.


5.  Aggregation in the IPFIX Architecture

   [TODO: describe these diagrams]

   +==========================================+
   | Exporting Process                        |
   +==========================================+
     |                                      |
     |             (Aggregated Flow Export) |
     V                                      |
   +=============================+          |
   | Mediator                    |          |
   +=============================+          |
     |                                      |
     | (Aggregating Mediator)               |
     V                                      V
   +==========================================+
   | Collecting Process                       |
   +==========================================+
           |
           | (Aggregation for Storage)
           V
   +--------------------+
   | IPFIX File Storage |
   +--------------------+

                 Figure 1: Potential Aggregation Locations



Trammell, et al.         Expires January 6, 2011                [Page 7]


Internet-Draft              IPFIX Aggregation                  July 2010


   packets --+                     +- IPFIX Messages -+
             |                     |                  |
             V                     V                  V
   +==================+ +====================+ +=============+
   | Metering Process | | Collecting Process | | File Reader |
   +==================+ +====================+ +=============+
             |            original | Flows            |
             V                     V                  V
   +=========================================================+
   |           Intermediate Aggregation Process (IAP)        |
   +=========================================================+
             | Aggregated                  Aggregated |
             | Flows                            Flows |
             V                                        V
   +===================+                       +=============+
   | Exporting Process |                       | File Writer |
   +===================+                       +=============+
             |                                        |
             +------------> IPFIX Messages <----------+

           Figure 2: Data flows through the aggregation process


         +----+  +-----+  +----+
 pkts -> | MP |->| IAP |->| EP |-> Export of Aggregated Flows
         +----+  +-----+  +----+
         +----+  +-----+  +----+
IPFIX -> | CP |->| IAP |->| EP |-> Aggregating Mediator
         +----+  +-----+  +----+
         +----+  +-----+  +----+
IPFIX -> | CP |->| IAP |->| FW |-> Aggregation for storage in IPFIX Files
         +----+  +-----+  +----+
         +----+  +-----+  +----+
IPFIX -> | FR |->| IAP |->| FW |-> Aggregation for analysis
 File    +----+  +-----+  +----+   (aggregating File manipulator)

   Figure 3: Possible aggregation arrangements in the IPFIX architecture


6.  Export of Aggregated IP Flows using IPFIX

   [TODO: frontmatter: here we talk about export specifics.]

6.1.  Flow Count Export

   [TODO: describe these four IEs. for now, see Section 4.3]





Trammell, et al.         Expires January 6, 2011                [Page 8]


Internet-Draft              IPFIX Aggregation                  July 2010


6.1.1.  originalFlowsPresent Information Element

   Description:   The non-conservative count of original Flows
      containing packets from which a this Aggregated Flow was
      aggregated.

   Abstract Data Type:   unsigned64

   ElementId:   TBD1

   Status:   Proposed

6.1.2.  originalFlowsInitiated InformationElement

   Description:   The conservative count of original Flows whose first
      packet is represented within this Aggregated Flow.

   Abstract Data Type:   unsigned64

   ElementId:   TBD2

   Status:   Proposed

6.1.3.  originalFlowsCompleted InformationElement

   Description:   The conservative count of original Flows whose last
      packet is represented within this Aggregated Flow.

   Abstract Data Type:   unsigned64

   ElementId:   TBD3

   Status:   Proposed

6.1.4.  originalFlows InformationElement

   Description:   The conservative count of original Flows represented
      within this Aggregated Flow; may be distributed via any of the
      methods described in Section 4.2

   Abstract Data Type:   float64

   ElementId:   TBD4

   Status:   Proposed






Trammell, et al.         Expires January 6, 2011                [Page 9]


Internet-Draft              IPFIX Aggregation                  July 2010


6.2.  Aggregate Counter Distibution Export

   [TODO: options template and IE for representing the distribution
   methods in Section 4.2]

6.3.  Time Interval Export

   Since an Aggregated Flow is simply a Flow, the existing timestamp
   Information Elements in the IPFIX Information Model (e.g.,
   flowStartMilliseconds, flowEndNanoseconds) are sufficient to specify
   the time interval for aggregation.  Therefore, this document
   specifies no new aggregation-specific Information Elements for
   exporting time interval information.

   Each Aggregated Flow SHOULD contain both an interval start and
   interval end timestamp.  If an exporter of Aggregated Flows omits the
   interval end timestamp from each Aggregated Flow, the time interval
   for Aggregated Flows within an Observation Domain and Transport
   Session MUST be regular and constant, and SHOULD be exported using
   the Options Record described in this section.  However, note that
   this approach might lead to interoperability problems when exporting
   Aggregated Flows to non-aggregation-aware Collecting Processes and
   downstream analysis tasks; therefore, an Exporting Process capable of
   exporting only interval start timestamps MUST provide a configuration
   option to export interval end timestamps as well.

   [TODO: options template and IE for binding a time interval to a
   session.]


7.  Examples

   [TODO]


8.  Security Considerations

   [TODO]


9.  IANA Considerations

   [TODO: add all IEs defined in Section 6.]


10.  References





Trammell, et al.         Expires January 6, 2011               [Page 10]


Internet-Draft              IPFIX Aggregation                  July 2010


10.1.  Normative References

   [RFC5101]  Claise, B., "Specification of the IP Flow Information
              Export (IPFIX) Protocol for the Exchange of IP Traffic
              Flow Information", RFC 5101, January 2008.

   [RFC5102]  Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
              Meyer, "Information Model for IP Flow Information Export",
              RFC 5102, January 2008.

   [RFC5610]  Boschi, E., Trammell, B., Mark, L., and T. Zseby,
              "Exporting Type Information for IP Flow Information Export
              (IPFIX) Information Elements", RFC 5610, July 2009.

   [RFC5655]  Trammell, B., Boschi, E., Mark, L., Zseby, T., and A.
              Wagner, "Specification of the IP Flow Information Export
              (IPFIX) File Format", RFC 5655, October 2009.

   [RFC3330]  IANA, "Special-Use IPv4 Addresses", RFC 3330,
              September 2002.

10.2.  Informative References

   [RFC5103]  Trammell, B. and E. Boschi, "Bidirectional Flow Export
              Using IP Flow Information Export (IPFIX)", RFC 5103,
              January 2008.

   [RFC5472]  Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP
              Flow Information Export (IPFIX) Applicability", RFC 5472,
              March 2009.

   [RFC5470]  Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
              "Architecture for IP Flow Information Export", RFC 5470,
              March 2009.

   [I-D.ietf-ipfix-mediators-framework]
              Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi,
              "IPFIX Mediation: Framework",
              draft-ietf-ipfix-mediators-framework-06 (work in
              progress), April 2010.

   [I-D.ietf-ipfix-mediators-problem-statement]
              Kobayashi, A., Claise, B., Nishida, H., Sommer, C.,
              Dressler, F., and E. Stephan, "IPFIX Mediation: Problem
              Statement",
              draft-ietf-ipfix-mediators-problem-statement-09 (work in
              progress), March 2010.




Trammell, et al.         Expires January 6, 2011               [Page 11]


Internet-Draft              IPFIX Aggregation                  July 2010


   [RFC5153]  Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and P.
              Aitken, "IP Flow Information Export (IPFIX) Implementation
              Guidelines", RFC 5153, April 2008.

   [RFC3917]  Quittek, J., Zseby, T., Claise, B., and S. Zander,
              "Requirements for IP Flow Information Export (IPFIX)",
              RFC 3917, October 2004.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.


Authors' Addresses

   Brian Trammell
   Swiss Federal Institute of Technology Zurich
   Gloriastrasse 35
   8092 Zurich
   Switzerland

   Phone: +41 44 632 70 13
   Email: trammell@tik.ee.ethz.ch


   Elisa Boschi
   Swiss Federal Institute of Technology Zurich
   Gloriastrasse 35
   8092 Zurich
   Switzerland

   Email: boschie@tik.ee.ethz.ch


   Arno Wagner
   Consecom AG
   Bellariastrasse 11
   8002 Zurich
   Switzerland

   Email: arno@wagner.name











Trammell, et al.         Expires January 6, 2011               [Page 12]


Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/