draft-ietf-rtfm-acct-experiences-00.txt   rfc2123.txt 
Internet Engineering Task Force Nevil Brownlee Network Working Group N. Brownlee
INTERNET-DRAFT The University of Auckland Request for Comments: 2123 The University of Auckland
Expire in six months May 1996 Category: Informational March 1997
Traffic Flow Measurement: Experiences with NeTraMet
<draft-ietf-rtfm-acct-experiences-00.txt> Traffic Flow Measurement: Experiences with NeTraMet
Status of this Memo Status of this Memo
This document is an Internet Draft. Internet Drafts are working This memo provides information for the Internet community. This memo
documents of the Internet Engineering Task Force (IETF), its Areas, and does not specify an Internet standard of any kind. Distribution of
its Working Groups. Note that other groups may also distribute working this memo is unlimited.
documents as Internet Drafts. This Internet Draft is a product of the
Realtime Traffic Flow Measurement Working Group of the IETF.
Internet Drafts are draft documents valid for a maximum of six months.
Internet Drafts may be updated, replaced, or obsoleted by other
documents at any time. It is not appropriate to use Internet Drafts as
reference material or to cite them other than as a "working draft" or
"work in progress."
Please check the I-D abstract listing contained in the internet-drafts
Shadow Directories on nic.ddn.mil, nnsc.nsf.net, nic.nordu.net,
ftp.nisc.sri.com or munnari.oz.au to learn the current status of this or
any other Internet Draft.
Abstract Abstract
This memo records experiences in implementing and using the Traffic Flow This memo records experiences in implementing and using the Traffic
Measurement Architecture and Meter MIB. It discusses the implementation Flow Measurement Architecture and Meter MIB. It discusses the
of NeTraMet (a traffic meter and combined manager / meter reader), implementation of NeTraMet (a traffic meter) and NeMaC (a combined
considers the writing of meter rule sets and gives some guidance on manager and meter reader), considers the writing of meter rule sets
setting up a traffic flow measurement system using NeTraMet. and gives some guidance on setting up a traffic flow measurement
system using NeTraMet.
Contents Table of Contents
1 Introduction 3 1 Introduction 2
1.1 NeTraMet structure and development . . . . . . . . . . . . . . 3 1.1 NeTraMet structure and development . . . . . . . . . . . . . . 3
1.2 Scope of this document . . . . . . . . . . . . . . . . . . . . 4 1.2 Scope of this document . . . . . . . . . . . . . . . . . . . . 4
2 Implementation 4
2 Implementation 5 2.1 Choice of meter platform . . . . . . . . . . . . . . . . . . . 4
2.1 Choice of meter platform . . . . . . . . . . . . . . . . . . . 5 2.2 Programming support requirements . . . . . . . . . . . . . . . 5
2.2 Programming support requirements . . . . . . . . . . . . . . . 6
2.2.1 DOS environment . . . . . . . . . . . . . . . . . . . . . 6 2.2.1 DOS environment . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Unix environment . . . . . . . . . . . . . . . . . . . . . 7 2.2.2 Unix environment . . . . . . . . . . . . . . . . . . . . . 7
2.3 Implementing the meter . . . . . . . . . . . . . . . . . . . . 7
2.3 Implementing the meter . . . . . . . . . . . . . . . . . . . . 8 2.3.1 Data structures . . . . . . . . . . . . . . . . . . . . . 7
2.3.1 Data structures . . . . . . . . . . . . . . . . . . . . . 8 2.3.2 Packet matching . . . . . . . . . . . . . . . . . . . . . 8
2.3.2 Packet matching . . . . . . . . . . . . . . . . . . . . . 9 2.3.3 Testing groups of rule addresses . . . . . . . . . . . . . 8
2.3.3 Testing groups of rule addresses . . . . . . . . . . . . . 9 2.3.4 Compression of address masks . . . . . . . . . . . . . . . 9
2.3.4 Compression of address masks . . . . . . . . . . . . . . 10 2.3.5 Ignoring unwanted flow data . . . . . . . . . . . . . . . 10
2.3.5 Ignoring unwanted flow data . . . . . . . . . . . . . . . 11 2.3.6 Observing meter reader activity . . . . . . . . . . . . . 11
2.3.6 Observing meter reader activity . . . . . . . . . . . . . 12 2.3.7 Meter memory management . . . . . . . . . . . . . . . . . 12
2.3.7 Meter memory management . . . . . . . . . . . . . . . . . 12 2.4 Data collection . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Restarting a meter . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Data Collection . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Restarting a meter . . . . . . . . . . . . . . . . . . . . . 15
2.6 Performance . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6 Performance . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 Writing rule sets 16
3.1 Rule set to observe all flows . . . . . . . . . . . . . . . . 17
3.2 Specifying flow direction, using computed attributes . . . . . 18
3.3 Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4 More complicated rule sets . . . . . . . . . . . . . . . . . . 23
3 Writing Rule Sets 16 4 Flow data files 26
3.1 Rule set to observe all flows . . . . . . . . . . . . . . . . 16 4.1 Sample flow data file . . . . . . . . . . . . . . . . . . . . 27
3.2 Specifying flow direction, using computed attributes . . . . 18 4.2 Flow data file features . . . . . . . . . . . . . . . . . . . 28
3.3 Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.3 Terminating and restarting meter reading . . . . . . . . . . . 29
3.4 More complicated rule sets . . . . . . . . . . . . . . . . . 22 5 Analysis applications 30
6 Using NeTraMet in a measurement system 31
4 Flow Data Files 25 6.1 Examples of NeTraMet in production use . . . . . . . . . . . . 31
4.1 Sample Flow Data File . . . . . . . . . . . . . . . . . . . . 25 7 Acknowledgments 33
4.2 Flow data file features . . . . . . . . . . . . . . . . . . . 27 8 References 33
4.3 Terminating and restarting meter reading . . . . . . . . . . 28 9 Security Considerations 34
10 Author's Address 34
5 Analysis applications 28
6 Using NeTraMet in a measurement system 29
7 References 30
8 Acknowledgments 30
9 References 30
10 Security Considerations 31
11 Author's Address 31
1 Introduction 1 Introduction
Early in 1992 my University needed to develop a system for recovering Early in 1992 my University needed to develop a system for recovering
the costs of its Internet traffic. In March that year I attended the the costs of its Internet traffic. In March of that year I attended
Internet Accounting Working Group's session at the San Diego IETF, where the Internet Accounting Working Group's session at the San Diego
I was delighted to find that the Group had produced a detailed IETF, where I was delighted to find that the Group had produced a
architecture for measuring network traffic and were waiting for someone detailed architecture for measuring network traffic and were waiting
to try implementing it. for someone to try implementing it.
During 1992 I produced a prototype measurement system, using balanced During 1992 I produced a prototype measurement system, using balanced
binary trees to store information about traffic flows. This work was binary trees to store information about traffic flows. This work was
reported at the Washington IETF in November 1992. The prototype reported at the Washington IETF in November 1992. The prototype
performed well, but it made no attempt to recover memory from old flows, performed well, but it made no attempt to recover memory from old
and the overheads in managing the balanced trees proved to be flows, and the overheads in managing the balanced trees proved to be
unacceptably high. I moved on to develop a production- quality system, unacceptably high. I moved on to develop a production-quality
this time using hash tables to index the flow information. system, this time using hash tables to index the flow information.
This version was called NeTraMet (the Network Traffic Meter), and was This version was called NeTraMet (the Network Traffic Meter), and was
released as free software in October 1993. Since then I have continued released as free software in October 1993. Since then I have
working on NeTraMet, producing new releases two or three times each continued working on NeTraMet, producing new releases two or three
year. NeTraMet is now in production use at many sites around the world. times each year. NeTraMet is now in production use at many sites
It is difficult to estimate the number of sites, but there is an active around the world. It is difficult to estimate the number of sites,
NeTraMet mailing list, which had about 130 subscribers in March 1996. but there is an active NeTraMet mailing list, which had about 130
subscribers in March 1996.
Early in 1996 the Realtime Traffic Flow Measurement Working Group (RTFM) Early in 1996 the Realtime Traffic Flow Measurement Working Group
was chartered to move the Traffic Flow Measurement Architecture on to (RTFM) was chartered to move the Traffic Flow Measurement
the IETF standards track. The present document records traffic flow Architecture on to the IETF standards track. This document records
measurement experience gained through three years experience with traffic flow measurement experience gained through three years
NeTraMet. experience with NeTraMet.
1.1 NeTraMet structure and development 1.1 NeTraMet structure and development
The Traffic Flow Architecture document [1] describes four components: The Traffic Flow Architecture document [1] describes four components:
- METERS, which are attached to the network at the points where it is - METERS, which are attached to the network at the points where
desired to measure the traffic, it is desired to measure the traffic,
- METER READERS, which read data from meters and store it for later - METER READERS, which read data from meters and store it for later
use, use,
- MANAGERS, which configure meters and control meter readers, and - MANAGERS, which configure meters and control meter readers, and
- ANALYSIS APPLICATIONS, which process the data from meter readers so - ANALYSIS APPLICATIONS, which process the data from meter readers
as to produce whatever reports are required. so as to produce whatever reports are required.
NeTraMet is a computer program which implements the Traffic Meter, NeTraMet is a computer program which implements the Traffic Meter,
stores the measured flow data in memory, and provides an SNMP agent so stores the measured flow data in memory, and provides an SNMP agent
as to make it available to Meter Readers. The NeTraMet distribution so as to make it available to Meter Readers. The NeTraMet
files include NeMaC, which is a combined Manager and Meter Reader distribution files include NeMaC, which is a combined Manager and
capable of managing an arbitrary number of meters, each of which may be Meter Reader capable of managing an arbitrary number of meters, each
using its own rule set, and having its flow data collected at its own of which may be using its own rule set, and having its flow data
specified intervals. The NeTraMet distribution also includes several collected at its own specified intervals. The NeTraMet distribution
rudimentary Analysis Applications, allowing users to produce simple also includes several rudimentary Analysis Applications, allowing
plots from NeMaC's flow data files (fd_filter and fd_extract) and to users to produce simple plots from NeMaC's flow data files (fd_filter
monitor the flows at a remote meter (nm_rc). and fd_extract) and to monitor - in real time - the flows at a remote
meter (nm_rc and nifty).
Since the first release the Traffic Meter MIB [2] has been both improved Since the first release the Traffic Meter MIB [2] has been both
and simplified. Significant changes have included better ways to improved and simplified. Significant changes have included better
specify traffic flows (i.e. more actions and better control structures ways to specify traffic flows (i.e. more actions and better control
for the Packet Matching Engine), and computed attributes (class and structures for the Packet Matching Engine), and computed attributes
kind). These changes have been prompted by operational requirements at (class and kind). These changes have been prompted by operational
sites using NeTraMet, and have been tested extensively in successive requirements at sites using NeTraMet, and have been tested
versions of NeTraMet. extensively in successive versions of NeTraMet.
NeTraMet is widely used to collect usage data for Internet Service NeTraMet is widely used to collect usage data for Internet Service
Providers. This is especially so in Australia and New Zealand, but Providers. This is especially so in Australia and New Zealand, but
there are also active users at sites around the world, for example in there are also active users at sites around the world, for example in
Canada, France, Germany and Poland. Canada, France, Germany and Poland.
NeTraMet is also very useful as a tool for understanding exactly where NeTraMet is very useful as a tool for understanding exactly where
traffic is flowing in large networks. Since the Traffic Meters perform traffic is flowing in large networks. Since the Traffic Meters
considerable data reduction (as specified by their rule sets) they perform considerable data reduction (as specified by their rule sets)
significantly reduce the volume of data to be read by Meter Readers. they significantly reduce the volume of data to be read by Meter
This characteristic makes NeTraMet particularly effective for networks Readers. This characteristic makes NeTraMet particularly effective
with many remote sites. An example of this (the Kawaihiko network) is for networks with many remote sites. An example of this (the
presented below. Kawaihiko network) is briefly described below.
As well as providing data for post-observation analysis, NeTraMet can
be used for real-time network monitoring and trouble-shooting. The
NeTraMet distribution includes 'nifty,' an X/Motif application which
monitors traffic flows and attempts to highlight those which are
'interesting.'
1.2 Scope of this document 1.2 Scope of this document
This document presents the experience gained from three years work with This document presents the experience gained from three years work
the Traffic Flow Measurement Architecture. Its contents are grouped as with the Traffic Flow Measurement Architecture. Its contents are
follows grouped as follows
- Implementation issues for NeTraMet and NeMaC, - Implementation issues for NeTraMet and NeMaC,
- How rule files work, and how to write them for particular purposes, - How rule files work, and how to write them for particular
and purposes, and
- How to use NeTraMet and NeMaC for short-term and log-term flow - How to use NeTraMet and NeMaC for short-term and long-term flow
measurement. measurement.
2 Implementation 2 Implementation
2.1 Choice of meter platform 2.1 Choice of meter platform
As pointed out in the Architecture document [1], the goal of the As pointed out in the Architecture document [1], the goal of the
Realtime Traffic Flow Measurement Working Group is to develop a standard Realtime Traffic Flow Measurement Working Group is to develop a
for the Traffic Meter, with the goal of seeing it implemented in network standard for the Traffic Meter, with the goal of seeing it
devices such as hubs, switches and routers. Until the Architecture is implemented in network devices such as hubs, switches and routers.
well enough developed to allow this, it has sufficed to implement the Until the Architecture is well enough developed to allow this, it has
meter as a program running on a general-purpose computer system. sufficed to implement the meter as a program running on a general-
purpose computer system.
The choice of computer system for NeTraMet was driven by the need to The choice of computer system for NeTraMet was driven by the need to
choose one which would be widely available within the Internet choose one which would be widely available within the Internet
community. One strong possibility was a Unix system, since these are community. One strong possibility was a Unix system, since these are
commonly used for a variety of network support and management tasks. commonly used for a variety of network support and management tasks.
For the initial implementation, however, Unix would have had some For the initial implementation, however, Unix would have had some
disadvantages: disadvantages:
- The wide variety of different Unix systems can increase the - The wide variety of different Unix systems can increase the
difficulties of software support. difficulties of software support.
- The cost of a Unix system as a meter is too high to allow users to - The cost of a Unix system as a meter is too high to allow users
run meters simultaneously at many points within their networks. to run meters simultaneously at many points within their
networks.
Another factor in choosing the platform was system performance. When I Another factor in choosing the platform was system performance. When
first started implementing NeTraMet it was impossible to predict how I first started implementing NeTraMet it was impossible to predict
much processing workload was needed for a viable meter. Similarly, I how much processing workload was needed for a viable meter.
had no idea how much memory would be required for code or data. I Similarly, I had no idea how much memory would be required for code
therefore chose to implement NeTraMet on a DOS PC. This was because: or data. I therefore chose to implement NeTraMet on a DOS PC. This
was because:
- It is a minimum system in all respects. If the meter works well on - It is a minimum system in all respects. If the meter works
such a system, it can be implemented on almost any hardware well on such a system, it can be implemented on almost any
(including routers, switches, etc.) hardware (including routers, switches, etc.)
- It is an inexpensive system. Sites can easily afford to have many - It is an inexpensive system. Sites can easily afford to have
meters around their networks. many meters around their networks.
- It is a simple system, and one which I had complete control over. - It is a simple system, and one which I had complete control over.
This allowed me to implement effective instrumentation to monitor This allowed me to implement effective instrumentation to monitor
the meter's performance, and to include a wide variety of the meter's performance, and to include a wide variety of
performance optimisations in the code. performance optimisations in the code.
Once the meter was running I needed a manager to download rule files to Once the meter was running I needed a manager to download rule files
it. Since a single manager and meter reader can effectively support a to it. Since a single manager and meter reader can effectively
large number of meters, a Unix environment for NeMaC was a natural support a large number of meters, a Unix environment for NeMaC was a
choice. There are fewer software support problems for NeMaC than for natural choice. There are fewer software support problems for NeMaC
NeTraMet since NeMaC has minimal support needs - it only needs to open a than for NeTraMet since NeMaC has minimal support needs - it only
UDP socket to the SNMP port on each controlled meter. needs to open a UDP socket to the SNMP port on each controlled meter.
Early NeTraMet distributions contained only the PC meter and Unix Early NeTraMet distributions contained only the PC meter and Unix
manager. In later releases I ported NeTraMet (the meter) to Unix, and manager. In later releases I ported NeTraMet (the meter) to Unix,
extended the control features of NeMaC (the combined manager and meter and extended the control features of NeMaC (the combined manager and
reader). I have also experimented with porting NeMaC to the DOS system. meter reader). I have also experimented with porting NeMaC to the
This is not difficult, but doesn't seem to be worth pursuing. DOS system. This is not difficult, but doesn't seem to be worth
pursuing.
The current version of NeTraMet is a production-quality traffic The current version of NeTraMet is a production-quality traffic
measurement system which has been in continuous use at the University of measurement system which has been in continuous use at the University
Auckland for nearly two years. of Auckland for nearly two years.
2.2 Programming support requirements 2.2 Programming support requirements
To implement the Traffic Flow Meter I needed an programming environment To implement the Traffic Flow Meter I needed a programming
providing good support for the following: environment providing good support for the following:
- observation of packet headers on the network; - observation of packet headers on the network;
- system timer with better than 10 ms resolution; - system timer with better than 10 ms resolution;
- IP (Internet Protocol), for communications with manager and meter - IP (Internet Protocol), for communications with manager and meter
reader; reader;
- SNMP, for the agent implementing the Meter MIB. - SNMP, for the agent implementing the Meter MIB.
2.2.1 DOS environment 2.2.1 DOS environment
For the PC I chose to use Ethernet as the physical network medium. This For the PC I chose to use Ethernet as the physical network medium.
is simply an initial choice, being the medium used within the University This is simply an initial choice, being the medium used within the
of Auckland's data network. Interfaces for other media could easily be University of Auckland's data network. Interfaces for other media
added as they are needed. could easily be added as they are needed.
In the PC environment a variety of 'generalised' network interfaces are In the PC environment a variety of 'generalised' network interfaces
available. I considered those available from companies such as Novell, are available. I considered those available from companies such as
DEC and Microsoft and decided against them, partly because they are Novell, DEC and Microsoft and decided against them, partly because
proprietary, and partly because they did not appear to be particularly they are proprietary, and partly because they did not appear to be
easy to use. Instead I chose the CRYNWR Packet Drivers [3] . These are particularly easy to use. Instead I chose the CRYNWR Packet Drivers
available for a wide variety of interface cards and are simple and [3] . These are available for a wide variety of interface cards and
clearly documented. They support Ethernet's promiscuous mode, allowing are simple and clearly documented. They support Ethernet's
one to observe headers for every passing packet in a straightforward promiscuous mode, allowing one to observe headers for every passing
manner. One disadvantage of the Packet Drivers is that it is harder to packet in a straightforward manner. One disadvantage of the Packet
use them with newer user shells (such as Microsoft Windows), but this Drivers is that it is harder to use them with newer user shells (such
was irrelevant since I intended to run the meter as the only program on as Microsoft Windows), but this was irrelevant since I intended to
a dedicated machine. run the meter as the only program on a dedicated machine.
Timing on the PC presented a challenge since the BIOS timer routines Timing on the PC presented a challenge since the BIOS timer routines
only provide a clock tick about 18 times each second, which limits the only provide a clock tick about 18 times each second, which limits
available time resolution. Initially I made do with a timing resolution the available time resolution. Initially I made do with a timing
of one second for packets, since I believed that most flows existed for resolution of one second for packets, since I believed that most
many seconds. In recent years it has become apparent that many flows flows existed for many seconds. In recent years it has become
have lifetimes well under a second. To measure them properly with the apparent that many flows have lifetimes well under a second. To
Traffic Flow Meter one needs times resolved to 10 millisecond intervals, measure them properly with the Traffic Flow Meter one needs times
this being the size of TimeTicks, the most common time unit within SNMP resolved to 10 millisecond intervals, this being the size of
[4]. Since all the details of the original PC are readily available TimeTicks, the most common time unit within SNMP [4]. Since all the
[5], it was not difficult to understand the underlying hardware. I have details of the original PC are readily available [5], it was not
written PC timer routines for NeTraMet which read the hardware timer difficult to understand the underlying hardware. I have written PC
with 256 times the resolution of the DOS clock ticks, i.e. about 5 timer routines for NeTraMet which read the hardware timer with 256
ticks per millisecond. times the resolution of the DOS clock ticks, i.e. about 5 ticks per
millisecond.
There are many TCP/IP implementations available for DOS, but most of There are many TCP/IP implementations available for DOS, but most of
them are commercial software. Instead I chose Waterloo TCP [6], since them are commercial software. Instead I chose Waterloo TCP [6],
this was available (including full source code) as public domain since this was available (including full source code) as public
software. This was necessary since I needed to modify it to allow me to domain software. This was necessary since I needed to modify it to
save incoming packet headers at the same time as forwarding packets allow me to save incoming packet headers at the same time as
destined for the meter to the IP handler routines. For SNMP I chose CMU forwarding packets destined for the meter to the IP handler routines.
SNMP [7], since again this was available (with full source code) as For SNMP I chose CMU SNMP [7], since again this was available (with
public domain software. This made it fairly simple to port it from Unix full source code) as public domain software. This made it fairly
to the PC. simple to port it from Unix to the PC.
Finally, for the NeTraMet development I used Borland's Turbo C and Turbo Finally, for the NeTraMet development I used Borland's Turbo C and
Assembler. Although many newer C programming environments are now Turbo Assembler. Although many newer C programming environments are
available, I have been perfectly happy with Turbo C version 2 for the now available, I have been perfectly happy with Turbo C version 2 for
NeTraMet project! the NeTraMet project!
2.2.2 Unix environment 2.2.2 Unix environment
In implementing the Unix meter, the one obvious problem was 'how do I In implementing the Unix meter, the one obvious problem was 'how do I
get access to packet headers?' Early versions of the Unix meter were get access to packet headers?' Early versions of the Unix meter were
implemented using various system-specific interfaces on a SunOS 4.2 implemented using various system-specific interfaces on a SunOS 4.2
system. Later versions use libpcap [8], which provides a portable system. Later versions use libpcap [8], which provides a portable
method of obtaining access to packet headers on a wide range of Unix method of obtaining access to packet headers on a wide range of Unix
systems. I have verified that this works very well for ethernet systems. I have verified that this works very well for ethernet
interfaces on Solaris, SunOS, Irix, DEC Unix and Linux, and for FDDI interfaces on Solaris, SunOS, Irix, DEC Unix and Linux, and for FDDI
interfaces on Solaris. libpcap provides timestamps for each packet interfaces on Solaris. libpcap provides timestamps for each packet
header with resolution determined by the system clock, which is header with resolution determined by the system clock, which is
certainly better than 10 ms! certainly better than 10 ms!
All Unix systems provide TCP/IP capabilities, so that was not an issue. All Unix systems provide TCP/IP capabilities, so that was not an
For SNMP I used CMU SNMP, exactly as on the PC. issue. For SNMP I used CMU SNMP, exactly as on the PC.
2.3 Implementing the meter 2.3 Implementing the meter
This section briefly discusses the data structures used by the meter, This section briefly discusses the data structures used by the meter,
and the packet matching process. One very strong concern during the and the packet matching process. One very strong concern during the
evolution of NeTraMet has been the need for the highest possible level evolution of NeTraMet has been the need for the highest possible
of meter performance. A variety of interesting optimisations have been level of meter performance. A variety of interesting optimisations
developed to achieve this; as discussed below. Another particular have been developed to achieve this; as discussed below. Another
concern was the need for efficient and effective memory managent; this particular concern was the need for efficient and effective memory
is discussed in detail below. managent; this is discussed in detail below.
2.3.1 Data structures 2.3.1 Data structures
All the programs in NeTraMet, NeMaC and their supporting utility All the programs in NeTraMet, NeMaC and their supporting utility
programs are written in C, partly because C and its run-time libraries programs are written in C, partly because C and its run-time
provides good access to the underlying hardware, and partly because I libraries provides good access to the underlying hardware, and partly
have found it to be a highly portable language. because I have found it to be a highly portable language.
The data for each flow is stored in a C structure. The structure The data for each flow is stored in a C structure. The structure
includes all the flow's attribute values (including packet and byte includes all the flow's attribute values (including packet and byte
counts), together with a link field which can be used to link flows into counts), together with a link field which can be used to link flows
lists. NeTraMet assumes that Adjacent addresses are 802 MAC Addresses, into lists. NeTraMet assumes that Adjacent addresses are 802 MAC
which are all six bytes long. Similarly, Transport addresses are Addresses, which are all six bytes long. Similarly, Transport
assumes to be two bytes long, which is the case for port numbers in IP. addresses are assumes to be two bytes long, which is the case for
Peer addresses are normally four bytes or less in length. They may, port numbers in IP. Peer addresses are normally four bytes or less
however, be as long as 20 bytes (for CLNS). I have chosen to use a fixed in length. They may, however, be as long as 20 bytes (for CLNS). I
Peer address size, defined at compile time, so as to avoid the have chosen to use a fixed Peer address size, defined at compile
complexity of having variable-sized flow structures. time, so as to avoid the complexity of having variable-sized flow
structures.
The flow table itself is an array of pointers to flow data structures, The flow table itself is an array of pointers to flow data
which allows indexed access to flows via their flow numbers. There is structures, which allows indexed access to flows via their flow
also a single large hash table, referred to in the Architecture document numbers. There is also a single large hash table, referred to in the
[1] as the flow table's 'search index'. Each hash value in the table Architecture document [1] as the flow table's 'search index'. Each
points to a circular chain of flows. To find a flow one computes its hash value in the table points to a circular chain of flows. To find
hash value then searches that value's flow chain. a flow one computes its hash value then searches that value's flow
chain.
The meter stores each rule in a C structure. All the rule components The meter stores each rule in a C structure. All the rule components
have fixed sizes, but addresses must be wide enough to hold any type of have fixed sizes, but address fields must be wide enough to hold any
address - Adjacent, Peer or Transport. The rule address width is type of address - Adjacent, Peer or Transport. The rule address
defined at compile time, in the same way as flow Peer addresses. Each width is defined at compile time, in the same way as flow Peer
rule set is implemented as an array of pointers to rule data structures, addresses. Each rule set is implemented as an array of pointers to
and the rule table is an array of pointers to the rule sets. The size rule data structures, and the rule table is an array of pointers to
of each rule set is specified by NeMaC (before it begins downloading the the rule sets. The size of each rule set is specified by NeMaC
rule set), but the maximum number of rule sets is defined at compile (before it begins downloading the rule set), but the maximum number
time. of rule sets is defined at compile time.
2.3.2 Packet matching 2.3.2 Packet matching
Packet matching is carried out in NeTraMet exactly as described in the Packet matching is carried out in NeTraMet exactly as described in
Architecture document [1]. Each incoming packet header is analysed so the Architecture document [1]. Each incoming packet header is
as to determine its attribute values. These values are stored in a analysed so as to determine its attribute values. These values are
structure which is passed to the Packet Matching Engine. To facilitate stored in a structure which is passed to the Packet Matching Engine.
matching with source and destination reversed this structure contains To facilitate matching with source and destination reversed this
two substructures, one containing the source Adjacent, Peer and structure contains two substructures, one containing the source
Transport address values, the other containing the destination address Adjacent, Peer and Transport address values, the other containing the
values. destination address values.
2.3.3 Testing groups of rule addresses 2.3.3 Testing groups of rule addresses
As described in the Architecture [1] each rule's address will usually be As described in the Architecture [1] each rule's address will usually
tested, i.e. ANDed with the rule's mask and compared with the rule's be tested, i.e. ANDed with the rule's mask and compared with the
value. If the comparison fails, the next rule in sequence is executed. rule's value. If the comparison fails, the next rule in sequence is
This allows one to write rule sets which use a group of rules to test an executed. This allows one to write rule sets which use a group of
incoming packet to see whether one of its addresses - e.g. its rules to test an incoming packet to see whether one of its addresses
SourcePeerAddress - is one of a set of specified IP addresses. Such - e.g. its SourcePeerAddress - is one of a set of specified IP
groups of related rules can grow quite large, containing hundreds of addresses. Such groups of related rules can grow quite large,
rules. It was clear that sequential execution of such groups of rules containing hundreds of rules. It was clear that sequential execution
would be slow, and that something better was essential. of such groups of rules would be slow, and that something better was
essential.
The optimisation implemented in NeTraMet is to find groups of rules The optimisation implemented in NeTraMet is to find groups of rules
which test the same attribute with the same mask, and convert them into which test the same attribute with the same mask, and convert them
a single hashed search of their addresses. The overhead of setting up into a single hashed search of their values. The overhead of setting
hash tables (one for each group) is incurred once, just before the meter up hash tables (one for each group) is incurred once, just before the
starts running a new rule set. When a 'group' test is to be performed, meter starts running a new rule set. When a 'group' test is to be
the meter ANDs the incoming attribute value, computes a hash value from performed, the meter ANDs the incoming attribute value, computes a
it, and uses this to search the group's hash table. Early tests showed hash value from it, and uses this to search the group's hash table.
that the rule hash chains were usually very short, usually having only Early tests showed that the rule hash chains were usually very short,
one or two members. The effect is to reduce large sequences of tests to usually having only one or two members. The effect is to reduce
a hash computation and lookup, with a very small number of compares; in large sequences of tests to a hash computation and lookup, with a
short this is an essential optimisation for any traffic meter! very small number of compares; in short this is an essential
optimisation for any traffic meter!
There is, of course, overhead associated with performing the hashed There is, of course, overhead associated with performing the hashed
compare. NeTraMet handles this by having a minimum group size defined compare. NeTraMet handles this by having a minimum group size
at compile time. If the group is too small it is not combined into a defined at compile time. If the group is too small it is not
hashed group. combined into a hashed group.
In early versions of NeTraMet I did not allow Gotos into a hashed group In early versions of NeTraMet I did not allow Gotos into a hashed
of rules, which proved to be an unnecessarily conservative position. group of rules, which proved to be an unnecessarily conservative
NeTraMet stores each group's hash table in a separate memory area, and position. NeTraMet stores each group's hash table in a separate
keeps a pointer to the hash table in the first rule of the group. (The memory area, and keeps a pointer to the hash table in the first rule
rules data structure has an extra component to hold this hash table of the group. (The rules data structure has an extra component to
pointer). Rules within the group don't have hash table pointers; when hold this hash table pointer). Rules within the group don't have
they are executed as the target of a Goto rule they behave as ordinary hash table pointers; when they are executed as the target of a Goto
rules, i.e. their tests are performed normally. rule they behave as ordinary rules, i.e. their tests are performed
normally.
2.3.4 Compression of address masks 2.3.4 Compression of address masks
When the Packet Matching Engine has decided that an incoming packet When the Packet Matching Engine has decided that an incoming packet
belongs to a flow which is to be measured, it searches the flow table to belongs to a flow which is to be measured, it searches the flow table
determine whether or not the flow is already current. It does this by to determine whether or not the flow is already present. It does
computing a hash from the packet and using it to for access to the flow this by computing a hash from the packet and using it for access to
table's search index. the flow table's search index.
When designing a hash table, one normally assumes that the objects in When designing a hash table, one normally assumes that the objects in
the table have a constant size. For NeTraMet's flow table this would the table have a constant size. For NeTraMet's flow table this would
mean that each flow would contain a value for every attribute. This, mean that each flow would contain a value for every attribute. This,
however, is not the case, since only those attribute values 'pushed' by however, is not the case, since only those attribute values 'pushed'
rules during packet matching are stored for a flow. by rules during packet matching are stored for a flow.
To demonstrate this problem , let us assume that every flow in the table To demonstrate this problem , let us assume that every flow in the
contains a value for only one attribute, SourcePeerAddress, and that the table contains a value for only one attribute, SourcePeerAddress, and
rule set decides whether flows belong to a specified list of IP that the rule set decides whether flows belong to a specified list of
networks, in which case only their network numbers are pushed. The IP networks, in which case only their network numbers are pushed.
rules perform this test using a variety of masks, since the network The rules perform this test using a variety of masks, since the
number allocations range from 16 to 24 bits in width. In searching the network number allocations range from 16 to 24 bits in width. In
flow table, the meter must distinguish between zeroes in the address and searching the flow table, the meter must distinguish between zeroes
'don't care' bits which had been ANDed out. To achieve this it must in the address and 'don't care' bits which had been ANDed out. To
store SourcePeerMask values in the flow table as well as the ANDed achieve this it must store SourcePeerMask values in the flow table as
SourcePeerAddress values. well as the ANDed SourcePeerAddress values.
In early versions of NeTraMet this problem was side-stepped by using In early versions of NeTraMet this problem was side-stepped by using
multiple hash tables and relying on the user to write rules which used multiple hash tables and relying on the user to write rules which
the same set of attributes and masks for all the flows in each table. used the same set of attributes and masks for all the flows in each
This was effective, but clumsy and difficult to explain. Later versions table. This was effective, but clumsy and difficult to explain.
changed to using a single hash table, and storing the mask values for Later versions changed to using a single hash table, and storing the
all the address attributes in each flow. mask values for all the address attributes in each flow.
The current version of the meter stores the address masks in compressed The current version of the meter stores the address masks in
form. After examining a large number of rule sets I realised that compressed form. After examining a large number of rule sets I
although a rule set may have many rules, it usually has a very small realised that although a rule set may have many rules, it usually has
number of address masks. It is a simple matter to build a table of a very small number of address masks. It is a simple matter to build
address masks, and store an index to this 'mask table' instead of a a table of address masks, and store an index to this 'mask table'
complete mask. NeTraMet's maximum number of masks is defined at compile instead of a complete mask. NeTraMet's maximum number of masks is
time, up to a maximum of 256. This allows me to use a single byte for defined at compile time, up to a maximum of 256. This allows me to
each mask in the flow data structure, significantly reducing the use a single byte for each mask in the flow data structure,
structure's size. As well as this size reduction, two masks can be significantly reducing the structure's size. As well as this size
compared by comparing their indices in the mask table, i.e. it reduces reduction, two masks can be compared by comparing their indices in
to a single-byte comparison. Overall, using a mask table seems to the mask table, i.e. it reduces to a single-byte comparison.
provide useful improvements in storage efficiency and execution speed. Overall, using a mask table seems to provide useful improvements in
storage efficiency and execution speed.
2.3.5 Ignoring unwanted flow data 2.3.5 Ignoring unwanted flow data
As described in the Architecture document [1], every incoming packet is As described in the Architecture document [1], every incoming packet
tested against the current rule set by the Packet Matching Engine. This is tested against the current rule set by the Packet Matching Engine.
section explains my efforts to improve NeTraMet performance on the PC by This section explains my efforts to improve NeTraMet performance on
reducing the amount of processing required by each incoming packet. the PC by reducing the amount of processing required by each incoming
packet.
On the PC each incoming packet causes an interrupt, which NeTraMet must On the PC each incoming packet causes an interrupt, which NeTraMet
process so as to collect information about the packet. In early must process so as to collect information about the packet. In early
versions I used a ring buffer with 512 slots for packet headers, and versions I used a ring buffer with 512 slots for packet headers, and
simply copied each packet's first 64 bytes into the next free slot. The simply copied each packet's first 64 bytes into the next free slot.
packet headers were later taken from the buffer, attribute values were The packet headers were later taken from the buffer, attribute values
extracted from them, and the resulting 'incoming attribute values' were extracted from them, and the resulting 'incoming attribute
record were passed to the Packet Matching Engine. values' records were passed to the Packet Matching Engine.
I modified the interrupt handling code to extract the attribute values I modified the interrupt handling code to extract the attribute
and stored those in a 'buffer slot.' This reduced the amount of storage values and store them in a 'buffer slot.' This reduced the amount of
required in each slot, allowing more space for storing flows. It did storage required in each slot, allowing more space for storing flows.
increase slightly the amount of processing done for each packet It did increase slightly the amount of processing done for each
interrupt, but this has not caused any problems. packet interrupt, but this has not caused any problems.
In later versions I realised that if one is only interested in measuring In later versions I realised that if one is only interested in
IP packets, there is no point in storing (and later processing) Novell measuring IP packets, there is no point in storing (and later
or EtherTalk packets! It is a simple matter for the meter to inspect a processing) Novell or EtherTalk packets! It is a simple matter for
rule set and determine which Peer types are of interest. If there are the meter to inspect a rule set and determine which Peer types are of
PushRule rules which test SourcePeerType (or DestPeerType), they specify interest. If there are PushRule rules which test SourcePeerType (or
which types are of interest. If there are no such rules, every packet DestPeerType), they specify which types are of interest. If there
type is of interest. The PC NeTraMet has a set of Boolean variables, are no such rules, every packet type is of interest. The PC NeTraMet
one for each protocol it can handle. The values of these 'protocol' has a set of Boolean variables, one for each protocol it can handle.
variables are determined when the meter begins running a new rule set. The values of these 'protocol' variables are determined when the
For each incoming packet, the interrupt handler determines the Peer meter begins running a new rule set. For each incoming packet, the
type. If the protocol is not of interest, no further processing is done interrupt handler determines the Peer type. If the protocol is not
- the packet is simply ignored. In a similar manner, if Adjacent of interest, no further processing is done - the packet is simply
addresses are never tested there is no point in copying them into the ignored. In a similar manner, if Adjacent addresses are never tested
packet buffer slot. there is no point in copying them into the packet buffer slot.
The overall effect of these optimisations is most noticeable for rule The overall effect of these optimisations is most noticeable for rule
files which measure IP flows on a network segment which also carries a files which measure IP flows on a network segment which also carries
lot of traffic for other network protocols; this situation is common on a lot of traffic for other network protocols; this situation is
multiprotocol Local Area networks. On the Unix version of NeTraMet, the common on multiprotocol Local Area networks. On the Unix version of
Operating System does all the packet interrupt processing, and libpcap NeTraMet the Operating System does all the packet interrupt
[8] delivers packet headers directly to NeTraMet. There seems little processing, and libpcap [8] delivers packet headers directly to
point in using the 'protocol' and 'adjacent address' optimisations for NeTraMet. The 'protocol' and 'adjacent address' optimisations are
Unix. still performed, at the point when NeTraMet receives the packet
headers from libpcap.
2.3.6 Observing meter reader activity 2.3.6 Observing meter reader activity
The Architecture document [1] explains that a flow data record must be The Architecture document [1] explains that a flow data record must
held in the meter until its data has been read by a meter reader. A be held in the meter until its data has been read by a meter reader.
meter must therefore have a reliable way of deciding when flow data has A meter must therefore have a reliable way of deciding when flow data
been read. The problem is complicated by the fact that there may be has been read. The problem is complicated by the fact that there may
more than one meter reader, and that meter readers collect their data be more than one meter reader, and that meter readers collect their
asynchronously. data asynchronously.
Early versions of NeTraMet solved this problem by having a single MIB Early versions of NeTraMet solved this problem by having a single MIB
variable which a meter reader could set to indicate that it was variable which a meter reader could set to indicate that it was
beginning a data collection. In response to such an SMP SET request, beginning a data collection. In response to such an SNMP SET
NeTraMet would update its 'collectors' table. This had an entry for request, NeTraMet would update its 'collectors' table. This had an
each meter reader, and variables recording the start time for the last entry for each meter reader, and variables recording the start time
two collections. The most recent collection might be still in progress, for the last two collections. The most recent collection might still
but its start time provides a safe estimate of a time when the one be in progress, but its start time provides a safe estimate of the
before it actually finished. Space used for flows which have been idle time when the one before it actually finished. Space used for flows
since the penultimate collection started can be recovered by the meter's which have been idle since the penultimate collection started can be
garbage collector, as described below. recovered by the meter's garbage collector, as described below.
The Meter MIB [2] specifies a more general table of meter reader The Meter MIB [2] specifies a more general table of meter reader
information. A meter reader wishing to collect data from a meter must information. A meter reader wishing to collect data from a meter
inform the meter of its intention by creating a row in the table, then must inform the meter of its intention by creating a row in the
setting a LastTime variable in that row to indicate the start of a table, then setting a LastTime variable in that row to indicate the
collection. The meter handles such a SET request exactly as described start of a collection. The meter handles such a SET request exactly
above. If there are multiple meter readers the meter can easily find as described above. If there are multiple meter readers the meter
the earliest time any of them started its penultimate collection, and can easily find the earliest time any of them started its penultimate
may recover flows idle since then. Should a meter reader fail, NeTraMet collection, and may recover flows idle since then. Should a meter
will eventually time out its entry in the meter reader info table, and reader fail, NeTraMet will eventually time out its entry in the meter
delete it. This avoids a situation where the meter can't recover flows reader info table, and delete it. This avoids a situation where the
until they have been collected by several meter readers, one of which meter can't recover flows until they have been collected by several
has failed. meter readers, one of which has failed.
2.3.7 Meter memory management 2.3.7 Meter memory management
In principle, the size of the flow table (i.e. the maximum number of In principle, the size of the flow table (i.e. the maximum number of
flows) could be changed dynamically. This would involve allocating flows) could be changed dynamically. This would involve allocating
space for the flow table's new pointer array and copying the old space for the flow table's new pointer array and copying the old
pointers into it. NeTraMet does not implement this. Instead the pointers into it. NeTraMet does not implement this. Instead the
maximum number of flows is set from the command line when it starts maximum number of flows is set from the command line when it starts
execution. If no maximum is specified, a compile-time default number is execution. If no maximum is specified, a compile-time default number
used. is used.
Memory for flow data structures (i.e. 'flows') is allocated Memory for flow data structures (i.e. 'flows') is allocated
dynamically. NeTraMet requests the C run-time system for blocks of dynamically. NeTraMet requests the C run-time system for blocks of
several hundred flows, and links them into a free list. When a new flow several hundred flows, and links them into a free list. When a new
is needed NeTraMet gets memory space from the free list, then searches flow is needed NeTraMet gets memory space from the free list, then
the flow table's pointer array for an unused flow pointer. In practice searches the flow table's pointer array for an unused flow pointer.
a 'last-allocated' index is used to point to the flow table, so a simple In practice a 'last-allocated' index is used to point to the flow
linear search suffices. The flow index is saved in the flow's data table, so a simple linear search suffices. The flow index is saved
record, and its other attribute values are set to zero. in the flow's data record, and its other attribute values are set to
zero.
To release a flow data record it must first be removed from any hash To release a flow data record it must first be removed from any hash
list it is part of - this is straightforward since those lists are list it is part of - this is straightforward since those lists are
circular. The flow's entry in the flow table pointer array is then set circular. The flow's entry in the flow table pointer array is then
to zero (NULL pointer), and its space is returned to the free list. set to zero (NULL pointer), and its space is returned to the free
list.
Once a flow data record is created it could continue to exist Once a flow data record is created it could continue to exist
indefinitely. In time, however, the meter would run out of space. To indefinitely. In time, however, the meter would run out of space.
deal with this problem NeTraMet uses an incremental garbage collector to To deal with this problem NeTraMet uses an incremental garbage
reclaim memory . collector to reclaim memory.
At regular intervals specified by a 'GarbageCollectInterval' variable At regular intervals specified by a 'GarbageCollectInterval' variable
the garbage collector procedure is invoked. This searches through the the garbage collector procedure is invoked. This searches through
flow table looking for flows which might be recovered. To control the the flow table looking for flows which might be recovered. To
resources consumed by garbage collection there are limits on the number control the resources consumed by garbage collection there are limits
of in-use and idle flows which the garbage collector may inspect these on the number of in-use and idle flows which the garbage collector
are set either when NeTraMet is started (as options on the command line) may inspect these are set either when NeTraMet is started (as
or dynamically by NeMaC (using variables in an Enterprise MIB for options on the command line) or dynamically by NeMaC (using variables
NeTraMet) in an Enterprise MIB for NeTraMet)
To decide whether a flow can be recovered, the garbage collector To decide whether a flow can be recovered, the garbage collector
considers how long it has been idle (no packets in either direction), considers how long it has been idle (no packets in either direction),
and when its data was last collected. If it has been collected by all and when its data was last collected. If it has been collected by
known meter readers since its LastTime, its memory may be recovered. all known meter readers since its LastTime, its memory may be
This alogrithm is implemented using a variable called recovered. This alogrithm is implemented using a variable called
'GarbageCollectTime,' which normally contains the meter's UpTime when 'GarbageCollectTime,' which normally contains the meter's UpTime when
the penultimate collection (i.e. the one before last) was started. See the penultimate collection (i.e. the one before last) was started.
the section on observing meter reader activity (above) for more details. See the section on observing meter reader activity (above) for more
details.
Should flows not be collected often enough the meter could run out of Should flows not be collected often enough the meter could run out of
space. NeTraMet attempts to prevent this by having a low-priority space. NeTraMet attempts to prevent this by having a low-priority
background process check the percentage of flows active and compare it background process check the percentage of flows active and compare
with the HighWaterMark MIB variable. If the percentage of active flows it with the HighWaterMark MIB variable. If the percentage of active
is greater than the high-water mark, 'GarbageCollectTime' is incremented flows is greater than the high-water mark, 'GarbageCollectTime' is
by the current value of the InactivityTimeout MIB variable. incremented by the current value of the InactivityTimeout MIB
variable.
The Meter MIB [2] specifies that a meter should switch to using a The Meter MIB [2] specifies that a meter should switch to using a
'standby' rule table if the percentage of flows active rises above 'standby' rule set if the percentage of active flows rises above
HighWaterMark. In using NeTraMet to measure traffic flows to and from HighWaterMark. In using NeTraMet to measure traffic flows to and
the University of Auckland it has not been difficult to create standby from the University of Auckland it has not been difficult to create
rules which are very similar to the 'production' rule file, differing standby rules which are very similar to the 'production' rule file,
only in that they push much less information about flows. This has, on differing only in that they push much less information about flows.
several occasions, allowed the meter to continue running for one or two This has, on several occasions, allowed the meter to continue running
days after the meter reader failed. When the meter reader restarted, it for one or two days after the meter reader failed. When the meter
was able ro collect all the accumulated flow data! reader restarted, it was able to collect all the accumulated flow
data!
The MIB also specifies that the meter should take some action when
the active flow percentage rises above its FloodMark value. If this
were not done, the meter could spend a rapidly increasing proportion
of its time garbage collecting, to the point where its ability to
respond to requests from its manager would be compromised. NeTraMet
switches to the default rule set when its FloodMark is reached.
The MIB also specifies that the meter should take some action when the A potentially large number of new flows may be created when the meter
active flow percentage rises above its FloodMark value. If this were switches to a standby rule set. It is important to set a
not done, the meter could spend a rapidly increasing proportion of its HighWaterMark so as to allow enough flow table space for this. In
time garbage collecting, to the point where its ability to respond to practice, a HighWaterMark of 65% and a FloodMark of 95% seem to work
requests from its manager would be compromised. NeTraMet switches to well.
the default rule set when its FloodMark is reached..
2.4 Data Collection 2.4 Data collection
As explained above, a meter reader wishing to collect flows begins each As explained above, a meter reader wishing to collect flows begins
collection by setting the LastTime variable in its ReaderInfoTable row, each collection by setting the LastTime variable in its
then works its way through the flow table collecting data. A number of ReaderInfoTable row, then works its way through the flow table
algorithms can be used to examine the flow table; these are presented collecting data. A number of algorithms can be used to examine the
below. flow table; these are presented below.
The simplest approach is a linear scan of the table, reading the The simplest approach is a linear scan of the table, reading the
LastTime variable for each row. If the read fails the row is inactive. LastTime variable for each row. If the read fails the row is
If it succeeds, it is of interest if its LastTime value is greater than inactive. If it succeeds, it is of interest if its LastTime value is
the time of the last collection. Although this method is simple it is greater than the time of the last collection. Although this method
also rather slow, requiring an SNMP GET request for every possible flow; is simple it is also rather slow, requiring an SNMP GET request for
this renders it impractical. every possible flow; this renders it impractical.
Early versions of NeTraMet used two 'windows' into the flow table to Early versions of NeTraMet used two 'windows' into the flow table to
find flows which were of interest. Both windows were SNMP tables, find flows which were of interest. Both windows were SNMP tables,
indexed by a variable which specified a time. A succession of GETNEXT indexed by a variable which specified a time. A succession of
requests on one of these windows allowed NeMaC (the meter reader) to GETNEXT requests on one of these windows allowed NeMaC (the meter
find the flow indices for all flows which had been active since the reader) to find the flow indices for all flows which had been active
specified time. The two windows were the ActivityTime window (which since the specified time. The two windows were the ActivityTime
located active flows), and the CreateTime window (which located new window (which located active flows), and the CreateTime window (which
flows). Knowing the index of an active flow, the meter reader can GET located new flows). Knowing the index of an active flow, the meter
the values for all the attributes of interest. NeMaC allows the user to reader can GET the values for all the attributes of interest. NeMaC
specify which these are, rather than simply read all the attributes. a allows the user to specify which these are, rather than simply read
all the attributes.
Having the two windows allowed NeMaC to read attributes which remain Having the two windows allowed NeMaC to read attributes which remain
constant - such as the flow's address attributes - when the flow is constant - such as the flow's address attributes - when the flow is
created, but to only read attributes which change with time - such as created, but to only read attributes which change with time - such as
its packet and byte counts - during later collections. Experience has its packet and byte counts - during later collections. Experience
shown, however, that many flows have rather short lifetimes; one effect has shown, however, that many flows have rather short lifetimes; one
of this is that the improved efficiency of using two windows does not effect of this is that the improved efficiency of using two windows
result in any worthwhile improvement in collection performance. does not result in any worthwhile improvement in collection
performance.
The current version of the Meter MIB [2] uses a TimeFilter variable in The current version of the Meter MIB [2] uses a TimeFilter variable
the flow table entries. This can be used with GETNEXT requests to find in the flow table entries. This can be used with GETNEXT requests to
all flows which have been active since a specified time directly, find all flows which have been active since a specified time
without requiring the extra 'window' SNMP variables. It can be combined directly, without requiring the extra 'window' SNMP variables. It
with SNMPv2's GETBULK request to further reduce the number of SNMP can be combined with SNMPv2's GETBULK request to further reduce the
packets needed for each collection; I have yet to implement this in number of SNMP packets needed for each collection; I have yet to
NeTraMet. implement this in NeTraMet.
A disadvantage of using SNMP to collect data from the meter is that SMP A disadvantage of using SNMP to collect data from the meter is that
packets impose a high overhead. For example, if we wish to read an SNMP packets impose a high overhead. For example, if we wish to read
Integer32 variable (four bytes of data), it will be returned with its an Integer32 variable (four bytes of data), it will be returned with
object identifier, type and length, i.e. at least ten bytes of its object identifier, type and length, i.e. at least ten bytes of
superfluous data. One way to reduce this overhead is to use an Opaque superfluous data. One way to reduce this overhead is to use an
object to return a collection of data. NeTraMet uses this approach to Opaque object to return a collection of data. NeTraMet uses this
retrieve 'column activity data' from the meter, as follows. approach to retrieve 'column activity data' from the meter, as
follows.
Each packet of column activity data contains data values for a specified Each packet of column activity data contains data values for a
attribute, and each value is preceded by its flow number. The flow specified attribute, and each value is preceded by its flow number.
table can be regarded as a two-dimensional array, with a column for each The flow table can be regarded as a two-dimensional array, with a
flow attribute. Column activity data objects allow the meter reader to column for each flow attribute. Column activity data objects allow
read columns of the flow table, so as to collect only those attributes the meter reader to read columns of the flow table, so as to collect
specified by the user. The actual implementation is complicated by the only those attributes specified by the user. The actual
fact that since the flow table is read column by column, rows can become implementation is complicated by the fact that since the flow table
active after the first column has been read. NeMaC reads the widest is read column by column, rows can become active after the first
columns (those with greatest size in bytes, e.g. PeerAddress) first, column has been read. NeMaC reads the widest columns (those with
and ignores any rows which appear in later columns. Newly active rows greatest size in bytes, e.g. PeerAddress) first, and ignores any rows
will, of course, be read in the next collection. which appear in later columns. Newly active rows will, of course, be
read in the next collection.
Using Opaque objects in this way dramatically reduces the number of SNMP Using Opaque objects in this way dramatically reduces the number of
packets required to read a meter. This has proved worthwhile in SNMP packets required to read a meter. This has proved worthwhile in
situations where the number of flows is large (for example on busy situations where the number of flows is large (for example on busy
LANs), and where the meter(s) are physically dispersed over slow WAN LANs), and where the meter(s) are physically dispersed over slow WAN
links. It has the disadvantage that general-purpose MIB browsers cannot links. It has the disadvantage that general-purpose MIB browsers
understand the column activity variables, but this seems a small price cannot understand the column activity variables, but this seems a
to pay for the improved data collection performance. small price to pay for the improved data collection performance.
2.5 Restarting a meter 2.5 Restarting a meter
If a meter fails, for example because of a power failure, it will If a meter fails, for example because of a power failure, it will
restart and begin running rule set 1, the default rule set which is restart and begin running rule set 1, the default rule set which is
built into the meter. Its manager must recognise that this has built into the meter. Its manager must recognise that this has
happened, and respond with some suitable action. happened, and respond with some suitable action.
NeMaC allows the user to specify a 'keepalive' interval. After every NeMaC allows the user to specify a 'keepalive' interval. After every
such interval NeMaC reads the meter's sysUptime and compares it with the such interval NeMaC reads the meter's sysUptime and compares it with
last sysUptime. If the new sysUptime is less than the last one, NeMaC the last sysUptime. If the new sysUptime is less than the last one,
decides that the meter has restarted. It downloads the meter's backup NeMaC decides that the meter has restarted. It downloads the meter's
rule set and production rule set, then requests the meter to start backup rule set and production rule set, then requests the meter to
running the production rule set. In normal use we use a keepalive start running the production rule set. In normal use we use a
interval of five minutes and a collection interval of 15 minutes. If a keepalive interval of five minutes and a collection interval of 15
meter restarts, we lose up to five minutes data before the rules sets minutes. If a meter restarts, we lose up to five minutes data before
are downloaded. the rules sets are downloaded.
Having the meter run the default rule set on startup is part of the Having the meter run the default rule set on startup is part of the
Traffic Flow Measurement Architecture [1], in keeping with the notion Traffic Flow Measurement Architecture [1], in keeping with the notion
that meters are very simple devices which do not have disk storage. that meters are very simple devices which do not have disk storage.
Since disks are now very cheap, it may be worth considering whether the Since disks are now very cheap, it may be worth considering whether
architecture should allow a meter to save its configuration (including the architecture should allow a meter to save its configuration
rule sets) on disk. (including rule sets) on disk.
2.6 Performance 2.6 Performance
The PC version of the meter, NeTraMet, continually measures how much The PC version of the meter, NeTraMet, continually measures how much
processor time is being used. Whenever there is no incoming packet data processor time is being used. Whenever there is no incoming packet
to process, 'dummy' packets are generated and placed in the input data to process, 'dummy' packets are generated and placed in the
buffer. These packets are processed normally by the Packet Matching input buffer. These packets are processed normally by the Packet
Engine; they have a PeerType of 'dummy.' The numbers of dummy and Matching Engine; they have a PeerType of 'dummy.' The numbers of
normal packets are counted by the meter; their ratio is used as an dummy and normal packets are counted by the meter; their ratio is
estimate of the processor time which is 'idle,' i.e. not being used to used as an estimate of the processor time which is 'idle,' i.e. not
process incoming packets. The Unix version is intended to run as a being used to process incoming packets. The Unix version is intended
process in a multiprocessing system, so it cannot busy-wait in this way. to run as a process in a multiprocessing system, so it cannot busy-
wait in this way.
The meter also collects several other performance measures; these can be The meter also collects several other performance measures; these can
displayed on the meter console in response to keyboard requests. be displayed on the meter console in response to keyboard requests.
The PC meter can be used with a 10 MHz 286 machine, on which it can The PC meter can be used with a 10 MHz 286 machine, on which it can
handle a steady load of about 750 packets per second. On a 25 MHz 386SX handle a steady load of about 750 packets per second. On a 25 MHz
it will handle about 1250 packets per second. Users have reported that 386SX it will handle about 1250 packets per second. Users have
a 40 MHz 486 can handle peaks of about 3,000 packets per second without reported that a 40 MHz 486 can handle peaks of about 3,000 packets
packet loss. The Unix meter has been tested metering traffic on a per second without packet loss. The Unix meter has been tested
(lightly loaded) FDDI interface; it uses about one percent of the metering traffic on a (lightly loaded) FDDI interface; it uses about
processor time on a SPARC 10 system running Solaris. one percent of the processor time on a SPARC 10 system running
Solaris.
3 Writing Rule Sets 3 Writing rule sets
The Traffic Meter provides a versatile device for measuring a The Traffic Meter provides a versatile device for measuring a user-
user-specified set of traffic flows, and performing useful data specified set of traffic flows, and performing useful data reduction
reduction on them. This data reduction capability not only minimises on them. This data reduction capability not only minimises the
the volume of data to be collected by meter readers, but also simplifies volume of data to be collected by meter readers, but also simplifies
the later processing of traffic flow data. the later processing of traffic flow data.
The flows of interest, and the processing to be performed, are specified The flows of interest, and the processing to be performed, are
in a 'rule set' which is downloaded to the meter (NeTraMet) by the specified in a 'rule set' which is downloaded to the meter (NeTraMet)
manager (NeMaC). This section explains what is involved in writing rule by the manager (NeMaC). This section explains what is involved in
sets. writing rule sets.
NeTraMet is limited to metering packets observed on a network
segment. This means that for all the observed flows, Source and Dest
Type attributes (e.g. SourcePeerType and DestPeerType) have the same
value.
The NeTraMet implementation uses single variables in its flow data
structure for AdjacentType, SourceType and TransType. Nonetheless,
the rule sets discussed below push values for both Source and Dest
Type attributes; this make sure that packet matching works properly
with the directions reversed, even for a meter which allows Source
and Dest Type values to be different.
3.1 Rule set to observe all flows 3.1 Rule set to observe all flows
NeMaC reads rule sets from text files which contain the rules, the set NeMaC reads rule sets from text files which contain the rules, the
number which the meter (and meter reader) will identify them by, and a set number which the meter (and meter reader) will identify them by,
'format,' i.e. a list specifying which attributes the meter reader and a 'format,' i.e. a list specifying which attributes the meter
should collect and write to the flow data file. The # character reader should collect and write to the flow data file. The #
indicates the start of a comment; NeMaC ignores the rest of the line. character indicates the start of a comment; NeMaC ignores the rest of
the line.
SET 2 SET 2
# #
RULES RULES
# #
SourcePeerType & 255 = Dummy: Ignore, 0; SourcePeerType & 255 = Dummy: Ignore, 0;
Null & 0 = 0: GotoAct, Next; Null & 0 = 0: GotoAct, Next;
# #
SourcePeerType & 255 = 0: PushPkttoAct, Next; SourcePeerType & 255 = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next; DestPeerType & 255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next; SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
SourceTransType & 255 = 0: PushPkttoAct, Next; DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: PushPkttoAct, Next; SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransAddress & 255.255 = 0: CountPkt, 0; DestTransType & 255 = 0: PushPkttoAct, Next;
# SourceTransAddress & 255.255 = 0: PushPkttoAct, Next;
FORMAT FlowRuleSet FlowIndex FirstTime " " DestTransAddress & 255.255 = 0: CountPkt, 0;
SourcePeerType SourcePeerAddress DestPeerAddress " " #
SourceTransType SourceTransAddress DestTransAddress " " FORMAT FlowRuleSet FlowIndex FirstTime " "
ToPDUs FromPDUs " " ToOctets FromOctets; SourcePeerType SourcePeerAddress DestPeerAddress " "
SourceTransType SourceTransAddress DestTransAddress " "
ToPDUs FromPDUs " " ToOctets FromOctets;
The first rule tests the incoming packet's SourcePeerType to see whether The first rule tests the incoming packet's SourcePeerType to see
it is 'dummy.' If it is, the packet is ignored, otherwise the next rule whether it is 'dummy.' If it is, the packet is ignored, otherwise
is executed. the next rule is executed.
The second rule tests the Null attribute. Such a test always succeeds, The second rule tests the Null attribute. Such a test always
so the rule simply jumps to the action of the next rule. (The keyword succeeds, so the rule simply jumps to the action of the next rule.
'next' is converted by NeMaC into the number of the following rule.) (The keyword 'next' is converted by NeMaC into the number of the
following rule.)
The third rule pushes the packet's SourcePeerType value, then jumps to The third rule pushes the packet's SourcePeerType value, then jumps
the action of the next rule. The user does not know in advance what the to the action of the next rule. The user does not know in advance
value of PushPkt rules will be, which is why the value appearing in them what the value of PushPkt rules will be, which is why the value
is always zero. The user must take care not to write rule sets which appearing in them is always zero. The user must take care not to
try to perform the test in a PushPkt rule. This is a very common error write rule sets which try to perform the test in a PushPkt rule.
in a rule set, so NeMaC tests for it and displays an error message. This is a very common error in a rule set, so NeMaC tests for it and
displays an error message.
The following rules push a series of attribute values from the packet, The following rules push a series of attribute values from the
and the last rule also Counts the packet, i.e. it tells the Packet packet, and the last rule also Counts the packet, i.e. it tells the
Matching Engine (PME) that the packet has been successfully matched. Packet Matching Engine (PME) that the packet has been successfully
The PME responds by searching the flow table to see whether the flow is matched. The PME responds by searching the flow table to see whether
already current (i.e. in the table), creating a new flow data record the flow is already current (i.e. in the table), creating a new flow
for it should this be necessary, and incrementing its packet and byte data record for it should this be necessary, and incrementing its
counters. packet and byte counters.
Overall this rule set simply classifies the packet (i.e. decides Overall this rule set simply classifies the packet (i.e. decides
whether or not it is to be counted), then pushes all the Peer and whether or not it is to be counted), then pushes all the Peer and
Transport attribute values for it. It makes no attempt to specify a Transport attribute values for it. It makes no attempt to specify a
direction for the flow - this is left to the PME, as described in [1]. direction for the flow - this is left to the PME, as described in
The resulting flow data file will have each flow's source and [1]. The resulting flow data file will have each flow's source and
destination addresses in the order of the first packet the meter destination addresses in the order of the first packet the meter
observed for the flow. observed for the flow.
3.2 Specifying flow direction, using computed attributes 3.2 Specifying flow direction, using computed attributes
As indicated above the Packet Matching Engine will reliably determine As indicated above, the Packet Matching Engine will reliably
the flow, and the direction within that flow, for every packet seen by a determine the flow, and the direction within that flow, for every
meter. If the rule set does not specify a direction for the flow, the packet seen by a meter. If the rule set does not specify a direction
PME simply assumes that the first packet observed for a flow is for the flow, the PME simply assumes that the first packet observed
travelling forward, i.e. from source to destination. In later analysis for a flow is travelling forward, i.e. from source to destination.
of the flow data, however, one is usually interested in traffic to or In later analysis of the flow data, however, one is usually
from a particular source. interested in traffic to or from a particular source.
One can achieve this in a simple manner by writing a rule set to specify One can achieve this in a simple manner by writing a rule set to
the source for flows. All that is required is to have rules which specify the source for flows. All that is required is to have rules
succeed if the packet is travelling in the required direction, and which which succeed if the packet is travelling in the required direction,
execute a Retry action otherwise. This is demonstrated in the following and which execute a 'Fail' action otherwise. This is demonstrated in
two examples .. the following two examples.
# Count IP packets from network 130.216.0.0 (Note that early versions of NeMaC allowed 'Retry' as a synonym for
# 'Fail.' The current version also allows 'NoMatch,' which seems a
SourcePeerType & 255 = IP: Pushto, ip`pkt; better way to imply "fail, allowing PME to try a second match with
Null & 0 = 0: Ignore, 0; directions reversed.")
#
ip`pkt:
SourcePeerAddress & 255.255.0.0 = 130.216.0.0: Goto c`pkt;
Null & 0 = 0: Retry, 0;
#
c`pkt:
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: CountPkt, 0;
The rule labelled ip_pkt tests whether the packet came from network # Count IP packets from network 130.216.0.0
130.216. If it did not, the test fails and the following rule executes #
a Retry action, causing the PME to retry the match with the directions SourcePeerType & 255 = IP: Pushto, ip_pkt;
reversed. If the second match fails the packet did not have 130.216 as Null & 0 = 0: Ignore, 0;
an end-point, and is ignored. #
ip_pkt:
SourcePeerAddress & 255.255.0.0 = 130.216.0.0: Goto c_pkt;
Null & 0 = 0: NoMatch, 0;
#
c_pkt:
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: CountPkt, 0;
The next rule set meters IP traffic on a network segment which connects The rule labelled ip_pkt tests whether the packet came from network
two routers, g1 and g2. It classifies flows into three groups - those 130.216. If it did not, the test fails and the following rule
travelling from g1 to g2, those whose source is g1 and those whose executes a NoMatch action, causing the PME to retry the match with
source is g2. the directions reversed. If the second match fails the packet did
not have 130.216 as an end-point, and is ignored.
# Count IP packets between two gateways The next rule set meters IP traffic on a network segment which
# connects two routers, g1 and g2. It classifies flows into three
# -------+-------------------+-----------------+------- groups - those travelling from g1 to g2, those whose source is g1 and
# $ $ $ those whose source is g2.
# +----+-----+ +----+-----+ +---+---+
# $ g1 $ $ g2 $ $ meter $
# +-+-+-+-+--+ +-+-+-+-+--+ +-------+
#
SourcePeerType & 255 = IP: Pushto, ip`pkt;
Null & 0 = 0: Ignore, 0;
#
ip`pkt:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
Goto, s1;
Null & 0 = 0: Goto, s2;
s1:
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A
GotoAct, g3;
Null & 0 = 0: GotoAct, g1;
s2:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A:
Goto, s3;
Null & 0 = 0: Retry, 0;
s3:
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
Retry, 0;
Null & 0 = 0: GotoAct, g2;
#
g1: FlowClass & 255 = 1: PushtoAct, c`pkt;
g2: FlowClass & 255 = 2: PushtoAct, c`pkt;
g3: FlowClass & 255 = 3: PushtoAct, c`pkt;
#
c`pkt:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 0:
PushPkttoAct, Next;
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
Null & 0 = 0: Count, 0
The first two rules ignore non-IP packets. The next two rules Goto s1 # Count IP packets between two gateways
if the packet's source was g1, or to s2 otherwise. The rule labelled s2 #
tests whether the packet's source was g2; if not a Retry action is # -------+-------------------+------------------+-------
executed, allowing the PME to try the match with the packet's direction # | | |
reversed. If the match fails on the second try the packet didn't come # +----+-----+ +----+-----+ +---+---+
from g1 or g2, and is ignored. # | g1 | | g2 | | meter |
# +-+-+-+-+--+ +-+-+-+-+--+ +-------+
#
SourcePeerType & 255 = IP: Pushto, ip_pkt;
Null & 0 = 0: Ignore, 0;
#
ip_pkt:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
Goto, s1;
Null & 0 = 0: Goto, s2;
s1:
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A
GotoAct, g3;
Null & 0 = 0: GotoAct, g1;
s2:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A:
Goto, s3;
Null & 0 = 0: NoMatch, 0;
s3:
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
NoMatch, 0;
Null & 0 = 0: GotoAct, g2;
#
g1: FlowClass & 255 = 1: PushtoAct, c_pkt; # From g1
g2: FlowClass & 255 = 2: PushtoAct, c_pkt; # From g2
g3: FlowClass & 255 = 3: PushtoAct, c_pkt; # g1 to g2
#
c_pkt:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 0:
PushPkttoAct, Next;
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
Null & 0 = 0: Count, 0
Packets which come from g1 are tested by the rule labelled s1, and the The first two rules ignore non-IP packets. The next two rules Goto
PME will Goto either g3 or g1. s1 if the packet's source was g1, or to s2 otherwise. The rule
labelled s2 tests whether the packet's source was g2; if not a
NoMatch action is executed, allowing the PME to try the match with
the packet's direction reversed. If the match fails on the second
try the packet didn't come from (or go to) g1 or g2, and is ignored.
Packets which came from g2 are tested by the rule labelled s3. If they Packets which come from g1 are tested by the rule labelled s1, and
are not going to g1 the PME will Goto g2. If they are going to g1 a the PME will Goto either g3 or g1.
Retry action is executed - we want them counted as backward-travelling
packets for the g1-g2 flow.
The rules at g1, g2 and g3 push the value 1, 2 or 3 from their rule into Packets which came from g2 are tested by the rule labelled s3. If
the flow's FlowClass attribute. This value can be used by an Analysis they are not going to g1 the PME will Goto g2. If they are going to
Application to separate the flows into the three groups of interest. g1 a NoMatch action is executed - we want them counted as backward-
FlowClass is an example of a 'computed' attribute, i.e. one whose value travelling packets for the g1-g2 flow.
is Pushed by the PME during rule matching.
The remaining rules Push the values of other attributes required for The rules at g1, g2 and g3 push the value 1, 2 or 3 from their rule
later analysis, then Count the flow. into the flow's FlowClass attribute. This value can be used by an
Analysis Application to separate the flows into the three groups of
interest. FlowClass is an example of a 'computed' attribute, i.e.
one whose value is Pushed by the PME during rule matching.
The remaining rules Push the values of other attributes required for
later analysis, then Count the flow.
3.3 Subroutines 3.3 Subroutines
Subroutines are implemented in the PME in much the same way as in BASIC. Subroutines are implemented in the PME in much the same way as in
A subroutine body is just a sequence of statements, supported by the BASIC. A subroutine body is just a sequence of statements, supported
GoSub and Return actions. 'GoSub' saves the PME's running environment by the GoSub and Return actions. 'GoSub' saves the PME's running
and jumps to the first rule of the subroutine body. Subroutine calls environment and jumps to the first rule of the subroutine body.
may be nested as required - NeTraMet defines the maximum nesting at Subroutine calls may be nested as required - NeTraMet defines the
compile time. 'Return n' restores the environment and jumps to the maximum nesting at compile time. 'Return n' restores the environment
action part of the nth rule after the Gosub, where n is the index value and jumps to the action part of the nth rule after the Gosub, where n
from the Return rule. is the index value from the Return rule.
The Return action provides a way of influencing the flow of control in a The Return action provides a way of influencing the flow of control
rule set, rather like a FORTRAN Computed Goto. This is one way in which in a rule set, rather like a FORTRAN Computed Goto. This is one way
a subroutine can return a result. The other way is by Pushing a value in which a subroutine can return a result. The other way is by
in either a computed attribute (as demonstrated in the preceding section Pushing a value in either a computed attribute (as demonstrated in
above), or in a flow attribute. the preceding section), or in a flow attribute.
One common use for a subroutine is to test whether a packet attribute One common use for a subroutine is to test whether a packet attribute
matches one of a set of values. Such a subroutine becomes much more matches one of a set of values. Such a subroutine becomes much more
useful if it can be used to test one of several attributes. The PME useful if it can be used to test one of several attributes. The PME
architecture provides for this by using 'meter variables' to hold the architecture provides for this by using 'meter variables' to hold the
names of the attributes to be tested. The meter variables are called names of the attributes to be tested. The meter variables are called
V1, V2, V3, V4 and V5, and the Assign action is provided to set their V1, V2, V3, V4 and V5, and the Assign action is provided to set their
values. If, for example, we need a subroutine to test either values. If, for example, we need a subroutine to test either
SourcePeerAddress or DestPeerAddress, we write its rules to test V1 SourcePeerAddress or DestPeerAddress, we write its rules to test V1
instead. Before calling the subroutine we Assign SourcePeerAddress to instead. Before calling the subroutine we Assign SourcePeerAddress
V1; later tests of V1 are converted by the PME into tests on to V1; later tests of V1 are converted by the PME into tests on
SourcePeerAddress. Note that since meter variables may be reassigned in SourcePeerAddress. Note that since meter variables may be reassigned
a subroutine, their values are part of the environment which must be in a subroutine, their values are part of the environment which must
saved by a Gosub action. be saved by a Gosub action.
The following rule set demonstrates the use of a subroutine .. The following rule set demonstrates the use of a subroutine ..
# Rule specification file to tally IP packets in three groups: # Rule specification file to tally IP packets in three groups:
# UA to AIT, UA to elsewhere, AIT to elsewhere # UA to AIT, UA to elsewhere, AIT to elsewhere
# #
# -------+-------------------+-----------------+-------- # -------+-------------------+-----------------+--------
# $ $ $ # | | |
# +----+-----+ +----+-----+ +---+---+ # +----+-----+ +----+-----+ +---+---+
# $ UA $ $ AIT $ $ meter $ # | UA | | AIT | | meter |
# +-+-+-+-+--+ +-+-+-+-+--+ +-------+ # +-+-+-+-+--+ +-+-+-+-+--+ +-------+
# #
SourcePeerType & 255 = IP: PushtoAct, ip`pkt; SourcePeerType & 255 = IP: PushtoAct, ip_pkt;
Null & 0 = 0: Ignore, 0; Null & 0 = 0: Ignore, 0;
# #
ip`pkt: ip_pkt:
v1 & 0 = SourcePeerAddress: AssignAct, Next; v1 & 0 = SourcePeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify; Null & 0 = 0: Gosub, classify;
Null & 0 = 0: GotoAct, from`ua; # 1 ua Null & 0 = 0: GotoAct, from_ua; # 1 ua
Null & 0 = 0: GotoAct, from`ait; # 2 ait Null & 0 = 0: GotoAct, from_ait; # 2 ait
Null & 0 = 0: Retry, 0; # 3 other Null & 0 = 0: NoMatch, 0; # 3 other
# #
from`ua: from_ua:
v1 & 0 = DestPeerAddress: AssignAct, Next; v1 & 0 = DestPeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify; Null & 0 = 0: Gosub, classify;
Null & 0 = 0: Ignore, 0; # 1 ua-ua Null & 0 = 0: Ignore, 0; # 1 ua-ua
Null & 0 = 0: GotoAct, ok`pkt; # 2 ua-ait Null & 0 = 0: GotoAct, ok_pkt; # 2 ua-ait
Null & 0 = 0: Gotoact, ok`pkt; # 3 ua-other Null & 0 = 0: Gotoact, ok_pkt; # 3 ua-other
# #
from`ait: from_ait:
v1 & 0 = DestPeerAddress: AssignAct, Next; v1 & 0 = DestPeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify; Null & 0 = 0: Gosub, classify;
Null & 0 = 0: Retry, 0; # 1 ait-ua Null & 0 = 0: NoMatch, 0; # 1 ait-ua
Null & 0 = 0: Ignore, 0; # 2 ait-ait Null & 0 = 0: Ignore, 0; # 2 ait-ait
Null & 0 = 0: GotoAct, ok`pkt; # 3 ait-other Null & 0 = 0: GotoAct, ok_pkt; # 3 ait-other
# #
ok`pkt: ok_pkt:
Null & 0 = 0: Count, 0; Null & 0 = 0: Count, 0;
The subroutine begins at the rule labelled classify (shown below). It The subroutine begins at the rule labelled classify (shown below).
returns to the first, second or third rule after the invoking Gosub It returns to the first, second or third rule after the invoking
rule, depending on whether the tested PeerAddress is in the UA, AIT, or Gosub rule, depending on whether the tested PeerAddress is in the UA,
'other' group of networks. In the listing below only one network is AIT, or 'other' group of networks. In the listing below only one
tested in each of the groups - it is trivial to add more rules (one per network is tested in each of the groups - it is trivial to add more
network) into either of the first two groups. In this example the rules (one per network) into either of the first two groups. In this
subroutine Pushes the network number from the packet into the tested example the subroutine Pushes the network number from the packet into
attribute before returning. the tested attribute before returning.
The first invocation of classify (above) begins at the rule labelled The first invocation of classify (above) begins at the rule labelled
ip_pkt. It Assigns SourcePeerAddress to V1 then executes a Gosub action. ip_pkt. It Assigns SourcePeerAddress to V1 then executes a Gosub
Classify returns to one of the three following rules. They will Goto action. Classify returns to one of the three following rules. They
from_ua or from_ait if the packet came from the UA or AIT groups, will Goto from_ua or from_ait if the packet came from the UA or AIT
otherwise the PME will Retry the match. This means that matched flows groups, otherwise the PME will retry the match. This means that
will have a UA or AIT network as their source, and flows between other matched flows will have a UA or AIT network as their source, and
networks will be ignored. flows between other networks will be ignored.
The next two invocations of 'classify' test the packet's The next two invocations of 'classify' test the packet's
DestPeerAddress. Packets from AIT to UA are Retried, forcing them to be DestPeerAddress. Packets from AIT to UA are Retried, forcing them to
counted as AU to AIT flows. Packets from UA to UA are ignored, as are be counted as AU to AIT flows. Packets from UA to UA are ignored, as
packets from AIT to AIT. are packets from AIT to AIT.
classify: classify:
v1 & 255.255.0.0 = 130.216.0.0: GotoAct, ua; # ua v1 & 255.255.0.0 = 130.216.0.0: GotoAct, ua; # ua
v1 & 255.255.0.0 = 156.62.0.0: GotoAct, ait; # ait v1 & 255.255.0.0 = 156.62.0.0: GotoAct, ait; # ait
Null & 0 = 0: Return, 3; # other Null & 0 = 0: Return, 3; # other
ua: ua:
v1 & 255.255.0.0 = 0: PushPkttoAct, Next; v1 & 255.255.0.0 = 0: PushPkttoAct, Next;
Null & 0 = 0: Return, 1; Null & 0 = 0: Return, 1;
ait: ait:
v1 & 255.255.0.0 = 0: PushPkttoAct, Next; v1 & 255.255.0.0 = 0: PushPkttoAct, Next;
Null & 0 = 0: Return, 2; Null & 0 = 0: Return, 2;
3.4 More complicated rule sets 3.4 More complicated rule sets
The next example demonstrates a way of grouping IP flows together The next example demonstrates a way of grouping IP flows together
depending on their Transport Address, i.e. their IP port number. depending on their Transport Address, i.e. their IP port number.
Simply Pushing every flow's SourceTransAddress and DestTransAddress Simply Pushing every flow's SourceTransAddress and DestTransAddress
would produce a large number of flows, most of which differ only in one would produce a large number of flows, most of which differ only in
of their transport addresses (the one which is not a well-known port). one of their transport addresses (the one which is not a well-known
Instead we Push the well-known port number into each flow's port).
SourceTransAddress; its DestTransAddress will be zero by default.
Instead we Push the well-known port number into each flow's
SourceTransAddress; its DestTransAddress will be zero by default.
SourcePeerType & 255 = dummy: Ignore, 0; SourcePeerType & 255 = dummy: Ignore, 0;
SourcePeerType & 255 = IP: Pushto, IP`pkt; SourcePeerType & 255 = IP: Pushto, IP_pkt;
Null & 0 = 0: GotoAct, Next; Null & 0 = 0: GotoAct, Next;
SourcePeerType & 255 = 0: PushPkttoAct, Next; SourcePeerType & 255 = 0: PushPkttoAct, Next;
Null & 0 = 0: Count, 0; # Count others by protocol type Null & 0 = 0: Count, 0; # Count others by protocol type
# #
IP`pkt: IP_pkt:
SourceTransType & 255 = tcp: Pushto, tcp`udp; SourceTransType & 255 = tcp: Pushto, tcp_udp;
SourceTransType & 255 = udp: Pushto, tcp`udp; SourceTransType & 255 = udp: Pushto, tcp_udp;
SourceTransType & 255 = icmp: CountPkt, 0; SourceTransType & 255 = icmp: CountPkt, 0;
SourceTransType & 255 = ospf: CountPkt, 0; SourceTransType & 255 = ospf: CountPkt, 0;
Null & 0 = 0: GotoAct, c`unknown; # Unknown transport type Null & 0 = 0: GotoAct, c_unknown; # Unknown transport type
#
tcp`udp:
s`domain:
SourceTransAddress & 255.255 = domain: PushtoAct, c`well`known;
s`ftp:
SourceTransAddress & 255.255 = ftp: PushtoAct, c`well`known;
s`imap:
SourceTransAddress & 255.255 = 113: PushtoAct, c`well`known;
s`nfs
SourceTransAddress & 255.255 = 2049: PushtoAct, c`well`known;
s`pop:
SourceTransAddress & 255.255 = 110: PushtoAct, c`well`known;
s`smtp:
SourceTransAddress & 255.255 = smtp: PushtoAct, c`well`known;
s`telnet:
SourceTransAddress & 255.255 = telnet: PushtoAct, c`well`known;
s`www:
SourceTransAddress & 255.255 = www: PushtoAct, c`well`known;
s`xwin
SourceTransAddress & 255.255 = 6000: PushtoAct, c`well`known;
# #
DestTransAddress & 255.255 = domain: GotoAct, s`domain; tcp_udp:
DestTransAddress & 255.255 = ftp: GotoAct, s`ftp; s_domain:
DestTransAddress & 255.255 = 113: GotoAct, s`imap; SourceTransAddress & 255.255 = domain: PushtoAct, c_well_known;
DestTransAddress & 255.255 = 2049: GotoAct, s`nfs; s_ftp:
DestTransAddress & 255.255 = 110: GotoAct, s`pop; SourceTransAddress & 255.255 = ftp: PushtoAct, c_well_known;
DestTransAddress & 255.255 = smtp: GotoAct, s`smtp; s_imap:
DestTransAddress & 255.255 = telnet: GotoAct, s`telnet; SourceTransAddress & 255.255 = 113: PushtoAct, c_well_known;
DestTransAddress & 255.255 = www: GotoAct, s`www; s_nfs
DestTransAddress & 255.255 = 6000: GotoAct, s`xwin; SourceTransAddress & 255.255 = 2049: PushtoAct, c_well_known;
s_pop:
SourceTransAddress & 255.255 = 110: PushtoAct, c_well_known;
s_smtp:
SourceTransAddress & 255.255 = smtp: PushtoAct, c_well_known;
s_telnet:
SourceTransAddress & 255.255 = telnet: PushtoAct, c_well_known;
s_www:
SourceTransAddress & 255.255 = www: PushtoAct, c_well_known;
s_xwin
SourceTransAddress & 255.255 = 6000: PushtoAct, c_well_known;
# #
Null & 0 = 0: GotoAct, c`unknown; # 'Unusual' port DestTransAddress & 255.255 = domain: GotoAct, s_domain;
DestTransAddress & 255.255 = ftp: GotoAct, s_ftp;
DestTransAddress & 255.255 = 113: GotoAct, s_imap;
DestTransAddress & 255.255 = 2049: GotoAct, s_nfs;
DestTransAddress & 255.255 = 110: GotoAct, s_pop;
DestTransAddress & 255.255 = smtp: GotoAct, s_smtp;
DestTransAddress & 255.255 = telnet: GotoAct, s_telnet;
DestTransAddress & 255.255 = www: GotoAct, s_www;
DestTransAddress & 255.255 = 6000: GotoAct, s_xwin;
# #
c`unknown: Null & 0 = 0: GotoAct, c_unknown; # 'Unusual' port
#
c_unknown:
SourceTransType & 255 = 0: PushPkttoAct, Next; SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: PushPkttoAct, Next; SourceTransAddress & 255.255 = 0: PushPkttoAct, Next;
DestTransAddress & 255.255 = 0: CountPkt, 0; DestTransAddress & 255.255 = 0: CountPkt, 0;
# #
c`well`known: c_well_known:
Null & 0 = 0: Count, 0 Null & 0 = 0: Count, 0
# #
The first few rules ignore dummy packets, select IP packets for further The first few rules ignore dummy packets, select IP packets for
processing, and count packets for other protocols in a single flow for further processing, and count packets for other protocols in a single
each PeerType. TCP and UDP packets cause the PME to Push their flow for each PeerType. TCP and UDP packets cause the PME to Push
TransType and Goto tcp_udp. ICMP and OSPF packets are counted in flows their TransType and Goto tcp_udp. ICMP and OSPF packets are counted
which have only their TransType Pushed. in flows which have only their TransType Pushed.
At tcp_udp the packets' SourceTransAddress is tested to see whether it is At tcp_udp the packets' SourceTransAddress is tested to see whether
included in a set of 'interesting' port numbers. If it is, the port it is included in a set of 'interesting' port numbers. If it is, the
number is pushed from the rule into the SourceTransAddress attribute, port number is pushed from the rule into the SourceTransAddress
and the packet is counted at c_well_known. (NeMaC accepts Pushto as a attribute, and the packet is counted at c_well_known. (NeMaC accepts
synonym for PushRuleto). Pushto as a synonym for PushRuleto).
This testing is repeated for the packet's DestTransAddress; if one of This testing is repeated for the packet's DestTransAddress; if one of
these tests succeeds the PME Goes to the corresponding rule above and these tests succeeds the PME Goes to the corresponding rule above and
Pushes the port number into the flow's SourceTransAddress. If these Pushes the port number into the flow's SourceTransAddress. If these
tests fail the packet is counted at c_unknown. tests fail the packet is counted at c_unknown, where all the flow's
Trans attributes are pushed. For production use more well-known
ports would need to be included in the tests above - c_unknown is
intended only for little-used exception flows!
Note that these rules only Push a value into a flow's Note that these rules only Push a value into a flow's
SourceTransAddress, and they don't contain any Retry actions. They SourceTransAddress, and they don't contain any NoMatch actions. They
therefore don't specify a packet's direction, and they could be used in therefore don't specify a packet's direction, and they could be used
other rule sets to group together flows for well-known ports. in other rule sets to group together flows for well-known ports.
The last example (below) meters flows from a remote router, and The last example (below) meters flows from a remote router, and
demonstrates another approach to grouping well-known ports. demonstrates another approach to grouping well-known ports.
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = SourceAdjacentAddress & FF-FF-FF-FF-FF-FF =
00-60-3E-10-E0-A1: Goto, gateway; # tmkr2 router 00-60-3E-10-E0-A1: Goto, gateway; # tmkr2 router
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-60-3E-10-E0-A1: DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-60-3E-10-E0-A1:
Goto, gateway; # Source is tmkr2 Goto, gateway; # Source is tmkr2
Null & 0 = 0: Ignore, 0; Null & 0 = 0: Ignore, 0;
# #
gateway: gateway:
SourcePeerType & 255 = IP: GotoAct, IP`pkt; SourcePeerType & 255 = IP: GotoAct, IP_pkt;
Null & 0 = 0: GotoAct, Next; Null & 0 = 0: GotoAct, Next;
SourcePeerType & 255 = 0: CountPkt, 0; SourcePeerType & 255 = 0: CountPkt, 0;
# #
IP`pkt: IP_pkt:
SourceTransType & 255 = tcp: PushRuleto, tcp`udp; SourceTransType & 255 = tcp: PushRuleto, tcp_udp;
SourceTransType & 255 = udp: PushRuleto, tcp`udp; SourceTransType & 255 = udp: PushRuleto, tcp_udp;
Null & 0 = 0: GotoAct, not`wkp; # Unknown transport type Null & 0 = 0: GotoAct, not_wkp; # Unknown transport type
# #
tcp`udp: tcp_udp:
SourceTransAddress & FC-00 = 0: GotoAct, well`known`port; SourceTransAddress & FC-00 = 0: GotoAct, well_known_port;
DestTransAddress & FC-00 = 0: Retry, 0; DestTransAddress & FC-00 = 0: NoMatch, 0;
Null & 0 = 0: GotoAct, not`wkp; Null & 0 = 0: GotoAct, not_wkp;
# #
not`wkp: not_wkp:
DestTransAddress & 255.255 = 0: PushPkttoAct, Next; DestTransAddress & 255.255 = 0: PushPkttoAct, Next;
well`known`port: well_known_port:
SourcePeerType & 255 = 0: PushPkttoAct, Next; SourcePeerType & 255 = 0: PushPkttoAct, Next;
DestPeerType & 255 = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next; SourcePeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next; DestPeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next;
SourceTransType & 255 = 0: PushPkttoAct, Next; SourceTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: CountPkt, 0; DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: CountPkt, 0;
The first group of rules test incoming packets' AdjacentAddresses to see The first group of rules test incoming packet's AdjacentAddresses to
whether they belong to a flow with an end point at the specified router. see whether they belong to a flow with an end point at the specified
Any which don't are ignored. Non-IP packets are counted in flows which router. Any which don't are ignored. Non-IP packets are counted in
only have their PeerType Pushed; these will produce one flow for each flows which only have their PeerType Pushed; these will produce one
non-IP protocol. IP packets with TransTypes other than UDP and TCP are flow for each non-IP protocol. IP packets with TransTypes other than
counted at not_wkp, where all their address attributes are pushed. UDP and TCP are counted at not_wkp, where all their address
attributes are pushed.
The high-order six bits of SourceTransAddress for UDP and TCP packets The high-order six bits of SourceTransAddress for UDP and TCP packets
are compared with zero. If this succeeds their source port number is are compared with zero. If this succeeds their source port number is
less than 1024, so they are from a well-known port. The port number is less than 1024, so they are from a well-known port. The port number
pushed from the rule into the flow's SourceTransAddress attribute, and is pushed from the rule into the flow's SourceTransAddress attribute,
the packet is counted at well_known_port. If the test fails, it is and the packet is counted at well_known_port. If the test fails, it
repeated on the packet's DestTransAddress. If the destination is a is repeated on the packet's DestTransAddress. If the destination is
well-known port the match is Retried, and will succeed with the a well-known port the match is Retried, and will succeed with the
well-known port as the flow's source. well-known port as the flow's source.
If later analysis shows were to show that a high proportion of the If later analysis were to show that a high proportion of the observed
observed flows were from non-well-known ports, further pairs of rules flows were from non-well-known ports, further pairs of rules could be
could be added to perform a test in each direction for other added to perform a test in each direction for other heavily-used
heavily-used ports. ports.
4 Flow Data Files 4 Flow data files
Although the Architecture document [1] specifies - in great detail - how Although the Architecture document [1] specifies - in great detail -
the Traffic Flow Meter works, and how a meter reader should collect flow how the Traffic Flow Meter works, and how a meter reader should
data from a meter, it does not say anything about how the collected data collect flow data from a meter, it does not say anything about how
should be stored. NeMaC uses a simple, self-documenting file format, the collected data should be stored. NeMaC uses a simple, self-
which has proved to be very effective in use. documenting file format, which has proved to be very effective in
use.
There are two kinds of records in a flow data file: flow records and There are two kinds of records in a flow data file: flow records and
information records. Each flow record is simply a sequence of attribute information records. Each flow record is simply a sequence of
values separators (these can be specified in a NeMaC rule file) or attribute values with separators (these can be specified in a NeMaC
spaces between them, terminated by a newline. rule file) or spaces between them, terminated by a newline.
Information records all start with a cross-hatch. The file's first Information records all start with a cross-hatch. The file's first
record begins with ##, and identifies the file as being a file of data record begins with ##, and identifies the file as being a file of
from NeTraMet. It records NeMaC's parameters and the time this data from NeTraMet. It records NeMaC's parameters and the time this
collection was started. The file's second record begins with #Format: collection was started. The file's second record begins with
and is a copy of the Format statement used by NeMaC to collect the data. #Format: and is a copy of the Format statement used by NeMaC to
collect the data.
The rest of the file is a sequence of collected data sets. Each of The rest of the file is a sequence of collected data sets. Each of
these starts with a #Time: record, giving the time of day the these starts with a #Time: record, giving the time-of-day the
collection was started, the meter name, and the range of meter times collection was started, the meter name, and the range of meter times
this collection represents. These from and to times are meter UpTimes, this collection represents. These from and to times are meter
i.e. they are times in hundredths of seconds since the meter commenced UpTimes, i.e. they are times in hundredths of seconds since the meter
operation. The flow records which comprise a data set follow the #Time commenced operation. Most analysis applications have simply used the
record. collection start times (which are ASCII time-of-day values), but the
from and to times could be used to convert Uptime values to time-of-
day. The flow records which comprise a data set follow the #Time
record.
4.1 Sample Flow Data File 4.1 Sample flow data file
A sample flow data file appears below. Most of the flow records have A sample flow data file appears below. Most of the flow records have
been deleted, but lines of dots show where they were. been deleted, but lines of dots show where they were.
##NeTraMet v3.2. -c300 -r rules.lan -e rules.default ##NeTraMet v3.2. -c300 -r rules.lan -e rules.default
test`meter -i eth0 4000 flows starting at 12:31:27 Wed 1 Feb 95 test_meter -i eth0 4000 flows starting at 12:31:27 Wed 1 Feb 95
#Format: flowruleset flowindex firsttime sourcepeertype #Format: flowruleset flowindex firsttime sourcepeertype
sourcepeeraddress destpeeraddress topdus frompdus sourcepeeraddress destpeeraddress topdus frompdus
tooctets fromoctets tooctets fromoctets
#Time: 12:31:27 Wed 1 Feb 95 130.216.14.251 Flows #Time: 12:31:27 Wed 1 Feb 95 130.216.14.251 Flows
from 1 to 3642 from 1 to 3642
1 2 13 5 31.32.0.0 33.34.0.0 1138 0 121824 0 1 2 13 5 31.32.0.0 33.34.0.0 1138 0 121824 0
1 3 13 2 11.12.0.0 13.14.0.0 4215 0 689711 0 1 3 13 2 11.12.0.0 13.14.0.0 4215 0 689711 0
1 4 13 7 41.42.0.0 43.34.0.0 1432 0 411712 0 1 4 13 7 41.42.0.0 43.34.0.0 1432 0 411712 0
1 5 13 6 21.22.0.0 23.24.0.0 8243 0 4338744 0 1 5 13 6 21.22.0.0 23.24.0.0 8243 0 4338744 0
3 6 3560 2 130.216.14.0 130.216.3.0 0 10 0 1053 3 6 3560 2 130.216.14.0 130.216.3.0 0 10 0 1053
skipping to change at page 27, line 23 skipping to change at page 28, line 42
3 9 3560 2 130.216.14.0 130.216.14.0 24980 1 5051834 60 3 9 3560 2 130.216.14.0 130.216.14.0 24980 1 5051834 60
3 10 3560 6 130.216.0.28 130.216.0.192 20087 1 8800070 66 3 10 3560 6 130.216.0.28 130.216.0.192 20087 1 8800070 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 164 0 32608 3 11 3560 7 0.0.255.0 0.128.113.0 0 164 0 32608
3 12 3560 5 59.3.13.0 4.1.152.0 41 41 2460 2460 3 12 3560 5 59.3.13.0 4.1.152.0 41 41 2460 2460
3 14 3560 5 59.3.40.0 4.1.153.0 82 82 4920 4920 3 14 3560 5 59.3.40.0 4.1.153.0 82 82 4920 4920
3 15 3560 5 0.0.0.0 4.1.153.0 0 60 0 3600 3 15 3560 5 0.0.0.0 4.1.153.0 0 60 0 3600
. . . . . . . . . . . . . . . . . .
4.2 Flow data file features 4.2 Flow data file features
Several features of NeMaC's flow data files (as indicated above) are Several features of NeMaC's flow data files (as indicated above) are
worthy of note: worthy of note:
- Collection times overlap slightly between samples. This allows for - Collection times overlap slightly between samples. This allows for
flows which were created after the collection started, and makes flows which were created after the collection started, and makes
sure that flows are not missed from a collection. sure that flows are not missed from a collection.
- The rule set may change during a run. The above shows flows from - The rule set may change during a run. The above shows flows from
rule set 1 - the default set - in the first collection, followed by rule set 1 - the default set - in the first collection, followed by
the first flows created by rule set 3 (which has just been the first flows created by rule set 3 (which has just been
downloaded by NeMaC). downloaded by NeMaC).
skipping to change at page 28, line 9 skipping to change at page 29, line 30
performing an unsigned subtraction and ignoring any carry. performing an unsigned subtraction and ignoring any carry.
- In the sample flow data file above I have used double spaces as - In the sample flow data file above I have used double spaces as
separators between the flow identifiers, peer addresses, pdu counts separators between the flow identifiers, peer addresses, pdu counts
and packet counts. and packet counts.
- The format of addresses in the flow data file depends on the type - The format of addresses in the flow data file depends on the type
of address. NeMaC always displays Adjacent addresses as six hex of address. NeMaC always displays Adjacent addresses as six hex
bytes separated by hyphens, and Transport addresses as (16-bit) bytes separated by hyphens, and Transport addresses as (16-bit)
integers. The format of a Peer address depends on its PeerType, integers. The format of a Peer address depends on its PeerType,
e.g. dotted decimal for IP. To facilitate this NeMaC needs to know e.g. dotted decimal for IP. To facilitate this NeMaC needs to know
the PeerType for each flow; the user must request NeMaC to collect the PeerType for each flow; the user must request NeMaC to collect
it. it.
4.3 Terminating and restarting meter reading 4.3 Terminating and restarting meter reading
When NeMaC first starts collecting from a meter, it reads the flow data When NeMaC first starts collecting from a meter, it reads the flow
for all active flows. This provides a starting point for analysis data for all active flows. This provides a starting point for
applications to compute the counts between successive collections. analysis applications to compute the counts between successive
collections.
From time to time the user needs to terminate a flow data file and begin From time to time the user needs to terminate a flow data file and
a new one. For example, a user might need to generate a separate file begin a new one. For example, a user might need to generate a
for each day of metering. NeMaC provides for this by closing the file separate file for each day of metering. NeMaC provides for this by
after each collection, then opening it and appending the data from the closing the file after each collection, then opening it and appending
next collection. To terminate a file the user simply renames it. The the data from the next collection. To terminate a file the user
Unix system will effect the name change either immediately (if the file simply renames it. The Unix system will effect the name change
was closed) or as soon as the current collection is complete (and the either immediately (if the file was closed) or as soon as the current
file is closed). collection is complete (and the file is closed).
When NeMaC begins its next collection it observes that the file has When NeMaC begins its next collection it observes that the file has
disappeared, so it creates a new one and writes the # header records disappeared, so it creates a new one and writes the # header records
before writing the collected data. before writing the collected data.
There is one aspect of the above which requires some care on the user's There is one aspect of the above which requires some care on the
part. The last data set in a file is not duplicated as the first data user's part. The last data set in a file is not duplicated as the
set of the next file. In other words, analysis applications must either first data set of the next file. In other words, analysis
look ahead at the first data set of the next file, or begin by reading applications must either look ahead at the first data set of the next
the last data set of the previous file. If they fail to do this they file, or begin by reading the last data set of the previous file. If
will loose one collection's worth of flow data at each change of file. they fail to do this they will loose one collection's worth of flow
data at each change of file.
5 Analysis applications 5 Analysis applications
Most analysis applications will be unique, taking data produced by a Most analysis applications will be unique, taking data produced by a
locally- developed rule set and producing reports to satisfy specific locally-developed rule set and producing reports to satisfy specific
local requirements. The NeTraMet distribution files include three local requirements. The NeTraMet distribution files include three
applications which are of general use, as follows: applications which are of general use, as follows:
- fd_filter computes data rates, i.e. the differences between - fd_filter computes data rates, i.e. the differences between
successive data sets in a flow data file. It also allows the user successive data sets in a flow data file. It also allows the user
to assign a 'tag' number to each flow; these are 'computed' to assign a 'tag' number to each flow; these are 'computed'
attributes similar to FlowClass and FlowKind - the only difference attributes similar to FlowClass and FlowKind - the only difference
is that they are computed from the collected data sets. is that they are computed from the collected data sets.
- fd_extract takes 'tagged' files from fd_filter and produces simple - fd_extract takes 'tagged' files from fd_filter and produces simple
'column list' files for use by other programs. One common use for 'column list' files for use by other programs. One common use for
fd_extract is to produce time-series data files which can be plotted fd_extract is to produce time-series data files which can be plotted
by utilities like GNUPlot. by utilities like GNUPlot.
- nm_rc is a 'remote console' for a NeTraMet meter. It is a slightly - nm_rc is a 'remote console' for a NeTraMet meter. It is a slightly
simplified version of NeMaC combined with fd_filter. It can be used simplified version of NeMaC combined with fd_filter. It can be used
to monitor any meter, and will display (as lines of text to monitor any meter, and will display (as lines of text
characters) information about the n busiest flows observed during characters) information about the n busiest flows observed during
each collection interval. each collection interval.
These applications are useful in themselves, and they provide a good - nifty is a traffic flow analyser, which (like nm_rc) displays data
starting point for users who wish to write their own analysis from a NeTraMet meter. nifty is an X/Motif application, which
applications. produces displays like 'Packet rate (pps) vs Flow lifetime
(minutes),' so as to highlight those flows which are 'interesting.'
These applications are useful in themselves, and they provide a good
starting point for users who wish to write their own analysis
applications.
6 Using NeTraMet in a measurement system 6 Using NeTraMet in a measurement system
This section gives a brief summary of the steps involved in setting up a This section gives a brief summary of the steps involved in setting
traffic measurement system using NeTraMet. These are: up a traffic measurement system using NeTraMet. These are:
- Decide what is to be measured. One good way to approach this is to - Decide what is to be measured. One good way to approach this is to
specify exactly which flows are to be measured, and what reports specify exactly which flows are to be measured, and what reports
will be required. Specifying the flows should make it obvious will be required. Specifying the flows should make it obvious
where meters will have to be placed so that the flows can be where meters will have to be placed so that the flows can be
observed, whether PCs will be adequate for the task, etc.. observed, whether PCs will be adequate for the task, etc..
- Install meters. As well as actually placing the meter hosts this - Install meters. As well as actually placing the meter hosts this
includes making sure that they are configured correctly, with includes making sure that they are configured correctly, with
appropriate IP addresses, SNMP community strings, etc. appropriate IP addresses, SNMP community strings, etc.
- Develop the rule set (and a standby rule set). The degree of - Develop the rule set (and a standby rule set). The degree of
difficulty here depends on how much is known in advance about the difficulty here depends on how much is known in advance about the
traffic. One possible approach is to start with the meter default traffic. One possible approach is to start with the meter default
rule set and measure how much traffic there is for each PeerType. rule set and measure how much traffic there is for each PeerType.
(This is a good way to verify that NeTraMet and NeMaC are working (This is a good way to verify that NeTraMet and NeMaC are working
properly). You can now add rules so as to increase the granularity properly). You can now add rules so as to increase the granularity
of the flows; this will of course increase the number of flows to of the flows; this will of course increase the number of flows to
be collected, and force the meter's garbage collector to work be collected, and force the meter's garbage collector to work
harder. Another approach is to try a rule set with very fine harder. Another approach is to try a rule set with very fine
granularity (i.e. one which Pushes all the address attributes), granularity (i.e. one which Pushes all the address attributes),
then observing how many flows are collected every few minutes. then observing how many flows are collected every few minutes.
- Develop a strategy for controlling meter reader. This means - Develop a strategy for controlling meter reader. This means
setting the meter's maximum number of flows, the collection setting the meter's maximum number of flows, the collection
interval, how breaks between flow data files will be handled, how interval, how breaks between flow data files will be handled, how
often NeMaC should check that the meter is running, etc. often NeMaC should check that the meter is running, etc.
- Develop application(s) to process the collected flow data and - Develop application(s) to process the collected flow data and
produce the required files and reports. produce the required files and reports.
- Test run. Monitor the system, then refine the rule sets and meter - Test run. Monitor the system, then refine the rule sets and meter
reading strategy until the overall system performance is reading strategy until the overall system performance is
satisfactory. satisfactory.
This process can take quite a long time, but the overall result is well This process can take quite a long time, but the overall result is
worth the effort. well worth the effort.
At the University of Auckland we run two sets of meters. One of these 6.1 Examples of NeTraMet in production use
measures the traffic entering and leaving our University network, and
generates usage reports for all our Internet users. This has been in
production since about mid-1995.
The other set consists of meters which are distributed at Universities At the University of Auckland we run two sets of meters. One of
throughout New Zealand. They provide continuous traffic flow these measures the traffic entering and leaving our University
measurements at five-minute intervals for all the links making up the network, and generates usage reports for all our Internet users.
Universities' network (Kawaihiko); this system has been in production This has been in production since early 1994.
since January 1996, and has already proved very useful in planning the
network's development.
7 References The other set consists of meters which are distributed at
Universities throughout New Zealand. They provide continuous traffic
flow measurements at five-minute intervals for all the links making
up the Universities' network (Kawaihiko); this system has been in
production since January 1996, and has already proved very useful in
planning the network's development.
8 Acknowledgments The Kawaihiko Network provides IP connectivity for the New Zealand
Universities. They are linked via a Frame Relay cloud, using a
partial mesh of permanent virtual circuits. There is a NeTraMet
meter at each site, metering inward and outward traffic. All the
meters are managed from Auckland, and they all run copies of the same
rule set.
This memo documents the implementation work on traffic flow measurement The rule set has about 650 rules, most of which are in a single
here at the University of Auckland. Many of my University colleagues subroutine which classifies PeerAddresses into three categories -
have contributed significantly to this work, especially Russell Fulton 'Kawaihiko network,' 'other New Zealand network' and 'non-New Zealand
(who developed the rules sets, Perl scripts and Cron jobs which produce network.' Inside New Zealand IP addresses lie within six CIDR
our traffic usage reports automatically week after week) and John White blocks, and there are about four hundred older networks which have
(for his patient help in documenting the project). addresses outside those blocks. The rules are arranged in groups by
subnet size, i.e. all the /24 networks are tested first, then the /23
networks, etc, finishing with the /16 networks. This means that
although there are about 600 networks, any PeerAddress can be
classified with only nine tests.
9 References The Kawaihiko rule set classifies flows, using computed attributes to
indicate the network 'kind' (Kawaihiko / New Zealand / international)
for each flow's SourcePeerAddress and DestPeerAddress, and to
indicate whether the flow is a 'network news' flow or not.
[1] Brownlee, N., Mills, C., and Ruth, G., "Traffic Flow Flow data is collected from all of the meters every five minutes, and
Measurement: Architecture," Internet Draft (work in progress), is used to produce weekly reports, as follows:
The University of Auckland, Bolt Beranek and Newman Inc., GTE
Laboratories, Inc, April 1996.
[2] Brownlee, N., "Traffic Flow Measurement: Meter MIB," - Traffic Plots. Plots of the 5-minute traffic rates for each site,
Internet Draft (work in progress), The University of Auckland, showing international traffic in and out, news traffic in and out,
April 1996. and total traffic in and out of the site.
- Traffic Matrices. Two of these are produced, one for news traffic,
the other for total traffic. They show the traffic rates from
every site (including 'other New Zealand' and 'international') to
every other site. The mean, third quartile and maximum are printed
for every cell in the matrices.
7 Acknowledgments
This memo documents the implementation work on traffic flow
measurement here at the University of Auckland. Many of my
University colleagues have contributed significantly to this work,
especially Russell Fulton (who developed the rules sets, Perl scripts
and Cron jobs which produce our traffic usage reports automatically
week after week) and John White (for his patient help in documenting
the project).
8 References
[1] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow
Measurement: Architecture", RFC 2063, The University of Auckland,
Bolt Beranek and Newman Inc., GTE Laboratories, Inc, January 1997.
[2] Brownlee, N., "Traffic Flow Measurement: Meter MIB",
RFC 2064, The University of Auckland, January 1997.
[3] CRYNWR Packer Drivers distribution site: [3] CRYNWR Packer Drivers distribution site:
http://www.crynwr.com/ http://www.crynwr.com/
[4] Case J., McCloghrie K., Rose M., and Waldbusser S., [4] Case J., McCloghrie K., Rose M., and Waldbusser S.,
"Structure of Management Information for version 2 of the "Structure of Management Information for version 2 of the
Simple Network Managemenet Protocol," RFC 1902, SNMP Research Simple Network Managemenet Protocol", RFC 1902, SNMP Research
Inc., Hughes LAN Systems, Dover Beach Consulting, Carnegie Inc., Hughes LAN Systems, Dover Beach Consulting, Carnegie
Mellon University, April 1993. Mellon University, April 1993.
[5] IBM Corporation, "IBM PC Technical Reference Manual," 1984. [5] IBM Corporation, "IBM PC Technical Reference Manual," 1984.
[6] Waterloo TCP distribution site: http://www.kiae.su:8090/ [6] Waterloo TCP distribution site:
___[Kiarchive_root_directory]/msdos/tcpip/waterloo http://mvmpc9.ciw.uni-karlsruhe.de:80/d:/public/tcp_ip/wattcp
[7] CMU SNMP distribution site: [7] CMU SNMP distribution site:
ftp://lancaster.andrew.cmu.edu/pub/snmp-dist ftp://lancaster.andrew.cmu.edu/pub/snmp-dist
[8] libpcap distribution site: [8] libpcap distribution site:
ftp://ftp.ee.lbl.gov/libpcap-*.tar.gz ftp://ftp.ee.lbl.gov/libpcap-*.tar.gz
10 Security Considerations 9 Security Considerations
Security issues are not discussed in detail in this document. The Security issues are not discussed in detail in this document. The
meter's management and collection protocols are responsible for meter's management and collection protocols are responsible for
providing sufficient data integrity and confidentiality. providing sufficient data integrity and confidentiality.
11 Author's Address 10 Author's Address
Nevil Brownlee Nevil Brownlee
The University of Auckland The University of Auckland
Phone: +64 9 373 7599 x8941 Phone: +64 9 373 7599 x8941
E-mail: n.brownlee@auckland.ac.nz Email: n.brownlee@auckland.ac.nz
 End of changes. 188 change blocks. 
1012 lines changed or deleted 1103 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/