draft-ietf-bmwg-mcast-08.txt   rfc2432.txt 
Network Working Group Debra Stopp Network Working Group K. Dubray
Hardev Soor Request for Comments: 2432 IronBridge Networks
INTERNET-DRAFT IXIA Category: Informational October 1998
Expires in: November 2002
Methodology for IP Multicast Benchmarking
<draft-ietf-bmwg-mcastm-08.txt>
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved.
Abstract
The purpose of this draft is to describe methodology specific to
the benchmarking of multicast IP forwarding devices. It builds upon
the tenets set forth in RFC 2544, RFC 2432 and other IETF
Benchmarking Methodology Working Group (BMWG) efforts. This
document seeks to extend these efforts to the multicast paradigm.
The BMWG produces two major classes of documents: Benchmarking
Terminology documents and Benchmarking Methodology documents. The
Terminology documents present the benchmarks and other related
terms. The Methodology documents define the procedures required to
collect the benchmarks cited in the corresponding Terminology
documents.
Table of Contents
1. INTRODUCTION...................................................3
2. KEY WORDS TO REFLECT REQUIREMENTS..............................3
3. TEST SET UP....................................................3
3.1. Test Considerations..........................................5
3.1.1. IGMP Support..............................................5
3.1.2. Group Addresses...........................................5
3.1.3. Frame Sizes...............................................5
3.1.4. TTL.......................................................6
3.1.5. Trial Duration............................................6
3.2. Layer 2 Support..............................................6
4. FORWARDING AND THROUGHPUT......................................6
4.1. Mixed Class Throughput.......................................6
4.2. Scaled Group Forwarding Matrix...............................7
4.3. Aggregated Multicast Throughput..............................8
4.4. Encapsulation/Decapsulation (Tunneling) Throughput...........9
4.4.1. Encapsulation Throughput..................................9
4.4.2. Decapsulation Throughput..................................9
4.4.3. Re-encapsulation Throughput..............................10
5. FORWARDING LATENCY............................................10
5.1. Multicast Latency...........................................11
5.2. Min/Max Multicast Latency...................................13
6. OVERHEAD......................................................14
6.1. Group Join Delay............................................14
6.2. Group Leave Delay...........................................15
7. CAPACITY......................................................16
7.1. Multicast Group Capacity....................................16
8. INTERACTION...................................................16
8.1. Forwarding Burdened Multicast Latency.......................17
8.2. Forwarding Burdened Group Join Delay........................17
9. SECURITY CONSIDERATIONS.......................................17
10. ACKNOWLEDGEMENTS.............................................17
11. REFERENCES...................................................18
12. AUTHOR'S ADDRESSES...........................................19
13. FULL COPYRIGHT STATEMENT.....................................19
1. Introduction
This document defines a specific set of tests that vendors can use
to measure and report the performance characteristics and
forwarding capabilities of network devices that support IP
multicast protocols. The results of these tests will provide the
user comparable data from different vendors with which to evaluate
these devices.
A previous document, " Terminology for IP Multicast Benchmarking"
(RFC 2432), defined many of the terms that are used in this
document. The terminology document should be consulted before
attempting to make use of this document.
This methodology will focus on one source to many destinations,
although many of the tests described may be extended to use
multiple source to multiple destination IP multicast communication.
2. Key Words to Reflect Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
in this document are to be interpreted as described in RFC 2119.
RFC 2119 defines the use of these key words to help make the intent
of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards
track document.
3. Test set up
The set of methodologies presented in this draft are for single
ingress, multiple egress scenarios as exemplified by Figures 1 and
2. Methodologies for multiple ingress, multiple egress scenarios
are beyond the scope of this document.
Figure 1 shows a typical setup for an IP multicast test, with one
source to multiple destinations.
+----------------+
+------------+ | Egress |
+--------+ | (-)-------->| destination(E1)|
| | | | | |
| source |------->(|)Ingress | +----------------+
| | | | +----------------+
+--------+ | D U T (-)-------->| Egress |
| | | destination(E2)|
| | | |
| | +----------------+
| | . . .
| | +----------------+
| | | Egress |
| (-)-------->| destination(En)|
| | | |
+------------+ +----------------+
Figure 1
---------
If the multicast metrics are to be taken across multiple devices
forming a System Under Test (SUT), then test packets are offered to
a single ingress interface on a device of the SUT, subsequently
routed across the SUT topology, and finally forwarded to the test
apparatus' packet-receiving components by the test egress
interface(s) of devices in the SUT. Figure 2 offers an example SUT
test topology. If a SUT is tested, the details of the test
topology MUST be disclosed with the corresponding test results.
+--------+ +----------------+ +--------+
| | +------------+ |DUT B Egress E0(-)-->| |
| | |DUT A |--->| | | |
| Test | | | | Egress E1(-)-->| Test |
| App. |--->(-)Ingress, I | +----------------+ | App. |
| Traffic| | | +----------------+ | Traffic|
| Src. | | |--->|DUT C Egress E2(-)-->| Dest. |
| | +------------+ | | | |
| | | Egress En(-)-->| |
+--------+ +----------------+ +--------+
Figure 2
---------
Generally, the destination ports first join the desired number of
multicast groups by sending IGMP Join Group messages to the
DUT/SUT. To verify that all destination ports successfully joined
the appropriate groups, the source port MUST transmit IP multicast
frames destined for these groups. The destination ports MAY send
IGMP Leave Group messages after the transmission of IP Multicast
frames to clear the IGMP table of the DUT/SUT.
In addition, test equipment MUST validate the correct and proper
forwarding actions of the devices they test in order to ensure the
receipt of only the frames that are involved in the test.
3.1. Test Considerations
The procedures outlined below are written without regard for
specific physical layer or link layer protocols. The methodology
further assumes a uniform medium topology. Issues regarding mixed
transmission media, such as speed mismatch, headers differences,
etc., are not specifically addressed. Flow control, QoS and other
traffic-affecting mechanisms MUST be disabled. Modifications to
the specified collection procedures might need to be made to
accommodate the transmission media actually tested. These
accommodations MUST be presented with the test results.
3.1.1. IGMP Support
Each of the destination ports should support and be able to test
all IGMP versions 1, 2 and 3. The minimum requirement, however, is
IGMP version 2.
Each destination port should be able to respond to IGMP queries
during the test.
Each destination port should also send LEAVE (running IGMP version
2) after each test.
3.1.2. Group Addresses
The Class D Group address SHOULD be changed between tests. Many
DUTs have memory or cache that is not cleared properly and can bias
the results.
The following group addresses are recommended by use in a test:
224.0.1.27-224.0.1.255
224.0.5.128-224.0.5.255
224.0.6.128-224.0.6.255
If the number of group addresses accommodated by these ranges does
not satisfy the requirements of the test, then these ranges may be
overlapped. The total number of configured group addresses must be
less than or equal to the IGMP table size of the DUT/SUT.
3.1.3. Frame Sizes
Each test SHOULD be run with different Multicast Frame Sizes. The
recommended frame sizes are 64, 128, 256, 512, 1024, 1280, and 1518
byte frames.
3.1.4. TTL Terminology for IP Multicast Benchmarking
The source frames should have a TTL value large enough to Status of this Memo
accommodate the DUT/SUT.
3.1.5. Trial Duration This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
The duration of the test portion of each trial SHOULD be at least Copyright Notice
30 seconds. This parameter MUST be included as part of the results
reporting for each methodology.
3.2. Layer 2 Support Copyright (C) The Internet Society (1998). All Rights Reserved.
Each of the destination ports should support GARP/GMRP protocols to Abstract
join groups on Layer 2 DUTs/SUTs.
4. Forwarding and Throughput The purpose of this document is to define terminology specific to the
benchmarking of multicast IP forwarding devices. It builds upon the
tenets set forth in RFC 1242, RFC 2285, and other IETF Benchmarking
Methodology Working Group (BMWG) efforts. This document seeks to
extend these efforts to the multicast paradigm.
This section contains the description of the tests that are related The BMWG produces two major classes of documents: Benchmarking
to the characterization of the packet forwarding of a DUT/SUT in a Terminology documents and Benchmarking Methodology documents. The
multicast environment. Some metrics extend the concept of throughput Terminology documents present the benchmarks and other related terms.
presented in RFC 1242. The notion of Forwarding Rate is cited in RFC The Methodology documents define the procedures required to collect
2285. the benchmarks cited in the corresponding Terminology documents.
4.1. Mixed Class Throughput 1. Introduction
Objective Network forwarding devices are being required to take a single frame
and support delivery to a number of destinations having membership to
a particular group. As such, multicast support may place a different
burden on the resources of these network forwarding devices than with
unicast or broadcast traffic types.
To determine the maximum throughput rate at which none of the Such burdens may not be readily apparent at first glance - the IP
offered frames, comprised from a unicast Class and a multicast multicast packet's Class D address may be the only noticeable
Class, to be forwarded are dropped by the device across a fixed difference from an IP unicast packet. However, there are many
number of ports as defined in RFC 2432. factors that may impact the treatment of IP multicast packets.
Procedure Consider how a device's architecture may impact the handling of a
multicast frame. For example, is the multicast packet subject to the
same processing as its unicast analog? Or is the multicast packet
treated as an exeception and processed on a different data path?
Consider, too, how a shared memory architecture may demonstrate a
different performance profile than an architecture which explicitly
passes each individual packet between the processing entities.
Multicast and unicast traffic are mixed together in the same In addition to forwarding device architecture, there are other
aggregated traffic stream in order to simulate the non-homogenous factors that may impact a device's or system's multicast related
networking environment. The DUT/SUT MUST learn the appropriate performance. Protocol requirements may demand that routers and
unicast IP addresses, either by sending ARP frames from each switches consider destination and source addressing in its multicast
unicast address, sending a RIP packet or by assigning static forwarding decisions. Capturing multicast source/destination
entries into the DUT/SUT address table. addressing information may impact forwarding table size and lengthen
lookups. Topological factors such as the degree of packet
replication, the number of multicast groups being supported by the
system, or the placement of multicast packets in unicast wrappers to
span non-multicast network paths may all potentially affect a
system's multicast related performance. For an overall understanding
of IP multicasting, the reader is directed to [Se98], [Hu95], and
[Mt98].
The mixture of multicast and unicast traffic MUST be set up in one By clearly identifying IP multicast benchmarks and related
of two ways: terminology in this document, it is hoped that detailed methodologies
can be generated in subsequent documents. Taken in tandem, these two
efforts endeavor to assist the clinical, empirical, and consistent
characterization of certain aspects of multicast technologies and
their individual implementations. Understanding the operational
profile of multicast forwarding devices may assist the network
designer to better deploy multicast in his or her networking
environment.
a) Input frame rate for each class of traffic [Br91] or as a Moreover, this document focuses on one source to many destinations
percentage of media_maximum-octets [Ma98]. Frame rate should profiling. Elements of this document may require extension when
be specified independently for each traffic class. considering multiple source to multiple destination IP multicast
communication.
b) As an aggregate rate (given either in frames per second or 2. Definition Format
as a percentage), with the ratio of multicast to unicast
traffic declared.
While the multicast traffic is transmitted from one source to This section cites the template suggested by RFC 1242 in the
multiple destinations, the unicast traffic MAY be evenly specification of a term to be defined.
distributed across the DUT/SUT architecture. Unicast traffic
distribution can either be non-meshed or meshed [Ma98] as specified
in RFC2544 or RFC2289.
Throughput measurement is defined in RFC1242 [Br91]. A search Term to be defined.
algorithm MUST be utilized to determine the maximum offered frame
rate with a zero frame loss rate.
Result Definition:
The specific definition for the term.
Parameters to be measured MUST include the aggregate offered load, Discussion:
number of multicast frames offered, number of unicast frames A brief discussion of the term, its application, or other
offered, number multicast frames received, number of unicast frames information that would build understanding.
received and transmit duration of offered frames.
4.2. Scaled Group Forwarding Matrix Measurement units:
Units used to record measurements of this term, if applicable.
Objective [Issues:]
List of issues or conditions that affect this term. This field can
present items the may impact the term's related methodology or
otherwise restrict its measurement procedures. This field is
optional in this document.
A table that demonstrates Forwarding Rate as a function of tested [See Also:]
multicast groups for a fixed number of tested DUT/SUT ports. List of other terms that are relevant to the discussion of this
term. This field is optional in this document.
Procedure 2.1 Existing Terminology
Multicast traffic is sent at a fixed percent of maximum offered This document draws on existing terminology defined in other BMWG
load with a fixed number of receive ports of the tester at a fixed work. Examples include, but are not limited to:
frame length.
On each iteration, the receive ports SHOULD incrementally join 10 Throughput [RFC 1242, section 3.17]
multicast groups until a user defined maximum number of groups is Latency [RFC 1242, section 3.8]
reached. Constant Load [RFC 1242, section 3.4]
Frame Loss Rate [RFC 1242, section 3.6]
Overhead behavior [RFC 1242, section 3.11]
Forwarding Rates [RFC 2285, section 3.6]
Loads [RFC 2285, section 3.5]
Device Under Test (DUT) [RFC 2285, section 3.1.1]
System Under Test (SUT) [RFC 2285, section 3.1.2]
Results Note: "DUT/SUT" refers to a metric that may be applicable to a DUT or
SUT.
Parameters to be measured MUST include the offered load and 3. Table of Defined Terms
forwarding rate as a function of the total number of multicast
groups, for each test iteration.
The nature of the traffic stream contributing to the result MUST be 3.1 General Nomenclature
reported, specifically number of source and destination ports
within the multicast group. In addition, all other reporting
parameters of the scaled group forwarding matrix methodology MUST
be reflected in the results report, such as the transmitted packet
size(s) and offered load of the packet stream for each source port.
Result reports MUST include the following parameters for each 3.1.1 Traffic Class. (TC)
iteration: the number of frames offered, number of frames received 3.1.2 Group Class. (GC)
per each group, number of multicast groups and forwarding rate, in 3.1.3 Service Class. (SC)
frames per second, and transmit duration of offered frames.
Constructing a table that contains the forwarding rate vs. number
of groups is desirable.
4.3. Aggregated Multicast Throughput 3.2 Forwarding and Throughput
3.2.1 Mixed Class Throughput (MCT).
3.2.2 Scaled Group Forwarding Matrix (SGFM).
3.2.3 Aggregated Multicast Throughput (AMT)
3.2.4 Encapsulation Throughput (ET)
3.2.5 Decapsulation Throughput (DT)
3.2.6 Re-encapsulation Throughput (RET)
Objective 3.3 Forwarding Latency
3.3.1 Multicast Latency (ML)
3.3.2 Min/Max Multicast Latency (Min/Max ML)
The maximum rate at which none of the offered frames to be 3.4 Overhead
forwarded through N destination interfaces of the same multicast 3.4.1 Group Join Delay. (GJD)
group is dropped. 3.4.2 Group Leave Delay. (GLD)
Procedure 3.5 Capacity
3.5.1 Multicast Group Capacity. (MGC)
Multicast traffic is sent at a fixed percent of maximum offered 3.6 Interaction
load with a fixed number of groups at a fixed frame length for a 3.6.1 Burdened Response
fixed duration of time. 3.6.2 Forwarding Burdened Multicast Latency (FBML)
3.6.3 Forwarding Burdened Join Delay (FBJD)
The initial number of receive ports of the tester will join the 3.1 General Nomenclature
group(s) and the sender will transmit to the same groups after a
certain delay (a few seconds).
If any frame loss is detected, one receive port MUST leave the This section will present general terminology to be used in this and
group(s) and the sender will transmit again. Continue in this other documents.
iterative fashion until either there are no ports left joined to
the multicast group(s) OR 0% frame loss is achieved.
Results 3.1.1 Traffic Class. (TC)
Parameters to be measured MUST include the maximum offered load at Definition:
which no frame loss occurred (as defined by RFC 2544) An equivalence class of packets comprising one or more data
streams.
The nature of the traffic stream contributing to the result MUST be Discussion:
reported. All required reporting parameters of aggregated In the scope of this document, Traffic Class will be considered a
throughput MUST be reflected in the results report, such as the logical identifier used to discriminate between a set or sets of
initial number of receive ports, the final number of receive ports, packets offered the DUT.
total number of multicast group addresses, the transmitted packet
size(s), offered load of the packet stream and transmit duration of
offered frames.
Constructing a table from the measurements might be useful in For example, one Traffic Class may identify a set of unicast
illustrating the effect of modifying the number of active egress packets offered to the DUT. Another Traffic Class may
ports on the tested system. differentiate the multicast packets destined to multicast group X.
Yet another Class may distinguish the set of multicast packets
destined to multicast group Y.
4.4. Encapsulation/Decapsulation (Tunneling) Throughput Unless otherwise qualified, the usage of the word "Class" in this
document will refer simply to a Traffic Class.
This sub-section provides the description of tests that help in Measurement units:
obtaining throughput measurements when a DUT/SUT or a set of DUTs Not applicable.
are acting as tunnel endpoints
4.4.1. Encapsulation Throughput 3.1.2 Group Class. (GC)
Objective Definition:
A specific type of Traffic Class where the packets comprising the
Class are destined to a particular multicast group.
The maximum rate at which frames offered a DUT/SUT are encapsulated Discussion:
and correctly forwarded by the DUT/SUT without loss.
Procedure Measurement units:
Not applicable.
Traffic is sent through a DUT/SUT that has been configured to 3.1.3 Service Class. (SC)
encapsulate the frames. Traffic is received on a test port prior to
decapsulation and throughput is calculated based on RFC2544.
Results Definition:
A specific type of Traffic Class where the packets comprising the
Class require particular treatment or treatments by the network
forwarding devices along the path to the packets' destination(s).
Parameters to be measured SHOULD include the measured throughput Discussion:
per tunnel,
The nature of the traffic stream contributing to the result MUST be Measurement units:
reported. All required reporting parameters of encapsulation Not applicable.
throughput MUST be reflected in the results report, such as the
transmitted packet size(s), offered load of the packet stream and
transmit duration of offered frames.
4.4.2. Decapsulation Throughput 3.2 Forwarding and Throughput.
Objective This section presents terminology related to the characterization of
the packet forwarding ability of a DUT/SUT in a multicast
environment. Some metrics extend the concept of throughput presented
in RFC 1242. The notion of Forwarding Rate is cited in RFC 2285.
The maximum rate at which frames offered a DUT/SUT are decapsulated 3.2.1 Mixed Class Throughput (MCT).
and correctly forwarded by the DUT/SUT without loss.
Procedure Definition:
The maximum rate at which none of the offered frames, comprised
from a unicast Class and a multicast Class, to be forwarded are
dropped by the device across a fixed number of ports.
Encapsulated traffic is sent through a DUT/SUT that has been Discussion:
configured to decapsulate the frames. Traffic is received on a test Often times, throughput is collected on a homogenous traffic class
port after decapsulation and throughput is calculated based on - the offered load to the DUT is either singularly unicast or
RFC2544. singularly multicast. In most networking environments, the
traffic mix is seldom so uniformly distributed.
Results Based on the RFC 1242 definition for throughput, the Mixed Class
Throughput benchmark attempts to characterize the DUT's ability to
process both unicast and multicast frames in the same aggregated
traffic stream.
Parameters to be measured SHOULD include the measured throughput Measurement units:
per tunnel. Frames per second
The nature of the traffic stream contributing to the result MUST be Issues:
reported. All required reporting parameters of decapsulation Related methodology may have to address the ratio of unicast
throughput MUST be reflected in the results report, such as the packets to multicast packets.
transmitted packet size(s), offered load of the packet stream and
transmit duration of offered frames.
4.4.3. Re-encapsulation Throughput Since frame size can sometimes be a factor in frame forwarding
benchmarks, the corresponding methodology for this metric will
need to consider frame size distribution(s).
Objective 3.2.2 Scaled Group Forwarding Matrix (SGFM).
The maximum rate at which frames of one encapsulated format offered Definition:
a DUT/SUT are converted to another encapsulated format and A table that demonstrates Forwarding Rate as a function of tested
correctly forwarded by the DUT/SUT without loss. multicast groups for a fixed number of tested DUT/SUT ports.
Procedure Discussion:
A desirable attribute of many Internet mechanisms is the ability
to "scale." This benchmark seeks to demonstrate the ability of a
SUT to forward as the number of multicast groups is scaled
upwards.
Traffic is sent through a DUT/SUT that has been configured to Measurement units:
encapsulate frames into one format, then re-encapsulate the frames Packets per second, with corresponding tested multicast group and
into another format. Traffic is received on a test port after all port configurations.
decapsulation is complete and throughput is calculated based on
RFC2544.
Results Issues:
The corresponding methodology may have to reflect the impact that
the pairing (source, group) has on many multicast routing
protocols.
Parameters to be measured SHOULD include the measured throughput Since frame size can sometimes be a factor in frame forwarding
per tunnel. benchmarks, the corresponding methodology for this metric will
need to consider frame size distribution(s).
The nature of the traffic stream contributing to the result MUST be 3.2.3 Aggregated Multicast Throughput (AMT)
reported. All required reporting parameters of re-encapsulation
throughput MUST be reflected in the results report, such as the
transmitted packet size(s), offered load of the packet stream and
transmit duration of offered frames.
5. Forwarding Latency Definition:
The maximum rate at which none of the offered frames to be
forwarded through N destination interfaces of the same multicast
group are dropped.
This section presents methodologies relating to the Discussion:
characterization of the forwarding latency of a DUT/SUT in a Another "scaling" type of exercise, designed to identify the
multicast environment. It extends the concept of latency DUT/SUT's ability to handle traffic as a function of the multicast
characterization presented in RFC 2544. destination ports it is required to support.
In order to lessen the effect of packet buffering in the DUT/SUT, Measurement units:
the latency tests MUST be run such that the offered load is less The ordered pair (N,t) where,
than the multicast throughput of the DUT/SUT as determined in the
previous section. The tests should also take into account the
DUT's/SUT's need to cache the traffic in its IP cache, fastpath
cache or shortcut tables since the initial part of the traffic will
be utilized to build these tables.
Lastly, RFC 1242 and RFC 2544 draw distinction between two classes N = the number of destination ports of the multicast group.
of devices: "store and forward" and "bit-forwarding." Each class t = the throughput, in frames per second, relative to the
impacts how latency is collected and subsequently presented. See source stream.
the related RFCs for more information. In practice, much of the
test equipment will collect the latency measurement for one class
or the other, and, if needed, mathematically derive the reported
value by the addition or subtraction of values accounting for
medium propagation delay of the packet, bit times to the timestamp
trigger within the packet, etc. Test equipment vendors SHOULD
provide documentation regarding the composition and calculation
latency values being reported. The user of this data SHOULD
understand the nature of the latency values being reported,
especially when comparing results collected from multiple test
vendors. (E.g., If test vendor A presents a "store and forward"
latency result and test vendor B presents a "bit-forwarding"
latency result, the user may erroneously conclude the DUT has two
differing sets of latency values.)
5.1. Multicast Latency Issues:
Since frame size can sometimes be a factor in frame forwarding
benchmarks, the corresponding methodology for this metric will
need to consider frame size distribution(s).
Objective 3.2.4 Encapsulation Throughput (ET)
To produce a set of multicast latency measurements from a single, Definition:
multicast ingress port of a DUT or SUT through multiple, egress The maximum rate at which frames offered a DUT are encapsulated
multicast ports of that same DUT or SUT as provided for by the and correctly forwarded by the DUT without loss.
metric "Multicast Latency" in RFC 2432.
The procedures highlighted below attempt to draw from the Discussion:
collection methodology for latency in RFC 2544 to the degree A popular technique in presenting a frame to a device that may not
possible. The methodology addresses two topological scenarios: one support a protocol feature is to encapsulate, or tunnel, the
for a single device (DUT) characterization; a second scenario is packet containing the unsupported feature in a format that is
presented or multiple device (SUT) characterization. supported by that device.
Procedure More specifically, encapsulation refers to the act of taking a
frame or part of a frame and embedding it as a payload of another
frame. This benchmark attempts to characterize the overhead
behavior associated with that translational process.
If the test trial is to characterize latency across a single Device Measurement units:
Under Test (DUT), an example test topology might take the form of Frames per second.
Figure 1 in section 3. That is, a single DUT with one ingress
interface receiving the multicast test traffic from packet-
transmitting component of the test apparatus and n egress
interfaces on the same DUT forwarding the multicast test traffic
back to the packet-receiving component of the test apparatus. Note
that n reflects the number of TESTED egress interfaces on the DUT
actually expected to forward the test traffic (as opposed to
configured but untested, non-forwarding interfaces, for example).
If the multicast latencies are to be taken across multiple devices Issues:
forming a System Under Test (SUT), an example test topology might Consideration may need to be given with respect to the impact of
take the form of Figure 2 in section 3. different frame formats on usable bandwidth.
The trial duration SHOULD be 120 seconds. Departures to the Since frame size can sometimes be a factor in frame forwarding
suggested traffic class guidelines MUST be disclosed with the benchmarks, the corresponding methodology for this metric will
respective trial results. The nature of the latency measurement, need to consider frame size distribution(s).
"store and forward" or "bit forwarding," MUST be associated with
the related test trial(s) and disclosed in the results report.
End-to-end reach ability of the test traffic path SHOULD be 3.2.5 Decapsulation Throughput (DT)
verified prior to the engagement of a test trial. This implies
that subsequent measurements are intended to characterize the
latency across the tested device's or devices' normal traffic
forwarding path (e.g., faster hardware-based engines) of the
device(s) as opposed a non-standard traffic processing path (e.g.
slower, software-based exception handlers). If the test trial is
to be executed with the intent of characterizing a non-optimal,
forwarding condition, then a description of the exception
processing conditions being characterized MUST be included with the
trial's results.
A test traffic stream is presented to the DUT. At the mid-point of Definition:
the trial's duration, the test apparatus MUST inject a uniquely The maximum rate at which frames offered a DUT are decapsulated
identifiable ("tagged") packet into the test traffic packets being and correctly forwarded by the DUT without loss.
presented. This tagged packet will be the basis for the latency
measurements. By "uniquely identifiable," it is meant that the test
apparatus MUST be able to discern the "tagged" packet from the
other packets comprising the test traffic set. A packet generation
timestamp, Timestamp A, reflecting the completion of the
transmission of the tagged packet by the test apparatus, MUST be
determined.
The test apparatus then monitors packets from the DUT's tested Discussion:
egress port(s) for the expected tagged packet(s) until the A popular technique in presenting a frame to a device that may not
cessation of traffic generation at the end of the configured trial support a protocol feature is to encapsulate, or tunnel, the
duration.A value of the Offered Load presented the DUT/SUT MUST be packet containing the unsupported feature in a format that is
noted. supported by that device. At some point, the frame may be required
to be returned its orginal format from its encapsulation wrapper
for use by the frame's next destination.
The test apparatus MUST record the time of the successful detection More specifically, decapsulation refers to the act of taking a
of a tagged packet from a tested egress interface with a timestamp, frame or part of a frame embedded as a payload of another frame
Timestamp B. A set of Timestamp B values MUST be collected for all and returning it to the payload's appropriate format. This
tested egress interfaces of the DUT/SUT. benchmark attempts to characterize the overhead behavior
associated with that translational process.
A trial MUST be considered INVALID should any of the following Measurement units:
conditions occur in the collection of the trial data: Frames per second.
. Forwarded test packets directed to improper destinations. Issues:
. Unexpected differences between Intended Load and Offered Load Consideration may need to be given with respect to the impact of
or unexpected differences between Offered Load and the different frame formats on usable bandwidth.
resulting Forwarding Rate(s) on the DUT/SUT egress ports.
. Forwarded test packets improperly formed or packet header
fields improperly manipulated.
. Failure to forward required tagged packet(s) on all expected
egress interfaces.
. Reception of a tagged packet by the test apparatus outside the
configured test duration interval or 5 seconds, whichever is
greater.
Data from invalid trials SHOULD be considered inconclusive. Data Since frame size can sometimes be a factor in frame forwarding
from invalid trials MUST not form the basis of comparison. benchmarks, the corresponding methodology for this metric will
need to consider frame size distribution(s).
The set of latency measurements, M, composed from each latency 3.2.6 Re-encapsulation Throughput (RET)
measurement taken from every ingress/tested egress interface
pairing MUST be determined from a valid test trial:
M = { (Timestamp B(E0) - Timestamp A),
(Timestamp B(E1) - Timestamp A), ...
(Timestamp B(En) - Timestamp A) }
where (E0 ... En) represents the range of all tested egress Definition:
interfaces and Timestamp B represents a tagged packet detection The maximum rate at which frames of one encapsulated format
event for a given DUT/SUT tested egress interface. offered a DUT are converted to another encapsulated format and
correctly forwarded by the DUT without loss.
Results Discussion:
A popular technique in presenting a frame to a device that may not
support a protocol feature is to encapsulate, or tunnel, the
packet containing the unsupported feature in a format that is
supported by that device. At some point, the frame may be required
to be converted from one encapsulation format to another
encapsulation format.
Two types of information MUST be reported: 1) the set of latency More specifically, re-encapsulation refers to the act of taking an
measurements and 2) the significant environmental, methodological, encapsulated payload of one format and replacing it with another
or device particulars giving insight into the test or its results. encapsulated format - all the while preserving the original
payload's contents. This benchmark attempts to characterize the
overhead behavior associated with that translational process.
Specifically, when reporting the results of a VALID test trial, the Measurement units:
set of ALL latencies related to the tested ingress interface and Frames per second.
each tested egress DUT/SUT interface of MUST be presented. The
time units of the presented latency MUST be uniform and with
sufficient precision for the medium or media being tested. Results
MAY be offered in tabular format and SHOULD preserve the
relationship of latency to ingress/egress interface to assist in
trending across multiple trials.
The Offered Load of the test traffic presented the DUT/SUT, size of Issues:
the "tagged" packet, transmit duration of offered frames and nature Consideration may need to be given with respect to the impact of
(i.e., store-and-forward or bit-forwarding) of the trial's different frame formats on usable bandwidth.
measurement MUST be associated with any reported test trial's
result.
5.2. Min/Max Multicast Latency Since frame size can sometimes be a factor in frame forwarding
benchmarks, the corresponding methodology for this metric will
need to consider frame size distribution(s).
Objective 3.3 Forwarding Latency.
The difference between the maximum latency measurement and the This section presents terminology relating to the characterization of
minimum latency measurement from a collected set of latencies the forwarding latency of a DUT/SUT in a multicast environment. It
produced by the Multicast Latency benchmark. extends the concept of latency presented in RFC 1242.
Procedure 3.3.1 Multicast Latency. (ML)
Collect a set of multicast latency measurements, as prescribed in Definition:
section 5.1. This will produce a set of multicast latencies, M, The set of individual latencies from a single input port on the
where M is composed of individual forwarding latencies between DUT DUT or SUT to all tested ports belonging to the destination
packet ingress and DUT packet egress port pairs. E.g.: multicast group.
M = {L(I,E1),L(I,E2), , L(I,En)} Discussion:
This benchmark is based on the RFC 1242 definition of latency.
While it is useful to collect latency between a pair of source and
destination multicast ports, it may be insightful to collect the
same type of measurements across a range of ports supporting that
Group Class.
where L is the latency between a tested ingress port, I, of the A variety of statistical exercises can be applied to the set of
DUT, and Ex a specific, tested multicast egress port of the DUT. latencies measurements.
E1 through En are unique egress ports on the DUT.
From the collected multicast latency measurements in set M, Measurement units:
identify MAX(M), where MAX is a function that yields the largest Time units with enough precision to reflect a latency measurement.
latency value from set M.
Identify MIN(M), when MIN is a function that yields the smallest 3.3.2 Min/Max Multicast Latency. (Min/Max ML)
latency value from set M.
The Max/Min value is determined from the following formula: Definition:
The difference between the maximum latency measurement and the
minimum latency measurement from the set of latencies produced by
the Multicast Latency benchmark.
Result = MAX(M) MIN(M) Discussion:
This statistic may yield some insight into how a particular
implementation handles its multicast traffic. This may be useful
to users of multicast synchronization types of applications.
Results Measurement units:
Time units with enough precision to reflect latency measurement.
The result MUST be represented as a single numerical value in time 3.4 Overhead
units consistent with the corresponding latency measurements. In
addition, the number of tested egress ports on the DUT MUST be
reported.
The nature of the traffic stream contributing to the result MUST be This section presents terminology relating to the characterization of
reported. All required reporting parameters of multicast latency the overhead delays associated with explicit operations found in
MUST be reflected in the min/max results report, such as the multicast environments.
transmitted packet size(s), offered load of the packet stream in
which the tagged packet was presented to the DUT and transmit
duration of offered frames.
6. Overhead 3.4.1 Group Join Delay. (GJD)
This section presents methodology relating to the characterization Definition:
of the overhead delays associated with explicit operations found in The time duration it takes a DUT to start forwarding multicast
multicast environments. packets from the time a successful IGMP group membership report
has been issued to the DUT.
6.1. Group Join Delay Discussion:
Many factors can contribute to different results, such as the
number or type of multicast-related protocols configured on the
device under test. Other factors are physical topology and "tree"
configuration.
Objective Because of the number of variables that could impact this metric,
the metric may be a better characterization tool for a device
rather than a basis for comparisons with other devices.
The time duration it takes a DUT/SUT to start forwarding multicast Issues:
packets from the time a successful IGMP group membership report has A consideration for the related methodology: possible need to
been issued to the DUT/SUT. differentiate a specifically-forwarded multicast frame from those
sprayed by protocols implementing a flooding tactic to solicit
prune feedback.
Procedure While this metric attempts to identify a simple delay, the
underlying and contributing delay components (e.g., propagation
delay, frame processing delay, etc.) make this a less than simple
measurement. The corresponding methodology will need to consider
this and similar factors to ensure a consistent and precise metric
result.
Traffic is sent on the source port at the same time as the IGMP Measurement units:
JOIN Group message is transmitted from the destination ports. The Microseconds.
join delay is the difference in time from when the IGMP Join is
sent (timestamp A) and the first frame is forwarded to a receiving
member port (timestamp B).
Group Join delay = timestamp B - timestamp A 3.4.2 Group Leave Delay. (GLD)
One of the keys is to transmit at the fastest rate the DUT/SUT can Definition:
handle multicast frames. This is to get the best resolution and The time duration it takes a DUT to cease forwarding multicast
the least margin of error in the Join Delay. packets after a corresponding IGMP "Leave Group" message has been
successfully offered to the DUT.
However, you do not want to transmit the frames so fast that frames Discussion:
are dropped by the DUT/SUT. Traffic should be sent at the While it is important to understand how quickly a device can
throughput rate determined by the forwarding tests of section 4. process multicast frames; it may be beneficial to understand how
quickly that same device can stop the process as well.
Results Because of the number of variables that could impact this metric,
the metric may be a better characterization tool for a device
rather than a basis for comparisons with other devices.
The parameter to be measured is the join delay time for each Measurement units:
multicast group address per destination port. In addition, the Microseconds.
number of frames transmitted and received and percent loss may be
reported.
6.2. Group Leave Delay Issues:
The Methodology may need to consider protocol-specific timeout
values.
Objective While this metric attempts to identify a simple delay, the
underlying and contributing delay components (e.g., propagation
delay, frame processing delay, etc.) make this a less than simple
measurement. Moreover, the cessation of traffic is a rather
unobservable event (i.e., at what point is the multicast forwarded
considered stopped on the DUT interface processing the Leave?).
The corresponding methodology will need to consider this and
similar factors to ensure a consistent and precise metric result.
The time duration it takes a DUT/SUT to cease forwarding multicast 3.5 Capacity
packets after a corresponding IGMP "Leave Group" message has been
successfully offered to the DUT/SUT.
Procedure This section offers terms relating to the identification of multicast
group limits of a DUT/SUT.
Traffic is sent on the source port at the same time as the IGMP 3.5.1 Multicast Group Capacity. (MGC)
Leave Group messages are transmitted from the destination ports.
The leave delay is the difference in time from when the IGMP leave
is sent (timestamp A) and the last frame is forwarded to a
receiving member port (timestamp B).
Group Leave delay = timestamp B - timestamp A Definition:
The maximum number of multicast groups a SUT/DUT can support while
maintaining the ability to forward multicast frames to all
multicast groups registered to that SUT/DUT.
One of the keys is to transmit at the fastest rate the DUT/SUT can Discussion:
handle multicast frames. This is to get the best resolution and
least margin of error in the Leave Delay. However, you do not want
to transmit the frames too fast that frames are dropped by the
DUT/SUT. Traffic should be sent at the throughput rate determined
by the forwarding tests of section 4.
Results Measurement units:
Multicast groups.
The parameter to be measured is the leave delay time for each Issues:
multicast group address per destination port. In addition, the The related methodology may have to consider the impact of
number of frames transmitted and received and percent loss may be multicast sources per group on the ability of a SUT/DUT to "scale
reported. up" the number of supportable multicast groups.
7. Capacity 3.6 Interaction
This section offers terms relating to the identification of Network forwarding devices are generally required to provide more
multicast group limits of a DUT/SUT. functionality than than the forwarding of traffic. Moreover, network
forwarding devices may be asked to provide those functions in a
variety of environments. This section offers terms to assist in the
charaterization of DUT/SUT behavior in consideration of potentially
interacting factors.
7.1. Multicast Group Capacity 3.6.1 Burdened Response.
Objective Definition:
A measured response collected from a DUT/SUT in light of
interacting, or potentially interacting, distinct stimulii.
The maximum number of multicast groups a DUT/SUT can support while Discussion:
maintaining the ability to forward multicast frames to all Many metrics provide a one dimensional view into an operating
multicast groups registered to that DUT/SUT. characteristic of a tested system. For example, the forwarding
rate metric may yield information about the packet processing
ability of a device. Collecting that same metric in view of
another control variable can oftentimes be very insightful. Taking
that same forwarding rate measurement, for instance, while the
device's address table is injected with an additional 50,000
entries may yield a different perspective.
Procedure Measurement units:
A burdened response is a type of metric. Metrics of this this
type must follow guidelines when reporting results.
One or more destination ports of DUT/SUT will join an initial The metric's principal result MUST be reported in conjunction with
number of groups. the contributing factors.
Then after a delay (enough time for all ports to join) the source For example, in reporting a Forwarding Burdened Latency, the
port will transmit to each group at a transmission rate that the latency measurement should be reported with respect to
DUT/SUT can handle without dropping IP Multicast frames. corresponding Offered Load and Forwarding Rates.
If all frames sent are forwarded by the DUT/SUT and received the Issues: A Burdened response may be very illuminating when trying to
test iteration is said to pass at the current capacity. characterize a single device or system. Extreme care must be
exercised when attempting to use that characterization as a basis
of comparison with other devices or systems. Test agents must
ensure that the measured response is a function of the controlled
stimulii, and not secondary factors. An example of of such an
interfering factor would be configuration mismatch of a timer
impacting a response process.
If the iteration passes at the capacity the test will add an user 3.6.2 Forwarding Burdened Multicast Latency. (FBML)
defined incremental value of groups to each receive port.
The iteration is to run again at the new group level and capacity Definition:
tested as stated above. A multicast latency taken from a DUT/SUT in the presence of a
traffic forwarding requirement.
Once the test fails at a capacity the capacity is stated to be the Discussion:
last Iteration that pass at a giving capacity. This burdened response metric builds on the Multicast Latency
definition offered in section 3.3.1. It mandates that the DUT be
subjected to an additional measure of traffic not required by the
non-burdened metric.
Results This metric attempts to provide a means by which to evaluate how
traffic load may or may not impact a device's or system's packet
processing delay.
The parameter to be measured is the total number of group addresses Measurement units:
that were successfully forwarded with no loss. Time units with enough precision to reflect the latencies
measurements.
In addition, the nature of the traffic stream contributing to the Latency measurements MUST be reported with the corresponding
result MUST be reported. All required reporting parameters MUST be sustained Forwarding Rate and associated Offered Load.
reflected in the results report, such as the transmitted packet
size(s) and offered load of the packet stream.
8. Interaction 3.6.3 Forwarding Burdened Group Join Delay. (FBGJD)
Network forwarding devices are generally required to provide more Definition:
functionality than just the forwarding of traffic. Moreover, A multicast Group Join Delay taken from a DUT in the presence of a
network-forwarding devices may be asked to provide those functions traffic forwarding requirement.
in a variety of environments. This section offers terms to assist
in the characterization of DUT/SUT behavior in consideration of
potentially interacting factors.
8.1. Forwarding Burdened Multicast Latency Discussion:
This burdened response metric builds on the Group Join Delay
definition offered in section 3.4.1. It mandates that the DUT be
subjected to an additional measure of traffic not required by the
non-burdened metric.
The Multicast Latency metrics can be influenced by forcing the Many factors can contribute to different results, such as the
DUT/SUT to perform extra processing of packets while multicast number or type of multicast-related protocols configured on the
traffic is being forwarded for latency measurements. In this test, device under test. Other factors could be physical topology or the
a set of ports on the tester will be designated to be source and logical multicast "tree" configuration.
destination similar to the generic IP Multicast test setup. In
addition to this setup, another set of ports will be selected to
transmit some multicast traffic that is destined to multicast group
addresses that have not been joined by these additional set of
ports.
For example, if ports 1,2, 3, and 4 form the burdened response Because of the number of variables that could impact this metric,
setup (setup A) which is used to obtain the latency metrics and the metric may be a better characterization tool for a device
ports 5, 6, 7, and 8 form the non-burdened response setup (setup B) rather than a basis for comparisons with other devices.
which will afflict the burdened response setup, then setup B
traffic will join multicast group addresses not joined by the ports
in this setup. By sending such multicast traffic, the DUT/SUT will
perform a lookup on the packets that will affect the processing of
setup A traffic.
8.2. Forwarding Burdened Group Join Delay Measurement units:
Time units with enough precision to reflect the delay
measurements.
The port configuration in this test is similar to the one described Delay measurements MUST be reported with the corresponding
in section 8.1, but in this test, the ports in setup B do not send sustained Forwarding Rate and associated Offered Load.
the multicast traffic. Rather, setup A traffic must be influenced
in such a way that will affect the DUT's/SUT's ability to process
Group Join messages. Therefore, in this test, the ports in setup B
will send a set of IGMP Group Join messages while the ports in
setup A are also joining its own set of group addresses. Since the
two sets of group addresses are independent of each other, the
group join delay for setup A may be different than in the case when
there were no other group addresses being joined.
9. Security Considerations Issues:
While this metric attempts to identify a simple delay, the
underlying and contributing delay components (e.g., propagation
delay, frame processing delay, etc.) make this a less than simple
measurement. The corresponding methodology will need to consider
this and similar factors to ensure a consistent and precise metric
result.
As this document is solely for the purpose of providing metric 4. Security Considerations
methodology and describes neither a protocol nor a protocol's
implementation, there are no security considerations associated
with this document.
10. Acknowledgements This document addresses metrics and terminology relating to the
performance benchmarking of IP Multicast forwarding devices. The
information contained in this document does not impact the security
of the Internet.
The authors would like to acknowledge the following individuals for Methodologies regarding the collection of the metrics described
their help and participation of the compilation and editing of this within this document may need to cite security considerations. This
document Ralph Daniels, Netcom Systems, who made significant document does not address methodological issues.
contributions to earlier versions of this draft, Daniel Bui, IXIA,
and Kevin Dubray, Juniper Networks.
11. References 5. Acknowledgments
[Br91] Bradner, S., "Benchmarking Terminology for Network The IETF BMWG participants have made several comments and suggestions
Interconnection Devices", RFC 1242, July 1991. regarding this work. Particular thanks goes to Harald Alvestrand,
Scott Bradner, Brad Cain, Eric Crawley, Bob Mandeville, David Newman,
Shuching Sheih, Dave Thaler, Chuck Winter, Zhaohui Zhang, and John
Galgay for their insightful review and assistance.
[Br96] Bradner, S., and J. McQuaid, "Benchmarking Methodology for 6. References
Network Interconnect Devices", RFC 2544, March 1999.
[Br97] Bradner, S. "Use of Keywords in RFCs to Reflect Requirement [Br91] Bradner, S., "Benchmarking Terminology for Network
Levels, RFC 2119, March 1997 Interconnection Devices", RFC 1242, July 1991.
[Du98] Dubray, K., "Terminology for IP Multicast Benchmarking", RFC [Br96] Bradner, S., and J. McQuaid, "Benchmarking Methodology for
2432, October 1998. Network Interconnect Devices", RFC 1944, May 1996.
[Hu95] Huitema, C. "Routing in the Internet." Prentice-Hall, 1995. [Hu95] Huitema, C. "Routing in the Internet." Prentice-Hall, 1995.
[Ka98] Kosiur, D., "IP Multicasting: the Complete Guide to [Se98] Semeria, C. and Maufer, T. "Introduction to IP Multicast
Interactive Corporate Networks", John Wiley & Sons, Inc, 1998. Routing." http://www.3com.com/nsc/501303.html 3Com Corp.,
1998.
[Ma98] Mandeville, R., "Benchmarking Terminology for LAN Switching [Ma98] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998. Devices", RFC 2285, February 1998.
[Mt98] Maufer, T. "Deploying IP Multicast in the Enterprise." [Mt98] Maufer, T. "Deploying IP Multicast in the Enterprise."
Prentice-Hall, 1998. Prentice-Hall, 1998.
[Se98] Semeria, C. and Maufer, T. "Introduction to IP Multicast 7. Author's Address
Routing." http://www.3com.com/nsc/501303.html 3Com Corp.,
1998.
12. Author's Addresses Kevin Dubray
IronBridge Networks
55 Hayden Avenue
Lexington, MA 02421
USA
Debra Stopp Phone: 781 372 8118
IXIA EMail: kdubray@ironbridgenetworks.com
26601 W. Agoura Rd.
Calabasas, CA 91302
USA
Phone: 818 871 1800 8. Full Copyright Statement
EMail: debby@ixiacom.com
Hardev Soor Copyright (C) The Internet Society (1998). All Rights Reserved.
IXIA
26601 W. Agoura Rd.
Calabasas, CA 91302
USA
Phone: 818 871 1800 This document and translations of it may be copied and furnished to
EMail: hardev@ixiacom.com others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
13. Full Copyright Statement The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
"Copyright (C) The Internet Society (date). All Rights Reserved. This document and the information contained herein is provided on an
This document and translations of it may be copied and furnished to "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
others, and derivative works that comment on or otherwise explain TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
it or assist in its implementation may be prepared, copied, BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
published and distributed, in whole or in part, without restriction HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
of any kind, provided that the above copyright notice and this MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
paragraph are included on all such copies and derivative works.
However, this document itself may not be modified in any way, such
as by removing the copyright notice or references to the Internet
Society or other Internet organizations, except as needed for the
purpose of developing Internet standards in which case the
procedures for copyrights defined in the Internet Standards process
must be followed, or as required to translate it into.
 End of changes. 171 change blocks. 
715 lines changed or deleted 523 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/