draft-ietf-bmwg-mcastm-10.txt   draft-ietf-bmwg-mcastm-11.txt 
Network Working Group Debra Stopp Network Working Group Debra Stopp
Hardev Soor INTERNET-DRAFT Ixia
INTERNET-DRAFT IXIA Expires in: August 2003 Brooks Hickman
Expires in: March 2003 Spirent Communications
February 2003
Methodology for IP Multicast Benchmarking Methodology for IP Multicast Benchmarking
<draft-ietf-bmwg-mcastm-10.txt> <draft-ietf-bmwg-mcastm-11.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 35 skipping to change at page 1, line 36
progress." progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved. Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract Abstract
The purpose of this draft is to describe methodology specific to The purpose of this document is to describe methodology specific to
the benchmarking of multicast IP forwarding devices. It builds upon the benchmarking of multicast IP forwarding devices. It builds upon
the tenets set forth in RFC 2544, RFC 2432 and other IETF the tenets set forth in RFC 2544, RFC 2432 and other IETF
Benchmarking Methodology Working Group (BMWG) efforts. This Benchmarking Methodology Working Group (BMWG) efforts. This
document seeks to extend these efforts to the multicast paradigm. document seeks to extend these efforts to the multicast paradigm.
The BMWG produces two major classes of documents: Benchmarking The BMWG produces two major classes of documents: Benchmarking
Terminology documents and Benchmarking Methodology documents. The Terminology documents and Benchmarking Methodology documents. The
Terminology documents present the benchmarks and other related Terminology documents present the benchmarks and other related
terms. The Methodology documents define the procedures required to terms. The Methodology documents define the procedures required to
collect the benchmarks cited in the corresponding Terminology collect the benchmarks cited in the corresponding Terminology
skipping to change at page 2, line 18 skipping to change at page 2, line 18
2. KEY WORDS TO REFLECT REQUIREMENTS..............................3 2. KEY WORDS TO REFLECT REQUIREMENTS..............................3
3. TEST SET UP....................................................3 3. TEST SET UP....................................................3
3.1. Test Considerations..........................................5 3.1. Test Considerations..........................................5
3.1.1. IGMP Support..............................................5 3.1.1. IGMP Support..............................................5
3.1.2. Group Addresses...........................................5 3.1.2. Group Addresses...........................................5
3.1.3. Frame Sizes...............................................6 3.1.3. Frame Sizes...............................................6
3.1.4. TTL.......................................................6 3.1.4. TTL.......................................................6
3.1.5. Trial Duration............................................6 3.1.5. Trial Duration............................................6
3.2. Layer 2 Support..............................................6
4. FORWARDING AND THROUGHPUT......................................6 4. FORWARDING AND THROUGHPUT......................................6
4.1. Mixed Class Throughput.......................................6 4.1. Mixed Class Throughput.......................................7
4.2. Scaled Group Forwarding Matrix...............................8 4.2. Scaled Group Forwarding Matrix...............................8
4.3. Aggregated Multicast Throughput..............................8 4.3. Aggregated Multicast Throughput..............................9
4.4. Encapsulation/Decapsulation (Tunneling) Throughput...........9 4.4. Encapsulation/Decapsulation (Tunneling) Throughput..........10
4.4.1. Encapsulation Throughput..................................9 4.4.1. Encapsulation Throughput.................................10
4.4.2. Decapsulation Throughput.................................10 4.4.2. Decapsulation Throughput.................................12
4.4.3. Re-encapsulation Throughput..............................11 4.4.3. Re-encapsulation Throughput..............................13
5. FORWARDING LATENCY............................................12 5. FORWARDING LATENCY............................................15
5.1. Multicast Latency...........................................12 5.1. Multicast Latency...........................................16
5.2. Min/Max Multicast Latency...................................15 5.2. Min/Max Multicast Latency...................................18
6. OVERHEAD......................................................16 6. OVERHEAD......................................................20
6.1. Group Join Delay............................................16 6.1. Group Join Delay............................................20
6.2. Group Leave Delay...........................................16 6.2. Group Leave Delay...........................................21
7. CAPACITY......................................................17 7. CAPACITY......................................................23
7.1. Multicast Group Capacity....................................17 7.1. Multicast Group Capacity....................................23
8. INTERACTION...................................................18 8. INTERACTION...................................................24
8.1. Forwarding Burdened Multicast Latency.......................18 8.1. Forwarding Burdened Multicast Latency.......................24
8.2. Forwarding Burdened Group Join Delay........................19 8.2. Forwarding Burdened Group Join Delay........................25
9. SECURITY CONSIDERATIONS.......................................20 9. SECURITY CONSIDERATIONS.......................................26
10. ACKNOWLEDGEMENTS.............................................20 10. ACKNOWLEDGEMENTS.............................................27
11. REFERENCES...................................................21 11. CONTRIBUTIONS................................................27
12. AUTHOR'S ADDRESSES...........................................22 12. REFERENCES...................................................28
13. FULL COPYRIGHT STATEMENT.....................................22 13. AUTHOR'S ADDRESSES...........................................29
14. FULL COPYRIGHT STATEMENT.....................................29
1. Introduction 1. Introduction
This document defines a specific set of tests that vendors can use This document defines tests for measuring and reporting the
to measure and report the performance characteristics and forwarding, latency and IGMP group membership characteristics of
forwarding capabilities of network devices that support IP devices that support IP multicast routing protocols. The results
multicast protocols. The results of these tests will provide the of these tests will provide the user with meaningful data on
user comparable data from different vendors with which to evaluate multicast performance.
these devices.
A previous document, " Terminology for IP Multicast Benchmarking" A previous document, " Terminology for IP Multicast Benchmarking"
(RFC 2432), defined many of the terms that are used in this (RFC 2432), defined many of the terms that are used in this
document. The terminology document should be consulted before document. The terminology document should be consulted before
attempting to make use of this document. attempting to make use of this document.
This methodology will focus on one source to many destinations, This methodology will focus on one source to many destinations,
although many of the tests described may be extended to use although many of the tests described may be extended to use
multiple source to multiple destination IP multicast communication. multiple source to multiple destination topologies.
2. Key Words to Reflect Requirements 2. Key Words to Reflect Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
in this document are to be interpreted as described in RFC 2119. in this document are to be interpreted as described in RFC 2119.
RFC 2119 defines the use of these key words to help make the intent RFC 2119 defines the use of these key words to help make the intent
of standards track documents as clear as possible. While this of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards document uses these keywords, this document is not a standards
track document. track document.
3. Test set up 3. Test set up
The set of methodologies presented in this draft are for single The set of methodologies presented in this document are for single
ingress, multiple egress scenarios as exemplified by Figures 1 and ingress, multiple egress scenarios as exemplified by Figures 1 and
2. Methodologies for multiple ingress, multiple egress scenarios 2. Methodologies for multiple ingress and multiple egress
are beyond the scope of this document. scenarios are beyond the scope of this document.
Figure 1 shows a typical setup for an IP multicast test, with one Figure 1 shows a typical setup for an IP multicast test, with one
source to multiple destinations. source to multiple destinations.
+----------------+
+------------+ | Egress | +------------+ +--------------+
+--------+ | (-)-------->| destination(E1)| | | | destination |
| | | | | | +--------+ | Egress(-)------->| test |
| source |------->(|)Ingress | +----------------+ | source | | | | port(E1) |
| | | | +----------------+ | test |------>(|)Ingress | +--------------+
+--------+ | D U T (-)-------->| Egress | | port | | | +--------------+
| | | destination(E2)| +--------+ | Egress(-)------->| destination |
| | | | | | | test |
| | +----------------+ | | | port(E2) |
| DUT | +--------------+
| | . . . | | . . .
| | +----------------+ | | +--------------+
| | | Egress | | | | destination |
| (-)-------->| destination(En)| | Egress(-)------->| test |
| | | | | | | port(En) |
+------------+ +----------------+ +------------+ +--------------+
Figure 1 Figure 1
--------- ---------
If the multicast metrics are to be taken across multiple devices If the multicast metrics are to be taken across multiple devices
forming a System Under Test (SUT), then test packets are offered to forming a System Under Test (SUT), then test frames are offered to
a single ingress interface on a device of the SUT, subsequently a single ingress interface on a device of the SUT, subsequently
routed across the SUT topology, and finally forwarded to the test forwarded across the SUT topology, and finally forwarded to the
apparatus' packet-receiving components by the test egress test apparatus' frame-receiving components by the test egress
interface(s) of devices in the SUT. Figure 2 offers an example SUT interface(s) of devices in the SUT. Figure 2 offers an example SUT
test topology. If a SUT is tested, the details of the test test topology. If a SUT is tested, the test topology and all
topology MUST be disclosed with the corresponding test results. relevant configuration details MUST be disclosed with the
corresponding test results.
+--------+ +----------------+ +--------+ *-----------------------------------------*
| | +------------+ |DUT B Egress E0(-)-->| | | |
| | |DUT A |--->| | | | +--------+ | +----------------+ | +--------+
| Test | | | | Egress E1(-)-->| Test | | | | +------------+ |DUT B Egress E0(-)-|->| |
| App. |--->(-)Ingress, I | +----------------+ | App. | | | | |DUT A |--->| | | | |
| Traffic| | | +----------------+ | Traffic| | Test | | | | | Egress E1(-)-|->| Test |
| Src. | | |--->|DUT C Egress E2(-)-->| Dest. | | App. |--|->(-)Ingress, I | +----------------+ | | App. |
| | +------------+ | | | | | Traffic| | | | +----------------+ | | Traffic|
| | | Egress En(-)-->| | | Src. | | | |--->|DUT C Egress E2(-)-|->| Dest. |
+--------+ +----------------+ +--------+ | | | +------------+ | | | | |
| | | | Egress En(-)--|->| |
+--------+ | +----------------+ | +--------+
| |
*------------------SUT--------------------*
Figure 2 Figure 2
--------- ---------
Generally, the destination ports first join the desired number of Generally, the destination test ports first join the desired number
multicast groups by sending IGMP Join Group messages to the of multicast groups by sending IGMP Group Report messages to the
DUT/SUT. To verify that all destination ports successfully joined DUT/SUT. To verify that all destination test ports successfully
the appropriate groups, the source port MUST transmit IP multicast joined the appropriate groups, the source port MUST transmit IP
frames destined for these groups. The destination ports MAY send multicast frames destined for these groups. The destination test
IGMP Leave Group messages after the transmission of IP Multicast ports MAY send IGMP Leave Group messages after the transmission of
frames to clear the IGMP table of the DUT/SUT. IP Multicast frames to clear the IGMP table of the DUT/SUT.
In addition, test equipment MUST validate the correct and proper In addition, test equipment MUST validate the correct and proper
forwarding actions of the devices they test in order to ensure the forwarding actions of the devices they test in order to ensure the
receipt of only the frames that are involved in the test. receipt of the frames that are involved in the test.
3.1. Test Considerations 3.1. Test Considerations
The procedures outlined below are written without regard for The methodology assumes a uniform medium topology. Issues regarding
specific physical layer or link layer protocols. The methodology mixed transmission media, such as speed mismatch, headers
further assumes a uniform medium topology. Issues regarding mixed differences, etc., are not specifically addressed. Flow control,
transmission media, such as speed mismatch, headers differences, QoS and other non-essential traffic or traffic-affecting mechanisms
etc., are not specifically addressed. Flow control, QoS and other affecting the variable under test MUST be disabled. Modifications
traffic-affecting mechanisms MUST be disabled. Modifications to to the collection procedures might need to be made to accommodate
the specified collection procedures might need to be made to the transmission media actually tested. These accommodations MUST
accommodate the transmission media actually tested. These be presented with the test results.
accommodations MUST be presented with the test results.
An actual flow of test traffic may be required to prime related An actual flow of test traffic may be required to prime related
mechanisms, (e.g., process RPF events, build device caches, etc.) mechanisms, (e.g., process RPF events, build device caches, etc.)
to optimally forward subsequent traffic. Therefore, before an to optimally forward subsequent traffic. Therefore, before an
initial, measured forwarding test trial, the test apparatus MUST initial, measured forwarding test trial, the test apparatus MUST
generate test traffic utilizing the same addressing characteristics generate test traffic utilizing the same addressing characteristics
to the DUT/SUT that will subsequently be used to measure the to the DUT/SUT that will subsequently be used to measure the
DUT/SUT response. The test monitor should ensure the correct DUT/SUT response. The test monitor should ensure the correct
forwarding of traffic by the DUT/SUT. The priming action need only forwarding of traffic by the DUT/SUT. The priming action need only
be repeated to keep the associated information current. be repeated to keep the associated information current.
3.1.1. IGMP Support 3.1.1. IGMP Support
Each of the destination ports should support and be able to test All of the ingress and egress interfaces MAY support any version of
all IGMP versions 1, 2 and 3. The minimum requirement, however, is IGMP. The IGMP version on the ingress interface MUST be the same
IGMP version 2. version of IGMP that is being tested on the egress interfaces.
Each destination port should be able to respond to IGMP queries Each of the ingress and egress interfaces SHOULD be able to respond
during the test. to IGMP queries during the test.
Each destination port should also send LEAVE (running IGMP version Each of the ingress and egress interfaces SHOULD also send LEAVE
2) after each test. (running IGMP version 2 or later) after each test.
3.1.2. Group Addresses 3.1.2. Group Addresses
It is intended that the collection of benchmarks prescribed in It is intended that the collection of benchmarks prescribed in this
this document be executed in an isolated lab environment. That document be executed in an isolated lab environment. That is to
is to say, the test traffic offered the tested devices MUST NOT say, the test traffic offered the tested devices MUST NOT traverse
traverse a live internet, intranet, or other user-oriented network. a live internet, intranet, or other production network.
Assuming the above, there is no restriction to the use of multicast Assuming the above, there is no restriction to the use of multicast
addresses to compose the test traffic other than those assignments addresses to compose the test traffic other than those assignments
imposed by IANA. The IANA assignments MUST be regarded for imposed by IANA. The IANA assignments MUST be regarded for
operational consistency. For multicast address assignments see: operational consistency. For multicast address assignments see:
http://www.iana.org/assignments/multicast-addresses http://www.iana.org/assignments/multicast-addresses
It should be noted that address selection need not be restricted to Address selection does not need to be restricted to
Administratively Scoped IP Multicast addresses. Administratively Scoped IP Multicast addresses.
3.1.3. Frame Sizes 3.1.3. Frame Sizes
Each test SHOULD be run with different Multicast Frame Sizes. The Each test SHOULD be run with different multicast frame sizes. For
recommended frame sizes are 64, 128, 256, 512, 1024, 1280, and 1518 Ethernet, the recommended sizes are 64, 128, 256, 512, 1024, 1280,
byte frames. and 1518 byte frames.
Other link layer technologies MAY be used. The minimum and maximum
frame lengths of the link layer technology in use SHOULD be tested.
When testing with different frame sizes, the DUT/SUT configuration
MUST remain the same.
3.1.4. TTL 3.1.4. TTL
The source frames should have a TTL value large enough to The data plane test traffic should have a TTL value large enough to
accommodate the DUT/SUT. traverse the DUT/SUT.
The TTL in IGMP control plane messages is in compliance with the
version of IGMP in use.
3.1.5. Trial Duration 3.1.5. Trial Duration
The duration of the test portion of each trial SHOULD be at least The duration of the test portion of each trial SHOULD be at least
30 seconds. This parameter MUST be included as part of the results 30 seconds. This parameter MUST be included as part of the results
reporting for each methodology. reporting for each methodology.
3.2. Layer 2 Support
Each of the destination ports should support GARP/GMRP protocols to
join groups on Layer 2 DUTs/SUTs.
4. Forwarding and Throughput 4. Forwarding and Throughput
This section contains the description of the tests that are related This section contains the description of the tests that are related
to the characterization of the packet forwarding of a DUT/SUT in a to the characterization of the frame forwarding of a DUT/SUT in a
multicast environment. Some metrics extend the concept of throughput multicast environment. Some metrics extend the concept of throughput
presented in RFC 1242. The notion of Forwarding Rate is cited in RFC presented in RFC 1242. Forwarding Rate is cited in RFC 2285.
2285.
4.1. Mixed Class Throughput 4.1. Mixed Class Throughput
Objective Objective:
To determine the throughput of a DUT/SUT when both unicast class To determine the throughput of a DUT/SUT when both unicast class
frames and multicast class frames are offered simultaneously to a frames and multicast class frames are offered simultaneously to a
fixed number of ports as defined in RFC 2432. fixed number of interfaces as defined in RFC 2432.
Procedure Procedure:
Multicast and unicast traffic are mixed together in the same Multicast and unicast traffic are mixed together in the same
aggregated traffic stream in order to simulate the non-homogenous aggregated traffic stream in order to simulate the non-homogenous
networking environment. The DUT/SUT MUST learn the appropriate networking environment.
unicast IP addresses, either by sending ARP frames from each
unicast address, sending a RIP packet or by assigning static
entries into the DUT/SUT address table.
The relationship between the intended load [Ma91] of multicast The following events MUST occur before offering test traffic:
class frames vs. unicast class frames MUST be specified:
a) As an independent rate for unicast class and multicast o All DUT/SUT egress interfaces configured to receive
class of traffic OR multicast traffic MUST join all configured multicast
b) As an aggregate rate comprised of a ratio of multicast groups;
class to unicast class of traffic. o The DUT/SUT MUST learn the appropriate unicast addresses;
and
o Group membership and unicast address learning MUST be
verified through some externally observable method.
The offered load per each DUT/SUT port MUST not exceed the maximum The intended load [Ma98] SHOULD be configured as alternating
bandwidth capacity of any configured receive DUT/SUT ports. multicast frames and unicast frames to a single ingress interface
in a 50-50 ratio. The unicast frames MUST be configured to
transmit in a round-robin fashion to all of the egress interfaces.
The multicast frames MUST be configured to transmit to all of the
egress interfaces.
All DUT/SUT ports configured to receive multicast traffic MUST join Mixed class throughput measurement is defined in RFC2432 [Du98]. A
all configured multicast groups prior to transmitting test frames. search algorithm MUST be utilized to determine the throughput for
Joining a group is accomplished by sending an IGMP Join Group both unicast class and multicast class traffic in a mixed class
messages. All DUT/SUT ports configured to receive unicast traffic environment.
MUST send learning frames prior to transmitting test frames (see
section 3 for more information).
Unicast traffic distribution can either be non-meshed or meshed Reporting Format:
[Ma98] as specified in RFC2544 or RFC2289. A minimum of one
unicast transmit port MUST be configured to transmit unicast
traffic to a DUT/SUT port that is configured to receive unicast and
multicast traffic.
Multicast traffic distribution MUST be configured to transmit The following configuration parameters MUST be reflected in the
traffic in a one-to-many mesh [Ma98] configuration. A minimum of results specific to this methodology:
one multicast transmit port MUST be configured to transmit
multicast traffic to a DUT/SUT port that is configured to receive
multicast traffic.
Throughput measurement is defined in RFC1242 [Br91]. A search o Frame size(s)
algorithm MUST be utilized to determine the maximum offered frame o Number of tested egress interfaces on the DUT/SUT
rate with a zero frame loss rate. o Test duration
o IGMP version
o Total number of multicast groups
o Traffic distribution for unicast and multicast traffic
classes
o The ratio of multicast and unicast traffic must be declared
The following results MUST be reflected in the results specific to
this methodology:
Result o Mixed Class Throughput as defined in RFC2432 [Du98],
including: Throughput per unicast and multicast traffic
classes.
Parameters to be measured MUST include the aggregate offered load, The Mixed Class Throughput results for each test SHOULD be reported
number of multicast frames offered, number of unicast frames in the form of a table with a row for each of the tested frame
offered, number of multicast frames received, number of unicast sizes per the recommendations in section 3.1.3. Each row SHOULD
frames received and transmit duration of offered frames. specify the intended load, number of multicast frames offered,
number of unicast frames offered and measured throughput per class.
4.2. Scaled Group Forwarding Matrix 4.2. Scaled Group Forwarding Matrix
Objective Objective:
To determine Forwarding Rate as a function of tested multicast To determine Forwarding Rate as a function of tested multicast
groups for a fixed number of tested DUT/SUT ports. groups for a fixed number of tested DUT/SUT ports.
Procedure Procedure:
Multicast traffic is sent at a fixed percent of maximum offered This is an iterative procedure. The destination test port(s) MUST
load with a fixed number of receive ports of the tester at a fixed join an initial number of multicast groups on the first iteration.
frame length. All DUT/SUT destination test port(s) configured to receive
multicast traffic MUST join all configured multicast groups. The
recommended number of groups to join on the first iteration is 10
groups. Multicast traffic is subsequently transmitted to all
groups joined on this iteration.
On each iteration, the receive ports SHOULD incrementally join 10 The number of multicast groups joined by each destination test port
multicast groups until a user defined maximum number of groups is is then incremented, or scaled, by an additional number of
reached. multicast groups. The recommended granularity of additional groups
to join per iteration is 10, although the tester MAY choose a finer
granularity. Multicast traffic is subsequently transmitted to all
groups joined during this iteration.
Results The total number of multicast groups joined MUST not exceed the
capacity of the DUT/SUT. Both Group Join Delay and Group Capacity
results MUST be known prior to running this test.
Parameters to be measured MUST include the offered load and Reporting Format:
forwarding rate as a function of the total number of multicast
groups, for each test iteration.
The nature of the traffic stream contributing to the result MUST be The following configuration parameters MUST be reflected in the
reported, specifically number of source and destination ports results specific to this methodology:
within the multicast group. In addition, all other reporting
parameters of the scaled group forwarding matrix methodology MUST
be reflected in the results report, such as the transmitted packet
size(s) and offered load of the packet stream for each source port.
Result reports MUST include the following parameters for each o Frame size(s)
iteration: the number of frames offered, number of frames received o Number of tested egress interfaces on the DUT/SUT
per each group, number of multicast groups and forwarding rate, in o Test duration
frames per second, and transmit duration of offered frames. o IGMP version
Constructing a table that contains the forwarding rate vs. number
of groups is desirable. The following results MUST be reflected in the results specific to
this methodology:
o The total number of multicast groups joined for that
iteration
o Total number of frames transmitted
o Total number of frames received
o Offered load
o Forwarding rate determined for that iteration
The Scaled Group Forwarding results for each test SHOULD be
reported in the form of a table with a row representing each
iteration of the test. Each row or iteration SHOULD specify the
total number of groups joined for that iteration, total number of
frames transmitted, total number of frames received and the
aggregate forwarding rate determined for that iteration.
4.3. Aggregated Multicast Throughput 4.3. Aggregated Multicast Throughput
Objective Objective:
To determine the maximum rate at which none of the offered frames To determine the maximum rate at which none of the offered frames
to be forwarded through N destination interfaces of the same to be forwarded through N destination interfaces of the same
multicast group is dropped. multicast groups are dropped.
Procedure Procedure:
Multicast traffic is sent at a fixed percent of maximum offered Offer multicast traffic at an initial fixed offered load to a fixed
load with a fixed number of groups at a fixed frame length for a set of interfaces with a fixed number of groups at a fixed frame
fixed duration of time. length for a fixed duration of time. All destination test ports
MUST join all specified multicast groups.
The initial number of receive ports of the tester will join the If any frame loss is detected, the offered load is decreased and
group(s) and the sender will transmit to the same groups after a the sender will transmit again. An iterative search algorithm MUST
certain delay (a few seconds). be utilized to determine the maximum offered frame rate with a zero
frame loss.
If any frame loss is detected, one receive port MUST leave the Each iteration will involve varying the offered load of the
group(s) and the sender will transmit again. Continue in this multicast traffic, while keeping the set of interfaces, number of
iterative fashion until either there are no ports left joined to multicast groups, frame length and test duration fixed, until the
the multicast group(s) OR 0% frame loss is achieved. maximum rate at which none of the offered frames are dropped is
determined.
Results Parameters to be measured MUST include the offered load at which no
frame loss occurred.
Parameters to be measured MUST include the maximum offered load at Reporting Format:
which no frame loss occurred (as defined by RFC 2544)
The nature of the traffic stream contributing to the result MUST be The following configuration parameters MUST be reflected in the
reported. All required reporting parameters of aggregated results specific to this methodology:
throughput MUST be reflected in the results report, such as the
initial number of receive ports, the final number of receive ports,
total number of multicast group addresses, the transmitted packet
size(s), offered load of the packet stream and transmit duration of
offered frames.
Constructing a table from the measurements might be useful in o Frame size(s)
illustrating the effect of modifying the number of active egress o Number of tested egress interfaces on the DUT/SUT
ports on the tested system. o Test duration
o IGMP version
o Total number of multicast groups
The following results MUST be reflected in the results specific to
this methodology:
o Aggregated Multicast Throughput as defined in RFC2432
[Du98]
The Aggregated Multicast Throughput results SHOULD be reported in
the format of a table with a row for each of the tested frame sizes
per the recommendations in section 3.1.3. Each row or iteration
SHOULD specify offered load, total number of offered frames and the
measured Aggregated Multicast Throughput.
4.4. Encapsulation/Decapsulation (Tunneling) Throughput 4.4. Encapsulation/Decapsulation (Tunneling) Throughput
This sub-section provides the description of tests that help in This sub-section provides the description of tests that help in
obtaining throughput measurements when a DUT/SUT or a set of DUTs obtaining throughput measurements when a DUT/SUT or a set of DUTs
are acting as tunnel endpoints. are acting as tunnel endpoints.
4.4.1. Encapsulation Throughput 4.4.1. Encapsulation Throughput
Objective Objective:
To determine the maximum rate at which frames offered a DUT/SUT are To determine the maximum rate at which frames offered to one
encapsulated and correctly forwarded by the DUT/SUT without loss. ingress interface of a DUT/SUT are encapsulated and correctly
forwarded on one or more egress interfaces of the DUT/SUT without
loss.
Procedure Procedure:
Traffic is offered to the ingress interface of a DUT/SUT <Figure 1> Source DUT/SUT Destination
that has been configured to encapsulate the frames and received on Test Port Test Port(s)
a test port prior to decapsulation at the egress interface. +---------+ +-----------+ +---------+
| | | | | |
| | | Egress|--(Tunnel)-->| |
| | | | | |
| |------->|Ingress | | |
| | | | | |
| | | Egress|--(Tunnel)-->| |
| | | | | |
+---------+ +-----------+ +---------+
The DUT/SUT SHOULD be configured such that the constitution of Figure 3
traffic will consist of either: ---------
Figure 3 shows the setup for testing the encapsulation throughput
of the DUT/SUT. One or more tunnels are created between each
egress interface of the DUT/SUT and a destination test port. Non-
Encapsulated multicast traffic will then be offered by the source
test port, encapsulated by the DUT/SUT and forwarded to the
destination test port(s).
The DUT/SUT SHOULD be configured such that the traffic across each
egress interface will consist of either:
a) A single tunnel encapsulating one or more multicast address a) A single tunnel encapsulating one or more multicast address
groups OR groups OR
b) Multiple tunnels, each encapsulating one or more multicast b) Multiple tunnels, each encapsulating one or more multicast
address groups. address groups.
Each tunnel created by the ingress DUT/SUT SHOULD contain the same The number of multicast groups per tunnel MUST be the same when the
number of multicast address groups per tunnel interface. DUT/SUT is configured in a multiple tunnel configuration. In
addition, it is RECOMMENDED to test with the same number of tunnels
on each egress interface. All destination test ports MUST join all
multicast group addresses offered by the source test port. Each
egress interface MUST be configured with the same MTU.
The offered load on the ingress port MUST not oversubscribe the A search algorithm MUST be utilized to determine the encapsulation
outbound link of the DUT/SUT with respect to the benchmarked throughput as defined in [Du98].
throughput at the encapsulated frame size.
Results Reporting Format:
Based on the resulting encapsulated frame size, parameters to be The following configuration parameters MUST be reflected in the
measured MUST include the maximum offered load at which no frame results specific to this methodology:
loss occurred, the number of encapsulated multicast frames offered
per tunnel interface, and the number of encapsulated multicast
frames received per tunnel interface.
The nature of the traffic stream contributing to the result MUST be o Number of tested egress interfaces on the DUT/SUT
reported. All required reporting parameters of multicast o Test duration
encapsulation throughput MUST be reflected in the results report, o IGMP version
such as the encapsulation format, transmitted packet size(s), o Total number of multicast groups
encapsulated frame size, and transmit duration of offered frames. o MTU size of DUT/SUT interfaces
The following results MUST be reflected in the results specific to
this methodology:
o Measured Encapsulated Throughput as defined in RFC2432
[Du98]
o Encapsulated frame size
o Originating un-encapsulated frame size
o Number of tunnels
o Number of multicast groups per tunnel
The Encapsulated Throughput results SHOULD be reported in the form
of a table and specific to this test there SHOULD be rows for each
originating un-encapsulated frame size. Each row or iteration
SHOULD specify the offered load, encapsulation method, encapsulated
frame size, total number of offered frames, and the encapsulation
throughput.
4.4.2. Decapsulation Throughput 4.4.2. Decapsulation Throughput
Objective Objective:
To determine the maximum rate at which frames offered a DUT/SUT are To determine the maximum rate at which frames offered to one
decapsulated and correctly forwarded by the DUT/SUT without loss. ingress interface of a DUT/SUT are decapsulated and correctly
forwarded by the DUT/SUT on one or more egress interfaces without
loss.
Procedure Procedure:
Encapsulated traffic is offered to the egress interface of a Source DUT/SUT Destination
DUT/SUT <Figure 1> that has been configured to decapsulate the Test Port Test Port(s)
frames. +---------+ +-----------+ +---------+
| | | | | |
| | | Egress|------->| |
| | | | | |
| |--(Tunnel)-->|Ingress | | |
| | | | | |
| | | Egress|------->| |
| | | | | |
+---------+ +-----------+ +---------+
The constitution of traffic SHOULD consist of either: Figure 4
---------
Figure 4 shows the setup for testing the decapsulation throughput
of the DUT/SUT. One or more tunnels are created between the source
test port and the DUT/SUT. Encapsulated multicast traffic will
then be offered by the source test port, decapsulated by the
DUT/SUT and forwarded to the destination test port(s).
The DUT/SUT SHOULD be configured such that the traffic across each
egress interface will consist of either:
a) A single tunnel encapsulating one or more multicast address a) A single tunnel encapsulating one or more multicast address
groups OR groups OR
b) Multiple tunnels, each encapsulating one or more multicast b) Multiple tunnels, each encapsulating one or more multicast
address groups. address groups.
Due to the nature of decapsulation, the offered load on the The number of multicast groups per tunnel MUST be the same when the
encapsulated egress port will be less than the offered load on the DUT/SUT is configured in a multiple tunnel configuration. All
decapsulated egress interface. destination test ports MUST join all multicast group addresses
offered by the source test port. Each egress interface MUST
be configured with the same MTU.
Results A search algorithm MUST be utilized to determine the decapsulation
throughput as defined in [Du98].
Based on the resulting decapsulated frame size, parameters to be Reporting Format:
measured MUST include the maximum offered load at which no frame
loss occurred, the number of encapsulated multicast frames offered
per tunnel interface, and the number of multicast frames received
after decapsulation.
The nature of the traffic stream contributing to the result MUST be The following configuration parameters MUST be reflected in the
reported. All required reporting parameters of multicast results specific to this methodology:
decapsulation throughput MUST be reflected in the results report,
such as the transmitted packet size(s), decapsulated frame size, o Number of tested egress interfaces on the DUT/SUT
and transmit duration of offered frames. o Test duration
o IGMP version
o Total number of multicast groups
o MTU size of DUT/SUT interfaces
The following results MUST be reflected in the results specific to
this methodology:
o Measured Decapsulated Throughput as defined in RFC2432
[Du98]
o Originating encapsulation format
o Decapsulated frame size
o Originating encapsulated frame size
o Number of tunnels
o Number of multicast groups per tunnel
The Decapsulated Throughput results SHOULD be reported in the
format of a table and specific to this test there SHOULD be rows
for each originating encapsulated frame size. Each row or
iteration SHOULD specify the offered load, decapsulated frame size,
total number of offered frames and the decapsulation throughput.
4.4.3. Re-encapsulation Throughput 4.4.3. Re-encapsulation Throughput
Objective Objective:
To determine the maximum rate at which frames of one encapsulated To determine the maximum rate at which frames of one encapsulated
format offered a DUT/SUT are converted to another encapsulated format offered to one ingress interface of a DUT/SUT are converted
format and correctly forwarded by the DUT/SUT without loss. to another encapsulated format and correctly forwarded by the
DUT/SUT to one or more egress interfaces without loss.
Procedure Procedure:
Encapsulated traffic of one type is offered to the egress interface Source DUT/SUT Destination
of a DUT/SUT <Figure 1> that has been configured to re-encapsulate Test Port Test Port(s)
the frames using a different encapsulation format. +---------+ +---------+ +---------+
| | | | | |
| | | Egress|-(Tunnel)->| |
| | | | | |
| |-(Tunnel)->|Ingress | | |
| | | | | |
| | | Egress|-(Tunnel)->| |
| | | | | |
+---------+ +---------+ +---------+
The DUT/SUT SHOULD be configured such that the constitution of Figure 5
traffic will consist of either: ---------
c) A single tunnel encapsulating one or more multicast address Figure 5 shows the setup for testing the Re-encapsulation
throughput of the DUT/SUT. The source test port will offer
encapsulated traffic of one type to the DUT/SUT, which has been
configured to re-encapsulate the offered frames using a different
encapsulation format. The DUT/SUT will then forward the re-
encapsulated frames to the destination test port(s).
The DUT/SUT SHOULD be configured such that the traffic across each
egress interface will consist of either:
a) A single tunnel encapsulating one or more multicast address
groups OR groups OR
d) Multiple tunnels, each encapsulating one or more multicast b) Multiple tunnels, each encapsulating one or more multicast
address groups. address groups.
Each tunnel created by the ingress DUT/SUT SHOULD contain the same The number of multicast groups per tunnel MUST be the same when the
number of multicast address groups per tunnel interface. DUT/SUT is configured in a multiple tunnel configuration.
The offered load on the ingress port MUST not oversubscribe the In addition, the DUT/SUT SHOULD be configured such that the number
outbound link with respect to the offered load at the higher end of of tunnels on the ingress and each egress interface are the same.
the DUT/SUTs capacity based on the encapsulated frame size. All destination test ports MUST join all multicast group addresses
offered by the source test port. Each egress interface MUST be
configured with the same MTU.
Results A search algorithm MUST be utilized to determine the re-
encapsulation throughput as defined in [Du98].
Based on the resulting encapsulated frame size, parameters to be Reporting Format:
measured MUST include the maximum offered load at which no frame
loss occurred, the number of encapsulated multicast frames offered
per tunnel interface, and the number of encapsulated multicast
frames received per tunnel interface.
The nature of the traffic stream contributing to the result MUST be The following configuration parameters MUST be reflected in the
reported. All required reporting parameters of multicast results specific to this methodology:
encapsulation throughput MUST be reflected in the results report,
such as the encapsulation format on the egress interface, o Number of tested egress interfaces on the DUT/SUT
transmitted packet size(s), encapsulated frame size, and transmit o Test duration
duration of offered frames. o IGMP version
o Total number of multicast groups
o MTU size of DUT/SUT interfaces
The following results MUST be reflected in the results specific to
this methodology:
o Measured Re-encapsulated Throughput as defined in RFC2432
[Du98]
o Originating encapsulation format
o Decapsulated frame size
o Originating encapsulated frame size
o Number of tunnels
o Number of multicast groups per tunnel
The Decapsulated Throughput results SHOULD be reported in the
format of a table and specific to this test there SHOULD be rows
for each originating encapsulated frame size. Each row or
iteration SHOULD specify the offered load, decapsulated frame size,
total number of offered frames and the decapsulation throughput
5. Forwarding Latency 5. Forwarding Latency
This section presents methodologies relating to the This section presents methodologies relating to the
characterization of the forwarding latency of a DUT/SUT in a characterization of the forwarding latency of a DUT/SUT in a
multicast environment. It extends the concept of latency multicast environment. It extends the concept of latency
characterization presented in RFC 2544. characterization presented in RFC 2544.
In order to lessen the effect of packet buffering in the DUT/SUT, To lessen the effect of frame buffering in the DUT/SUT, the latency
the latency tests MUST be run at the measured multicast throughput tests MUST be run at the measured multicast throughput level of the
level of the DUT; multicast latency at other offered loads is DUT; multicast latency at other offered loads is optional.
optional.
Lastly, RFC 1242 and RFC 2544 draw distinction between two classes Lastly, RFC 1242 and RFC 2544 draw a distinction between device
of devices: "store and forward" and "bit-forwarding." Each class types: "store and forward" and "bit-forwarding." Each type impacts
impacts how latency is collected and subsequently presented. See how latency is collected and subsequently presented. See the
the related RFCs for more information. In practice, much of the related RFCs for more information. In practice, much of the test
test equipment will collect the latency measurement for one class equipment will collect the latency measurement for one type or the
or the other, and, if needed, mathematically derive the reported other, and, if needed, mathematically derive the reported value by
value by the addition or subtraction of values accounting for the addition or subtraction of values accounting for medium
medium propagation delay of the packet, bit times to the timestamp propagation delay of the frame, bit times to the timestamp trigger
trigger within the packet, etc. Test equipment vendors SHOULD within the frame, etc.
provide documentation regarding the composition and calculation
latency values being reported. The user of this data SHOULD
understand the nature of the latency values being reported,
especially when comparing results collected from multiple test
vendors. (E.g., If test vendor A presents a "store and forward"
latency result and test vendor B presents a "bit-forwarding"
latency result, the user may erroneously conclude the DUT has two
differing sets of latency values.)
5.1. Multicast Latency 5.1. Multicast Latency
Objective Objective:
To produce a set of multicast latency measurements from a single, To produce a set of multicast latency measurements from a single,
multicast ingress port of a DUT/SUT through multiple, egress multicast ingress interface of a DUT/SUT through multiple, egress
multicast ports of that same DUT/SUT as provided for by the metric multicast interfaces of that same DUT/SUT as provided for by the
"Multicast Latency" in RFC 2432. metric "Multicast Latency" in RFC 2432.
The procedures highlighted below attempt to draw from the The Procedures highlighted below attempt to draw from the
collection methodology for latency in RFC 2544 to the degree collection methodology for latency in RFC 2544 to the degree
possible. The methodology addresses two topological scenarios: one possible. The methodology addresses two topological scenarios: one
for a single device (DUT) characterization; a second scenario is for a single device (DUT) characterization; a second scenario is
presented or multiple device (SUT) characterization. presented or multiple device (SUT) characterization.
Procedure Procedure:
If the test trial is to characterize latency across a single Device If the test trial is to characterize latency across a single Device
Under Test (DUT), an example test topology might take the form of Under Test (DUT), an example test topology might take the form of
Figure 1 in section 3. That is, a single DUT with one ingress Figure 1 in section 3. That is, a single DUT with one ingress
interface receiving the multicast test traffic from packet- interface receiving the multicast test traffic from frame-
transmitting component of the test apparatus and n egress transmitting component of the test apparatus and n egress
interfaces on the same DUT forwarding the multicast test traffic interfaces on the same DUT forwarding the multicast test traffic
back to the packet-receiving component of the test apparatus. Note back to the frame-receiving component of the test apparatus. Note
that n reflects the number of TESTED egress interfaces on the DUT that n reflects the number of TESTED egress interfaces on the DUT
actually expected to forward the test traffic (as opposed to actually expected to forward the test traffic (as opposed to
configured but untested, non-forwarding interfaces, for example). configured but untested, non-forwarding interfaces, for example).
If the multicast latencies are to be taken across multiple devices If the multicast latencies are to be taken across multiple devices
forming a System Under Test (SUT), an example test topology might forming a System Under Test (SUT), an example test topology might
take the form of Figure 2 in section 3. take the form of Figure 2 in section 3.
The trial duration SHOULD be 120 seconds. Departures to the The trial duration SHOULD be 120 seconds to be consistent with RFC
suggested traffic class guidelines MUST be disclosed with the 2544. The nature of the latency measurement, "store and forward"
respective trial results. The nature of the latency measurement, or "bit forwarding," MUST be associated with the related test
"store and forward" or "bit forwarding," MUST be associated with trial(s) and disclosed in the results report.
the related test trial(s) and disclosed in the results report.
End-to-end reach ability of the test traffic path MUST be verified End-to-end reach ability of the test traffic path MUST be verified
prior to the engagement of a test trial. This implies that prior to the engagement of a test trial. This implies that
subsequent measurements are intended to characterize the latency subsequent measurements are intended to characterize the latency
across the tested device's or devices' normal traffic forwarding across the tested device's or devices' normal traffic forwarding
path (e.g., faster hardware-based engines) of the device(s) as path (e.g., faster hardware-based engines) of the device(s) as
opposed a non-standard traffic processing path (e.g. slower, opposed a non-standard traffic processing path (e.g. slower,
software-based exception handlers). If the test trial is to be software-based exception handlers). If the test trial is to be
executed with the intent of characterizing a non-optimal, executed with the intent of characterizing a non-optimal,
forwarding condition, then a description of the exception forwarding condition, then a description of the exception
processing conditions being characterized MUST be included with the processing conditions being characterized MUST be included with the
trial's results. trial's results.
A test traffic stream is presented to the DUT. At the mid-point of A test traffic stream is presented to the DUT. It is RECOMMENDED to
the trial's duration, the test apparatus MUST inject a uniquely offer traffic at the measured aggregated multicast throughput rate
identifiable ("tagged") packet into the test traffic packets being (Section 4.3). At the mid-point of the trial's duration, the test
presented. This tagged packet will be the basis for the latency apparatus MUST inject a uniquely identifiable ("tagged") frame into
measurements. By "uniquely identifiable," it is meant that the test the test traffic frames being presented. This tagged frame will be
apparatus MUST be able to discern the "tagged" packet from the the basis for the latency measurements. By "uniquely identifiable,"
other packets comprising the test traffic set. A packet generation it is meant that the test apparatus MUST be able to discern the
timestamp, Timestamp A, reflecting the completion of the "tagged" frame from the other frames comprising the test traffic
transmission of the tagged packet by the test apparatus, MUST be set. A frame generation timestamp, Timestamp A, reflecting the
determined. completion of the transmission of the tagged frame by the test
apparatus, MUST be determined.
The test apparatus then monitors packets from the DUT's tested The test apparatus then monitors frames from the DUT's tested
egress port(s) for the expected tagged packet(s) until the egress interface(s) for the expected tagged frame(s) until the
cessation of traffic generation at the end of the configured trial cessation of traffic generation at the end of the configured trial
duration.A value of the Offered Load presented the DUT/SUT MUST be duration.
noted.
The test apparatus MUST record the time of the successful detection The test apparatus MUST record the time of the successful detection
of a tagged packet from a tested egress interface with a timestamp, of a tagged frame from a tested egress interface with a timestamp,
Timestamp B. A set of Timestamp B values MUST be collected for all Timestamp B. A set of Timestamp B values MUST be collected for all
tested egress interfaces of the DUT/SUT. tested egress interfaces of the DUT/SUT. See RFC 1242 [Br91] for
additional discussion regarding store and forward devices and bit
forwarding devices.
A trial MUST be considered INVALID should any of the following A trial MUST be considered INVALID should any of the following
conditions occur in the collection of the trial data: conditions occur in the collection of the trial data:
. Forwarded test packets directed to improper destinations. o Forwarded test frames directed to improper destinations.
. Unexpected differences between Intended Load and Offered Load o Unexpected differences between Intended Load and Offered
or unexpected differences between Offered Load and the Load or unexpected differences between Offered Load and the
resulting Forwarding Rate(s) on the DUT/SUT egress ports. resulting Forwarding Rate(s) on the DUT/SUT egress ports.
. Forwarded test packets improperly formed or packet header o Forwarded test frames improperly formed or frame header
fields improperly manipulated. fields improperly manipulated.
. Failure to forward required tagged packet(s) on all expected o Failure to forward required tagged frame(s) on all expected
egress interfaces. egress interfaces.
. Reception of a tagged packet by the test apparatus outside the o Reception of a tagged frame by the test apparatus outside
configured test duration interval or 5 seconds, whichever is the configured test duration interval or 5 seconds,
greater. whichever is greater.
Data from invalid trials SHOULD be considered inconclusive. Data Data from invalid trials SHOULD be considered inconclusive. Data
from invalid trials MUST not form the basis of comparison. from invalid trials MUST not form the basis of comparison.
The set of latency measurements, M, composed from each latency The set of latency measurements, M, composed from each latency
measurement taken from every ingress/tested egress interface measurement taken from every ingress/tested egress interface
pairing MUST be determined from a valid test trial: pairing MUST be determined from a valid test trial:
M = { (Timestamp B(E0) - Timestamp A), M = { (Timestamp B(E0) - Timestamp A),
(Timestamp B(E1) - Timestamp A), ... (Timestamp B(E1) - Timestamp A), ...
(Timestamp B(En) - Timestamp A) } (Timestamp B(En) - Timestamp A) }
where (E0 ... En) represents the range of all tested egress where (E0 ... En) represents the range of all tested egress
interfaces and Timestamp B represents a tagged packet detection interfaces and Timestamp B represents a tagged frame detection
event for a given DUT/SUT tested egress interface. event for a given DUT/SUT tested egress interface.
Results A more continuous profile MAY be built from a series of individual
measurements.
Two types of information MUST be reported: 1) the set of latency Reporting Format:
measurements and 2) the significant environmental, methodological,
or device particulars giving insight into the test or its results.
Specifically, when reporting the results of a VALID test trial, the The following configuration parameters MUST be reflected in the
set of ALL latencies related to the tested ingress interface and results specific to this methodology:
each tested egress DUT/SUT interface of MUST be presented. The
time units of the presented latency MUST be uniform and with
sufficient precision for the medium or media being tested. Results
MAY be offered in tabular format and SHOULD preserve the
relationship of latency to ingress/egress interface to assist in
trending across multiple trials.
The Offered Load of the test traffic presented the DUT/SUT, size of o Frame size(s)
the "tagged" packet, transmit duration of offered frames and nature o Number of tested egress interfaces on the DUT/SUT
(i.e., store-and-forward or bit-forwarding) of the trial's o Test duration
measurement MUST be associated with any reported test trial's o IGMP version
result. o Offered load
o Total number of multicast groups
The following results MUST be reflected in the results specific to
this methodology:
o The time units of the presented latency MUST be uniform and
with sufficient precision for the medium or media being
tested.
o Specifically, when reporting the results of a valid test
trial, the set of all latencies related to the tested
ingress and each tested egress DUT/SUT interface of MUST be
presented.
The latency results for each test SHOULD be reported in the form of
a table, with a row for each of the tested frame sizes per the
recommended frame sizes in section 3.1.3, and SHOULD preserve the
relationship of latency to ingress/egress interface(s) to assist in
trending across multiple trials.
5.2. Min/Max Multicast Latency 5.2. Min/Max Multicast Latency
Objective Objective:
To determine the difference between the maximum latency measurement To determine the difference between the maximum latency measurement
and the minimum latency measurement from a collected set of and the minimum latency measurement from a collected set of
latencies produced by the Multicast Latency benchmark. latencies produced by the Multicast Latency benchmark.
Procedure Procedure:
Collect a set of multicast latency measurements, as prescribed in Collect a set of multicast latency measurements over a single test
section 5.1. This will produce a set of multicast latencies, M, duration, as prescribed in section 5.1. This will produce a set of
where M is composed of individual forwarding latencies between DUT multicast latencies, M, where M is composed of individual
packet ingress and DUT packet egress port pairs. E.g.: forwarding latencies between DUT frame ingress and DUT frame egress
port pairs. E.g.:
M = {L(I,E1),L(I,E2), , L(I,En)} M = {L(I,E1),L(I,E2), ..., L(I,En)}
where L is the latency between a tested ingress port, I, of the where L is the latency between a tested ingress interface, I, of
DUT, and Ex a specific, tested multicast egress port of the DUT. the DUT, and Ex a specific, tested multicast egress interface of
E1 through En are unique egress ports on the DUT. the DUT. E1 through En are unique egress interfaces on the DUT.
From the collected multicast latency measurements in set M, From the collected multicast latency measurements in set M,
identify MAX(M), where MAX is a function that yields the largest identify MAX(M), where MAX is a function that yields the largest
latency value from set M. latency value from set M.
Identify MIN(M), when MIN is a function that yields the smallest Identify MIN(M), when MIN is a function that yields the smallest
latency value from set M. latency value from set M.
The Max/Min value is determined from the following formula: The Max/Min value is determined from the following formula:
Result = MAX(M) MIN(M) Result = MAX(M) - MIN(M)
Results A more continuous profile MAY be built from a series of individual
measurements.
The result MUST be represented as a single numerical value in time Reporting Format:
units consistent with the corresponding latency measurements. In
addition, the number of tested egress ports on the DUT MUST be
reported.
The nature of the traffic stream contributing to the result MUST be The following configuration parameters MUST be reflected in the
reported. All required reporting parameters of multicast latency results specific to this methodology:
MUST be reflected in the min/max results report, such as the
transmitted packet size(s), offered load of the packet stream in o Frame size(s)
which the tagged packet was presented to the DUT and transmit o Number of tested egress interfaces on the DUT/SUT
duration of offered frames. o Test duration
o IGMP version
o Offered load
o Total number of multicast groups
The following results MUST be reflected in the results specific to
this methodology:
o The result of the min/max value represented as a single
numerical value in time units consistent with the
corresponding latency measurements.
o Specifically, when reporting the results of a valid test
trial, the set of all latencies related to the tested
ingress interface MUST be reported.
The time units of the presented latency MUST be uniform and with
sufficient precision for the medium or media being tested. The
latency results for each test SHOULD be reported in the form of a
table, with a row for each of the tested frame sizes per the
recommendations in section 3.1.3, and SHOULD preserve the
relationship of latency to ingress/egress interface(s) to assist in
trending across multiple trials.
6. Overhead 6. Overhead
This section presents methodology relating to the characterization This section presents methodology relating to the characterization
of the overhead delays associated with explicit operations found in of the overhead delays associated with explicit operations found in
multicast environments. multicast environments.
6.1. Group Join Delay 6.1. Group Join Delay
Objective Objective:
To determine the time duration it takes a DUT/SUT to start To determine the time duration it takes a DUT/SUT to start
forwarding multicast packets from the time a successful IGMP group forwarding multicast frames from the time a successful IGMP group
membership report has been issued to the DUT/SUT. membership report has been issued to the DUT/SUT.
Procedure Procedure:
Traffic is sent on the source port at the same time as the IGMP Prior to sending any IGMP Group Membership Reports used to
JOIN Group message is transmitted from the destination ports. The calculate the group join delay, it MUST be verified through
join delay is the difference in time from when the IGMP Join is externally observable means that the destination test ports are not
sent (timestamp A) and the first frame is forwarded to a receiving currently a member of any of the specified multicast groups. If
member port (timestamp B). any of the egress interfaces forward multicast frames, the test is
not valid.
Group Join delay = timestamp B - timestamp A Once verification is complete, multicast traffic for all relevant
multicast group addresses MUST be offered to the ingress interface
prior to receipt or processing of any IGMP Group Membership Report
messages. It is RECOMMENDED to offer traffic at the measured
aggregated multicast throughput rate (Section 4.3).
One of the keys is to transmit at the fastest rate the DUT/SUT can After the multicast traffic has been started, each destination test
handle multicast frames. This is to get the best resolution and port (See Figure 1) SHOULD send one IGMP Group Membership Report
the least margin of error in the Join Delay. with one or more (IGMP V3) multicast group(s) specified. All
destination test ports MUST join all multicast groups offered on
the ingress interface of the DUT/SUT. The test MUST be performed
with one multicast group and SHOULD be performed with multiple
groups.
However, you do not want to transmit the frames so fast that frames The join delay is the difference in time from when the IGMP Group
are dropped by the DUT/SUT. Traffic should be sent at the Membership message is sent (timestamp A) and the first frame of the
throughput rate determined by the forwarding tests of section 4. multicast group is forwarded to a receiving egress interface
(timestamp B).
Results Group Join delay time = timestamp B - timestamp A
The parameter to be measured is the join delay time for each Timestamp A MUST be the time the last bit of the IGMP group
multicast group address per destination port. In addition, the membership report is sent from the destination test port; timestamp
number of frames transmitted and received and percent loss may be B MUST be the time the first bit of the first valid multicast frame
reported. is forwarded on the egress interface of the DUT/SUT.
Reporting Format:
The following configuration parameters MUST be reflected in the
results specific to this methodology:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Total number of multicast groups
The following results MUST be reflected in the results specific to
this methodology:
o The group join delay time per multicast group address
o The group join delay time per egress interface(s)
The Group Join Delay results for each test SHOULD be reported in
the form of a table, with a row for each of the tested frame sizes
per the recommendations in section 3.1.3. Each row or iteration
SHOULD specify the group join delay time for each multicast group
per destination interface, number of frames transmitted and number
of frames received for that iteration.
6.2. Group Leave Delay 6.2. Group Leave Delay
Objective Objective:
To determine the time duration it takes a DUT/SUT to cease To determine the time duration it takes a DUT/SUT to cease
forwarding multicast packets after a corresponding IGMP "Leave forwarding multicast frames after a corresponding IGMP Leave Group
Group" message has been successfully offered to the DUT/SUT. message has been successfully offered to the DUT/SUT.
Procedure Procedure:
Traffic is sent on the source port at the same time as the IGMP Prior to sending any IGMP Group Leave Group messages used to
Leave Group messages are transmitted from the destination ports. calculate the group leave delay, it MUST be verified through
The leave delay is the difference in time from when the IGMP leave externally observable means that the destination test ports are
is sent (timestamp A) and the last frame is forwarded to a currently a member of all the specified multicast groups. If any
receiving member port (timestamp B). of the destination test ports do not receive multicast frames, the
test is not valid.
Group Leave delay = timestamp B - timestamp A Once verification is complete, multicast traffic for all relevant
multicast group addresses MUST be offered to the ingress interface
prior to receipt or processing of any IGMP Leave Group messages.
It is RECOMMENDED to offer traffic at the measured aggregated
multicast throughput rate (Section 4.3).
One of the keys is to transmit at the fastest rate the DUT/SUT can After the multicast traffic has been started, each destination test
handle multicast frames. This is to get the best resolution and port (See Figure 1) MUST send one IGMP Leave Group message for each
least margin of error in the Leave Delay. However, you do not want multicast group specified. All destination test ports MUST leave
to transmit the frames too fast that frames are dropped by the all relevant multicast groups offered on the ingress interface of
DUT/SUT. Traffic should be sent at the throughput rate determined the DUT/SUT. The test MUST be performed with one multicast group
by the forwarding tests of section 4. and SHOULD be performed with multiple groups.
Results The leave delay is the difference in time from when the IGMP Leave
Group message is sent (timestamp A) and the last frame of the
multicast group is forwarded to a receiving egress interface
(timestamp B).
The parameter to be measured is the leave delay time for each Group Leave delay time = timestamp B - timestamp A
multicast group address per destination port. In addition, the
number of frames transmitted and received and percent loss may be Timestamp A MUST be the time the last bit of the IGMP Leave Group
reported. message is sent from the destination test port; timestamp B MUST be
the time the last bit of the last valid multicast frame is
forwarded on the egress interface of the DUT/SUT.
Reporting Format:
The following configuration parameters MUST be reflected in the
results specific to this methodology:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Total number of multicast groups
The following results MUST be reflected in the results specific to
this methodology:
o The group leave delay time per multicast group address
o The group leave delay time per egress interface(s)
The Group Leave Delay results for each test SHOULD be reported in
the form of a table, with a row for each of the tested frame sizes
per the recommendations in section 3.1.3. Each row or iteration
SHOULD specify the group leave delay time for each multicast group
per destination interface, number of frames transmitted and number
of frames received for that iteration.
7. Capacity 7. Capacity
This section offers terms relating to the identification of This section offers terms relating to the identification of
multicast group limits of a DUT/SUT. multicast group limits of a DUT/SUT.
7.1. Multicast Group Capacity 7.1. Multicast Group Capacity
Objective Objective:
To determine the maximum number of multicast groups a DUT/SUT can To determine the maximum number of multicast groups a DUT/SUT can
support while maintaining the ability to forward multicast frames support while maintaining the ability to forward multicast frames
to all multicast groups registered to that DUT/SUT. to all multicast groups registered to that DUT/SUT.
Procedure Procedure:
One or more destination ports of DUT/SUT will join an initial One or more egress interfaces of DUT/SUT will join an initial
number of groups. number of multicast groups.
Then after a delay (enough time for all ports to join) the source After a delay as determined by section 6.1, the ingress interface
port will transmit to each group at a transmission rate that the MUST transmit to each group at a specified offered load.
DUT/SUT can handle without dropping IP Multicast frames.
If all frames sent are forwarded by the DUT/SUT and received the If at least one frame for each multicast group is forwarded
test iteration is said to pass at the current capacity. properly by the DUT/SUT to each participating egress interface, the
iteration is said to pass at the current capacity.
If the iteration passes at the capacity the test will add an user If the iteration passes, the test will add a user defined
defined incremental value of groups to each receive port. incremental value of groups to each egress interface. At the new
group level and resultant capacity as stated above, run the
iteration again.
The iteration is to run again at the new group level and capacity Once the test fails, the last/previous iteration capacity that
tested as stated above. passed is the stated Maximum Group Capacity result.
Once the test fails at a capacity the capacity is stated to be the Reporting Format:
last Iteration that pass at a giving capacity.
Results The following configuration parameters MUST be reflected in the
results specific to this methodology:
The parameter to be measured is the total number of group addresses o Frame size(s)
that were successfully forwarded with no loss. o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Offered load
In addition, the nature of the traffic stream contributing to the The following results MUST be reflected in the results specific to
result MUST be reported. All required reporting parameters MUST be this methodology:
reflected in the results report, such as the transmitted packet
size(s) and offered load of the packet stream. o The total number of multicast group addresses that were
successfully forwarded through the DUT/SUT
The Multicast Group Capacity results for each test SHOULD be
reported in the form of a table, with a row for each of the tested
frame sizes per the recommendations in section 3.1.3. Each row or
iteration SHOULD specify the number of multicast groups joined per
destination interface, number of frames transmitted and number of
frames received for that iteration.
8. Interaction 8. Interaction
Network forwarding devices are generally required to provide more Network forwarding devices are generally required to provide more
functionality than just the forwarding of traffic. Moreover, functionality than just the forwarding of traffic. Moreover,
network-forwarding devices may be asked to provide those functions network-forwarding devices may be asked to provide those functions
in a variety of environments. This section offers terms to assist in a variety of environments. This section offers procedures to
in the characterization of DUT/SUT behavior in consideration of assist in the characterization of DUT/SUT behavior in consideration
potentially interacting factors. of potentially interacting factors.
8.1. Forwarding Burdened Multicast Latency 8.1. Forwarding Burdened Multicast Latency
Objective Objective:
To produce a set of multicast latency measurements from a single, To produce a set of multicast latency measurements from a single
multicast ingress port of a DUT/SUT through multiple, egress multicast ingress interface of a DUT/SUT through multiple egress
multicast ports of that same DUT/SUT as provided for by the metric multicast interfaces of that same DUT/SUT as provided for by the
"Multicast Latency" in RFC 2432, while burdening the DUT/SUT by metric "Multicast Latency" in RFC 2432 while under the influence of
injecting addresses into the DUT/SUT address table. a traffic forwarding requirement.
Procedure Procedure:
The Multicast Latency metrics can be influenced by forcing the The Multicast Latency metrics can be influenced by forcing the
DUT/SUT to perform extra processing of packets while multicast DUT/SUT to perform extra processing of packets while multicast
class traffic is being forwarded for latency measurements. As class traffic is being forwarded for latency measurements.
described in Section 5.1, a set of ports on the tester will be
designated to be the source and destination in this test. In
addition to this setup, another set of ports will be selected to
transmit some multicast class traffic that is destined to multicast
group addresses that have not been joined by these additional set
of ports.
For example, ports 1,2, 3, and 4 form the burdened response setup The Burdened Forwarding Latency test MUST follow the described
(setup A) which is used to obtain the latency metrics and ports 5, setup in Section 5.1.
6, 7, and 8 form the non-burdened response setup (setup B) which
will afflict the burdened response setup. Setup B traffic will
then join multicast group addresses not joined by the ports in this
setup. By sending such multicast class traffic, the DUT/SUT will
perform a lookup on the packets that will affect the processing of
setup A traffic.
Results Perform a baseline measurement of latency as described in Section
5.1. After the baseline measurement is obtained, the test is
repeated with the ingress interface offering an additional set of
user specified multicast group addresses which have not been joined
by the destination test port(s). The offered load MUST be the same
as was used in the baseline measurement.
Result reports MUST include the following parameters for each By sending such multicast class traffic, the DUT/SUT may perform a
iteration: transmitted packet size, the number of frames offered, lookup on the frames that may affect the processing of traffic
number of frames received per each group, number of multicast destined for the egress interface(s).
groups and forwarding rate in frames per second, number of
addresses injected into address table for that iteration, and
transmit duration of offered frames. The result report must also
specify the number of source and destination ports within the
multicast group, as well as the ports designated to inject
addresses throughout the test.
The following metrics MUST be reported: Reporting Format:
1) The set of latency measurements
2) The nature of latency measured (i.e., store-and-forward or
bit-forwarding)
3) The significant environmental, methodological, or device
particulars giving insight into the test or its results.
Constructing a table that contains the latency vs. number of Similar to Section 5.1, the following configuration parameters MUST
injected addresses is desirable. be reflected in the results specific to this methodology:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Total number of multicast groups in the baseline setup
o Total number of additional multicast groups used to burden
the setup
The following results MUST be reflected in the results specific to
this methodology:
o The time units of the presented latency MUST be uniform and
with sufficient precision for the medium or media being
tested
o Specifically, when reporting the results of a valid test
trial, the set of all latencies related to the tested
ingress and each tested egress DUT/SUT interface of MUST be
presented
o Reported results from baseline measurement; section 5.1
The latency results for each test SHOULD be reported in the form of
a table, with a row for each of the tested frame sizes per the
recommended frame sizes in section 3.1.3, and SHOULD preserve the
relationship of latency to ingress/egress interface(s) to assist in
trending across multiple trials.
8.2. Forwarding Burdened Group Join Delay 8.2. Forwarding Burdened Group Join Delay
Objective Objective:
To determine the time duration it takes a DUT/SUT to start To determine the time duration it takes a DUT/SUT to start
forwarding multicast packets from the time a successful IGMP group forwarding multicast frames from the time a successful IGMP Group
membership report has been issued to the DUT/SUT while burdening Membership Report has been issued to the DUT/SUT while under the
the DUT/SUT by injecting addresses into the DUT/SUT address table influence of a traffic forwarding requirement.
on a unrelated set of ports.
Procedure Procedure:
The port configuration in this test is similar to the one described The Group Join Delay metrics can be influenced by forcing the
in Sections 6.1 and 8.1, however, the additional set of transmit DUT/SUT to perform extra process of packets while attempting to
ports, which comprise setup B, do not send multicast class update and maintain the IP multicast address forwarding table.
traffic. Setup A traffic must be influenced in such a way that will
affect the DUT's/SUT's ability to process Group Join messages.
Therefore, in this test, the ports in setup B will send a set of
IGMP Group Join messages while the ports in setup A are also
simultaneously joining its own set of group addresses. Since the
two sets of group addresses are independent of each other, the
group join delay for setup A may be different than in the case when
there were no other group addresses being joined.
Results The Forwarding Burdened Group Join Delay test MUST follow the
described setup in Section 6.1.
Similar to Section 6.1, the parameter to be measured is the leave Perform a baseline measurement of group join delay as described in
delay time for each multicast group address per destination port. Section 6.1. After the baseline measurement is obtained, the test
Result reports MUST specify the number of multicast groups joined is repeated with the ingress interface offering an additional set
in the join delay port group, the number of groups joined by the of user specified multicast group address which have not been
unrelated ports, the number of source and destination ports within joined by the destination test port(s). The offered load MUST be
the join delay port group, and the number of unrelated ports the same as was used in the baseline measurement.
designated to inject addresses throughout the test.
Constructing a table that contains the join delay time vs. number By sending such multicast class traffic, the DUT/SUT may perform a
of injected addresses is desirable. lookup on the frames that may affect the processing of the IGMP
Group Report messages.
Reporting Format:
Similar to Section 6.1, the following configuration parameters MUST
be reflected in the results specific to this methodology:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Total number of multicast groups in the baseline setup
o Total number of additional multicast groups used to burden
the setup
The following results MUST be reflected in the results specific to
this methodology:
o The group join delay time per multicast group address
o The group join delay time per egress interface(s)
o Reported results from baseline measurement; section 6.1
The Group Join Delay results for each test SHOULD be reported in
the form of a table, with a row for each of the tested frame sizes
per the recommendations in section 3.1.3. Each row or iteration
SHOULD specify the group join delay time for each multicast group
per destination interface, number of frames transmitted and number
of frames received for that iteration.
9. Security Considerations 9. Security Considerations
As this document is solely for the purpose of providing metric As this document is solely for the purpose of providing metric
methodology and describes neither a protocol nor a protocol's methodology and describes neither a protocol nor a protocol's
implementation, there are no security considerations associated implementation, there are no security considerations associated
with this document. with this document.
10. Acknowledgements 10. Acknowledgements
The Benchmarking Methodology Working Group of the IETF and
particularly Kevin Dubray, Juniper Networks, are to be thanked for
the many suggestions they collectively made to help complete this
document.
11. Contributions
The authors would like to acknowledge the following individuals for The authors would like to acknowledge the following individuals for
their help and participation of the compilation and editing of this their help and participation of the compilation of this document:
document Ralph Daniels, Netcom Systems, who made significant Hardev Soor, Ixia, and Ralph Daniels, Spirent Communications, both
contributions to earlier versions of this draft, Michele Bustos, who made significant contributions to the earlier versions of this
IXIA, and Kevin Dubray, Juniper Networks. document. In addition, the authors would like to acknowledge the
members of the task team who helped bring this document to
fruition: Michele Bustos, Tony De La Rosa, David Newman and Jerry
Perser.
11. References 12. References
Normative References
[Br91] Bradner, S., "Benchmarking Terminology for Network [Br91] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991. Interconnection Devices", RFC 1242, July 1991.
[Br96] Bradner, S., and J. McQuaid, "Benchmarking Methodology for [Br96] Bradner, S., and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[Br97] Bradner, S. "Use of Keywords in RFCs to Reflect Requirement [Br97] Bradner, S. "Use of Keywords in RFCs to Reflect Requirement
Levels, RFC 2119, March 1997 Levels, RFC 2119, March 1997
[Du98] Dubray, K., "Terminology for IP Multicast Benchmarking", RFC [Du98] Dubray, K., "Terminology for IP Multicast Benchmarking", RFC
2432, October 1998. 2432, October 1998.
[Ma98] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998.
Informative References
[Ca02] Cain, B., et al., "Internet Group Management Protocol, Version
3", RFC 3376, October 2002.
[De89] Deering, S., "Host Extensions for IP Multicasting", STD 5, RFC
1112, August 1989.
[Fe97] Fenner, W., "Internet Group Management Protocol, Version 2",
RFC 2236, November 1997.
[Hu95] Huitema, C. "Routing in the Internet." Prentice-Hall, 1995. [Hu95] Huitema, C. "Routing in the Internet." Prentice-Hall, 1995.
[Ka98] Kosiur, D., "IP Multicasting: the Complete Guide to [Ka98] Kosiur, D., "IP Multicasting: the Complete Guide to
Interactive Corporate Networks", John Wiley & Sons, Inc, 1998. Interactive Corporate Networks", John Wiley & Sons, Inc, 1998.
[Ma98] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998.
[Mt98] Maufer, T. "Deploying IP Multicast in the Enterprise." [Mt98] Maufer, T. "Deploying IP Multicast in the Enterprise."
Prentice-Hall, 1998. Prentice-Hall, 1998.
[Se98] Semeria, C. and Maufer, T. "Introduction to IP Multicast 13. Author's Addresses
Routing." http://www.3com.com/nsc/501303.html 3Com Corp.,
1998.
12. Author's Addresses
Debra Stopp Debra Stopp
IXIA Ixia
26601 W. Agoura Rd. 26601 W. Agoura Rd.
Calabasas, CA 91302 Calabasas, CA 91302
USA USA
Phone: 818 871 1800 Phone: + 1 818 871 1800
EMail: debby@ixiacom.com EMail: debby@ixiacom.com
Hardev Soor Brooks Hickman
IXIA Spirent Communications
26601 W. Agoura Rd. 26750 Agoura Rd.
Calabasas, CA 91302 Calabasas, CA 91302
USA USA
Phone: 818 871 1800 Phone: + 1 818 676 2412
EMail: hardev@ixiacom.com EMail: brooks.hickman@spirentcom.com
13. Full Copyright Statement 14. Full Copyright Statement
"Copyright (C) The Internet Society (date). All Rights Reserved. "Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain others, and derivative works that comment on or otherwise explain
it or assist in its implementation may be prepared, copied, it or assist in its implementation may be prepared, copied,
published and distributed, in whole or in part, without restriction published and distributed, in whole or in part, without restriction
of any kind, provided that the above copyright notice and this of any kind, provided that the above copyright notice and this
paragraph are included on all such copies and derivative works. paragraph are included on all such copies and derivative works.
However, this document itself may not be modified in any way, such However, this document itself may not be modified in any way, such
as by removing the copyright notice or references to the Internet as by removing the copyright notice or references to the Internet
Society or other Internet organizations, except as needed for the Society or other Internet organizations, except as needed for the
purpose of developing Internet standards in which case the purpose of developing Internet standards in which case the
procedures for copyrights defined in the Internet Standards process procedures for copyrights defined in the Internet Standards process
must be followed, or as required to translate it into. must be followed, or as required to translate it into.
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/