draft-ietf-bmwg-sip-bench-meth-08.txt   draft-ietf-bmwg-sip-bench-meth-09.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working Group C. Davids
Group Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Internet-Draft V. Gurbani Intended status: Informational V. Gurbani
Intended status: Informational Bell Laboratories, Alcatel-Lucent Expires: August 18, 2014 Bell Laboratories,
Expires: July 12, 2013 S. Poretsky Alcatel-Lucent
S. Poretsky
Allot Communications Allot Communications
January 8, 2013 February 14, 2014
Methodology for Benchmarking SIP Networking Devices Methodology for Benchmarking Session Initiation Protocol (SIP) Devices:
draft-ietf-bmwg-sip-bench-meth-08 Basic session setup and registration
draft-ietf-bmwg-sip-bench-meth-09
Abstract Abstract
This document describes the methodology for benchmarking Session This document provides a methodology for benchmarking the Session
Initiation Protocol (SIP) performance as described in SIP Initiation Protocol (SIP) performance of devices. Terminology
benchmarking terminology document. The methodology and terminology related to benchmarking SIP devices is described in the companion
are to be used for benchmarking signaling plane performance with terminology document. Using these two documents, benchmarks can be
varying signaling and media load. Both scale and establishment rate obtained and compared for different types of devices such as SIP
are measured by signaling plane performance. The SIP Devices to be Proxy Servers, Registrars and Session Border Controllers. The term
benchmarked may be a single device under test (DUT) or a system under "performance" in this context means the capacity of the device-under-
test (SUT). Benchmarks can be obtained and compared for different test (DUT) to process SIP messages. Media streams are used only to
types of devices such as SIP Proxy Server, SBC, and server paired study how they impact the signaling behavior. The intent of the two
with a media relay or Firewall/NAT device. documents is to provide a normalized set of tests that will enable an
objective comparison of the capacity of SIP devices. Test setup
parameters and a methodology is necessary because SIP allows a wide
range of configuration and operational conditions that can influence
performance benchmark measurements. A standard terminology and
methodology will ensure that benchmarks have consistent definition
and were obtained following the same procedures.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2014.
This Internet-Draft will expire on July 12, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5
4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 5 4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 6
4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 6
4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 6
4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 7
4.4. Selection of Associated Media Protocol . . . . . . . . . . 6 4.4. Selection of Associated Media Protocol . . . . . . . . . . 7
4.5. Number of Associated Media Streams per SIP Session . . . . 6 4.5. Number of Associated Media Streams per SIP Session . . . . 7
4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 7
4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 7
4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6 4.8. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 7
4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 10
5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 10
5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9
5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10
5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10
6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.1. Baseline Session Establishment Rate of the test bed . . . 10 6.1. Baseline Session Establishment Rate of the test bed . . . 11
6.2. Session Establishment Rate without media . . . . . . . . . 11 6.2. Session Establishment Rate without media . . . . . . . . . 11
6.3. Session Establishment Rate with Media not on DUT/SUT . . . 11 6.3. Session Establishment Rate with Media not on DUT . . . . . 11
6.4. Session Establishment Rate with Media on DUT/SUT . . . . . 12 6.4. Session Establishment Rate with Media on DUT . . . . . . . 12
6.5. Session Establishment Rate with Loop Detection Enabled . . 13 6.5. Session Establishment Rate with TLS Encrypted SIP . . . . 12
6.6. Session Establishment Rate with Forking . . . . . . . . . 13 6.6. Session Establishment Rate with IPsec Encrypted SIP . . . 13
6.7. Session Establishment Rate with Forking and Loop 6.7. Registration Rate . . . . . . . . . . . . . . . . . . . . 13
Detection . . . . . . . . . . . . . . . . . . . . . . . . 14 6.8. Re-Registration Rate . . . . . . . . . . . . . . . . . . . 14
6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14
6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15 8. Security Considerations . . . . . . . . . . . . . . . . . . . 14
6.10. Session Establishment Rate with SIP Flooding . . . . . . . 16 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 14
6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16 10.1. Normative References . . . . . . . . . . . . . . . . . . . 15
6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17 10.2. Informative References . . . . . . . . . . . . . . . . . . 15
6.14. Session Capacity without Media . . . . . . . . . . . . . . 17 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15
6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18
6.16. Session Capacity with Media and a Media Relay/NAT
and/or Firewall . . . . . . . . . . . . . . . . . . . . . 19
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.1. Normative References . . . . . . . . . . . . . . . . . . . 20
10.2. Informative References . . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20
1. Terminology 1. Terminology
In this document, the key words "MUST", "MUST NOT", "REQUIRED", In this document, the key words "MUST", "MUST NOT", "REQUIRED",
"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as
described in BCP 14, conforming to [RFC2119] and indicate requirement described in BCP 14, conforming to [RFC2119] and indicate requirement
levels for compliant implementations. levels for compliant implementations.
Terms specific to SIP [RFC3261] performance benchmarking are defined
in [I-D.sip-bench-term].
RFC 2119 defines the use of these key words to help make the intent RFC 2119 defines the use of these key words to help make the intent
of standards track documents as clear as possible. While this of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track document uses these keywords, this document is not a standards track
document. The term Throughput is defined in [RFC2544]. document. The term Throughput is defined in [RFC2544].
Terms specific to SIP [RFC3261] performance benchmarking are defined
in [I-D.sip-bench-term].
2. Introduction 2. Introduction
This document describes the methodology for benchmarking Session This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in Terminology Initiation Protocol (SIP) performance as described in Terminology
document [I-D.sip-bench-term]. The methodology and terminology are document [I-D.sip-bench-term]. The methodology and terminology are
to be used for benchmarking signaling plane performance with varying to be used for benchmarking signaling plane performance with varying
signaling and media load. Both scale and establishment rate are signaling and media load. Media streams, when used, are used only to
measured by signaling plane performance. study how they impact the signaling behavior. This document
concentrates on benchmarking SIP session setup and SIP registrations
only.
The SIP Devices to be benchmarked may be a single device under test The device-under-test (DUT) is a SIP server, which may be any SIP
(DUT) or a system under test (SUT). The DUT is a SIP Server, which conforming [RFC3261] device. Benchmarks can be obtained and compared
may be any [RFC3261] conforming device. The SUT can be any device or for different types of devices such as SIP Proxy Server, Session
group of devices containing RFC 3261 conforming functionality along Border Controllers (SBC), SIP registrars and SIP proxy server paired
with Firewall and/or NAT functionality. This enables benchmarks to with a media relay.
be obtained and compared for different types of devices such as SIP
Proxy Server, SBC, SIP proxy server paired with a media relay or
Firewall/NAT device. SIP Associated Media benchmarks can also be
made when testing SUTs.
The test cases provide benchmarks metrics of Registration Rate, SIP The test cases provide metrics for benchmarking the maximum 'SIP
Session Establishment Rate, Session Capacity, and IM Rate. These can Registration Rate' and maximum 'SIP Session Establishment Rate' that
be benchmarked with or without associated Media. Some cases are also the DUT can sustain over an extended period of time without failures.
included to cover Forking, Loop detection, Encrypted SIP, and SIP Some cases are included to cover Encrypted SIP. The test topologies
Flooding. The test topologies that can be used are described in the that can be used are described in the Test Setup section. Topologies
Test Setup section. Topologies are provided for benchmarking of a in which the DUT handles media as well as those in which the DUT does
DUT or SUT. Benchmarking with Associated Media can be performed when not handle media are both considered. The measurement of the
using a SUT. performance characteristics of the media itself is outside the scope
of these documents.
SIP permits a wide range of configuration options that are explained
in Section 4 and Section 2 of [I-D.sip-bench-term]. Benchmark values
could possibly be impacted by Associated Media. The selected values
for Session Duration and Media Streams per Session enable benchmark
metrics to be evaluated without Associated Media. Session
Establishment Rate could possibly be impacted by the selected value
for Maximum Sessions Attempted. The benchmark for Session
Establishment Rate is measured with a fixed value for maximum Session
Attempts.
SIP permits a wide range of configuration options that are explained SIP permits a wide range of configuration options that are explained
in Section 4 and Section 2 of [I-D.sip-bench-term]. Benchmark in Section 4 and Section 2 of [I-D.sip-bench-term]. Benchmark
metrics could possibly be impacted by Associated Media. The selected metrics could possibly be impacted by Associated Media. The selected
values for Session Duration and Media Streams per Session enable values for Session Duration and Media Streams per Session enable
benchmark metrics to be benchmarked without Associated Media. benchmark metrics to be benchmarked without Associated Media.
Session Setup Rate could possibly be impacted by the selected value Session Setup Rate could possibly be impacted by the selected value
for Maximum Sessions Attempted. The benchmark for Session for Maximum Sessions Attempted. The benchmark for Session
Establishment Rate is measured with a fixed value for maximum Session Establishment Rate is measured with a fixed value for maximum Session
Attempts. Attempts.
skipping to change at page 5, line 21 skipping to change at page 5, line 30
Finally, the overall value of these tests is to serve as a comparison Finally, the overall value of these tests is to serve as a comparison
function between multiple SIP implementations. One way to use these function between multiple SIP implementations. One way to use these
tests is to derive benchmarks with SIP devices from Vendor-A, derive tests is to derive benchmarks with SIP devices from Vendor-A, derive
a new set of benchmarks with similar SIP devices from Vendor-B and a new set of benchmarks with similar SIP devices from Vendor-B and
perform a comparison on the results of Vendor-A and Vendor-B. This perform a comparison on the results of Vendor-A and Vendor-B. This
document does not make any claims on the interpretation of such document does not make any claims on the interpretation of such
results. results.
3. Benchmarking Topologies 3. Benchmarking Topologies
Familiarity with the benchmarking models in Section 2.2 of There are two test topologies; one in which the DUT does not process
[I-D.sip-bench-term] is assumed. Figures 1 through 10 in the media (Figure 1) and the other in which it does process media
[I-D.sip-bench-term] contain the canonical topologies that can be (Figure 2). In both cases, the tester or EA sends traffic into the
used to perform the benchmarking tests listed in this document. DUT and absorbs traffic from the DUT. The diagrams in Figure 1 and
Figure 2 represent the logical flow of information and do not dictate
a particular physical arrangements of the entities.
Test organizations need to be aware that these tests generate large
volumes of data and consequently ensure that networking devices like
hubs, switches or routers are able to handle the generated volume.
Figure 1 depicts a layout in which the DUT as an intermediary between
the two interfaces of the EA. If the test case requires the exchange
of media, the media does not flow through the DUT but rather passes
directly between the two endpoints. Figure 2 shows the DUT as an
intermediary between the two interfaces of the EA. If the test case
requires the exchange of media, the media flows through the DUT
between the endpoints.
+--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |------------>+ |------------>+ |
| | | | | |
| | Response | | Response | |
| Tester +<------------| DUT +<------------| Tester |
| (EA) | | | | (EA) |
| | | | | |
+--------+ +--------+ +--------+
/|\ /|\
| Media (optional) |
+==============================================+
Figure 1: DUT as an intermediary, end-to-end media
+--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |------------>+ |------------>+ |
| | | | | |
| | Response | | Response | |
| Tester +<------------| DUT +<------------| Tester |
| (EA) | | | | (EA) |
| |<===========>| |<===========>| |
+--------+ Media +--------+ Media +--------+
(Optional) (Optional)
Figure 2: DUT as an intermediary forwarding media
4. Test Setup Parameters 4. Test Setup Parameters
4.1. Selection of SIP Transport Protocol 4.1. Selection of SIP Transport Protocol
Test cases may be performed with any transport protocol supported by Test cases may be performed with any transport protocol supported by
SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and SIP. This includes, but is not limited to, TCP, UDP, TLS and
TLS. The protocol used for the SIP transport protocol must be websockets. The protocol used for the SIP transport protocol must be
reported with benchmarking results. reported with benchmarking results.
4.2. Signaling Server 4.2. Signaling Server
The Signaling Server is defined in the companion terminology The Signaling Server is defined in the companion terminology
document, ([I-D.sip-bench-term], Section 3.2.2) It is a SIP-speaking document, ([I-D.sip-bench-term], Section 3.2.2). The Signaling
device that complies with RFC 3261. Conformance to [RFC3261] is Server is a DUT.
assumed for all tests. The Signaling Server may be the DUT or a
component of a SUT. The Signaling Server may include Firewall and/or
NAT functionality. The components of the SUT may be a single
physical device or separate devices.
4.3. Associated Media 4.3. Associated Media
Some tests require Associated Media to be present for each SIP Some tests require Associated Media to be present for each SIP
session. The test topologies to be used when benchmarking SUT session. The test topologies to be used when benchmarking DUT
performance for Associated Media are shown in [I-D.sip-bench-term], performance for Associated Media are shown in Figure 1 and Figure 2.
Figures 4 and 5.
4.4. Selection of Associated Media Protocol 4.4. Selection of Associated Media Protocol
The test cases specified in this document provide SIP performance The test cases specified in this document provide SIP performance
independent of the protocol used for the media stream. Any media independent of the protocol used for the media stream. Any media
protocol supported by SIP may be used. This includes, but is not protocol supported by SIP may be used. This includes, but is not
limited to, RTP, RTSP, and SRTP. The protocol used for Associated limited to, RTP, RTSP, and SRTP. The protocol used for Associated
Media MUST be reported with benchmarking results. Media MUST be reported with benchmarking results.
4.5. Number of Associated Media Streams per SIP Session 4.5. Number of Associated Media Streams per SIP Session
Benchmarking results may vary with the number of media streams per Benchmarking results may vary with the number of media streams per
SIP session. When benchmarking a SUT for voice, a single media SIP session. When benchmarking a DUT for voice, a single media
stream is used. When benchmarking a SUT for voice and video, two stream is used. When benchmarking a DUT for voice and video, two
media streams are used. The number of Associated Media Streams MUST media streams are used. The number of Associated Media Streams MUST
be reported with benchmarking results. be reported with benchmarking results.
4.6. Session Duration 4.6. Session Duration
SUT performance benchmarks may vary with the duration of SIP The value of the DUT's performance benchmarks may vary with the
sessions. Session Duration MUST be reported with benchmarking duration of SIP sessions. Session Duration MUST be reported with
results. A Session Duration of zero seconds indicates transmission benchmarking results. A Session Duration of zero seconds indicates
of a BYE immediately following successful SIP establishment indicate transmission of a BYE immediately following successful SIP
by receipt of a 200 OK. An infinite Session Duration indicates that establishment indicate by receipt of a 200 OK. An infinite Session
a BYE is never transmitted. Duration indicates that a BYE is never transmitted.
4.7. Attempted Sessions per Second 4.7. Attempted Sessions per Second
DUT and SUT performance benchmarks may vary with the the rate of The value of the DUT's performance benchmarks may vary with the
attempted sessions offered by the Tester. Attempted Sessions per Session Attempt Rate offered by the tester. Session Attempt Rate
Second MUST be reported with benchmarking results. MUST be reported with the benchmarking results.
4.8. Stress Testing
The purpose of this document is to benchmark SIP performance; this
document does not benchmark stability of SIP systems under stressful
conditions such as a high rate of Attempted Sessions per Second.
4.9. Benchmarking algorithm 4.8. Benchmarking algorithm
In order to benchmark the test cases uniformly in Section 6, the In order to benchmark the test cases uniformly in Section 6, the
algorithm described in this section should be used. Both, a prosaic algorithm described in this section should be used. Both, a prosaic
description of the algorithm and a pseudo-code description are description of the algorithm and a pseudo-code description are
provided. provided.
The goal is to find the largest value of a SIP session-request-rate, The goal is to find the largest value, R, of a SIP Session Attempt
measured in sessions-per-second, which the DUT/SUT can process with Rate, measured in sessions-per-second, which the DUT can process with
zero errors. To discover that number, an iterative process (defined zero errors over a defined, extended period. This period is defined
below) is used to find a candidate for this rate. Once the candidate as the amount of time needed to attempt N SIP sessions, where N is a
rate has been found, the DUT/SUT is subjected to an offered load parameter of test, at the attempt rate, R. An iterative process is
whose arrival rate is set to that of the candidate rate. This test used to find this rate. The iterative process is divided into two
is run for an extended period of time, which is referred to as distinct phases: Candidate Identification and Steady State Testing.
infinity, and which is, itself, a parameter of the test labeled T in
the pseudo-code. This latter phase of testing is called the steady-
state phase. If errors are encountered during this steady-state
phase, then the candidate rate is reduced by a defined percent, also
a parameter of test, and the steady-state phase is entered again
until a final (new) steady-state rate is achieved.
The iterative process itself is defined as follows: a starting rate
of 100 sessions per second (sps) is selected. The test is executed
for the time period identified by t in the pseudo-code below. If no
failures occur, the rate is increased to 150 sps and again tested for
time period t. The attempt rate is continuously ramped up until a
failure is encountered before the end of the test time t. Then an
attempt rate is calculated that is higher than the last successful
attempt rate by a quantity equal to half the difference between the
rate at which failures occurred and the last successful rate. If
this new attempt rate also results in errors, a new attempt rate is
tried that is higher than the last successful attempt rate by a
quantity equal to half the difference between the rate at which
failures occurred and the last successful rate. Continuing in this
way, an attempt rate without errors is found. The operator can
specify margin of error using the parameter G, measured in units of
sessions per second.
The pseudo-code corresponding to the description above follows.
; ---- Parameters of test, adjust as needed
t := 5000 ; local maximum; used to figure out largest
; value
T := 50000 ; global maximum; once largest value has been
; figured out, pump this many requests before calling
; the test a success
m := {...} ; other attributes that affect testing, such
; as media streams, etc.
s := 100 ; Initial session attempt rate (in sessions/sec)
G := 5 ; granularity of results - the margin of error in sps
C := 0.05 ; calibration amount: How much to back down if we
; have found candidate s but cannot send at rate s for
; time T without failures
; ---- End of parameters of test
; ---- Initialization of flags, candidate values and upper bounds
f := false ; indicates that you had a success after the upper limit
F := false ; indicates that test is done
c := 0 ; indicates that we have found an upper limit
proc main During the Candidate Identification phase, the test runs until n
find_largest_value ; First, figure out the largest value. sessions have been attempted, at session attempt rates, r, which vary
according to the algorithm below, where n is also a parameter of test
and is a relatively large number, but an order of magnitude smaller
than N. If no errors occur during the time it takes to attempt n
sessions, we increment r according to the algorithm. If errors are
encountered during the test, we decrement r according to the
algorithm. The algorithm provides a variable, G, that allows us to
control how the accuracy, in sessions per second, that we require of
the test.
; Now that the largest value (saved in s) has been figured out, After this candidate rate has been discovered, the test enters the
; use it for sending out s requests/s and send out T requests. Steady State phase. In the Steady State phase, N session Attempts
are made at the candidate rate. The goal is to find a rate at which
the DUT can process calls "forever" with no errors and the test
organization can choose N as large as it deems appropriate. If
errors are encountered during this steady-state phase, the candidate
rate is reduced by a defined percent, also a parameter of test, and
the steady-state phase is entered again until a final (new) steady-
state rate is achieved.
do { The iterative process itself is defined as follows: A starting rate
send_traffic(s, m, T) ; send_traffic not shown of r = 100 sessions per second is used and we place calls at that
if (all requests succeeded) { rate until n = 5000 calls have been placed. If all n calls are
F := true ; test is done successful, the rate is increased to 150 sps and again we place calls
} else if (one or more requests fail) { at that rate until n = 5000 calls have been placed. The attempt rate
s := s - (C * s) ; Reduce s by calibration amount is continuously ramped up until a failure is encountered before n =
steady_state 5000 calls have been placed. Then an attempt rate is calculated that
} is higher than the last successful attempt rate by a quantity equal
} while (F == false) to half the difference between the rate at which failures occurred
end proc and the last successful rate. If this new attempt rate also results
in errors, a new attempt rate is tried that is higher than the last
successful attempt rate by a quantity equal to half the difference
between the rate at which failures occurred and the last successful
rate. Continuing in this way, an attempt rate without errors is
found. The tester can specify margin of error using the parameter G,
measured in units of sessions per second.
proc find_largest_value The pseudo-code corresponding to the description above follows.
; Iterative process to figure out the largest value we can
; handle with no failures
do {
send_traffic(s, m, t) ; Send s request/sec with m
; characteristics until t requests have
; been sent
if (all requests succeeded) {
s' := s ; save candidate value of metric
if ( c == 0 ) { ; ---- Parameters of test, adjust as needed
s := s + (0.5 * s) n := 5000 ; local maximum; used to figure out largest
; value (number of sessions attempted)
N := 50000 ; global maximum; once largest session rate has
; been established, send this many requests before
; calling the test a success
m := {...} ; other attributes that affect testing, such
; as media streams, etc.
r := 100 ; Initial session attempt rate (in sessions/sec)
G := 5 ; granularity of results - the margin of error in
; sps
C := 0.05 ; calibration amount: How much to back down if we
; have found candidate s but cannot send at rate s
; for time T without failures
}else if ((c == 1) && (s''-s')) > 2*G ) { ; ---- End of parameters of test
s := s + ( 0.5 * (s'' - s ); ; ---- Initialization of flags, candidate values and upper bounds
}else if ((c == 1) && ((s''-s') <= 2*G ) { f := false ; indicates a success after the upper limit
f := true; F := false ; indicates that test is done
c := 0 ; indicates that we have found an upper limit
} proc find_largest_value
else if (one or more requests fail) { ; Iterative process to figure out the largest value we can
c := 1 ; we have found an upper bound for the metric ; handle with no failures
s'' := s ; save new upper bound do {
s := s - (0.5 * (s - s')) send_traffic(r, m, n) ; Send r request/sec with m
} ; characteristics until n
} while (f == false) ; requests have been sent
end proc if (all requests succeeded) {
r' := r ; save candidate value of metric
if ( c == 0 ) {
r := r + (0.5 * r)
}
else if ((c == 1) && (r''-r')) > 2*G ) {
r := r + ( 0.5 * (r'' - r );
}
else if ((c == 1) && ((r''-r') <= 2*G ) {
f := true;
}
else if (one or more requests fail) {
c := 1 ; found upper bound for the metric
r'' := r ; save new upper bound
r := r - (0.5 * (r - r'))
}
} while (f == false)
end proc
5. Reporting Format 5. Reporting Format
5.1. Test Setup Report 5.1. Test Setup Report
SIP Transport Protocol = ___________________________ SIP Transport Protocol = ___________________________
(valid values: TCP|UDP|TLS|SCTP|specify-other) (valid values: TCP|UDP|TLS|SCTP|websockets|specify-other)
Session Attempt Rate = _____________________________ Session Attempt Rate = _____________________________
(session attempts/sec) (session attempts/sec)
IS Media Attempt Rate = ____________________________
(IS media attempts/sec)
Total Sessions Attempted = _________________________ Total Sessions Attempted = _________________________
(total sessions to be created over duration of test) (total sessions to be created over duration of test)
Media Streams Per Session = _______________________ Media Streams Per Session = _______________________
(number of streams per session) (number of streams per session)
Associated Media Protocol = _______________________ Associated Media Protocol = _______________________
(RTP|RTSP|specify-other) (RTP|RTSP|specify-other)
Media Packet Size = _______________________________ Media Packet Size = _______________________________
(bytes) (bytes)
Media Offered Load = ______________________________
(packets per second)
Media Session Hold Time = _________________________
(seconds)
Establishment Threshold time = ____________________ Establishment Threshold time = ____________________
(seconds) (seconds)
Loop Detecting Option = ___________________________ TLS ciphersuite used
(on|off) (for tests involving TLS) = ________________________
Forking Option (e.g., TLS_RSA_WITH_AES_128_CBC_SHA)
Number of endpoints request sent to = ___________ IPSec profile used
(1, means forking is not enabled) (for tests involving IPSEC) = _____________________
Type of forking = _______________________________
(serial|parallel)
Authentication option = ___________________________________
(on|off; if on, please see Notes 2 and 3 below).
Note 1: Total Sessions Attempted is used in the calculation of the
Session Establishment Performance ([I-D.sip-bench-term], Section
3.4.5). It is the number of session attempts ([I-D.sip-bench-term],
Section 3.1.6) that will be made over the duration of the test.
Note 2: When the Authentication Option is "on" the test tool must be
set to ignore 401 and 407 failure responses in any test described as
a "test to failure." If this is not done, all such tests will yield
trivial benchmarks, as all attempt rates will lead to a failure after
the first attempt.
Note 3: When the Authentication Option is "on" the DUT/SUT uses two
transactions instead of one when it is establishing a session or
accomplishing a registration. The first transaction ends with the
401 or 407. The second ends with the 200 OK or another failure
message. The Test Organization interested in knowing how many times
the EA was intended to send a REGISTER as distinct from how many
times the EA wound up actually sending a REGISTER may wish to record
the following data as well:
Number of responses of the following type:
401: _____________ (if authentication turned on; N/A
otherwise)
407: _____________ (if authentication turned on; N/A
otherwise)
5.2. Device Benchmarks for IS 5.2. Device Benchmarks for IS
Registration Rate = _______________________________
(registrations per second)
Re-registration Rate = ____________________________
(registrations per second)
Session Capacity = _________________________________
(sessions)
Session Overload Capacity = ________________________
(sessions)
Session Establishment Rate = ______________________ Session Establishment Rate = ______________________
(sessions per second) (sessions per second)
Session Establishment Performance = _______________ Is DUT acting as a media relay (yes/no) = _________
(total established sessions/total sessions attempted)(no units)
Session Attempt Delay = ___________________________
(seconds)
5.3. Device Benchmarks for NS 5.3. Device Benchmarks for NS
IM Rate = _______________________________ (IM messages per second) Registration Rate = ____________________________
(registrations per second)
Re-registration Rate = ____________________________
(registrations per second)
6. Test Cases 6. Test Cases
6.1. Baseline Session Establishment Rate of the test bed 6.1. Baseline Session Establishment Rate of the test bed
Objective: Objective:
To benchmark the Session Establishment Rate of the Emulated Agent To benchmark the Session Establishment Rate of the Emulated Agent
(EA) with zero failures. (EA) with zero failures.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 in 1. Configure the DUT in the test topology shown in Figure 1.
[I-D.sip-bench-term].
2. Set media streams per session to 0. 2. Set media streams per session to 0.
3. Execute benchmarking algorithm as defined in Section 4.9 to 3. Execute benchmarking algorithm as defined in Section 4.8 to
get the baseline session establishment rate. This rate MUST get the baseline session establishment rate. This rate MUST
be recorded using any pertinent parameters as shown in the be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1. reporting format of Section 5.1.
Expected Results: This is the scenario to obtain the maximum Session Expected Results: This is the scenario to obtain the maximum Session
Establishment Rate of the EA and the test bed when no DUT/SUT is Establishment Rate of the EA and the test bed when no DUT is
present. The results of this test might be used to normalize test present. The results of this test might be used to normalize test
results performed on different test beds or simply to better results performed on different test beds or simply to better
understand the impact of the DUT/SUT on the test bed in question. understand the impact of the DUT on the test bed in question.
6.2. Session Establishment Rate without media 6.2. Session Establishment Rate without media
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with no To benchmark the Session Establishment Rate of the DUT with no
associated media and zero failures. associated media and zero failures.
Procedure: Procedure:
1. If the DUT/SUT is being benchmarked as a user agent client or 1. Configure a DUT according to the test topology shown in
a user agent server, configure the DUT in the test topology Figure 1 or Figure 2.
shown in Figure 1 or Figure 2 in [I-D.sip-bench-term]. 2. Set media streams per session to 0.
Alternatively, if the DUT is being benchmarked as a proxy or a 3. Execute benchmarking algorithm as defined in Section 4.8 to
B2BUA, configure the DUT in the test topology shown in Figure
5 in [I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
7 in [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate. This rate MUST be get the session establishment rate. This rate MUST be
recorded using any pertinent parameters as shown in the recorded using any pertinent parameters as shown in the
reporting format of Section 5.1. reporting format of Section 5.1.
Expected Results: This is the scenario to obtain the maximum Session Expected Results: Find the Session Establishment Rate of the DUT
Establishment Rate of the DUT/SUT. when the EA is not sending media streams.
6.3. Session Establishment Rate with Media not on DUT
6.3. Session Establishment Rate with Media not on DUT/SUT
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT with zero
zero failures when Associated Media is included in the benchmark failures when Associated Media is included in the benchmark test
test but the media is not running through the DUT/SUT. but the media is not running through the DUT.
Procedure: Procedure:
1. If the DUT is being benchmarked as proxy or B2BUA, configure 1. Configure a DUT according to the test topology shown in
the DUT in the test topology shown in Figure 7 in Figure 1.
[I-D.sip-bench-term]. 2. Set media streams per session to 1.
2. Configure a SUT according to the test topology shown in Figure 3. Execute benchmarking algorithm as defined in Section 4.8 to
8 in [I-D.sip-bench-term].
3. Set media streams per session to 1.
4. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with media. This rate MUST get the session establishment rate with media. This rate MUST
be recorded using any pertinent parameters as shown in the be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1. reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
Associated Media with any number of media streams per SIP session Associated Media with any number of media streams per SIP session
are expected to be identical to the Session Establishment Rate are expected to be identical to the Session Establishment Rate
results obtained without media in the case where the server is results obtained without media in the case where the DUT is
running on a platform separate from the platform on which the running on a platform separate from the platform on which the
Media Relay, NAT or Firewall is running. Media Relay.
6.4. Session Establishment Rate with Media on DUT/SUT 6.4. Session Establishment Rate with Media on DUT
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT with zero
zero failures when Associated Media is included in the benchmark failures when Associated Media is included in the benchmark test
test and the media is running through the DUT/SUT. and the media is running through the DUT.
Procedure: Procedure:
1. If the DUT is being benchmarked as a user agent client or a 1. Configure a DUT according to the test topology shown in
user agent server, configure the DUT in the test topology Figure 2.
shown in Figure 3 or Figure 4 of [I-D.sip-bench-term]. 2. Set media streams per session to 1.
Alternatively, if the DUT is being benchmarked as a B2BUA, 3. Execute benchmarking algorithm as defined in Section 4.8 to
configure the DUT in the test topology shown in Figure 6 in
[I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
9 in [I-D.sip-bench-term].
3. Set media streams per session to 1.
4. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with media. This rate MUST get the session establishment rate with media. This rate MUST
be recorded using any pertinent parameters as shown in the be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1. reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
Associated Media may be lower than those obtained without media in Associated Media may be lower than those obtained without media in
the case where the server and the NAT, Firewall or Media Relay are the case where the DUT and the Media Relay are running on the same
running on the same platform. platform.
6.5. Session Establishment Rate with Loop Detection Enabled
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Loop Detection option is enabled and no
media streams are present.
Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, and loop
detection is supported in the DUT, then configure the DUT in
the test topology shown in Figure 5 in [I-D.sip-bench-term].
If the DUT does not support loop detection, then this step can
be skipped.
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Turn on the Loop Detection option in the DUT or SUT.
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with loop detection
enabled. This rate MUST be recorded using any pertinent
parameters as shown in the reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with
Loop Detection may be lower than those obtained without Loop
Detection enabled.
6.6. Session Establishment Rate with Forking
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Forking Option is enabled.
Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, and
forking is supported in the DUT, then configure the DUT in the
test topology shown in Figure 5 in [I-D.sip-bench-term]. If
the DUT does not support forking, then this step can be
skipped.
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Set the number of endpoints that will receive the forked
invitation to a value of 2 or more (subsequent tests may
increase this value at the discretion of the tester.)
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with forking. This rate
MUST be recorded using any pertinent parameters as shown in
the reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with
Forking may be lower than those obtained without Forking enabled.
6.7. Session Establishment Rate with Forking and Loop Detection
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when both the Forking and Loop Detection Options are
enabled.
Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, then
configure the DUT in the test topology shown in Figure 5 in
[I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Enable the Loop Detection Options on the DUT.
5. Set the number of endpoints that will receive the forked
invitation to a value of 2 or more (subsequent tests may
increase this value at the discretion of the tester.)
6. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with forking and loop
detection. This rate MUST be recorded using any pertinent
parameters as shown in the reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with
Forking and Loop Detection may be lower than those obtained with
only Forking or Loop Detection enabled.
6.8. Session Establishment Rate with TLS Encrypted SIP 6.5. Session Establishment Rate with TLS Encrypted SIP
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT with zero
zero failures when using TLS encrypted SIP signaling. failures when using TLS encrypted SIP signaling.
Procedure: Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, then 1. If the DUT is being benchmarked as a proxy or B2BUA, then
configure the DUT in the test topology shown in Figure 5 in configure the DUT in the test topology shown in Figure 1 or
[I-D.sip-bench-term]. Figure 2.
2. Configure a SUT according to the test topology shown in Figure 2. Configure the tester to enable TLS over the transport being
8 of [I-D.sip-bench-term]. used during benchmarking. Note the ciphersuite being used for
TLS and record it in Section 5.1.
3. Set media streams per session to 0 (media is not used in this 3. Set media streams per session to 0 (media is not used in this
test). test).
4. Configure Tester to enable TLS over the transport being 4. Execute benchmarking algorithm as defined in Section 4.8 to
benchmarked. Make a note the transport when compiling get the session establishment rate with TLS encryption.
results. May need to run for each transport of interest.
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with encryption. This rate
MUST be recorded using any pertinent parameters as shown in
the reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
TLS Encrypted SIP may be lower than those obtained with plaintext TLS Encrypted SIP may be lower than those obtained with plaintext
SIP. SIP.
6.9. Session Establishment Rate with IPsec Encrypted SIP 6.6. Session Establishment Rate with IPsec Encrypted SIP
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT with zero
zero failures when using IPsec Encrypted SIP signaling. failures when using IPsec Encrypted SIP signaling.
Procedure: Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, then 1. Configure a DUT according to the test topology shown in
configure the DUT in the test topology shown in Figure 5 in Figure 1 or Figure 2.
[I-D.sip-bench-term]. 2. Set media streams per session to 0 (media is not used in this
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0 (media is not used in this
test). test).
4. Configure Tester for IPSec. 3. Configure tester for IPSec. Note the IPSec profile being used
5. Execute benchmarking algorithm as defined in Section 4.9 to for and record it in Section 5.1.
get the session establishment rate with encryption. This rate 4. Execute benchmarking algorithm as defined in Section 4.8 to
MUST be recorded using any pertinent parameters as shown in get the session establishment rate with encryption.
the reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
IPSec Encrypted SIP may be lower than those obtained with IPSec Encrypted SIP may be lower than those obtained with
plaintext SIP. plaintext SIP.
6.10. Session Establishment Rate with SIP Flooding 6.7. Registration Rate
Objective: Objective:
To benchmark the Session Establishment Rate of the SUT with zero To benchmark the maximum registration rate the DUT can handle over
failures when SIP Flooding is occurring. an extended time period with zero failures.
Procedure: Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, then 1. Configure a DUT according to the test topology shown in
configure the DUT in the test topology shown in Figure 5 in Figure 1 or Figure 2.
[I-D.sip-bench-term]. 2. Set the registration timeout value to at least 3600 seconds.
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Set s to a high value (e.g., 500) (c.f. Section 4.9).
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with flooding. This rate
MUST be recorded using any pertinent parameters as shown in
the reporting format of Section 5.1.
Expected Results: Session Establishment Rate results obtained with
SIP Flooding may be degraded.
6.11. Maximum Registration Rate
Objective:
To benchmark the maximum registration rate of the DUT/SUT with
zero failures.
Procedure: 3. Execute benchmarking algorithm as defined in Section 4.8 to
1. If the DUT is being benchmarked as a proxy or B2BUA, then
configure the DUT in the test topology shown in Figure 5 in
[I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Set the registration timeout value to at least 3600 seconds.
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the maximum registration rate. This rate MUST be recorded get the maximum registration rate. This rate MUST be recorded
using any pertinent parameters as shown in the reporting using any pertinent parameters as shown in the reporting
format of Section 5.1. format of Section 5.1. For example, the use of TLS or IPSec
Expected Results: during registration must be noted in the reporting format.
6.12. Maximum Re-Registration Rate
Objective:
To benchmark the maximum re-registration rate of the DUT/SUT with
zero failures.
Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, then
configure the DUT in the test topology shown in Figure 5 in
[I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. First, execute test detailed in Section 6.11 to register the
endpoints with the registrar.
4. After at least 5 minutes of Step 2, but no more than 10
minutes after Step 2 has been performed, execute test detailed
in Section 6.11 again (this will count as a re-registration).
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the maximum re-registration rate. This rate MUST be
recorded using any pertinent parameters as shown in the
reporting format of Section 5.1.
Expected Results: The rate should be at least equal to but not more
than the result of Section 6.11.
6.13. Maximum IM Rate
Objective:
To benchmark the maximum IM rate of the SUT with zero failures.
Procedure:
1. If the DUT/SUT is being benchmarked as a user agent client or
a user agent server, configure the DUT in the test topology
shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
Alternatively, if the DUT is being benchmarked as a proxy or a
B2BUA, configure the DUT in the test topology shown in Figure
5 in [I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
5 in [I-D.sip-bench-term].
3. Execute benchmarking algorithm as defined in Section 4.9 to
get the maximum IM rate. This rate MUST be recorded using any
pertinent parameters as shown in the reporting format of
Section 5.1.
Expected Results:
6.14. Session Capacity without Media Expected Results: Provides a maximum registration rate.
Objective:
To benchmark the Session Capacity of the SUT without Associated
Media.
Procedure:
1. If the DUT/SUT is being benchmarked as a user agent client or
a user agent server, configure the DUT in the test topology
shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
Alternatively, if the DUT is being benchmarked as a proxy or a
B2BUA, configure the DUT in the test topology shown in Figure
5 in [I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
7 in [I-D.sip-bench-term].
3. Set the media treams per session to be 0.
4. Set the Session Duration to be a value greater than T.
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the baseline session establishment rate. This rate MUST
be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1.
6. The Session Capacity is the product of T and the Session
Establishment Rate.
Expected Results: The maximum rate at which the DUT/SUT can handle
session establishment requests with no media for an infinitely
long period with no errors. This is the SIP "throughput" of the
system with no media.
6.15. Session Capacity with Media 6.8. Re-Registration Rate
Objective: Objective:
To benchmark the session capacity of the DUT/SUT with Associated To benchmark the re-registration rate of the DUT with zero
Media. failures.
Procedure:
1. Configure the DUT in the test topology shown in Figure 3 or
Figure 4 of [I-D.sip-bench-term] depending on whether the DUT
is being benchmarked as a user agent client or user agent
server. Alternatively, configure the DUT in the test topology
shown in Figure 6 or Figure 7 in [I-D.sip-bench-term]
depending on whether the DUT is being benchmarked as a B2BUA
or as a proxy. If a SUT is being benchmarked, configure the
SUT as shown in Figure 9 of [I-D.sip-bench-term].
2. Set the media streams per session to 1.
3. Set the Session Duration to be a value greater than T.
4. Execute benchmarking algorithm as defined in Section 4.9 to
get the baseline session establishment rate. This rate MUST
be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1.
5. The Session Capacity is the product of T and the Session
Establishment Rate.
Expected Results: Session Capacity results obtained with Associated
Media with any number of media streams per SIP session will be
less than the Session Capacity results obtained without media.
6.16. Session Capacity with Media and a Media Relay/NAT and/or Firewall
Objective:
To benchmark the Session Establishment Rate of the SUT with
Associated Media.
Procedure: Procedure:
1. Configure the SUT as shown in Figure 7 or Figure 10 in 1. Configure a DUT according to the test topology shown in
[I-D.sip-bench-term]. Figure 1 or Figure 2.
2. Set media streams per session to 1. 2. First, execute test detailed in Section 6.7 to register the
3. Execute benchmarking algorithm as defined in Section 4.9 to endpoints with the registrar and obtain the registration rate.
get the session establishment rate with media. This rate MUST 3. After at least 5 minutes of Step 2, but no more than 10
be recorded using any pertinent parameters as shown in the minutes after Step 2 has been performed, execute Step 3 of the
reporting format of Section 5.1. test in Section 6.7. This will count as a re-registration
because the SIP address of records have not yet expired.
Expected Results: Session Capacity results obtained with Associated Expected Results: The rate should be at least equal to but not more
Media with any number of media streams per SIP session may be than the result of Section 6.7.
lower than the Session Capacity without Media result if the Media
Relay, NAT or Firewall is sharing a platform with the server.
7. IANA Considerations 7. IANA Considerations
This document does not requires any IANA considerations. This document does not requires any IANA considerations.
8. Security Considerations 8. Security Considerations
Documents of this type do not directly affect the security of Documents of this type do not directly affect the security of
Internet or corporate networks as long as benchmarking is not Internet or corporate networks as long as benchmarking is not
performed on devices or systems connected to production networks. performed on devices or systems connected to production networks.
skipping to change at page 20, line 18 skipping to change at page 15, line 22
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[I-D.sip-bench-term] [I-D.sip-bench-term]
Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance
Benchmarking Terminology", Benchmarking Terminology",
draft-ietf-bmwg-sip-bench-term-08 (work in progress), draft-ietf-bmwg-sip-bench-term-04 (work in progress),
January 2013. March 2012.
10.2. Informative References 10.2. Informative References
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
Authors' Addresses Authors' Addresses
 End of changes. 76 change blocks. 
557 lines changed or deleted 324 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/