draft-ietf-bmwg-sip-bench-meth-03.txt   draft-ietf-bmwg-sip-bench-meth-04.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working C. Davids
Group Illinois Institute of Technology Group Illinois Institute of Technology
Internet-Draft V. Gurbani Internet-Draft V. Gurbani
Expires: September 15, 2011 Bell Laboratories, Alcatel-Lucent Expires: September 13, 2012 Bell Laboratories, Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
March 14, 2011 March 12, 2012
Methodology for Benchmarking SIP Networking Devices Methodology for Benchmarking SIP Networking Devices
draft-ietf-bmwg-sip-bench-meth-03 draft-ietf-bmwg-sip-bench-meth-04
Abstract Abstract
This document describes the methodology for benchmarking Session This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document. The methodology and terminology benchmarking terminology document. The methodology and terminology
are to be used for benchmarking signaling plane performance with are to be used for benchmarking signaling plane performance with
varying signaling and media load. Both scale and establishment rate varying signaling and media load. Both scale and establishment rate
are measured by signaling plane performance. The SIP Devices to be are measured by signaling plane performance. The SIP Devices to be
benchmarked may be a single device under test (DUT) or a system under benchmarked may be a single device under test (DUT) or a system under
skipping to change at page 1, line 42 skipping to change at page 1, line 42
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 15, 2011. This Internet-Draft will expire on September 13, 2012.
Copyright Notice Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 3, line 19 skipping to change at page 3, line 19
3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5
4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5
4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5
4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5
4.4. Selection of Associated Media Protocol . . . . . . . . . . 6 4.4. Selection of Associated Media Protocol . . . . . . . . . . 6
4.5. Number of Associated Media Streams per SIP Session . . . . 6 4.5. Number of Associated Media Streams per SIP Session . . . . 6
4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6
4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6
4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6
5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 6 4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6
5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 6 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9
5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 7 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9
5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 8 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 9
6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10
6.1. Baseline Session Establishment Rate . . . . . . . . . . . 8 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.2. Session Establishment Rate . . . . . . . . . . . . . . . . 9 6.1. Baseline Session Establishment Rate of the test bed . . . 10
6.3. Session Establishment Rate with Media . . . . . . . . . . 9 6.2. Session Establishment Rate without media . . . . . . . . . 11
6.4. Session Establishment Rate with Loop Detection Enabled . . 10 6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11
6.5. Session Establishment Rate with Forking . . . . . . . . . 11 6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12
6.6. Session Establishment Rate with Forking and Loop 6.5. Session Establishment Rate with Loop Detection Enabled . . 12
Detection . . . . . . . . . . . . . . . . . . . . . . . . 11 6.6. Session Establishment Rate with Forking . . . . . . . . . 13
6.7. Session Establishment Rate with TLS Encrypted SIP . . . . 12 6.7. Session Establishment Rate with Forking and Loop
6.8. Session Establishment Rate with IPsec Encrypted SIP . . . 13 Detection . . . . . . . . . . . . . . . . . . . . . . . . 14
6.9. Session Establishment Rate with SIP Flooding . . . . . . . 13 6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14
6.10. Maximum Registration Rate . . . . . . . . . . . . . . . . 14 6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15
6.11. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 14 6.10. Session Establishment Rate with SIP Flooding . . . . . . . 15
6.12. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 15 6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16
6.13. Session Capacity without Media . . . . . . . . . . . . . . 16 6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16
6.14. Session Capacity with Media . . . . . . . . . . . . . . . 16 6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17
6.15. Session Capacity with Media and a Media Relay/NAT 6.14. Session Capacity without Media . . . . . . . . . . . . . . 17
and/or Firewall . . . . . . . . . . . . . . . . . . . . . 17 6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 6.16. Session Capacity with Media and a Media Relay/NAT
8. Security Considerations . . . . . . . . . . . . . . . . . . . 18 and/or Firewall . . . . . . . . . . . . . . . . . . . . . 18
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 18 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
10.1. Normative References . . . . . . . . . . . . . . . . . . . 18 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19
10.2. Informative References . . . . . . . . . . . . . . . . . . 19 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 19 10.1. Normative References . . . . . . . . . . . . . . . . . . . 19
10.2. Informative References . . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20
1. Terminology 1. Terminology
In this document, the key words "MUST", "MUST NOT", "REQUIRED", In this document, the key words "MUST", "MUST NOT", "REQUIRED",
"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as
described in BCP 14, conforming to [RFC2119] and indicate requirement described in BCP 14, conforming to [RFC2119] and indicate requirement
levels for compliant implementations. levels for compliant implementations.
Terms specific to SIP [RFC3261] performance benchmarking are defined Terms specific to SIP [RFC3261] performance benchmarking are defined
skipping to change at page 5, line 11 skipping to change at page 5, line 11
SIP permits a wide range of configuration options that are also SIP permits a wide range of configuration options that are also
explained in the Test Setup section. Benchmark metrics could explained in the Test Setup section. Benchmark metrics could
possibly be impacted by Associated Media. The selected values for possibly be impacted by Associated Media. The selected values for
Session Duration and Media Streams Per Session enable benchmark Session Duration and Media Streams Per Session enable benchmark
metrics to be benchmarked without Associated Media. Session Setup metrics to be benchmarked without Associated Media. Session Setup
Rate could possibly be impacted by the selected value for Maximum Rate could possibly be impacted by the selected value for Maximum
Sessions Attempted. The benchmark for Session Establishment Rate is Sessions Attempted. The benchmark for Session Establishment Rate is
measured with a fixed value for maximum Session Attempts. measured with a fixed value for maximum Session Attempts.
Finally, the overall value of these tests is to serve as a comparison
function between multiple SIP implementations. One way to use these
tests is to derive benchmarks with SIP devices from Vendor-A, derive
a new set of benchmarks with similar SIP devices from Vendor-B and
perform a comparison on the results of Vendor-A and Vendor-B. This
document does not make any claims on the interpretation of such
results.
3. Benchmarking Topologies 3. Benchmarking Topologies
Familiarity with the benchmarking models in Section 2.2 of Familiarity with the benchmarking models in Section 2.2 of
[I-D.sip-bench-term] is assumed. Figures 1 through 9 in [I-D.sip-bench-term] is assumed. Figures 1 through 10 in
[I-D.sip-bench-term] contain the canonical topologies that can be [I-D.sip-bench-term] contain the canonical topologies that can be
used to perform the benchmarking tests listed in this document. used to perform the benchmarking tests listed in this document.
4. Benchmarking Considerations 4. Benchmarking Considerations
4.1. Selection of SIP Transport Protocol 4.1. Selection of SIP Transport Protocol
Test cases may be performed with any transport protocol supported Test cases may be performed with any transport protocol supported by
by SIP. This includes, but is not limited to, SIP TCP, SIP UDP, SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and
and TLS. The protocol used for the SIP transport protocol must be TLS. The protocol used for the SIP transport protocol must be
reported with benchmarking results. reported with benchmarking results.
4.2. Signaling Server 4.2. Signaling Server
The Server is a SIP-speaking device that complies with RFC 3261. The Server is a SIP-speaking device that complies with RFC 3261. The
The purpose of this document is to benchmark SIP performance, not purpose of this document is to benchmark SIP performance, not
conformance. Conformance to [RFC3261] is assumed for all tests. conformance. Conformance to [RFC3261] is assumed for all tests. The
The Server may be the DUT or a component of a SUT that includes Server may be the DUT or a component of a SUT that includes Firewall
Firewall and/or NAT functionality. The components of the SUT may and/or NAT functionality. The components of the SUT may be a single
be a single physical device or separate devices. physical device or separate devices.
4.3. Associated Media 4.3. Associated Media
Some tests may require associated media to be present for each SIP Some tests may require associated media to be present for each SIP
session. The Server is not involved in the forwarding of media. session. The Server is not involved in the forwarding of media.
Associated Media can be benchmarked only with a SUT in which the Associated Media can be benchmarked only with a SUT in which the
media traverses a Media Relay, Firewall, NAT, or Firewall/NAT media traverses a Media Relay, Firewall, NAT, or Firewall/NAT device.
device. The test topologies to be used when benchmarking SUT
performance for Associated Media are shown in Figures 4 and 5, in The test topologies to be used when benchmarking SUT performance for
which the SIP signaling is bidirectional and the Associated Media Associated Media are shown in Figures 4 and 5, in which the SIP
is unidirectional. signaling is bidirectional and the Associated Media is
unidirectional.
4.4. Selection of Associated Media Protocol 4.4. Selection of Associated Media Protocol
The test cases specified in this document provide SIP performance The test cases specified in this document provide SIP performance
independent of the protocol used for the media stream. Any media independent of the protocol used for the media stream. Any media
protocol supported by SIP may be used. This includes, but is not protocol supported by SIP may be used. This includes, but is not
limited to, RTP, RTSP, and SRTP. The protocol used for Associated limited to, RTP, RTSP, and SRTP. The protocol used for Associated
Media MUST be reported with benchmarking results. Media MUST be reported with benchmarking results.
4.5. Number of Associated Media Streams per SIP Session 4.5. Number of Associated Media Streams per SIP Session
Benchmarking results may vary with the number of media streams per Benchmarking results may vary with the number of media streams per
SIP session. When benchmarking a SUT for voice, a single media SIP session. When benchmarking a SUT for voice, a single media
stream is used. When benchmarking a SUT for voice and video, two stream is used. When benchmarking a SUT for voice and video, two
media streams are used. The number of Associated Media Streams media streams are used. The number of Associated Media Streams MUST
MUST be reported with benchmarking results. be reported with benchmarking results.
4.6. Session Duration 4.6. Session Duration
SUT performance benchmarks may vary with the duration of SIP SUT performance benchmarks may vary with the duration of SIP
sessions. Session Duration MUST be reported with benchmarking sessions. Session Duration MUST be reported with benchmarking
results. A Session Duration of zero seconds indicates results. A Session Duration of zero seconds indicates transmission
transmission of a BYE immediately following successful SIP of a BYE immediately following successful SIP establishment indicate
establishment indicate by receipt of a 200 OK. An infinite by receipt of a 200 OK. An infinite Session Duration indicates that
Session Duration indicates that a BYE is never transmitted. a BYE is never transmitted.
4.7. Attempted Sessions per Second 4.7. Attempted Sessions per Second
DUT and SUT performance benchmarks may vary with the the rate of DUT and SUT performance benchmarks may vary with the the rate of
attempted sessions offered by the Tester. Attempted Sessions per attempted sessions offered by the Tester. Attempted Sessions per
Second MUST be reported with benchmarking results. Second MUST be reported with benchmarking results.
4.8. Stress Testing 4.8. Stress Testing
Discussion: The purpose of this document is to benchmark SIP performance; this
The purpose of this document is to benchmark SIP performance, not document does not benchmark stability of SIP systems under stressful
system stability under stressful conditions such as a high rate of conditions such as a high rate of Attempted Sessions per Second.
Attempted Sessions per Second.
4.9. Benchmarking algorithm
In order to benchmark the test cases uniformly in Section 6, the
algorithm described in this section should be used. Both, a prosaic
description of the algorithm and a pseudo-code description are
provided.
The goal is to find the largest value of a SIP session-request-rate,
measured in sessions-per-second, which the DUT/SUT can process with
zero errors. To discover that number, an iterative process (defined
below) is used to find a candidate for this rate. Once the candidate
rate has been found, the DUT/SUT is subjected to an offered load
whose arrival rate is set to that of the candidate rate. This test
is run for an extended period of time, which is referred to as
infinity, and which is, itself, a parameter of the test labeled T in
the pseudo-code). This latter phase of testing is called the steady-
state phase. If errors are encountered during this steady-state
phase, then the candidate rate is reduced by a defined percent, also
a parameter of test, and the steady-state phase is entered again
until a final (new) steady-state rate is achieved.
The iterative process itself is defined as follows: a starting rate
of 100 sessions per second (sps) is selected. The test is executed
for the time period identified by t in the pseudo-code below. If no
failures occur, the rate is increased to 150 sps and again tested for
time period t. The attempt rate is continuously ramped up until a
failure is encountered before the end of the test time t. Then an
attempt rate is calculated that is higher than the last successful
attempt rate by a quantity equal to half the difference between the
rate at which failures occurred and the last successful rate. If
this new attempt rate also results in errors, a new attempt rate is
tried that is higher than the last successful attempt rate by a
quantity equal to half the difference between the rate at which
failures occurred and the last successful rate. Continuing in this
way, an attempt rate without errors is found. The operator can
specify margin of error using the parameter G, measured in units of
sessions per second.
The pseudo-code corresponding to the description above follows.
; ---- Parameters of test, adjust as needed
t := 5000 ; local maximum; used to figure out largest
; value
T := 50000 ; global maximum; once largest value has been
; figured out, pump this many requests before calling
; the test a success
m := {...} ; other attributes that affect testing, such
; as media streams, etc.
s := 100 ; Initial session attempt rate (in sessions/sec)
G := 5 ; granularity of results - the margin of error in sps
C := 0.05 ; caliberation amount: How much to back down if we
; have found candidate s but cannot send at rate s for
; time T without failures
; ---- End of parameters of test
; ---- Initialization of flags, candidate values and upper bounds
f := false ; indicates that you had a success after the upper limit
F := false ; indicates that test is done
c := 0 ; indicates that we have found an upper limit
proc main
find_largest_value ; First, figure out the largest value.
; Now that the largest value (saved in s) has been figured out,
; use it for sending out s requests/s and send out T requests.
do {
send_traffic(s, m, T) ; send_traffic not shown
if (all requests succeeded) {
F := true ; test is done
} else if (one or more requests fail) {
s := s - (C * s) ; Reduce s by calibration amount
steady_state
}
} while (F == false)
end proc
proc find_largest_value
; Iterative process to figure out the largest value we can
; handle with no failures
do {
send_traffic(s, m, t) ; Send s request/sec with m
; characteristics until t requests have
; been sent
if (all requests succeeded) {
s' := s ; save candidate value of metric
if ( c == 0 ) {
s := s + (0.5 * s)
}else if ((c == 1) && (s''-s')) > 2*G ) {
s := s + ( 0.5 * (s'' - s );
}else if ((c == 1) && ((s''-s') <= 2*G ) {
f := true;
}
else if (one or more requests fail) {
c := 1 ; we have found an upper bound for the metric
s'' := s ; save new upper bound
s := s - (0.5 * (s - s'))
}
} while (f == false)
end proc
5. Reporting Format 5. Reporting Format
5.1. Test Setup Report 5.1. Test Setup Report
SIP Transport Protocol = ___________________________ SIP Transport Protocol = ___________________________
(valid values: TCP|UDP|TLS|SCTP|specify-other) (valid values: TCP|UDP|TLS|SCTP|specify-other)
Session Attempt Rate = _____________________________ Session Attempt Rate = _____________________________
(session attempts/sec) (session attempts/sec)
IS Media Attempt Rate = ____________________________ IS Media Attempt Rate = ____________________________
(IS media attempts/sec) (IS media attempts/sec)
Total Sessions Attempted = _________________________ Total Sessions Attempted = _________________________
(total sessions to be created over duration of test) (total sessions to be created over duration of test)
Media Streams Per Session = _______________________ Media Streams Per Session = _______________________
(number of streams per session) (number of streams per session)
skipping to change at page 8, line 25 skipping to change at page 10, line 25
(total established sessions/total sessions attempted)(no units) (total established sessions/total sessions attempted)(no units)
Session Attempt Delay = ___________________________ Session Attempt Delay = ___________________________
(seconds) (seconds)
5.3. Device Benchmarks for NS 5.3. Device Benchmarks for NS
IM Rate = _______________________________ (IM messages per second) IM Rate = _______________________________ (IM messages per second)
6. Test Cases 6. Test Cases
6.1. Baseline Session Establishment Rate 6.1. Baseline Session Establishment Rate of the test bed
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the Emulated Agent
zero failures. (EA) with zero failures.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. Configure the DUT in the test topology shown in Figure 1 in
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT [I-D.sip-bench-term].
is being benchmarked as a user agent client or user agent 2. Set media streams per session to 0.
server. If a SUT is being benchmarked, configure the SUT as 3. Execute benchmarking algorithm as defined in Section 4.9 to
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. get the baseline session establishment rate. This rate MUST
2. Configure Tester with a Session Attempt Rate = 100 SPS, be recorded using any pertinent parameters as shown in the
maximum Session Attempts = 100,000 and Media Streams Per reporting format of Section 5.1.
Session=0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: This is the scenario to obtain the maximum Session Expected Results: This is the scenario to obtain the maximum Session
Establishment Rate of the test bed when no DUT?SUT is present. Establishment Rate of the EA and the test bed when no DUT/SUT is
The results of this test might be used to normalize test results present. The results of this test might be used to normalize test
performed on different test beds or simply to better understand results performed on different test beds or simply to better
the impact of the DUT/SUT on the test bed in question. understand the impact of the DUT/SUT on the test bed in question.
6.2. Session Establishment Rate 6.2. Session Establishment Rate without media
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT/SUT with no
zero failures. associated media and zero failures.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT/SUT is being benchmarked as a user agent client or
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT a user agent server, configure the DUT in the test topology
is being benchmarked as a user agent client or user agent shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as Alternatively, if the DUT is being benchmarked as a proxy or a
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. B2BUA, configure the DUT in the test topology shown in Figure
2. Configure Tester with a Session Attempt Rate = 100 SPS, 5 in [I-D.sip-bench-term].
maximum Session Attempts = 100,000 and Media Streams Per 2. Configure a SUT according to the test topology shown in Figure
Session=0. 7 in [I-D.sip-bench-term].
3. Start Tester to initiate SIP Session establishment with the 3. Set media streams per session to 0.
DUT. 4. Execute benchmarking algorithm as defined in Section 4.9 to
4. Measure Session Attempt Failures and total Established get the session establishment rate. This rate MUST be
Sessions at the Tester. recorded using any pertinent parameters as shown in the
5. If a Session Attempt Failure is recorded then reduce the reporting format of Section 5.1.
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: This is the scenario to obtain the maximum Session Expected Results: This is the scenario to obtain the maximum Session
Establishment Rate of the DUT/SUT. Establishment Rate of the DUT/SUT.
6.3. Session Establishment Rate with Media 6.3. Session Establishment Rate with Media on DUT/SUT
Objective: Objective:
To benchmark the Session Establishment Rate of the SUT with zero To benchmark the Session Establishment Rate of the DUT/SUT with
failures when Associated Media is included in the benchmark test. zero failures when Associated Media is included in the benchmark
test and the media is running through the DUT/SUT.
Procedure: Procedure:
1. Configure the SUT as shown in Figure 6 or Figure 9 in 1. If the DUT is being benchmarked as a user agent client or a
user agent server, configure the DUT in the test topology
shown in Figure 3 or Figure 4 of [I-D.sip-bench-term].
Alternatively, if the DUT is being benchmarked as a B2BUA,
configure the DUT in the test topology shown in Figure 6 in
[I-D.sip-bench-term]. [I-D.sip-bench-term].
2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum 2. Configure a SUT according to the test topology shown in Figure
Session Attempts = 100,000 and Media Streams Per Session = 1. 9 in [I-D.sip-bench-term].
The rate of offered load for each media stream SHOULD be (eq 3. Set media streams per session to 1.
1) Offered Load per Media Stream = Throughput / maximum 4. Execute benchmarking algorithm as defined in Section 4.9 to
sessions attempted, where Throughput is defined in [RFC2544]. get the session establishment rate with media. This rate MUST
be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1.
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. At the Tester measure Session Attempt Failures, total
Established Sessions, and Packet Loss [RFC2544] of the media.
5. If a Session Attempt Failure or Packet Loss is recorded then
reduce the Session Attempt Rate configured on the Tester by
50%.
6. If no Session Attempt Failure or Packet Loss is recorded then
increase the Session Attempt Rate configured on the Tester by
50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
8. Repeat steps 1 through 7 for multimedia in which Media Streams
Per Session = 2.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
Associated Media with any number of media streams per SIP session Associated Media with any number of media streams per SIP session
are expected to be identical to the Session Establishment Rate are expected to be identical to the Session Establishment Rate
results obtained without media in the case where the server is results obtained without media in the case where the server is
running on a platform separate from the platform on which the running on a platform separate from the platform on which the
Media Relay, NAT or Firewall is running. Session Establishment Media Relay, NAT or Firewall is running. Session Establishment
Rate results obtained with Associated Media may be lower than Rate results obtained with Associated Media may be lower than
those obtained without media in the case where the server and the those obtained without media in the case where the server and the
NAT, Firewall or Media Relay are running on the same platform. NAT, Firewall or Media Relay are running on the same platform.
6.4. Session Establishment Rate with Loop Detection Enabled 6.4. Session Establishment Rate with Media not on DUT/SUT
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Loop Detection option is enabled. zero failures when Associated Media is included in the benchmark
test but the media is not running through the DUT/SUT.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as proxy or B2BUA, configure
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT the DUT in the test topology shown in Figure 7 in
is being benchmarked as a user agent client or user agent [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as 2. Configure a SUT according to the test topology shown in Figure
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. 8 in [I-D.sip-bench-term].
2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum 3. Set media streams per session to 1.
Session Attempts = 100,000 and Media Streams Per Session=0. 4. Execute benchmarking algorithm as defined in Section 4.9 to
3. Turn on the Loop Detection option in the DUT or SUT. get the session establishment rate with media. This rate MUST
4. Start Tester to initiate SIP Session establishment with the be recorded using any pertinent parameters as shown in the
DUT. reporting format of Section 5.1.
5. Measure Session Attempt Failures and total Established
Sessions at the Tester. Expected Results: Session Establishment Rate results obtained with
6. If a Session Attempt Failure is recorded then reduce the Associated Media with any number of media streams per SIP session
Session Attempt Rate configured on the Tester by 50%. are expected to be identical to the Session Establishment Rate
7. If no Session Attempt Failure is recorded then increase the results obtained without media in the case where the server is
Session Attempt Rate configured on the Tester by 50%. running on a platform separate from the platform on which the
Media Relay, NAT or Firewall is running. Session Establishment
Rate results obtained with Associated Media may be lower than
those obtained without media in the case where the server and the
NAT, Firewall or Media Relay are running on the same platform.
6.5. Session Establishment Rate with Loop Detection Enabled
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Loop Detection option is enabled and no
media streams are present.
Procedure:
1. If the DUT is being benchmarked as a proxy or B2BUA, and loop
detection is supported in the DUT, then configure the DUT in
the test topology shown in Figure 5 in [I-D.sip-bench-term].
If the DUT does not support loop detection, then this step can
be skipped.
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Turn on the Loop Detection option in the DUT or SUT.
5. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with loop detection
enabled. This rate MUST be recorded using any pertinent
parameters as shown in the reporting format of Section 5.1.
8. Repeat steps 4 through 7 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
Loop Detection may be lower than those obtained without Loop Loop Detection may be lower than those obtained without Loop
Detection enabled. Detection enabled.
6.5. Session Establishment Rate with Forking 6.6. Session Establishment Rate with Forking
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Forking Option is enabled. zero failures when the Forking Option is enabled.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as a proxy or B2BUA, and
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT forking is supported in the DUT, then configure the DUT in the
is being benchmarked as a user agent client or user agent test topology shown in Figure 5 in [I-D.sip-bench-term]. If
server. If a SUT is being benchmarked, configure the SUT as the DUT does not support forking, then this step can be
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. skipped.
2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum 2. Configure a SUT according to the test topology shown in Figure
Session Attempts = 100,000 and Media Streams Per Session=0. 8 of [I-D.sip-bench-term].
3. Set the number of endpoints that will receive the forked 3. Set media streams per session to 0.
4. Set the number of endpoints that will receive the forked
invitation to a value of 2 or more (subsequent tests may invitation to a value of 2 or more (subsequent tests may
increase this value at the discretion of the tester.) increase this value at the discretion of the tester.)
4. Start Tester to initiate SIP Session establishment with the 5. Execute benchmarking algorithm as defined in Section 4.9 to
DUT. get the session establishment rate with forking. This rate
5. Measure Session Attempt Failures and total Established MUST be recorded using any pertinent parameters as shown in
Sessions at the Tester. the reporting format of Section 5.1.
6. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
7. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
8. Repeat steps 4 through 7 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
Forking may be lower than those obtained without Forking enabled. Forking may be lower than those obtained without Forking enabled.
6.6. Session Establishment Rate with Forking and Loop Detection 6.7. Session Establishment Rate with Forking and Loop Detection
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when both the Forking and Loop Detection Options are zero failures when both the Forking and Loop Detection Options are
enabled. enabled.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT
is being benchmarked as a user agent client or user agent
server. If a SUT is being benchmarked, configure the SUT as
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term].
2. Configure Tester for a Session Attempt Rate = 100 SPS, maximum Procedure:
Session Attempts = 100,000 and Media Streams Per Session=0. 1. If the DUT is being benchmarked as a proxy or B2BUA, then
3. Start Tester to initiate SIP Session establishment with the configure the DUT in the test topology shown in Figure 5 in
DUT. [I-D.sip-bench-term].
2. Configure a SUT according to the test topology shown in Figure
8 of [I-D.sip-bench-term].
3. Set media streams per session to 0.
4. Enable the Loop Detection Options on the DUT. 4. Enable the Loop Detection Options on the DUT.
5. Set the number of endpoints that will receive the forked 5. Set the number of endpoints that will receive the forked
invitation to a value of 2 or more (subsequent tests may invitation to a value of 2 or more (subsequent tests may
increase this value at the discretion of the tester.) increase this value at the discretion of the tester.)
6. Measure Session Attempt Failures and total Established 6. Execute benchmarking algorithm as defined in Section 4.9 to
Sessions at the Tester. get the session establishment rate with forking and loop
7. If a Session Attempt Failure is recorded then reduce the detection. This rate MUST be recorded using any pertinent
Session Attempt Rate configured on the Tester by 50%. parameters as shown in the reporting format of Section 5.1.
8. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
9. Repeat steps 4 through 7 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
Forking and Loop Detection may be lower than those obtained with Forking and Loop Detection may be lower than those obtained with
only Forking or Loop Detection enabled. only Forking or Loop Detection enabled.
6.7. Session Establishment Rate with TLS Encrypted SIP 6.8. Session Establishment Rate with TLS Encrypted SIP
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when using TLS encrypted SIP. zero failures when using TLS encrypted SIP.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as a proxy or B2BUA, then
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT configure the DUT in the test topology shown in Figure 5 in
is being benchmarked as a user agent client or user agent [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as 2. Configure a SUT according to the test topology shown in Figure
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. 8 of [I-D.sip-bench-term].
2. Configure Tester for SIP TCP, enable TLS, Session Attempt Rate 3. Set media streams per session to 0.
= 100 SPS, maximum Session Attempts = 100,000 and Media 4. Configure Tester to enable TLS over the transport being
Streams Per Session = 0. benchmarked. Make a note the transport when compiling
3. Start Tester to initiate SIP Session establishment with the results. May need to run for each transport of interest.
DUT. 5. Execute benchmarking algorithm as defined in Section 4.9 to
4. Measure Session Attempt Failures and total Established get the session establishment rate with encryption. This rate
Sessions at the Tester. MUST be recorded using any pertinent parameters as shown in
5. If a Session Attempt Failure is recorded then reduce the the reporting format of Section 5.1.
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
TLS Encrypted SIP may be lower than those obtained with plaintext TLS Encrypted SIP may be lower than those obtained with plaintext
SIP. SIP.
6.8. Session Establishment Rate with IPsec Encrypted SIP 6.9. Session Establishment Rate with IPsec Encrypted SIP
Objective: Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when using IPsec Encryoted SIP. zero failures when using IPsec Encryoted SIP.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as a proxy or B2BUA, then
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT configure the DUT in the test topology shown in Figure 5 in
is being benchmarked as a user agent client or user agent [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as 2. Configure a SUT according to the test topology shown in Figure
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. 8 of [I-D.sip-bench-term].
2. Configure Tester for SIP TCP, enable IPSec, Session Attempt 3. Set media streams per session to 0.
Rate = 100 SPS, maximum Session Attempts = 100,000 and Media 4. Configure Tester for IPSec.
Streams Per Session = 0. 5. Execute benchmarking algorithm as defined in Section 4.9 to
3. Start Tester to initiate SIP Session establishment with the get the session establishment rate with encryption. This rate
DUT. MUST be recorded using any pertinent parameters as shown in
4. Measure Session Attempt Failures and total Established the reporting format of Section 5.1.
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
IPSec Encrypted SIP may be lower than those obtained with IPSec Encrypted SIP may be lower than those obtained with
plaintext SIP. plaintext SIP.
6.9. Session Establishment Rate with SIP Flooding 6.10. Session Establishment Rate with SIP Flooding
Objective: Objective:
To benchmark the Session Establishment Rate of the SUT with zero To benchmark the Session Establishment Rate of the SUT with zero
failures when SIP Flooding is occurring. failures when SIP Flooding is occurring.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as a proxy or B2BUA, then
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT configure the DUT in the test topology shown in Figure 5 in
is being benchmarked as a user agent client or user agent [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as 2. Configure a SUT according to the test topology shown in Figure
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. 8 of [I-D.sip-bench-term].
2. Configure Tester for SIP UDP with an Session Attempt Rate = 3. Set media streams per session to 0.
100 SPS, maximum Session Attempts = 100,000, Associated Media 4. Set s = 500 (c.f. Section 4.9).
Streams Per Session = 0, and SIP INVITE Message Flood = 500 5. Execute benchmarking algorithm as defined in Section 4.9 to
per second. get the session establishment rate with flooding. This rate
MUST be recorded using any pertinent parameters as shown in
the reporting format of Section 5.1.
3. Start Tester to initiate SIP Session establishment with the
SUT and SIP Flood targetted at the Server.
4. At the Tester measure Session Attempt Failures, total
Established Sessions, and Packet Loss [RFC2544] of the media.
5. If a Session Attempt Failure or Packet Loss is recorded then
reduce the Session Attempt Rate configured on the Tester by
50%.
6. If no Session Attempt Failure or Packet Loss is recorded then
increase the Session Attempt Rate configured on the Tester by
50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
8. Repeat steps 1 through 7 with SIP INVITE Message Flood = 1000
per second.
Expected Results: Session Establishment Rate results obtained with Expected Results: Session Establishment Rate results obtained with
SIP Flooding may be degraded. SIP Flooding may be degraded.
6.10. Maximum Registration Rate 6.11. Maximum Registration Rate
Objective: Objective:
To benchmark the maximum registration rate of the DUT/SUT with To benchmark the maximum registration rate of the DUT/SUT with
zero failures. zero failures.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as a proxy or B2BUA, then
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT configure the DUT in the test topology shown in Figure 5 in
is being benchmarked as a user agent client or user agent [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as 2. Configure a SUT according to the test topology shown in Figure
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. 8 of [I-D.sip-bench-term].
2. Configure Tester with a Registration Rate = 100 SPS and 3. Set media streams per session to 0.
maximum registrations attempted = 100,000. 4. Set the registration timeout value to at least 3600 seconds.
3. Set the registration timeout value to at least 3600 seconds. 5. Execute benchmarking algorithm as defined in Section 4.9 to
4. At the Tester measure failed registration attempts, total get the maximum registration rate. This rate MUST be recorded
registrations and packet loss. using any pertinent parameters as shown in the reporting
5. If a Failed Registration Attempt or Packet Loss is recorded format of Section 5.1.
then reduce the Attempted Registration Rate configured on the
Tester by 50%.
6. If no Failed Registration or Packet Loss is recorded then
increase the Attempted Registration Rate configured on the
Tester by 50%.
7. Repeat steps 5 and 6 until the all registrations have
succeeded. This number is obtained and recorded.
Expected Results: Expected Results:
6.11. Maximum Re-Registration Rate 6.12. Maximum Re-Registration Rate
Objective: Objective:
To benchmark the maximum re-registration rate of the DUT/SUT with To benchmark the maximum re-registration rate of the DUT/SUT with
zero failures. zero failures.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT is being benchmarked as a proxy or B2BUA, then
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT configure the DUT in the test topology shown in Figure 5 in
is being benchmarked as a user agent client or user agent [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as 2. Configure a SUT according to the test topology shown in Figure
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. 8 of [I-D.sip-bench-term].
2. Execute test detailed in Section 6.10 to register the 3. First, execute test detailed in Section 6.11 to register the
endpoints with the registrar. The rest of the steps below endpoints with the registrar.
MUST be performed at least 5 minutes after, but no more than 4. After at least 5 mintes of Step 2, but no more than 10 minutes
15 minutes after the test performed in Section 6.10. after Step 2 has been performed, execute test detailed in
3. Configure Tester for an attempted Registration Rate = 100 SPS Section 6.11 again (this will count as a re-registration).
and maximum registrations attempted = 100,000. 5. Execute benchmarking algorithm as defined in Section 4.9 to
4. Configure Tester to re-register the same address-of-records get the maximum re-registration rate. This rate MUST be
that were registered in Section 6.10. recorded using any pertinent parameters as shown in the
5. At the Tester measure failed registration attempts, total reporting format of Section 5.1.
registrations and packet loss.
6. If a Failed Registration Attempt or Packet Loss is recorded
then reduce the Attempted Registration Rate configured on the
Tester by 50%.
7. If no Failed Registration or Packet Loss is recorded then
increase the Attempted Registration Rate configured on the
Tester by 50%.
8. Repeat steps 6 and 7 until the all re-registrations have
succeeded. This number is obtained and recorded.
Expected Results: The rate should be at least equal to but not more Expected Results: The rate should be at least equal to but not more
than the result of Section 6.10. than the result of Section 6.11.
6.12. Maximum IM Rate 6.13. Maximum IM Rate
Objective: Objective:
To benchmark the maximum IM rate of the SUT with zero failures. To benchmark the maximum IM rate of the SUT with zero failures.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT/SUT is being benchmarked as a user agent client or
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT a user agent server, configure the DUT in the test topology
is being benchmarked as a user agent client or user agent shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as Alternatively, if the DUT is being benchmarked as a proxy or a
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. B2BUA, configure the DUT in the test topology shown in Figure
2. Configure Tester for an IM Rate = 100 SPS, Maximum IM 5 in [I-D.sip-bench-term].
Attempted = 100,000. 2. Configure a SUT according to the test topology shown in Figure
3. At the Tester measure Failed IM Attempts, Total IM and Packet 5 in [I-D.sip-bench-term].
Loss. 3. Execute benchmarking algorithm as defined in Section 4.9 to
4. If a Failed IM Attempt or Packet Loss is recorded then reduce get the maximum IM rate. This rate MUST be recorded using any
the Attempted IM Rate configured on the Tester by 50%. pertinent parameters as shown in the reporting format of
Section 5.1.
5. If no Failed IM or Packet Loss is recorded then increase the
Attempted IM Rate configured on the Tester by 50%.
6. Repeat steps 3 through 6 until the Maximum IM Rate is obtained
and recorded.
Expected Results: Expected Results:
6.13. Session Capacity without Media 6.14. Session Capacity without Media
Objective: Objective:
To benchmark the Session Capacity of the SUT without Associated To benchmark the Session Capacity of the SUT without Associated
Media. Media.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or 1. If the DUT/SUT is being benchmarked as a user agent client or
Figure 2 in [I-D.sip-bench-term] depending on whether the DUT a user agent server, configure the DUT in the test topology
is being benchmarked as a user agent client or user agent shown in Figure 1 or Figure 2 in [I-D.sip-bench-term].
server. If a SUT is being benchmarked, configure the SUT as Alternatively, if the DUT is being benchmarked as a proxy or a
shown in Figure 5 or Figure 6 in [I-D.sip-bench-term]. B2BUA, configure the DUT in the test topology shown in Figure
2. Configure Tester for a Session Attempt Rate = Session 5 in [I-D.sip-bench-term].
Establishment Rate, maximum Session Attempts = 10,000 and 2. Configure a SUT according to the test topology shown in Figure
Media Streams Per Session = 0. 7 in [I-D.sip-bench-term].
3. Start Tester to initiate SIP Session establishment with the 3. Set the media treams per session to be 0.
DUT. 4. Set the Session Duration to be a value greater than T.
4. Measure Session Attempt Failures, total Established Sessions, 5. Execute benchmarking algorithm as defined in Section 4.9 to
and Packet Loss [RFC2544] at the Tester. get the baseline session establishment rate. This rate MUST
5. If a Session Attempt Failure or Packet Loss is recorded then be recorded using any pertinent parameters as shown in the
reduce the maximum Session Attempts configured on the Tester reporting format of Section 5.1.
by 5,000. 6. The Session Capacity is the product of T and the Session
6. If no Session Attempt Failure or Packet Loss is recorded then Establishment Rate.
increase the maximum Session Attempts configured on the Tester
by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained and recorded.
8. Repeat steps 1 through 7 for multimedia in which media streams
per session = 2.
Expected Results: This is the scenario to obtain the maximum Session
Capacity of the DUT/SUT.
6.14. Session Capacity with Media Expected Results: The maximum rate at which the DUT/SUT can handle
session establishment requests with no media for an infinitely
long period with no errors. This is the SIP "throughput" of the
system with no media.
6.15. Session Capacity with Media
Objective: Objective:
To benchmark the session capacity of the DUT/SUT with Associated To benchmark the session capacity of the DUT/SUT with Associated
Media. Media.
Procedure: Procedure:
1. Configure the DUT in the test topology shown in Figure 3 or 1. Configure the DUT in the test topology shown in Figure 3 or
Figure 4 in [I-D.sip-bench-term] depending on whether the DUT Figure 4 of [I-D.sip-bench-term] depending on whether the DUT
is being benchmarked as a user agent client or user agent is being benchmarked as a user agent client or user agent
server. If a SUT is being benchmarked, configure the SUT as server. Alternatively, configure the DUT in the test topology
shown in Figure 9 of [I-D.sip-bench-term]. shown in Figure 6 or Figure 7 in [I-D.sip-bench-term]
depending on whether the DUT is being benchmarked as a B2BUA
2. Configure Tester for a Session Attempt Rate = 100 SPS, Session or as a proxy. If a SUT is being benchmarked, configure the
Duration = 30 sec, maximum Session Attempts = 100,000 and SUT as shown in Figure 9 of [I-D.sip-bench-term].
Media Streams Per Session = 1. 2. Set the media streams per session to 1.
NOTE: The total offered load to the DUT/SUT SHOULD be equal 3. Set the Session Duration to be a value greater than T.
to the Throughput of the DUT/SUT as defined in [RFC2544]. 4. Execute benchmarking algorithm as defined in Section 4.9 to
The offered load to the DUT/SUT for each media stream get the baseline session establishment rate. This rate MUST
SHOULD be equal to be recorded using any pertinent parameters as shown in the
Throughput/Maximum Session Attemps. reporting format of Section 5.1.
3. Start Tester to initiate SIP Session establishment with the 5. The Session Capacity is the product of T and the Session
SUT and transmit media through the SUT to a destination other Establishment Rate.
than the server.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
maximum Session Attempts configured on the Tester by 5,000.
6. If no Session Attempt Failure is recorded then increase the
maximum Session Attempts configured on the Tester by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained and recorded.
Expected Results: Session Capacity results obtained with Associated Expected Results: Session Capacity results obtained with Associated
Media with any number of media streams per SIP session will be Media with any number of media streams per SIP session will be
identical to the Session Capacity results obtained without media. identical to the Session Capacity results obtained without media.
6.15. Session Capacity with Media and a Media Relay/NAT and/or Firewall 6.16. Session Capacity with Media and a Media Relay/NAT and/or Firewall
Objective: Objective:
To benchmark the Session Establishment Rate of the SUT with To benchmark the Session Establishment Rate of the SUT with
Associated Media. Associated Media.
Procedure: Procedure:
1. Configure the SUT as shown in Figure 6 or Figure 9 in 1. Configure the SUT as shown in Figure 7 or Figure 10 in
[I-D.sip-bench-term]. [I-D.sip-bench-term].
2. Configure Tester for a Session Attempt Rate = 100 SPS, Session 2. Set media streams per session to 1.
Duration = 30 sec, maximum Session Attempts = 100,000 and 3. Execute benchmarking algorithm as defined in Section 4.9 to
Media Streams Per Session = 1. get the session establishment rate with media. This rate MUST
NOTE: The offered load for each media stream SHOULD be as be recorded using any pertinent parameters as shown in the
in Equation 1. reporting format of Section 5.1.
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
maximum Session Attempts configured on the Tester by 5,000.
6. If no Session Attempt Failure is recorded then increase the
maximum Session Attempts configured on the Tester by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained and recorded.
Expected Results: Session Capacity results obtained with Associated Expected Results: Session Capacity results obtained with Associated
Media with any number of media streams per SIP session may be Media with any number of media streams per SIP session may be
lower than the Session Capacity without Media result if the Media lower than the Session Capacity without Media result if the Media
Relay, NAT or Firewall is sharing a platform with the server. Relay, NAT or Firewall is sharing a platform with the server.
7. IANA Considerations 7. IANA Considerations
This document does not requires any IANA considerations. This document does not requires any IANA considerations.
skipping to change at page 18, line 43 skipping to change at page 19, line 43
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[I-D.sip-bench-term] [I-D.sip-bench-term]
Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance
Benchmarking Terminology", Benchmarking Terminology",
draft-ietf-bmwg-sip-bench-term-03 (work in progress), draft-ietf-bmwg-sip-bench-term-04 (work in progress),
March 2011. March 2012.
10.2. Informative References 10.2. Informative References
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
Authors' Addresses Authors' Addresses
 End of changes. 72 change blocks. 
395 lines changed or deleted 453 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/