draft-ietf-bmwg-sip-bench-meth-04.txt   draft-ietf-bmwg-sip-bench-meth-05.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working Group C. Davids
Group Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Internet-Draft V. Gurbani Expires: April 25, 2013 V. Gurbani
Expires: September 13, 2012 Bell Laboratories, Alcatel-Lucent Bell Laboratories,
Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
March 12, 2012 October 22, 2012
Methodology for Benchmarking SIP Networking Devices Methodology for Benchmarking SIP Networking Devices
draft-ietf-bmwg-sip-bench-meth-04 draft-ietf-bmwg-sip-bench-meth-05
Abstract Abstract
This document describes the methodology for benchmarking Session This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document. The methodology and terminology benchmarking terminology document. The methodology and terminology
are to be used for benchmarking signaling plane performance with are to be used for benchmarking signaling plane performance with
varying signaling and media load. Both scale and establishment rate varying signaling and media load. Both scale and establishment rate
are measured by signaling plane performance. The SIP Devices to be are measured by signaling plane performance. The SIP Devices to be
benchmarked may be a single device under test (DUT) or a system under benchmarked may be a single device under test (DUT) or a system under
skipping to change at page 1, line 42 skipping to change at page 1, line 43
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 13, 2012. This Internet-Draft will expire on April 25, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 3, line 10 skipping to change at page 3, line 10
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5 3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5 4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 5
4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5 4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 5
4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5 4.2. Signaling Server . . . . . . . . . . . . . . . . . . . . . 5
4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5 4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 5
4.4. Selection of Associated Media Protocol . . . . . . . . . . 6 4.4. Selection of Associated Media Protocol . . . . . . . . . . 6
4.5. Number of Associated Media Streams per SIP Session . . . . 6 4.5. Number of Associated Media Streams per SIP Session . . . . 6
4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6 4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 6
4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6 4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 6
4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6 4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 6
4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6 4.9. Benchmarking algorithm . . . . . . . . . . . . . . . . . . 6
5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9 5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9
5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9 5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9
5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 9 5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10
5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10 5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10
6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10 6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.1. Baseline Session Establishment Rate of the test bed . . . 10 6.1. Baseline Session Establishment Rate of the test bed . . . 10
6.2. Session Establishment Rate without media . . . . . . . . . 11 6.2. Session Establishment Rate without media . . . . . . . . . 11
6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11 6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11
6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12 6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12
6.5. Session Establishment Rate with Loop Detection Enabled . . 12 6.5. Session Establishment Rate with Loop Detection Enabled . . 13
6.6. Session Establishment Rate with Forking . . . . . . . . . 13 6.6. Session Establishment Rate with Forking . . . . . . . . . 13
6.7. Session Establishment Rate with Forking and Loop 6.7. Session Establishment Rate with Forking and Loop
Detection . . . . . . . . . . . . . . . . . . . . . . . . 14 Detection . . . . . . . . . . . . . . . . . . . . . . . . 14
6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14 6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14
6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15 6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15
6.10. Session Establishment Rate with SIP Flooding . . . . . . . 15 6.10. Session Establishment Rate with SIP Flooding . . . . . . . 15
6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16 6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16
6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16 6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16
6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17 6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17
6.14. Session Capacity without Media . . . . . . . . . . . . . . 17 6.14. Session Capacity without Media . . . . . . . . . . . . . . 17
skipping to change at page 4, line 42 skipping to change at page 4, line 42
may be any [RFC3261] conforming device. The SUT can be any device or may be any [RFC3261] conforming device. The SUT can be any device or
group of devices containing RFC 3261 conforming functionality along group of devices containing RFC 3261 conforming functionality along
with Firewall and/or NAT functionality. This enables benchmarks to with Firewall and/or NAT functionality. This enables benchmarks to
be obtained and compared for different types of devices such as SIP be obtained and compared for different types of devices such as SIP
Proxy Server, SBC, SIP proxy server paired with a media relay or Proxy Server, SBC, SIP proxy server paired with a media relay or
Firewall/NAT device. SIP Associated Media benchmarks can also be Firewall/NAT device. SIP Associated Media benchmarks can also be
made when testing SUTs. made when testing SUTs.
The test cases covered in this methodology document provide The test cases covered in this methodology document provide
benchmarks metrics of Registration Rate, SIP Session Establishment benchmarks metrics of Registration Rate, SIP Session Establishment
Rate, Session Capacity, IM Rate, and Presence Rate. These can be Rate, Session Capacity, and IM Rate. These can be benchmarked with
benchmarked with or without associated Media. Some cases are also or without associated Media. Some cases are also included to cover
included to cover Forking, Loop detecion, Encrypted SIP, and SIP Forking, Loop detection, Encrypted SIP, and SIP Flooding. The test
Flooding. The test topologies that can be used are described in the topologies that can be used are described in the Test Setup section.
Test Setup section. Topologies are provided for benchmarking of a Topologies are provided for benchmarking of a DUT or SUT.
DUT or SUT. Benchmarking with Associated Media can be performed when Benchmarking with Associated Media can be performed when using a SUT.
using a SUT.
SIP permits a wide range of configuration options that are also SIP permits a wide range of configuration options that are also
explained in the Test Setup section. Benchmark metrics could explained in the Test Setup section. Benchmark metrics could
possibly be impacted by Associated Media. The selected values for possibly be impacted by Associated Media. The selected values for
Session Duration and Media Streams Per Session enable benchmark Session Duration and Media Streams Per Session enable benchmark
metrics to be benchmarked without Associated Media. Session Setup metrics to be benchmarked without Associated Media. Session Setup
Rate could possibly be impacted by the selected value for Maximum Rate could possibly be impacted by the selected value for Maximum
Sessions Attempted. The benchmark for Session Establishment Rate is Sessions Attempted. The benchmark for Session Establishment Rate is
measured with a fixed value for maximum Session Attempts. measured with a fixed value for maximum Session Attempts.
skipping to change at page 5, line 26 skipping to change at page 5, line 25
document does not make any claims on the interpretation of such document does not make any claims on the interpretation of such
results. results.
3. Benchmarking Topologies 3. Benchmarking Topologies
Familiarity with the benchmarking models in Section 2.2 of Familiarity with the benchmarking models in Section 2.2 of
[I-D.sip-bench-term] is assumed. Figures 1 through 10 in [I-D.sip-bench-term] is assumed. Figures 1 through 10 in
[I-D.sip-bench-term] contain the canonical topologies that can be [I-D.sip-bench-term] contain the canonical topologies that can be
used to perform the benchmarking tests listed in this document. used to perform the benchmarking tests listed in this document.
4. Benchmarking Considerations 4. Test Setup Parameters
4.1. Selection of SIP Transport Protocol 4.1. Selection of SIP Transport Protocol
Test cases may be performed with any transport protocol supported by Test cases may be performed with any transport protocol supported by
SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and SIP. This includes, but is not limited to, SIP TCP, SIP UDP, and
TLS. The protocol used for the SIP transport protocol must be TLS. The protocol used for the SIP transport protocol must be
reported with benchmarking results. reported with benchmarking results.
4.2. Signaling Server 4.2. Signaling Server
The Server is a SIP-speaking device that complies with RFC 3261. The The Signaling Server is defined in the companion terminology
purpose of this document is to benchmark SIP performance, not document, ([I-D.sip-bench-term], Section 3.2.2) It is a SIP-speaking
conformance. Conformance to [RFC3261] is assumed for all tests. The device that complies with RFC 3261. Conformance to [RFC3261] is
Server may be the DUT or a component of a SUT that includes Firewall assumed for all tests. The Signaling Server may be the DUT or a
and/or NAT functionality. The components of the SUT may be a single component of a SUT. The Signaling Server may include Firewall and/or
NAT functionality. The components of the SUT may be a single
physical device or separate devices. physical device or separate devices.
4.3. Associated Media 4.3. Associated Media
Some tests may require associated media to be present for each SIP Some tests require Associated Media to be present for each SIP
session. The Server is not involved in the forwarding of media. session. The test topologies to be used when benchmarking SUT
Associated Media can be benchmarked only with a SUT in which the performance for Associated Media are shown in [I-D.sip-bench-term],
media traverses a Media Relay, Firewall, NAT, or Firewall/NAT device. Figures 4 and 5.
The test topologies to be used when benchmarking SUT performance for
Associated Media are shown in Figures 4 and 5, in which the SIP
signaling is bidirectional and the Associated Media is
unidirectional.
4.4. Selection of Associated Media Protocol 4.4. Selection of Associated Media Protocol
The test cases specified in this document provide SIP performance The test cases specified in this document provide SIP performance
independent of the protocol used for the media stream. Any media independent of the protocol used for the media stream. Any media
protocol supported by SIP may be used. This includes, but is not protocol supported by SIP may be used. This includes, but is not
limited to, RTP, RTSP, and SRTP. The protocol used for Associated limited to, RTP, RTSP, and SRTP. The protocol used for Associated
Media MUST be reported with benchmarking results. Media MUST be reported with benchmarking results.
4.5. Number of Associated Media Streams per SIP Session 4.5. Number of Associated Media Streams per SIP Session
skipping to change at page 7, line 13 skipping to change at page 7, line 8
provided. provided.
The goal is to find the largest value of a SIP session-request-rate, The goal is to find the largest value of a SIP session-request-rate,
measured in sessions-per-second, which the DUT/SUT can process with measured in sessions-per-second, which the DUT/SUT can process with
zero errors. To discover that number, an iterative process (defined zero errors. To discover that number, an iterative process (defined
below) is used to find a candidate for this rate. Once the candidate below) is used to find a candidate for this rate. Once the candidate
rate has been found, the DUT/SUT is subjected to an offered load rate has been found, the DUT/SUT is subjected to an offered load
whose arrival rate is set to that of the candidate rate. This test whose arrival rate is set to that of the candidate rate. This test
is run for an extended period of time, which is referred to as is run for an extended period of time, which is referred to as
infinity, and which is, itself, a parameter of the test labeled T in infinity, and which is, itself, a parameter of the test labeled T in
the pseudo-code). This latter phase of testing is called the steady- the pseudo-code. This latter phase of testing is called the steady-
state phase. If errors are encountered during this steady-state state phase. If errors are encountered during this steady-state
phase, then the candidate rate is reduced by a defined percent, also phase, then the candidate rate is reduced by a defined percent, also
a parameter of test, and the steady-state phase is entered again a parameter of test, and the steady-state phase is entered again
until a final (new) steady-state rate is achieved. until a final (new) steady-state rate is achieved.
The iterative process itself is defined as follows: a starting rate The iterative process itself is defined as follows: a starting rate
of 100 sessions per second (sps) is selected. The test is executed of 100 sessions per second (sps) is selected. The test is executed
for the time period identified by t in the pseudo-code below. If no for the time period identified by t in the pseudo-code below. If no
failures occur, the rate is increased to 150 sps and again tested for failures occur, the rate is increased to 150 sps and again tested for
time period t. The attempt rate is continuously ramped up until a time period t. The attempt rate is continuously ramped up until a
skipping to change at page 9, line 40 skipping to change at page 9, line 36
(seconds) (seconds)
Establishment Threshold time = ____________________ Establishment Threshold time = ____________________
(seconds) (seconds)
Loop Detecting Option = ___________________________ Loop Detecting Option = ___________________________
(on|off) (on|off)
Forking Option Forking Option
Number of endpoints request sent to = ___________ Number of endpoints request sent to = ___________
(1, means forking is not enabled) (1, means forking is not enabled)
Type of forking = _______________________________ Type of forking = _______________________________
(serial|parallel) (serial|parallel)
Authentication option = ___________________________________
(on|off; if on, please see Notes 2 and 3 below).
Note: Total Sessions Attempted is used in the calculation of the Note 1: Total Sessions Attempted is used in the calculation of the
Session Establishment Performance ([I-D.sip-bench-term], Section Session Establishment Performance ([I-D.sip-bench-term], Section
3.4.5). It is the number of session attempts ([I-D.sip-bench-term], 3.4.5). It is the number of session attempts ([I-D.sip-bench-term],
Section 3.1.6) that will be made over the duration of the test. Section 3.1.6) that will be made over the duration of the test.
Note 2: When the Authentication Option is "on" the test tool must be
set to ignore 401 and 407 failure responses in any test described as
a "test to failure." If this is not done, all such tests will yield
trivial benchmarks, as all attempt rates will lead to a failure after
the first attempt.
Note 3: When the Authentication Option is "on" the DUT/SUT uses two
transactions instead of one when it is establishing a session or
accomplishing a registration. The first transaction ends with the
401 or 407. The second ends with the 200 OK or another failure
message. The Test Organization interested in knowing how many times
the EA was intended to send a REGISTER as distinct from how many
times the EA wound up actually sending a REGISTER may wish to record
the following data as well: Number of responses of the following
type: 401: _____________ (if authentication turned on; N/A otherwise)
407: _____________ (if authentication turned on; N/A otherwise)
5.2. Device Benchmarks for IS 5.2. Device Benchmarks for IS
Registration Rate = _______________________________ Registration Rate = _______________________________
(registrations per second) (registrations per second)
Re-registration Rate = ____________________________ Re-registration Rate = ____________________________
(registrations per second) (registrations per second)
Session Capacity = _________________________________ Session Capacity = _________________________________
(sessions) (sessions)
Session Overload Capacity = ________________________ Session Overload Capacity = ________________________
(sessions) (sessions)
Session Establishment Rate = ______________________ Session Establishment Rate = ______________________
(sessions per second) (sessions per second)
 End of changes. 16 change blocks. 
34 lines changed or deleted 50 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/