Benchmarking Methodology Working                             S. Poretsky
Group                                               Allot Communications
Internet-Draft                                                V. Gurbani
Expires: August 12, 2010 January 13, 2011              Bell Laboratories, Alcatel-Lucent
                                                               C. Davids
                                        Illinois Institute of Technology
                                                        February 8,
                                                           July 12, 2010

          Methodology for Benchmarking SIP Networking Devices
                   draft-ietf-bmwg-sip-bench-meth-01
                   draft-ietf-bmwg-sip-bench-meth-02

Abstract

   This document describes the methodology for benchmarking Session
   Initiation Protocol (SIP) performance as described in SIP
   benchmarking terminology document.  The methodology and terminology
   are to be used for benchmarking signaling plane performance with
   varying signaling and media load.  Both scale and establishment rate
   are measured by signaling plane performance.  The SIP Devices to be
   benchmarked may be a single device under test (DUT) or a system under
   test (SUT).  Benchmarks can be obtained and compared for different
   types of devices such as SIP Proxy Server, SBC, and server paired
   with a media relay or Firewall/NAT device.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on August 12, 2010. January 13, 2011.

Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the BSD License.

Table of Contents

   1.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Test Topologies  . . . . . . . . . . . . . . . . . . . . . . .  5
   4.  Test Considerations  . . . . . . . . . . . . . . . . . . . . .  7  6
     4.1.  Selection of SIP Transport Protocol  . . . . . . . . . . .  7  6
     4.2.  Signaling Server . . . . . . . . . . . . . . . . . . . . . . . . . .  8  6
     4.3.  Associated Media . . . . . . . . . . . . . . . . . . . . .  8  7
     4.4.  Selection of Associated Media Protocol . . . . . . . . . .  8  7
     4.5.  Number of Associated Media Streams per SIP Session . . . .  8  7
     4.6.  Session Duration . . . . . . . . . . . . . . . . . . . . .  8  7
     4.7.  Attempted Sessions per Second  . . . . . . . . . . . . . .  9  7
     4.8.  Stress Testing . . . . . . . . . . . . . . . . . . . . . .  9  8
   5.  Reporting Format . . . . . . . . . . . . . . . . . . . . . . .  9  8
     5.1.  Test Setup Report  . . . . . . . . . . . . . . . . . . . .  9  8
     5.2.  Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10  9
     5.3.  Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10  9
   6.  Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10  9
     6.1.  Baseline Session Establishment Rate  . . . . . . . . . . .  9
     6.2.  Session Establishment Rate . . . . . . . . . . . . . . . . 10
     6.2.
     6.3.  Session Establishment Rate with Media  . . . . . . . . . . 10
     6.3.
     6.4.  Session Establishment Rate with Loop Detection Enabled . . 11
     6.4.
     6.5.  Session Establishment Rate with Forking  . . . . . . . . . 12
     6.5.
     6.6.  Session Establishment Rate with Forking and Loop
           Detection  . . . . . . . . . . . . . . . . . . . . . . . . 12
     6.6.
     6.7.  Session Establishment Rate with TLS Encrypted SIP  . . . . 13
     6.7.
     6.8.  Session Establishment Rate with IPsec Encrypted SIP  . . . 14
     6.8. 13
     6.9.  Session Establishment Rate with SIP Flooding . . . . . . . 14
     6.9.
     6.10. Maximum Registration Rate  . . . . . . . . . . . . . . . . 15
     6.10. 14
     6.11. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 15
     6.11.
     6.12. Maximum IM Rate  . . . . . . . . . . . . . . . . . . . . . 16
     6.12.
     6.13. Session Capacity without Media . . . . . . . . . . . . . . 16
     6.13.
     6.14. Session Capacity with Media  . . . . . . . . . . . . . . . 17
     6.14.
     6.15. Session Capacity with Media and a Media Relay/NAT
           and/or Firewall  . . . . . . . . . . . . . . . . . . . . . 18 17
   7.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 18
   8.  Security Considerations  . . . . . . . . . . . . . . . . . . . 18
   9.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 19 18
   10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 18
     10.1. Normative References . . . . . . . . . . . . . . . . . . . 19 18
     10.2. Informative References . . . . . . . . . . . . . . . . . . 19
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 19

1.  Terminology

   In this document, the key words "MUST", "MUST NOT", "REQUIRED",
   "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
   RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as
   described in BCP 14, conforming to [RFC2119] and indicate requirement
   levels for compliant implementations.

   Terms specific to SIP [RFC3261] performance benchmarking are defined
   in [I-D.sip-bench-term].

   RFC 2119 defines the use of these key words to help make the intent
   of standards track documents as clear as possible.  While this
   document uses these keywords, this document is not a standards track
   document.  The term Throughput is defined in [RFC2544].

2.  Introduction

   This document describes the methodology for benchmarking Session
   Initiation Protocol (SIP) performance as described in Terminology
   document [I-D.sip-bench-term].  The methodology and terminology are
   to be used for benchmarking signaling plane performance with varying
   signaling and media load.  Both scale and establishment rate are
   measured by signaling plane performance.

   The SIP Devices to be benchmarked may be a single device under test
   (DUT) or a system under test (SUT).  The DUT is a SIP Server, which
   may be any [RFC3261] conforming device.  The SUT can be any device or
   group of devices containing RFC 3261 conforming functionality along
   with Firewall and/or NAT functionality.  This enables benchmarks to
   be obtained and compared for different types of devices such as SIP
   Proxy Server, SBC, SIP proxy server paired with a media relay or
   Firewall/NAT device.  SIP Associated Media benchmarks can also be
   made when testing SUTs.

   The test cases covered in this methodology document provide
   benchmarks metrics of Registration Rate, SIP Session Establishment
   Rate, Session Capacity, IM Rate, and Presence Rate.  These can be
   benchmarked with or without associated Media.  Some cases are also
   included to cover Forking, Loop detecion, Encrypted SIP, and SIP
   Flooding.  The test topologies that can be used are described in the
   Test Setup section.  Topologies are provided for benchmarking of a
   DUT or SUT.  Benchmarking with Associated Media can be performed when
   using a SUT.

   SIP permits a wide range of configuration options that are also
   explained in the Test Setup section.  Benchmark metrics could
   possibly be impacted by Associated Media.  The selected values for
   Session Duration and Media Streams Per Session enable benchmark
   metrics to be benchmarked without Associated Media.  Session Setup
   Rate could possibly be impacted by the selected value for Maximum
   Sessions Attempted.  The benchmark for Session Establishment Rate is
   measured with a fixed value for maximum Session Attempts.

3.  Test Topologies

   Figures 1 through 6 3 below provide various topologies to perform the
   SIP Performance Benchmarking.  These figures show the Device Under
   Test (DUT) to be a single server or a System Under Test (SUT).  Test
   Topology options to include benchmarking with Associated Media
   require use of a SUT and are shown in Figures 4 and 5. 3.

              DUT
            ---------               ---------
            |       |               |       |
            |signal-|
            |ing    |               |       |       |
            |
            |server |      SIP      |       |
            |Server
            |       |<------------->| Tester|   EA  |
            |       |               |       |
            |       |               |       |
            |       |               |       |
            ---------               ---------

                     Figure 1: Basic SIP Test Topology

                      SUT
            ------------------------
            ---------      ---------         ---------
            |       |      |media- |         |       |
            |signal-|      | relay |         |       |       |         |       |
            |       |  SIP |Fire-
            |ing    |  SIP |   OR  |   SIP   | Server|<---------------------->| Tester|       |
            |server |<---------------------->|   EA  |      |Wall
            |       |      |fire   |         |       |
            |       |      |wall OR|         |       |
            |       |      |  NAT  |         |       |
            ---------      ---------         ---------

                 Figure 2: SIP Test Topology with Firewall
                      SUT
            ------------------------
            ---------      ---------         ---------
            |       |      | media-|         |       |
            |signal-|      |  relay|         |       |
            |ing    |       |         |       |
            |       |  SIP | NAT   |   SIP   |       |
            | Server|<---------------------->| Tester|
            |       |      |       |         |       |
            |       |      |       |         |       |
            |       |      |       |         |       |
            ---------      ---------         ---------

                Figure 3:  SIP Test Topology with NAT Device

                      SUT
            ------------------------
            ---------      ---------         ---------
            |       |      |       |         |       |
            |       |      |       |         |       | |   OR  |   SIP |Fire-  |   SIP   |       |   | Server|<---------------------->| Tester|       |
            |server |<---------------------->|   EA  |      |Wall
            |       |      |fire   |         |       |
            |       |      |wall OR|  Media  |       |
            |       |   ---|  NAT  |---------|       |
            ---------   |  ---------         ---------
                        |             Media      ^
                        -------------------------|

          Figure 4: 3: SIP Test Topology with Media through Firewall
                       SUT
            ------------------------
            ---------

                                    ---------         ---------
            |       |      |       |         |       |
            |       |      |       |         |       |
            |       |  SIP |  NAT  |   SIP   |       |
            | Server|<---------------------->| Tester|
                                    |       |
                     |------------->|       |       |         |       |
            |       |      |       |  Media  |       |
            |       |   ---|       |---------|       |
            ---------   |  ---------         ---------
                        |             Media
                     ^
                        -------------------------|

         Figure 5: SIP Test Topology with Media through NAT Device

              SUT
              ------------------------
              ---------      ---------         ---------
              |       |      |       |         |       |
              |       |      |       |         |              |       |
                     |      SIP |Media  |   SIP   |       |
              | Server|<---------------------->| Tester|
              |       |      |Relay  |         |       |
              |     |       |
                     |<-------------|   EA  |  Media
                                    |       |
                                    |       |   ---|       |---------|
                                    |
              ---------       |
                                    ---------         ---------
                          |             Media      ^
                          -------------------------|

                     Figure 6: SIP 4: Baseline Test Topology with Media Through a Media Relay

4.  Test Considerations

4.1.  Selection of SIP Transport Protocol

   Discussion:

      Test cases may be performed with any transport protocol supported
      by SIP.  This includes, but is not limited to, SIP TCP, SIP UDP,
      and TLS.  The protocol used for the SIP transport protocol must be
      reported with benchmarking results.

4.2.  Signaling Server

   Discussion:
      The Server is a SIP-speaking device that complies with RFC 3261.
      The purpose of this document is to benchmark SIP performance, not
      conformance.  Conformance to [RFC3261] is assumed for all tests.
      The Server may be the DUT or a component of a SUT that includes
      Firewall and/or NAT functionality.  The components of the SUT may
      be a single physical device or separate devices.

4.3.  Associated Media

   Discussion:

      Some tests may require associated media to be present for each SIP
      session.  The Server is not involved in the forwarding of media.
      Associated Media can be benchmarked only with a SUT in which the
      media traverses a Media Relay, Firewall, NAT, or Firewall/NAT
      device.  The test topologies to be used when benchmarking SUT
      performance for Associated Media are shown in Figures 4 and 5, in
      which the SIP signaling is bidirectional and the Associated Media
      is unidirectional.

4.4.  Selection of Associated Media Protocol

   Discussion:

      The test cases specified in this document provide SIP performance
      independent of the protocol used for the media stream.  Any media
      protocol supported by SIP may be used.  This includes, but is not
      limited to, RTP, RTSP, and SRTP.  The protocol used for Associated
      Media must MUST be reported with benchmarking results.

4.5.  Number of Associated Media Streams per SIP Session

   Discussion:

      Benchmarking results may vary with the number of media streams per
      SIP session.  When benchmarking a SUT for voice, a single media
      stream is used.  When benchmarking a SUT for voice and video, two
      media streams are used.  The number of Associated Media Streams
      must
      MUST be reported with benchmarking results.

4.6.  Session Duration
   Discussion:

      SUT performance benchmarks may vary with the duration of SIP
      sessions.  Session Duration must MUST be reported with benchmarking
      results.  A Session Duration of zero seconds indicates
      transmission of a BYE immediately following successful SIP
      establishment indicate by receipt of a 200 OK.  An infinite
      Session Duration indicates that a BYE is never transmitted.

4.7.  Attempted Sessions per Second

   Discussion:
      DUT and SUT performance benchmarks may vary with the the rate of
      attempted sessions offered by the Tester.  Attempted Sessions per
      Second must MUST be reported with benchmarking results.

4.8.  Stress Testing

   Discussion:
      The purpose of this document is to benchmark SIP performance, not
      system stability under stressful conditions such as a high rate of
      Attempted Sessions per Second.

5.  Reporting Format

5.1.  Test Setup Report

     SIP Transport Protocol = ___________________________
     (valid values: TCP|UDP|TLS|SCTP|specify-other)
     Session Attempt Rate = _____________________________
     (session attempts/sec)
     IS Media Attempt Rate = ____________________________
     (IS media attempts/sec)
     Total Sessions Attempted = _________________________
     (total sessions to be created over duration of test)
     Media Streams Per Session =  _______________________
     (number of streams per session)
     Associated Media Protocol =  _______________________
     (RTP|RTSP|specify-other)
     Media Packet Size =  _______________________________
     (bytes)
     Media Offered Load =  ______________________________
     (packets per second)
     Media Session Hold Time =  _________________________
     (seconds)
     Establishment Threshold Time time =  ____________________
     (seconds)
     Loop Detecting Option =  ___________________________
     (on|off)
     Forking Option
        Number of endpoints request sent to = ___________
     (1, means forking is not enabled)
        Type of forking = _______________________________
     (serial|parallel)

   Note: Total Sessions Attempted Attempted is used in the calculation of the
   Session Establishment Performance ([I-D.sip-bench-term], Section
   3.4.5).  It is the number of session attempts ([I-D.sip-bench-term],
   Section 3.1.6) that will be made over the duration of the test.

5.2.  Device Benchmarks for IS

     Registration Rate =  _______________________________
     (registrations per second)
     Re-registration Rate =  ____________________________
     (registrations per second)
     Session Capacity = _________________________________
     (sessions)
     Session Overload Capacity = ________________________
     (sessions)
     Session Establishment Rate =  ______________________
     (sessions per second)
     Session Establishment Performance =  _______________
     (total established sessions/total sessions attempted)(no units)
     Session Attempt Delay =  ___________________________
     (seconds)

5.3.  Device Benchmarks for NS

     IM Rate =  _______________________________ (IM messages per second)

6.  Test Cases

6.1.  Baseline Session Establishment Rate

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures.
   Procedure:
      1.  Configure the test bed in the test topology shown in Figure 4.
      2.  Configure Tester with a Session Attempt Rate = 100 SPS,
          maximum Session Attempts = 100,000 and Media Streams Per
          Session=0.
      3.  Start Tester to initiate SIP Session establishment with the
          DUT.
      4.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      5.  If a Session Attempt Failure is used in recorded then reduce the calculation of
          Session Attempt Rate configured on the Tester by 50%.
      6.  If no Session Establishment Performance ([I-D.sip-bench-term], Section
   3.4.5).  It Attempt Failure is recorded then increase the number of session attempts ([I-D.sip-bench-term],
   Section 3.1.6) that will be made over
          Session Attempt Rate configured on the duration of Tester by 50%.

      7.  Repeat steps 3 through 6 until the test.

5.2.  Device Benchmarks for IS

     Registration Rate =  _______________________________
     Re-registration Rate =  ____________________________
     Session Capacity = _________________________________
     Session Overload Capacity = ________________________ Session Establishment Rate =  ______________________
          is obtained and recorded.
   Expected Results:  This is the scenario to obtain the maximum Session
      Establishment Performance =  _______________
     Session Attempt Delay =  ___________________________
     Session Disconnect Delay =  ________________________

5.3.  Device Benchmarks for NS

     IM Rate =  _______________________________

6.  Test Cases

6.1. of the test bed when no DUT?SUT is present.
      The results of this test might be used to normalize test results
      performed on different test beds or simply to better understand
      the impact of the DUT/SUT on the test bed in question.

6.2.  Session Establishment Rate

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an a Session Attempt Rate = 100 SPS,
          maximum Session Attempts = 100,000 and Media Streams Per
          Session=0.
      3.  Start Tester to initiate SIP Session establishment with the
          DUT.
      4.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      5.  If a Session Attempt Failure is recorded then reduce the
          Session Attempt Rate configured on the Tester by 50%.
      6.  If no Session Attempt Failure is recorded then increase the
          Session Attempt Rate configured on the Tester by 50%.
      7.  Repeat steps 3 through 6 until the Session Establishment Rate
          is obtained and recorded.
   Expected Results:  This is the scenario to obtain the maximum Session
      Establishment Rate of the DUT/SUT.

6.2.

6.3.  Session Establishment Rate with Media

   Objective:
      To benchmark the Session Establishment Rate of the SUT with zero
      failures when Associated Media is included in the benchmark test.
   Procedure:
      1.  Configure the SUT in the test topology shown in Figure 4, 5 2 or
          6. 3.
      2.  Configure Tester for SIP UDP with an a Session Attempt Rate = 100 SPS, maximum
          Session Attempts = 100,000 and Media Streams Per Session = 1.
          The rate of offered load for each media stream SHOULD be (eq
          1) Offered Load per Media Stream = Throughput / maximum
          sessions attempted, where Throughput is defined in [RFC2544].
      3.  Start Tester to initiate SIP Session establishment with the
          SUT and transmit media through the SUT to a destination other
          than the server.

      4.  At the Tester measure Session Attempt Failures, total
          Established Sessions, and Packet Loss [RFC2544] of the media.
      5.  If a Session Attempt Failure or Packet Loss is recorded then
          reduce the Session Attempt Rate configured on the Tester by
          50%.
      6.  If no Session Attempt Failure or Packet Loss is recorded then
          increase the Session Attempt Rate configured on the Tester by
          50%.
      7.  Repeat steps 3 through 6 until the Session Establishment Rate
          is obtained and recorded.
      8.  Repeat steps 1 through 7 for multimedia in which Media Streams
          Per Session = 2.
   Expected Results:  Session Establishment Rate results obtained with
      Associated Media with any number of media streams per SIP session
      are expected to be identical to the Session Establishment Rate
      results obtained without media in the case where the server is
      running on a platform separate from the platform on which the
      Media Relay, NAT or Firewall is running.  Session Establishment
      Rate results obtained with Associated Media may be lower than
      those obtained without media in the case where the server and the
      NAT, Firewall or Media Relay are running on the same platform.

6.3.

6.4.  Session Establishment Rate with Loop Detection Enabled

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when the Loop Detection option is enabled.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an a Session Attempt Rate = 100 SPS, maximum
          Session Attempts = 100,000 and Media Streams Per Session=0.
      3.  Turn on the Loop Detection option in the DUT or SUT.
      4.  Start Tester to initiate SIP Session establishment with the
          DUT.
      5.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      6.  If a Session Attempt Failure is recorded then reduce the
          Session Attempt Rate configured on the Tester by 50%.
      7.  If no Session Attempt Failure is recorded then increase the
          Session Attempt Rate configured on the Tester by 50%.
      8.  Repeat steps 4 through 7 until the Session Establishment Rate
          is obtained and recorded.
   Expected Results:  Session Establishment Rate results obtained with
      Loop Detection may be lower than those obtained without Loop
      Detection enabled.

6.4.

6.5.  Session Establishment Rate with Forking

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when the Forking Option is enabled.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an a Session Attempt Rate = 100 SPS, maximum
          Session Attempts = 100,000 and Media Streams Per Session=0.
      3.  Set the number of endpoints that will receive the forked
          invitation to a value of 2 or more (subsequent tests may
          increase this value at the discretion of the tester.)
      4.  Start Tester to initiate SIP Session establishment with the
          DUT.
      5.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      6.  If a Session Attempt Failure is recorded then reduce the
          Session Attempt Rate configured on the Tester by 50%.
      7.  If no Session Attempt Failure is recorded then increase the
          Session Attempt Rate configured on the Tester by 50%.
      8.  Repeat steps 4 through 7 until the Session Establishment Rate
          is obtained and recorded.
   Expected Results:  Session Establishment Rate results obtained with
      Forking may be lower than those obtained without Forking enabled.

6.5.

6.6.  Session Establishment Rate with Forking and Loop Detection

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when both the Forking and Loop Detection Options are
      enabled.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an a Session Attempt Rate = 100 SPS, maximum
          Session Attempts = 100,000 and Media Streams Per Session=0.
      3.  Start Tester to initiate SIP Session establishment with the
          DUT.
      4.  Enable the Loop Detection Options on the DUT.
      5.  Set the number of endpoints that will receive the forked
          invitation to a value of 2 or more (subsequent tests may
          increase this value at the discretion of the tester.)
      6.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      7.  If a Session Attempt Failure is recorded then reduce the
          Session Attempt Rate configured on the Tester by 50%.

      8.  If no Session Attempt Failure is recorded then increase the
          Session Attempt Rate configured on the Tester by 50%.
      9.  Repeat steps 4 through 7 until the Session Establishment Rate
          is obtained and recorded.
   Expected Results:  Session Establishment Rate results obtained with
      Forking and Loop Detection may be lower than those obtained with
      only Forking or Loop Detection enabled.

6.6.

6.7.  Session Establishment Rate with TLS Encrypted SIP

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when using TLS encrypted SIP.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP TCP, enable TLS, Session Attempt Rate
          = 100 SPS, maximum Session Attempts = 100,000 and Media
          Streams Per Session = 0.
      3.  Start Tester to initiate SIP Session establishment with the
          DUT.
      4.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      5.  If a Session Attempt Failure is recorded then reduce the
          Session Attempt Rate configured on the Tester by 50%.
      6.  If no Session Attempt Failure is recorded then increase the
          Session Attempt Rate configured on the Tester by 50%.
      7.  Repeat steps 3 through 6 until the Session Establishment Rate
          is obtained and recorded.
   Expected Results:  Session Establishment Rate results obtained with
      TLS Encrypted SIP may be lower than those obtained with plaintext
      SIP.

6.7.

6.8.  Session Establishment Rate with IPsec Encrypted SIP

   Objective:
      To benchmark the Session Establishment Rate of the DUT/SUT with
      zero failures when using IPsec Encryoted SIP.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP TCP, enable IPSec, Session Attempt
          Rate = 100 SPS, maximum Session Attempts = 100,000 and Media
          Streams Per Session = 0.
      3.  Start Tester to initiate SIP Session establishment with the
          DUT.

      4.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      5.  If a Session Attempt Failure is recorded then reduce the
          Session Attempt Rate configured on the Tester by 50%.
      6.  If no Session Attempt Failure is recorded then increase the
          Session Attempt Rate configured on the Tester by 50%.
      7.  Repeat steps 3 through 6 until the Session Establishment Rate
          is obtained and recorded.
   Expected Results:  Session Establishment Rate results obtained with
      IPSec Encrypted SIP may be lower than those obtained with
      plaintext SIP.

6.8.

6.9.  Session Establishment Rate with SIP Flooding

   Objective:
      To benchmark the Session Establishment Rate of the SUT with zero
      failures when SIP Flooding is occurring.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          the SUT as shown in Figure 2.
      2.  Configure Tester for SIP UDP with an Session Attempt Rate =
          100 SPS, maximum Session Attempts = 100,000, Associated Media
          Streams Per Session = 0, and SIP INVITE Message Flood = 500
          per second.
      3.  Start Tester to initiate SIP Session establishment with the
          SUT and SIP Flood targetted at the Server.
      4.  At the Tester measure Session Attempt Failures, total
          Established Sessions, and Packet Loss [RFC2544] of the media.
      5.  If a Session Attempt Failure or Packet Loss is recorded then
          reduce the Session Attempt Rate configured on the Tester by
          50%.
      6.  If no Session Attempt Failure or Packet Loss is recorded then
          increase the Session Attempt Rate configured on the Tester by
          50%.
      7.  Repeat steps 3 through 6 until the Session Establishment Rate
          is obtained and recorded.
      8.  Repeat steps 1 through 7 with SIP INVITE Message Flood = 1000
          per second.
   Expected Results:  Session Establishment Rate results obtained with
      SIP Flooding may be degraded.

6.9.

6.10.  Maximum Registration Rate

   Objective:

      To benchmark the maximum registration rate of the DUT/SUT with
      zero failures.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an attempted a Registration Rate = 100 SPS and
          maximum registrations attempted = 100,000.
      3.  Set the registration timeout value to at least 3600 seconds.
      4.  At the Tester measure failed registration attempts, total
          registrations and packet loss.
      5.  If a Failed Registration Attempt or Packet Loss is recorded
          then reduce the Attempted Registration Rate configured on the
          Tester by 50%.
      6.  If no Failed Registration or Packet Loss is recorded then
          increase the Attempted Registration Rate configured on the
          Tester by 50%.
      7.  Repeat steps 5 and 6 until the all registrations have
          succeeded.  This number is obtained and recorded.
   Expected Results:

6.10.

6.11.  Maximum Re-Registration Rate

   Objective:
      To benchmark the maximum re-registration rate of the DUT/SUT with
      zero failures.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Execute test detailed in Section 6.9 6.10 to register the
          endpoints with the registrar.  The rest of the steps below
          MUST be performed at least 5 minutes after, but no more than
          15 minutes after the test performed in Section 6.9. 6.10.
      3.  Configure Tester for SIP UDP with an attempted Registration Rate = 100 SPS
          and maximum registrations attempted = 100,000.
      4.  Configure Tester to re-register the same address-of-records
          that were registered in Section 6.9. 6.10.
      5.  At the Tester measure failed registration attempts, total
          registrations and packet loss.
      6.  If a Failed Registration Attempt or Packet Loss is recorded
          then reduce the Attempted Registration Rate configured on the
          Tester by 50%.
      7.  If no Failed Registration or Packet Loss is recorded then
          increase the Attempted Registration Rate configured on the
          Tester by 50%.
      8.  Repeat steps 6 and 7 until the all re-registrations have
          succeeded.  This number is obtained and recorded.

   Expected Results:  The rate should be at least equal to but not more
      than the result of Section 6.9.

6.11. 6.10.

6.12.  Maximum IM Rate

   Objective:
      To benchmark the maximum IM rate of the SUT with zero failures.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an Attempted IM Rate = 100 SPS, Maximum IM
          Attempted = 100,000.
      3.  At the Tester measure Failed IM Attempts, Total IM and Packet
          Loss.
      4.  If a Failed IM Attempt or Packet Loss is recorded then reduce
          the Attempted IM Rate configured on the Tester by 50%.
      5.  If no Failed IM or Packet Loss is recorded then increase the
          Attempted IM Rate configured on the Tester by 50%.
      6.  Repeat steps 3 through 6 until the Maximum IM Rate is obtained
          and recorded.
   Expected Results:

6.12.

6.13.  Session Capacity without Media

   Objective:
      To benchmark the Session Capacity of the SUT without Associated
      Media.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with an a Session Attempt Rate = Session
          Establishment Rate, maximum Session Attempts = 10,000 and
          Media Streams Per Session = 0.
      3.  Start Tester to initiate SIP Session establishment with the
          DUT.
      4.  Measure Session Attempt Failures, total Established Sessions,
          and Packet Loss [RFC2544] at the Tester.
      5.  If a Session Attempt Failure or Packet Loss is recorded then
          reduce the maximum Session Attempts configured on the Tester
          by 5,000.
      6.  If no Session Attempt Failure or Packet Loss is recorded then
          increase the maximum Session Attempts configured on the Tester
          by 10,000.
      7.  Repeat steps 3 through 6 until the Session Capacity is
          obtained and recorded.
      8.  Repeat steps 1 through 7 for multimedia in which media streams
          per session = 2.

   Expected Results:  This is the scenario to obtain the maximum Session
      Capacity of the DUT/SUT.

6.13.

6.14.  Session Capacity with Media

   Objective:
      To benchmark the session capacity of the DUT/SUT with Associated
      Media.
   Procedure:
      1.  Configure the DUT in the test topology shown in Figure 1 or
          SUT as shown in Figures 2 or 3.
      2.  Configure Tester for SIP UDP with a Session Attempt Rate = 100 SPS, Session
          Duration = 30 sec, maximum Session Attempts = 100,000 and
          Media Streams Per Session = 1.
             NOTE: The total offered load to the DUT/SUT SHOULD be equal
             to the Throughput of the DUT/SUT as defined in [RFC2544].
             The offered load to the DUT/SUT for each media stream
             SHOULD be equal to
             Throughput/Maximum Session Attemps.
      3.  Start Tester to initiate SIP Session establishment with the
          SUT and transmit media through the SUT to a destination other
          than the server.
      4.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      5.  If a Session Attempt Failure is recorded then reduce the
          maximum Session Attempts configured on the Tester by 5,000.
      6.  If no Session Attempt Failure is recorded then increase the
          maximum Session Attempts configured on the Tester by 10,000.
      7.  Repeat steps 3 through 6 until the Session Capacity is
          obtained and recorded.
   Expected Results:  Session Capacity results obtained with Associated
      Media with any number of media streams per SIP session will be
      identical to the Session Capacity results obtained without media.

6.14.

6.15.  Session Capacity with Media and a Media Relay/NAT and/or Firewall

   Objective:
      To benchmark the Session Establishment Rate of the SUT with
      Associated Media.
   Procedure:
      1.  Configure the SUT in the test topology shown in Figure 4, 5 or
          6. 3.
      2.  Configure Tester for SIP UDP with a Session Attempt Rate = 100 SPS, Session
          Duration = 30 sec, maximum Session Attempts = 100,000 and
          Media Streams Per Session = 1.
             NOTE: The offered load for each media stream SHOULD be as
             in Equation 1.

      3.  Start Tester to initiate SIP Session establishment with the
          SUT and transmit media through the SUT to a destination other
          than the server.
      4.  Measure Session Attempt Failures and total Established
          Sessions at the Tester.
      5.  If a Session Attempt Failure is recorded then reduce the
          maximum Session Attempts configured on the Tester by 5,000.
      6.  If no Session Attempt Failure is recorded then increase the
          maximum Session Attempts configured on the Tester by 10,000.
      7.  Repeat steps 3 through 6 until the Session Capacity is
          obtained and recorded.
   Expected Results:  Session Capacity results obtained with Associated
      Media with any number of media streams per SIP session may be
      lower than the Session Capacity without Media result if the Media
      Relay, NAT or Firewall is sharing a platform with the server.

7.  IANA Considerations

   This document does not requires any IANA considerations.

8.  Security Considerations

   Documents of this type do not directly affect the security of
   Internet or corporate networks as long as benchmarking is not
   performed on devices or systems connected to production networks.
   Security threats and how to counter these in SIP and the media layer
   is discussed in RFC3261, RFC3550, and RFC3711 and various other
   drafts.  This document attempts to formalize a set of common
   methodology for benchmarking performance of SIP devices in a lab
   environment.

9.  Acknowledgments

   The authors would like to thank Keith Drage and Daryl Malas for their
   contributions to this document.

10.  References

10.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

   [I-D.sip-bench-term]
              Poretsky, S., Gurbani, V., and C. Davids, "SIP Performance
              Benchmarking Terminology",
              draft-ietf-bmwg-sip-bench-term-01
              draft-ietf-bmwg-sip-bench-term-02 (work in progress),
              January
              July 2010.

10.2.  Informative References

   [RFC3261]  Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
              A., Peterson, J., Sparks, R., Handley, M., and E.
              Schooler, "SIP: Session Initiation Protocol", RFC 3261,
              June 2002.

Authors' Addresses

   Scott Poretsky
   Allot Communications
   300 TradeCenter, Suite 4680
   Woburn, MA  08101
   USA

   Phone: +1 508 309 2179
   Email: sporetsky@allot.com

   Vijay K. Gurbani
   Bell Laboratories, Alcatel-Lucent
   1960 Lucent Lane
   Rm 9C-533
   Naperville, IL  60566
   USA

   Phone: +1 630 224 0216
   Email: vkg@bell-labs.com

   Carol Davids
   Illinois Institute of Technology
   201 East Loop Road
   Wheaton, IL  60187
   USA

   Phone: +1 630 682 6024
   Email: davids@iit.edu