draft-ietf-bmwg-sip-bench-meth-06.txt   draft-ietf-bmwg-sip-bench-meth-07.txt 
Benchmarking Methodology Working C. Davids Benchmarking Methodology Working Group C. Davids
Group Illinois Institute of Technology Internet-Draft Illinois Institute of Technology
Internet-Draft V. Gurbani Intended status: Informational V. Gurbani
Expires: May 12, 2013 Bell Laboratories, Alcatel-Lucent Expires: July 10, 2013 Bell Laboratories,
Alcatel-Lucent
S. Poretsky S. Poretsky
Allot Communications Allot Communications
November 8, 2012 January 6, 2013
Methodology for Benchmarking SIP Networking Devices Methodology for Benchmarking SIP Networking Devices
draft-ietf-bmwg-sip-bench-meth-06 draft-ietf-bmwg-sip-bench-meth-07
Abstract Abstract
This document describes the methodology for benchmarking Session This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document. The methodology and terminology benchmarking terminology document. The methodology and terminology
are to be used for benchmarking signaling plane performance with are to be used for benchmarking signaling plane performance with
varying signaling and media load. Both scale and establishment rate varying signaling and media load. Both scale and establishment rate
are measured by signaling plane performance. The SIP Devices to be are measured by signaling plane performance. The SIP Devices to be
benchmarked may be a single device under test (DUT) or a system under benchmarked may be a single device under test (DUT) or a system under
skipping to change at page 1, line 42 skipping to change at page 1, line 43
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 12, 2013. This Internet-Draft will expire on July 10, 2013.
Copyright Notice Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
skipping to change at page 3, line 35 skipping to change at page 3, line 35
6.1. Baseline Session Establishment Rate of the test bed . . . 10 6.1. Baseline Session Establishment Rate of the test bed . . . 10
6.2. Session Establishment Rate without media . . . . . . . . . 11 6.2. Session Establishment Rate without media . . . . . . . . . 11
6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11 6.3. Session Establishment Rate with Media on DUT/SUT . . . . . 11
6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12 6.4. Session Establishment Rate with Media not on DUT/SUT . . . 12
6.5. Session Establishment Rate with Loop Detection Enabled . . 13 6.5. Session Establishment Rate with Loop Detection Enabled . . 13
6.6. Session Establishment Rate with Forking . . . . . . . . . 13 6.6. Session Establishment Rate with Forking . . . . . . . . . 13
6.7. Session Establishment Rate with Forking and Loop 6.7. Session Establishment Rate with Forking and Loop
Detection . . . . . . . . . . . . . . . . . . . . . . . . 14 Detection . . . . . . . . . . . . . . . . . . . . . . . . 14
6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14 6.8. Session Establishment Rate with TLS Encrypted SIP . . . . 14
6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15 6.9. Session Establishment Rate with IPsec Encrypted SIP . . . 15
6.10. Session Establishment Rate with SIP Flooding . . . . . . . 15 6.10. Session Establishment Rate with SIP Flooding . . . . . . . 16
6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16 6.11. Maximum Registration Rate . . . . . . . . . . . . . . . . 16
6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16 6.12. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 16
6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17 6.13. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 17
6.14. Session Capacity without Media . . . . . . . . . . . . . . 17 6.14. Session Capacity without Media . . . . . . . . . . . . . . 17
6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18 6.15. Session Capacity with Media . . . . . . . . . . . . . . . 18
6.16. Session Capacity with Media and a Media Relay/NAT 6.16. Session Capacity with Media and a Media Relay/NAT
and/or Firewall . . . . . . . . . . . . . . . . . . . . . 18 and/or Firewall . . . . . . . . . . . . . . . . . . . . . 19
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
8. Security Considerations . . . . . . . . . . . . . . . . . . . 19 8. Security Considerations . . . . . . . . . . . . . . . . . . . 19
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.1. Normative References . . . . . . . . . . . . . . . . . . . 19 10.1. Normative References . . . . . . . . . . . . . . . . . . . 20
10.2. Informative References . . . . . . . . . . . . . . . . . . 20 10.2. Informative References . . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20
1. Terminology 1. Terminology
In this document, the key words "MUST", "MUST NOT", "REQUIRED", In this document, the key words "MUST", "MUST NOT", "REQUIRED",
"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as
described in BCP 14, conforming to [RFC2119] and indicate requirement described in BCP 14, conforming to [RFC2119] and indicate requirement
levels for compliant implementations. levels for compliant implementations.
skipping to change at page 8, line 37 skipping to change at page 8, line 37
do { do {
send_traffic(s, m, t) ; Send s request/sec with m send_traffic(s, m, t) ; Send s request/sec with m
; characteristics until t requests have ; characteristics until t requests have
; been sent ; been sent
if (all requests succeeded) { if (all requests succeeded) {
s' := s ; save candidate value of metric s' := s ; save candidate value of metric
if ( c == 0 ) { if ( c == 0 ) {
s := s + (0.5 * s) s := s + (0.5 * s)
}else if ((c == 1) && (s''-s')) > 2*G ) { }else if ((c == 1) && (s??-s?)) > 2*G ) {
s := s + ( 0.5 * (s'' - s ); s := s + ( 0.5 * (s?? ? s );
}else if ((c == 1) && ((s''-s') <= 2*G ) { }else if ((c == 1) && ((s??-s?) <= 2*G ) {
f := true; f := true;
} }
else if (one or more requests fail) { else if (one or more requests fail) {
c := 1 ; we have found an upper bound for the metric c := 1 ; we have found an upper bound for the metric
s'' := s ; save new upper bound s?? := s ; save new upper bound
s := s - (0.5 * (s - s')) s := s - (0.5 * (s ? s?))
} }
} while (f == false) } while (f == false)
end proc end proc
5. Reporting Format 5. Reporting Format
5.1. Test Setup Report 5.1. Test Setup Report
SIP Transport Protocol = ___________________________ SIP Transport Protocol = ___________________________
(valid values: TCP|UDP|TLS|SCTP|specify-other) (valid values: TCP|UDP|TLS|SCTP|specify-other)
skipping to change at page 10, line 11 skipping to change at page 10, line 11
the first attempt. the first attempt.
Note 3: When the Authentication Option is "on" the DUT/SUT uses two Note 3: When the Authentication Option is "on" the DUT/SUT uses two
transactions instead of one when it is establishing a session or transactions instead of one when it is establishing a session or
accomplishing a registration. The first transaction ends with the accomplishing a registration. The first transaction ends with the
401 or 407. The second ends with the 200 OK or another failure 401 or 407. The second ends with the 200 OK or another failure
message. The Test Organization interested in knowing how many times message. The Test Organization interested in knowing how many times
the EA was intended to send a REGISTER as distinct from how many the EA was intended to send a REGISTER as distinct from how many
times the EA wound up actually sending a REGISTER may wish to record times the EA wound up actually sending a REGISTER may wish to record
the following data as well: Number of responses of the following the following data as well: Number of responses of the following
type: 401: _____________ (if authentication turned on; N/A otherwise) type:
401: _____________ (if authentication turned on; N/A otherwise)
407: _____________ (if authentication turned on; N/A otherwise) 407: _____________ (if authentication turned on; N/A otherwise)
5.2. Device Benchmarks for IS 5.2. Device Benchmarks for IS
Registration Rate = _______________________________ Registration Rate = _______________________________
(registrations per second) (registrations per second)
Re-registration Rate = ____________________________ Re-registration Rate = ____________________________
(registrations per second) (registrations per second)
Session Capacity = _________________________________ Session Capacity = _________________________________
(sessions) (sessions)
skipping to change at page 18, line 32 skipping to change at page 19, line 4
or as a proxy. If a SUT is being benchmarked, configure the or as a proxy. If a SUT is being benchmarked, configure the
SUT as shown in Figure 9 of [I-D.sip-bench-term]. SUT as shown in Figure 9 of [I-D.sip-bench-term].
2. Set the media streams per session to 1. 2. Set the media streams per session to 1.
3. Set the Session Duration to be a value greater than T. 3. Set the Session Duration to be a value greater than T.
4. Execute benchmarking algorithm as defined in Section 4.9 to 4. Execute benchmarking algorithm as defined in Section 4.9 to
get the baseline session establishment rate. This rate MUST get the baseline session establishment rate. This rate MUST
be recorded using any pertinent parameters as shown in the be recorded using any pertinent parameters as shown in the
reporting format of Section 5.1. reporting format of Section 5.1.
5. The Session Capacity is the product of T and the Session 5. The Session Capacity is the product of T and the Session
Establishment Rate. Establishment Rate.
Expected Results: Session Capacity results obtained with Associated Expected Results: Session Capacity results obtained with Associated
Media with any number of media streams per SIP session will be Media with any number of media streams per SIP session will be
identical to the Session Capacity results obtained without media. identical to the Session Capacity results obtained without media.
6.16. Session Capacity with Media and a Media Relay/NAT and/or Firewall 6.16. Session Capacity with Media and a Media Relay/NAT
and/or Firewall
Objective: Objective:
To benchmark the Session Establishment Rate of the SUT with To benchmark the Session Establishment Rate of the SUT with
Associated Media. Associated Media.
Procedure: Procedure:
1. Configure the SUT as shown in Figure 7 or Figure 10 in 1. Configure the SUT as shown in Figure 7 or Figure 10 in
[I-D.sip-bench-term]. [I-D.sip-bench-term].
2. Set media streams per session to 1. 2. Set media streams per session to 1.
3. Execute benchmarking algorithm as defined in Section 4.9 to 3. Execute benchmarking algorithm as defined in Section 4.9 to
get the session establishment rate with media. This rate MUST get the session establishment rate with media. This rate MUST
skipping to change at page 19, line 44 skipping to change at page 20, line 18
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[I-D.sip-bench-term] [I-D.sip-bench-term]
Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance Davids, C., Gurbani, V., and S. Poretsky, "SIP Performance
Benchmarking Terminology", Benchmarking Terminology",
draft-ietf-bmwg-sip-bench-term-04 (work in progress), draft-ietf-bmwg-sip-bench-term-07 (work in progress),
March 2012. March 2012.
10.2. Informative References 10.2. Informative References
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261, Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002. June 2002.
Authors' Addresses Authors' Addresses
 End of changes. 15 change blocks. 
20 lines changed or deleted 25 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/