draft-ietf-bmwg-ngfw-performance-10.txt   draft-ietf-bmwg-ngfw-performance-11.txt 
Benchmarking Methodology Working Group B. Balarajah Benchmarking Methodology Working Group B. Balarajah
Internet-Draft Internet-Draft
Obsoletes: 3511 (if approved) C. Rossenhoevel Obsoletes: 3511 (if approved) C. Rossenhoevel
Intended status: Informational EANTC AG Intended status: Informational EANTC AG
Expires: 30 March 2022 B. Monkman Expires: 23 April 2022 B. Monkman
NetSecOPEN NetSecOPEN
September 2021 October 2021
Benchmarking Methodology for Network Security Device Performance Benchmarking Methodology for Network Security Device Performance
draft-ietf-bmwg-ngfw-performance-10 draft-ietf-bmwg-ngfw-performance-11
Abstract Abstract
This document provides benchmarking terminology and methodology for This document provides benchmarking terminology and methodology for
next-generation network security devices including next-generation next-generation network security devices including next-generation
firewalls (NGFW), next-generation intrusion prevention systems firewalls (NGFW), next-generation intrusion prevention systems
(NGIPS), and unified threat management (UTM) implementations. This (NGIPS), and unified threat management (UTM) implementations. This
document aims to improve the applicability, reproducibility, and document aims to improve the applicability, reproducibility, and
transparency of benchmarks and to align the test methodology with transparency of benchmarks and to align the test methodology with
today's increasingly complex layer 7 security centric network today's increasingly complex layer 7 security centric network
skipping to change at page 1, line 42 skipping to change at page 1, line 42
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on 5 March 2022. This Internet-Draft will expire on 4 April 2022.
Copyright Notice Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/ Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document. license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights Please review these documents carefully, as they describe your rights
skipping to change at page 4, line 32 skipping to change at page 4, line 32
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119], [RFC8174] when, and only when, they appear in all 14 [RFC2119], [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
3. Scope 3. Scope
This document provides testing terminology and testing methodology This document provides testing terminology and testing methodology
for modern and next-generation network security devices that are for modern and next-generation network security devices that are
configured in Active ("Inline") mode. It covers the validation of configured in Active ("Inline", see Figure 1 and Figure 2) mode. It
security effectiveness configurations of network security devices, covers the validation of security effectiveness configurations of
followed by performance benchmark testing. This document focuses on network security devices, followed by performance benchmark testing.
advanced, realistic, and reproducible testing methods. Additionally, This document focuses on advanced, realistic, and reproducible
it describes testbed environments, test tool requirements, and test testing methods. Additionally, it describes testbed environments,
result formats. test tool requirements, and test result formats.
4. Test Setup 4. Test Setup
Test setup defined in this document applies to all benchmarking tests Test setup defined in this document applies to all benchmarking tests
described in Section 7. The test setup MUST be contained within an described in Section 7. The test setup MUST be contained within an
Isolated Test Environment (see Section 3 of [RFC6815]). Isolated Test Environment (see Section 3 of [RFC6815]).
4.1. Testbed Configuration 4.1. Testbed Configuration
Testbed configuration MUST ensure that any performance implications Testbed configuration MUST ensure that any performance implications
skipping to change at page 7, line 5 skipping to change at page 7, line 5
Table 1 and Table 2 below describe the RECOMMENDED and OPTIONAL sets Table 1 and Table 2 below describe the RECOMMENDED and OPTIONAL sets
of network security feature list for NGFW and NGIPS respectively. of network security feature list for NGFW and NGIPS respectively.
The selected security features SHOULD be consistently enabled on the The selected security features SHOULD be consistently enabled on the
DUT/SUT for all benchmarking tests described in Section 7. DUT/SUT for all benchmarking tests described in Section 7.
To improve repeatability, a summary of the DUT/SUT configuration To improve repeatability, a summary of the DUT/SUT configuration
including a description of all enabled DUT/SUT features MUST be including a description of all enabled DUT/SUT features MUST be
published with the benchmarking results. published with the benchmarking results.
+------------------------+ +============================+=============+==========+
| NGFW | | DUT/SUT (NGFW) Features | RECOMMENDED | OPTIONAL |
+--------------- +-------------+----------+ +============================+=============+==========+
| | | | | SSL Inspection | x | |
|DUT/SUT Features| RECOMMENDED | OPTIONAL | +----------------------------+-------------+----------+
| | | | | IDS/IPS | x | |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|SSL Inspection | x | | | Anti-Spyware | x | |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|IDS/IPS | x | | | Anti-Virus | x | |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|Anti-Spyware | x | | | Anti-Botnet | x | |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|Anti-Virus | x | | | Web Filtering | | x |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|Anti-Botnet | x | | | Data Loss Protection (DLP) | | x |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|Web Filtering | | x | | DDoS | | x |
+----------------+-------------+----------+ +----------------------------+-------------+----------+
|Data Loss | | | | Certificate Validation | | x |
|Protection (DLP)| | x | +----------------------------+-------------+----------+
+----------------+-------------+----------+ | Logging and Reporting | x | |
|DDoS | | x | +----------------------------+-------------+----------+
+----------------+-------------+----------+ | Application Identification | x | |
|Certificate | | x | +----------------------------+-------------+----------+
|Validation | | |
+----------------+-------------+----------+
|Logging and | x | |
|Reporting | | |
+----------------+-------------+----------+
|Application | x | |
|Identification | | |
+----------------+-------------+----------+
Figure 3: Table 1: NGFW Security Features Table 1: NGFW Security Features
+------------------------+
| NGIPS |
+----------------+-------------+----------+
| | | |
|DUT/SUT Features| RECOMMENDED | OPTIONAL |
| | | |
+----------------+-------------+----------+
|SSL Inspection | x | |
+----------------+-------------+----------+
|Anti-Malware | x | |
+----------------+-------------+----------+
|Anti-Spyware | x | |
+----------------+-------------+----------+
|Anti-Botnet | x | |
+----------------+-------------+----------+
|Logging and | x | |
|Reporting | | |
+----------------+-------------+----------+
|Application | x | |
|Identification | | |
+----------------+-------------+----------+
|Deep Packet | x | |
|Inspection | | |
+----------------+-------------+----------+
|Anti-Evasion | x | |
+----------------+-------------+----------+
Figure 4: Table 2: NGIPS Security Features +============================+=============+==========+
| DUT/SUT (NGIPS) Features | RECOMMENDED | OPTIONAL |
+============================+=============+==========+
| SSL Inspection | x | |
+----------------------------+-------------+----------+
| Anti-Malware | x | |
+----------------------------+-------------+----------+
| Anti-Spyware | x | |
+----------------------------+-------------+----------+
| Anti-Botnet | x | |
+----------------------------+-------------+----------+
| Logging and Reporting | x | |
+----------------------------+-------------+----------+
| Application Identification | x | |
+----------------------------+-------------+----------+
| Deep Packet Inspection | x | |
+----------------------------+-------------+----------+
| Anti-Evasion | x | |
+----------------------------+-------------+----------+
Table 2: NGIPS Security Features
The following table provides a brief description of the security The following table provides a brief description of the security
features. features.
+------------------+------------------------------------------------+ +================+================================================+
| DUT/SUT Features | Description | | DUT/SUT | Description |
+------------------+------------------------------------------------+ | Features | |
| SSL Inspection | DUT/SUT intercepts and decrypts inbound HTTPS | +================+================================================+
| | traffic between servers and clients. Once the | | SSL Inspection | DUT/SUT intercepts and decrypts inbound HTTPS |
| | content inspection has been completed, DUT/SUT | | | traffic between servers and clients. Once the |
| | encrypts the HTTPS traffic with ciphers | | | content inspection has been completed, DUT/SUT |
| | and keys used by the clients and servers. | | | encrypts the HTTPS traffic with ciphers and |
+------------------+------------------------------------------------+ | | keys used by the clients and servers. |
| IDS/IPS | DUT/SUT detects and blocks exploits | +----------------+------------------------------------------------+
| | targeting known and unknown vulnerabilities | | IDS/IPS | DUT/SUT detects and blocks exploits targeting |
| | across the monitored network. | | | known and unknown vulnerabilities across the |
+------------------+------------------------------------------------+ | | monitored network. |
| Anti-Malware | DUT/SUT detects and prevents the transmission | +----------------+------------------------------------------------+
| | of malicious executable code and any associated| | Anti-Malware | DUT/SUT detects and prevents the transmission |
| | communications across the monitored network. | | | of malicious executable code and any |
| | This includes data exfiltration as well as | | | associated communications across the monitored |
| | command and control channels. | | | network. This includes data exfiltration as |
+------------------+------------------------------------------------+ | | well as command and control channels. |
| Anti-Spyware | Anti-Spyware is a subcategory of Anti Malware. | +----------------+------------------------------------------------+
| | Spyware transmits information without the | | Anti-Spyware | Anti-Spyware is a subcategory of Anti Malware. |
| | user's knowledge or permission. DUT/SUT detects| | | Spyware transmits information without the |
| | and block initial infection or transmission of | | | user's knowledge or permission. DUT/SUT |
| | data. | | | detects and block initial infection or |
+------------------+------------------------------------------------+ | | transmission of data. |
| Anti-Botnet | DUT/SUT detects traffic to or from botnets. | +----------------+------------------------------------------------+
+------------------+------------------------------------------------+ | Anti-Botnet | DUT/SUT detects traffic to or from botnets. |
| Anti-Evasion | DUT/SUT detects and mitigates attacks that have| +----------------+------------------------------------------------+
| | been obfuscated in some manner. | | Anti-Evasion | DUT/SUT detects and mitigates attacks that |
+------------------+------------------------------------------------+ | | have been obfuscated in some manner. |
| Web Filtering | DUT/SUT detects and blocks malicious website | +----------------+------------------------------------------------+
| | including defined classifications of website | | Web Filtering | DUT/SUT detects and blocks malicious website |
| | across the monitored network. | | | including defined classifications of website |
+------------------+------------------------------------------------+ | | across the monitored network. |
| DLP | DUT/SUT detects and prevents data breaches and | +----------------+------------------------------------------------+
| | data exfiltration, or it detects and blocks the| | DLP | DUT/SUT detects and prevents data breaches and |
| | transmission of sensitive data across the | | | data exfiltration, or it detects and blocks |
| | monitored network. | | | the transmission of sensitive data across the |
+------------------+------------------------------------------------+ | | monitored network. |
| Certificate | DUT/SUT validates certificates used in | +----------------+------------------------------------------------+
| Validation | encrypted communications across the monitored | | Certificate | DUT/SUT validates certificates used in |
| | network. | | Validation | encrypted communications across the monitored |
+------------------+------------------------------------------------+ | | network. |
| Logging and | DUT/SUT logs and reports all traffic at the | +----------------+------------------------------------------------+
| Reporting | flow level across the monitored network. | | Logging and | DUT/SUT logs and reports all traffic at the |
+------------------+------------------------------------------------+ | Reporting | flow level across the monitored network. |
| Application | DUT/SUT detects known applications as defined | +----------------+------------------------------------------------+
| Identification | within the traffic mix selected across | | Application | DUT/SUT detects known applications as defined |
| | the monitored network. | | Identification | within the traffic mix selected across the |
+------------------+------------------------------------------------+ | | monitored network. |
+----------------+------------------------------------------------+
Figure 5: Table 3: Security Feature Description Table 3: Security Feature Description
Below is a summary of the DUT/SUT configuration: Below is a summary of the DUT/SUT configuration:
* DUT/SUT MUST be configured in "inline" mode. * DUT/SUT MUST be configured in "inline" mode.
* "Fail-Open" behavior MUST be disabled. * "Fail-Open" behavior MUST be disabled.
* All RECOMMENDED security features are enabled. * All RECOMMENDED security features are enabled.
* Logging SHOULD be enabled. DUT/SUT SHOULD log all traffic at the * Logging SHOULD be enabled. DUT/SUT SHOULD log all traffic at the
skipping to change at page 10, line 17 skipping to change at page 10, line 13
application from the defined traffic mix. application from the defined traffic mix.
In addition, a realistic number of access control rules (ACL) SHOULD In addition, a realistic number of access control rules (ACL) SHOULD
be configured on the DUT/SUT where ACLs are configurable and be configured on the DUT/SUT where ACLs are configurable and
reasonable based on the deployment scenario. This document reasonable based on the deployment scenario. This document
determines the number of access policy rules for four different determines the number of access policy rules for four different
classes of DUT/SUT: Extra Small (XS), Small (S), Medium (M), and classes of DUT/SUT: Extra Small (XS), Small (S), Medium (M), and
Large (L). A sample DUT/SUT classification is described in Large (L). A sample DUT/SUT classification is described in
Appendix B. Appendix B.
The Access Control Rules (ACL) defined in Table 4 MUST be configured The Access Control Rules (ACL) defined in Figure 3 MUST be configured
from top to bottom in the correct order as shown in the table. This from top to bottom in the correct order as shown in the table. This
is due to ACL types listed in specificity decreasing order, with is due to ACL types listed in specificity decreasing order, with
"block" first, followed by "allow", representing a typical ACL based "block" first, followed by "allow", representing a typical ACL based
security policy. The ACL entries SHOULD be configured with routable security policy. The ACL entries SHOULD be configured with routable
IP subnets by the DUT/SUT. (Note: There will be differences between IP subnets by the DUT/SUT. (Note: There will be differences between
how security vendors implement ACL decision making.) The configured how security vendors implement ACL decision making.) The configured
ACL MUST NOT block the security and measurement traffic used for the ACL MUST NOT block the security and measurement traffic used for the
benchmarking tests. benchmarking tests.
+---------------+ +---------------+
skipping to change at page 11, line 52 skipping to change at page 11, line 52
+-------------------------------------------------------------------+ +-------------------------------------------------------------------+
|IP layer |SRC IP | The rest of the | allow| >1| >1| >1| >1| |IP layer |SRC IP | The rest of the | allow| >1| >1| >1| >1|
| | | SRC IP subnet | | | | | | | | | SRC IP subnet | | | | | |
| | | range used in the | | | | | | | | | range used in the | | | | | |
| | | measurement | | | | | | | | | measurement | | | | | |
| | | traffic | | | | | | | | | traffic | | | | | |
| | | (one rule per | | | | | | | | | (one rule per | | | | | |
| | | subnet) | | | | | | | | | subnet) | | | | | |
+-----------+-----------+--------------------+------+---+---+---+---+ +-----------+-----------+--------------------+------+---+---+---+---+
Figure 6: Table 4: DUT/SUT Access List Figure 3: DUT/SUT Access List
Note: If half of the applications included in the measurement traffic Note: If half of the applications included in the measurement traffic
is less than 10, the missing number of ACL entries (dummy rules) can is less than 10, the missing number of ACL entries (dummy rules) can
be configured for any application traffic not included in the be configured for any application traffic not included in the
measurement traffic. measurement traffic.
4.2.1. Security Effectiveness Configuration 4.2.1. Security Effectiveness Configuration
The Security features (defined in table 1 and 2) of the DUT/SUT MUST The Security features (defined in Table 1 and Table 2) of the DUT/SUT
be configured effectively to detect, prevent, and report the defined MUST be configured effectively to detect, prevent, and report the
security vulnerability sets. This section defines the selection of defined security vulnerability sets. This section defines the
the security vulnerability sets from Common vulnerabilities and selection of the security vulnerability sets from Common
Exposures (CVE) list for the testing. The vulnerability set SHOULD vulnerabilities and Exposures (CVE) list for the testing. The
reflect a minimum of 500 CVEs from no older than 10 calendar years to vulnerability set SHOULD reflect a minimum of 500 CVEs from no older
the current year. These CVEs SHOULD be selected with a focus on in- than 10 calendar years to the current year. These CVEs SHOULD be
use software commonly found in business applications, with a Common selected with a focus on in-use software commonly found in business
vulnerability Scoring System (CVSS) Severity of High (7-10). applications, with a Common vulnerability Scoring System (CVSS)
Severity of High (7-10).
This document is primarily focused on performance benchmarking. This document is primarily focused on performance benchmarking.
However, it is RECOMMENDED to validate the security features However, it is RECOMMENDED to validate the security features
configuration of the DUT/SUT by evaluating the security effectiveness configuration of the DUT/SUT by evaluating the security effectiveness
as a prerequisite for performance benchmarking tests defined in the as a prerequisite for performance benchmarking tests defined in the
section 7. In case the benchmarking tests are performed without section 7. In case the benchmarking tests are performed without
evaluating security effectiveness, the test report MUST explain the evaluating security effectiveness, the test report MUST explain the
implications of this. The methodology for evaluating security implications of this. The methodology for evaluating security
effectiveness is defined in Appendix A. effectiveness is defined in Appendix A.
skipping to change at page 14, line 42 skipping to change at page 14, line 42
For HTTP traffic emulation, the emulated browser MUST negotiate HTTP For HTTP traffic emulation, the emulated browser MUST negotiate HTTP
version 1.1 or higher. Depending on test scenarios and chosen HTTP version 1.1 or higher. Depending on test scenarios and chosen HTTP
version, the emulated browser MAY open multiple TCP connections per version, the emulated browser MAY open multiple TCP connections per
Server endpoint IP at any time depending on how many sequential Server endpoint IP at any time depending on how many sequential
transactions need to be processed. For HTTP/2 or HTTP/3, the transactions need to be processed. For HTTP/2 or HTTP/3, the
emulated browser MAY open multiple concurrent streams per connection emulated browser MAY open multiple concurrent streams per connection
(multiplexing). If HTTP/3 is used the emulated browser MUST open (multiplexing). If HTTP/3 is used the emulated browser MUST open
Quick UDP Internet Connections (QUIC). HTTP settings such as number Quick UDP Internet Connections (QUIC). HTTP settings such as number
of connection per server IP, number of requests per connection, and of connection per server IP, number of requests per connection, and
number of streams per connection MUST be documented. This document number of streams per connection MUST be documented. This document
refers to [RFC8446] for HTTP/2 and [RFC9000]for QUIC. The emulated refers to [RFC8446] for HTTP/2 and [RFC9000] for QUIC. The emulated
browser SHOULD advertise a User-Agent header. The emulated browser browser SHOULD advertise a User-Agent header. The emulated browser
SHOULD enforce content length validation. Depending on test SHOULD enforce content length validation. Depending on test
scenarios and selected HTTP version, HTTP header compression MAY be scenarios and selected HTTP version, HTTP header compression MAY be
set to enable or disable. This setting (compression enabled or set to enable or disable. This setting (compression enabled or
disabled) MUST be documented in the report. disabled) MUST be documented in the report.
For encrypted traffic, the following attributes SHALL define the For encrypted traffic, the following attributes SHALL define the
negotiated encryption parameters. The test clients MUST use TLS negotiated encryption parameters. The test clients MUST use TLS
version 1.2 or higher. TLS record size MAY be optimized for the version 1.2 or higher. TLS record size MAY be optimized for the
HTTPS response object size up to a record size of 16 KByte. If HTTPS response object size up to a record size of 16 KByte. If
Server Name Indication (SNI) is required in the traffic mix profile, Server Name Indication (SNI) is required in the traffic mix profile,
the client endpoint MUST send TLS extension Server Name Indication the client endpoint MUST send TLS extension Server Name Indication
(SNI) information when opening a security tunnel. Each client (SNI) information when opening a security tunnel. Each client
connection MUST perform a full handshake with server certificate and connection MUST perform a full handshake with server certificate and
MUST NOT use session reuse or resumption. MUST NOT use session reuse or resumption.
The following TLS 1.2 supported ciphers and keys are RECOMMENDED to The following TLS 1.2 supported ciphers and keys are RECOMMENDED to
use for HTTPS based benchmarking tests defined in Section 7. use for HTTPS based benchmarking tests defined in Section 7.
1. ECDHE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash 1. ECDHE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash
Algorithm: ecdsa_secp256r1_sha256 and Supported group: sepc256r1) Algorithm: ecdsa_secp256r1_sha256 and Supported group: secp256r1)
2. ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash 2. ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash
Algorithm: rsa_pkcs1_sha256 and Supported group: sepc256r1) Algorithm: rsa_pkcs1_sha256 and Supported group: secp256r1)
3. ECDHE-ECDSA-AES256-GCM-SHA384 with Secp521 (Signature Hash 3. ECDHE-ECDSA-AES256-GCM-SHA384 with Secp521 (Signature Hash
Algorithm: ecdsa_secp384r1_sha384 and Supported group: sepc521r1) Algorithm: ecdsa_secp384r1_sha384 and Supported group: secp521r1)
4. ECDHE-RSA-AES256-GCM-SHA384 with RSA 4096 (Signature Hash 4. ECDHE-RSA-AES256-GCM-SHA384 with RSA 4096 (Signature Hash
Algorithm: rsa_pkcs1_sha384 and Supported group: secp256r1) Algorithm: rsa_pkcs1_sha384 and Supported group: secp256r1)
Note: The above ciphers and keys were those commonly used enterprise Note: The above ciphers and keys were those commonly used enterprise
grade encryption cipher suites for TLS 1.2. It is recognized that grade encryption cipher suites for TLS 1.2. It is recognized that
these will evolve over time. Individual certification bodies SHOULD these will evolve over time. Individual certification bodies SHOULD
use ciphers and keys that reflect evolving use cases. These choices use ciphers and keys that reflect evolving use cases. These choices
MUST be documented in the resulting test reports with detailed MUST be documented in the resulting test reports with detailed
information on the ciphers and keys used along with reasons for the information on the ciphers and keys used along with reasons for the
skipping to change at page 17, line 16 skipping to change at page 17, line 16
This section describes the traffic pattern between client and server This section describes the traffic pattern between client and server
endpoints. At the beginning of the test, the server endpoint endpoints. At the beginning of the test, the server endpoint
initializes and will be ready to accept connection states including initializes and will be ready to accept connection states including
initialization of the TCP stack as well as bound HTTP and HTTPS initialization of the TCP stack as well as bound HTTP and HTTPS
servers. When a client endpoint is needed, it will initialize and be servers. When a client endpoint is needed, it will initialize and be
given attributes such as a MAC and IP address. The behavior of the given attributes such as a MAC and IP address. The behavior of the
client is to sweep through the given server IP space, generating a client is to sweep through the given server IP space, generating a
recognizable service by the DUT. Sequential and pseudorandom sweep recognizable service by the DUT. Sequential and pseudorandom sweep
methods are acceptable. The method used MUST be stated in the final methods are acceptable. The method used MUST be stated in the final
report. Thus, a balanced\ mesh between client endpoints and server report. Thus, a balanced mesh between client endpoints and server
endpoints will be generated in a client IP and port to server IP and endpoints will be generated in a client IP and port to server IP and
port combination. Each client endpoint performs the same actions as port combination. Each client endpoint performs the same actions as
other endpoints, with the difference being the source IP of the other endpoints, with the difference being the source IP of the
client endpoint and the target server IP pool. The client MUST use client endpoint and the target server IP pool. The client MUST use
the server IP address or FQDN in the host header [RFC2616]. the server IP address or FQDN in the host header [RFC2616].
4.3.3.1. Description of Intra-Client Behavior 4.3.3.1. Description of Intra-Client Behavior
Client endpoints are independent of other clients that are Client endpoints are independent of other clients that are
concurrently executing. When a client endpoint initiates traffic, concurrently executing. When a client endpoint initiates traffic,
skipping to change at page 23, line 24 skipping to change at page 23, line 24
7. Benchmarking Tests 7. Benchmarking Tests
7.1. Throughput Performance with Application Traffic Mix 7.1. Throughput Performance with Application Traffic Mix
7.1.1. Objective 7.1.1. Objective
Using a relevant application traffic mix, determine the sustainable Using a relevant application traffic mix, determine the sustainable
inspected throughput supported by the DUT/SUT. inspected throughput supported by the DUT/SUT.
Based on customer use case, users can choose the relevant application Based on the test customer's specific use case, testers can choose
traffic mix for this test. The details about the traffic mix MUST be the relevant application traffic mix for this test. The details
documented in the report. At least the following traffic mix details about the traffic mix MUST be documented in the report. At least the
MUST be documented and reported together with the test results: following traffic mix details MUST be documented and reported
together with the test results:
Name of applications and layer 7 protocols Name of applications and layer 7 protocols
Percentage of emulated traffic for each application and layer 7 Percentage of emulated traffic for each application and layer 7
protocol protocol
Percentage of encrypted traffic and used cipher suites and keys Percentage of encrypted traffic and used cipher suites and keys
(The RECOMMENDED ciphers and keys are defined in Section 4.3.1.3.) (The RECOMMENDED ciphers and keys are defined in Section 4.3.1.3.)
Used object sizes for each application and layer 7 protocols Used object sizes for each application and layer 7 protocols
skipping to change at page 31, line 13 skipping to change at page 31, line 13
specific deployment scenario specific deployment scenario
Initial throughput: 10% of "Target inspected throughput" Note: Initial throughput: 10% of "Target inspected throughput" Note:
Initial throughput is not a KPI to report. This value is configured Initial throughput is not a KPI to report. This value is configured
on the traffic generator and used to perform Step 1: "Test on the traffic generator and used to perform Step 1: "Test
Initialization and Qualification" described under Section 7.3.4. Initialization and Qualification" described under Section 7.3.4.
Number of HTTP response object requests (transactions) per Number of HTTP response object requests (transactions) per
connection: 10 connection: 10
RECOMMENDED HTTP response object size: 1, 16, 64, 256 KByte, and RECOMMENDED HTTP response object size: 1, 16, 64, 256 KByte, and
mixed objects defined in table 5. mixed objects defined in Table 4.
+---------------------+---------------------+ +=====================+============================+
| Object size (KByte) | Number of requests/ | | Object size (KByte) | Number of requests/ Weight |
| | Weight | +=====================+============================+
+---------------------+---------------------+ | 0.2 | 1 |
| 0.2 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 6 | 1 |
| 6 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 8 | 1 |
| 8 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 9 | 1 |
| 9 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 10 | 1 |
| 10 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 25 | 1 |
| 25 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 26 | 1 |
| 26 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 35 | 1 |
| 35 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 59 | 1 |
| 59 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+ | 347 | 1 |
| 347 | 1 | +---------------------+----------------------------+
+---------------------+---------------------+
Figure 7: Table 5: Mixed Objects Table 4: Mixed Objects
7.3.3.3. Test Results Validation Criteria 7.3.3.3. Test Results Validation Criteria
The following criteria are the test results validation criteria. The The following criteria are the test results validation criteria. The
test results validation criteria MUST be monitored during the whole test results validation criteria MUST be monitored during the whole
sustain phase of the traffic load profile. sustain phase of the traffic load profile.
a. Number of failed application transactions (receiving any HTTP a. Number of failed application transactions (receiving any HTTP
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions. of 100,000 transactions) of attempt transactions.
skipping to change at page 44, line 27 skipping to change at page 44, line 27
Initial throughput is not a KPI to report. This value is configured Initial throughput is not a KPI to report. This value is configured
on the traffic generator and used to perform the Step1: "Test on the traffic generator and used to perform the Step1: "Test
Initialization and Qualification" described under the Section 7.7.4. Initialization and Qualification" described under the Section 7.7.4.
Number of HTTPS response object requests (transactions) per Number of HTTPS response object requests (transactions) per
connection: 10 connection: 10
RECOMMENDED ciphers and keys defined in Section 4.3.1.3 RECOMMENDED ciphers and keys defined in Section 4.3.1.3
RECOMMENDED HTTPS response object size: 1, 16, 64, 256 KByte, and RECOMMENDED HTTPS response object size: 1, 16, 64, 256 KByte, and
mixed objects defined in Table 5 under Section 7.3.3.2. mixed objects defined in Table 4 under Section 7.3.3.2.
7.7.3.3. Test Results Validation Criteria 7.7.3.3. Test Results Validation Criteria
The following criteria are the test results validation criteria. The The following criteria are the test results validation criteria. The
test results validation criteria MUST be monitored during the whole test results validation criteria MUST be monitored during the whole
sustain phase of the traffic load profile. sustain phase of the traffic load profile.
a. Number of failed Application transactions (receiving any HTTP a. Number of failed Application transactions (receiving any HTTP
response code other than 200 OK) MUST be less than 0.001% (1 out response code other than 200 OK) MUST be less than 0.001% (1 out
of 100,000 transactions) of attempt transactions. of 100,000 transactions) of attempt transactions.
 End of changes. 23 change blocks. 
169 lines changed or deleted 157 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/