draft-ietf-bmwg-firewall-08.txt   rfc3511.txt 
Benchmarking Working Group Brooks Hickman
Internet-Draft Spirent Communications
Expiration Date: June 2003 David Newman
Network Test
Saldju Tadjudin
Spirent Communications
Terry Martin
GVNW Consulting Inc
January 2003
Benchmarking Methodology for Firewall Performance
<draft-ietf-bmwg-firewall-08.txt>
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Network Working Group B. Hickman
Task Force (IETF), its areas, and its working groups. Note that Request for Comments: 3511 Spirent Communications
other groups may also distribute working documents as Internet- Category: Informational D. Newman
Drafts. Network Test
S. Tadjudin
Spirent Communications
T. Martin
GVNW Consulting Inc
April 2003
Internet-Drafts are draft documents valid for a maximum of six Benchmarking Methodology for Firewall Performance
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at Status of this Memo
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at This memo provides information for the Internet community. It does
http://www.ietf.org/shadow.html. not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved. Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract Abstract
This document discusses and defines a number of tests that may be This document discusses and defines a number of tests that may be
used to describe the performance characteristics of firewalls. In
used to describe the performance characteristics of firewalls. In addition to defining the tests, this document also describes specific
addition to defining the tests, this document also describes formats for reporting the results of the tests.
specific formats for reporting the results of the tests.
This document is a product of the Benchmarking Methodology Working This document is a product of the Benchmarking Methodology Working
Group (BMWG) of the Internet Engineering Task Force (IETF). Group (BMWG) of the Internet Engineering Task Force (IETF).
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1 Test Considerations . . . . . . . . . . . . . . . . . . . 4 4.1 Test Considerations. . . . . . . . . . . . . . . . . . . 4
4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 4 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 4
4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4 4.3 Test Traffic Requirements. . . . . . . . . . . . . . . . 5
4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 5 4.4 DUT/SUT Traffic Flows. . . . . . . . . . . . . . . . . . 5
4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5
4.6 Network Address Translation (NAT) . . . . . . . . . . . . 5 4.6 Network Address Translation (NAT). . . . . . . . . . . . 6
4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 5 4.7 Rule Sets. . . . . . . . . . . . . . . . . . . . . . . . 6
4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6 4.8 Web Caching. . . . . . . . . . . . . . . . . . . . . . . 6
4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6 4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 7
4.10 TCP Stack Considerations . . . . . . . . . . . . . . . . 6 4.10 TCP Stack Considerations. . . . . . . . . . . . . . . . 7
5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 7
5.1 IP throughput . . . . . . . . . . . . . . . . . . . . . . 6 5.1 IP throughput. . . . . . . . . . . . . . . . . . . . . . 7
5.2 Concurrent TCP Connection Capacity . . . . . . . . . . . 8 5.2 Concurrent TCP Connection Capacity . . . . . . . . . . . 9
5.3 Maximum TCP Connection Establishment Rate . . . . . . . . 10 5.3 Maximum TCP Connection Establishment Rate. . . . . . . . 12
5.4 Maximum TCP Connection Tear Down Rate . . . . . . . . . . 12 5.4 Maximum TCP Connection Tear Down Rate. . . . . . . . . . 14
5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 14 5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 16
5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 15 5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 18
5.7 Maximum HTTP Transaction Rate . . . . . . . . . . . . . . 18 5.7 Maximum HTTP Transaction Rate. . . . . . . . . . . . . . 21
5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 20 5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 23
5.9 IP Fragmentation Handling . . . . . . . . . . . . . . . . 21 5.9 IP Fragmentation Handling. . . . . . . . . . . . . . . . 24
5.10 Latency . . . . . . . . . . . . . . . . . . . . . . . . 23 5.10 Latency . . . . . . . . . . . . . . . . . . . . . . . . 26
6. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 6. References . . . . . . . . . . . . . . . . . . . . . . . . . 29
7. Security Consideration . . . . . . . . . . . . . . . . . . . 26 6.1 Normative References . . . . . . . . . . . . . . . . . . 29
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 26 6.2 Informative References . . . . . . . . . . . . . . . . . 30
9. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 27 7. Security Consideration . . . . . . . . . . . . . . . . . . . 30
Appendix A - HyperText Transfer Protocol (HTTP) . . . . . . . . 28 Appendix A - HyperText Transfer Protocol (HTTP) . . . . . . . . 31
Appendix B - Connection Establishment Time Measurements . . . . 28 Appendix B - Connection Establishment Time Measurements . . . . 31
Appendix C - Connection Tear Down Time Measurements . . . . . . 29 Appendix C - Connection Tear Down Time Measurements . . . . . . 32
Full Copyright Statement . . . . . . . . . . . . . . . . . . . 30 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 33
Full Copyright Statement . . . . . . . . . . . . . . . . . . . 34
1. Introduction 1. Introduction
This document provides methodologies for the performance This document provides methodologies for the performance benchmarking
benchmarking of firewalls. It covers four areas: forwarding, of firewalls. It covers four areas: forwarding, connection, latency
connection, latency and filtering. In addition to and filtering. In addition to defining tests, this document also
defining tests, this document also describes specific formats describes specific formats for reporting test results.
for reporting test results.
A previous document, "Benchmarking Terminology for Firewall A previous document, "Benchmarking Terminology for Firewall
Performance" [1], defines many of the terms that are used in this Performance" [1], defines many of the terms that are used in this
document. The terminology document SHOULD be consulted before document. The terminology document SHOULD be consulted before
attempting to make use of this document. attempting to make use of this document.
2. Requirements 2. Requirements
In this document, the words that are used to define the significance In this document, the words that are used to define the significance
of each particular requirement are capitalized. These words are: of each particular requirement are capitalized. These words are:
* "MUST" This word, or the words "REQUIRED" and "SHALL" mean that * "MUST" This word, or the words "REQUIRED" and "SHALL" mean that
the item is an absolute requirement of the specification. the item is an absolute requirement of the specification.
* "SHOULD" This word or the adjective "RECOMMENDED" means that * "SHOULD" This word or the adjective "RECOMMENDED" means that there
there may exist valid reasons in particular circumstances to may exist valid reasons in particular circumstances to ignore this
ignore this item, but the full implications should be understood item, but the full implications should be understood and the case
and the case carefully weighed before choosing a different carefully weighed before choosing a different course.
course.
* "MAY" This word or the adjective "OPTIONAL" means that this item * "MAY" This word or the adjective "OPTIONAL" means that this item
is truly optional. One vendor may choose to include the item is truly optional. One vendor may choose to include the item
because a particular marketplace requires it or because it because a particular marketplace requires it or because it
enhances the product, for example; another vendor may omit the enhances the product, for example; another vendor may omit the
same item. same item.
An implementation is not compliant if it fails to satisfy one or more An implementation is not compliant if it fails to satisfy one or more
of the MUST requirements. An implementation that satisfies all the of the MUST requirements. An implementation that satisfies all the
MUST and all the SHOULD requirements is said to be "unconditionally MUST and all the SHOULD requirements is said to be "unconditionally
compliant"; one that satisfies all the MUST requirements but not all compliant"; one that satisfies all the MUST requirements but not all
the SHOULD requirements is said to be "conditionally compliant". the SHOULD requirements is said to be "conditionally compliant".
3. Scope 3. Scope
Firewalls can control access between networks. Usually, a firewall Firewalls can control access between networks. Usually, a firewall
protects a private networks from public or shared network(s) to protects a private network from public or shared network(s) to which
which it is connected. A firewall can be as simple as a single it is connected. A firewall can be as simple as a single device that
device that filters packets or as complex as a group of devices that filters packets or as complex as a group of devices that combine
combine packet filtering and application-level proxy and network packet filtering and application-level proxy and network translation
translation services. This document focuses on benchmarking firewall services. This document focuses on benchmarking firewall
performance - wherever possible, independent of implementation. performance, wherever possible, independent of implementation.
4. Test Setup 4. Test Setup
Test configurations defined in this document will be confined to Test configurations defined in this document will be confined to
dual-homed and tri-homed as shown in figure 1 and figure 2 dual-homed and tri-homed as shown in figure 1 and figure 2
respectively. respectively.
Firewalls employing dual-homed configurations connect two networks. Firewalls employing dual-homed configurations connect two networks.
One interface of the firewall is attached to the unprotected One interface of the firewall is attached to the unprotected network
network[1], typically the public network(Internet). The other [1], typically the public network (Internet). The other interface is
interface is connected to the protected network[1], typically the connected to the protected network [1], typically the internal LAN.
internal LAN.
In the case of dual-homed configurations, servers which are made In the case of dual-homed configurations, servers which are made
accessible to the public(Unprotected) network are attached to the accessible to the public (Unprotected) network are attached to the
private(Protected) network. private (Protected) network.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Servers/ |----| | | |------| Servers/ | | Servers/ |----| | | |------| Servers/ |
| Clients | | | | | | Clients | | Clients | | | | | | Clients |
| | |-------| DUT/SUT |--------| | | | | |-------| DUT/SUT |--------| | |
+----------+ | | | | +----------+ +----------+ | | | | +----------+
Protected | +----------+ | Unprotected Protected | +----------+ | Unprotected
Network | | Network Network | | Network
Figure 1 (Dual-Homed) Figure 1 (Dual-Homed)
Tri-homed[1] configurations employ a third segment called a Tri-homed [1] configurations employ a third segment called a
Demilitarized Zone(DMZ). With tri-homed configurations, servers Demilitarized Zone (DMZ). With tri-homed configurations, servers
accessible to the public network are attached to the DMZ. Tri-Homed accessible to the public network are attached to the DMZ. Tri-Homed
configurations offer additional security by separating server(s) configurations offer additional security by separating server(s)
accessible to the public network from internal hosts. accessible to the public network from internal hosts.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Clients |----| | | |------| Servers/ | | Clients |----| | | |------| Servers/ |
| | | | | | | Clients | | | | | | | | Clients |
+----------+ |-------| DUT/SUT |--------| | | +----------+ |-------| DUT/SUT |--------| | |
| | | | +----------+ | | | | +----------+
| +----------+ | | +----------+ |
Protected | | | Unprotected Protected | | | Unprotected
Network | Network Network | Network
| |
----------------- -----------------
| DMZ | DMZ
| |
| |
+-----------+ +-----------+
| | | |
| Servers | | Servers |
| | | |
+-----------+ +-----------+
Figure 2 (Tri-Homed) Figure 2 (Tri-Homed)
4.1 Test Considerations 4.1 Test Considerations
4.2 Virtual Clients/Servers 4.2 Virtual Clients/Servers
Since firewall testing may involve data sources which emulate Since firewall testing may involve data sources which emulate
multiple users or hosts, the methodology uses the terms virtual multiple users or hosts, the methodology uses the terms virtual
clients/servers. For these firewall tests, virtual clients/servers clients/servers. For these firewall tests, virtual clients/servers
specify application layer entities which may not be associated with specify application layer entities which may not be associated with a
a unique physical interface. For example, four virtual clients may unique physical interface. For example, four virtual clients may
originate from the same data source[1]. The test report MUST originate from the same data source [1]. The test report MUST
indicate the number of virtual clients and virtual servers indicate the number of virtual clients and virtual servers
participating in the test. participating in the test.
4.3 Test Traffic Requirements 4.3 Test Traffic Requirements
While the function of a firewall is to enforce access control While the function of a firewall is to enforce access control
policies, the criteria by which those policies are defined vary policies, the criteria by which those policies are defined vary
depending on the implementation. Firewalls may use network layer, depending on the implementation. Firewalls may use network layer,
transport layer or, in many cases, application-layer criteria to transport layer or, in many cases, application-layer criteria to make
make access-control decisions. access-control decisions.
For the purposes of benchmarking firewall performance, this document For the purposes of benchmarking firewall performance, this document
references HTTP 1.1 or higher as the application layer entity. The references HTTP 1.1 or higher as the application layer entity. The
methodologies MAY be used as a template for benchmarking with other methodologies MAY be used as a template for benchmarking with other
applications. Since testing may involve proxy based DUT/SUTs, HTTP applications. Since testing may involve proxy based DUT/SUTs, HTTP
version considerations are discussed in appendix A. version considerations are discussed in appendix A.
4.4 DUT/SUT Traffic Flows 4.4 DUT/SUT Traffic Flows
Since the number of interfaces are not fixed, the traffic flows will Since the number of interfaces are not fixed, the traffic flows will
be dependent upon the configuration used in benchmarking the be dependent upon the configuration used in benchmarking the DUT/SUT.
DUT/SUT. Note that the term "traffic flows" is associated with Note that the term "traffic flows" is associated with client-to-
client-to-server requests. server requests.
For Dual-Homed configurations, there are two unique traffic flows: For Dual-Homed configurations, there are two unique traffic flows:
Client Server Client Server
------ ------ ------ ------
Protected -> Unprotected Protected -> Unprotected
Unprotected -> Protected Unprotected -> Protected
For Tri-Homed configurations, there are three unique traffic flows: For Tri-Homed configurations, there are three unique traffic flows:
Client Server Client Server
------ ------ ------ ------
Protected -> Unprotected Protected -> Unprotected
Protected -> DMZ Protected -> DMZ
Unprotected -> DMZ Unprotected -> DMZ
4.5 Multiple Client/Server Testing 4.5 Multiple Client/Server Testing
One or more clients may target multiple servers for a given One or more clients may target multiple servers for a given
application. Each virtual client MUST initiate connections in a application. Each virtual client MUST initiate connections in a
round-robin fashion. For example, if the test consisted of six round-robin fashion. For example, if the test consisted of six
virtual clients targeting three servers, the pattern would be as virtual clients targeting three servers, the pattern would be as
follows: follows:
Client Target Server(In order of request) Client Target Server (In order of request)
#1 1 2 3 1... #1 1 2 3 1...
#2 2 3 1 2... #2 2 3 1 2...
#3 3 1 2 3... #3 3 1 2 3...
#4 1 2 3 1... #4 1 2 3 1...
#5 2 3 1 2... #5 2 3 1 2...
#6 3 1 2 3... #6 3 1 2 3...
4.6 Network Address Translation (NAT) 4.6 Network Address Translation (NAT)
Many firewalls implement network address translation(NAT)[1], a Many firewalls implement network address translation (NAT) [1], a
function which translates private internet addresses to public function which translates private internet addresses to public
internet addresses. This involves additional processing on the part internet addresses. This involves additional processing on the part
of the DUT/SUT and may impact performance. Therefore, tests SHOULD of the DUT/SUT and may impact performance. Therefore, tests SHOULD
be ran with NAT disabled and NAT enabled to determine the be ran with NAT disabled and NAT enabled to determine the performance
performance differential, if any. The test report MUST indicate differential, if any. The test report MUST indicate whether NAT was
whether NAT was enabled or disabled. enabled or disabled.
4.7 Rule Sets 4.7 Rule Sets
Rule sets[1] are a collection of access control policies that Rule sets [1] are a collection of access control policies that
determine which packets the DUT/SUT will forward and which it will determine which packets the DUT/SUT will forward and which it will
reject[1]. Since criteria by which these access control policies may reject [1]. Since criteria by which these access control policies
be defined will vary depending on the capabilities of the DUT/SUT, may be defined will vary depending on the capabilities of the
the following is limited to providing guidelines for configuring DUT/SUT, the following is limited to providing guidelines for
rule sets when benchmarking the performance of the DUT/SUT. configuring rule sets when benchmarking the performance of the
DUT/SUT.
It is RECOMMENDED that a rule be entered for each host(Virtual It is RECOMMENDED that a rule be entered for each host (Virtual
client). In addition, testing SHOULD be performed using different client). In addition, testing SHOULD be performed using different
size rule sets to determine its impact on the performance of the size rule sets to determine its impact on the performance of the
DUT/SUT. Rule sets MUST be configured in a manner, such that, rules DUT/SUT. Rule sets MUST be configured in a manner, such that, rules
associated with actual test traffic are configured at the end of the associated with actual test traffic are configured at the end of the
rule set and not the beginning. rule set and not at the beginning.
The DUT/SUT SHOULD be configured to deny access to all traffic which The DUT/SUT SHOULD be configured to deny access to all traffic which
was not previously defined in the rule set. The test report SHOULD was not previously defined in the rule set. The test report SHOULD
include the DUT/SUT configured rule set(s). include the DUT/SUT configured rule set(s).
4.8 Web Caching 4.8 Web Caching
Some firewalls include caching agents to reduce network load. When Some firewalls include caching agents to reduce network load. When
making a request through a caching agent, the caching agent attempts making a request through a caching agent, the caching agent attempts
to service the response from its internal memory. The cache itself to service the response from its internal memory. The cache itself
saves responses it receives, such as responses for HTTP GET saves responses it receives, such as responses for HTTP GET requests.
requests. Testing SHOULD be performed with any caching agents on the Testing SHOULD be performed with any caching agents on the DUT/SUT
DUT/SUT disabled. disabled.
4.9 Authentication 4.9 Authentication
Access control may involve authentication processes such as user, Access control may involve authentication processes such as user,
client or session authentication. Authentication is usually client or session authentication. Authentication is usually
performed by devices external to the firewall itself, such as an performed by devices external to the firewall itself, such as an
authentication server(s) and may add to the latency of the system. authentication server(s) and may add to the latency of the system.
Any authentication processes MUST be included as part of connection Any authentication processes MUST be included as part of connection
setup process. setup process.
4.10 TCP Stack Considerations 4.10 TCP Stack Considerations
Some test instruments allow configuration of one or more TCP stack Some test instruments allow configuration of one or more TCP stack
parameters, thereby influencing the traffic flows which will be parameters, thereby influencing the traffic flows which will be
offered and impacting performance measurements. While this document offered and impacting performance measurements. While this document
does not attempt to specify which TCP parameters should be does not attempt to specify which TCP parameters should be
configurable, any such TCP parameter(s) MUST be noted in the test configurable, any such TCP parameter(s) MUST be noted in the test
report. In addition, when comparing multiple DUT/SUTs, the same TCP report. In addition, when comparing multiple DUT/SUTs, the same TCP
parameters MUST be used. parameters MUST be used.
5. Benchmarking Tests 5. Benchmarking Tests
5.1 IP Throughput 5.1 IP Throughput
5.1.1 Objective 5.1.1 Objective
To determine the throughput of network-layer data traversing the To determine the throughput of network-layer data traversing the
DUT/SUT, as defined in RFC1242[3]. Note that while RFC1242 uses the DUT/SUT, as defined in RFC 1242 [3]. Note that while RFC 1242 uses
term frames, which is associated with the link layer, the procedure the term frames, which is associated with the link layer, the
uses the term packets, since it is referencing the network layer. procedure uses the term packets, since it is referencing the network
layer.
5.1.2 Setup Parameters 5.1.2 Setup Parameters
The following parameters MUST be defined: The following parameters MUST be defined:
Packet size - Number of bytes in the IP packet, exclusive of any Packet size - Number of bytes in the IP packet, exclusive of any
link layer header or checksums. link layer header or checksums.
Test Duration - Duration of the test, expressed in seconds. Test Duration - Duration of the test, expressed in seconds.
5.1.3 Procedure 5.1.3 Procedure
The tester MUST offer unicast IP packets traffic to the DUT/SUT at a The test instrument MUST offer unicast IP packets to the DUT/SUT at a
constant rate. The test MAY consist of either bi-directional or constant rate. The test MAY consist of either bi-directional or
unidirectional traffic; for example, an emulated client may offer a unidirectional traffic; for example, an emulated client may offer a
unicast stream of packets to an emulated server, or the tester may unicast stream of packets to an emulated server, or the test
simulate a client/server exchange by offering bidirectional traffic. instrument may simulate a client/server exchange by offering
bidirectional traffic.
The test MAY employ an iterative search algorithm. Each iteration This test will employ an iterative search algorithm. Each iteration
will involve the tester varying the intended load until the maximum will involve the test instrument varying the intended load until the
rate, at which no packet loss occurs, is found. Since backpressure maximum rate, at which no packet loss occurs, is found. Since
mechanisms may be employed, resulting in the intended load and backpressure mechanisms may be employed, resulting in the intended
offered load being different, the test SHOULD be performed in either load and offered load being different, the test SHOULD be performed
a packet based or time based manner as described in RFC2889[5]. As in either a packet based or time based manner as described in RFC
with RFC1242, the term packet is used in place of frame. The 2889 [5]. As with RFC 1242, the term packet is used in place of
duration of the test portion of each trial MUST be at least 30 frame. The duration of the test portion of each trial MUST be at
seconds. least 30 seconds.
It is RECOMMENDED to perform the throughput measurements with It is RECOMMENDED to perform the throughput measurements with
different packet sizes. When testing with different packet sizes the different packet sizes. When testing with different packet sizes the
DUT/SUT configuration MUST remain the same. DUT/SUT configuration MUST remain the same.
5.1.4 Measurement 5.1.4 Measurement
5.1.4.1 Network Layer 5.1.4.1 Network Layer
Throughput - Maximum offered load, expressed in either bits per Throughput:
second or packets per second, at which no packet loss is detected. Maximum offered load, expressed in either bits per second or
The bits to be counted are in the IP packet (header plus payload); packets per second, at which no packet loss is detected. The bits
other fields, such as link-layer headers and trailers, MUST NOT be to be counted are in the IP packet (header plus payload); other
included in the measurement. fields, such as link-layer headers and trailers, MUST NOT be
included in the measurement.
Forwarding Rate - Forwarding rate, expressed in either bits per Forwarding Rate:
second or packets per second, the device is observed to successfully Forwarding rate, expressed in either bits per second or packets
forward to the correct destination interface in response to a per second, the device is observed to successfully forward to the
specified offered load. The bits to be counted are in the IP packet correct destination interface in response to a specified offered
(header plus payload); other fields, such as link-layer headers and load. The bits to be counted are in the IP packet (header plus
trailers, MUST NOT be included in the measurement. payload); other fields, such as link-layer headers and trailers,
MUST NOT be included in the measurement.
5.1.4 Reporting Format 5.1.5 Reporting Format
The test report MUST note the packet size(s), test duration, The test report MUST note the packet size(s), test duration,
throughput and forwarding rate. If the test involved offering throughput and forwarding rate. In addition, the test report MUST
packets which target more than one segment(Protected, Unprotected conform to the reporting requirements set in section 4, Test Setup.
or DMZ), the report MUST identify the results as an aggregate If the test involved offering packets which target more than one
throughput measurement. segment (Protected, Unprotected or DMZ), the report MUST identify the
results as an aggregate throughput measurement.
The throughput results SHOULD be reported in the format of a table The throughput results SHOULD be reported in the format of a table
with a row for each of the tested packet sizes. There SHOULD be with a row for each of the tested packet sizes. There SHOULD be
columns for the packet size, the intended load, the offered load, columns for the packet size, the intended load, the offered load,
resultant throughput and forwarding rate for each test. resultant throughput and forwarding rate for each test.
The intermediate results of the search algorithm MAY be saved in log The intermediate results of the search algorithm MAY be saved in log
file which includes the packet size, test duration and for each file which includes the packet size, test duration and for each
iteration: iteration:
- Step Iteration - Step Iteration
- Pass/Fail Status - Pass/Fail Status
- Total packets offered - Total packets offered
- Total packets forwarded - Total packets forwarded
- Intended load - Intended load
- Offered load(If applicable) - Offered load (If applicable)
- Forwarding rate - Forwarding rate
5.2 Concurrent TCP Connection Capacity 5.2 Concurrent TCP Connection Capacity
5.2.1 Objective 5.2.1 Objective
To determine the maximum number of concurrent TCP connections To determine the maximum number of concurrent TCP connections
supported through or with the DUT/SUT, as defined in RFC2647[1]. supported through or with the DUT/SUT, as defined in RFC 2647 [1].
This test is intended to find the maximum number of entries the This test is intended to find the maximum number of entries the
DUT/SUT can store in its connection table. DUT/SUT can store in its connection table.
5.2.2 Setup Parameters 5.2.2 Setup Parameters
The following parameters MUST be defined for all tests: The following parameters MUST be defined for all tests:
5.2.2.1 Transport-Layer Setup Parameters 5.2.2.1 Transport-Layer Setup Parameters
Connection Attempt Rate - The aggregate rate, expressed in Connection Attempt Rate:
connections per second, at which TCP connection requests are The aggregate rate, expressed in connections per second, at which
attempted. The rate SHOULD be set at or lower than the maximum TCP connection requests are attempted. The rate SHOULD be set at
rate at which the DUT/SUT can accept connection requests. or lower than the maximum rate at which the DUT/SUT can accept
connection requests.
Aging Time - The time, expressed in seconds, the DUT/SUT will Aging Time:
keep a connection in its connection table after receiving a TCP The time, expressed in seconds, the DUT/SUT will keep a connection
FIN or RST packet. in its connection table after receiving a TCP FIN or RST packet.
5.2.2.2 Application-Layer Setup Parameters 5.2.2.2 Application-Layer Setup Parameters
Validation Method - HTTP 1.1 or higher MUST be used for this Validation Method:
test for both clients and servers. The client and server MUST use HTTP 1.1 or higher MUST be used for this test for both clients and
the same HTTP version. servers. The client and server MUST use the same HTTP version.
Object Size - Defines the number of bytes, excluding any bytes Object Size:
associated with the HTTP header, to be transferred in response to Defines the number of bytes, excluding any bytes associated with
an HTTP 1.1 or higher GET request. the HTTP header, to be transferred in response to an HTTP 1.1 or
higher GET request.
5.2.3 Procedure 5.2.3 Procedure
An iterative search algorithm MAY be used to determine the maximum This test will employ an iterative search algorithm to determine the
number of concurrent TCP connections supported through or with the maximum number of concurrent TCP connections supported through or
DUT/SUT. with the DUT/SUT.
For each iteration, the aggregate number of concurrent TCP For each iteration, the aggregate number of concurrent TCP
connections attempted by the virtual client(s) will be varied. The connections attempted by the virtual client(s) will be varied. The
destination address will be that of the server or that of the NAT destination address will be that of the server or that of the NAT
proxy. The aggregate rate will be defined by connection attempt proxy. The aggregate rate will be defined by connection attempt
rate, and will be attempted in a round-robin fashion(See 4.5). rate, and will be attempted in a round-robin fashion (See 4.5).
To validate all connections, the virtual client(s) MUST request an To validate all connections, the virtual client(s) MUST request an
object using an HTTP 1.1 or higher GET request. The requests MUST be object using an HTTP 1.1 or higher GET request. The requests MUST be
initiated on each connection after all of the TCP connections have initiated on each connection after all of the TCP connections have
been established. been established.
When testing proxy-based DUT/SUTs, the virtual client(s) MUST When testing proxy-based DUT/SUTs, the virtual client(s) MUST request
request two objects using HTTP 1.1 or higher GET requests. The first two objects using HTTP 1.1 or higher GET requests. The first GET
GET request is required for connection time establishment[1] request is required for connection time establishment [1]
measurements as specified in appendix B. The second request is used measurements as specified in appendix B. The second request is used
for validation as previously mentioned. When comparing proxy and for validation as previously mentioned. When comparing proxy and
non-proxy based DUT/SUTs, the test MUST be performed in the same non-proxy based DUT/SUTs, the test MUST be performed in the same
manner. manner.
Between each iteration, it is RECOMMENDED that the tester issue a Between each iteration, it is RECOMMENDED that the test instrument
TCP RST referencing each connection attempted for the previous issue a TCP RST referencing each connection attempted for the
iteration, regardless of whether or not the connection attempt was previous iteration, regardless of whether or not the connection
successful. The tester will wait for aging time before continuing to attempt was successful. The test instrument will wait for aging time
the next iteration. before continuing to the next iteration.
5.2.4 Measurements 5.2.4 Measurements
5.2.4.1 Application-Layer measurements 5.2.4.1 Application-Layer measurements
Number of objects requested Number of objects requested
Number of objects returned Number of objects returned
5.2.4.2 Transport-Layer measurements 5.2.4.2 Transport-Layer measurements
Maximum concurrent connections - Total number of TCP connections Maximum concurrent connections:
open for the last successful iteration performed in the search Total number of TCP connections open for the last successful
algorithm. iteration performed in the search algorithm.
Minimum connection establishment time - Lowest TCP connection Minimum connection establishment time:
establishment time measured as defined in appendix B. Lowest TCP connection establishment time measured, as defined in
appendix B.
Maximum connection establishment time - Highest TCP connection Maximum connection establishment time:
establishment time measured as defined in appendix B. Highest TCP connection establishment time measured, as defined in
appendix B.
Average connection establishment time - The mean of all Average connection establishment time:
measurements of connection establishment times. The mean of all measurements of connection establishment times.
Aggregate connection establishment time - The total of all Aggregate connection establishment time:
measurements of connection establishment times. The total of all measurements of connection establishment times.
5.2.5 Reporting Format 5.2.5 Reporting Format
5.2.5.1 Application-Layer Reporting: The test report MUST conform to the reporting requirements set in
section 4, Test Setup.
The test report MUST note the object size, number of completed 5.2.5.1 Application-Layer Reporting:
requests and number of completed responses.
The intermediate results of the search algorithm MAY be reported The test report MUST note the object size, number of completed
in a tabular format with a column for each iteration. There requests and number of completed responses.
SHOULD be rows for the number of requests attempted, number and
percentage requests completed, number of responses attempted,
number and percentage of responses completed. The table MAY be
combined with the transport-layer reporting, provided that the
table identify this as an application layer measurement.
Version information: The intermediate results of the search algorithm MAY be reported in a
tabular format with a column for each iteration. There SHOULD be
rows for the number of requests attempted, number and percentage
requests completed, number of responses attempted, number and
percentage of responses completed. The table MAY be combined with
the transport-layer reporting, provided that the table identify this
as an application layer measurement.
Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.2.5.2 Transport-Layer Reporting: 5.2.5.2 Transport-Layer Reporting:
The test report MUST note the connection attempt rate, aging time, The test report MUST note the connection attempt rate, aging time,
minimum TCP connection establishment time, maximum TCP connection minimum TCP connection establishment time, maximum TCP connection
establishment time, average connection establishment time, aggregate establishment time, average connection establishment time, aggregate
connection establishment time and maximum concurrent connections connection establishment time and maximum concurrent connections
measured. measured.
The intermediate results of the search algorithm MAY be reported in The intermediate results of the search algorithm MAY be reported in
the format of a table with a column for each iteration. There SHOULD the format of a table with a column for each iteration. There SHOULD
be rows for the total number of TCP connections attempted, number be rows for the total number of TCP connections attempted, number and
and percentage of TCP connections completed, minimum TCP connection percentage of TCP connections completed, minimum TCP connection
establishment time, maximum TCP connection establishment time, establishment time, maximum TCP connection establishment time,
average connection establishment time and the aggregate connection average connection establishment time and the aggregate connection
establishment time. establishment time.
5.3 Maximum TCP Connection Establishment Rate 5.3 Maximum TCP Connection Establishment Rate
5.3.1 Objective 5.3.1 Objective
To determine the maximum TCP connection establishment rate through To determine the maximum TCP connection establishment rate through or
or with the DUT/SUT, as defined by RFC2647[1]. This test is intended with the DUT/SUT, as defined by RFC 2647 [1]. This test is intended
to find the maximum rate the DUT/SUT can update its connection to find the maximum rate the DUT/SUT can update its connection table.
table.
5.3.2 Setup Parameters 5.3.2 Setup Parameters
The following parameters MUST be defined for all tests: The following parameters MUST be defined for all tests:
5.3.2.1 Transport-Layer Setup Parameters 5.3.2.1 Transport-Layer Setup Parameters
Number of Connections - Defines the aggregate number of TCP
connections that must be established.
Aging Time - The time, expressed in seconds, the DUT/SUT will Number of Connections:
keep a connection in it's state table after receiving a TCP FIN Defines the aggregate number of TCP connections that must be
or RST packet. established.
5.3.2.2 Application-Layer Setup Parameters Aging Time:
The time, expressed in seconds, the DUT/SUT will keep a connection
in it's state table after receiving a TCP FIN or RST packet.
Validation Method - HTTP 1.1 or higher MUST be used for this 5.3.2.2 Application-Layer Setup Parameters
test for both clients and servers. The client and server MUST use
the same HTTP version.
Object Size - Defines the number of bytes, excluding any bytes Validation Method:
associated with the HTTP header, to be transferred in response to HTTP 1.1 or higher MUST be used for this test for both clients and
an HTTP 1.1 or higher GET request. servers. The client and server MUST use the same HTTP version.
Object Size:
Defines the number of bytes, excluding any bytes associated with
the HTTP header, to be transferred in response to an HTTP 1.1 or
higher GET request.
5.3.3 Procedure 5.3.3 Procedure
An iterative search algorithm MAY be used to determine the maximum This test will employ an iterative search algorithm to determine the
rate at which the DUT/SUT can accept TCP connection requests. maximum rate at which the DUT/SUT can accept TCP connection requests.
For each iteration, the aggregate rate at which TCP connection For each iteration, the aggregate rate at which TCP connection
requests are attempted by the virtual client(s) will be varied. The requests are attempted by the virtual client(s) will be varied. The
destination address will be that of the server or that of the NAT destination address will be that of the server or that of the NAT
proxy. The aggregate number of connections, defined by number of proxy. The aggregate number of connections, defined by number of
connections, will be attempted in a round-robin fashion(See 4.5). connections, will be attempted in a round-robin fashion (See 4.5).
The same application-layer object transfers required for validation The same application-layer object transfers required for validation
and establishment time measurements as described in the concurrent and establishment time measurements as described in the concurrent
TCP connection capacity test MUST be performed. TCP connection capacity test MUST be performed.
Between each iteration, it is RECOMMENDED that the tester issue a Between each iteration, it is RECOMMENDED that the test instrument
TCP RST referencing each connection attempted for the previous issue a TCP RST referencing each connection attempted for the
iteration, regardless of whether or not the connection attempt was previous iteration, regardless of whether or not the connection
successful. The tester will wait for aging time before continuing to attempt was successful. The test instrument will wait for aging time
the next iteration. before continuing to the next iteration.
5.3.4 Measurements 5.3.4 Measurements
5.3.4.1 Application-Layer measurements 5.3.4.1 Application-Layer measurements
Number of objects requested Number of objects requested
Number of objects returned Number of objects returned
5.3.4.2 Transport-Layer measurements 5.3.4.2 Transport-Layer measurements
Highest connection rate - Highest rate, in connections per Highest connection rate:
second, for which all connections successfully opened in the Highest rate, in connections per second, for which all connections
search algorithm. successfully opened in the search algorithm.
Minimum connection establishment time - Lowest TCP connection Minimum connection establishment time:
establishment time measured as defined in appendix B. Lowest TCP connection establishment time measured, as defined in
appendix B.
Maximum connection establishment time - Highest TCP connection Maximum connection establishment time:
establishment time measured as defined in appendix B. Highest TCP connection establishment time measured, as defined in
appendix B.
Average connection establishment time - The mean of all Average connection establishment time:
measurements of connection establishment times. The mean of all measurements of connection establishment times.
Aggregate connection establishment time - The total of all Aggregate connection establishment time:
measurements of connection establishment times. The total of all measurements of connection establishment times.
5.3.5 Reporting Format 5.3.5 Reporting Format
5.3.5.1 Application-Layer Reporting: The test report MUST conform to the reporting requirements set in
section 4, Test Setup.
The test report MUST note object size(s), number of completed 5.3.5.1 Application-Layer Reporting:
requests and number of completed responses.
The intermediate results of the search algorithm MAY be reported The test report MUST note object size(s), number of completed
in a tabular format with a column for each iteration. There requests and number of completed responses.
SHOULD be rows for the number of requests attempted, number and
percentage requests completed, number of responses attempted,
number and percentage of responses completed. The table MAY be
combined with the transport-layer reporting, provided that the
table identify this as an application layer measurement.
Version information: The intermediate results of the search algorithm MAY be reported in a
tabular format with a column for each iteration. There SHOULD be
rows for the number of requests attempted, number and percentage
requests completed, number of responses attempted, number and
percentage of responses completed. The table MAY be combined with
the transport-layer reporting, provided that the table identify this
as an application layer measurement.
Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.3.5.2 Transport-Layer Reporting: 5.3.5.2 Transport-Layer Reporting:
The test report MUST note the number of connections, aging time, The test report MUST note the number of connections, aging time,
minimum TCP connection establishment time, maximum TCP connection minimum TCP connection establishment time, maximum TCP connection
establishment time, average connection establishment time, establishment time, average connection establishment time, aggregate
aggregate connection establishment time and highest connection connection establishment time and highest connection rate measured.
rate measured.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported in
in the format of a table with a column for each iteration. There the format of a table with a column for each iteration. There SHOULD
SHOULD be rows for the connection attempt rate, total number of be rows for the connection attempt rate, total number of TCP
TCP connections attempted, total number of TCP connections connections attempted, total number of TCP connections completed,
completed, minimum TCP connection establishment time, maximum TCP minimum TCP connection establishment time, maximum TCP connection
connection establishment time, average connection establishment establishment time, average connection establishment time and the
time and the aggregate connection establishment time. aggregate connection establishment time.
5.4 Maximum TCP Connection Tear Down Rate 5.4 Maximum TCP Connection Tear Down Rate
5.4.1 Objective 5.4.1 Objective
To determine the maximum TCP connection tear down rate through or To determine the maximum TCP connection tear down rate through or
with the DUT/SUT, as defined by RFC2647[1]. with the DUT/SUT, as defined by RFC 2647 [1].
5.4.2 Setup Parameters 5.4.2 Setup Parameters
Number of Connections - Defines the number of TCP connections that Number of Connections:
will be attempted to be torn down. Defines the number of TCP connections that will be attempted to be
torn down.
Aging Time - The time, expressed in seconds, the DUT/SUT will keep a Aging Time:
connection in it's state table after receiving a TCP FIN or RST The time, expressed in seconds, the DUT/SUT will keep a connection
packet. in it's state table after receiving a TCP FIN or RST packet.
Close Method - Defines method for closing TCP connections. The test Close Method:
MUST be performed with either a three-way or four-way handshake. In Defines method for closing TCP connections. The test MUST be
a four-way handshake, each side sends separate FIN and ACK messages. performed with either a three-way or four-way handshake. In a
In a three-way handshake, one side sends a combined FIN/ACK message four-way handshake, each side sends separate FIN and ACK messages.
upon receipt of a FIN. In a three-way handshake, one side sends a combined FIN/ACK
message upon receipt of a FIN.
Close Direction - Defines whether closing of connections are to be Close Direction:
initiated from the client or from the server. Defines whether closing of connections are to be initiated from
the client or from the server.
5.4.3 Procedure 5.4.3 Procedure
An iterative search algorithm MAY be used to determine the maximum This test will employ an iterative search algorithm to determine the
TCP connection tear down rate. The test iterates through different maximum TCP connection tear down rate supported by the DUT/SUT. The
TCP connection tear down rates with a fixed number of TCP test iterates through different TCP connection tear down rates with a
connections. fixed number of TCP connections.
In the case of proxy based DUT/SUTs, the DUT/SUT will itself receive In the case of proxy based DUT/SUTs, the DUT/SUT will itself receive
the ACK in response to issuing a FIN packet to close its side of the the ACK in response to issuing a FIN packet to close its side of the
TCP connection. For validation purposes, the virtual client or TCP connection. For validation purposes, the virtual client or
server, whichever is applicable, MAY verify that the DUT/SUT server, whichever is applicable, MAY verify that the DUT/SUT received
received the final ACK by re-transmitting the final ACK. A TCP RST the final ACK by re-transmitting the final ACK. A TCP RST should be
should be received in response to the retransmitted ACK. received in response to the retransmitted ACK.
Between each iteration, it is RECOMMENDED that the virtual client(s) Between each iteration, it is RECOMMENDED that the virtual client(s)
or server(s), whichever is applicable, issue a TCP RST referencing or server(s), whichever is applicable, issue a TCP RST referencing
each connection which was attempted to be torn down, regardless of each connection which was attempted to be torn down, regardless of
whether or not the connection tear down attempt was successful. The whether or not the connection tear down attempt was successful. The
test will wait for aging time before continuing to the next test will wait for aging time before continuing to the next
iteration. iteration.
5.4.4 Measurements 5.4.4 Measurements
Highest connection tear down rate - Highest rate, in connections per Highest connection tear down rate:
second, for which all TCP connections were successfully torn down in Highest rate, in connections per second, for which all TCP
the search algorithm. connections were successfully torn down in the search algorithm.
The following tear down time[1] measurements MUST only include The following tear down time [1] measurements MUST only include
connections for which both sides of the connection were successfully connections for which both sides of the connection were successfully
torn down. For example, tear down times for connections which are torn down. For example, tear down times for connections which are
left in a FINWAIT-2[8] state should not be included: left in a FINWAIT-2 [8] state should not be included:
Minimum connection tear down time - Lowest TCP connection tear down Minimum connection tear down time:
time measured as defined in appendix C. Lowest TCP connection tear down time measured as defined in
appendix C.
Maximum connection tear down time - Highest TCP connection tear down Maximum connection tear down time:
time measured as defined in appendix C. Highest TCP connection tear down time measured as defined in
appendix C.
Average connection tear down time - The mean of all measurements of Average connection tear down time:
connection tear down times. The mean of all measurements of connection tear down times.
Aggregate connection tear down time - The total of all measurements Aggregate connection tear down time:
of connection tear down times. The total of all measurements of connection tear down times.
5.4.5 Reporting Format 5.4.5 Reporting Format
The test report MUST note the number of connections, aging time, The test report MUST note the number of connections, aging time,
close method, close direction, minimum TCP connection tear down close method, close direction, minimum TCP connection tear down time,
time, maximum TCP connection tear down time, average TCP connection maximum TCP connection tear down time, average TCP connection tear
tear down time and the aggregate TCP connection tear down time and down time and the aggregate TCP connection tear down time and highest
highest connection tear down rate measured. connection tear down rate measured. In addition, the test report MUST
conform to the reporting requirements set in section 4, Test Setup.
The intermediate results of the search algorithm MAY be reported in The intermediate results of the search algorithm MAY be reported in
the format of a table with a column for each iteration. There SHOULD the format of a table with a column for each iteration. There SHOULD
be rows for the number of TCP tear downs attempted, number and be rows for the number of TCP tear downs attempted, number and
percentage of TCP connection tear downs completed, minimum TCP percentage of TCP connection tear downs completed, minimum TCP
connection tear down time, maximum TCP connection tear down time, connection tear down time, maximum TCP connection tear down time,
average TCP connection tear down time, aggregate TCP connection tear average TCP connection tear down time, aggregate TCP connection tear
down time and validation failures, if required. down time and validation failures, if required.
5.5 Denial Of Service Handling 5.5 Denial Of Service Handling
5.5.1 Objective 5.5.1 Objective
To determine the effect of a denial of service attack on a DUT/SUT To determine the effect of a denial of service attack on a DUT/SUT
TCP connection establishment and/or HTTP transfer rates. The denial TCP connection establishment and/or HTTP transfer rates. The denial
of service handling test MUST be run after obtaining baseline of service handling test MUST be run after obtaining baseline
measurements from sections 5.3 and/or 5.6. measurements from sections 5.3 and/or 5.6.
The TCP SYN flood attack exploits TCP's three-way handshake The TCP SYN flood attack exploits TCP's three-way handshake mechanism
mechanism by having an attacking source host generate TCP SYN by having an attacking source host generate TCP SYN packets with
packets with random source addresses towards a victim host, thereby random source addresses towards a victim host, thereby consuming that
consuming that host's resources. host's resources.
5.5.2 Setup Parameters 5.5.2 Setup Parameters
Use the same setup parameters as defined in section 5.3.2 or 5.6.2, Use the same setup parameters as defined in section 5.3.2 or 5.6.2,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
establishment rate test or HTTP transfer rate test, respectfully. establishment rate test or HTTP transfer rate test, respectfully.
In addition, the following setup parameters MUST be defined: In addition, the following setup parameters MUST be defined:
SYN attack rate - Rate, expressed in packets per second, at which SYN attack rate:
the server(s) or NAT proxy address is targeted with TCP SYN packets. Rate, expressed in packets per second, at which the server(s) or
NAT proxy address is targeted with TCP SYN packets.
5.5.3 Procedure 5.5.3 Procedure
Use the same procedure as defined in section 5.3.3 or 5.6.3, Use the same procedure as defined in section 5.3.3 or 5.6.3,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
establishment rate or HTTP transfer rate test, respectfully. In establishment rate or HTTP transfer rate test, respectfully. In
addition, the tester will generate TCP SYN packets targeting the addition, the test instrument will generate TCP SYN packets targeting
server(s) IP address or NAT proxy address at a rate defined by SYN the server(s) IP address or NAT proxy address at a rate defined by
attack rate. SYN attack rate.
The tester originating the TCP SYN attack MUST be attached to the The test instrument originating the TCP SYN attack MUST be attached
unprotected network. In addition, the tester MUST not respond to the to the unprotected network. In addition, the test instrument MUST
SYN/ACK packets sent by target server or NAT proxy in response to not respond to the SYN/ACK packets sent by target server or NAT proxy
the SYN packet. in response to the SYN packet.
Some firewalls employ mechanisms to guard against SYN attacks. If Some firewalls employ mechanisms to guard against SYN attacks. If
such mechanisms exist on the DUT/SUT, tests SHOULD be run with these such mechanisms exist on the DUT/SUT, tests SHOULD be run with these
mechanisms enabled and disabled to determine how well the DUT/SUT mechanisms enabled and disabled to determine how well the DUT/SUT can
can maintain, under such attacks, the baseline connection maintain, under such attacks, the baseline connection establishment
establishment rates and HTTP transfer rates determined in section rates and HTTP transfer rates determined in section 5.3 and section
5.3 and section 5.6, respectively. 5.6, respectively.
5.5.4 Measurements 5.5.4 Measurements
Perform the same measurements as defined in section 5.3.4 or 5.6.4, Perform the same measurements as defined in section 5.3.4 or 5.6.4,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
establishment rate test or HTTP transfer rate, respectfully. establishment rate test or HTTP transfer rate, respectfully.
In addition, the tester SHOULD track TCP SYN packets associated with In addition, the test instrument SHOULD track TCP SYN packets
the SYN attack which the DUT/SUT forwards on the protected or DMZ associated with the SYN attack which the DUT/SUT forwards on the
interface(s). protected or DMZ interface(s).
5.5.5 Reporting Format 5.5.5 Reporting Format
The test SHOULD use the same reporting format as described in The test SHOULD use the same reporting format as described in section
section 5.3.5 or 5.6.5, depending on whether testing against the 5.3.5 or 5.6.5, depending on whether testing against the baseline TCP
baseline TCP connection establishment rate test or HTTP transfer connection establishment rate test or HTTP transfer rate,
rate, respectfully. respectfully.
In addition, the report MUST indicate a denial of service handling In addition, the report MUST indicate a denial of service handling
test, SYN attack rate, number of TCP SYN attack packets transmitted test, SYN attack rate, number of TCP SYN attack packets transmitted
and the number of TCP SYN attack packets forwarded by the DUT/SUT. and the number of TCP SYN attack packets forwarded by the DUT/SUT.
The report MUST indicate whether or not the DUT has any SYN attack The report MUST indicate whether or not the DUT has any SYN attack
mechanisms enabled. mechanisms enabled.
5.6 HTTP Transfer Rate 5.6 HTTP Transfer Rate
5.6.1 Objective 5.6.1 Objective
To determine the transfer rate of HTTP requested object traversing To determine the transfer rate of HTTP requested object traversing
the DUT/SUT. the DUT/SUT.
5.6.2 Setup Parameters 5.6.2 Setup Parameters
The following parameters MUST be defined for all tests: The following parameters MUST be defined for all tests:
5.6.2.1 Transport-Layer Setup Parameters 5.6.2.1 Transport-Layer Setup Parameters
Number of connections - Defines the aggregate number of Number of connections:
connections attempted. The number SHOULD be a multiple of the Defines the aggregate number of connections attempted. The number
number of virtual clients participating in the test. SHOULD be a multiple of the number of virtual clients
participating in the test.
Close Method - Defines the method for closing TCP connections. Close Method:
The test MUST be performed with either a three-way or four-way Defines the method for closing TCP connections. The test MUST be
handshake. In a four-way handshake, each side sends separate FIN performed with either a three-way or four-way handshake. In a
and ACK messages. In a three-way handshake, one side sends a four-way handshake, each side sends separate FIN and ACK messages.
combined FIN/ACK message upon receipt of a FIN. In a three-way handshake, one side sends a combined FIN/ACK
message upon receipt of a FIN.
Close Direction - Defines whether closing of connections are to Close Direction:
be initiated from the client or from the server. Defines whether closing of connections are to be initiated from
the client or from the server.
5.6.2.2 Application-Layer Setup Parameters 5.6.2.2 Application-Layer Setup Parameters
Session Type - The virtual clients/servers MUST use HTTP 1.1 or Session Type:
higher. The client and server MUST use the same HTTP version. The virtual clients/servers MUST use HTTP 1.1 or higher. The
client and server MUST use the same HTTP version.
GET requests per connection - Defines the number of HTTP 1.1 or GET requests per connection:
higher GET requests attempted per connection. Defines the number of HTTP 1.1 or higher GET requests attempted
per connection.
Object Size - Defines the number of bytes, excluding any bytes Object Size:
associated with the HTTP header, to be transferred in response to Defines the number of bytes, excluding any bytes associated with
an HTTP 1.1 or higher GET request. the HTTP header, to be transferred in response to an HTTP 1.1 or
higher GET request.
5.6.3 Procedure 5.6.3 Procedure
Each HTTP 1.1 or higher virtual client will request one or more Each HTTP 1.1 or higher virtual client will request one or more
objects from an HTTP 1.1 or higher server using one or more HTTP objects from an HTTP 1.1 or higher server using one or more HTTP GET
GET requests over each connection. The aggregate number of requests over each connection. The aggregate number of connections
connections attempted, defined by number of connections, MUST be attempted, defined by number of connections, MUST be evenly divided
evenly divided among all of the participating virtual clients. among all of the participating virtual clients.
If the virtual client(s) make multiple HTTP GET requests per If the virtual client(s) make multiple HTTP GET requests per
connection, it MUST request the same object size for each GET connection, it MUST request the same object size for each GET
request. Multiple iterations of this test may be run with objects request. Multiple iterations of this test may be run with objects of
of different sizes. different sizes.
5.6.4 Measurements 5.6.4 Measurements
5.6.4.1 Application-Layer measurements 5.6.4.1 Application-Layer measurements
Average Transfer Rate - The average transfer rate of the DUT/SUT Average Transfer Rate :
MUST be measured and shall be referenced to the requested The average transfer rate of the DUT/SUT MUST be measured and
object(s). The measurement will start on transmission of the shall be referenced to the requested object(s). The measurement
first bit of the first requested object and end on transmission will start on transmission of the first bit of the first requested
of the last bit of the last requested object. The average object and end on transmission of the last bit of the last
transfer rate, in bits per second, will be calculated using the requested object. The average transfer rate, in bits per second,
following formula: will be calculated using the following formula:
OBJECTS * OBJECTSIZE * 8 OBJECTS * OBJECTSIZE * 8
TRANSFER RATE(bit/s) = -------------------------- TRANSFER RATE (bit/s) = --------------------------
DURATION DURATION
OBJECTS - Total number of objects successfully transferred across OBJECTS - Total number of objects successfully transferred across
all connections. all connections.
OBJECTSIZE - Object size in bytes OBJECTSIZE - Object size in bytes
DURATION - Aggregate transfer time based on aforementioned time DURATION - Aggregate transfer time based on aforementioned time
references. references.
5.6.4.2 Measurements at or below the Transport-Layer 5.6.4.2 Measurements at or below the Transport-Layer
The following measurements SHOULD be performed for each The following measurements SHOULD be performed for each connection-
connection-oriented protocol: oriented protocol:
Goodput [1]:
Goodput as defined in section 3.17 of RFC 2647. Measurements MUST
only reference the protocol payload, excluding any of the protocol
header. In addition, the test instrument MUST exclude any bits
associated with the connection establishment, connection tear
down, security associations [1] or connection maintenance [1].
Goodput[1] - Goodput as defined in section 3.17 of RFC2647.
Measurements MUST only reference the protocol payload, excluding
any of the protocol header. In addition, the tester MUST exclude
any bits associated with the connection establishment, connection
tear down, security associations[1] or connection maintenance[1].
Since connection-oriented protocols require that data be Since connection-oriented protocols require that data be
acknowledged, the offered load[4] will be varying. Therefore, the acknowledged, the offered load [4] will be varying. Therefore,
tester should measure the average forwarding rate over the the test instrument should measure the average forwarding rate
duration of the test. Measurement should start on transmission of over the duration of the test. Measurement should start on
the first bit of the payload of the first datagram and end on transmission of the first bit of the payload of the first datagram
transmission of the last bit of the payload of the last datagram. and end on transmission of the last bit of the payload of the last
datagram.
Number of bytes transferred - Total payload bytes transferred. Number of bytes transferred - Total payload bytes transferred.
Number of Timeouts - Total number of timeout events. Number of Timeouts - Total number of timeout events.
Retransmitted bytes - Total number of retransmitted bytes. Retransmitted bytes - Total number of retransmitted bytes.
5.6.5 Reporting Format 5.6.5 Reporting Format
5.6.5.1 Application-Layer reporting The test report MUST conform to the reporting requirements set in
section 4, Test Setup.
The test report MUST note number of GET requests per connection 5.6.5.1 Application-Layer reporting
and object size(s).
The transfer rate results SHOULD be reported in tabular form with The test report MUST note number of GET requests per connection and
a column for each of the object sizes tested. There SHOULD be a object size(s).
row for the number and percentage of completed requests, number
and percentage of completed responses, and the resultant transfer
rate for each iteration of the test.
Failure analysis: The transfer rate results SHOULD be reported in tabular form with a
column for each of the object sizes tested. There SHOULD be a row
for the number and percentage of completed requests, number and
percentage of completed responses, and the resultant transfer rate
for each iteration of the test.
Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP The test report SHOULD indicate the number and percentage of HTTP
GET request and responses that failed to complete. GET request and responses that failed to complete.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.6.5.2 Transport-Layer and below reporting 5.6.5.2 Transport-Layer and below reporting
The test report MUST note the number of connections, close
method, close direction and the protocol for which the
measurement was made.
The results SHOULD be reported in tabular form for each of the The test report MUST note the number of connections, close method,
HTTP object sizes tested. There SHOULD be a row for the total close direction and the protocol for which the measurement was made.
bytes transferred, total timeouts, total retransmitted bytes and
and resultant goodput. Note that total bytes refers to total
datagram payload bytes transferred. The table MAY be combined
with the application layer reporting, provided the table clearly
identify the protocol for which the measurement was made.
Failure analysis: The results SHOULD be reported in tabular form for each of the HTTP
object sizes tested. There SHOULD be a row for the total bytes
transferred, total timeouts, total retransmitted bytes and and
resultant goodput. Note that total bytes refers to total datagram
payload bytes transferred. The table MAY be combined with the
application layer reporting, provided the table clearly identifies
the protocol for which the measurement was made.
Failure analysis:
The test report SHOULD indicate the number and percentage of The test report SHOULD indicate the number and percentage of
connection establishment failures as well as number and connection establishment failures as well as number and percentage
percentage of TCP tear down failures. of TCP tear down failures.
It is RECOMMENDED that the report include a graph to plot the It is RECOMMENDED that the report include a graph to plot the
distribution of both connection establishment failures and distribution of both connection establishment failures and connection
connection tear down failures. The x coordinate SHOULD be the tear down failures. The x coordinate SHOULD be the elapsed test
elapsed test time, the y coordinate SHOULD be the number of time, the y coordinate SHOULD be the number of failures for a given
failures for a given sampling period. There SHOULD be two lines sampling period. There SHOULD be two lines on the graph, one for
on the graph, one for connection failures and one for tear down connection failures and one for tear down failures. The graph MUST
failures. The graph MUST note the sampling period. note the sampling period.
5.7 Maximum HTTP Transaction Rate 5.7 Maximum HTTP Transaction Rate
5.7.1 Objective 5.7.1 Objective
Determine the maximum transaction rate the DUT/SUT can sustain. This Determine the maximum transaction rate the DUT/SUT can sustain. This
test is intended to find the maximum rate at which users can access test is intended to find the maximum rate at which users can access
objects. objects.
5.7.2 Setup Parameters 5.7.2 Setup Parameters
5.7.2.1 Transport-Layer Setup Parameters 5.7.2.1 Transport-Layer Setup Parameters
Close Method - Defines method for closing TCP connections. The Close Method:
test MUST be performed with either a three-way or four-way Defines method for closing TCP connections. The test MUST be
handshake. In a four-way handshake, each side sends separate FIN performed with either a three-way or four-way handshake. In a
and ACK messages. In a three-way handshake, one side sends a four-way handshake, each side sends separate FIN and ACK messages.
combined FIN/ACK message upon receipt of a FIN. In a three-way handshake, one side sends a combined FIN/ACK
message upon receipt of a FIN.
Close Direction - Defines whether closing of connections are to Close Direction:
be initiated from the client or from the server. Defines whether closing of connections are to be initiated from
the client or from the server.
5.7.2.2 Application-Layer Setup Parameters 5.7.2.2 Application-Layer Setup Parameters
Session Type - HTTP 1.1 or higher MUST be used for this test. The Session Type:
client and server MUST use the same HTTP version. HTTP 1.1 or higher MUST be used for this test. The client and
server MUST use the same HTTP version.
Test Duration - Time, expressed in seconds, for which the Test Duration:
virtual client(s) will sustain the attempted GET request rate. Time, expressed in seconds, for which the virtual client(s) will
It is RECOMMENDED that the duration be at least 30 seconds. sustain the attempted GET request rate. It is RECOMMENDED that
the duration be at least 30 seconds.
Requests per connection - Number of object requests per Requests per connection:
connection. Number of object requests per connection.
Object Size - Defines the number of bytes, excluding any bytes Object Size:
associated with the HTTP header, to be transferred in response to Defines the number of bytes, excluding any bytes associated with
an HTTP 1.1 or higher GET request. the HTTP header, to be transferred in response to an HTTP 1.1 or
higher GET request.
5.7.3 Procedure 5.7.3 Procedure
An iterative search algorithm MAY be used to determine the maximum This test will employ an iterative search algorithm to determine the
transaction rate that the DUT/SUT can sustain. maximum transaction rate that the DUT/SUT can sustain.
For each iteration, HTTP 1.1 or higher virtual client(s) will vary For each iteration, HTTP 1.1 or higher virtual client(s) will vary
the aggregate GET request rate offered to HTTP 1.1 or higher the aggregate GET request rate offered to HTTP 1.1 or higher
server(s). The virtual client(s) will maintain the offered request server(s). The virtual client(s) will maintain the offered request
rate for the defined test duration. rate for the defined test duration.
If the virtual client(s) make multiple HTTP GET requests per If the virtual client(s) make multiple HTTP GET requests per
connection, it MUST request the same object size for each GET connection, it MUST request the same object size for each GET
request. Multiple tests MAY be performed with different object request. Multiple tests MAY be performed with different object
sizes. sizes.
5.7.4 Measurements 5.7.4 Measurements
Maximum Transaction Rate - The maximum rate at which all Maximum Transaction Rate:
transactions -- that is all requests/responses cycles -- are The maximum rate at which all transactions, that is all
completed. requests/responses cycles, are completed.
Transaction Time - The tester SHOULD measure minimum, maximum and Transaction Time:
average transaction times. The transaction time will start when the The test instrument SHOULD measure minimum, maximum and average
virtual client issues the GET request and end when the requesting transaction times. The transaction time will start when the
virtual client receives the last bit of the requested object. virtual client issues the GET request and end when the requesting
virtual client receives the last bit of the requested object.
5.7.5 Reporting Format 5.7.5 Reporting Format
5.7.5.1 Application-Layer reporting The test report MUST conform to the reporting requirements set in
section 4, Test Setup.
The test report MUST note the test duration, object size, 5.7.5.1 Application-Layer reporting
requests per connection, minimum transaction time, maximum
transaction time, average transaction time and maximum
transaction rate measured
The intermediate results of the search algorithm MAY be reported The test report MUST note the test duration, object size, requests
in a table format with a column for each iteration. There SHOULD per connection, minimum transaction time, maximum transaction time,
be rows for the GET request attempt rate, number of requests average transaction time and maximum transaction rate measured
attempted, number and percentage of requests completed, number of
responses attempted, number and percentage of responses
completed, minimum transaction time, average transaction time and
maximum transaction time.
Version information: The intermediate results of the search algorithm MAY be reported in a
table format with a column for each iteration. There SHOULD be rows
for the GET request attempt rate, number of requests attempted,
number and percentage of requests completed, number of responses
attempted, number and percentage of responses completed, minimum
transaction time, average transaction time and maximum transaction
time.
Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.7.5.2 Transport-Layer 5.7.5.2 Transport-Layer
The test report MUST note the close method, close direction, The test report MUST note the close method, close direction, number
number of connections established and number of connections torn of connections established and number of connections torn down.
down.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported in a
in a table format with a column for each iteration. There SHOULD table format with a column for each iteration. There SHOULD be rows
be rows for the number of connections attempted, number and for the number of connections attempted, number and percentage of
percentage of connections completed, number and percentage of connections completed, number and percentage of connection tear downs
connection tear downs completed. The table MAY be combined with completed. The table MAY be combined with the application layer
the application layer reporting, provided the table identify this reporting, provided the table identify this as transport layer
as transport layer measurement. measurement.
5.8 Illegal Traffic Handling 5.8 Illegal Traffic Handling
5.8.1 Objective 5.8.1 Objective
To characterize the behavior of the DUT/SUT when presented with a To characterize the behavior of the DUT/SUT when presented with a
combination of both legal and Illegal[1] traffic. Note that Illegal combination of both legal and Illegal [1] traffic. Note that Illegal
traffic does not refer to an attack, but traffic which has been traffic does not refer to an attack, but traffic which has been
explicitly defined by a rule(s) to drop. explicitly defined by a rule(s) to drop.
5.8.2 Setup Parameters 5.8.2 Setup Parameters
Setup parameters will use the same parameters as specified in the Setup parameters will use the same parameters as specified in the
HTTP transfer rate test(Section 5.6.2). In addition, the following HTTP transfer rate test (Section 5.6.2). In addition, the following
setup parameters MUST be defined: setup parameters MUST be defined:
Illegal traffic percentage - Percentage of HTTP 1.1 or higher Illegal traffic percentage:
connections which have been explicitly defined in a rule(s) to drop. Percentage of HTTP 1.1 or higher connections which have been
explicitly defined in a rule(s) to drop.
5.8.3 Procedure 5.8.3 Procedure
Each HTTP 1.1 or higher client will request one or more objects from Each HTTP 1.1 or higher client will request one or more objects from
an HTTP 1.1 or higher server using one or more HTTP GET requests an HTTP 1.1 or higher server using one or more HTTP GET requests over
over each connection. The aggregate number of connections attempted, each connection. The aggregate number of connections attempted,
defined by number of connections, MUST be evenly divided among all defined by number of connections, MUST be evenly divided among all of
of the participating virtual clients. the participating virtual clients.
The virtual client(s) MUST offer the connection requests, both legal The virtual client(s) MUST offer the connection requests, both legal
and illegal, in an evenly distributed manner. Many firewalls have and illegal, in an evenly distributed manner. Many firewalls have
the capability to filter on different traffic criteria( IP the capability to filter on different traffic criteria (IP addresses,
addresses, Port numbers, etc). Multiple iterations of this test MAY Port numbers, etc.). Multiple iterations of this test MAY be run
be run with the DUT/SUT configured to filter on different traffic with the DUT/SUT configured to filter on different traffic criteria.
criteria.
5.8.4 Measurements 5.8.4 Measurements
The same measurements as defined in HTTP transfer rate test(Section The same measurements as defined in HTTP transfer rate test (Section
5.6.4) SHOULD be performed. Any forwarding rate measurements MUST 5.6.4) SHOULD be performed. Any forwarding rate measurements MUST
only include bits which are associated with legal traffic. only include bits which are associated with legal traffic.
5.8.5 Reporting Format 5.8.5 Reporting Format
Test reporting format SHOULD be the same as specified in the HTTP Test reporting format SHOULD be the same as specified in the HTTP
transfer rate test(Section 5.6.5). transfer rate test (Section 5.6.5).
In addition, the report MUST note the percentage of illegal HTTP In addition, the report MUST note the percentage of illegal HTTP
connections. connections.
Failure analysis: Failure analysis:
Test report MUST note the number and percentage of illegal
Test report MUST note the number and percentage of illegal connections that were allowed by the DUT/SUT.
connections that were allowed by the DUT/SUT.
5.9 IP Fragmentation Handling 5.9 IP Fragmentation Handling
5.9.1 Objective 5.9.1 Objective
To determine the performance impact when the DUT/SUT is presented To determine the performance impact when the DUT/SUT is presented
with IP fragmented traffic. IP packets which have been with IP fragmented traffic. IP packets which have been fragmented,
fragmented, due to crossing a network that supports a smaller due to crossing a network that supports a smaller MTU (Maximum
MTU(Maximum Transmission Unit) than the actual IP packet, may Transmission Unit) than the actual IP packet, may require the
require the firewall to perform re-assembly prior to the rule set firewall to perform re-assembly prior to the rule set being applied.
being applied.
While IP fragmentation is a common form of attack, either on the While IP fragmentation is a common form of attack, either on the
firewall itself or on internal hosts, this test will focus on firewall itself or on internal hosts, this test will focus on
determining how the additional processing associated with the determining how the additional processing associated with the re-
re-assembly of the packets have on the forwarding rate of the assembly of the packets have on the forwarding rate of the DUT/SUT.
DUT/SUT. RFC 1858 addresses some fragmentation attacks that RFC 1858 addresses some fragmentation attacks that get around IP
get around IP filtering processes used in routers and hosts. filtering processes used in routers and hosts.
5.9.2 Setup Parameters 5.9.2 Setup Parameters
The following parameters MUST be defined. The following parameters MUST be defined.
5.9.2.1 Non-Fragmented Traffic Parameters 5.9.2.1 Non-Fragmented Traffic Parameters
Setup parameters will be the same as defined in the HTTP transfer Setup parameters will be the same as defined in the HTTP transfer
rate test(Sections 5.6.2.1 and 5.6.2.2). rate test (Sections 5.6.2.1 and 5.6.2.2).
5.9.2.2 Fragmented Traffic Parameters 5.9.2.2 Fragmented Traffic Parameters
Packet size - Number of bytes in the IP/UDP packet, exclusive of Packet size:
link-layer headers and checksums, prior to fragmentation. Number of bytes in the IP/UDP packet, exclusive of link-layer
headers and checksums, prior to fragmentation.
MTU - Maximum transmission unit, expressed in bytes. For testing MTU:
Maximum transmission unit, expressed in bytes. For testing
purposes, this MAY be configured to values smaller than the MTU purposes, this MAY be configured to values smaller than the MTU
supported by the link layer. supported by the link layer.
Intended Load - Intended load, expressed as percentage of media Intended Load:
utilization. Intended load, expressed as percentage of media utilization.
5.9.3 Procedure 5.9.3 Procedure
Each HTTP 1.1 or higher client will request one or more objects from Each HTTP 1.1 or higher client will request one or more objects from
an HTTP 1.1 or higher server using one or more HTTP GET requests an HTTP 1.1 or higher server using one or more HTTP GET requests over
over each connection. The aggregate number of connections attempted, each connection. The aggregate number of connections attempted,
defined by number of connections, MUST be evenly divided among all defined by number of connections, MUST be evenly divided among all of
of the participating virtual clients. If the virtual client(s) make the participating virtual clients. If the virtual client(s) make
multiple HTTP GET requests per connection, it MUST request the same multiple HTTP GET requests per connection, it MUST request the same
object size for each GET request. object size for each GET request.
A tester attached to the unprotected side of the network, will offer A test instrument attached to the unprotected side of the network,
a unidirectional stream of unicast fragmented IP/UDP traffic, will offer a unidirectional stream of unicast fragmented IP/UDP
targeting a server attached to either the protected or DMZ segment. traffic, targeting a server attached to either the protected or DMZ
The tester MUST offer the unidirectional stream over the duration of segment. The test instrument MUST offer the unidirectional stream
the test -- that is, duration over which the HTTP traffic is being over the duration of the test, that is, duration over which the HTTP
offered. traffic is being offered.
Baseline measurements SHOULD be performed with IP filtering deny Baseline measurements SHOULD be performed with IP filtering deny
rule(s) to filter fragmented traffic. If the DUT/SUT has logging rule(s) to filter fragmented traffic. If the DUT/SUT has logging
capability, the log SHOULD be checked to determine if it contains capability, the log SHOULD be checked to determine if it contains the
the correct information regarding the fragmented traffic. correct information regarding the fragmented traffic.
The test SHOULD be repeated with the DUT/SUT rule set changed to The test SHOULD be repeated with the DUT/SUT rule set changed to
allow the fragmented traffic through. When running multiple allow the fragmented traffic through. When running multiple
iterations of the test, it is RECOMMENDED to vary the MTU while iterations of the test, it is RECOMMENDED to vary the MTU while
keeping all other parameters constant. keeping all other parameters constant.
Then setup the DUT/SUT to the policy or rule set the manufacturer Then setup the DUT/SUT to the policy or rule set the manufacturer
required to be defined to protect against fragmentation attacks and required to be defined to protect against fragmentation attacks and
repeat the measurements outlined in the baseline procedures. repeat the measurements outlined in the baseline procedures.
5.9.4 Measurements 5.9.4 Measurements
Tester SHOULD perform the same measurements as defined in HTTP Test instrument SHOULD perform the same measurements as defined in
test(Section 5.6.4). HTTP test (Section 5.6.4).
Transmitted UDP/IP Packets - Number of UDP packets transmitted by Transmitted UDP/IP Packets:
client. Number of UDP packets transmitted by client.
Received UDP/IP Packets - Number of UDP/IP Packets received by Received UDP/IP Packets:
server. Number of UDP/IP Packets received by server.
5.9.5 Reporting Format 5.9.5 Reporting Format
5.10.1 Non-Fragmented Traffic 5.9.5.1 Non-Fragmented Traffic
The test report SHOULD be the same as described in section 5.6.5. The test report SHOULD be the same as described in section 5.6.5.
Note that any forwarding rate measurements for the HTTP traffic Note that any forwarding rate measurements for the HTTP traffic
excludes any bits associated with the fragmented traffic which excludes any bits associated with the fragmented traffic which may be
may be forward by the DUT/SUT. forward by the DUT/SUT.
5.9.2 Fragmented Traffic 5.9.5.2 Fragmented Traffic
The test report MUST note the packet size, MTU size, intended The test report MUST note the packet size, MTU size, intended load,
load, number of UDP/IP packets transmitted and number of UDP/IP number of UDP/IP packets transmitted and number of UDP/IP packets
packets forwarded. The test report SHOULD also note whether or forwarded. The test report SHOULD also note whether or not the
not the DUT/SUT forwarded the offered UDP/IP traffic fragmented. DUT/SUT forwarded the offered UDP/IP traffic fragmented.
5.10 Latency 5.10 Latency
5.10.1 Objective 5.10.1 Objective
To determine the latency of network-layer or application-layer data To determine the latency of network-layer or application-layer data
traversing the DUT/SUT. RFC 1242 [3] defines latency. traversing the DUT/SUT. RFC 1242 [3] defines latency.
5.10.2 Setup Parameters 5.10.2 Setup Parameters
The following parameters MUST be defined: The following parameters MUST be defined:
5.10.2.1 Network-layer Measurements 5.10.2.1 Network-layer Measurements
Packet size, expressed as the number of bytes in the IP packet, Packet size, expressed as the number of bytes in the IP packet,
exclusive of link-layer headers and checksums. exclusive of link-layer headers and checksums.
Intended load, expressed as percentage of media utilization. Intended load, expressed as percentage of media utilization.
Test duration, expressed in seconds. Test duration, expressed in seconds.
Test instruments MUST generate packets with unique timestamp The test instruments MUST generate packets with unique timestamp
signatures. signatures.
5.10.2.2 Application-layer Measurements 5.10.2.2 Application-layer Measurements
Object Size - Defines the number of bytes, excluding any bytes Object Size:
associated with the HTTP header, to be transferred in response to Defines the number of bytes, excluding any bytes associated with
an HTTP 1.1 or higher GET request. Testers SHOULD use the minimum the HTTP header, to be transferred in response to an HTTP 1.1 or
object size supported by the media, but MAY use other object higher GET request. The minimum object size supported by the
sizes as well. media SHOULD be used, but other object sizes MAY be used as well.
Connection type. The tester MUST use one HTTP 1.1 or higher Connection type:
connection for latency measurements. The test instrument MUST use one HTTP 1.1 or higher connection for
latency measurements.
Number of objects requested. Number of objects requested.
Number of objects transferred. Number of objects transferred.
Test duration, expressed in seconds. Test duration, expressed in seconds.
Test instruments MUST generate packets with unique timestamp Test instruments MUST generate packets with unique timestamp
signatures. signatures.
5.10.3 Network-layer procedure 5.10.3 Network-layer procedure
A client will offer a unidirectional stream of unicast packets to A client will offer a unidirectional stream of unicast packets to a
a server. The packets MUST use a connectionless protocol like IP server. The packets MUST use a connectionless protocol like IP or
or UDP/IP. UDP/IP.
The tester MUST offer packets in a steady state. As noted in the The test instrument MUST offer packets in a steady state. As noted
latency discussion in RFC 2544 [2], latency measurements MUST be in the latency discussion in RFC 2544 [2], latency measurements MUST
taken at the throughput level -- that is, at the highest offered be taken at the throughput level, that is, at the highest offered
load with zero packet loss. Measurements taken at the throughput load with zero packet loss. Measurements taken at the throughput
level are the only ones that can legitimately be termed latency. level are the only ones that can legitimately be termed latency.
It is RECOMMENDED that implementers use offered loads not only at It is RECOMMENDED that implementers use offered loads not only at the
the throughput level, but also at load levels that are less than throughput level, but also at load levels that are less than or
or greater than the throughput level. To avoid confusion with greater than the throughput level. To avoid confusion with existing
existing terminology, measurements from such tests MUST be labeled terminology, measurements from such tests MUST be labeled as delay
as delay rather than latency. rather than latency.
It is RECOMMENDED to perform the latency measurements with It is RECOMMENDED to perform the latency measurements with different
different packet sizes. When testing with different packet sizes packet sizes. When testing with different packet sizes the DUT/SUT
the DUT/SUT configuration MUST remain the same. configuration MUST remain the same.
If desired, the tester MAY use a step test in which offered loads If desired, a step test MAY be used in which offered loads increment
increment or decrement through a range of load levels. or decrement through a range of load levels.
The duration of the test portion of each trial MUST be at least The duration of the test portion of each trial MUST be at least 30
30 seconds. seconds.
5.10.4 Application layer procedure 5.10.4 Application layer procedure
An HTTP 1.1 or higher client will request one or more objects from An HTTP 1.1 or higher client will request one or more objects from an
an HTTP 1.1 or higher server using one or more HTTP GET requests. If HTTP 1.1 or higher server using one or more HTTP GET requests. If
the tester makes multiple HTTP GET requests, it MUST request the the test instrument makes multiple HTTP GET requests, it MUST request
same-sized object each time. Testers may run multiple iterations of the same-sized object each time. Multiple iterations of this test
this test with objects of different sizes. may be performed with objects of different sizes.
Implementers MAY configure the tester to run for a fixed duration. Implementers MAY configure the test instrument to run for a fixed
In this case, the tester MUST report the number of objects requested duration. In this case, the test instrument MUST report the number
and returned for the duration of the test. For fixed-duration tests of objects requested and returned for the duration of the test. For
it is RECOMMENDED that the duration be at least 30 seconds. fixed-duration tests it is RECOMMENDED that the duration be at least
30 seconds.
5.10.5 Measurements 5.10.5 Measurements
Minimum delay - The smallest delay incurred by data traversing the Minimum delay:
DUT/SUT at the network layer or application layer, as appropriate. The smallest delay incurred by data traversing the DUT/SUT at the
network layer or application layer, as appropriate.
Maximum delay - The largest delay incurred by data traversing the Maximum delay:
DUT/SUT at the network layer or application layer, as appropriate. The largest delay incurred by data traversing the DUT/SUT at the
network layer or application layer, as appropriate.
Average delay - The mean of all measurements of delay incurred by Average delay:
data traversing the DUT/SUT at the network layer or application The mean of all measurements of delay incurred by data traversing
layer, as appropriate. the DUT/SUT at the network layer or application layer, as
appropriate.
Delay distribution - A set of histograms of all delay measurements Delay distribution:
observed for data traversing the DUT/SUT at the network layer or A set of histograms of all delay measurements observed for data
application layer, as appropriate. traversing the DUT/SUT at the network layer or application layer,
as appropriate.
5.10.6 Network-layer reporting format 5.10.6 Network-layer reporting format
The test report MUST note the packet size(s), offered load(s) and The test report MUST note the packet size(s), offered load(s) and
test duration used. test duration used. In addition, the test report MUST conform to the
reporting requirements set in section 4, Test Setup.
The latency results SHOULD be reported in the format of a table with The latency results SHOULD be reported in the format of a table with
a row for each of the tested packet sizes. There SHOULD be columns a row for each of the tested packet sizes. There SHOULD be columns
for the packet size, the intended rate, the offered rate, and the for the packet size, the intended rate, the offered rate, and the
resultant latency or delay values for each test. resultant latency or delay values for each test.
5.10.7 Application-layer reporting format 5.10.7 Application-layer reporting format
The test report MUST note the object size(s) and number of requests The test report MUST note the object size(s) and number of requests
and responses completed. If applicable, the report MUST note the and responses completed. If applicable, the report MUST note the
test duration if a fixed duration was used. test duration if a fixed duration was used. In addition, the test
report MUST conform to the reporting requirements set in section 4,
Test Setup.
The latency results SHOULD be reported in the format of a table with The latency results SHOULD be reported in the format of a table with
a row for each of the object sizes. There SHOULD be columns for the a row for each of the object sizes. There SHOULD be columns for the
object size, the number of completed requests, the number of object size, the number of completed requests, the number of
completed responses, and the resultant latency or delay values for completed responses, and the resultant latency or delay values for
each test. each test.
Failure analysis: Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP
The test report SHOULD indicate the number and percentage of HTTP GET request or responses that failed to complete within the test
GET request or responses that failed to complete within the test duration.
duration.
Version information: Version information:
The test report MUST note the version of HTTP client and server.
The test report MUST note the version of HTTP client and server.
6. References 6. References
Normative References 6.1 Normative References
[1] D. Newman, "Benchmarking Terminology for Firewall Devices", [1] Newman, D., "Benchmarking Terminology for Firewall Devices", RFC
RFC 2647, August 1999. 2647, August 1999.
[2] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network [2] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Interconnect Devices," RFC 2544, March 1999. Network Interconnect Devices", RFC 2544, March 1999.
[3] S. Bradner, editor. "Benchmarking Terminology for Network [3] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices," RFC 1242, July 1991. Interconnection Devices", RFC 1242, July 1991.
[4] Mandeville, R., "Benchmarking Terminology for LAN Switching [4] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998. Devices", RFC 2285, February 1998.
[5] Mandeville, R., Perser,J., "Benchmarking Methodology for LAN [5] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN
Switching Devices", RFC 2889, August 2000. Switching Devices", RFC 2889, August 2000.
Informative References 6.2 Informative References
[6] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter, [6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L.,
P. Leach, T. Berners-Lee , "Hypertext Transfer Protocol - Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -
HTTP/1.1", RFC 2616, June 1999. HTTP/1.1", RFC 2616, June 1999.
[7] David C. Clark, "IP Datagram Reassembly Algorithm", RFC 815 , [7] Clark, D., "IP Datagram Reassembly Algorithm", RFC 815, July
July 1982. 1982.
[8] Postel, J. (ed.), "Transmission Control Protocol", RFC 793, [8] Postel, J., "Transmission Control Protocol", STD 7, RFC 793,
USC/Information Sciences Institute, September 1981. September 1981.
7. Security Considerations 7. Security Considerations
The primary goal of this document is to provide methodologies in The primary goal of this document is to provide methodologies in
benchmarking firewall performance. While there is some overlap benchmarking firewall performance. While there is some overlap
between performance and security issues, assessment of firewall between performance and security issues, assessment of firewall
security is outside the scope of this document. security is outside the scope of this document.
8. Acknowledgement APPENDIX A: HTTP (HyperText Transfer Protocol)
Funding for the RFC Editor function is currently provided by the The most common versions of HTTP in use today are HTTP/1.0 and
Internet Society. HTTP/1.1 with the main difference being in regard to persistent
connections. HTTP 1.0, by default, does not support persistent
connections. A separate TCP connection is opened up for each GET
request the client wants to initiate and closed after the requested
object transfer is completed. While some implementations HTTP/1.0
supports persistence through the use of a keep-alive, there is no
official specification for how the keep-alive operates. In addition,
HTTP 1.0 proxies do support persistent connection as they do not
recognize the connection header.
9. Authors' Addresses HTTP/1.1, by default, does support persistent connection and is
therefore the version that is referenced in this methodology. Proxy
based DUT/SUTs may monitor the TCP connection and after a timeout,
close the connection if no activity is detected. The duration of
this timeout is not defined in the HTTP/1.1 specification and will
vary between DUT/SUTs. If the DUT/SUT closes inactive connections,
the aging timer on the DUT SHOULD be configured for a duration that
exceeds the test time.
While this document cannot foresee future changes to HTTP and it
impact on the methodologies defined herein, such changes should be
accommodated for so that newer versions of HTTP may be used in
benchmarking firewall performance.
APPENDIX B: Connection Establishment Time Measurements
Some connection oriented protocols, such as TCP, involve an odd
number of messages when establishing a connection. In the case of
proxy based DUT/SUTs, the DUT/SUT will terminate the connection,
setting up a separate connection to the server. Since, in such
cases, the test instrument does not own both sides of the connection,
measurements will be made two different ways. While the following
describes the measurements with reference to TCP, the methodology may
be used with other connection oriented protocols which involve an odd
number of messages.
When testing non-proxy based DUT/SUTs , the establishment time shall
be directly measured and is considered to be from the time the first
bit of the first SYN packet is transmitted by the client to the time
the last bit of the final ACK in the three-way handshake is received
by the target server.
If the DUT/SUT is proxy based, the connection establishment time is
considered to be from the time the first bit of the first SYN packet
is transmitted by the client to the time the client transmits the
first bit of the first acknowledged TCP datagram (t4-t0 in the
following timeline).
t0: Client sends a SYN.
t1: Proxy sends a SYN/ACK.
t2: Client sends the final ACK.
t3: Proxy establishes separate connection with server.
t4: Client sends TCP datagram to server.
*t5: Proxy sends ACK of the datagram to client.
* While t5 is not considered part of the TCP connection
establishment, acknowledgement of t4 must be received for the
connection to be considered successful.
APPENDIX C: Connection Tear Down Time Measurements
While TCP connections are full duplex, tearing down of such
connections are performed in a simplex fashion, that is, FIN segments
are sent by each host/device terminating each side of the TCP
connection.
When making connection tear down times measurements, such
measurements will be made from the perspective of the entity, that
is, virtual client/server initiating the connection tear down
request. In addition, the measurement will be performed in the same
manner, independent of whether or not the DUT/SUT is proxy-based. The
connection tear down will be considered the interval between the
transmission of the first bit of the first TCP FIN packet transmitted
by the virtual client or server, whichever is applicable, requesting
a connection tear down to receipt of the last bit of the
corresponding ACK packet on the same virtual client/server interface.
Authors' Addresses
Brooks Hickman Brooks Hickman
Spirent Communications Spirent Communications
26750 Agoura Road 26750 Agoura Road
Calabasas, CA 91302 Calabasas, CA 91302
USA USA
Phone: + 1 818 676 2412 Phone: + 1 818 676 2412
Email: brooks.hickman@spirentcom.com EMail: brooks.hickman@spirentcom.com
David Newman David Newman
Network Test Inc. Network Test Inc.
31324 Via Colinas, Suite 113 31324 Via Colinas, Suite 113
Westlake Village, CA 91362-6761 Westlake Village, CA 91362-6761
USA USA
Phone: + 1 818 889-0011 Phone: + 1 818 889-0011
Email: dnewman@networktest.com EMail: dnewman@networktest.com
Saldju Tadjudin Saldju Tadjudin
Spirent Communications Spirent Communications
26750 Agoura Road 26750 Agoura Road
Calabasas, CA 91302 Calabasas, CA 91302
USA USA
Phone: + 1 818 676 2468 Phone: + 1 818 676 2468
Email: Saldju.Tadjudin@spirentcom.com EMail: Saldju.Tadjudin@spirentcom.com
Terry Martin Terry Martin
GVNW Consulting Inc. GVNW Consulting Inc.
8050 SW Warm Springs Road 8050 SW Warm Springs Road
Tualatin Or. 97062 Tualatin Or. 97062
USA USA
Phone: + 1 503 612 4422 Phone: + 1 503 612 4422
Email: tmartin@gvnw.com EMail: tmartin@gvnw.com
APPENDIX A: HTTP (HyperText Transfer Protocol)
The most common versions of HTTP in use today are HTTP/1.0 and
HTTP/1.1 with the main difference being in regard to persistent
connections. HTTP 1.0, by default, does not support persistent
connections. A separate TCP connection is opened up for each
GET request the client wants to initiate and closed after the
requested object transfer is completed. While some implementations
HTTP/1.0 supports persistence through the use of a keep-alive,
there is no official specification for how the keep-alive operates.
In addition, HTTP 1.0 proxies do support persistent connection as
they do not recognize the connection header.
HTTP/1.1, by default, does support persistent connection and
is therefore the version that is referenced in this methodology.
Proxy based DUT/SUTs may monitor the TCP connection and after a
timeout, close the connection if no activity is detected. The
duration of this timeout is not defined in the HTTP/1.1
specification and will vary between DUT/SUTs. If the DUT/SUT
closes inactive connections, the aging timer on the DUT SHOULD
be configured for a duration that exceeds the test time.
While this document cannot foresee future changes to HTTP
and it impact on the methodologies defined herein, such
changes should be accommodated for so that newer versions of
HTTP may be used in benchmarking firewall performance.
APPENDIX B: Connection Establishment Time Measurements
Some connection oriented protocols, such as TCP, involve an odd
number of messages when establishing a connection. In the case of
proxy based DUT/SUTs, the DUT/SUT will terminate the connection,
setting up a separate connection to the server. Since, in such
cases, the tester does not own both sides of the connection,
measurements will be made two different ways. While the following
describes the measurements with reference to TCP, the methodology
may be used with other connection oriented protocols which involve
an odd number of messages.
When testing non-proxy based DUT/SUTs , the establishment time shall
be directly measured and is considered to be from the time the first
bit of the first SYN packet is transmitted by the client to the
time the last bit of the final ACK in the three-way handshake is
received by the target server.
If the DUT/SUT is proxy based, the connection establishment time is
considered to be from the time the first bit of the first SYN packet
is transmitted by the client to the time the client transmits the
first bit of the first acknowledged TCP datagram(t4-t0 in the
following timeline).
t0: Client sends a SYN.
t1: Proxy sends a SYN/ACK.
t2: Client sends the final ACK.
t3: Proxy establishes separate connection with server.
t4: Client sends TCP datagram to server.
*t5: Proxy sends ACK of the datagram to client.
* While t5 is not considered part of the TCP connection
establishment, acknowledgement of t4 must be received for the
connection to be considered successful.
APPENDIX C: Connection Tear Time Measurements
While TCP connections are full duplex, tearing down of such
connections are performed in a simplex fashion -- that is - FIN
segments are sent by each host/device terminating each side of
the TCP connection.
When making connection tear down times measurements, such
measurements will be made from the perspective of the entity - that
is -- virtual client/server initiating the connection tear down
request. In addition, the measurement will be performed in the same
manner, independent of whether or not the DUT/SUT is proxy-based.
The connection tear down will be considered the interval between the
transmission of the first bit of the first TCP FIN packet transmitted
by the virtual client or server, whichever is applicable, requesting
a connection tear down to receipt of the last bit of the
corresponding ACK packet on the same virtual client/server interface.
Full Copyright Statement Full Copyright Statement
Copyright (C) The Internet Society (2002). All Rights Reserved. Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph kind, provided that the above copyright notice and this paragraph are
are included on all such copies and derivative works. However, this included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than followed, or as required to translate it into languages other than
english. The limited permissions granted above are perpetual and English.
will not be revoked by the Internet Society or its successors or
assigns. This document and the information contained herein is The limited permissions granted above are perpetual and will not be
provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE revoked by the Internet Society or its successors or assigns.
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF This document and the information contained herein is provided on an
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
 End of changes. 271 change blocks. 
791 lines changed or deleted 813 lines changed or added

This html diff was produced by rfcdiff 1.41. The latest version is available from http://tools.ietf.org/tools/rfcdiff/