draft-ietf-bmwg-firewall-01.txt   draft-ietf-bmwg-firewall-02.txt 
Network Working Group Terry Martin
Internet-Draft M2networx INC
Expiration Date: B. Hickman
Netcom Systems
November 2000
Benchmarking Methodology for Firewalls Network Working Group Brooks Hickman
<draft-ietf-bmwg-firewall-01.txt> Internet-Draft Spirent Communications
Expiration Date: December 2001 David Newman
Network Test
Terry Martin
M2networx INC
June 2001
Benchmarking Methodology for Firewall Performance
<draft-ietf-bmwg-firewall-02.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six
and may be updated, replaced, or obsoleted by other documents at any months and may be updated, replaced, or obsoleted by other documents
time. It is inappropriate to use Internet-Drafts as reference at any time. It is inappropriate to use Internet-Drafts as
material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.1 Test Considerations . . . . . . . . . . . . . . . . . . . . . 4 4.1 Test Considerations . . . . . . . . . . . . . . . . . . 3
4.1.1 Virtual Client/Servers . . . . . . . . . . . . . . . . . . 4 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 3
4.1.2 Test Traffic Requirements . . . . . . . . . . . . . . . . 4 4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4
4.1.3 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 5 4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 4
4.1.4 Multiple Client/Server Testing . . . . . . . . . . . . . . 5 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5
4.1.5 NAT(Network Address Translation) . . . . . . . . . . . . . 6 4.6 NAT(Network Address Translation) . . . . . . . . . . . . 5
4.1.6 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 6 4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 5
4.1.7 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6 4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 5
4.1.8 Authentication . . . . . . . . . . . . . . . . . . . . . . 6 4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6
5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . . . 7 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6
5.1 Concurrent Connection Capacity . . . . . . . . . . . . . . . . 7 5.1 Concurrent Connection Capacity . . . . . . . . . . . . . 6
5.2 Maximum Connection Rate . . . . . . . . . . . . . . . . . . . 8 5.2 Maximum Connection Setup Rate . . . . . . . . . . . . . . 7
5.3 Connection Establishment Time . . . . . . . . . . . . . . . . 10 5.3 Connection Establishment Time . . . . . . . . . . . . . . 9
5.4 Denial Of Service Handling . . . . . . . . . . . . . . . . . . 11 5.4 Connection Teardown Time . . . . . . . . . . . . . . . . 11
5.5 Single Application Goodput . . . . . . . . . . . . . . . . . . 12 5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 13
5.5.1 FTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 12 5.6 HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.5.2 SMTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 14 5.7 IP Fragmentation Handling . . . . . . . . . . . . . . . . 16
5.5.3 HTTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 15 5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 18
5.6 Concurrent Application Goodput . . . . . . . . . . . . . . . . 17 5.9 Latency . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.7 Illegal Traffic Handling . . . . . . . . . . . . . . . . . . . 19 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.8 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . 22
Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B. References . . . . . . . . . . . . . . . . . . . . . . . . 23
A. File Transfer Protocol(FTP) . . . . . . . . . . . . . . . . . . . 19
A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 19
A.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 20
A.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 20
B. Simple Mail Transfer Protocol(SMTP) . . . . . . . . . . . . . . . 21
B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 21
B.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 21
B.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 22
C. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . . . . . 22
C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 22
C.2 Version Considerations . . . . . . . . . . . . . . . . . . . . 23
C.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 23
E. TCP establishment/teardown . . . . . . . . . . . . . . . . . . . . 23
D. GoodPut Measurements . . . . . . . . . . . . . . . . . . . . . . . 23
F. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1. Introduction 1. Introduction
This document is intended to provide methodology for the benchmarking This document provides methodologies for the performance
of firewalls. It provides methodologies for benchmarking forwarding benchmarking of firewalls. It provides methodologies in four areas:
performance, connection performance, latency and filtering. In forwarding, connection, latency and filtering. In addition to
addition to defining the tests, this document also describes specific defining the tests, this document also describes specific formats
formats for reporting the results of the tests. for reporting the results of the tests.
A previous document, "Benchmarking Terminology for Firewall A previous document, "Benchmarking Terminology for Firewall
Performance" [1], defines many of the terms that are used in this Performance" [1], defines many of the terms that are used in this
document. The terminology document SHOULD be consulted before document. The terminology document SHOULD be consulted before
attempting to make use of this document. attempting to make use of this document.
2. Requirements 2. Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
document are to be interpreted as described in RFC 2119. this document are to be interpreted as described in RFC 2119.
3. Scope 3. Scope
Firewalls can provide a single point of defense between two Firewalls can provide a single point of defense between networks.
networks--it protects one network from the other. Usually, a firewall Usually, a firewall protects private networks from the public or
protects the company's private network from the public or shared shared networks to which it is connected. A firewall can be as
networks to which it is connected. A firewall can be as simple as a simple as a device that filters different packets or as complex
device that filters different packets or as complex as a group of as a group of devices that combine packet filtering and
devices providing solutions that offers combined packet filtering application-level proxy or network translation services. This RFC
and application-level proxy or network translation services. This RFC will focus on developing benchmark testing of DUT/SUTs, wherever
will focus on developing benchmark testing of systems from an possible, independent of their implementation.
application perspective and will be developed independent of any
firewall implementation. These tests will evaluate the ability of
firewall devices to control and manage applications services used
by today's businesses such as applications like the World Wide Web,
File transfer procedures and e-mail.
Even through there are many different control methods of managing
application level being implemented, this RFC does not condone or
promote any aforementioned process or procedure. It's goal is to
present a procedure that will evaluate firewall performance
independent of their implementation.
4. Test Setup 4. Test Setup
Test configurations defined in this document will be confined to Test configurations defined in this document will be confined to
dual-homed and tri-homed as shown in figure 1 and figure 2 dual-homed and tri-homed as shown in figure 1 and figure 2
respectively. respectively.
Firewalls employing Dual-Homed configurations connect two networks. Firewalls employing dual-homed configurations connect two networks.
One interface of the firewall is attached to the unprotected network, One interface of the firewall is attached to the unprotected
typically the public network(i.e. - Internet). The other interface is network, typically the public network(Internet). The other interface
connected to the protected network, typically the internal LAN. In is connected to the protected network, typically the internal LAN.
the case of Dual-Homed configurations, servers which are made
In the case of dual-homed configurations, servers which are made
accessible to the public(Unprotected) network are attached to the accessible to the public(Unprotected) network are attached to the
private(Protected) network. private(Protected) network.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Servers/ |----| | | |------| Servers/ | | Servers/ |----| | | |------| Servers/ |
| Clients | | | | | | Clients | | Clients | | | | | | Clients |
| | |-------| DUT/SUT |--------| | | | | |-------| DUT/SUT |--------| | |
+----------+ | | | | +----------+ +----------+ | | | | +----------+
| +----------+ | | +----------+ |
Protected | | Unprotected Protected | | Unprotected
Network Network Network Network
Figure 1(Dual-Homed) Figure 1(Dual-Homed)
Tri-homed[1] configurations employ a third segment called a DMZ. With Tri-homed[1] configurations employ a third segment called a DMZ.
tri-homed configurations, servers accessible to the public network With tri-homed configurations, servers accessible to the public
are attached to the DMZ. Tri-Homed configurations offer additional network are attached to the DMZ. Tri-Homed configurations offer
security by separating server accessible to the public network from additional security by separating server accessible to the public
internal hosts. network from internal hosts.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Clients |----| | | |------| Servers/ | | Clients |----| | | |------| Servers/ |
| | | | | | | Clients | | | | | | | | Clients |
+----------+ |-------| DUT/SUT |--------| | | +----------+ |-------| DUT/SUT |--------| | |
| | | | +----------+ | | | | +----------+
| +----------+ | | +----------+ |
Protected | | | Unprotected Protected | | | Unprotected
Network | Network Network | Network
skipping to change at page 4, line 30 skipping to change at page 3, line 51
+-----------+ +-----------+
| | | |
| Servers | | Servers |
| | | |
+-----------+ +-----------+
Figure 2(Tri-Homed) Figure 2(Tri-Homed)
4.1 Test Considerations 4.1 Test Considerations
4.1.1 Virtual Clients/Servers 4.2 Virtual Clients/Servers
Since firewall testing may involve data sources which emulate multiple Since firewall testing may involve data sources which emulate
users or hosts, the methodology uses the terms virtual clients/servers. multiple users or hosts, the methodology uses the terms virtual
For these firewall tests, virtual clients/servers specify application clients/servers. For these firewall tests, virtual clients/servers
layer entities which may not be associated with a unique physical specify application layer entities which may not be associated with
interface. For example, four virtual clients may originate from the a unique physical interface. For example, four virtual clients may
same data source[1]. The test report SHOULD indicate the number of originate from the same data source[1]. The test report SHOULD
virtual clients and virtual servers participating in the test on a per indicate the number of virtual clients and virtual servers
interface(See 4.1.3) basis. participating in the test on a per interface(See 4.1.3) basis.
Need to include paragraph for synchronize start of test. Data sources Testers MUST synchronize all data sources participating in a test.
MUST be synchronized to start initiating connections within a specified
time of each other.
4.1.2 Test Traffic Requirements 4.3 Test Traffic Requirements
While the function of a firewall is to enforce access control While the function of a firewall is to enforce access control
policies, the criteria by which those policies are defined vary policies, the criteria by which those policies are defined vary
depending on the implementation. Firewalls may use network layer depending on the implementation. Firewalls may use network layer,
and/or, in many cases, application-layer criteria to make transport layer or, in many cases, application-layer criteria to
access-control decisions. Therefore, the test equipment used to make access-control decisions. Therefore, the test equipment used to
benchmark the DUT/SUT performance MUST consist of real clients and benchmark the DUT/SUT performance MUST consist of real clients and
servers generating legitimate layer 7 conversations. servers generating legitimate layer seven conversations.
The tests defined in this document use HTTP, FTP, and SMTP sessions For the purposes of benchmarking firewall performance, HTTP 1.1
for benchmarking the performance of the DUT/SUT. Other layer 7 will be referenced in this document, although the methodologies
conversations are outside the scope of this document. See appendices may be used as a template for benchmarking with other applications.
for specific information regarding the transactions involved in Since testing may involve proxy based DUT/SUTs, HTTP version
establishing/tearing down connections as well as object formats considerations are discussed in appendix A.
for each of the aforementioned protocols.
4.1.3 DUT/SUT Traffic Flows 4.4 DUT/SUT Traffic Flows
Since the number of interfaces are not fixed, the traffic flows will Since the number of interfaces are not fixed, the traffic flows will
be dependent upon the configuration used in benchmarking the DUT/SUT. be dependent upon the configuration used in benchmarking the
Note that the term "traffic flows" is associated with client-to- DUT/SUT. Note that the term "traffic flows" is associated with
server requests. client-to- server requests.
For Dual-Homed configurations, there are two unique traffic flows: For Dual-Homed configurations, there are two unique traffic flows:
Client Server Client Server
------ ------ ------ ------
Protected -> Unprotected Protected -> Unprotected
Unprotected -> Protected Unprotected -> Protected
For Tri-Homed configurations, there are three unique traffic flows: For Tri-Homed configurations, there are three unique traffic flows:
Client Server Client Server
------ ------ ------ ------
Protected -> Unprotected Protected -> Unprotected
Protected -> DMZ Protected -> DMZ
Unprotected -> DMZ Unprotected -> DMZ
4.1.4 Multiple Client/Server Testing 4.5 Multiple Client/Server Testing
One or more clients may target multiple servers for a given One or more clients may target multiple servers for a given
application. Each virtual client MUST initiate requests(Connection, application. Each virtual client MUST initiate requests(Connection,
file transfers, etc.) in a round-robin fashion. For example, if object transfers, etc.) in a round-robin fashion. For example, if
the test consisted of six virtual clients targeting three servers, the test consisted of six virtual clients targeting three servers,
the pattern would be as follows: the pattern would be as follows:
Client Target Server(In order of request) Client Target Server(In order of request)
#1 1 2 3 1... #1 1 2 3 1...
#2 2 3 1 2... #2 2 3 1 2...
#3 3 1 2 3... #3 3 1 2 3...
#4 1 2 3 1... #4 1 2 3 1...
#5 2 3 1 2... #5 2 3 1 2...
#6 3 1 2 3... #6 3 1 2 3...
4.1.5 NAT(Network Address Translation) 4.6 NAT(Network Address Translation)
Most firewalls come with Network Address Translation(NAT)networks Many firewalls implement network address translation(NAT), a
built in which translates internal host IP addresses attached to the function which translates internal host IP addresses attached to
protected network to a virtual IP address for communicating across the the protected network to a virtual IP address for communicating
unprotected network(Internet). This involves additional processing across the unprotected network(Internet). This involves additional
on the part of the DUT/SUT and may impact on performance. Therefore, processing on the part of the DUT/SUT and may impact performance.
tests SHOULD be ran with NAT disabled and NAT enabled to determine the Therefore, tests SHOULD be ran with NAT disabled and NAT enabled
performance differentials. The test report SHOULD indicate whether NAT to determine the performance differentials. The test report SHOULD
was enabled or disabled. indicate whether NAT was enabled or disabled.
4.1.6 Rule Sets 4.7 Rule Sets
Rule sets[1] are a collection of access control policies that Rule sets[1] are a collection of access control policies that
determines which packets the DUT/SUT will forward and which it will determines which packets the DUT/SUT will forward and which it will
reject. The criteria by which these access control policies may be reject. The criteria by which these access control policies may be
defined will vary depending on the capabilities of the DUT/SUT. The defined will vary depending on the capabilities of the DUT/SUT. The
scope of this document is limited to how the rule sets should be scope of this document is limited to how the rule sets should be
applied when testing the DUT/SUT. applied when testing the DUT/SUT.
The firewall monitors the incoming traffic and checks to make sure The firewall monitors the incoming traffic and checks to make sure
that the traffic meets one of the defined rules before allowing it to that the traffic meets one of the defined rules before allowing it
be forwarded. It is RECOMMENDED that a rule be entered for each to be forwarded. It is RECOMMENDED that a rule be entered for each
host(i.e. - Virtual client). Although many firewalls permit groups host(Virtual client). Although many firewalls permit groups of IP
of IP addresses to be defined for a given rule, tests SHOULD be addresses to be defined for a given rule, tests SHOULD be performed
performed with large rule sets, which are more stressful to the with large rule sets, which are more stressful to the DUT/SUT.
DUT/SUT.
The DUT/SUT SHOULD be configured to denies access to all traffic which The DUT/SUT SHOULD be configured to denies access to all traffic
was not previously defined in the rule set. which was not previously defined in the rule set.
4.8 Web Caching 4.7 Web Caching
Some firewalls include caching agents in order to reduce network Some firewalls include caching agents in order to reduce network
load. When making a request through a caching agent, the caching load. When making a request through a caching agent, the caching
agent attempts to service the response from its internal resources. agent attempts to service the response from its internal memory.
The cache itself saves responses it receives, such as responses for
HTTP GET requests. The report SHOULD indicate whether web caching
was enabled or disabled on the DUT/SUT. The test report SHOULD
indicate whether NAT was enabled or disabled.
4.9 Authentication The cache itself saves responses it receives, such as responses
for HTTP GET requests. The report SHOULD indicate whether caching
was enabled or disabled on the DUT/SUT.
4.8 Authentication
Access control may involve authentication processes such as user, Access control may involve authentication processes such as user,
client or session authentication. Authentication is usually performed client or session authentication. Authentication is usually
by devices external to the firewall itself, such as an authentication performed by devices external to the firewall itself, such as an
servers and may add to the latency of the system. Any authentication authentication servers and may add to the latency of the system.
processes MUST be included as part of connection setup process. Any authentication processes MUST be included as part of connection
setup process.
5. Benchmarking Tests 5. Benchmarking Tests
5.1 Concurrent Connection Capacity 5.1 Concurrent Connection Capacity
5.1.1 Objective 5.1.1 Objective
To determine the maximum number of concurrent connections supported To determine the maximum number of concurrent connections through
by the DUT/SUT, as defined in RFC2647[1]. This test will employ a or with the DUT/SUT, as defined in RFC2647[1]. This test will employ
step search algorithm to obtain the maximum number of concurrent a step algorithm to obtain the maximum number of concurrent TCP
FTP,HTTP or SMTP connections the DUT/SUT can maintain. connections that the DUT/SUT can maintain.
5.1.2 Setup Parameters 5.1.2 Setup Parameters
The following parameters MUST be defined. Each parameters is The following parameters MUST be defined for all tests.
configured with the following considerations.
Connection Attempt Rate - The rate at which new connection requests Connection Attempt Rate - The rate, expressed in connections per
are attempted. The rate SHOULD be set lower than maximum rate at second, at which new TCP connection requests are attempted. The
which the DUT/SUT can accept new connection requests. rate SHOULD be set lower than maximum rate at which the DUT/SUT can
accept connection requests.
Connection Step count - Defines the number of additional connections Connection Step Count - Defines the number of additional TCP
attempted for each iteration of the step search algorithm. connections attempted for each iteration of the step search
algorithm.
Object/Message - Defines the number of bytes to be transferred across Object Size - Defines the number of bytes to be transferred in
each connection. response to a HTTP 1.1 GET request . It is RECOMMENDED to use the
minimum object size supported by the media.
5.1.3 Procedure 5.1.3 Procedure
Each virtual client will attempt to establish connections to their Each virtual client will attempt to establish TCP connections to its
target server(s) at a fixed rate in a round robin fashion. Each target server(s), using either the target server's IP address or NAT
proxy address, at a fixed rate in a round robin fashion. Each
iteration will involve the virtual clients attempting to establish a iteration will involve the virtual clients attempting to establish a
fixed number of additional connections. This search algorithm will fixed number of additional TCP connections. This search algorithm
be repeated until either: will be repeated until either:
- One or more of the additional connection attempts fail to - One or more of the additional connection attempts fail to
complete complete.
- One or more of the previously established connections failed. - One or more of the previously established connections fail.
Data transfers SHOULD be performed on each connection after the The test MUST also include application layer data transfers in
given connection is established. Data transfers MUST be performed on order to validate the TCP connections since, in the case of proxy
all connections after all of the addition connection have been based DUT/SUTs, the tester does not own both sides of the
established. connection. For the purposes of validation, the virtual client(s)
will request an object from its target server(s) using an HTTP 1.1
GET request, with both the client request and server response
excluding the connection-token close in the connection header. In
addition, periodic HTTP GET requests MAY be required to keep the
underlying TCP connection open(See Appendix A).
When benchmarking with FTP, virtual clients will issue NOOP command's 5.1.4 Measurements
to validate that work can be performed across each connection. The
virtual clients must receive a Command Successful reply from the
target server in order to be considered a valid connection.
When benchmarking with other applications such as HTTP or SMTP, Maximum concurrent connections - Total number of TCP connections
validation of the connection will be performed by initiating open for the last successful iteration performed in the search
object/message transfers. All bytes associated with the object/message algorithm.
transfers MUST be received by the requesting virtual client in order
to be considered a valid connection.
5.1.5 Measurements 5.1.5 Reporting Format
Total number of connections that were successfully completed in a 5.1.5.1 Transport-Layer Reporting:
step. Test equipment MUST be able to track each connection to verify
all required transaction between the virtual client and server
completed successfully. This includes successful completion of both
the command sequences and exchanging of any data across each of those
connections.
5.1.6 Reporting Format The test report MUST note the connection attempt rate, connection
step count and maximum concurrent connections measured.
Maximum concurrent connections reported MUST be the aggregate number 5.1.5.2 Application-Layer Reporting:
of connections completed for the last successful iteration. Report
SHOULD also include:
- Connection Attempt Rate. The test report MUST note the object size(s) and the use of
- Connection Step Count. HTTP 1.1 client and server.
A log file MAY be generated which includes for each step iteration: 5.1.5.3 Log Files
A log file MAY be generated which includes the TCP connection
attempt rate, HTTP object size and for each iteration:
- Step Iteration
- Pass/Fail Status. - Pass/Fail Status.
- Total connections established. - Total TCP connections established.
- Number of previously established connections dropped. - Number of previously established TCP connections dropped.
- Number of the additional connections that failed to complete. - Number of the additional TCP connections that failed to
complete.
5.2 Maximum Connection Setup Rate 5.2 Maximum Connection Setup Rate
5.2.1 Objective 5.2.1 Objective
To determine the maximum connection rate which can be supported To determine the maximum TCP connection setup rate through or with
through the DUT/SUT. As with the Concurrent Connection Capacity test, the DUT/SUT, as defined by RFC2647[1]. This test will employ a
FTP,HTTP and SMTP sessions will be used to determine this metric. search algorithm to obtain the maximum rate at which TCP connections
can be established through or with the DUT/SUT.
5.2.2 Setup Parameters 5.2.2 Setup Parameters
The following parameters MUST be defined. Each test parameter is The following parameters MUST be defined.
configured with the following considerations.
Initial Attempt Rate - The rate at which the initial connection Initial Attempt Rate - The rate, expressed in connections per
requests are attempted. second, at which the initial TCP connection requests are attempted.
Number of Connections - Defines the number of connections that must Number of Connections - Defines the number of TCP connections that
be established. The number MUST be between the number of must be established. The number MUST be between the number of
participating virtual clients and the maximum number supported by the participating virtual clients and the maximum number supported by
DUT/SUT. It is RECOMMENDED not to exceed the concurrent connection the DUT/SUT. It is RECOMMENDED not to exceed the concurrent
capacity found in section 5.1. The connection rate may vary depending connection capacity found in section 5.1.
on the number of connections attempted.
Object/Message - Defines the number of bytes to be transferred across Connection Teardown Rate - The rate, expressed in connections per
each connection. second, at which the tester will attempt to teardown TCP connections
between each iteration. The connection teardown rate SHOULD be set
lower than rate at which the DUT/SUT can teardown TCP connections.
Age Time - The time, expressed in seconds, the DUT/SUT will keep a
connection in it's state table after receiving a TCP FIN or RST
packet.
Object Size - Defines the number of bytes to be transferred in
response to a HTTP 1.1 GET request . It is RECOMMENDED to use the
minimum object size supported by the media.
5.2.3 Procedure 5.2.3 Procedure
An iterative search algorithm will be used to determine the maximum An iterative search algorithm will be used to determine the maximum
connection rate. This test iterates through different connection rates connection rate. This test iterates through different connection rates
with a fixed number of connections attempted by the virtual clients to with a fixed number of connections attempted by the virtual clients to
their associated server(s). their associated server(s).
Each iteration will use the same connection establishment and Each iteration will use the same connection establishment and
connection validation algorithms defined in the concurrent capacity connection validation algorithms defined in the concurrent capacity
test(See 5.1). After each iteration, the tester MUST close all test(See section 5.1).
connections prior to continuing to the next iteration.
Between each iteration of the test, the tester must close all
connections completed for the previous iteration. In addition,
it is RECOMMENDED to abort all unsuccessful connections attempted.
The tester will wait for the period of time, specified by age time,
before continuing to the next iteration.
5.2.4 Measurements 5.2.4 Measurements
The highest connection rate, in connections per second, for which Highest connection rate - Highest rate, in connections per second,
all connections completed successfully. Test equipment MUST be able for which all TCP connections completed successfully.
to track each connection to verify all required transaction between
the virtual client and server completed successfully. This includes
successful completion of both the command sequences and exchanging
of any data across each of those connections.
5.2.5 Reporting Format 5.2.5 Reporting Format
The maximum connection rate reported MUST be the maximum rate for 5.2.5.1 Transport-Layer Reporting:
which all connections successfully completed.
A log file MAY be generated which includes for each step iteration: The test report MUST note the number of connections attempted,
connection teardown rate, age time, and highest connection rate
measured.
5.1.5.2 Application-Layer Reporting:
The test report MUST note the object size(s) and the use of
HTTP 1.1 client and server.
5.1.5.3 Log Files
A log file MAY be generated which includes the total TCP connections
attempt, TCP connection teardown rate, age time, HTTP object size and
for each iteration:
- Step Iteration
- Pass/Fail Status. - Pass/Fail Status.
- Connection attempt rate. - Total TCP connections established.
- Number of the connections that failed to complete. - Number of TCP connections that failed to complete.
- Total connections established.
5.3 Connection Establishment Time 5.3 Connection Establishment Time
5.3.1 Objective 5.3.1 Objective
To characterize the connection establishment time[1] through or with To determine the connection establishment times[1] through or with
the DUT/SUT as a function of the number of open connections. the DUT/SUT as a function of the number of open connections.
5.3.2 Setup Parameters A connection for a client/server application is not atomic, in that
it not only involves transactions at the application layer, but
The following parameters MUST be defined. Each parameters is involves first establishing a connection using one or more underlying
configured with the following considerations. connection oriented protocols(TCP, ATM, etc). Therefore, it is
encouraged to make separate measurements for each connection oriented
Connection Attempt Rate - The rate at which new connection requests protocol required in order to perform the application layer
are attempted. The rate SHOULD be set lower than maximum rate at transaction.
which the DUT/SUT can accept new connection requests.
Connection Step count - Defines the number of additional connections
attempted for each iteration of the step algorithm.
Object/Message - Defines the number of bytes to be transferred across
each connection.
5.3.3 Procedure
The test will use the same algorithm as defined in the Concurrent
Capacity Test. This includes both the connection establishment and
validation of each connection by transferring data across each
connection.
5.3.4 Measurement
For each iteration, the tester MUST measure the Min/Avg/Max connection
times for the additional connections. It is RECOMMENDED that in addition
to the application layer, the tester include measurements at the lower
layer protocols(i.e. - TCP, ATM) when applicable. For each of the
protocols which the tester is measuring, the connection establishment
time shall consist of all transactions required to enable data to be
transferred across the given connection.
For example, FTP requires the user to login prior to being able to get
files, view directories and so forth. Connection establishment times
MUST include all of these transactions. In the case of TCP, the
connection establishment time would consist of the three-way handshake
between the two hosts(See Appendix D).
5.3.5 Reporting Format
Graph of the min/avg/maximum connection establishment times versus the
number of open connections. The report MUST identify the layer for which
the measurement was taken(i.e. - Application, transport, etc).
5.4 Denial Of Service Handling
5.4.1 Objective
To determine the effect of a denial of service attack on connection
establishment rates through the DUT/SUT. The Denial Of Service Handling
test should be ran after obtaining baseline measurements from section
5.2.
When a normal TCP connection starts, a destination host receives a SYN
(synchronize/start)packet from a source host and sends back a SYN ACK
(synchronize acknowledge). The destination host must then hear an ACK
(acknowledge) of the SYN ACK before the connection is established. The
TCP SYN attack exploits this design by having an attacking source host
generate TCP SYN packets with random source addresses towards a victim
host, thereby consuming that hosts resources.
Some firewalls employ one or more mechanisms to guard against SYN
attacks. If such mechanisms exist on the DUT/SUT, tests SHOULD be ran
with these mechanisms enabled to determine how well the DUT/SUT can
maintain the baseline connection rates determined in section 5.2
under such attacks.
5.4.2 Setup Parameters
The following parameters MUST be defined. Each parameter is configured
with the following considerations.
Initial Attempt Rate - The rate at which the initial connection
requests are attempted.
Number of Connections - Defines the number of connections that must 5.3.2 Setup Parameters
be established. The number MUST be between the number of
participating clients and the maximum number supported by the
DUT/SUT. It is RECOMMENDED not to exceed the concurrent connection
capacity found in section 5.1.
SYN Attack Rate - Defines the rate at which the server(s) are targeted The following parameters MUST be defined.
with TCP SYN packets.
5.4.3 Procedure Connection Attempt Rate - The rate, expressed in connections per
second, at which new TCP connection requests are attempted. It is
RECOMMENDED not to exceed the maximum connection rate found in
section 5.2.
This test uses the same procedure as defined in the maximum connection Connection Attempt Step count - Defines the number of additional
setup rate, with the addition of TCP SYN packets targeting the TCP connections attempted for each iteration of the step algorithm.
server(s) IP address or NAT proxy address.
The tester originating the TCP SYN attack MUST be attached to the Maximum Attempt Connection Count - Defines the maximum number of
Unprotected network. In addition, the tester MUST not respond to the TCP connections attempted in the test. It is RECOMMENDED not to
SYN ACK packets sent by target server in response to the SYN packet. exceed the concurrent connection capacity found in section 5.1.
5.4.4 Measurements Hickman, Newman, Martin [Page 9]
Object Size - Defines the number of bytes to be transferred in
response to a HTTP 1.1 GET request.
The highest connection rate, in connections per second, for which Number of requests - Defines the number of HTTP 1.1 GET requests
all legitimate connections completed successfully. Test equipment per connection. Note that connection, in this case, refers to the
MUST be able to track each connection to verify all required underlying transport protocol.
transaction between the virtual client and server completed
successfully. This includes successful completion of both the command
sequences and exchanging of any data across each of those connections.
In addition, the tester SHOULD track SYN packets associated with the 5.3.3 Procedure
SYN attack which the DUT/SUT forwards on the protected or DMZ
interface(s).
5.4.5 Reporting Format Each virtual client will attempt to establish TCP connections to its
target server(s) at a fixed rate in a round robin fashion. Each
iteration will involve the virtual clients attempting to establish
a fixed number of additional connections until the maximum attempt
connection count is reached.
The maximum connection rate reported MUST be the maximum rate for As with the concurrent capacity tests, application layer data
which all connections successfully completed. The report SHOULD transfers will be performed. Each virtual client(s) will request
include what percentage of TCP SYN packets were forwarded by the one or more objects from its target server(s) using one or more
DUT/SUT. HTTP 1.1 GET request, with both the client request and server
response excluding the connection-token close in the connection
header. In addition, periodic HTTP GET requests MAY be required to
keep the underlying TCP connection open(See appendix A).
A log file MAY be generated which includes for each step iteration: Since testing may involve proxy based DUT/SUTs, which terminates the
TCP connection, making a direct measurement of the TCP connection
establishment time is not possible since the protocol involves an
odd number of messages in establishing a connection. Therefore, when
testing with proxy based firewalls, the datagram following the final
ACK on the three-way handshake will be used in determining the
connection setup time.
- Pass/Fail Status. The following shows the timeline for the TCP connection setup
- Connection attempt rate. involving a proxy DUT/SUT and is referenced in the measurement
- Number of the connections that failed to complete. section. Note that this method may be applied when measuring other
- Total connections established. connection oriented protocols involving an odd number of messages
in establishing a connection.
5.5 Single Application Goodput t0: Client sends a SYN.
t1: Proxy sends a SYN/ACK.
t2: Client sends the final ACK.
t3: Proxy establishes separate connection with server.
t4: Client sends TCP datagram to server.
*t5: Proxy sends ACK of the datagram to client.
This section defined the procedures for base lining the Goodput[1] of the * While t5 is not considered part of the TCP connection establishment,
DUT/SUT for FTP, HTTP and SMTP traffic. acknowledgement of t4 must be received for the connection to be
considered successful.
5.5.1 FTP Goodput 5.3.4 Measurements
For each iteration of the test, the tester MUST measure the minimum,
maximum and average TCP connection establishment times. Measuring TCP
connection establishment times will be made two different ways,
depending on whether or not the DUT/SUT is proxy based. If proxy
based, the connection establishment time is considered to be from the
time the first bit of the SYN packet is transmitted by the client to
the time the client transmits the first bit of the TCP datagram,
provided that the TCP datagram gets acknowledged(t4-t0 in the above
timeline). For DUT/SUTs that are not proxy based, the establishment
time shall be directly measured and is considered to be from the time
the first bit of the SYN packet is transmitted by the client to the
time the last bit of the final ACK in the three-way handshake is
received by the target server.
5.5.1.1 Objective In addition, the tester SHOULD measure the minimum, maximum and
average connection establishment times for all other underlying
connection oriented protocols which are required to be established
for the client/server application to transfer an object. Each
connection oriented protocol has its own set of transactions
required for establishing a connection between two hosts or a host
and DUT/SUT. For purposes of benchmarking firewall performance, the
connection establishment time will be considered the interval
between the transmission of the first bit of the first octet of the
packet carrying the connection request to receipt of the last bit of
the last octet of the last packet of the connection setup traffic
received on the client or server, depending on whether a given
connection requires an even or odd number of messages, respectfully.
The File Transfer Protocol is a common application used by companies 5.3.5 Reporting Format
to transfer files from one device to another. Evaluating FTP Goodput
will allow individuals to measure how much successful traffic has
been forwarded by the DUT/SUT.
5.5.1.2 Setup Parameters The test report MUST note the TCP connection attempt rate, TCP
connection attempt step count and maximum TCP connections attempted,
HTTP object size and number of requests per connection.
The following parameters MUST be defined. Each parameter is configured For each connection oriented protocol the tester measured, the
with the following considerations. connection establishment time results SHOULD be in tabular form
with a row for each iteration of the test. There SHOULD be a column
for the iteration count, minimum connection establishment time,
average connection establishment time, maximum connection
establishment time, attempted connections completed, attempted
connections failed.
Number of Connections - Defines the number of connections to be 5.4 Connection Teardown Time
opened for transferring data objects. Number MUST be equal or
greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual client
participating in the test.
Connection Rate - Defines the rate at which connections are established. 5.4.1 Objective
Object Size - Defines the number of bytes to be transferred across each To determine the connection teardown time[1] through or with the
DUT/SUT as a function of the number of open connections. As with the
connection establishment time, separate measurements will be taken
for each connection oriented protocol involved in closing a
connection. connection.
5.5.1.3 Procedure 5.4.2 Setup Parameters
Each virtual client will establish a FTP connection to its respective
server(s) in a round robin fashion at the connection rate. The
transaction involved in establishing the FTP connection should follow
the procedure defined in Appendix A.
After the login process has been completed, the virtual client will
initiate a file transfer by issuing a "Get" command. The "Get"
command will target a predefined file of Object Size bytes.
Once the file transfer has completed, the virtual client will close
the FTP connection by issuing the QUIT command.
5.5.1.4 Measurement
The Goodput for each interface of the DUT/SUT MUST be measured. See The following parameters MUST be defined. Each parameters is
appendix D for details in regards to measuring the Goodput of the configured with the following considerations.
DUT/SUT. Only file transfers which have been completed are to be
included in the Goodput measurements.
The average transaction time each object successfully transferred MAY Initial connections - Defines the number of TCP connections to
be measured. The start time will begin when the time the "Get" initialize the test with. It is RECOMMENDED not to exceed the
commands is initiated and end at the time when the client receives concurrent connection capacity found in section 5.1.
an acknowledgment from the server that file transfer has completed.
5.5.1.5 Reporting Format Initial connection rate - Defines the rate, in connections per
second, at which the initial TCP connections are attempted. It is
RECOMMENDED not to exceed the maximum Connection setup rate found
in section 5.2.
The Goodput for each interface of the DUT/SUT MUST be reported in Teardown attempt rate - The rate at which the tester will attempt
bits per second. This will be the aggregate of session Goodput's to teardown TCP connections.
measured for a given interface.
Failure analysis: Teardown step count - Defines the number of TCP connections the
tester will attempt to teardown for each iteration of the step
algorithm.
The report SHOULD include the percentage of connections that Object size - Defines the number of bytes to be transferred across
failed. This includes: each connection in response to an HTTP 1.1 GET request during the
initialization phase of the test as well as periodic GET requests,
if required.
- Connections which failed to establish 5.4.3 Procedure
- Connections which failed to complete the object transfer
Transaction Processing analysis: Prior to beginning a step algorithm, the tester will initialize
the test by establishing connections defined by initial connections.
The test will use the same algorithm for establishing the connection
as described in the connection capacity test(Section 5.1).
The report SHOULD include average transaction time in transactions For each iteration of the step algorithm, the tester will attempt
per second. teardown the number of connections defined by teardown step count
at a rate defined by teardown attempt rate. This will be repeated
until the tester has attempted to teardown all of the connections.
The report SHOULD also include the object size(Bytes) being transferred. 5.4.4 Measurements
5.5.2 SMTP Goodput For each iteration of the test, the tester MUST measure the minimum,
average and maximum connection teardown times. As with the
connection establishment time test, the tester SHOULD measure all
connection oriented protocols which are being torn down.
5.5.2.1 Objective 5.4.5 Reporting Format
Another application commonly in use today is the mail transfer. One The test report MUST note the initial connections, initial
the common transport mechanisms for mail messages is the Simple Mail connection rate, teardown attempt rate, teardown step count and
Transfer Protocol(SMTP). The SMTP Goodput will allow individuals to object size.
measure how much successful SMTP traffic has been forwarded by the
DUT/SUT.
5.5.2.2 Setup Parameters For each connection oriented protocol the tester measured, the
connection teardown time results SHOULD be in tabular form
with a row for each iteration of the test. There SHOULD be a column
for the iteration count, minimum connection teardown time,
average connection teardown time, maximum connection teardown
time, attempted teardowns completed, attempted teardown failed.
The following parameters MUST be defined. Each parameter is 5.5 Denial Of Service Handling
configured with the following considerations.
Number of Connections - Defines the number of connections to be 5.5.1 Objective
opened for transferring data objects. Number MUST be equal or
greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual client
participating in the test.
Connection Rate - Defines the rate at which connections are To determine the effect of a denial of service attack on a DUT/SUTs
attempted. connection establishment rates and/or goodput. The Denial Of Service
Handling test MUST be run after obtaining baseline measurements
from sections 5.2 and/or 5.6.
Message Size - Defines the number of bytes to be transferred across The TCP SYN flood attack exploits TCP's three-way handshake mechanism
each connection. by having an attacking source host generate TCP SYN packets with
random source addresses towards a victim host, thereby consuming that
host's resources.
5.5.2.3 Procedure Some firewalls employ mechanisms to guard against SYN attacks. If such
mechanisms exist on the DUT/SUT, tests SHOULD be run with these
mechanisms enabled to determine how well the DUT/SUT can maintain,
under such attacks, the baseline connection rates and goodput determined
in section 5.2 and section 5.6, respectively.
Each virtual client will establish a SMTP connection to its 5.5.2 Setup Parameters
respective server(s) in a round robin fashion at the connection rate.
The transaction involved in establishing the SMTP connection should
follow the procedure defined in Appendix B.
After the greeting exchanges have been completed, the client will Use the same setup parameters as defined in section 5.2.2 or 5.6.2,
initiate the transfer of the message by initiating the MAIL command. depending on whether testing against the baseline connection setup
The client will then send the predefined message of Object Size. rate test or goodput test, respectfully.
Once the message has been acknowledged as being received by the In addition, the following setup parameters MUST be defined.
server, the virtual client will then close the connection.
5.5.2.4 Measurement SYN Attack Rate - Defines the rate, in packets per second at which
the server(s) are targeted with TCP SYN packets.
The Goodput for each interface of the DUT/SUT MUST be measured. See 5.5.3 Procedure
appendix D for details in regards to measuring the Goodput of the
DUT/SUT. Only message transfers which have been completed are to be
included in the Goodput measurements.
The average transaction time for each message transferred MAY be Use the same procedure as defined in section 5.2.3 or 5.6.3, depending
measured. The start time will begin when the time the "MAIL" command on whether testing against the baseline connection setup rate test or
is initiated and end at the time when the client receives an goodput test, respectfully. In addition, the tester will generate TCP
acknowledgment from the server that the message has been received. SYN packets targeting the server(s) IP address or NAT proxy address at
a rate defined by SYN attack rate.
5.5.2.5 Reporting Format The tester originating the TCP SYN attack MUST be attached to the
unprotected network. In addition, the tester MUST not respond to the
SYN/ACK packets sent by target server in response to the SYN packet.
Goodput analysis: 5.5.4 Measurements
Perform the same measurements as defined in section 5.2.4 or 5.6.4,
depending on whether testing against the baseline connection setup
rate test or goodput test, respectfully.
The Goodput for each interface of the DUT/SUT MUST be reported in In addition, the tester SHOULD track SYN packets associated with the
bits per second. This will be the aggregate of session Goodput's SYN attack which the DUT/SUT forwards on the protected or DMZ
measured for a given interface. interface(s).
Failure analysis: 5.5.5 Reporting Format
The report SHOULD include the percentage of connections that The test SHOULD use the same reporting format as described in
failed. This includes: section 5.2.5 or 5.6.5, depending on whether testing against
baseline throughput rates or goodput, respectively.
- Connections which failed to establish In addition, the report MUST indicate a denial of service handling
- Connections which failed to complete the object transfer test, SYN attack rate, number SYN attack packets transmitted and
number of SYN attack packets received and whether or not the DUT
has any SYN attack mechanisms enabled.
Transaction Processing analysis: 5.6 HTTP
The report SHOULD include average transaction time in transactions 5.6.1 Objective
per second.
The report SHOULD also include the object size(Bytes) being transferred. To determine the goodput, as defined by RFC2647, of the DUT/SUT
when presented with HTTP traffic flows. The goodput measurement
will be based on HTTP objects forwarded to the correct destination
interface of the DUT/SUT.
5.5.3 HTTP Goodput Goodput 5.6.2 Setup Parameters
5.5.3.1 Objective The following parameters MUST be defined.
Another common application is the World Wide Web (WWW) application Number of sessions - Defines the number of HTTP 1.1 sessions to be
that can access documents over the Internet. This application uses attempted for transferring an HTTP object(s). Number MUST be equal
the Hypertext Transfer Control Protocol (HTTP) to copy information or greater than the number of virtual clients participating in the
from one system to another. test. The number SHOULD be a multiple of the virtual clients
participating in the test. Note that each session will use one
underlying transport layer connection.
Session rate - Defines the rate, in sessions per second, that the
HTTP sessions are attempted.
HTTP Goodput measurement is actually determined by evaluating the Requests per session - Defines the number of HTTP GET requests per
Forwarding rate of packets. This is based on measuring only data session.
that has successfully been forwarded to the destination interface.
When benchmarking the performance of the DUT/SUT, consideration of Object Size - Defines the number of bytes to be transferred in
the HTTP version being used must be taken into account. Appendix C response to an HTTP GET request.
of this document discusses enhancements to the HTTP protocol which
may impact performance results.
5.5.3.2 Setup Parameters 5.6.3 HTTP Procedure
The following parameters MUST be defined. Each variable is Each HTTP 1.1 virtual client will attempt to establish sessions
configured with the following considerations. to its HTTP 1.1 target server(s), using either the target server's
IP address or NAT proxy address, at a fixed rate in a round robin
fashion.
Number of Connections - Defines the number of HTTP connections Baseline measurements SHOULD be performed using a single GET request
to be opened for transferring data objects. Number MUST be equal or per HTTP session with the minimal object size supported by the media.
greater than the number of virtual clients participating in the If the tester makes multiple HTTP GET requests per session, it MUST
test. The number SHOULD be a multiple of the virtual client request the same-sized object each time. Testers may run multiple
participating in the test. iterations of this test with objects of different sizes. See
appendix A when testing proxy based DUT/SUT regarding HTTP version
considerations. 5.6.4 Measurement
Connection Rate - Defines the rate at which connections are Aggregate Goodput - The aggregate bit forwarding rate of the
attempted. requested HTTP objects. The measurement will start on receipt of the
first bit of the first packet containing a requested object which
has been successfully transferred and end on receipt of the last
packet containing the last requested object that has been
successfully transferred. The goodput, in bits per second, can be
calculated using the following formula:
Object Size - Defines the number of bytes to be transferred OBJECTS * OBJECTSIZE * 8
across each connection. Goodput = --------------------------
DURATION
5.5.3.3 HTTP Procedure OBJECTS - Objects successfully transferred
For the HTTP Goodput tests, it is RECOMMENDED to determine which OBJECTSIZE - Object size in bytes
version of HTTP the DUT/SUT has implemented and use the same
version for the test. To determine the version of HTTP, the user
documentation of the DUT/SUT SHOULD be consulted.
Each client will attempt to establish HTTP connection's to their DURATION - Aggregate transfer time based on aforementioned time
respective servers a user defined rate. The clients will attach to references.
the servers using either the servers IP address or NAT proxy
address.
After the client has established the connection with the server, 5.6.5 Reporting Format
the client will initiate GET command(s) to retrieve predefined
web page(s).
When employing HTTP/1.0 in benchmarking the performance of the The test report MUST note the object size(s), number of sessions,
DUT/SUT, only one object will be retrieved for each of the defined session rate and requests per session.
object sizes. After the object has been transferred, the connection
should then be torn down. When defining multiple objects, object
transfers must be completed and the connections closed for all
of the participating clients prior testing the next object size.
This process is repeated until all of the defined objects are
tested.
When employing HTTP/1.1, all objects defined by the user will The goodput results SHOULD be reported in tabular form with a row
be requested with a GET request over the same connection. The for each of the object sizes. There SHOULD be columns for the object
connection should then be torn down after all of the objects size, measured goodput and number of successfully transferred
have been transferred. objects.
5.5.3.4 Measurement Failure analysis:
The Goodput for each of the objects sizes transferred MUST be The test report SHOULD indicate the number and percentage of HTTP
measured. See appendix D for details in regards to measuring the sessions that failed to complete the requested number of
Goodput of the DUT/SUT. Only objects which have been successfully transactions, with a transaction being the GET request and
acknowledged by the server are to be included in the Goodput successfully returned object.
measurements.
The transaction times for each object transferred MUST measured. Version information:
The transaction connection time starts when the connection is
made and will end when the web pages is completely mapped on the
virtual client (when the client sends an acknowledgment packet is
sent from the client).
5.5.3.5 Reporting Format The test report MUST note the use of an HTTP 1.1 client and server.
Goodput analysis: 5.7 IP Fragmentation
The Goodput for each interface of the DUT/SUT MUST be reported in 5.7.1 Objective
bits per second. This will be the aggregate of session Goodput's
measured for a given interface.
Failure analysis: To determine the performance impact when the DUT/SUT is presented
with IP fragmented[5] traffic. IP datagrams which have been
fragmented, due to crossing a network that supports a smaller
MTU(Maximum Transmission Unit) than the actual datagram, may
require the firewall to perform re-assembly prior to the datagram
being applied to the rule set.
The report SHOULD include the percentage of connections that While IP fragmentation is a common form of attack, either on the
failed. This includes: firewall itself or on internal hosts, this test will focus on
determining how the additional processing associated with the
re-assembly of the datagrams has on the goodput of the DUT/SUT.
- Connections which failed to establish 5.7.2 Setup Parameters
- Connections which failed to complete the object transfer
Transaction Processing analysis: The following parameters MUST be defined.
The report SHOULD include average transaction time in transactions Trial duration - Trial duration SHOULD be set for 30 seconds.
per second.
The report SHOULD also include the object size(Bytes) being transferred. 5.7.2.1 Non-Fragmented Traffic Parameters
Version Information Session rate - Defines the rate, in sessions per second, that the
HTTP sessions are attempted.
Report MUST include the version of HTTP used for the test. In Requests per session - Defines the number of HTTP GET requests per
addition, if the HTTP/1.1 is used, the number of concurrent GET's session.
allowable(Pipelining) SHOULD be reported.
5.6 Concurrent Application Goodput Object Size - Defines the number of bytes to be transferred in
response to an HTTP GET request.
5.6.1 Objective 5.7.2.1 Fragmented Traffic Parameters
To determine the Goodput of the DUT/SUT when offering a mix of FTP, Packet size, expressed as the number of bytes in the IP/UDP packet,
SMTP and HTTP traffic flows. Real world traffic does not consist exclusive of link-layer headers and checksums.
of a single protocol, but a mix of different applications. This
test will allow an individual to determine how well the DUT/SUT
handles a mix of applications by comparing the results to the
individual baseline measurements.
5.6.2 Setup Parameters Fragmentation Length - Defines the length of the data portion of the
IP datagram and MUST be multiple of 8. Testers SHOULD use the minimum
value, but MAY use other sizes as well.
The following parameters MUST be defined. Each variable is Intended Load - Intended load, expressed as percentage of media
configured with the following considerations. utilization.
Number of Connections - Defines the aggregate number of connections 5.7.3 Procedure
to be opened for transferring data/message objects. Number MUST be
equal to or greater than the number of virtual clients participating
in the test. The number SHOULD be a multiple of the virtual client
participating in the test.
Connection Rate - Defines the rate at which connections attempts are Each HTTP 1.1 virtual client will attempt to establish sessions
opened. Number MUST be evenly divided among all of the virtual to its HTTP 1.1 target server(s), using either the target server's
clients participating in the test. IP address or NAT proxy address, at a fixed rate in a round robin
fashion. At the same time, a client attached to the unprotected side
of the network will offer a unidirectional stream of unicast UDP/IP
packets to a server connected to the protected side of the network.
The tester MUST offer IP/UDP packets in a steady state.
Object/Message Size - Defines the number of bytes to be transferred Baseline measurements SHOULD be performed with a deny rule(s) that
across each connection. RECOMMENDED message sizes still needs to be filters the fragmented traffic. If the DUT/SUT has logging
determined. capability, the log SHOULD be checked to determine if it contains
the correct information regarding the fragmented traffic.
At a minimum, at least one of the following parameters MUST be The test SHOULD be repeated with the DUT/SUT rule set changed to
defined. In addition, the cumulative percentage all the defined allow the fragmented traffic through. When running multiple
percentages MUST equal 100%. iterations of the test, it is RECOMMENDED to vary the fragment
length while keeping all other parameters constant.
FTP Percentage - Defines the percentage of traffic connections 5.7.4 Measurements
which are to consist of FTP file transfers.
SMTP Percentage - Defines the percentage of traffic connections Aggregate Goodput - The aggregate bit forwarding rate of the
which are to consist of SMTP Message transfers. requested HTTP objects.(See section 5.6). Only objects which have
successfully completed transferring within the trial duration are
to be included in the goodput measurement.
HTTP Percentage - Defines the percentage of traffic connections Transmitted UDP/IP Packets - Number of UDP packets transmitted by
which are to consist of HTTP GET requests. client.
5.6.3 Procedure Received UDP/IP Packets - Number of UDP/IP Packets received by
server.
This test will run each of the single application Goodput tests, 5.7.5 Reporting Format
for which there is a defined percentage, concurrently. For each
of the defined traffic types, the connection establishment,
data/message transfer and teardown procedures will be the same
as defined in the individual tests.
5.6.4 Measurements The test report MUST note the test duration.
As with the individual tests, the Goodput for each of the defined The test report MUST note the packet size(s), offered load(s) and
traffic types MUST be measured. See appendix D for details in IP fragmentation length of the UDP/IP traffic. It SHOULD also note
regards to measuring the Goodput of the DUT/SUT. Only messages/data whether the DUT/SUT egresses the offered UDP/IP traffic fragmented
which have been successfully acknowledged as being transferred are or not.
to be included in the Goodput measurements.
The transaction times for each of the defined applications MUST be The test report MUST note the object size(s), session rate and
measured. See the appropriate single application Goodput test for requests per session.
the specifics of measuring the transaction times for each of the
defined traffic types.
5.6.5 Reporting Format The results SHOULD be reported in the format of a table with a
row for each of the fragmentation lengths. There SHOULD be columns
for the fragmentation length, IP/UDP packets transmitted by client,
IP/UDP packets received by server, HTTP object size, and measured
goodput.
Goodput analysis: 5.8 Illegal Traffic Handling
Reporting SHOULD include a graph of the number of connections
versus the measured Goodput in Mbps for each of the defined
traffic types(FTP, SMTP, HTTP).
Failure analysis: 5.8.1 Objective
Reporting should include a graph of number of connections versus
percent success for each of the defined traffic types.
Transaction Processing analysis: To determine the behavior of the DUT/SUT when presented with a
Reporting should include a graph of number of virtual connections combination of both legal and Illegal traffic flows. Note that
versus average transaction for each of the defined traffic types. Illegal traffic does not refer to an attack, but to traffic which
has been explicitly defined by a rule(s) to drop.
5.7 Illegal Traffic Handling 5.8.2 Setup Parameters
To determine the behavior of the DUT/SUT when presented with a The following parameters MUST be defined.
combination of both legal and Illegal traffic.
5.7.1 Procedure Number of sessions - Defines the number of HTTP 1.1 sessions to be
attempted for transferring an HTTP object(s). Number MUST be equal
or greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual clients
participating in the test. Note that each session will use one
underlying transport layer connection.
Session rate - Defines the rate, in sessions per second, that the
HTTP sessions are attempted.
Still Needs to be determined Requests per session - Defines the number of HTTP GET requests per
session.
5.7.2 Measurements Object size - Defines the number of bytes to be transferred in
response to an HTTP GET request.
Still Needs to be determined Illegal traffic percentage - Percentage of HTTP 1.1 sessions which
have been explicitly defined in a rule(s) to drop.
5.7.3 Reporting Format 5.8.3 Procedure
Still Needs to be determined Each HTTP 1.1 virtual client will attempt to establish sessions
to its HTTP 1.1 target server(s), using either the target server's
IP address or NAT proxy address, at a fixed rate in a round robin
fashion.
5.8 Latency The tester MUST present the connection requests, both legal and
illegal, in an evenly distributed manner. Many firewalls have
the capability to filter on different traffic criteria( IP
addresses, Port numbers, etc). Testers may run multiple
iterations of this test with the DUT/SUT configured to filter
on different traffic criteria.
Determine the latency of application layer data through the DUT/SUT. 5.8.4 Measurements
5.8.1 Procedure Legal sessions allowed - Number and percentage of legal HTTP
sessions which completed.
Still Needs to be determined Illegal session allowed - Number and percentage of illegal HTTP
session which completed.
5.8.2 Measurements 5.8.5 Reporting Format
Still needs to be determined. The test report MUST note the number of sessions, session rate,
requests per session, percentage of illegal sessions and measurement
results. The results SHOULD be reported in the form of a table with a row
for each of the object sizes. There SHOULD be columns for the
object size, number of legal sessions attempted, number of legal
sessions successful, number of illegal sessions attempted and number
of illegal sessions successful.
5.8.3 Reporting format 5.9 Latency
Still needs to be determined. 5.9.1 Objective
APPENDICES To determine the latency of network-layer or application-layer data
traversing the DUT/SUT. RFC 1242 [3] defines latency.
APPENDIX A: FTP(File Transfer Protocol) 5.9.2 Setup Parameters
A.1 Introduction The following parameters MUST be defined:
The FTP protocol was designed to be operated by interactive end users 5.9.2.1 Network-layer Measurements
or application programs. The communication protocol to transport this
service is TCP. The core functions of this application enable users
to copy files between systems, view directory listings and perform
house keeping chores - such as renaming, deleting and copying files.
Unlike other protocols, FTP uses two connections. One connection,
called the control connection, is used to pass commands between
the client and the server. The other, called the data connection, is
used to transfer the actual data(Files, directory lists, etc.).
A.2 Connection Establishment/Teardown(Control) Packet size, expressed as the number of bytes in the IP packet,
exclusive of link-layer headers and checksums.
FTP control connections are established by issuing OPEN command Intended load, expressed as percentage of media utilization.
targeting either the URL or a specific IP address. Since the
methodology does not include DNS servers, OPEN commands should
target specific IP address of target server. A TCP connection
will be established between the client and target server.
The client will then begin the login process. When logging in, Offered load, expressed as percentage of media utilization.
it is RECOMMENDED to perform the test using Anonymous FTP Login
and should use the following syntax:
User ID: Anonymous Test duration, expressed in seconds.
Password: will correspond to the System ID
(client1_1@test.net through client 1_8@test.net)
Once a successful login acknowledgment is received from the server, Test instruments MUST generate packets with unique timestamp signatures.
the client may then initiate a file transfer. After all transfer
operations have been completed, the FTP connection may be closed by
issuing a QUIT command.
A.3 Data Connection 5.9.2.2 Application-layer Measurements
The data connection is established each time the user requests a file Object size, expressed as the number of bytes to be transferred across a
transfer and torn down when the transfer is completed. FTP supports connection in response to an HTTP GET request. Testers SHOULD use the
two modes of operation, namely normal mode and passive mode, which minimum object size supported by the media, but MAY use other object
determine who initiates the data connection. In normal mode sizes as well.
operation, the server initiates the data connection, targeting a
predefined PortID specified in the PORT command. In passive mode, the
client initiates the data connection, targeting the PortID returned
in response to the PASV Command. It is RECOMMENDED to perform the
tests in normal mode operation.
File transfers are initiated by using the "Get" or "Put" command and Connection type. The tester MUST use one HTTP 1.1 connection for latency
specifying the desired filename. The tests defined in this document measurements.
will use the "Get" command to initiate file transfers from the target
server to the client.
A.4 Object Format Number of objects requested.
Need to define the object format. Number of objects transferred.
APPENDIX B: SMTP (Simple Mail Transfer Protocol) Test duration, expressed in seconds.
B.1 Introduction Test instruments MUST generate packets with unique timestamp signatures.
The SMTP defines a simple straight forward way to move messages 5.9.3 Network-layer procedure
between hosts. There are two roles in the SMTP protocol, one is the
sender and one is the receiver. The sender acts like a client and
establishes a TCP connection with the receiver which acts like a
server. The transactions defined in this section will use the terms
client and server in place of sender and receiver.
B.2 Connection Establishment/Teardown A client will offer a unidirectional stream of unicast packets to a server.
The packets MUST use a connectionless protocol like IP or UDP/IP.
Each connection involves a connection greeting between the The tester MUST offer packets in a steady state. As noted in the latency
sender(Client) and receiver(Server). The syntax used to identify each discussion in RFC 2544 [4], latency measurements MUST be taken at the
other's hostnames during this greeting exchange SHOULD be: throughput level -- that is, at the highest offered load with zero packet
loss. Measurements taken at the throughput level are the only ones that can
legitimately be termed latency.
"SMTPRcv_<Virtual_Server>.com" It is RECOMMENDED that implementers use offered loads not only at the
"SMTPSender_<Virtual Client>.com" throughput level, but also at load levels that are less than or greater
than the throughput level. To avoid confusion with existing terminology,
measurements from such tests MUST be labeled as delay rather than latency.
If desired, the tester MAY use a step test in which offered loads increment
or decrement through a range of load levels.
where <Virtual_Client> and <Virtual_Server> represent a The duration of the test portion of each trial MUST be at least 30 seconds.
unique virtual number for the client and server
respectively.
The basic transactions in moving mail between two hosts involve three 5.9.4 Application layer procedure
basic steps which are outlined below. These are:
1) Client issuing a MAIL command identifying the message originator An HTTP 1.1 client will request one or more objects from an HTTP 1.1 server
for that session. Syntax used to identify the originator SHOULD using one or more HTTP GET requests. If the tester makes multiple HTTP GET
be as follows: requests, it MUST request the same-sized object each time. Testers may run
multiple iterations of this test with objects of different sizes.
connection1,2,3...@hostname Implementers MAY configure the tester to run for a fixed duration. In this
case, the tester MUST report the number of objects requested and returned
for the duration of the test. For fixed-duration tests it is RECOMMENDED
that the duration be at least 30 seconds.
2) Client issues an RCPT command identifying the recipient of the 5.9.5 Measurements
message for that session. Syntax used to identify the recipient
of the message SHOULD be as follows:
reciever1,2,3...@hostname Minimum delay - The smallest delay incurred by data traversing the DUT/SUT
at the network layer or application layer, as appropriate.
3) Client issues a DATA command. After receiving the Maximum delay - The largest delay incurred by data traversing the DUT/SUT
acknowledgment from the server, the client will then transfer at the network layer or application layer, as appropriate.
the message which MUST include a line with a period to
indicate to the server the end of the message. Once the end of
message is received by the server, it will acknowledge the end
of message.
The client may initiate another message transfer or close the session Average delay - The mean of all measurements of delay incurred by data
by initiating the QUIT command. traversing the DUT/SUT at the network layer or application layer, as
appropriate.
B.3 Message Format Delay distribution - A set of histograms of all delay measurements observed
for data traversing the DUT/SUT at the network layer or application layer,
as appropriate.
As Internet e-mail has evolved, SMTP extensions have been added to 5.9.6 Network-layer reporting format
support both audio and video message transfers. For these firewall
tests, messages SHOULD consist of plain text ASCII.
APPENDIX C: HTTP(HyperText Transfer Protocol) The test report MUST note the packet size(s), offered load(s) and test
duration used.
C.1 Introduction The latency results SHOULD be reported in the format of a table with a row
for each of the tested packet sizes. There SHOULD be columns for the
packet size, the intended rate, the offered rate, and the resultant latency
or delay values for each test.
As HTTP has evolved over the years, changes to the protocol have 5.9.7 Application-layer reporting format
occurred to both fix problems of previous versions as well as improve
performance. The most common versions in use today are HTTP/1.0 and
HTTP/1.1 and are and are discussed below.
C.2 Version Considerations The test report MUST note the object size(s) and number of requests and
responses completed. If applicable, the report MUST note the test duration
if a fixed duration was used.
HTTP/1.1 was approved by the WWW Consortium in July 1999 as an IETF The latency results SHOULD be reported in the format of a table with a row
Draft Standard. This is a formal recognition of the fact that all for each of the object sizes. There SHOULD be columns for the object size,
known technical issues have been resolved in the specification which the number of completed requests, the number of completed responses, and the
was brought out in June 1997. HTTP/1.1 is also downward compatible resultant latency or delay values for each test.
with HTTP/1.0. Both protocols on the popular browsers in use today.
The following is a list of features that are offered in HTTP 1.1 that
are not in HTTP 1.0.
- Persistent connections and pipelining Failure analysis:
Though both use TCP for data transfer, but differ in the way it is The test report SHOULD indicate the number and percentage of HTTP GET
used, with the later version being more efficient. Once a connection request or responses that failed to complete within the test duration.
is opened, it is not closed until the HTML document and all objects
referred by it are downloaded. This technique is called persistent
connection. By serving multiple requests on the same TCP segment,
many control packets (which are not part of actual data transfer)
are avoided. The technique of containing multiple requests and
responses within the same TCP segment over a persistent connection
is called pipelining.
- Data compression Version information:
HTTP/1.1 provides for compression of documents before file transfer. The test report MUST note the use of an HTTP 1.1 client and server.
Since most other objects like images and binaries are already
compressed, this feature applies only to HTML and plain text
documents.
- Range request and validation APPENDICES
Bandwidth saving measure is the introduction of two new fields in APPENDIX A: HTTP(HyperText Transfer Protocol)
an HTTP request header, viz. If-Modified-Since: and If-Unmodified-
Since:. The significance of this feature is that if a browser
identifies a file in its cache, it needn't reload it unless it has
changed since the last time it was used.
- Support for multiple hosts The most common versions of HTTP in use today are HTTP/1.0 and
HTTP/1.1 with the main difference being in regard to persistent
connections. HTTP 1.0, by default, does not support persistent
connections. A separate TCP connection is opened up for each
GET request the client wants to initiate and closed after the
requested object transfer is completed. Some implementations of
HTTP/1.0 supports persistence by adding an additional header
to the request/response:
It is common for an ISP to host more than one Web site on a single Connection: Keep-Alive
server. In such a case, each domain requires its own IP address.
C.3 Object Format However, under HTTP 1.0, there is no official specification for
how the keep-alive operates. In addition, HTTP 1.0 proxies do
support persistent connection as they do not recognize the
connection header.
Object SHOULD be an HTML formatted object. HTTP/1.1, by default, does support persistent connection and
is therefore the version that is referenced in this methodology.
When HTTP/1.1 entities want the underlying transport layer
connection closed after a transaction has completed, the
request/response will include a connection-token close in the
connection header:
Append D. GOODPUT Measurements. Connection: close
The Goodput will measure the number of bits per second forwarded by If no such connection-token is present, the connection remains
the DUT/SUT and will be referenced to the application level data. The open after the transaction is completed. In addition, proxy
formula for determining Goodput of the DUT/SUT is as follows: based DUT/SUTs may monitor the TCP connection and after a
timeout, close the connection if no activity is detected. The
duration of this timeout is not defined in the HTTP/1.1
specification and will vary between DUT/SUTs. When performing
concurrent connection testing, GET requests MAY need to be
issued at a periodic rate so that the proxy does not close the
TCP connection.
ObjectSize(Bytes) * 8 While this document cannot foresee future changes to HTTP
Goodput(Bits/Sec) = Transfer Time(Seconds) and it's impact on the methodologies defined herein, such
changes should be accommodated for so that newer versions of
HTTP may be used in benchmarking firewall performance.
Transfer Time starts when the first bit of the object/message is Appendix B. References
received at the destination port of the tester. The transfer time ends
when the last bit of the object/message is received at the destination
port of the tester.
Appendix E. References [1] D. Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647,
August 1999.
[1] Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, [2] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter, P. Leach,
February 1998. T. Berners-Lee , "Hypertext Transfer Protocol -- HTTP/1.1",
RFC 2616 June 1999
[2] J. Postel, "Simple Mail Transfer Protocol", RFC 821, August 1982. [3] S. Bradner, editor. "Benchmarking Terminology for Network
Interconnection Devices," RFC 1242, July 1991.
[3] R. Fielding, J. Gettys, J. Mogul, H Frystyk, T. Berners, "Hypertext [4] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network
Transfer Protocol -- HTTP/1.1", January 1997 Interconnect Devices," RFC 2544, March 1999.
[4] J. Postel, J. Reynolds, "File Transfer Protocol(FTP)", October 1985 [5] David C. Clark, "IP Datagram Reassembly Algorithm", RFC 815 ,
July 1982.
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/