draft-ietf-bmwg-firewall-04.txt   draft-ietf-bmwg-firewall-05.txt 
Network Working Group Brooks Hickman Benchmarking Working Group Brooks Hickman
Internet-Draft Spirent Communications Internet-Draft Spirent Communications
Expiration Date: November 2002 David Newman Expiration Date: December 2002 David Newman
Network Test Network Test
Saldju Tadjudin Saldju Tadjudin
Spirent Communications Spirent Communications
Terry Martin Terry Martin
M2networx INC GVNW Consulting Inc
May 2002 June 2002
Benchmarking Methodology for Firewall Performance Benchmarking Methodology for Firewall Performance
<draft-ietf-bmwg-firewall-04.txt> <draft-ietf-bmwg-firewall-05.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 36 skipping to change at page 1, line 36
months and may be updated, replaced, or obsoleted by other documents months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved.
Abstract
This document discusses and defines a number of tests that may be
used to describe the performance characteristics of firewalls. In
addition to defining the tests this document also describes specific
formats for reporting the results of the tests.
This document is a product of the Benchmarking Methodology Working
Group (BMWG) of the Internet Engineering Task Force (IETF).
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 2 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1 Test Considerations . . . . . . . . . . . . . . . . . . 3 4.1 Test Considerations . . . . . . . . . . . . . . . . . . 4
4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 3 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 4
4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4 4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4
4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 4 4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 5
4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5
4.6 NAT(Network Address Translation) . . . . . . . . . . . . 5 4.6 NAT(Network Address Translation) . . . . . . . . . . . . 5
4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 5 4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 6
4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 5 4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6
4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6 4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6
5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6
5.1 IP throughput . . . . . . . . . . . . . . . . . . . . . . 6 5.1 IP throughput . . . . . . . . . . . . . . . . . . . . . . 6
5.2 Concurrent TCP Connection Capacity . . . . . . . . . . . 7 5.2 Concurrent TCP Connection Capacity . . . . . . . . . . . 8
5.3 Maximum TCP Connection Establishment Rate . . . . . . . . 10 5.3 Maximum TCP Connection Establishment Rate . . . . . . . . 10
5.4 Maximum TCP Connection Tear Down Rate . . . . . . . . . . 12 5.4 Maximum TCP Connection Tear Down Rate . . . . . . . . . . 12
5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 13 5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 14
5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 14 5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 15
5.7 HTTP Concurrent Transaction Capacity . . . . . . . . . . 17 5.7 HTTP Concurrent Transaction Capacity . . . . . . . . . . 17
5.8 HTTP Transaction Rate . . . . . . . . . . . . . . . . . . 18 5.8 HTTP Transaction Rate . . . . . . . . . . . . . . . . . . 18
5.9 Illegal Traffic Handling . . . . . . . . . . . . . . . . 19 5.9 Illegal Traffic Handling . . . . . . . . . . . . . . . . 20
5.10 IP Fragmentation Handling . . . . . . . . . . . . . . . 20 5.10 IP Fragmentation Handling . . . . . . . . . . . . . . . 21
5.11 Latency . . . . . . . . . . . . . . . . . . . . . . . . 22 5.11 Latency . . . . . . . . . . . . . . . . . . . . . . . . 23
Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . 25 6. References . . . . . . . . . . . . . . . . . . . . . . . . . 25
A. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . 25 7. Security Consideration . . . . . . . . . . . . . . . . . . . 26
B. Connection Establishment Time Measurements . . . . . . . . 25 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 26
C. Connection Tear Down Time Measurements . . . . . . . . . . 26 9. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 26
C. References . . . . . . . . . . . . . . . . . . . . . . . . 26 Appendix A - HyperText Transfer Protocol(HTTP) . . . . . . . . 27
Appendix B - Connection Establishment Time Measurements . . . . 27
Appendix C - Connection Tear Down Time Measurements . . . . . . 28
Full Copy Statement . . . . . . . . . . . . . . . . . . . . . . 28
1. Introduction 1. Introduction
This document provides methodologies for the performance This document provides methodologies for the performance
benchmarking of firewalls. It provides methodologies in four areas: benchmarking of firewalls. It provides methodologies in four areas:
forwarding, connection, latency and filtering. In addition to forwarding, connection, latency and filtering. In addition to
defining the tests, this document also describes specific formats defining the tests, this document also describes specific formats
for reporting the results of the tests. for reporting the results of the tests.
A previous document, "Benchmarking Terminology for Firewall A previous document, "Benchmarking Terminology for Firewall
Performance" [1], defines many of the terms that are used in this Performance" [1], defines many of the terms that are used in this
document. The terminology document SHOULD be consulted before document. The terminology document SHOULD be consulted before
attempting to make use of this document. attempting to make use of this document.
2. Requirements 2. Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", In this document, the words that are used to define the significance
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in of each particular requirement are capitalized. These words are:
this document are to be interpreted as described in RFC 2119.
* "MUST" This word, or the words "REQUIRED" and "SHALL" mean that
the item is an absolute requirement of the specification.
* "SHOULD" This word or the adjective "RECOMMENDED" means that there
may exist valid reasons in particular circumstances to ignore this
item, but the full implications should be understood and the case
carefully weighed before choosing a different course.
* "MAY" This word or the adjective "OPTIONAL" means that this item
is truly optional. One vendor may choose to include the item
because a particular marketplace requires it or because it
enhances the product, for example; another vendor may omit the
same item.
An implementation is not compliant if it fails to satisfy one or more
of the MUST requirements for the protocols it implements. An
implementation that satisfies all the MUST and all the SHOULD
requirements for its protocols is said to be "unconditionally
compliant"; one that satisfies all the MUST requirements but not all
the SHOULD requirements for its protocols is said to be
"conditionally compliant".
3. Scope 3. Scope
Firewalls can provide a single point of defense between networks. Firewalls can provide a single point of defense between networks.
Usually, a firewall protects private networks from the public or Usually, a firewall protects private networks from the public or
shared networks to which it is connected. A firewall can be as shared networks to which it is connected. A firewall can be as
simple as a device that filters different packets or as complex simple as a device that filters different packets or as complex
as a group of devices that combine packet filtering and as a group of devices that combine packet filtering and
application-level proxy or network translation services. This RFC application-level proxy or network translation services. This RFC
will focus on developing benchmark testing of DUT/SUTs, wherever will focus on developing benchmark testing of DUT/SUTs, wherever
skipping to change at page 3, line 15 skipping to change at page 3, line 53
In the case of dual-homed configurations, servers which are made In the case of dual-homed configurations, servers which are made
accessible to the public(Unprotected) network are attached to the accessible to the public(Unprotected) network are attached to the
private(Protected) network. private(Protected) network.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Servers/ |----| | | |------| Servers/ | | Servers/ |----| | | |------| Servers/ |
| Clients | | | | | | Clients | | Clients | | | | | | Clients |
| | |-------| DUT/SUT |--------| | | | | |-------| DUT/SUT |--------| | |
+----------+ | | | | +----------+ +----------+ | | | | +----------+
| +----------+ | Protected | +----------+ | Unprotected
Protected | | Unprotected Network | | Network
Network Network
Figure 1(Dual-Homed) Figure 1(Dual-Homed)
Tri-homed[1] configurations employ a third segment called a Tri-homed[1] configurations employ a third segment called a
Demilitarized Zone(DMZ). With tri-homed configurations, servers Demilitarized Zone(DMZ). With tri-homed configurations, servers
accessible to the public network are attached to the DMZ. Tri-Homed accessible to the public network are attached to the DMZ. Tri-Homed
configurations offer additional security by separating server(s) configurations offer additional security by separating server(s)
accessible to the public network from internal hosts. accessible to the public network from internal hosts.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Clients |----| | | |------| Servers/ | | Clients |----| | | |------| Servers/ |
| | | | | | | Clients | | | | | | | | Clients |
skipping to change at page 8, line 5 skipping to change at page 8, line 40
Connection Attempt Rate - The aggregate rate, expressed in Connection Attempt Rate - The aggregate rate, expressed in
connections per second, at which new TCP connection requests are connections per second, at which new TCP connection requests are
attempted. The rate SHOULD be set at or lower than the maximum attempted. The rate SHOULD be set at or lower than the maximum
rate at which the DUT/SUT can accept connection requests. rate at which the DUT/SUT can accept connection requests.
Age Time - The time, expressed in seconds, the DUT/SUT will keep a Age Time - The time, expressed in seconds, the DUT/SUT will keep a
connection in its connection table after receiving a TCP FIN or RST connection in its connection table after receiving a TCP FIN or RST
packet. packet.
5.2.2.2 Transport-Layer Setup Parameters 5.2.2.2 Application-Layer Setup Parameters
Validation Method - HTTP 1.1 or higher MUST be used for this test. Validation Method - HTTP 1.1 or higher MUST be used for this test.
Object Size - Defines the number of bytes, excluding any bytes Object Size - Defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an associated with the HTTP header, to be transferred in response to an
HTTP 1.1 or higher GET request. HTTP 1.1 or higher GET request.
5.2.3 Procedure 5.2.3 Procedure
An iterative search algorithm MAY be used to determine the maximum An iterative search algorithm MAY be used to determine the maximum
skipping to change at page 10, line 25 skipping to change at page 11, line 5
5.3.2.1 Transport-Layer Setup Parameters 5.3.2.1 Transport-Layer Setup Parameters
Number of Connections - Defines the aggregate number of TCP Number of Connections - Defines the aggregate number of TCP
connections that must be established. connections that must be established.
Age Time - The time, expressed in seconds, the DUT/SUT will keep a Age Time - The time, expressed in seconds, the DUT/SUT will keep a
connection in it's state table after receiving a TCP FIN or RST connection in it's state table after receiving a TCP FIN or RST
packet. packet.
5.3.2.2 Transport-Layer Setup Parameters 5.3.2.2 Application-Layer Setup Parameters
Validation Method - HTTP 1.1 or higher MUST be used for this test. Validation Method - HTTP 1.1 or higher MUST be used for this test.
Object Size - Defines the number of bytes, excluding any bytes Object Size - Defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an associated with the HTTP header, to be transferred in response to an
HTTP 1.1 or higher GET request. HTTP 1.1 or higher GET request.
5.3.3 Procedure Test 5.3.3 Procedure
An iterative search algorithm MAY be used to determine the maximum An iterative search algorithm MAY be used to determine the maximum
rate at which the DUT/SUT can accept TCP connection requests. rate at which the DUT/SUT can accept TCP connection requests.
For each iteration, the aggregate rate at which TCP connection For each iteration, the aggregate rate at which TCP connection
requests are attempted by the virtual client(s) will be varied. The requests are attempted by the virtual client(s) will be varied. The
destination address will be that of the server or that of the NAT destination address will be that of the server or that of the NAT
proxy. The aggregate number of connections, defined by number of proxy. The aggregate number of connections, defined by number of
connections, will be attempted in a round-robin fashion(See 4.5). connections, will be attempted in a round-robin fashion(See 4.5).
skipping to change at page 12, line 29 skipping to change at page 13, line 9
the tester will attempt to tear down. the tester will attempt to tear down.
Age Time - The time, expressed in seconds, the DUT/SUT will keep a Age Time - The time, expressed in seconds, the DUT/SUT will keep a
connection in it's state table after receiving a TCP FIN or RST connection in it's state table after receiving a TCP FIN or RST
packet. packet.
5.4.3 Procedure 5.4.3 Procedure
An iterative search algorithm MAY be used to determine the maximum An iterative search algorithm MAY be used to determine the maximum
TCP connection tear down rate. The test iterates through different TCP connection tear down rate. The test iterates through different
5.4.3 Procedure
An iterative search algorithm MAY be used to determine the maximum
TCP connection tear down rate. The test iterates through different
TCP connection tear down rates with a fixed number of TCP TCP connection tear down rates with a fixed number of TCP
connections. connections.
The virtual client(s) will initialize the test by establishing TCP The virtual client(s) will initialize the test by establishing TCP
connections defined by number of connections. The virtual client(s) connections defined by number of connections. The virtual client(s)
will then attempt to tear down all of TCP connections, at a rate will then attempt to tear down all of TCP connections, at a rate
defined by tear down attempt rate. For benchmarking purposes, the defined by tear down attempt rate. For benchmarking purposes, the
tester MUST use a TCP FIN when initiating the connection tear down. tester MUST use a TCP FIN when initiating the connection tear down.
In the case of proxy based DUT/SUTs, the DUT/SUT will itself receive In the case of proxy based DUT/SUTs, the DUT/SUT will itself receive
skipping to change at page 12, line 51 skipping to change at page 13, line 36
verify that the DUT/SUT received the final ACK in the connection tear verify that the DUT/SUT received the final ACK in the connection tear
down exchange for all connections by transmitting a TCP datagram down exchange for all connections by transmitting a TCP datagram
referencing the previously town down connection. A TCP RST should be referencing the previously town down connection. A TCP RST should be
received in response to the TCP datagram. received in response to the TCP datagram.
5.4.4 Measurements 5.4.4 Measurements
Highest connection tear down rate - Highest rate, in connections per Highest connection tear down rate - Highest rate, in connections per
second, for which all TCP connections were successfully torn down. second, for which all TCP connections were successfully torn down.
The following measurements SHOULD performed on a per iteration
basis. The tester MUST only include such measurements for which both
sides of the connection were successfully torn down. For example,
tear down times for connections which are left in a FINWAIT-2[8]
state should not be included:
Minimum connection tear down time - Lowest TCP connection tear down Minimum connection tear down time - Lowest TCP connection tear down
time measured as defined in appendix C. time measured as defined in appendix C.
Maximum connection tear down time - Highest TCP connection tear down Maximum connection tear down time - Highest TCP connection tear down
time measured as defined in appendix C. time measured as defined in appendix C.
Average connection tear down time - The mean of all measurements of Average connection tear down time - The mean of all measurements of
connection tear down times. connection tear down times.
Aggregate connection tear down time - The total of all measurements Aggregate connection tear down time - The total of all measurements
skipping to change at page 13, line 40 skipping to change at page 14, line 29
of service handling test MUST be run after obtaining baseline of service handling test MUST be run after obtaining baseline
measurements from sections 5.3 and/or 5.6. measurements from sections 5.3 and/or 5.6.
The TCP SYN flood attack exploits TCP's three-way handshake The TCP SYN flood attack exploits TCP's three-way handshake
mechanism by having an attacking source host generate TCP SYN mechanism by having an attacking source host generate TCP SYN
packets with random source addresses towards a victim host, thereby packets with random source addresses towards a victim host, thereby
consuming that host's resources. consuming that host's resources.
5.5.2 Setup Parameters 5.5.2 Setup Parameters
Use the same setup parameters as defined in section 5.2.2 or 5.6.2, Use the same setup parameters as defined in section 5.3.2 or 5.6.2,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
setup rate test or HTTP transfer rate test, respectfully. establishment rate test or HTTP transfer rate test, respectfully.
In addition, the following setup parameters MUST be defined. In addition, the following setup parameters MUST be defined.
SYN attack rate - Rate, expressed in packets per second, at which SYN attack rate - Rate, expressed in packets per second, at which
the server(s) or NAT proxy address is targeted with TCP SYN packets. the server(s) or NAT proxy address is targeted with TCP SYN packets.
5.5.3 Procedure 5.5.3 Procedure
Use the same procedure as defined in section 5.3.3 or 5.6.3, Use the same procedure as defined in section 5.3.3 or 5.6.3,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
skipping to change at page 15, line 44 skipping to change at page 16, line 33
MUST be measured and shall be referenced to the requested object(s). MUST be measured and shall be referenced to the requested object(s).
The measurement will start on transmission of the first bit of the The measurement will start on transmission of the first bit of the
first requested object and end on transmission of the last bit of first requested object and end on transmission of the last bit of
the last requested object. The average transfer rate, in bits per the last requested object. The average transfer rate, in bits per
second, will be calculated using the following formula: second, will be calculated using the following formula:
OBJECTS * OBJECTSIZE * 8 OBJECTS * OBJECTSIZE * 8
TRANSFER RATE(bit/s) = -------------------------- TRANSFER RATE(bit/s) = --------------------------
DURATION DURATION
OBJECTS - Objects successfully transferred OBJECTS - Total number of objects successfully transferred across
all connections.
OBJECTSIZE - Object size in bytes OBJECTSIZE - Object size in bytes
DURATION - Aggregate transfer time based on aforementioned time DURATION - Aggregate transfer time based on aforementioned time
references. references.
5.6.4.2 Measurements at or below the Transport-Layer 5.6.4.2 Measurements at or below the Transport-Layer
The tester SHOULD make goodput[1] measurements for connection- The tester SHOULD make goodput[1] measurements for connection-
oriented protocols at or below the transport layer. Goodput oriented protocols at or below the transport layer. Goodput
skipping to change at page 16, line 17 skipping to change at page 17, line 9
Since connection-oriented protocols require that data be Since connection-oriented protocols require that data be
acknowledged, the offered load[6] will vary over the duration of the acknowledged, the offered load[6] will vary over the duration of the
test. When performing forwarding rate measurements, the tester test. When performing forwarding rate measurements, the tester
should measure the average forwarding rate over the duration of the should measure the average forwarding rate over the duration of the
test. test.
5.6.5 Reporting Format 5.6.5 Reporting Format
5.6.5.1 Application-Layer reporting 5.6.5.1 Application-Layer reporting
The test report MUST note number of GET requests per connection, The test report MUST note number of GET requests per connection and
and object size. object size.
The transfer rate results SHOULD be reported in tabular form with The transfer rate results SHOULD be reported in tabular form with a
a row for each of the object sizes. There SHOULD be a column for the row for each of the object sizes. There SHOULD be a column for the
object size, the number of completed requests, the number of object size, the number of completed requests, the number of
completed responses, and the transfer rate results for each test. completed responses, and the transfer rate results for each test.
Failure analysis: Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP The test report SHOULD indicate the number and percentage of HTTP
GET request or responses that failed to complete. GET request or responses that failed to complete.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.6.5.2 Transport-Layer and below reporting 5.6.5.2 Transport-Layer and below reporting
The test report MUST note the aggregate number of connections. In The test report MUST note the aggregate number of connections. In
addition, the report MUST identify the layer/protocol for which the addition, the report MUST identify the protocol for which the
measurement was made. measurement was made.
The results SHOULD be in tabular form with a column for each The results SHOULD be in tabular form with a column for each
iteration of the test. There should be columns for transmitted bits, iteration of the test. There should be columns for transmitted bits,
retransmitted bits and the measured goodput. retransmitted bits and the measured goodput.
Failure analysis: Failure analysis:
The test report SHOULD indicate the number and percentage of The test report SHOULD indicate the number and percentage of
connections that failed to complete. connections that failed to complete.
skipping to change at page 18, line 12 skipping to change at page 18, line 51
in a table format with a column for each iteration. There SHOULD be in a table format with a column for each iteration. There SHOULD be
rows for the number of concurrent transactions attempted, GET rows for the number of concurrent transactions attempted, GET
request rate, number of aborted transactions and number of request rate, number of aborted transactions and number of
transactions active at the end of the test iteration. transactions active at the end of the test iteration.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.8 HTTP Transaction Rate 5.8 Maximum HTTP Transaction Rate
5.8.1 Objective 5.8.1 Objective
Determine the maximum HTTP transaction rate that a DUT/SUT can Determine the maximum HTTP transaction rate that a DUT/SUT can
sustain. sustain.
5.8.2 Setup Parameters 5.8.2 Setup Parameters
Session Type - HTTP 1.1 or higher MUST be used for this test. Session Type - HTTP 1.1 or higher MUST be used for this test.
skipping to change at page 19, line 9 skipping to change at page 19, line 48
completed. completed.
Transaction Time - The tester SHOULD measure minimum, maximum and Transaction Time - The tester SHOULD measure minimum, maximum and
average transaction times. The transaction time will start when the average transaction times. The transaction time will start when the
virtual client issues the GET request and end when the requesting virtual client issues the GET request and end when the requesting
virtual client receives the last bit of the requested object. virtual client receives the last bit of the requested object.
5.8.5 Reporting Format 5.8.5 Reporting Format
The test report MUST note the test duration, object size, requests The test report MUST note the test duration, object size, requests
per connection and the measured maximum transaction rate. per connection and the measured minimum, maximum and average
transaction rate.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported
in a table format with a column for each iteration. There SHOULD be in a table format with a column for each iteration. There SHOULD be
rows for the GET request attempt rate, number of requests attempted, rows for the GET request attempt rate, number of requests attempted,
number and percentage of requests completed, number of responses number and percentage of requests completed, number of responses
attempted, number and percentage of responses completed, minimum attempted, number and percentage of responses completed, minimum
transaction time, average transaction time and maximum transaction transaction time, average transaction time and maximum transaction
time. time.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.9 Illegal Traffic Handling 5.9 Illegal Traffic Handling
5.9.1 Objective 5.9.1 Objective
To determine the behavior of the DUT/SUT when presented with a To determine the behavior of the DUT/SUT when presented with a
combination of both legal and Illegal traffic flows. Note that combination of both legal and Illegal traffic. Note that Illegal
Illegal traffic does not refer to an attack, but traffic which traffic does not refer to an attack, but traffic which has been
has been explicitly defined by a rule(s) to drop. explicitly defined by a rule(s) to drop.
5.9.2 Setup Parameters 5.9.2 Setup Parameters
Setup parameters will use the same parameters as specified in the Setup parameters will use the same parameters as specified in the
HTTP transfer rate test. In addition, the following setup HTTP transfer rate test(Section 5.6.2). In addition, the following
parameters MUST be defined: setup parameters MUST be defined:
Illegal traffic percentage - Percentage of HTTP 1.1 or higher Illegal traffic percentage - Percentage of HTTP 1.1 or higher
connections which have been explicitly defined in a rule(s) to drop. connections which have been explicitly defined in a rule(s) to drop.
5.9.3 Procedure 5.9.3 Procedure
Each HTTP 1.1 or higher client will request one or more objects from Each HTTP 1.1 or higher client will request one or more objects from
an HTTP 1.1 or higher server using one or more HTTP GET requests. an HTTP 1.1 or higher server using one or more HTTP GET requests.
The aggregate number of connections attempted, defined by number of The aggregate number of connections attempted, defined by number of
connections, MUST be evenly divided among all of the participating connections, MUST be evenly divided among all of the participating
skipping to change at page 20, line 8 skipping to change at page 20, line 46
The virtual client(s) MUST offer the connection requests, both legal The virtual client(s) MUST offer the connection requests, both legal
and illegal, in an evenly distributed manner. Many firewalls have and illegal, in an evenly distributed manner. Many firewalls have
the capability to filter on different traffic criteria( IP the capability to filter on different traffic criteria( IP
addresses, Port numbers, etc). Testers may run multiple addresses, Port numbers, etc). Testers may run multiple
iterations of this test with the DUT/SUT configured to filter iterations of this test with the DUT/SUT configured to filter
on different traffic criteria. on different traffic criteria.
5.9.4 Measurements 5.9.4 Measurements
Tester SHOULD perform the same measurements as defined in HTTP Tester SHOULD perform the same measurements as defined in HTTP
test(Section 5.6.4). Unlike the HTTP transfer rate test, the transfer rate test(Section 5.6.4). Unlike the HTTP transfer rate
tester MUST not include any bits which are associated with illegal test, the tester MUST not include any bits which are associated
traffic in its forwarding rate measurements. with illegal traffic in its forwarding rate measurements.
5.9.5 Reporting Format 5.9.5 Reporting Format
Test report SHOULD be the same as specified in the HTTP Test report SHOULD be the same as specified in the HTTP
test(Section 5.6.5). test(Section 5.6.5).
In addition, the report MUST note the percentage of illegal HTTP In addition, the report MUST note the percentage of illegal HTTP
connections. connections.
Failure analysis: Failure analysis:
skipping to change at page 21, line 23 skipping to change at page 22, line 6
Each HTTP 1.1 or higher client will request one or more objects from Each HTTP 1.1 or higher client will request one or more objects from
an HTTP 1.1 or higher server using one or more HTTP GET requests. an HTTP 1.1 or higher server using one or more HTTP GET requests.
The aggregate number of connections attempted, defined by number of The aggregate number of connections attempted, defined by number of
connections, MUST be evenly divided among all of the participating connections, MUST be evenly divided among all of the participating
virtual clients. If the virtual client(s) make multiple HTTP GET virtual clients. If the virtual client(s) make multiple HTTP GET
requests per connection, it MUST request the same object size for requests per connection, it MUST request the same object size for
each GET request. each GET request.
A tester attached to the unprotected side of the network, will offer A tester attached to the unprotected side of the network, will offer
a unidirectional stream of unicast IP/UDP targeting a server a unidirectional stream of unicast fragmented IP/UDP traffic,
attached to either the protected or DMZ. The tester MUST offer the targeting a server attached to either the protected or DMZ segment.
unidirectional stream over the duration of the test. The tester MUST offer the unidirectional stream over the duration of
the test -- that is, duration over which the HTTP traffic is being
offered.
Baseline measurements SHOULD be performed with IP filtering deny Baseline measurements SHOULD be performed with IP filtering deny
rule(s) to filter fragmented traffic. If the DUT/SUT has logging rule(s) to filter fragmented traffic. If the DUT/SUT has logging
capability, the log SHOULD be checked to determine if it contains capability, the log SHOULD be checked to determine if it contains
the correct information regarding the fragmented traffic. the correct information regarding the fragmented traffic.
The test SHOULD be repeated with the DUT/SUT rule set changed to The test SHOULD be repeated with the DUT/SUT rule set changed to
allow the fragmented traffic through. When running multiple allow the fragmented traffic through. When running multiple
iterations of the test, it is RECOMMENDED to vary the MTU while iterations of the test, it is RECOMMENDED to vary the MTU while
keeping all other parameters constant. keeping all other parameters constant.
skipping to change at page 25, line 5 skipping to change at page 25, line 27
Failure analysis: Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP The test report SHOULD indicate the number and percentage of HTTP
GET request or responses that failed to complete within the test GET request or responses that failed to complete within the test
duration. duration.
Version information: Version information:
The test report MUST note the version of HTTP client and server. The test report MUST note the version of HTTP client and server.
6. References
[1] D. Newman, "Benchmarking Terminology for Firewall Devices",
RFC 2647, August 1999.
[2] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter,
P. Leach, T. Berners-Lee , "Hypertext Transfer Protocol -
HTTP/1.1", RFC 2616 June 1999.
[3] S. Bradner, editor. "Benchmarking Terminology for Network
Interconnection Devices," RFC 1242, July 1991.
[4] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network
Interconnect Devices," RFC 2544, March 1999.
[5] David C. Clark, "IP Datagram Reassembly Algorithm", RFC 815 ,
July 1982.
[6] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998.
[7] Mandeville, R., Perser,J., "Benchmarking Methodology for LAN
Switching Devices", RFC 2889, August 2000.
[8] Postel, J. (ed.), "Internet Protocol - DARPA Internet Program
Protocol Specification", RFC 793, USC/Information Sciences
Institute, September 1981.
7. Security Considerations
The primary goal of this document is to provide methodologies in
benchmarking firewall performance. While there is some overlap
between performance and security issues, assessment of firewall
security is outside the scope of this document.
8. Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
9. Authors' Addresses
Brooks Hickman
Spirent Communications
26750 Agoura Road
Calabasas, CA 91302
USA
Phone: + 1 818 676 2412
Email: brooks.hickman@spirentcom.com
David Newman
Network Test Inc.
31324 Via Colinas, Suite 113
Westlake Village, CA 91362-6761
USA
Phone: + 1 818 889-0011
Email: dnewman@networktest.com
Saldju Tadjudin
Spirent Communications
26750 Agoura Road
Calabasas, CA 91302
USA
Phone: + 1 818 676 2468
Email: saldju.Tadjudin@spirentcom.com
Terry Martin
GVNW Consulting Inc.
8050 SW Warm Springs Road
Tualatin Or. 97062
USA
Phone: + 1 503 612 4422
Email: tmartin@gvnw.com
APPENDIX A: HTTP(HyperText Transfer Protocol) APPENDIX A: HTTP(HyperText Transfer Protocol)
The most common versions of HTTP in use today are HTTP/1.0 and The most common versions of HTTP in use today are HTTP/1.0 and
HTTP/1.1 with the main difference being in regard to persistent HTTP/1.1 with the main difference being in regard to persistent
connections. HTTP 1.0, by default, does not support persistent connections. HTTP 1.0, by default, does not support persistent
connections. A separate TCP connection is opened up for each connections. A separate TCP connection is opened up for each
GET request the client wants to initiate and closed after the GET request the client wants to initiate and closed after the
requested object transfer is completed. While some implementations requested object transfer is completed. While some implementations
HTTP/1.0 supports persistence through the use of a keep-alive, HTTP/1.0 supports persistence through the use of a keep-alive,
there is no official specification for how the keep-alive operates. there is no official specification for how the keep-alive operates.
skipping to change at page 25, line 34 skipping to change at page 27, line 34
closes inactive connections, the aging timer on the DUT SHOULD closes inactive connections, the aging timer on the DUT SHOULD
be configured for a duration that exceeds the test time. be configured for a duration that exceeds the test time.
While this document cannot foresee future changes to HTTP While this document cannot foresee future changes to HTTP
and it impact on the methodologies defined herein, such and it impact on the methodologies defined herein, such
changes should be accommodated for so that newer versions of changes should be accommodated for so that newer versions of
HTTP may be used in benchmarking firewall performance. HTTP may be used in benchmarking firewall performance.
APPENDIX B: Connection Establishment Time Measurements APPENDIX B: Connection Establishment Time Measurements
For purposes of benchmarking firewall performance, the connection
establishment time will be considered the interval between the
transmission of the first bit of the first octet of the packet
carrying the connection request to the DUT/SUT interface to
receipt of the last bit of the last octet of the last packet of
the connection setup traffic received on the client or server,
depending on whether a given connection requires an even or odd
number of messages, respectfully.
Some connection oriented protocols, such as TCP, involve an odd Some connection oriented protocols, such as TCP, involve an odd
number of messages when establishing a connection. In the case of number of messages when establishing a connection. In the case of
proxy based DUT/SUTs, the DUT/SUT will terminate the connection, proxy based DUT/SUTs, the DUT/SUT will terminate the connection,
setting up a separate connection to the server. Since, in such setting up a separate connection to the server. Since, in such
cases, the tester does not own both sides of the connection, cases, the tester does not own both sides of the connection,
measurements will be made two different ways. While the following measurements will be made two different ways. While the following
describes the measurements with reference to TCP, the methodology describes the measurements with reference to TCP, the methodology
may be used with other connection oriented protocols which involve may be used with other connection oriented protocols which involve
an odd number of messages. an odd number of messages.
For non-proxy based DUT/SUTs , the establishment time shall be When testing non-proxy based DUT/SUTs , the establishment time shall
directly measured and is considered to be from the time the first be directly measured and is considered to be from the time the first
bit of the first SYN packet is transmitted by the client to the bit of the first SYN packet is transmitted by the client to the
time the last bit of the final ACK in the three-way handshake is time the last bit of the final ACK in the three-way handshake is
received by the target server. received by the target server.
If the DUT/SUT is proxy based, the connection establishment time is If the DUT/SUT is proxy based, the connection establishment time is
considered to be from the time the first bit of the first SYN packet considered to be from the time the first bit of the first SYN packet
is transmitted by the client to the time the client transmits the first is transmitted by the client to the time the client transmits the
bit of the first acknowledged TCP datagram(t4-t0 in the following first bit of the first acknowledged TCP datagram(t4-t0 in the
timeline). following timeline).
t0: Client sends a SYN. t0: Client sends a SYN.
t1: Proxy sends a SYN/ACK. t1: Proxy sends a SYN/ACK.
t2: Client sends the final ACK. t2: Client sends the final ACK.
t3: Proxy establishes separate connection with server. t3: Proxy establishes separate connection with server.
t4: Client sends TCP datagram to server. t4: Client sends TCP datagram to server.
*t5: Proxy sends ACK of the datagram to client. *t5: Proxy sends ACK of the datagram to client.
* While t5 is not considered part of the TCP connection establishment, * While t5 is not considered part of the TCP connection
acknowledgement of t4 must be received for the connection to be establishment, acknowledgement of t4 must be received for the
considered successful. connection to be considered successful.
APPENDIX C: Connection Tear Time Measurements APPENDIX C: Connection Tear Time Measurements
The TCP connection tear down time will be considered the interval While TCP connections are full duplex, tearing down of such connections
between the transmission of the first TCP FIN packet transmitted are performed in a simplex fashion -- that is, FIN segments are sent by
by the tester requesting a connection tear down to receipt of the each host/device terminating each side of the TCP connection.
ACK packet on the same tester interface.
Appendix D. References
[1] D. Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647,
August 1999.
[2] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter, P. Leach,
T. Berners-Lee , "Hypertext Transfer Protocol -- HTTP/1.1",
RFC 2616 June 1999
[3] S. Bradner, editor. "Benchmarking Terminology for Network
Interconnection Devices," RFC 1242, July 1991.
[4] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network When making connection tear down times measurements, such measurements
Interconnect Devices," RFC 2544, March 1999. will be made from the perspective of the client and will be performed
in the same manner, independent of whether or not the DUT/SUT is
proxy-based. The connection tear down will be considered the interval
between the transmission of the first bit of the first TCP FIN packet
transmitted by the tester requesting a connection tear down to receipt
of the last bit of the corresponding ACK packet on the same tester
interface.
[5] David C. Clark, "IP Datagram Reassembly Algorithm", RFC 815 , Full Copyright Statement
July 1982.
[6] Mandeville, R., "Benchmarking Terminology for LAN Switching Copyright (C) The Internet Society (2002). All Rights Reserved.
Devices", RFC 2285, February 1998.
[7] Mandeville, R., Perser,J., "Benchmarking Methodology for LAN This document and translations of it may be copied and furnished to
Switching Devices", RFC 2889, August 2000. others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph
are included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
english. The limited permissions granted above are perpetual and
will not be revoked by the Internet Society or its successors or
assigns. This document and the information contained herein is
provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/