draft-ietf-bmwg-firewall-02.txt   draft-ietf-bmwg-firewall-03.txt 
Network Working Group Brooks Hickman Network Working Group Brooks Hickman
Internet-Draft Spirent Communications Internet-Draft Spirent Communications
Expiration Date: December 2001 David Newman Expiration Date: April 2002 David Newman
Network Test Network Test
Saldju Tadjudin
Spirent Communications
Terry Martin Terry Martin
M2networx INC M2networx INC
June 2001 October 2001
Benchmarking Methodology for Firewall Performance Benchmarking Methodology for Firewall Performance
<draft-ietf-bmwg-firewall-02.txt> <draft-ietf-bmwg-firewall-03.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 4, line 7 skipping to change at page 4, line 7
4.2 Virtual Clients/Servers 4.2 Virtual Clients/Servers
Since firewall testing may involve data sources which emulate Since firewall testing may involve data sources which emulate
multiple users or hosts, the methodology uses the terms virtual multiple users or hosts, the methodology uses the terms virtual
clients/servers. For these firewall tests, virtual clients/servers clients/servers. For these firewall tests, virtual clients/servers
specify application layer entities which may not be associated with specify application layer entities which may not be associated with
a unique physical interface. For example, four virtual clients may a unique physical interface. For example, four virtual clients may
originate from the same data source[1]. The test report SHOULD originate from the same data source[1]. The test report SHOULD
indicate the number of virtual clients and virtual servers indicate the number of virtual clients and virtual servers
participating in the test on a per interface(See 4.1.3) basis. participating in the test on a per interface(See 4.0) basis.
Testers MUST synchronize all data sources participating in a test. Testers MUST synchronize all data sources participating in a test.
4.3 Test Traffic Requirements 4.3 Test Traffic Requirements
While the function of a firewall is to enforce access control While the function of a firewall is to enforce access control
policies, the criteria by which those policies are defined vary policies, the criteria by which those policies are defined vary
depending on the implementation. Firewalls may use network layer, depending on the implementation. Firewalls may use network layer,
transport layer or, in many cases, application-layer criteria to transport layer or, in many cases, application-layer criteria to
make access-control decisions. Therefore, the test equipment used to make access-control decisions. Therefore, the test equipment used to
skipping to change at page 5, line 36 skipping to change at page 5, line 36
across the unprotected network(Internet). This involves additional across the unprotected network(Internet). This involves additional
processing on the part of the DUT/SUT and may impact performance. processing on the part of the DUT/SUT and may impact performance.
Therefore, tests SHOULD be ran with NAT disabled and NAT enabled Therefore, tests SHOULD be ran with NAT disabled and NAT enabled
to determine the performance differentials. The test report SHOULD to determine the performance differentials. The test report SHOULD
indicate whether NAT was enabled or disabled. indicate whether NAT was enabled or disabled.
4.7 Rule Sets 4.7 Rule Sets
Rule sets[1] are a collection of access control policies that Rule sets[1] are a collection of access control policies that
determines which packets the DUT/SUT will forward and which it will determines which packets the DUT/SUT will forward and which it will
reject. The criteria by which these access control policies may be reject[1]. The criteria by which these access control policies may
defined will vary depending on the capabilities of the DUT/SUT. The be defined will vary depending on the capabilities of the DUT/SUT.
scope of this document is limited to how the rule sets should be The scope of this document is limited to how the rule sets should
applied when testing the DUT/SUT. be applied when testing the DUT/SUT.
The firewall monitors the incoming traffic and checks to make sure The firewall monitors the incoming traffic and checks to make sure
that the traffic meets one of the defined rules before allowing it that the traffic meets one of the defined rules before allowing it
to be forwarded. It is RECOMMENDED that a rule be entered for each to be forwarded. It is RECOMMENDED that a rule be entered for each
host(Virtual client). Although many firewalls permit groups of IP host(Virtual client). Although many firewalls permit groups of IP
addresses to be defined for a given rule, tests SHOULD be performed addresses to be defined for a given rule, tests SHOULD be performed
with large rule sets, which are more stressful to the DUT/SUT. with large rule sets, which are more stressful to the DUT/SUT.
The DUT/SUT SHOULD be configured to denies access to all traffic The DUT/SUT SHOULD be configured to denies access to all traffic
which was not previously defined in the rule set. which was not previously defined in the rule set.
skipping to change at page 6, line 24 skipping to change at page 6, line 24
authentication servers and may add to the latency of the system. authentication servers and may add to the latency of the system.
Any authentication processes MUST be included as part of connection Any authentication processes MUST be included as part of connection
setup process. setup process.
5. Benchmarking Tests 5. Benchmarking Tests
5.1 Concurrent Connection Capacity 5.1 Concurrent Connection Capacity
5.1.1 Objective 5.1.1 Objective
To determine the maximum number of concurrent connections through To determine the maximum number of TCP concurrent connections through
or with the DUT/SUT, as defined in RFC2647[1]. This test will employ or with the DUT/SUT, as defined in RFC2647[1]. This test will employ
a step algorithm to obtain the maximum number of concurrent TCP a step algorithm to obtain the maximum number of concurrent TCP
connections that the DUT/SUT can maintain. connections that the DUT/SUT can maintain.
5.1.2 Setup Parameters 5.1.2 Setup Parameters
The following parameters MUST be defined for all tests. The following parameters MUST be defined for all tests.
Connection Attempt Rate - The rate, expressed in connections per Connection Attempt Rate - The rate, expressed in connections per
second, at which new TCP connection requests are attempted. The second, at which new TCP connection requests are attempted. The
rate SHOULD be set lower than maximum rate at which the DUT/SUT can rate SHOULD be set lower than maximum rate at which the DUT/SUT can
accept connection requests. accept connection requests.
Connection Step Count - Defines the number of additional TCP Connection Step Count - Defines the number of additional TCP
connections attempted for each iteration of the step search connections attempted for each iteration of the step search
algorithm. algorithm.
Object Size - Defines the number of bytes to be transferred in Object Size - Defines the number of bytes to be transferred in
response to a HTTP 1.1 GET request . It is RECOMMENDED to use the response to a HTTP 1.1 GET request . It is RECOMMENDED to use
minimum object size supported by the media. 1-byte object sizes for this test.
5.1.3 Procedure 5.1.3 Procedure
Each virtual client will attempt to establish TCP connections to its Each virtual client will attempt to establish TCP connections to its
target server(s), using either the target server's IP address or NAT target server(s), using either the target server's IP address or NAT
proxy address, at a fixed rate in a round robin fashion. Each proxy address, at a fixed rate in a round robin fashion. Each
iteration will involve the virtual clients attempting to establish a iteration will involve the virtual clients attempting to establish a
fixed number of additional TCP connections. This search algorithm fixed number of additional TCP connections. This search algorithm
will be repeated until either: will be repeated until either:
- One or more of the additional connection attempts fail to - One or more of the additional connection attempts fail to
complete. complete.
- One or more of the previously established connections fail. - One or more of the previously established connections fail.
The test MUST also include application layer data transfers in The test MUST include application layer data transfers in order to
order to validate the TCP connections since, in the case of proxy validate the TCP connections since, in the case of proxy based
based DUT/SUTs, the tester does not own both sides of the DUT/SUTs, the tester does not own both sides of the connection.
connection. For the purposes of validation, the virtual client(s) After all the addition connections have been attempted for each
will request an object from its target server(s) using an HTTP 1.1 iteration of the test, the virtual client(s) will request an
GET request, with both the client request and server response object from its target server(s) using an HTTP 1.1 GET request on
excluding the connection-token close in the connection header. In the additional connections as well as all previously established
addition, periodic HTTP GET requests MAY be required to keep the connections. Both the client request and server response MUST exclude
underlying TCP connection open(See Appendix A). the connection-token close in the connection header(See Appendix A).
5.1.4 Measurements 5.1.4 Measurements
Maximum concurrent connections - Total number of TCP connections Maximum concurrent connections - Total number of TCP connections
open for the last successful iteration performed in the search open for the last successful iteration performed in the search
algorithm. algorithm.
5.1.5 Reporting Format 5.1.5 Reporting Format
5.1.5.1 Transport-Layer Reporting: 5.1.5.1 Application-Layer Reporting:
The test report MUST note the connection attempt rate, connection The test report MUST note the use of HTTP 1.1 client and server
step count and maximum concurrent connections measured. and the object size.
5.1.5.2 Application-Layer Reporting: 5.1.5.2 Transport-Layer Reporting:
The test report MUST note the object size(s) and the use of The test report MUST note the connection attempt rate, connection
HTTP 1.1 client and server. step count and maximum concurrent connections measured.
5.1.5.3 Log Files 5.1.5.3 Log Files
A log file MAY be generated which includes the TCP connection A log file MAY be generated which includes the TCP connection
attempt rate, HTTP object size and for each iteration: attempt rate, connection step count, object size and for each
iteration:
- Step Iteration - Step Iteration
- Pass/Fail Status. - Pass/Fail Status.
- Total TCP connections established. - Total TCP connections established.
- Number of previously established TCP connections dropped. - Number of previously established TCP connections that failed.
- Number of the additional TCP connections that failed to - Number of the additional TCP connections that failed to
complete. complete.
5.2 Maximum Connection Setup Rate 5.2 Maximum Connection Setup Rate
5.2.1 Objective 5.2.1 Objective
To determine the maximum TCP connection setup rate through or with To determine the maximum TCP connection setup rate through or with
the DUT/SUT, as defined by RFC2647[1]. This test will employ a the DUT/SUT, as defined by RFC2647[1]. This test will employ a
search algorithm to obtain the maximum rate at which TCP connections search algorithm to obtain the maximum rate at which TCP connections
skipping to change at page 8, line 15 skipping to change at page 8, line 15
5.2.2 Setup Parameters 5.2.2 Setup Parameters
The following parameters MUST be defined. The following parameters MUST be defined.
Initial Attempt Rate - The rate, expressed in connections per Initial Attempt Rate - The rate, expressed in connections per
second, at which the initial TCP connection requests are attempted. second, at which the initial TCP connection requests are attempted.
Number of Connections - Defines the number of TCP connections that Number of Connections - Defines the number of TCP connections that
must be established. The number MUST be between the number of must be established. The number MUST be between the number of
participating virtual clients and the maximum number supported by participating virtual clients and the maximum number supported by
the DUT/SUT. It is RECOMMENDED not to exceed the concurrent the DUT/SUT.
connection capacity found in section 5.1.
Connection Teardown Rate - The rate, expressed in connections per Object Size - Defines the number of bytes to be transferred in
second, at which the tester will attempt to teardown TCP connections response to a HTTP 1.1 GET request . It is RECOMMENDED to use
between each iteration. The connection teardown rate SHOULD be set 1-byte object sizes for this test.
lower than rate at which the DUT/SUT can teardown TCP connections.
Age Time - The time, expressed in seconds, the DUT/SUT will keep a Age Time - The time, expressed in seconds, the DUT/SUT will keep a
connection in it's state table after receiving a TCP FIN or RST connection in it's state table after receiving a TCP FIN or RST
packet. packet.
Object Size - Defines the number of bytes to be transferred in
response to a HTTP 1.1 GET request . It is RECOMMENDED to use the
minimum object size supported by the media.
5.2.3 Procedure 5.2.3 Procedure
An iterative search algorithm will be used to determine the maximum An iterative search algorithm will be used to determine the maximum
connection rate. This test iterates through different connection rates connection rate. This test iterates through different connection rates
with a fixed number of connections attempted by the virtual clients to with a fixed number of connections attempted by the virtual clients to
their associated server(s). their associated server(s).
Each iteration will use the same connection establishment and Each iteration will use the same connection establishment and
connection validation algorithms defined in the concurrent capacity connection validation algorithms defined in the concurrent capacity
test(See section 5.1). test(See section 5.1).
skipping to change at page 10, line ? skipping to change at page 8, line 49
The tester will wait for the period of time, specified by age time, The tester will wait for the period of time, specified by age time,
before continuing to the next iteration. before continuing to the next iteration.
5.2.4 Measurements 5.2.4 Measurements
Highest connection rate - Highest rate, in connections per second, Highest connection rate - Highest rate, in connections per second,
for which all TCP connections completed successfully. for which all TCP connections completed successfully.
5.2.5 Reporting Format 5.2.5 Reporting Format
5.2.5.1 Transport-Layer Reporting: 5.1.5.1 Application-Layer Reporting:
The test report MUST note the number of connections attempted, The test report MUST note the use of HTTP 1.1 client and server
connection teardown rate, age time, and highest connection rate and the object size.
measured.
5.1.5.2 Application-Layer Reporting: 5.2.5.2 Transport-Layer Reporting:
The test report MUST note the object size(s) and the use of The test report MUST note the number of connections, age time
HTTP 1.1 client and server. and highest connection rate measured.
5.1.5.3 Log Files
A log file MAY be generated which includes the total TCP connections 5.2.5.3 Log Files
attempt, TCP connection teardown rate, age time, HTTP object size and
A log file MAY be generated which includes the number of TCP
connections attempt, age time, object size and
for each iteration: for each iteration:
- Step Iteration - Step Iteration
- Pass/Fail Status. - Pass/Fail Status.
- Attempted Connection Establishment Rate
- Total TCP connections established. - Total TCP connections established.
- Number of TCP connections that failed to complete. - Number of TCP connections that failed to complete.
5.3 Connection Establishment Time 5.3 Connection Establishment Time
5.3.1 Objective 5.3.1 Objective
To determine the connection establishment times[1] through or with To determine the connection establishment times[1] through or with
the DUT/SUT as a function of the number of open connections. the DUT/SUT.
A connection for a client/server application is not atomic, in that A connection for a client/server application is not atomic, in that
it not only involves transactions at the application layer, but it not only involves transactions at the application layer, but
involves first establishing a connection using one or more underlying involves first establishing a connection using one or more underlying
connection oriented protocols(TCP, ATM, etc). Therefore, it is connection oriented protocols(TCP, ATM, etc). Therefore, it is
encouraged to make separate measurements for each connection oriented encouraged to make separate measurements for each connection oriented
protocol required in order to perform the application layer protocol required in order to perform the application layer
transaction. transaction.
5.3.2 Setup Parameters 5.3.2 Setup Parameters
skipping to change at page 10, line ? skipping to change at page 10, line ?
Connection Attempt Rate - The rate, expressed in connections per Connection Attempt Rate - The rate, expressed in connections per
second, at which new TCP connection requests are attempted. It is second, at which new TCP connection requests are attempted. It is
RECOMMENDED not to exceed the maximum connection rate found in RECOMMENDED not to exceed the maximum connection rate found in
section 5.2. section 5.2.
Connection Attempt Step count - Defines the number of additional Connection Attempt Step count - Defines the number of additional
TCP connections attempted for each iteration of the step algorithm. TCP connections attempted for each iteration of the step algorithm.
Maximum Attempt Connection Count - Defines the maximum number of Maximum Attempt Connection Count - Defines the maximum number of
TCP connections attempted in the test. It is RECOMMENDED not to TCP connections attempted in the test.
exceed the concurrent connection capacity found in section 5.1.
Hickman, Newman, Martin [Page 9]
Object Size - Defines the number of bytes to be transferred in
response to a HTTP 1.1 GET request.
Number of requests - Defines the number of HTTP 1.1 GET requests
per connection. Note that connection, in this case, refers to the
underlying transport protocol.
5.3.3 Procedure 5.3.3 Procedure
Each virtual client will attempt to establish TCP connections to its Each virtual client will attempt to establish TCP connections to its
target server(s) at a fixed rate in a round robin fashion. Each target server(s) at a fixed rate in a round robin fashion. Each
iteration will involve the virtual clients attempting to establish iteration will involve the virtual clients attempting to establish
a fixed number of additional connections until the maximum attempt a fixed number of additional connections until the maximum attempt
connection count is reached. connection count is reached.
As with the concurrent capacity tests, application layer data After each connection has been completed, the virtual client(s) MUST
transfers will be performed. Each virtual client(s) will request request a 1-byte object from its target server(s) using an HTTP 1.1
one or more objects from its target server(s) using one or more GET request. Both the client request and server response MUST exclude
HTTP 1.1 GET request, with both the client request and server
response excluding the connection-token close in the connection [Page 9]
header. In addition, periodic HTTP GET requests MAY be required to the connection-token close in the connection header(See Appendix A).
keep the underlying TCP connection open(See appendix A).
Since testing may involve proxy based DUT/SUTs, which terminates the Since testing may involve proxy based DUT/SUTs, which terminates the
TCP connection, making a direct measurement of the TCP connection TCP connection, making a direct measurement of the TCP connection
establishment time is not possible since the protocol involves an establishment time is not possible since the protocol involves an
odd number of messages in establishing a connection. Therefore, when odd number of messages in establishing a connection. Therefore, when
testing with proxy based firewalls, the datagram following the final testing with proxy based firewalls, the datagram following the final
ACK on the three-way handshake will be used in determining the ACK on the three-way handshake will be used in determining the
connection setup time. connection setup time.
The following shows the timeline for the TCP connection setup The following shows the timeline for the TCP connection setup
involving a proxy DUT/SUT and is referenced in the measurement involving a proxy DUT/SUT and is referenced in the measurements
section. Note that this method may be applied when measuring other section(5.3.4). Note that this methodology may be applied when
connection oriented protocols involving an odd number of messages measuring other connection oriented protocols involving an odd number
in establishing a connection. of messages in establishing a connection.
t0: Client sends a SYN. t0: Client sends a SYN.
t1: Proxy sends a SYN/ACK. t1: Proxy sends a SYN/ACK.
t2: Client sends the final ACK. t2: Client sends the final ACK.
t3: Proxy establishes separate connection with server. t3: Proxy establishes separate connection with server.
t4: Client sends TCP datagram to server. t4: Client sends TCP datagram to server.
*t5: Proxy sends ACK of the datagram to client. *t5: Proxy sends ACK of the datagram to client.
* While t5 is not considered part of the TCP connection establishment, * While t5 is not considered part of the TCP connection establishment,
acknowledgement of t4 must be received for the connection to be acknowledgement of t4 must be received for the connection to be
considered successful. considered successful.
When comparing firewalls with different architectures, such as proxy
based and stateful packet filtering, the same method SHOULD be used
when measuring establishment times.
5.3.4 Measurements 5.3.4 Measurements
For each iteration of the test, the tester MUST measure the minimum, For each iteration of the test, the tester MUST measure the minimum,
maximum and average TCP connection establishment times. Measuring TCP maximum and average TCP connection establishment times. If the DUT/SUT
connection establishment times will be made two different ways, is proxy based, the connection establishment time is considered to be
depending on whether or not the DUT/SUT is proxy based. If proxy from the time the first bit of the first SYN packet is transmitted by
based, the connection establishment time is considered to be from the the client to the time the client transmits the first bit of the first
time the first bit of the SYN packet is transmitted by the client to acknowledged TCP datagram(t4-t0 in the above timeline). For non-proxy
the time the client transmits the first bit of the TCP datagram, based DUT/SUTs , the establishment time shall be directly measured and
provided that the TCP datagram gets acknowledged(t4-t0 in the above is considered to be from the time the first bit of the first SYN packet
timeline). For DUT/SUTs that are not proxy based, the establishment is transmitted by the client to the time the last bit of the final ACK
time shall be directly measured and is considered to be from the time in the three-way handshake is received by the target server.
the first bit of the SYN packet is transmitted by the client to the
time the last bit of the final ACK in the three-way handshake is
received by the target server.
In addition, the tester SHOULD measure the minimum, maximum and In addition, the tester SHOULD measure the minimum, maximum and
average connection establishment times for all other underlying average connection establishment times for all other underlying
connection oriented protocols which are required to be established connection oriented protocols. For purposes of benchmarking
for the client/server application to transfer an object. Each firewall performance, the connection establishment time will be
connection oriented protocol has its own set of transactions considered the interval between the transmission of the first bit
required for establishing a connection between two hosts or a host of the first octet of the packet carrying the connection request
and DUT/SUT. For purposes of benchmarking firewall performance, the to the DUT/SUT interface to receipt of the last bit of the last
connection establishment time will be considered the interval octet of the last packet of the connection setup traffic received
between the transmission of the first bit of the first octet of the on the client or server, depending on whether a given connection
packet carrying the connection request to receipt of the last bit of requires an even or odd number of messages, respectfully.
the last octet of the last packet of the connection setup traffic
received on the client or server, depending on whether a given Tester SHOULD measure the aggregate connection time and the total
connection requires an even or odd number of messages, respectfully. number of connections completed for all measured protocols for each
iteration of the test.
5.3.5 Reporting Format 5.3.5 Reporting Format
5.3.5.1 Application-Layer Reporting:
The test report MUST note the use of HTTP 1.1 client and server.
5.3.5.2 Transport-Layer and Below Reporting:
The test report MUST note the TCP connection attempt rate, TCP The test report MUST note the TCP connection attempt rate, TCP
connection attempt step count and maximum TCP connections attempted, connection attempt step count and maximum TCP connections attempted.
HTTP object size and number of requests per connection.
For each connection oriented protocol the tester measured, the For each connection oriented protocol the tester measured, the
connection establishment time results SHOULD be in tabular form connection establishment time results SHOULD be in tabular form
with a row for each iteration of the test. There SHOULD be a column with a row for each iteration of the test. There SHOULD be a column
for the iteration count, minimum connection establishment time, for the iteration count, minimum connection establishment time,
average connection establishment time, maximum connection average connection establishment time, maximum connection
establishment time, attempted connections completed, attempted establishment time, attempted connections completed and aggregate
connections failed. connection time.
The report MUST also identify the layer/protocol for which the
measurements were made.
5.4 Connection Teardown Time 5.4 Connection Teardown Time
5.4.1 Objective 5.4.1 Objective
To determine the connection teardown time[1] through or with the To determine the connection teardown time[1] through or with the
DUT/SUT as a function of the number of open connections. As with the DUT/SUT. As with the connection establishment time, separate
connection establishment time, separate measurements will be taken measurements SHOULD be taken for each connection oriented protocol
for each connection oriented protocol involved in closing a involved in closing a connection.
connection.
5.4.2 Setup Parameters 5.4.2 Setup Parameters
The following parameters MUST be defined. Each parameters is The following parameters MUST be defined. Each parameters is
configured with the following considerations. configured with the following considerations.
Initial connections - Defines the number of TCP connections to Initial connections - Defines the number of TCP connections to
initialize the test with. It is RECOMMENDED not to exceed the initialize the test with.
concurrent connection capacity found in section 5.1.
Initial connection rate - Defines the rate, in connections per
second, at which the initial TCP connections are attempted. It is
RECOMMENDED not to exceed the maximum Connection setup rate found
in section 5.2.
Teardown attempt rate - The rate at which the tester will attempt Teardown attempt rate - The rate at which the tester will attempt
to teardown TCP connections. to teardown TCP connections.
Teardown step count - Defines the number of TCP connections the
tester will attempt to teardown for each iteration of the step
algorithm.
Object size - Defines the number of bytes to be transferred across
each connection in response to an HTTP 1.1 GET request during the
initialization phase of the test as well as periodic GET requests,
if required.
5.4.3 Procedure 5.4.3 Procedure
Prior to beginning a step algorithm, the tester will initialize The virtual clients will initialize the test by establishing TCP
the test by establishing connections defined by initial connections. connections defined by initial connections. The test will use the
The test will use the same algorithm for establishing the connection same algorithm for establishing the TCP connections as described in
as described in the connection capacity test(Section 5.1). the connection capacity test(Section 5.1) with the exception that
no object transfers are required.
For each iteration of the step algorithm, the tester will attempt The virtual clients will then attempt to tear down all of TCP
teardown the number of connections defined by teardown step count connections at a rate defined by tear down attempt rate. The
at a rate defined by teardown attempt rate. This will be repeated tester(Virtual Clients) MUST exclude any connections which do not
until the tester has attempted to teardown all of the connections. properly close in its measurements. For example, connections in
which the DUT/SUT transmits a TCP RST in response to a TCP FIN
packet or connections which do not acknowledge the FIN packet
requesting the connection be closed.
In the case of proxy based DUT/SUTs, the DUT/SUT will itself receive
the final ACK when closing out it's side of the TCP connection. For
validation purposes, the virtual client(s) SHOULD verify that the
DUT/SUT received the final ACK in the connection tear down exchange
for all connections by transmitting a TCP datagram(s) referencing
the previously town down connection(s). A TCP RST should be received
in response to the TCP datagram(s), if the ACK was received by the
DUT/SUT.
5.4.4 Measurements 5.4.4 Measurements
For each iteration of the test, the tester MUST measure the minimum, The tester MUST measure the minimum, average and maximum TCP
average and maximum connection teardown times. As with the connection tear down times. The TCP connection tear down time will
connection establishment time test, the tester SHOULD measure all be considered the interval between the transmission of the first TCP
connection oriented protocols which are being torn down. FIN packet transmitted by the tester requesting a connection tear
down to receipt of the ACK packet on the same tester interface.
The tester SHOULD measure the minimum, maximum and average tear down
times for all other underlying connection oriented protocols. For
purposes of benchmarking firewall performance, the connection tear down
time will be considered the interval between the transmission of the
first bit of the first octet of the packet carrying the tear down
request to the DUT/SUT interface to receipt of the last bit of the
last octet of the last packet of the connection tear down traffic
headed in the opposite direction.
The tester SHOULD measure the aggregate connection tear down time and
the total number of connections torn down for each protocol measured.
5.4.5 Reporting Format 5.4.5 Reporting Format
The test report MUST note the initial connections, initial The test report MUST note the initial connections , tear down attempt
connection rate, teardown attempt rate, teardown step count and rate and tear down step count.
object size.
For each connection oriented protocol the tester measured, the For each connection oriented protocol the tester measured, the
connection teardown time results SHOULD be in tabular form report MUST note the minimum, average and maximum connection tear
with a row for each iteration of the test. There SHOULD be a column down. In addition, it SHOULD include the aggregate connection tear
for the iteration count, minimum connection teardown time, down time and attempted tear downs completed. The report MUST
average connection teardown time, maximum connection teardown identify the layer/protocol for which the measurements were made.
time, attempted teardowns completed, attempted teardown failed.
Failure analysis:
The test report SHOULD indicate the number of connections which failed
the validation step.
5.5 Denial Of Service Handling 5.5 Denial Of Service Handling
5.5.1 Objective 5.5.1 Objective
To determine the effect of a denial of service attack on a DUT/SUTs To determine the effect of a denial of service attack on a DUT/SUTs
connection establishment rates and/or goodput. The Denial Of Service connection establishment and/or forwarding rate. The Denial Of
Handling test MUST be run after obtaining baseline measurements Service Handling test MUST be run after obtaining baseline
from sections 5.2 and/or 5.6. Measurements from sections 5.2 and/or 5.6.
The TCP SYN flood attack exploits TCP's three-way handshake mechanism The TCP SYN flood attack exploits TCP's three-way handshake mechanism
by having an attacking source host generate TCP SYN packets with by having an attacking source host generate TCP SYN packets with
random source addresses towards a victim host, thereby consuming that random source addresses towards a victim host, thereby consuming that
host's resources. host's resources.
Some firewalls employ mechanisms to guard against SYN attacks. If such Some firewalls employ mechanisms to guard against SYN attacks. If such
mechanisms exist on the DUT/SUT, tests SHOULD be run with these mechanisms exist on the DUT/SUT, tests SHOULD be run with these
mechanisms enabled to determine how well the DUT/SUT can maintain, mechanisms enabled to determine how well the DUT/SUT can maintain,
under such attacks, the baseline connection rates and goodput determined under such attacks, the baseline connection rates and HTTP forwarding
in section 5.2 and section 5.6, respectively. rates determined in section 5.2 and section 5.6, respectively.
5.5.2 Setup Parameters 5.5.2 Setup Parameters
Use the same setup parameters as defined in section 5.2.2 or 5.6.2, Use the same setup parameters as defined in section 5.2.2 or 5.6.2,
depending on whether testing against the baseline connection setup depending on whether testing against the baseline connection setup
rate test or goodput test, respectfully. rate test or HTTP test, respectfully.
In addition, the following setup parameters MUST be defined. In addition, the following setup parameters MUST be defined.
SYN Attack Rate - Defines the rate, in packets per second at which SYN Attack Rate - Defines the rate, in packets per second at which
the server(s) are targeted with TCP SYN packets. the server(s) are targeted with TCP SYN packets.
5.5.3 Procedure 5.5.3 Procedure
Use the same procedure as defined in section 5.2.3 or 5.6.3, depending Use the same procedure as defined in section 5.2.3 or 5.6.3,
on whether testing against the baseline connection setup rate test or Depending on whether testing against the baseline connection setup
goodput test, respectfully. In addition, the tester will generate TCP rate test or HTTP test, respectfully. In addition, the tester will
SYN packets targeting the server(s) IP address or NAT proxy address at generate TCP SYN packets targeting the server(s) IP address or NAT
a rate defined by SYN attack rate. proxy address at a rate defined by SYN attack rate.
The tester originating the TCP SYN attack MUST be attached to the The tester originating the TCP SYN attack MUST be attached to the
unprotected network. In addition, the tester MUST not respond to the unprotected network. In addition, the tester MUST not respond to the
SYN/ACK packets sent by target server in response to the SYN packet. SYN/ACK packets sent by target server in response to the SYN packet.
5.5.4 Measurements 5.5.4 Measurements
Perform the same measurements as defined in section 5.2.4 or 5.6.4, Perform the same measurements as defined in section 5.2.4 or 5.6.4,
depending on whether testing against the baseline connection setup depending on whether testing against the baseline connection setup
rate test or goodput test, respectfully. rate test or HTTP test, respectfully.
In addition, the tester SHOULD track SYN packets associated with the In addition, the tester SHOULD track SYN packets associated with the
SYN attack which the DUT/SUT forwards on the protected or DMZ SYN attack which the DUT/SUT forwards on the protected or DMZ
interface(s). interface(s).
5.5.5 Reporting Format 5.5.5 Reporting Format
The test SHOULD use the same reporting format as described in The test SHOULD use the same reporting format as described in
section 5.2.5 or 5.6.5, depending on whether testing against section 5.2.5 or 5.6.5, depending on whether testing against
baseline throughput rates or goodput, respectively. baseline throughput rates or HTTP test, respectively.
In addition, the report MUST indicate a denial of service handling In addition, the report MUST indicate a denial of service handling
test, SYN attack rate, number SYN attack packets transmitted and test, SYN attack rate, number SYN attack packets transmitted and
number of SYN attack packets received and whether or not the DUT number of SYN attack packets received. The report SHOULD indicate
has any SYN attack mechanisms enabled. whether or not the DUT has any SYN attack mechanisms enabled.
5.6 HTTP 5.6 HTTP
5.6.1 Objective 5.6.1 Objective
To determine the goodput, as defined by RFC2647, of the DUT/SUT To determine the bit forwarding rate, as defined by RFC2647, of the
when presented with HTTP traffic flows. The goodput measurement DUT/SUT when presented with HTTP traffic flows.
will be based on HTTP objects forwarded to the correct destination
interface of the DUT/SUT.
5.6.2 Setup Parameters 5.6.2 Setup Parameters
The following parameters MUST be defined. Connection type - The tester MUST use HTTP 1.1 for HTTP measurements.
Number of sessions - Defines the number of HTTP 1.1 sessions to be Number of GET Requests - Defines the number of HTTP 1.1 GET
attempted for transferring an HTTP object(s). Number MUST be equal requests attempted per connection.
or greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual clients
participating in the test. Note that each session will use one
underlying transport layer connection.
Session rate - Defines the rate, in sessions per second, that the
HTTP sessions are attempted.
Requests per session - Defines the number of HTTP GET requests per GET Request Rate - Defines the rate, in GET requests per second, at
session. which HTTP GET requests are attempted on any given connection.
Object Size - Defines the number of bytes to be transferred in Object Size - Defines the number of bytes to be transferred in
response to an HTTP GET request. response to an HTTP GET request.
5.6.3 HTTP Procedure 5.6.3 HTTP Procedure
Each HTTP 1.1 virtual client will attempt to establish sessions Each HTTP 1.1 virtual client will attempt to establish each
to its HTTP 1.1 target server(s), using either the target server's connection to its HTTP 1.1 target server(s), using either the target
IP address or NAT proxy address, at a fixed rate in a round robin server's IP address or NAT proxy address, in a round robin fashion.
fashion. The tester will initiate GET requests for each connection at a
constant rate defined by GET request rate, regardless of the state
of the DUT/SUT.
Baseline measurements SHOULD be performed using a single GET request Baseline measurements SHOULD be performed using a single GET request
per HTTP session with the minimal object size supported by the media. per connection with a 1-byte object size. If the tester makes multiple
If the tester makes multiple HTTP GET requests per session, it MUST HTTP GET requests per connection, it MUST request the same object size
request the same-sized object each time. Testers may run multiple for each GET request. Testers SHOULD run multiple iterations of this
iterations of this test with objects of different sizes. See test using other object sizes and/or multiple requests per connection.
appendix A when testing proxy based DUT/SUT regarding HTTP version See appendix A when testing proxy based DUT/SUT regarding HTTP version
considerations. 5.6.4 Measurement considerations.
Aggregate Goodput - The aggregate bit forwarding rate of the 5.6.4 Measurements
requested HTTP objects. The measurement will start on receipt of the
first bit of the first packet containing a requested object which Version information:
has been successfully transferred and end on receipt of the last
packet containing the last requested object that has been The test report MUST note the use of an HTTP 1.1 client and server.
successfully transferred. The goodput, in bits per second, can be
Application Layer
Bit Forwarding Rate - The bit forwarding rate of the DUT/SUT MUST,
at a minimum, be measured at the application layer and shall be
referenced to the requested object(s). The measurement will
start on transmission of the first bit of the first requested object
and end on transmission of the last bit of the last requested object.
The aggregate bit forwarding rate, in bits per second, will be
calculated using the following formula: calculated using the following formula:
OBJECTS * OBJECTSIZE * 8 OBJECTS * OBJECTSIZE * 8
Goodput = -------------------------- FORWARDING RATE(bit/s) = --------------------------
DURATION DURATION
OBJECTS - Objects successfully transferred OBJECTS - Objects successfully transferred
OBJECTSIZE - Object size in bytes OBJECTSIZE - Object size in bytes
DURATION - Aggregate transfer time based on aforementioned time DURATION - Aggregate transfer time based on aforementioned time
references. references.
5.6.5 Reporting Format Transport-Layer and Below
The test report MUST note the object size(s), number of sessions, Bit forwarding rate for layers at or below the transport layer SHOULD
session rate and requests per session. also be performed. Bit forwarding rate for these underlying
layers/protocols MAY be measured in either bits per seconds or UOTs
per second. In both cases, the measurement will start on transmission
of the first packet containing the first HTTP GET request and end on
transmission of the last packet containing the last octet of the last
requested object. The aggregate bit forwarding rate, in bits per second,
will be calculated using the following formula:
The goodput results SHOULD be reported in tabular form with a row TX - RETX
for each of the object sizes. There SHOULD be columns for the object FORWARDING RATE(bit/s or UOT/s) = -----------
size, measured goodput and number of successfully transferred DURATION
objects. TX - If measuring in units of bits per seconds, TOTAL is the total
bits transmitted including header and optional data for a
given protocol. If measuring in UOTs per seconds, total is the
total number of UOTs transmitted. This excludes any bits or UOT
that are associated with connection maintenance[1], such as TCP
keep-alives.
Failure analysis: RETX - If measuring in units of bits per seconds, RETX is the total
number of bits, including header and optional data for a given
protocol, that was retransmitted. If measuring in units of UOTs
per second, RETX is the number of UOTs retransmitted. This
excludes any bits or UOTs that are associated with connection
maintenance, such as TCP keep-alives.
The test report SHOULD indicate the number and percentage of HTTP DURATION - Test duration based on aforementioned time references.
sessions that failed to complete the requested number of
transactions, with a transaction being the GET request and
successfully returned object.
Version information: 5.6.5 Reporting Format
The test report MUST note the use of an HTTP 1.1 client and server. The test report MUST note the number of GET requests, GET request
rate and object size.
Application layer bit forwarding rate results SHOULD be reported in
tabular form with a row for each of the object sizes. There SHOULD
be columns for the object size, the number of completed requests,
the number of completed responses, and the bit forwarding rate
results for each test.
When reporting bit forwarding measurements for layers below the
application layer, such as TCP or IP, the report MUST note whether
the measurements are in bit per second or UOTs per second and the
object size transferred. The report SHOULD be in tabular form with
a row for each layer/protocol. There should be columns for
transmitted bits/UOTs, retransmitted bits/UOTs and the measured
forwarding rate.
Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP GET
request or responses that failed to complete.
5.7 IP Fragmentation 5.7 IP Fragmentation
5.7.1 Objective 5.7.1 Objective
To determine the performance impact when the DUT/SUT is presented To determine the performance impact when the DUT/SUT is presented
with IP fragmented[5] traffic. IP datagrams which have been with IP fragmented[5] traffic. IP datagrams which have been
fragmented, due to crossing a network that supports a smaller fragmented, due to crossing a network that supports a smaller
MTU(Maximum Transmission Unit) than the actual datagram, may MTU(Maximum Transmission Unit) than the actual datagram, may
require the firewall to perform re-assembly prior to the datagram require the firewall to perform re-assembly prior to the datagram
skipping to change at page 18, line 16 skipping to change at page 18, line 47
5.8.1 Objective 5.8.1 Objective
To determine the behavior of the DUT/SUT when presented with a To determine the behavior of the DUT/SUT when presented with a
combination of both legal and Illegal traffic flows. Note that combination of both legal and Illegal traffic flows. Note that
Illegal traffic does not refer to an attack, but to traffic which Illegal traffic does not refer to an attack, but to traffic which
has been explicitly defined by a rule(s) to drop. has been explicitly defined by a rule(s) to drop.
5.8.2 Setup Parameters 5.8.2 Setup Parameters
The following parameters MUST be defined. Connection type - The tester MUST use HTTP 1.1 for HTTP measurements.
Number of sessions - Defines the number of HTTP 1.1 sessions to be Number of GET Requests - Defines the number of HTTP 1.1 GET
attempted for transferring an HTTP object(s). Number MUST be equal requests attempted per connection.
or greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual clients
participating in the test. Note that each session will use one
underlying transport layer connection.
Session rate - Defines the rate, in sessions per second, that the
HTTP sessions are attempted.
Requests per session - Defines the number of HTTP GET requests per GET Request Rate - Defines the rate, in GET requests per second, at
session. which HTTP GET requests are attempted on any given connection.
Object size - Defines the number of bytes to be transferred in Object Size - Defines the number of bytes to be transferred in
response to an HTTP GET request. response to an HTTP GET request.
Illegal traffic percentage - Percentage of HTTP 1.1 sessions which Illegal traffic percentage - Percentage of HTTP connections which
have been explicitly defined in a rule(s) to drop. have been explicitly defined in a rule(s) to drop.
5.8.3 Procedure 5.8.3 Procedure
Each HTTP 1.1 virtual client will attempt to establish sessions Each HTTP 1.1 virtual client will attempt to establish sessions
to its HTTP 1.1 target server(s), using either the target server's to its HTTP 1.1 target server(s), using either the target server's
IP address or NAT proxy address, at a fixed rate in a round robin IP address or NAT proxy address, at a fixed rate in a round robin
fashion. fashion.
The tester MUST present the connection requests, both legal and The tester MUST present the connection requests, both legal and
illegal, in an evenly distributed manner. Many firewalls have illegal, in an evenly distributed manner. Many firewalls have
the capability to filter on different traffic criteria( IP the capability to filter on different traffic criteria( IP
addresses, Port numbers, etc). Testers may run multiple addresses, Port numbers, etc). Testers may run multiple
iterations of this test with the DUT/SUT configured to filter iterations of this test with the DUT/SUT configured to filter
on different traffic criteria. on different traffic criteria.
5.8.4 Measurements 5.8.4 Measurements
Legal sessions allowed - Number and percentage of legal HTTP Tester SHOULD perform the same bit forwarding measurements as defined
sessions which completed. in HTTP test(Section 5.6.4).
Illegal session allowed - Number and percentage of illegal HTTP
session which completed.
5.8.5 Reporting Format 5.8.5 Reporting Format
The test report MUST note the number of sessions, session rate, Tester SHOULD report SHOULD be the same as specified in the HTTP
requests per session, percentage of illegal sessions and measurement test(Section 5.6.5).
results. The results SHOULD be reported in the form of a table with a row
for each of the object sizes. There SHOULD be columns for the In addition, the report MUST note the percentage of illegal HTTP
object size, number of legal sessions attempted, number of legal connections.
sessions successful, number of illegal sessions attempted and number
of illegal sessions successful. Failure analysis
Test report MUST note the number and percentage of illegal connections
allowed by the DUT/SUT.
5.9 Latency 5.9 Latency
5.9.1 Objective 5.9.1 Objective
To determine the latency of network-layer or application-layer data To determine the latency of network-layer or application-layer data
traversing the DUT/SUT. RFC 1242 [3] defines latency. traversing the DUT/SUT. RFC 1242 [3] defines latency.
5.9.2 Setup Parameters 5.9.2 Setup Parameters
skipping to change at page 22, line 39 skipping to change at page 22, line 39
request/response will include a connection-token close in the request/response will include a connection-token close in the
connection header: connection header:
Connection: close Connection: close
If no such connection-token is present, the connection remains If no such connection-token is present, the connection remains
open after the transaction is completed. In addition, proxy open after the transaction is completed. In addition, proxy
based DUT/SUTs may monitor the TCP connection and after a based DUT/SUTs may monitor the TCP connection and after a
timeout, close the connection if no activity is detected. The timeout, close the connection if no activity is detected. The
duration of this timeout is not defined in the HTTP/1.1 duration of this timeout is not defined in the HTTP/1.1
specification and will vary between DUT/SUTs. When performing specification and will vary between DUT/SUTs. If the DUT/SUT
concurrent connection testing, GET requests MAY need to be closes inactive connections, the aging timer on the DUT SHOULD
issued at a periodic rate so that the proxy does not close the be configured for a duration that exceeds the test time.
TCP connection.
While this document cannot foresee future changes to HTTP While this document cannot foresee future changes to HTTP
and it's impact on the methodologies defined herein, such and it impact on the methodologies defined herein, such
changes should be accommodated for so that newer versions of changes should be accommodated for so that newer versions of
HTTP may be used in benchmarking firewall performance. HTTP may be used in benchmarking firewall performance.
Appendix B. References Appendix B. References
[1] D. Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, [1] D. Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647,
August 1999. August 1999.
[2] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter, P. Leach, [2] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter, P. Leach,
T. Berners-Lee , "Hypertext Transfer Protocol -- HTTP/1.1", T. Berners-Lee , "Hypertext Transfer Protocol -- HTTP/1.1",
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/