draft-ietf-bmwg-firewall-06.txt   draft-ietf-bmwg-firewall-07.txt 
Benchmarking Working Group Brooks Hickman Benchmarking Working Group Brooks Hickman
Internet-Draft Spirent Communications Internet-Draft Spirent Communications
Expiration Date: March 2003 David Newman Expiration Date: April 2003 David Newman
Network Test Network Test
Saldju Tadjudin Saldju Tadjudin
Spirent Communications Spirent Communications
Terry Martin Terry Martin
GVNW Consulting Inc GVNW Consulting Inc
September 2002 October 2002
Benchmarking Methodology for Firewall Performance Benchmarking Methodology for Firewall Performance
<draft-ietf-bmwg-firewall-06.txt> <draft-ietf-bmwg-firewall-07.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 56 skipping to change at page 2, line 4
This document is a product of the Benchmarking Methodology Working This document is a product of the Benchmarking Methodology Working
Group (BMWG) of the Internet Engineering Task Force (IETF). Group (BMWG) of the Internet Engineering Task Force (IETF).
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1 Test Considerations . . . . . . . . . . . . . . . . . . 4 4.1 Test Considerations . . . . . . . . . . . . . . . . . . . 4
4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 4 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 4
4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4 4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4
4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 4 4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 5
4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5
4.6 NAT(Network Address Translation) . . . . . . . . . . . . 5 4.6 NAT(Network Address Translation) . . . . . . . . . . . . 5
4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 5 4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 5
4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6 4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6
4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6 4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6
4.10 TCP Stack Considerations . . . . . . . . . . . . . . . . 6 4.10 TCP Stack Considerations . . . . . . . . . . . . . . . . 6
5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6
5.1 IP throughput . . . . . . . . . . . . . . . . . . . . . . 6 5.1 IP throughput . . . . . . . . . . . . . . . . . . . . . . 6
5.2 Concurrent TCP Connection Capacity . . . . . . . . . . . 8 5.2 Concurrent TCP Connection Capacity . . . . . . . . . . . 8
5.3 Maximum TCP Connection Establishment Rate . . . . . . . . 10 5.3 Maximum TCP Connection Establishment Rate . . . . . . . . 10
5.4 Maximum TCP Connection Tear Down Rate . . . . . . . . . . 12 5.4 Maximum TCP Connection Tear Down Rate . . . . . . . . . . 12
5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 14 5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 14
5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 15 5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 15
5.7 Maximum HTTP Transaction Rate . . . . . . . . . . . . . . 18 5.7 Maximum HTTP Transaction Rate . . . . . . . . . . . . . . 18
5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 20 5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 20
5.9 IP Fragmentation Handling . . . . . . . . . . . . . . . . 21 5.9 IP Fragmentation Handling . . . . . . . . . . . . . . . . 21
5.10 Latency . . . . . . . . . . . . . . . . . . . . . . . . 22 5.10 Latency . . . . . . . . . . . . . . . . . . . . . . . . 23
6. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 6. References . . . . . . . . . . . . . . . . . . . . . . . . . 25
7. Security Consideration . . . . . . . . . . . . . . . . . . . 25 7. Security Consideration . . . . . . . . . . . . . . . . . . . 26
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 25 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 26
9. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 26 9. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 27
Appendix A - HyperText Transfer Protocol(HTTP) . . . . . . . . 27 Appendix A - HyperText Transfer Protocol(HTTP) . . . . . . . . 28
Appendix B - Connection Establishment Time Measurements . . . . 27 Appendix B - Connection Establishment Time Measurements . . . . 28
Appendix C - Connection Tear Down Time Measurements . . . . . . 28 Appendix C - Connection Tear Down Time Measurements . . . . . . 29
Full Copy Statement . . . . . . . . . . . . . . . . . . . . . . 29 Full Copy Statement . . . . . . . . . . . . . . . . . . . . . . 30
1. Introduction 1. Introduction
This document provides methodologies for the performance This document provides methodologies for the performance
benchmarking of firewalls. It provides methodologies in four areas: benchmarking of firewalls. It provides methodologies in four areas:
forwarding, connection, latency and filtering. In addition to forwarding, connection, latency and filtering. In addition to
defining the tests, this document also describes specific formats defining the tests, this document also describes specific formats
for reporting the results of the tests. for reporting the results of the tests.
A previous document, "Benchmarking Terminology for Firewall A previous document, "Benchmarking Terminology for Firewall
skipping to change at page 2, line 57 skipping to change at page 3, line 4
In this document, the words that are used to define the significance In this document, the words that are used to define the significance
of each particular requirement are capitalized. These words are: of each particular requirement are capitalized. These words are:
* "MUST" This word, or the words "REQUIRED" and "SHALL" mean that * "MUST" This word, or the words "REQUIRED" and "SHALL" mean that
the item is an absolute requirement of the specification. the item is an absolute requirement of the specification.
* "SHOULD" This word or the adjective "RECOMMENDED" means that * "SHOULD" This word or the adjective "RECOMMENDED" means that
there may exist valid reasons in particular circumstances to there may exist valid reasons in particular circumstances to
ignore this item, but the full implications should be understood ignore this item, but the full implications should be understood
and the case carefully weighed before choosing a different course. and the case carefully weighed before choosing a different
course.
* "MAY" This word or the adjective "OPTIONAL" means that this item * "MAY" This word or the adjective "OPTIONAL" means that this item
is truly optional. One vendor may choose to include the item is truly optional. One vendor may choose to include the item
because a particular marketplace requires it or because it because a particular marketplace requires it or because it
enhances the product, for example; another vendor may omit the enhances the product, for example; another vendor may omit the
same item. same item.
An implementation is not compliant if it fails to satisfy one or more An implementation is not compliant if it fails to satisfy one or more
of the MUST requirements. An implementation that satisfies all the of the MUST requirements. An implementation that satisfies all the
MUST and all the SHOULD requirements is said to be "unconditionally MUST and all the SHOULD requirements is said to be "unconditionally
skipping to change at page 4, line 15 skipping to change at page 4, line 20
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Clients |----| | | |------| Servers/ | | Clients |----| | | |------| Servers/ |
| | | | | | | Clients | | | | | | | | Clients |
+----------+ |-------| DUT/SUT |--------| | | +----------+ |-------| DUT/SUT |--------| | |
| | | | +----------+ | | | | +----------+
| +----------+ | | +----------+ |
Protected | | | Unprotected Protected | | | Unprotected
Network | Network Network | Network
| |
|
----------------- -----------------
| DMZ | DMZ
| |
| |
+-----------+ +-----------+
| | | |
| Servers | | Servers |
| | | |
+-----------+ +-----------+
skipping to change at page 5, line 41 skipping to change at page 5, line 46
#1 1 2 3 1... #1 1 2 3 1...
#2 2 3 1 2... #2 2 3 1 2...
#3 3 1 2 3... #3 3 1 2 3...
#4 1 2 3 1... #4 1 2 3 1...
#5 2 3 1 2... #5 2 3 1 2...
#6 3 1 2 3... #6 3 1 2 3...
4.6 Network Address Translation(NAT) 4.6 Network Address Translation(NAT)
Many firewalls implement network address translation(NAT)[1], a Many firewalls implement network address translation(NAT)[1], a
function which translates internal host IP addresses attached to function which translates private internet addresses to public
the protected network to a virtual IP address for communicating internet addresses. This involves additional processing on the part
across the unprotected network(Internet). This involves additional of the DUT/SUT and may impact performance. Therefore, tests SHOULD
processing on the part of the DUT/SUT and may impact performance. be ran with NAT disabled and NAT enabled to determine the
Therefore, tests SHOULD be ran with NAT disabled and NAT enabled performance differential, if any. The test report MUST indicate
to determine the performance differentials. The test report MUST whether NAT was enabled or disabled.
indicate whether NAT was enabled or disabled.
4.7 Rule Sets 4.7 Rule Sets
Rule sets[1] are a collection of access control policies that Rule sets[1] are a collection of access control policies that
determine which packets the DUT/SUT will forward and which it will determine which packets the DUT/SUT will forward and which it will
reject[1]. Since criteria by which these access control policies may reject[1]. Since criteria by which these access control policies may
be defined will vary depending on the capabilities of the DUT/SUT, be defined will vary depending on the capabilities of the DUT/SUT,
the following is limited to providing guidelines for configuring the following is limited to providing guidelines for configuring
rule sets when benchmarking the performance of the DUT/SUT. rule sets when benchmarking the performance of the DUT/SUT.
It is RECOMMENDED that a rule be entered for each host(Virtual It is RECOMMENDED that a rule be entered for each host(Virtual
client). In addition, testing SHOULD be performed using different client). In addition, testing SHOULD be performed using different
size rule sets to determine its impact on the performance of the size rule sets to determine its impact on the performance of the
DUT/SUT. Rule sets MUST be configured in a manner, such that, rules DUT/SUT. Rule sets MUST be configured in a manner, such that, rules
associated with actual test traffic are configured at the end of the associated with actual test traffic are configured at the end of the
rule set and not the beginning. rule set and not the beginning.
The DUT/SUT SHOULD be configured to deny access to all traffic The DUT/SUT SHOULD be configured to deny access to all traffic which
which was not previously defined in the rule set. The test report was not previously defined in the rule set. The test report SHOULD
SHOULD include the DUT/SUT configured rule set(s). include the DUT/SUT configured rule set(s).
4.8 Web Caching 4.8 Web Caching
Some firewalls include caching agents to reduce network load. When Some firewalls include caching agents to reduce network load. When
making a request through a caching agent, the caching agent attempts making a request through a caching agent, the caching agent attempts
to service the response from its internal memory. The cache itself to service the response from its internal memory. The cache itself
saves responses it receives, such as responses for HTTP GET saves responses it receives, such as responses for HTTP GET
requests. Testing SHOULD be performed with any caching agents on the requests. Testing SHOULD be performed with any caching agents on the
DUT/SUT disabled. DUT/SUT disabled.
skipping to change at page 6, line 50 skipping to change at page 6, line 52
configurable, any such TCP parameter(s) MUST be noted in the test configurable, any such TCP parameter(s) MUST be noted in the test
report. In addition, when comparing multiple DUT/SUTs, the same TCP report. In addition, when comparing multiple DUT/SUTs, the same TCP
parameters MUST be used. parameters MUST be used.
5. Benchmarking Tests 5. Benchmarking Tests
5.1 IP Throughput 5.1 IP Throughput
5.1.1 Objective 5.1.1 Objective
To determine the throughput of network-layer data transversing To determine the throughput of network-layer data transversing the
the DUT/SUT, as defined in RFC1242[1]. Note that while RFC1242 DUT/SUT, as defined in RFC1242[1]. Note that while RFC1242 uses the
uses the term frames, which is associated with the link layer, the term frames, which is associated with the link layer, the procedure
procedure uses the term packets, since it is referencing the uses the term packets, since it is referencing the network layer.
network layer.
5.1.2 Setup Parameters 5.1.2 Setup Parameters
The following parameters MUST be defined: The following parameters MUST be defined:
Packet size - Number of bytes in the IP packet, exclusive of any Packet size - Number of bytes in the IP packet, exclusive of any
link layer header or checksums. link layer header or checksums.
Test Duration - Duration of the test, expressed in seconds. Test Duration - Duration of the test, expressed in seconds.
5.1.3 Procedure 5.1.3 Procedure
The tester will offer client/server traffic to the DUT/SUT, The tester MUST offer unicast IP packets traffic to the DUT/SUT at a
consisting of unicast IP packets. The tester MUST offer the packets constant rate. The test MAY consist of either bi-directional or
at a constant rate. The test MAY consist of either bi-directional or unidirectional traffic; for example, an emulated client may offer a
unidirectional traffic, with the client offering a unicast stream of unicast stream of packets to an emulated server, or the tester may
packets to the server for the latter. simulate a client/server exchange by offering bidirectional traffic.
The test MAY employ an iterative search algorithm. Each iteration The test MAY employ an iterative search algorithm. Each iteration
will involve the tester varying the intended load until the maximum will involve the tester varying the intended load until the maximum
rate, at which no packet loss occurs, is found. Since backpressure rate, at which no packet loss occurs, is found. Since backpressure
mechanisms may be employed, resulting in the intended load and mechanisms may be employed, resulting in the intended load and
offered load being different, the test SHOULD be performed in either offered load being different, the test SHOULD be performed in either
a packet based or time based manner as described in RFC2889[7]. As a packet based or time based manner as described in RFC2889[7]. As
with RFC1242, the term packet is used in place of frame. The with RFC1242, the term packet is used in place of frame. The
duration of the test portion of each trial MUST be at least 30 seconds. duration of the test portion of each trial MUST be at least 30
seconds.
It is RECOMMENDED to perform the throughput measurements with It is RECOMMENDED to perform the throughput measurements with
different packet sizes. When testing with different packet sizes the different packet sizes. When testing with different packet sizes the
DUT/SUT configuration MUST remain the same. DUT/SUT configuration MUST remain the same.
5.1.4 Measurement 5.1.4 Measurement
5.1.4.1 Network Layer 5.1.4.1 Network Layer
Throughput - Maximum offered load, expressed in either bits per Throughput - Maximum offered load, expressed in either bits per
second or packets per second, at which no packet loss is detected. second or packets per second, at which no packet loss is detected.
The bits to be counted are in the IP packet (header plus payload);
other fields, such as link-layer headers and trailers, MUST NOT be
included in the measurement.
Forwarding Rate - Forwarding rate, expressed in either bits per Forwarding Rate - Forwarding rate, expressed in either bits per
second or packets per second, the device is observed to second or packets per second, the device is observed to successfully
successfully forward to the correct destination interface in forward to the correct destination interface in response to a
response to a specified offered load. specified offered load. The bits to be counted are in the IP packet
(header plus payload); other fields, such as link-layer headers and
trailers, MUST NOT be included in the measurement.
5.1.4 Reporting Format 5.1.4 Reporting Format
The test report MUST note the packet size(s), test duration, The test report MUST note the packet size(s), test duration,
throughput and forwarding rate. If the test involved offering throughput and forwarding rate. If the test involved offering
packets which target more than one segment(Protected, Unprotected packets which target more than one segment(Protected, Unprotected
or DMZ), the report MUST identify the results as an aggregate or DMZ), the report MUST identify the results as an aggregate
throughput measurement. throughput measurement.
The throughput results SHOULD be reported in the format of a table The throughput results SHOULD be reported in the format of a table
with a row for each of the tested packet sizes. There SHOULD be with a row for each of the tested packet sizes. There SHOULD be
columns for the packet size, the intended load, the offered load, columns for the packet size, the intended load, the offered load,
resultant throughput and forwarding rate for each test. resultant throughput and forwarding rate for each test.
The intermediate results of the search algorithm MAY be saved in The intermediate results of the search algorithm MAY be saved in log
log file which includes the packet size, test duration and for file which includes the packet size, test duration and for each
each iteration: iteration:
- Step Iteration - Step Iteration
- Pass/Fail Status - Pass/Fail Status
- Total packets offered - Total packets offered
- Total packets forwarded - Total packets forwarded
- Intended load - Intended load
- Offered load(If applicable) - Offered load(If applicable)
- Forwarding rate - Forwarding rate
5.2 Concurrent TCP Connection Capacity 5.2 Concurrent TCP Connection Capacity
5.2.1 Objective 5.2.1 Objective
To determine the maximum number of concurrent TCP connections To determine the maximum number of concurrent TCP connections
supported through or with the DUT/SUT, as defined in RFC2647[1]. supported through or with the DUT/SUT, as defined in RFC2647[1].
This test is indented to find the maximum number of entries This test is intended to find the maximum number of entries the
the DUT/SUT can store in its connection table. DUT/SUT can store in its connection table.
5.2.2 Setup Parameters 5.2.2 Setup Parameters
The following parameters MUST be defined for all tests: The following parameters MUST be defined for all tests:
5.2.2.1 Transport-Layer Setup Parameters 5.2.2.1 Transport-Layer Setup Parameters
Connection Attempt Rate - The aggregate rate, expressed in Connection Attempt Rate - The aggregate rate, expressed in
connections per second, at which TCP connection requests are connections per second, at which TCP connection requests are
attempted. The rate SHOULD be set at or lower than the maximum attempted. The rate SHOULD be set at or lower than the maximum
rate at which the DUT/SUT can accept connection requests. rate at which the DUT/SUT can accept connection requests.
Aging Time - The time, expressed in seconds, the DUT/SUT will keep a Aging Time - The time, expressed in seconds, the DUT/SUT will
connection in its connection table after receiving a TCP FIN or RST keep a connection in its connection table after receiving a TCP
packet. FIN or RST packet.
5.2.2.2 Application-Layer Setup Parameters 5.2.2.2 Application-Layer Setup Parameters
Validation Method - HTTP 1.1 or higher MUST be used for this test. Validation Method - HTTP 1.1 or higher MUST be used for this
test.
Object Size - Defines the number of bytes, excluding any bytes Object Size - Defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an associated with the HTTP header, to be transferred in response to
HTTP 1.1 or higher GET request. an HTTP 1.1 or higher GET request.
5.2.3 Procedure 5.2.3 Procedure
An iterative search algorithm MAY be used to determine the maximum An iterative search algorithm MAY be used to determine the maximum
number of concurrent TCP connections supported through or with the number of concurrent TCP connections supported through or with the
DUT/SUT. DUT/SUT.
For each iteration, the aggregate number of concurrent TCP For each iteration, the aggregate number of concurrent TCP
connections attempted by the virtual client(s) will be varied. The connections attempted by the virtual client(s) will be varied. The
destination address will be that of the server or that of the NAT destination address will be that of the server or that of the NAT
skipping to change at page 9, line 45 skipping to change at page 9, line 56
Maximum concurrent connections - Total number of TCP connections Maximum concurrent connections - Total number of TCP connections
open for the last successful iteration performed in the search open for the last successful iteration performed in the search
algorithm. algorithm.
Minimum connection establishment time - Lowest TCP connection Minimum connection establishment time - Lowest TCP connection
establishment time measured as defined in appendix B. establishment time measured as defined in appendix B.
Maximum connection establishment time - Highest TCP connection Maximum connection establishment time - Highest TCP connection
establishment time measured as defined in appendix B. establishment time measured as defined in appendix B.
Average connection establishment time - The mean of all measurements Average connection establishment time - The mean of all
of connection establishment times. measurements of connection establishment times.
Aggregate connection establishment time - The total of all Aggregate connection establishment time - The total of all
measurements of connection establishment times. measurements of connection establishment times.
5.2.5 Reporting Format 5.2.5 Reporting Format
5.2.5.1 Application-Layer Reporting: 5.2.5.1 Application-Layer Reporting:
The test report MUST note the object size, number of completed The test report MUST note the object size, number of completed
requests and number of completed responses. requests and number of completed responses.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported
in a tabular format with a column for each iteration. There SHOULD in a tabular format with a column for each iteration. There
be rows for the number of requests attempted, number and percentage SHOULD be rows for the number of requests attempted, number and
requests completed, number of responses attempted, number and percentage requests completed, number of responses attempted,
percentage of responses completed. The table MAY be combined with number and percentage of responses completed. The table MAY be
the transport-layer reporting, provided that the table identify this combined with the transport-layer reporting, provided that the
as an application layer measurement. table identify this as an application layer measurement.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.2.5.2 Transport-Layer Reporting: 5.2.5.2 Transport-Layer Reporting:
The test report MUST note the connection attempt rate, aging time, The test report MUST note the connection attempt rate, aging time,
minimum TCP connection establishment time, maximum TCP connection minimum TCP connection establishment time, maximum TCP connection
establishment time, average connection establishment time, aggregate establishment time, average connection establishment time, aggregate
connection establishment time and maximum concurrent connections connection establishment time and maximum concurrent connections
measured. measured.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported in
in the format of a table with a column for each iteration. There the format of a table with a column for each iteration. There SHOULD
SHOULD be rows for the total number of TCP connections attempted, be rows for the total number of TCP connections attempted, number
number and percentage of TCP connections completed, minimum TCP and percentage of TCP connections completed, minimum TCP connection
connection establishment time, maximum TCP connection establishment establishment time, maximum TCP connection establishment time,
time, average connection establishment time and the aggregate average connection establishment time and the aggregate connection
connection establishment time. establishment time.
5.3 Maximum TCP Connection Establishment Rate 5.3 Maximum TCP Connection Establishment Rate
5.3.1 Objective 5.3.1 Objective
To determine the maximum TCP connection establishment rate through To determine the maximum TCP connection establishment rate through
or with the DUT/SUT, as defined by RFC2647[1]. This test is indented or with the DUT/SUT, as defined by RFC2647[1]. This test is intended
to find the maximum rate the DUT/SUT can update its connection to find the maximum rate the DUT/SUT can update its connection
table. table.
5.3.2 Setup Parameters 5.3.2 Setup Parameters
The following parameters MUST be defined for all tests: The following parameters MUST be defined for all tests:
5.3.2.1 Transport-Layer Setup Parameters 5.3.2.1 Transport-Layer Setup Parameters
Number of Connections - Defines the aggregate number of TCP Number of Connections - Defines the aggregate number of TCP
connections that must be established. connections that must be established.
Aging Time - The time, expressed in seconds, the DUT/SUT will keep a Aging Time - The time, expressed in seconds, the DUT/SUT will
connection in it's state table after receiving a TCP FIN or RST keep a connection in it's state table after receiving a TCP FIN
packet. or RST packet.
5.3.2.2 Application-Layer Setup Parameters 5.3.2.2 Application-Layer Setup Parameters
Validation Method - HTTP 1.1 or higher MUST be used for this test. Validation Method - HTTP 1.1 or higher MUST be used for this
test.
Object Size - Defines the number of bytes, excluding any bytes Object Size - Defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an associated with the HTTP header, to be transferred in response to
HTTP 1.1 or higher GET request. an HTTP 1.1 or higher GET request.
5.3.3 Procedure 5.3.3 Procedure
An iterative search algorithm MAY be used to determine the maximum An iterative search algorithm MAY be used to determine the maximum
rate at which the DUT/SUT can accept TCP connection requests. rate at which the DUT/SUT can accept TCP connection requests.
For each iteration, the aggregate rate at which TCP connection For each iteration, the aggregate rate at which TCP connection
requests are attempted by the virtual client(s) will be varied. The requests are attempted by the virtual client(s) will be varied. The
destination address will be that of the server or that of the NAT destination address will be that of the server or that of the NAT
proxy. The aggregate number of connections, defined by number of proxy. The aggregate number of connections, defined by number of
skipping to change at page 11, line 40 skipping to change at page 11, line 51
5.3.4 Measurements 5.3.4 Measurements
5.3.4.1 Application-Layer measurements 5.3.4.1 Application-Layer measurements
Number of objects requested Number of objects requested
Number of objects returned Number of objects returned
5.3.4.2 Transport-Layer measurements 5.3.4.2 Transport-Layer measurements
Highest connection rate - Highest rate, in connections per second, Highest connection rate - Highest rate, in connections per
for which all connections successfully opened in the search second, for which all connections successfully opened in the
algorithm. search algorithm.
Minimum connection establishment time - Lowest TCP connection Minimum connection establishment time - Lowest TCP connection
establishment time measured as defined in appendix B. establishment time measured as defined in appendix B.
Maximum connection establishment time - Highest TCP connection Maximum connection establishment time - Highest TCP connection
establishment time measured as defined in appendix B. establishment time measured as defined in appendix B.
Average connection establishment time - The mean of all measurements Average connection establishment time - The mean of all
of connection establishment times. measurements of connection establishment times.
Aggregate connection establishment time - The total of all Aggregate connection establishment time - The total of all
measurements of connection establishment times. measurements of connection establishment times.
5.3.5 Reporting Format 5.3.5 Reporting Format
5.3.5.1 Application-Layer Reporting: 5.3.5.1 Application-Layer Reporting:
The test report MUST note object size(s), number of completed The test report MUST note object size(s), number of completed
requests and number of completed responses. requests and number of completed responses.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported
in a tabular format with a column for each iteration. There SHOULD in a tabular format with a column for each iteration. There
be rows for the number of requests attempted, number and percentage SHOULD be rows for the number of requests attempted, number and
requests completed, number of responses attempted, number and percentage requests completed, number of responses attempted,
percentage of responses completed. The table MAY be combined with number and percentage of responses completed. The table MAY be
the transport-layer reporting, provided that the table identify this combined with the transport-layer reporting, provided that the
as an application layer measurement. table identify this as an application layer measurement.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.3.5.2 Transport-Layer Reporting: 5.3.5.2 Transport-Layer Reporting:
The test report MUST note the number of connections, aging time, The test report MUST note the number of connections, aging time,
minimum TCP connection establishment time, maximum TCP connection minimum TCP connection establishment time, maximum TCP connection
establishment time, average connection establishment time, aggregate establishment time, average connection establishment time,
connection establishment time and highest connection rate measured. aggregate connection establishment time and highest connection
rate measured.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported
in the format of a table with a column for each iteration. There in the format of a table with a column for each iteration. There
SHOULD be rows for the connection attempt rate, total number of SHOULD be rows for the connection attempt rate, total number of
TCP connections attempted, total number of TCP connections TCP connections attempted, total number of TCP connections
completed, minimum TCP connection establishment time, maximum TCP completed, minimum TCP connection establishment time, maximum TCP
connection establishment time, average connection establishment time connection establishment time, average connection establishment
and the aggregate connection establishment time. time and the aggregate connection establishment time.
5.4 Maximum TCP Connection Tear Down Rate 5.4 Maximum TCP Connection Tear Down Rate
5.4.1 Objective 5.4.1 Objective
To determine the maximum TCP connection tear down rate through or To determine the maximum TCP connection tear down rate through or
with the DUT/SUT, as defined by RFC2647[1]. with the DUT/SUT, as defined by RFC2647[1].
5.4.2 Setup Parameters 5.4.2 Setup Parameters
skipping to change at page 14, line 8 skipping to change at page 14, line 20
of connection tear down times. of connection tear down times.
5.4.5 Reporting Format 5.4.5 Reporting Format
The test report MUST note the number of connections, aging time, The test report MUST note the number of connections, aging time,
close method, close direction, minimum TCP connection tear down close method, close direction, minimum TCP connection tear down
time, maximum TCP connection tear down time, average TCP connection time, maximum TCP connection tear down time, average TCP connection
tear down time and the aggregate TCP connection tear down time and tear down time and the aggregate TCP connection tear down time and
highest connection tear down rate measured. highest connection tear down rate measured.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported in
in the format of a table with a column for each iteration. There the format of a table with a column for each iteration. There SHOULD
SHOULD be rows for the number of TCP tear downs attempted, number be rows for the number of TCP tear downs attempted, number and
and percentage of TCP connection tear downs completed, minimum percentage of TCP connection tear downs completed, minimum TCP
TCP connection tear down time, maximum TCP connection tear down connection tear down time, maximum TCP connection tear down time,
time, average TCP connection tear down time, aggregate TCP average TCP connection tear down time, aggregate TCP connection tear
connection tear down time and validation failures, if required. down time and validation failures, if required.
5.5 Denial Of Service Handling 5.5 Denial Of Service Handling
5.5.1 Objective 5.5.1 Objective
To determine the effect of a denial of service attack on a DUT/SUT To determine the effect of a denial of service attack on a DUT/SUT
TCP connection establishment and/or HTTP transfer rates. The denial TCP connection establishment and/or HTTP transfer rates. The denial
of service handling test MUST be run after obtaining baseline of service handling test MUST be run after obtaining baseline
measurements from sections 5.3 and/or 5.6. measurements from sections 5.3 and/or 5.6.
skipping to change at page 14, line 36 skipping to change at page 14, line 48
mechanism by having an attacking source host generate TCP SYN mechanism by having an attacking source host generate TCP SYN
packets with random source addresses towards a victim host, thereby packets with random source addresses towards a victim host, thereby
consuming that host's resources. consuming that host's resources.
5.5.2 Setup Parameters 5.5.2 Setup Parameters
Use the same setup parameters as defined in section 5.3.2 or 5.6.2, Use the same setup parameters as defined in section 5.3.2 or 5.6.2,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
establishment rate test or HTTP transfer rate test, respectfully. establishment rate test or HTTP transfer rate test, respectfully.
In addition, the following setup parameters MUST be defined. In addition, the following setup parameters MUST be defined:
SYN attack rate - Rate, expressed in packets per second, at which SYN attack rate - Rate, expressed in packets per second, at which
the server(s) or NAT proxy address is targeted with TCP SYN packets. the server(s) or NAT proxy address is targeted with TCP SYN packets.
5.5.3 Procedure 5.5.3 Procedure
Use the same procedure as defined in section 5.3.3 or 5.6.3, Use the same procedure as defined in section 5.3.3 or 5.6.3,
depending on whether testing against the baseline TCP connection depending on whether testing against the baseline TCP connection
establishment rate or HTTP transfer rate test, respectfully. In establishment rate or HTTP transfer rate test, respectfully. In
addition, the tester will generate TCP SYN packets targeting the addition, the tester will generate TCP SYN packets targeting the
skipping to change at page 15, line 22 skipping to change at page 15, line 34
establishment rate test or HTTP transfer rate, respectfully. establishment rate test or HTTP transfer rate, respectfully.
In addition, the tester SHOULD track TCP SYN packets associated with In addition, the tester SHOULD track TCP SYN packets associated with
the SYN attack which the DUT/SUT forwards on the protected or DMZ the SYN attack which the DUT/SUT forwards on the protected or DMZ
interface(s). interface(s).
5.5.5 Reporting Format 5.5.5 Reporting Format
The test SHOULD use the same reporting format as described in The test SHOULD use the same reporting format as described in
section 5.3.5 or 5.6.5, depending on whether testing against the section 5.3.5 or 5.6.5, depending on whether testing against the
baseline TCP connection establishment rate test or HTTP transfer rate, baseline TCP connection establishment rate test or HTTP transfer
respectfully. rate, respectfully.
In addition, the report MUST indicate a denial of service handling In addition, the report MUST indicate a denial of service handling
test, SYN attack rate, number TCP SYN attack packets transmitted test, SYN attack rate, number TCP SYN attack packets transmitted
and the number of TCP SYN attack packets forwarded by the DUT/SUT. and the number of TCP SYN attack packets forwarded by the DUT/SUT.
The report MUST indicate whether or not the DUT has any SYN attack The report MUST indicate whether or not the DUT has any SYN attack
mechanisms enabled. mechanisms enabled.
5.6 HTTP Transfer Rate 5.6 HTTP Transfer Rate
5.6.1 Objective 5.6.1 Objective
To determine the transfer rate of HTTP requested object transversing To determine the transfer rate of HTTP requested object transversing
the DUT/SUT. the DUT/SUT.
5.6.2 Setup Parameters 5.6.2 Setup Parameters
The following parameters MUST be defined for all tests: The following parameters MUST be defined for all tests:
5.6.2.1 Transport-Layer Setup Parameters 5.6.2.1 Transport-Layer Setup Parameters
Number of connections - Defines the aggregate number of connections Number of connections - Defines the aggregate number of
attempted. The number SHOULD be a multiple of the number of virtual connections attempted. The number SHOULD be a multiple of the
clients participating in the test. number of virtual clients participating in the test.
Close Method - Defines method for closing TCP connections. The test Close Method - Defines method for closing TCP connections. The
MUST be performed with either a three-way or four-way handshake. In test MUST be performed with either a three-way or four-way
a four-way handshake, each side sends separate FIN and ACK messages. handshake. In a four-way handshake, each side sends separate FIN
In a three-way handshake, one side sends a combined FIN/ACK message and ACK messages. In a three-way handshake, one side sends a
upon receipt of a FIN. combined FIN/ACK message upon receipt of a FIN.
Close Direction - Defines whether closing of connections are to be Close Direction - Defines whether closing of connections are to
initiated from the client or from the server. be initiated from the client or from the server.
5.6.2.2 Application-Layer Setup Parameters 5.6.2.2 Application-Layer Setup Parameters
Session Type - The virtual clients/servers MUST use HTTP 1.1 or Session Type - The virtual clients/servers MUST use HTTP 1.1 or
higher. higher.
GET requests per connection - Defines the number of HTTP 1.1 or GET requests per connection - Defines the number of HTTP 1.1 or
higher GET requests attempted per connection. higher GET requests attempted per connection.
Object Size - Defines the number of bytes, excluding any bytes Object Size - Defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an associated with the HTTP header, to be transferred in response to
HTTP 1.1 or higher GET request. an HTTP 1.1 or higher GET request.
5.6.3 Procedure 5.6.3 Procedure
Each HTTP 1.1 or higher virtual client will request one or more Each HTTP 1.1 or higher virtual client will request one or more
objects from an HTTP 1.1 or higher server using one or more HTTP objects from an HTTP 1.1 or higher server using one or more HTTP
GET requests over each connection. The aggregate number of GET requests over each connection. The aggregate number of
connections attempted, defined by number of connections, MUST be connections attempted, defined by number of connections, MUST be
evenly divided among all of the participating virtual clients. evenly divided among all of the participating virtual clients.
If the virtual client(s) make multiple HTTP GET requests per If the virtual client(s) make multiple HTTP GET requests per
connection, it MUST request the same object size for each GET connection, it MUST request the same object size for each GET
request. Multiple iterations of this test may be run with objects request. Multiple iterations of this test may be run with objects
of different sizes. of different sizes.
5.6.4 Measurements 5.6.4 Measurements
5.6.4.1 Application-Layer measurements 5.6.4.1 Application-Layer measurements
Average Transfer Rate - The average transfer rate of the DUT/SUT Average Transfer Rate - The average transfer rate of the DUT/SUT
MUST be measured and shall be referenced to the requested object(s). MUST be measured and shall be referenced to the requested
The measurement will start on transmission of the first bit of the object(s). The measurement will start on transmission of the
first requested object and end on transmission of the last bit of first bit of the first requested object and end on transmission
the last requested object. The average transfer rate, in bits per of the last bit of the last requested object. The average
second, will be calculated using the following formula: transfer rate, in bits per second, will be calculated using the
following formula:
OBJECTS * OBJECTSIZE * 8 OBJECTS * OBJECTSIZE * 8
TRANSFER RATE(bit/s) = -------------------------- TRANSFER RATE(bit/s) = --------------------------
DURATION DURATION
OBJECTS - Total number of objects successfully transferred across OBJECTS - Total number of objects successfully transferred across
all connections. all connections.
OBJECTSIZE - Object size in bytes OBJECTSIZE - Object size in bytes
DURATION - Aggregate transfer time based on aforementioned time DURATION - Aggregate transfer time based on aforementioned time
references. references.
5.6.4.2 Measurements at or below the Transport-Layer 5.6.4.2 Measurements at or below the Transport-Layer
The following measurements SHOULD be performed for each connection- The following measurements SHOULD be performed for each
oriented protocol: connection-oriented protocol:
Goodput[1] - Goodput as defined in section 3.17 of RFC2647. Goodput[1] - Goodput as defined in section 3.17 of RFC2647.
Measurements MUST only reference the protocol payload, excluding Measurements MUST only reference the protocol payload, excluding
any of the protocol header. In addition, the tester MUST exclude any of the protocol header. In addition, the tester MUST exclude
any bits associated with the connection establishment, connection any bits associated with the connection establishment, connection
tear down, security associations[1] or connection maintenance[1]. tear down, security associations[1] or connection maintenance[1].
Since connection-oriented protocols require that data be Since connection-oriented protocols require that data be
acknowledged, the offered load[6] will be varying. Therefore, the acknowledged, the offered load[6] will be varying. Therefore, the
tester should measure the average forwarding rate over the tester should measure the average forwarding rate over the
duration of the test. Measurement should start on transmission of duration of the test. Measurement should start on transmission of
the first bit of the payload of the first datagram and end on the first bit of the payload of the first datagram and end on
transmission of the last bit of the payload of the last datagram. transmission of the last bit of the payload of the last datagram.
skipping to change at page 17, line 26 skipping to change at page 17, line 37
Number of bytes transferred - Total payload bytes transferred. Number of bytes transferred - Total payload bytes transferred.
Number of Timeouts - Total number of timeout events. Number of Timeouts - Total number of timeout events.
Retransmitted bytes - Total number of retransmitted bytes. Retransmitted bytes - Total number of retransmitted bytes.
5.6.5 Reporting Format 5.6.5 Reporting Format
5.6.5.1 Application-Layer reporting 5.6.5.1 Application-Layer reporting
The test report MUST note number of GET requests per connection and The test report MUST note number of GET requests per connection
object size(s). and object size(s).
The transfer rate results SHOULD be reported in tabular form with a The transfer rate results SHOULD be reported in tabular form with
column for each of the object sizes tested. There SHOULD be a row a column for each of the object sizes tested. There SHOULD be a
for the object size, number and percentage of completed requests, row for the object size, number and percentage of completed
number and percentage of completed responses, and the resultant requests, number and percentage of completed responses, and the
transfer rate for each iteration of the test. resultant transfer rate for each iteration of the test.
Failure analysis: Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP The test report SHOULD indicate the number and percentage of HTTP
GET request and responses that failed to complete. GET request and responses that failed to complete.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.6.5.2 Transport-Layer and below reporting 5.6.5.2 Transport-Layer and below reporting
The test report MUST note the number of connections, close method, The test report MUST note the number of connections, close
close direction and the protocol for which the measurement was method, close direction and the protocol for which the
made. measurement was made.
The results SHOULD be reported in tabular form for each of the HTTP The results SHOULD be reported in tabular form for each of the
object sizes tested. There SHOULD be a row for the HTTP object size, HTTP object sizes tested. There SHOULD be a row for the HTTP
resultant goodput, total timeouts, total retransmitted bytes and object size, resultant goodput, total timeouts, total
total bytes transferred. Note that total bytes refers to total retransmitted bytes and total bytes transferred. Note that total
datagram payload bytes transferred. The table MAY be combined with bytes refers to total datagram payload bytes transferred. The
the application layer reporting, provided the table clearly identify table MAY be combined with the application layer reporting,
the protocol for which the measurement was made. provided the table clearly identify the protocol for which the
measurement was made.
Failure analysis: Failure analysis:
The test report SHOULD indicate the number and percentage of The test report SHOULD indicate the number and percentage of
connection establishment failures as well as number and percentage connection establishment failures as well as number and
of TCP tear down failures. percentage of TCP tear down failures.
It is RECOMMENDED that the report include a graph to plot the It is RECOMMENDED that the report include a graph to plot the
distribution of both connection establishment failures and distribution of both connection establishment failures and
connection tear down failures. The x coordinate SHOULD be the connection tear down failures. The x coordinate SHOULD be the
elapsed test time, the y coordinate SHOULD be the number of failures elapsed test time, the y coordinate SHOULD be the number of
for a given sampling period. There SHOULD be two lines on the graph, failures for a given sampling period. There SHOULD be two lines
one for connection failures and one for tear down failures. The on the graph, one for connection failures and one for tear down
graph MUST note the sampling period. failures. The graph MUST note the sampling period.
5.7 Maximum HTTP Transaction Rate 5.7 Maximum HTTP Transaction Rate
5.7.1 Objective 5.7.1 Objective
Determine the maximum transaction rate the DUT/SUT can sustain. Determine the maximum transaction rate the DUT/SUT can sustain. This
This test is intended to find the maximum rate at which users can test is intended to find the maximum rate at which users can access
access objects. objects.
5.7.2 Setup Parameters 5.7.2 Setup Parameters
5.7.2.1 Transport-Layer Setup Parameters 5.7.2.1 Transport-Layer Setup Parameters
Close Method - Defines method for closing TCP connections. The test Close Method - Defines method for closing TCP connections. The
MUST be performed with either a three-way or four-way handshake. In test MUST be performed with either a three-way or four-way
a four-way handshake, each side sends separate FIN and ACK messages. handshake. In a four-way handshake, each side sends separate FIN
In a three-way handshake, one side sends a combined FIN/ACK message and ACK messages. In a three-way handshake, one side sends a
upon receipt of a FIN. combined FIN/ACK message upon receipt of a FIN.
Close Direction - Defines whether closing of connections are to be Close Direction - Defines whether closing of connections are to
initiated from the client or from the server. be initiated from the client or from the server.
5.7.2.2 Application-Layer Setup Parameters 5.7.2.2 Application-Layer Setup Parameters
Session Type - HTTP 1.1 or higher MUST be used for this test. Session Type - HTTP 1.1 or higher MUST be used for this test.
Test Duration - Time, expressed in seconds, for which the Test Duration - Time, expressed in seconds, for which the
virtual client(s) will sustain the attempted GET request rate. virtual client(s) will sustain the attempted GET request rate.
It is RECOMMENDED that the duration be at least 30 seconds. It is RECOMMENDED that the duration be at least 30 seconds.
Requests per connection - Number of object requests per connection. Requests per connection - Number of object requests per
connection.
Object Size - Defines the number of bytes, excluding any bytes Object Size - Defines the number of bytes, excluding any bytes
associated with the HTTP header, to be transferred in response to an associated with the HTTP header, to be transferred in response to
HTTP 1.1 or higher GET request. an HTTP 1.1 or higher GET request.
5.7.3 Procedure 5.7.3 Procedure
An iterative search algorithm MAY be used to determine the maximum An iterative search algorithm MAY be used to determine the maximum
transaction rate that the DUT/SUT can sustain. transaction rate that the DUT/SUT can sustain.
For each iteration, HTTP 1.1 or higher virtual client(s) will For each iteration, HTTP 1.1 or higher virtual client(s) will vary
vary the aggregate GET request rate offered to HTTP 1.1 or higher the aggregate GET request rate offered to HTTP 1.1 or higher
server(s). The virtual client(s) will maintain the offered request server(s). The virtual client(s) will maintain the offered request
rate for the defined test duration. rate for the defined test duration.
If the virtual client(s) make multiple HTTP GET requests per If the virtual client(s) make multiple HTTP GET requests per
connection, it MUST request the same object size for each GET connection, it MUST request the same object size for each GET
request. Multiple tests MAY be performed with different object request. Multiple tests MAY be performed with different object
sizes. sizes.
5.7.4 Measurements 5.7.4 Measurements
skipping to change at page 19, line 30 skipping to change at page 19, line 42
Transaction Time - The tester SHOULD measure minimum, maximum and Transaction Time - The tester SHOULD measure minimum, maximum and
average transaction times. The transaction time will start when the average transaction times. The transaction time will start when the
virtual client issues the GET request and end when the requesting virtual client issues the GET request and end when the requesting
virtual client receives the last bit of the requested object. virtual client receives the last bit of the requested object.
5.7.5 Reporting Format 5.7.5 Reporting Format
5.7.5.1 Application-Layer reporting 5.7.5.1 Application-Layer reporting
The test report MUST note the test duration, object size, requests The test report MUST note the test duration, object size,
per connection and the measured minimum, maximum and average requests per connection and the measured minimum, maximum and
transaction rate. average transaction rate.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported
in a table format with a column for each iteration. There SHOULD be in a table format with a column for each iteration. There SHOULD
rows for the GET request attempt rate, number of requests attempted, be rows for the GET request attempt rate, number of requests
number and percentage of requests completed, number of responses attempted, number and percentage of requests completed, number of
attempted, number and percentage of responses completed, minimum responses attempted, number and percentage of responses
transaction time, average transaction time and maximum transaction completed, minimum transaction time, average transaction time and
time. maximum transaction time.
Version information: Version information:
The test report MUST note the version of HTTP client(s) and The test report MUST note the version of HTTP client(s) and
server(s). server(s).
5.7.5.2 Transport-Layer 5.7.5.2 Transport-Layer
The test report MUST note the close method, close direction, number The test report MUST note the close method, close direction,
of connections established and number of connections torn down. number of connections established and number of connections torn
down.
The intermediate results of the search algorithm MAY be reported The intermediate results of the search algorithm MAY be reported
in a table format with a column for each iteration. There SHOULD be in a table format with a column for each iteration. There SHOULD
rows for the number of connections attempted, number and percentage be rows for the number of connections attempted, number and
of connections completed, number and percentage of connection tear percentage of connections completed, number and percentage of
downs completed. The table MAY be combined with the application connection tear downs completed. The table MAY be combined with
layer reporting, provided the table identify this as transport layer the application layer reporting, provided the table identify this
measurement. as transport layer measurement.
5.8 Illegal Traffic Handling 5.8 Illegal Traffic Handling
5.8.1 Objective 5.8.1 Objective
To character the behavior of the DUT/SUT when presented with a To character the behavior of the DUT/SUT when presented with a
combination of both legal and Illegal[1] traffic. Note that Illegal combination of both legal and Illegal[1] traffic. Note that Illegal
traffic does not refer to an attack, but traffic which has been traffic does not refer to an attack, but traffic which has been
explicitly defined by a rule(s) to drop. explicitly defined by a rule(s) to drop.
skipping to change at page 22, line 42 skipping to change at page 23, line 7
5.10.1 Non-Fragmented Traffic 5.10.1 Non-Fragmented Traffic
The test report SHOULD be the same as described in section 5.6.5. The test report SHOULD be the same as described in section 5.6.5.
Note that any forwarding rate measurements for the HTTP traffic Note that any forwarding rate measurements for the HTTP traffic
excludes any bits associated with the fragmented traffic which excludes any bits associated with the fragmented traffic which
may be forward by the DUT/SUT. may be forward by the DUT/SUT.
5.9.2 Fragmented Traffic 5.9.2 Fragmented Traffic
The test report MUST note the packet size, MTU size, intended load, The test report MUST note the packet size, MTU size, intended
number of UDP/IP packets transmitted and number of UDP/IP packets load, number of UDP/IP packets transmitted and number of UDP/IP
forwarded. The test report SHOULD also note whether or not the packets forwarded. The test report SHOULD also note whether or
DUT/SUT forwarded the offered UDP/IP traffic fragmented. not the DUT/SUT forwarded the offered UDP/IP traffic fragmented.
5.10 Latency 5.10 Latency
5.10.1 Objective 5.10.1 Objective
To determine the latency of network-layer or application-layer data To determine the latency of network-layer or application-layer data
traversing the DUT/SUT. RFC 1242 [3] defines latency. traversing the DUT/SUT. RFC 1242 [3] defines latency.
5.10.2 Setup Parameters 5.10.2 Setup Parameters
skipping to change at page 23, line 39 skipping to change at page 23, line 57
Number of objects transferred. Number of objects transferred.
Test duration, expressed in seconds. Test duration, expressed in seconds.
Test instruments MUST generate packets with unique timestamp Test instruments MUST generate packets with unique timestamp
signatures. signatures.
5.10.3 Network-layer procedure 5.10.3 Network-layer procedure
A client will offer a unidirectional stream of unicast packets to a A client will offer a unidirectional stream of unicast packets to
server. The packets MUST use a connectionless protocol like IP or a server. The packets MUST use a connectionless protocol like IP
UDP/IP. or UDP/IP.
The tester MUST offer packets in a steady state. As noted in the The tester MUST offer packets in a steady state. As noted in the
latency discussion in RFC 2544 [4], latency measurements MUST be latency discussion in RFC 2544 [4], latency measurements MUST be
taken at the throughput level -- that is, at the highest offered taken at the throughput level -- that is, at the highest offered
load with zero packet loss. Measurements taken at the throughput load with zero packet loss. Measurements taken at the throughput
level are the only ones that can legitimately be termed latency. level are the only ones that can legitimately be termed latency.
It is RECOMMENDED that implementers use offered loads not only at It is RECOMMENDED that implementers use offered loads not only at
the throughput level, but also at load levels that are less than the throughput level, but also at load levels that are less than
or greater than the throughput level. To avoid confusion with or greater than the throughput level. To avoid confusion with
existing terminology, measurements from such tests MUST be labeled existing terminology, measurements from such tests MUST be labeled
as delay rather than latency. as delay rather than latency.
It is RECOMMENDED to perform the latency measurements with different It is RECOMMENDED to perform the latency measurements with
packet sizes. When testing with different packet sizes the DUT/SUT different packet sizes. When testing with different packet sizes
configuration MUST remain the same. the DUT/SUT configuration MUST remain the same.
If desired, the tester MAY use a step test in which offered loads If desired, the tester MAY use a step test in which offered loads
increment or decrement through a range of load levels. increment or decrement through a range of load levels.
The duration of the test portion of each trial MUST be at least 30 The duration of the test portion of each trial MUST be at least
seconds. 30 seconds.
5.10.4 Application layer procedure 5.10.4 Application layer procedure
An HTTP 1.1 or higher client will request one or more objects from An HTTP 1.1 or higher client will request one or more objects from
an HTTP or higher 1.1 server using one or more HTTP GET requests. If an HTTP 1.1 or higher server using one or more HTTP GET requests. If
the tester makes multiple HTTP GET requests, it MUST request the the tester makes multiple HTTP GET requests, it MUST request the
same-sized object each time. Testers may run multiple iterations of same-sized object each time. Testers may run multiple iterations of
this test with objects of different sizes. this test with objects of different sizes.
Implementers MAY configure the tester to run for a fixed duration. Implementers MAY configure the tester to run for a fixed duration.
In this case, the tester MUST report the number of objects requested In this case, the tester MUST report the number of objects requested
and returned for the duration of the test. For fixed-duration tests and returned for the duration of the test. For fixed-duration tests
it is RECOMMENDED that the duration be at least 30 seconds. it is RECOMMENDED that the duration be at least 30 seconds.
5.10.5 Measurements 5.10.5 Measurements
skipping to change at page 26, line 32 skipping to change at page 27, line 32
Phone: + 1 818 889-0011 Phone: + 1 818 889-0011
Email: dnewman@networktest.com Email: dnewman@networktest.com
Saldju Tadjudin Saldju Tadjudin
Spirent Communications Spirent Communications
26750 Agoura Road 26750 Agoura Road
Calabasas, CA 91302 Calabasas, CA 91302
USA USA
Phone: + 1 818 676 2468 Phone: + 1 818 676 2468
Email: saldju.Tadjudin@spirentcom.com Email: Saldju.Tadjudin@spirentcom.com
Terry Martin Terry Martin
GVNW Consulting Inc. GVNW Consulting Inc.
8050 SW Warm Springs Road 8050 SW Warm Springs Road
Tualatin Or. 97062 Tualatin Or. 97062
USA USA
Phone: + 1 503 612 4422 Phone: + 1 503 612 4422
Email: tmartin@gvnw.com Email: tmartin@gvnw.com
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/