draft-ietf-bmwg-firewall-00.txt   draft-ietf-bmwg-firewall-01.txt 
Network Working Group Terry Martin Network Working Group Terry Martin
Internet-Draft M2networx INC Internet-Draft M2networx INC
Expiration Date: B. Hickman Expiration Date: B. Hickman
Netcom Systems Netcom Systems
June 2000 November 2000
Benchmarking Methodology for Firewalls Benchmarking Methodology for Firewalls
<draft-ietf-bmwg-firewall-00.txt> <draft-ietf-bmwg-firewall-01.txt>
Status of this Memo Status of this Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at page 1, line 39 skipping to change at page 1, line 39
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/shadow.html.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.1 Test Considerations . . . . . . . . . . . . . . . . . . . . . 4 4.1 Test Considerations . . . . . . . . . . . . . . . . . . . . . 4
4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . . . 4 4.1.1 Virtual Client/Servers . . . . . . . . . . . . . . . . . . 4
4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . . . 4 4.1.2 Test Traffic Requirements . . . . . . . . . . . . . . . . 4
4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . . . 5 4.1.3 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 5
4.5 Multiple Client/Server Testing . . . . . . . . . . . . . . . 5 4.1.4 Multiple Client/Server Testing . . . . . . . . . . . . . . 5
4.6 NAT(Network Address Translation) . . . . . . . . . . . . . . 6 4.1.5 NAT(Network Address Translation) . . . . . . . . . . . . . 6
4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1.6 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 6
4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1.7 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6
4.1.8 Authentication . . . . . . . . . . . . . . . . . . . . . . 6
5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . . . 7 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . . . 7
5.1 Concurrent Connection Capacity . . . . . . . . . . . . . . . 7 5.1 Concurrent Connection Capacity . . . . . . . . . . . . . . . . 7
5.2 Maximum Connection Rate . . . . . . . . . . . . . . . . . . . 9 5.2 Maximum Connection Rate . . . . . . . . . . . . . . . . . . . 8
5.3 Denial Of Service Handling . . . . . . . . . . . . . . . . . 10 5.3 Connection Establishment Time . . . . . . . . . . . . . . . . 10
5.3 Single Application Goodput . . . . . . . . . . . . . . . . . 12 5.4 Denial Of Service Handling . . . . . . . . . . . . . . . . . . 11
5.4 FTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.5 Single Application Goodput . . . . . . . . . . . . . . . . . . 12
5.5 SMTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . 13 5.5.1 FTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 12
5.6 HTTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . 15 5.5.2 SMTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 14
5.7 Mixed Application Goodput . . . . . . . . . . . . . . . . . . 17 5.5.3 HTTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 15
5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . . . 18 5.6 Concurrent Application Goodput . . . . . . . . . . . . . . . . 17
5.9 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.7 Illegal Traffic Handling . . . . . . . . . . . . . . . . . . . 19
5.8 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
A. File Transfer Protocol(FTP) . . . . . . . . . . . . . . . . . . . 19 A. File Transfer Protocol(FTP) . . . . . . . . . . . . . . . . . . . 19
A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 19
A.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 19 A.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 20
A.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 20 A.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 20
B. Simple Mail Transfer Protocol(SMTP) . . . . . . . . . . . . . . . 20 B. Simple Mail Transfer Protocol(SMTP) . . . . . . . . . . . . . . . 21
B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 20 B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 21
B.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 20 B.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 21
B.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 21 B.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 22
C. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . . . . . 21 C. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . . . . . 22
C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 21 C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 22
C.2 Version Considerations . . . . . . . . . . . . . . . . . . . . 21 C.2 Version Considerations . . . . . . . . . . . . . . . . . . . . 23
C.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 21 C.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 23
D. GoodPut Measurements . . . . . . . . . . . . . . . . . . . . . . . 22 E. TCP establishment/teardown . . . . . . . . . . . . . . . . . . . . 23
E. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 D. GoodPut Measurements . . . . . . . . . . . . . . . . . . . . . . . 23
F. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1. Introduction 1. Introduction
This document is intended to provide methodology for the benchmarking This document is intended to provide methodology for the benchmarking
of firewalls. It provides methodologies for benchmarking forwarding of firewalls. It provides methodologies for benchmarking forwarding
performance, connection performance, latency and filtering. In performance, connection performance, latency and filtering. In
addition to defining the tests, this document also describes specific addition to defining the tests, this document also describes specific
formats for reporting the results of the tests. formats for reporting the results of the tests.
A previous document, "Benchmarking Terminology for Firewall A previous document, "Benchmarking Terminology for Firewall
skipping to change at page 3, line 38 skipping to change at page 3, line 39
| | | +----------+ | | | | | | +----------+ | | |
| Servers/ |----| | | |------| Servers/ | | Servers/ |----| | | |------| Servers/ |
| Clients | | | | | | Clients | | Clients | | | | | | Clients |
| | |-------| DUT/SUT |--------| | | | | |-------| DUT/SUT |--------| | |
+----------+ | | | | +----------+ +----------+ | | | | +----------+
| +----------+ | | +----------+ |
Protected | | Unprotected Protected | | Unprotected
Network Network Network Network
Figure 1(Dual-Homed) Figure 1(Dual-Homed)
Tri-homed configurations employ a third segment called a DMZ. With Tri-homed[1] configurations employ a third segment called a DMZ. With
tri-homed configurations, servers accessible to the public network tri-homed configurations, servers accessible to the public network
are attached to the DMZ. Tri-Homed configurations offer additional are attached to the DMZ. Tri-Homed configurations offer additional
security by separating server accessible to the public network from security by separating server accessible to the public network from
internal hosts. internal hosts.
+----------+ +----------+ +----------+ +----------+
| | | +----------+ | | | | | | +----------+ | | |
| Clients |----| | | |------| Servers/ | | Clients |----| | | |------| Servers/ |
| | | | | | | Clients | | | | | | | | Clients |
+----------+ |-------| DUT/SUT |--------| | | +----------+ |-------| DUT/SUT |--------| | |
skipping to change at page 4, line 30 skipping to change at page 4, line 30
+-----------+ +-----------+
| | | |
| Servers | | Servers |
| | | |
+-----------+ +-----------+
Figure 2(Tri-Homed) Figure 2(Tri-Homed)
4.1 Test Considerations 4.1 Test Considerations
4.2 Virtual Clients/Servers 4.1.1 Virtual Clients/Servers
Since firewall testing may involve data sources which emulate multiple Since firewall testing may involve data sources which emulate multiple
users or hosts, the methodology uses the terms virtual clients/servers. users or hosts, the methodology uses the terms virtual clients/servers.
For these firewall tests, virtual clients/servers specify application For these firewall tests, virtual clients/servers specify application
layer entities which may not be associated with a unique physical layer entities which may not be associated with a unique physical
interface. For example, four virtual clients may originate from the interface. For example, four virtual clients may originate from the
same data source. same data source[1]. The test report SHOULD indicate the number of
virtual clients and virtual servers participating in the test on a per
interface(See 4.1.3) basis.
4.3 Test Traffic Requirements Need to include paragraph for synchronize start of test. Data sources
MUST be synchronized to start initiating connections within a specified
time of each other.
4.1.2 Test Traffic Requirements
While the function of a firewall is to enforce access control While the function of a firewall is to enforce access control
policies, the criteria by which those policies are defined vary policies, the criteria by which those policies are defined vary
depending on the implementation. Firewalls may use network layer depending on the implementation. Firewalls may use network layer
and/or, in many cases, application-layer criteria to make and/or, in many cases, application-layer criteria to make
access-control decisions. Therefore, the test equipment used to access-control decisions. Therefore, the test equipment used to
benchmark the DUT/SUT performance MUST consist of real clients and benchmark the DUT/SUT performance MUST consist of real clients and
servers generating legitimate layer 7 conversations. Specifically, servers generating legitimate layer 7 conversations.
the tests defined in this document use HTTP, FTP, and SMTP sessions
for benchmarking the performance of the DUT/SUT. See appendices for The tests defined in this document use HTTP, FTP, and SMTP sessions
specific information regarding the transactions involved in for benchmarking the performance of the DUT/SUT. Other layer 7
conversations are outside the scope of this document. See appendices
for specific information regarding the transactions involved in
establishing/tearing down connections as well as object formats establishing/tearing down connections as well as object formats
for each of the aforementioned protocols. for each of the aforementioned protocols.
4.4 DUT/SUT Traffic Flows 4.1.3 DUT/SUT Traffic Flows
This section discusses the client -> server traffic flow Since the number of interfaces are not fixed, the traffic flows will
requirements associated with the source -> destination interfaces be dependent upon the configuration used in benchmarking the DUT/SUT.
of the firewall. Since the number of interfaces are not fixed, the Note that the term "traffic flows" is associated with client-to-
traffic flows will be dependent upon the configuration used in server requests.
benchmarking the DUT/SUT. Note that the term "traffic flows" is in
the context of client requests since traffic between clients and
servers are, by nature, bi-directional with transactions occurring
between both entities.
In the case of Dual-Homed configurations, traffic flows presented For Dual-Homed configurations, there are two unique traffic flows:
to the DUT/SUT SHOULD consist of client -> server requests associated
with the following interfaces:
Client Server
------ ------
Protected -> Unprotected Protected -> Unprotected
Unprotected -> Protected Unprotected -> Protected
In the case of Tri-Homed configurations, traffic flows presented For Tri-Homed configurations, there are three unique traffic flows:
to the DUT/SUT SHOULD consist of client -> server requests associated
with the following interfaces:
Client Server
------ ------
Protected -> Unprotected Protected -> Unprotected
Protected -> DMZ Protected -> DMZ
Unprotected -> DMZ Unprotected -> DMZ
4.5 Multiple Client/Server Testing 4.1.4 Multiple Client/Server Testing
The methodologies defined in this document MAY be performed with one One or more clients may target multiple servers for a given
or more clients targeting multiple servers for a given application. application. Each virtual client MUST initiate requests(Connection,
When performing tests with one or more virtual clients targeting file transfers, etc.) in a round-robin fashion. For example, if
multiple servers, each virtual client MUST initiate requests the test consisted of six virtual clients targeting three servers,
(Connection, file transfers, etc.) in a round-robin fashion. For the pattern would be as follows:
example, if the test consisted of six virtual clients targeting
three servers, the pattern would be as follows:
Client Target Server(In order of request) Client Target Server(In order of request)
#1 1 2 3 1... #1 1 2 3 1...
#2 2 3 1 2... #2 2 3 1 2...
#3 3 1 2 3... #3 3 1 2 3...
#4 1 2 3 1... #4 1 2 3 1...
#5 2 3 1 2... #5 2 3 1 2...
#6 3 1 2 3... #6 3 1 2 3...
4.6 NAT(Network Address Translation) 4.1.5 NAT(Network Address Translation)
Most firewalls come with Network Address Translation(NAT)networks Most firewalls come with Network Address Translation(NAT)networks
built in which translates internal host IP addresses attached to the built in which translates internal host IP addresses attached to the
protected network to a virtual IP address for communicating across the protected network to a virtual IP address for communicating across the
unprotected network(Internet). Since this involves additional processing unprotected network(Internet). This involves additional processing
on the part of the DUT/SUT, its impact on performance should be measured. on the part of the DUT/SUT and may impact on performance. Therefore,
Therefore, tests defined in this document SHOULD first be ran with NAT tests SHOULD be ran with NAT disabled and NAT enabled to determine the
disabled and then repeated with NAT enabled to determine the performance performance differentials. The test report SHOULD indicate whether NAT
differentials. was enabled or disabled.
4.7 Rule Sets 4.1.6 Rule Sets
Rule sets are a collection of access control policies that determines Rule sets[1] are a collection of access control policies that
which packets the DUT/SUT will forward and which it will reject. The determines which packets the DUT/SUT will forward and which it will
criteria by which these access control policies may be defined will reject. The criteria by which these access control policies may be
vary depending on the capabilities of the product. Therefore, this defined will vary depending on the capabilities of the DUT/SUT. The
document will not attempt to define the specifics of the rule sets, scope of this document is limited to how the rule sets should be
but instead define how the rule sets should be applied when testing applied when testing the DUT/SUT.
the DUT/SUT.
The firewall monitors the incoming traffic and checks to make sure The firewall monitors the incoming traffic and checks to make sure
that the traffic meets one of the defined rules before allowing it to that the traffic meets one of the defined rules before allowing it to
be forwarded. It is RECOMMENDED that a rule be entered for each be forwarded. It is RECOMMENDED that a rule be entered for each
host(i.e. - Virtual client). Although many firewalls permit groups host(i.e. - Virtual client). Although many firewalls permit groups
of IP addresses to be defined for a given rule, tests SHOULD be of IP addresses to be defined for a given rule, tests SHOULD be
performed with as large a rule set as possible to better stress test performed with large rule sets, which are more stressful to the
the DUT/SUT. In addition, a rule MUST be defined which denies access DUT/SUT.
to all traffic which was not previously defined in the rule set,
provided such a rule can be defined in the DUT/SUT. The DUT/SUT SHOULD be configured to denies access to all traffic which
was not previously defined in the rule set.
4.8 Web Caching 4.8 Web Caching
Some firewalls include caching agents in order to reduce network Some firewalls include caching agents in order to reduce network
load. When making a request through a caching agent, the caching load. When making a request through a caching agent, the caching
agent attempts to service the response from its internal resources. agent attempts to service the response from its internal resources.
The cache itself saves responses it receives, such as responses for The cache itself saves responses it receives, such as responses for
HTTP GET requests. Therefore, caching SHOULD be disabled on the HTTP GET requests. The report SHOULD indicate whether web caching
DUT/SUT prior to performing the tests. was enabled or disabled on the DUT/SUT. The test report SHOULD
indicate whether NAT was enabled or disabled.
5. Benchmarking Tests 4.9 Authentication
The following tests offer objectives, procedures, and reporting Access control may involve authentication processes such as user,
formats for benchmarking firewalls. client or session authentication. Authentication is usually performed
by devices external to the firewall itself, such as an authentication
servers and may add to the latency of the system. Any authentication
processes MUST be included as part of connection setup process.
5. Benchmarking Tests
5.1 Concurrent Connection Capacity 5.1 Concurrent Connection Capacity
5.1.1 Objective 5.1.1 Objective
To determine the maximum number of concurrent connections supported To determine the maximum number of concurrent connections supported
by the DUT/SUT, as defined in RFC2647[1]. This test will employ a by the DUT/SUT, as defined in RFC2647[1]. This test will employ a
step search algorithm to obtain the maximum number of concurrent FTP step search algorithm to obtain the maximum number of concurrent
connections the DUT/SUT can maintain. FTP,HTTP or SMTP connections the DUT/SUT can maintain.
5.1.2 Setup Parameters 5.1.2 Setup Parameters
The following parameters MUST be defined. Each variable is configured The following parameters MUST be defined. Each parameters is
with the following considerations. configured with the following considerations.
Connection Attempt Rate - The rate at which new connection requests Connection Attempt Rate - The rate at which new connection requests
are attempted. The rate SHOULD be set lower than maximum rate at are attempted. The rate SHOULD be set lower than maximum rate at
which the DUT/SUT can accept new connection requests. The connection which the DUT/SUT can accept new connection requests.
attempt rate MUST be evenly divided among all of the participating
virtual clients.
Connection Step count - Defines the number of additional connections Connection Step count - Defines the number of additional connections
attempted for each iteration of the search algorithm. The step count attempted for each iteration of the step search algorithm.
SHOULD be a multiple of the number of virtual clients participating
in the test. In addition, the number MUST be greater than or equal to Object/Message - Defines the number of bytes to be transferred across
the number of virtual clients participating in the test. each connection.
5.1.3 Procedure 5.1.3 Procedure
Each virtual client will attempt to establish FTP control connections Each virtual client will attempt to establish connections to their
by issuing an OPEN command to their target server(s) at a fixed rate. target server(s) at a fixed rate in a round robin fashion. Each
Each iteration will involve the virtual clients attempting to iteration will involve the virtual clients attempting to establish a
establish a fixed number of additional connections. This search fixed number of additional connections. This search algorithm will
algorithm will be repeated until either: be repeated until either:
- One or more of the additional connection attempts fail to - One or more of the additional connection attempts fail to
complete complete
- One or more of the previously established connections drop. - One or more of the previously established connections failed.
The transactions between the client and server associated with
establishing FTP connections are defined in Appendix A. In order to
verify that work can be performed across that connection, the virtual
client MUST issue a NOOP command. This command is in addition to the
transactions define in Appendix A, and must be accompanied by the
Command Successful reply from the target server in order to be
considered a successful connection.
After each iteration, all virtual clients MUST issue a NOP command
across all of their open connections to verify that none of the
previously opened connections were dropped by the DUT/SUT.
The algorithm for the test is as follows:
CONSTANT
STEP = Connection Step Count; {# of additional connection attempts Data transfers SHOULD be performed on each connection after the
per iteration} given connection is established. Data transfers MUST be performed on
MAX = ...; {maximum connections supported by test tool all connections after all of the addition connection have been
implementation} established.
VARIABLE
CURRENT := 0; {Highest passed value} When benchmarking with FTP, virtual clients will issue NOOP command's
STATUS = PASS; to validate that work can be performed across each connection. The
virtual clients must receive a Command Successful reply from the
target server in order to be considered a valid connection.
BEGIN When benchmarking with other applications such as HTTP or SMTP,
RESET; {RESET all connections} validation of the connection will be performed by initiating
DO object/message transfers. All bytes associated with the object/message
BEGIN transfers MUST be received by the requesting virtual client in order
EstablishStepConnections; { See Appendix A.3} to be considered a valid connection.
IF(AttemptsSucessful) THEN
BEGIN
VerifyAllConnections { Send NOP command over all open
connections }
IF(Received replies for all NO0P commands) THEN
BEGIN { Maximum number of connections NOT found }
CURRENT := CURRENT + Step Count
END
ELSE
BEGIN { Maximum number of connections < Step Count }
STATUS := FAIL
END
END
ELSE
BEGIN { Maximum number of connections < Step Count }
STATUS := FAIL
END
END
END WHILE(STATUS == PASS && [[CURRENT + Step Count] < MAX])
END
END {Value of CURRENT equals number connections supported by DUT/SUT}
5.1.4 Measurements 5.1.5 Measurements
Whether all of the connection attempts were successfully completed. Total number of connections that were successfully completed in a
Test equipment MUST be able to track each connection to verify all step. Test equipment MUST be able to track each connection to verify
required transaction between the virtual client and server completed all required transaction between the virtual client and server
successfully. This includes successful completion of both the completed successfully. This includes successful completion of both
command sequences and exchanging of any data across each of those the command sequences and exchanging of any data across each of those
connections. connections.
5.1.5 Reporting Format 5.1.6 Reporting Format
Maximum concurrent connections reported MUST be the aggregate number Maximum concurrent connections reported MUST be the aggregate number
of connections completed for the last successful iteration. Report of connections completed for the last successful iteration. Report
SHOULD also include: SHOULD also include:
- Number of virtual clients and servers participating in the test - Connection Attempt Rate.
- Whether NAT was enabled or disabled. - Connection Step Count.
A log file MAY be generated which includes for each step iteration:
- Pass/Fail Status.
- Total connections established.
- Number of previously established connections dropped.
- Number of the additional connections that failed to complete.
5.2 Maximum Connection Setup Rate 5.2 Maximum Connection Setup Rate
5.2.1 Objective 5.2.1 Objective
To determine the maximum connection rate which the DUT/SUT can To determine the maximum connection rate which can be supported
successfully establish connections. As with the Concurrent Connection through the DUT/SUT. As with the Concurrent Connection Capacity test,
Capacity test, FTP session will be used to determine this metric. FTP,HTTP and SMTP sessions will be used to determine this metric.
5.2.2 Setup Parameters 5.2.2 Setup Parameters
The following parameters MUST be defined. Each variable is configured The following parameters MUST be defined. Each test parameter is
with the following considerations. configured with the following considerations.
Initial Attempt Rate - The rate at which the initial connection Initial Attempt Rate - The rate at which the initial connection
requests are attempted. The connection attempt rate MUST be evenly requests are attempted.
divided among all of the participating virtual clients.
Rate Step Count - The rate at which connection attempts are either
increased or decreased during the search algorithm. The rate step
count MUST be evenly divided among all of the participating virtual
clients.
Number of Connections - Defines the number of connections that must Number of Connections - Defines the number of connections that must
be established. The number MUST be between the number of be established. The number MUST be between the number of
participating virtual clients and the maximum number supported by the participating virtual clients and the maximum number supported by the
implementation. It is RECOMMENDED not to exceed the concurrent DUT/SUT. It is RECOMMENDED not to exceed the concurrent connection
connection capacity found in section 5.1. capacity found in section 5.1. The connection rate may vary depending
on the number of connections attempted.
5.2.3 Procedure Object/Message - Defines the number of bytes to be transferred across
each connection.
An algorithm similar to the one used to determine the concurrent 5.2.3 Procedure
connection capacity will be used to determine the maximum connection
rate. This test iterates the rate at which connections are attempted
by the virtual clients to their associated server(s).
The connection establishment rate might be determined for different An iterative search algorithm will be used to determine the maximum
numbers of connections but in each test run, the number MUST remain connection rate. This test iterates through different connection rates
constant and SHOULD be equal to or less than the maximum connection with a fixed number of connections attempted by the virtual clients to
capacity. their associated server(s).
As with the connection capacity test, NOOP commands will be Each iteration will use the same connection establishment and
generated when both establishing the connection and after all connection validation algorithms defined in the concurrent capacity
connections have been established to verify none of the previously test(See 5.1). After each iteration, the tester MUST close all
established connections have dropped. connections prior to continuing to the next iteration.
5.2.4 Measurements 5.2.4 Measurements
Min/Avg/Max connection times MUST be measured. The connection time The highest connection rate, in connections per second, for which
SHALL begin when the client initiates the OPEN command and end when all connections completed successfully. Test equipment MUST be able
the client received a successful login acknowledgment from the to track each connection to verify all required transaction between
server. Note that although the NOOP command is included as part of the virtual client and server completed successfully. This includes
the procedure for establishing the connection, it MUST not be successful completion of both the command sequences and exchanging
included as part of the connection establishment time. of any data across each of those connections.
5.2.5 Reporting Format 5.2.5 Reporting Format
The results for these tests SHOULD be reported in the form of a The maximum connection rate reported MUST be the maximum rate for
graph. The x coordinate SHOULD be the connection count. The y which all connections successfully completed.
coordinate should be the connection establishment time. The graph
SHOULD include the Minimum, Average and Maximum connection times.
Report SHOULD also include: A log file MAY be generated which includes for each step iteration:
- Number of virtual clients and servers participating in the test - Pass/Fail Status.
- Whether NAT was enabled or disabled. - Connection attempt rate.
- Number of the connections that failed to complete.
- Total connections established.
5.3 Denial Of Service Handling 5.3 Connection Establishment Time
5.3.1 Objective 5.3.1 Objective
To determine the effect of a denial of service attack in so far as To characterize the connection establishment time[1] through or with
it's impacts on connection establishment rates through the DUT/SUT. the DUT/SUT as a function of the number of open connections.
The Denial Of Service Handling test should be ran after obtaining
baseline measurements from section 5.2. The results can then be
compared with the baseline measurements to obtain the performance
differentials.
5.3.2 Setup Parameters 5.3.2 Setup Parameters
The following parameters MUST be defined. Each variable is The following parameters MUST be defined. Each parameters is
configured with the following considerations. configured with the following considerations.
Initial Attempt Rate - The rate at which the initial connection Connection Attempt Rate - The rate at which new connection requests
requests are attempted. The connection attempt rate MUST be evenly are attempted. The rate SHOULD be set lower than maximum rate at
divided among all of the participating virtual clients. which the DUT/SUT can accept new connection requests.
Rate Step Count - The rate at which connection attempts are either Connection Step count - Defines the number of additional connections
increased or decreased during the search algorithm. The rate step attempted for each iteration of the step algorithm.
count MUST be evenly divided among all of the virtual clients
participating in the test. Object/Message - Defines the number of bytes to be transferred across
each connection.
5.3.3 Procedure
The test will use the same algorithm as defined in the Concurrent
Capacity Test. This includes both the connection establishment and
validation of each connection by transferring data across each
connection.
5.3.4 Measurement
For each iteration, the tester MUST measure the Min/Avg/Max connection
times for the additional connections. It is RECOMMENDED that in addition
to the application layer, the tester include measurements at the lower
layer protocols(i.e. - TCP, ATM) when applicable. For each of the
protocols which the tester is measuring, the connection establishment
time shall consist of all transactions required to enable data to be
transferred across the given connection.
For example, FTP requires the user to login prior to being able to get
files, view directories and so forth. Connection establishment times
MUST include all of these transactions. In the case of TCP, the
connection establishment time would consist of the three-way handshake
between the two hosts(See Appendix D).
5.3.5 Reporting Format
Graph of the min/avg/maximum connection establishment times versus the
number of open connections. The report MUST identify the layer for which
the measurement was taken(i.e. - Application, transport, etc).
5.4 Denial Of Service Handling
5.4.1 Objective
To determine the effect of a denial of service attack on connection
establishment rates through the DUT/SUT. The Denial Of Service Handling
test should be ran after obtaining baseline measurements from section
5.2.
When a normal TCP connection starts, a destination host receives a SYN
(synchronize/start)packet from a source host and sends back a SYN ACK
(synchronize acknowledge). The destination host must then hear an ACK
(acknowledge) of the SYN ACK before the connection is established. The
TCP SYN attack exploits this design by having an attacking source host
generate TCP SYN packets with random source addresses towards a victim
host, thereby consuming that hosts resources.
Some firewalls employ one or more mechanisms to guard against SYN
attacks. If such mechanisms exist on the DUT/SUT, tests SHOULD be ran
with these mechanisms enabled to determine how well the DUT/SUT can
maintain the baseline connection rates determined in section 5.2
under such attacks.
5.4.2 Setup Parameters
The following parameters MUST be defined. Each parameter is configured
with the following considerations.
Initial Attempt Rate - The rate at which the initial connection
requests are attempted.
Number of Connections - Defines the number of connections that must Number of Connections - Defines the number of connections that must
be established. The number MUST be between the number of be established. The number MUST be between the number of
participating clients and the maximum number supported by the participating clients and the maximum number supported by the
implementation. It is RECOMMENDED not to exceed the concurrent DUT/SUT. It is RECOMMENDED not to exceed the concurrent connection
connection capacity found in section 5.1. capacity found in section 5.1.
SYN Attack Rate - Defines the rate at which the server(s) are
targeted with TCP SYN packets.
5.3.3 Procedure SYN Attack Rate - Defines the rate at which the server(s) are targeted
with TCP SYN packets.
This test will use the same procedure as defined in the maximum 5.4.3 Procedure
connection setup rate, except that the test traffic will include
TCP SYN packets targeting the server(s) IP address or NAT proxy
address.
TCP employs a "three-way handshake" when establishing a connection This test uses the same procedure as defined in the maximum connection
between two hosts. When a normal TCP connection starts, a destination setup rate, with the addition of TCP SYN packets targeting the
host receives a SYN (synchronize/start)packet from a source host and server(s) IP address or NAT proxy address.
sends back a SYN ACK (synchronize acknowledge). The destination host
must then hear an ACK (acknowledge) of the SYN ACK before the
connection is established. The TCP SYN attack exploits this design
by having an attacking source host generate TCP SYN packets with
random source addresses towards a victim host, thereby consuming
that hosts resources.
The tester originating the TCP SYN attack MUST be attached to the The tester originating the TCP SYN attack MUST be attached to the
Unprotected network. In addition, the tester MUST not respond to the Unprotected network. In addition, the tester MUST not respond to the
SYN ACK packets sent by target server in response to the SYN packet. SYN ACK packets sent by target server in response to the SYN packet.
5.3.4 Measurements 5.4.4 Measurements
Min/Avg/Max connection times MUST be measured. The connection time The highest connection rate, in connections per second, for which
SHALL begin when the client initiates the OPEN command and end when all legitimate connections completed successfully. Test equipment
the client received a successful login acknowledgment from the MUST be able to track each connection to verify all required
server. Note that although the NOOP command is included as part of transaction between the virtual client and server completed
the procedure for establishing the connection, it MUST not be successfully. This includes successful completion of both the command
included as part of the connection establishment time. sequences and exchanging of any data across each of those connections.
5.2.5 Reporting Format In addition, the tester SHOULD track SYN packets associated with the
SYN attack which the DUT/SUT forwards on the protected or DMZ
interface(s).
The results for these tests SHOULD be reported in the form of a 5.4.5 Reporting Format
graph. The x coordinate SHOULD be the connection count. The y
coordinate should be the connection establishment time. The graph
SHOULD include the Minimum, Average and Maximum connection times.
Report SHOULD also include: The maximum connection rate reported MUST be the maximum rate for
which all connections successfully completed. The report SHOULD
include what percentage of TCP SYN packets were forwarded by the
DUT/SUT.
- Number of virtual clients and servers participating in the test A log file MAY be generated which includes for each step iteration:
- Whether NAT was enabled or disabled.
- TCP SYN Attack Rate
5.4 Single Application Goodput - Pass/Fail Status.
- Connection attempt rate.
- Number of the connections that failed to complete.
- Total connections established.
This section defined the procedures for baselining the Goodput of the 5.5 Single Application Goodput
This section defined the procedures for base lining the Goodput[1] of the
DUT/SUT for FTP, HTTP and SMTP traffic. DUT/SUT for FTP, HTTP and SMTP traffic.
5.4.1.1 FTP Goodput 5.5.1 FTP Goodput
5.4.1.2 Objective 5.5.1.1 Objective
The File Transfer Protocol is a common application used by companies The File Transfer Protocol is a common application used by companies
to transfer files from one device to another. Evaluating FTP Goodput to transfer files from one device to another. Evaluating FTP Goodput
will allow individuals to measure how much successful traffic has will allow individuals to measure how much successful traffic has
been forwarded by the DUT/SUT. been forwarded by the DUT/SUT.
5.4.1.3 Setup Parameters 5.5.1.2 Setup Parameters
The following parameters MUST be defined. Each variable is The following parameters MUST be defined. Each parameter is configured
configured with the following considerations. with the following considerations.
Number of Connections - Defines the number of connections to be Number of Connections - Defines the number of connections to be
opened for transferring data objects. Number MUST be equal or opened for transferring data objects. Number MUST be equal or
greater than the number of virtual clients participating in the greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual client test. The number SHOULD be a multiple of the virtual client
participating in the test. participating in the test.
Connection Rate - Defines the rate at which connections are Connection Rate - Defines the rate at which connections are established.
established. Number MUST be evenly divided among all of the
virtual clients participating in the test.
Object Size - Defines the number of bytes to be transferred across Object Size - Defines the number of bytes to be transferred across each
each connection. RECOMMENDED object sizes still need to be connection.
determined.
5.4.1.4 Procedure 5.5.1.3 Procedure
Each virtual client will establish a FTP connection to its respective Each virtual client will establish a FTP connection to its respective
server(s) at a user specified rate. The transaction involved in server(s) in a round robin fashion at the connection rate. The
establishing the FTP connection should follow the procedure defined transaction involved in establishing the FTP connection should follow
in Appendix A. the procedure defined in Appendix A.
After the login process has been completed, the virtual client will After the login process has been completed, the virtual client will
initiate a file transfer by issuing a "Get" command. The "Get" initiate a file transfer by issuing a "Get" command. The "Get"
command will target a predefined file of object size bytes. command will target a predefined file of Object Size bytes.
Once the file transfer has completed, the virtual client will close Once the file transfer has completed, the virtual client will close
the FTP connection by issuing the QUIT command. When testing multiple the FTP connection by issuing the QUIT command.
object sizes, all connections established for the previous file
transfer MUST have been torn down before testing the next object
size.
5.4.2.2 Measurement 5.5.1.4 Measurement
The Goodput for each of the object sizes transferred MUST be The Goodput for each interface of the DUT/SUT MUST be measured. See
measured. See appendix B for details in regards to measuring the appendix D for details in regards to measuring the Goodput of the
Goodput of the DUT/SUT. Only file transfers which have been DUT/SUT. Only file transfers which have been completed are to be
successfully acknowledged by the server are to be included in the included in the Goodput measurements.
Goodput measurements.
The transaction time for each object transferred MUST be measured. The average transaction time each object successfully transferred MAY
The start time will begin when the time the "Get" commands is be measured. The start time will begin when the time the "Get"
initiated and end at the time when the client receives an commands is initiated and end at the time when the client receives
acknowledgment from the server that file transfer has completed. an acknowledgment from the server that file transfer has completed.
5.4.2.3 Reporting Format 5.5.1.5 Reporting Format
Goodput analysis: The Goodput for each interface of the DUT/SUT MUST be reported in
Reporting SHOULD include a graph of the number of connections bits per second. This will be the aggregate of session Goodput's
versus the measured Goodput in Mbps. measured for a given interface.
Failure analysis: Failure analysis:
Reporting should include a graph of number of connections versus
percent success. The report SHOULD include the percentage of connections that
failed. This includes:
- Connections which failed to establish
- Connections which failed to complete the object transfer
Transaction Processing analysis: Transaction Processing analysis:
Reporting should include a graph of number of virtual connections
versus average transaction time.
6.4.3 SMTP Goodput The report SHOULD include average transaction time in transactions
per second.
The report SHOULD also include the object size(Bytes) being transferred.
5.5.2 SMTP Goodput
5.5.2.1 Objective
Another application commonly in use today is the mail transfer. One Another application commonly in use today is the mail transfer. One
the common transport mechanisms for mail messages is the Simple Mail the common transport mechanisms for mail messages is the Simple Mail
Transfer Protocol(SMTP). As with the FTP Goodput, the SMTP Goodput Transfer Protocol(SMTP). The SMTP Goodput will allow individuals to
will allow individuals to measure how much successful SMTP traffic measure how much successful SMTP traffic has been forwarded by the
has been forwarded by the DUT/SUT. DUT/SUT.
5.4.1.3 Setup Parameters 5.5.2.2 Setup Parameters
The following parameters MUST be defined. Each variable is The following parameters MUST be defined. Each parameter is
configured with the following considerations. configured with the following considerations.
Number of Connections - Defines the number of connections to be Number of Connections - Defines the number of connections to be
opened for transferring data objects. Number MUST be equal or opened for transferring data objects. Number MUST be equal or
greater than the number of virtual clients participating in the greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual client test. The number SHOULD be a multiple of the virtual client
participating in the test. participating in the test.
Connection Rate - Defines the rate at which connections are Connection Rate - Defines the rate at which connections are
attempted. Number MUST be evenly divided among all of the attempted.
virtual clients participating in the test.
Message Size - Defines the number of bytes to be transferred across Message Size - Defines the number of bytes to be transferred across
each connection. RECOMMENDED message sizes still need to be each connection.
determined.
5.4.1.4 Procedure 5.5.2.3 Procedure
Each virtual client will establish a SMTP connection to its Each virtual client will establish a SMTP connection to its
respective server(s) at a user specified rate. The transaction respective server(s) in a round robin fashion at the connection rate.
involved in establishing the SMTP connection should follow the The transaction involved in establishing the SMTP connection should
procedure defined in Appendix B. follow the procedure defined in Appendix B.
After the greeting exchanges have been completed, the client will After the greeting exchanges have been completed, the client will
initiate the transfer of the message by initiating the MAIL command. initiate the transfer of the message by initiating the MAIL command.
The client will then send the predefined message. The client will then send the predefined message of Object Size.
Once the message has been acknowledged as being received by the Once the message has been acknowledged as being received by the
server, the virtual client will then close the connection. When server, the virtual client will then close the connection.
testing multiple message sizes, all connections established for the
previous test MUST be torn down before testing the next message
size. This process is to be repeated for each of the defined message
sizes.
5.4.2.2 Measurement 5.5.2.4 Measurement
The Goodput for each of the message sizes transferred MUST be The Goodput for each interface of the DUT/SUT MUST be measured. See
measured. See appendix D for details in regards to measuring the appendix D for details in regards to measuring the Goodput of the
Goodput of the DUT/SUT. Only messages which have been successfully DUT/SUT. Only message transfers which have been completed are to be
acknowledged by the server are to be included in the Goodput included in the Goodput measurements.
measurements.
The transaction time for each message transferred MUST be measured. The average transaction time for each message transferred MAY be
The start time will begin when the time the "MAIL" command is measured. The start time will begin when the time the "MAIL" command
initiated and end at the time when the client receives an is initiated and end at the time when the client receives an
acknowledgment from the server that the message has been received. acknowledgment from the server that the message has been received.
6.4.2.3 Reporting Format 5.5.2.5 Reporting Format
Goodput analysis: Goodput analysis:
Reporting SHOULD include a graph of the number of connections
versus the measured Goodput in Mbps. The Goodput for each interface of the DUT/SUT MUST be reported in
bits per second. This will be the aggregate of session Goodput's
measured for a given interface.
Failure analysis: Failure analysis:
Reporting should include a graph of number of connections versus
percent success. The report SHOULD include the percentage of connections that
failed. This includes:
- Connections which failed to establish
- Connections which failed to complete the object transfer
Transaction Processing analysis: Transaction Processing analysis:
Reporting should include a graph of number of virtual connections
versus average transaction time.
6.4.3 HTTP Goodput The report SHOULD include average transaction time in transactions
per second.
The report SHOULD also include the object size(Bytes) being transferred.
5.5.3 HTTP Goodput Goodput
5.5.3.1 Objective
Another common application is the World Wide Web (WWW) application Another common application is the World Wide Web (WWW) application
that can access documents over the Internet. This application uses that can access documents over the Internet. This application uses
the Hypertext Transfer Control Protocol (HTTP) to copy information the Hypertext Transfer Control Protocol (HTTP) to copy information
from one system to another. from one system to another.
HTTP Goodput measurement is actually determined by evaluating the HTTP Goodput measurement is actually determined by evaluating the
Forwarding rate of packets. This is based on measuring only data Forwarding rate of packets. This is based on measuring only data
that has successfully been forwarded to the destination interface. that has successfully been forwarded to the destination interface.
When benchmarking the performance of the DUT/SUT, consideration of When benchmarking the performance of the DUT/SUT, consideration of
the HTTP version being used must be taken into account. Appendix C the HTTP version being used must be taken into account. Appendix C
of this document discusses enhancements to the HTTP protocol which of this document discusses enhancements to the HTTP protocol which
may impact performance results. may impact performance results.
6.4.1.3 Setup Parameters 5.5.3.2 Setup Parameters
The following parameters MUST be defined. Each variable is The following parameters MUST be defined. Each variable is
configured with the following considerations. configured with the following considerations.
Number of Connections - Defines the number of HTTP connections Number of Connections - Defines the number of HTTP connections
to be opened for transferring data objects. Number MUST be equal or to be opened for transferring data objects. Number MUST be equal or
greater than the number of virtual clients participating in the greater than the number of virtual clients participating in the
test. The number SHOULD be a multiple of the virtual client test. The number SHOULD be a multiple of the virtual client
participating in the test. participating in the test.
Connection Rate - Defines the rate at which connections are Connection Rate - Defines the rate at which connections are
attempted. Number MUST be evenly divided among all of the attempted.
virtual clients participating in the test.
Object Size - Defines the number of bytes to be transferred Object Size - Defines the number of bytes to be transferred
across each connection. Need to determine the RECOMMENDED across each connection.
object sizes.
6.4.2.1 HTTP Procedure 5.5.3.3 HTTP Procedure
For the HTTP Goodput tests, it is RECOMMENDED to determine which For the HTTP Goodput tests, it is RECOMMENDED to determine which
version of HTTP the DUT/SUT has implemented and use the same version of HTTP the DUT/SUT has implemented and use the same
version for the test. To determine the version of HTTP, the user version for the test. To determine the version of HTTP, the user
documentation of the DUT/SUT SHOULD be consulted. documentation of the DUT/SUT SHOULD be consulted.
Each client will attempt to establish HTTP connection's to their Each client will attempt to establish HTTP connection's to their
respective servers a user defined rate. The clients will attach to respective servers a user defined rate. The clients will attach to
the servers using either the servers IP address or NAT proxy the servers using either the servers IP address or NAT proxy
address. address.
skipping to change at page 16, line 19 skipping to change at page 16, line 41
transfers must be completed and the connections closed for all transfers must be completed and the connections closed for all
of the participating clients prior testing the next object size. of the participating clients prior testing the next object size.
This process is repeated until all of the defined objects are This process is repeated until all of the defined objects are
tested. tested.
When employing HTTP/1.1, all objects defined by the user will When employing HTTP/1.1, all objects defined by the user will
be requested with a GET request over the same connection. The be requested with a GET request over the same connection. The
connection should then be torn down after all of the objects connection should then be torn down after all of the objects
have been transferred. have been transferred.
6.4.2.2 Measurement 5.5.3.4 Measurement
The Goodput for each of the objects sizes transferred MUST be The Goodput for each of the objects sizes transferred MUST be
measured. See appendix D for details in regards to measuring the measured. See appendix D for details in regards to measuring the
Goodput of the DUT/SUT. Only objects which have been successfully Goodput of the DUT/SUT. Only objects which have been successfully
acknowledged by the server are to be included in the Goodput acknowledged by the server are to be included in the Goodput
measurements. measurements.
The transaction times for each object transferred MUST measured. The transaction times for each object transferred MUST measured.
The transaction connection time starts when the connection is The transaction connection time starts when the connection is
made and will end when the web pages is completely mapped on the made and will end when the web pages is completely mapped on the
virtual client (when the client sends an acknowledgment packet is virtual client (when the client sends an acknowledgment packet is
sent from the client). sent from the client).
6.4.2.3 Reporting Format 5.5.3.5 Reporting Format
Goodput analysis: Goodput analysis:
Reporting SHOULD include a graph of the number of connections
versus the measured Goodput in Mbps. The Goodput for each interface of the DUT/SUT MUST be reported in
bits per second. This will be the aggregate of session Goodput's
measured for a given interface.
Failure analysis: Failure analysis:
Reporting should include a graph of number of connections versus
percent success. The report SHOULD include the percentage of connections that
failed. This includes:
- Connections which failed to establish
- Connections which failed to complete the object transfer
Transaction Processing analysis: Transaction Processing analysis:
Reporting should include a graph of number of virtual connections
versus average transaction time. The report SHOULD include average transaction time in transactions
per second.
The report SHOULD also include the object size(Bytes) being transferred.
Version Information Version Information
Report MUST include the version of HTTP used for the test. In Report MUST include the version of HTTP used for the test. In
addition, if the HTTP/1.1 is used, the number of concurrent GET's addition, if the HTTP/1.1 is used, the number of concurrent GET's
allowable(Pipelining) SHOULD be reported. allowable(Pipelining) SHOULD be reported.
6.5 Concurrent Application Goodput 5.6 Concurrent Application Goodput
5.6.1 Objective
To determine the Goodput of the DUT/SUT when offering a mix of FTP, To determine the Goodput of the DUT/SUT when offering a mix of FTP,
SMTP and HTTP traffic flows. Real world traffic does not consist SMTP and HTTP traffic flows. Real world traffic does not consist
of a single protocol, but a mix of different applications. This of a single protocol, but a mix of different applications. This
test will allow an individual to determine how well the DUT/SUT test will allow an individual to determine how well the DUT/SUT
handles a mix of applications by comparing the results to the handles a mix of applications by comparing the results to the
individual baseline measurements. individual baseline measurements.
6.5.1 Setup Parameters 5.6.2 Setup Parameters
The following parameters MUST be defined. Each variable is The following parameters MUST be defined. Each variable is
configured with the following considerations. configured with the following considerations.
Number of Connections - Defines the aggregate number of connections Number of Connections - Defines the aggregate number of connections
to be opened for transferring data/message objects. Number MUST be to be opened for transferring data/message objects. Number MUST be
equal to or greater than the number of virtual clients participating equal to or greater than the number of virtual clients participating
in the test. The number SHOULD be a multiple of the virtual client in the test. The number SHOULD be a multiple of the virtual client
participating in the test. participating in the test.
skipping to change at page 17, line 46 skipping to change at page 18, line 26
FTP Percentage - Defines the percentage of traffic connections FTP Percentage - Defines the percentage of traffic connections
which are to consist of FTP file transfers. which are to consist of FTP file transfers.
SMTP Percentage - Defines the percentage of traffic connections SMTP Percentage - Defines the percentage of traffic connections
which are to consist of SMTP Message transfers. which are to consist of SMTP Message transfers.
HTTP Percentage - Defines the percentage of traffic connections HTTP Percentage - Defines the percentage of traffic connections
which are to consist of HTTP GET requests. which are to consist of HTTP GET requests.
6.5.1 Procedure 5.6.3 Procedure
This test will run each of the single application Goodput tests, This test will run each of the single application Goodput tests,
for which there is a defined percentage, concurrently. For each for which there is a defined percentage, concurrently. For each
of the defined traffic types, the connection establishment, of the defined traffic types, the connection establishment,
data/message transfer and teardown procedures will be the same data/message transfer and teardown procedures will be the same
as defined in the individual tests. as defined in the individual tests.
6.5.2 Measurements 5.6.4 Measurements
As with the individual tests, the Goodput for each of the defined As with the individual tests, the Goodput for each of the defined
traffic types MUST be measured. See appendix D for details in traffic types MUST be measured. See appendix D for details in
regards to measuring the Goodput of the DUT/SUT. Only messages/data regards to measuring the Goodput of the DUT/SUT. Only messages/data
which have been successfully acknowledged as being transferred are which have been successfully acknowledged as being transferred are
to be included in the Goodput measurements. to be included in the Goodput measurements.
The transaction times for each of the defined applications MUST be The transaction times for each of the defined applications MUST be
measured. See the appropriate single application Goodput test for measured. See the appropriate single application Goodput test for
the specifics of measuring the transaction times for each of the the specifics of measuring the transaction times for each of the
defined traffic types. defined traffic types.
6.5.3 Reporting Format 5.6.5 Reporting Format
Goodput analysis: Goodput analysis:
Reporting SHOULD include a graph of the number of connections Reporting SHOULD include a graph of the number of connections
versus the measured Goodput in Mbps for each of the defined versus the measured Goodput in Mbps for each of the defined
traffic types(FTP, SMTP, HTTP). traffic types(FTP, SMTP, HTTP).
Failure analysis: Failure analysis:
Reporting should include a graph of number of connections versus Reporting should include a graph of number of connections versus
percent success for each of the defined traffic types. percent success for each of the defined traffic types.
Transaction Processing analysis: Transaction Processing analysis:
Reporting should include a graph of number of virtual connections Reporting should include a graph of number of virtual connections
versus average transaction for each of the defined traffic types. versus average transaction for each of the defined traffic types.
6.6 Illegal Traffic Handling 5.7 Illegal Traffic Handling
To determine the behavior of the DUT/SUT when presented with a To determine the behavior of the DUT/SUT when presented with a
combination of both legal and Illegal traffic. combination of both legal and Illegal traffic.
6.6.1 Procedure 5.7.1 Procedure
Still Needs to be determined Still Needs to be determined
6.6.2 Measurements 5.7.2 Measurements
Still Needs to be determined Still Needs to be determined
6.6.3 Reporting Format 5.7.3 Reporting Format
Still Needs to be determined Still Needs to be determined
6.7 Latency 5.8 Latency
Determine the latency of application layer data through the DUT/SUT. Determine the latency of application layer data through the DUT/SUT.
6.7.1 Procedure 5.8.1 Procedure
Still Needs to be determined Still Needs to be determined
6.7.2 Measurements 5.8.2 Measurements
Still needs to be determined. Still needs to be determined.
6.7.3 Reporting format 5.8.3 Reporting format
Still needs to be determined. Still needs to be determined.
APPENDICES APPENDICES
APPENDIX A: FTP(File Transfer Protocol) APPENDIX A: FTP(File Transfer Protocol)
A.1 Introduction A.1 Introduction
The FTP protocol was designed to be operated by interactive end users The FTP protocol was designed to be operated by interactive end users
skipping to change at page 19, line 42 skipping to change at page 20, line 10
Unlike other protocols, FTP uses two connections. One connection, Unlike other protocols, FTP uses two connections. One connection,
called the control connection, is used to pass commands between called the control connection, is used to pass commands between
the client and the server. The other, called the data connection, is the client and the server. The other, called the data connection, is
used to transfer the actual data(Files, directory lists, etc.). used to transfer the actual data(Files, directory lists, etc.).
A.2 Connection Establishment/Teardown(Control) A.2 Connection Establishment/Teardown(Control)
FTP control connections are established by issuing OPEN command FTP control connections are established by issuing OPEN command
targeting either the URL or a specific IP address. Since the targeting either the URL or a specific IP address. Since the
methodology does not include DNS servers, OPEN commands should methodology does not include DNS servers, OPEN commands should
target specific IP address of target server. target specific IP address of target server. A TCP connection
will be established between the client and target server.
It is RECOMMENDED to perform the test using Anonymous FTP Login and The client will then begin the login process. When logging in,
should use the following syntax: it is RECOMMENDED to perform the test using Anonymous FTP Login
and should use the following syntax:
User ID: Anonymous User ID: Anonymous
Password: will correspond to the System ID Password: will correspond to the System ID
(client1_1@test.net through client 1_8@test.net) (client1_1@test.net through client 1_8@test.net)
Once a successful login acknowledgment is received from the server, Once a successful login acknowledgment is received from the server,
the client may then initiate a file transfer. After all transfer the client may then initiate a file transfer. After all transfer
operations have been completed, the FTP connection may be closed by operations have been completed, the FTP connection may be closed by
issuing a QUIT command. issuing a QUIT command.
skipping to change at page 20, line 40 skipping to change at page 21, line 19
The SMTP defines a simple straight forward way to move messages The SMTP defines a simple straight forward way to move messages
between hosts. There are two roles in the SMTP protocol, one is the between hosts. There are two roles in the SMTP protocol, one is the
sender and one is the receiver. The sender acts like a client and sender and one is the receiver. The sender acts like a client and
establishes a TCP connection with the receiver which acts like a establishes a TCP connection with the receiver which acts like a
server. The transactions defined in this section will use the terms server. The transactions defined in this section will use the terms
client and server in place of sender and receiver. client and server in place of sender and receiver.
B.2 Connection Establishment/Teardown B.2 Connection Establishment/Teardown
Each connection involves a connection greeting between the Each connection involves a connection greeting between the
sender(Client) and receiver(Server). After a connection is first sender(Client) and receiver(Server). The syntax used to identify each
established, both the server and the client will exchange greetings other's hostnames during this greeting exchange SHOULD be:
identifying themselves. The syntax used to identify each other's
hostnames SHOULD be:
"SMTPRcv_<Virtual_Server>.com" "SMTPRcv_<Virtual_Server>.com"
"SMTPSender_<Virtual Client>.com" "SMTPSender_<Virtual Client>.com"
where <Virtual_Client> and <Virtual_Server> represent a where <Virtual_Client> and <Virtual_Server> represent a
unique virtual number for the client and server unique virtual number for the client and server
respectively. respectively.
The basic transactions in moving mail between two hosts involve three The basic transactions in moving mail between two hosts involve three
basic steps which are outlined below. These are: basic steps which are outlined below. These are:
skipping to change at page 22, line 39 skipping to change at page 23, line 12
identifies a file in its cache, it needn't reload it unless it has identifies a file in its cache, it needn't reload it unless it has
changed since the last time it was used. changed since the last time it was used.
- Support for multiple hosts - Support for multiple hosts
It is common for an ISP to host more than one Web site on a single It is common for an ISP to host more than one Web site on a single
server. In such a case, each domain requires its own IP address. server. In such a case, each domain requires its own IP address.
C.3 Object Format C.3 Object Format
Object SHOULD be an HTLM formatted object. Object SHOULD be an HTML formatted object.
Append D GOODPUT Measurements Append D. GOODPUT Measurements.
Methodology for determining the "Goodput" of the DUT still needs The Goodput will measure the number of bits per second forwarded by
to be determined. Note that the methodology should be independent the DUT/SUT and will be referenced to the application level data. The
of the application which is initiating the transfer of the object formula for determining Goodput of the DUT/SUT is as follows:
(File, message, Web Page, etc.).
ObjectSize(Bytes) * 8
Goodput(Bits/Sec) = Transfer Time(Seconds)
Transfer Time starts when the first bit of the object/message is
received at the destination port of the tester. The transfer time ends
when the last bit of the object/message is received at the destination
port of the tester.
Appendix E. References Appendix E. References
Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, [1] Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647,
February 1998. February 1998.
J. Postel, "Simple Mail Transfer Protocol", RFC 821, August 1982. [2] J. Postel, "Simple Mail Transfer Protocol", RFC 821, August 1982.
R. Fielding, J. Gettys, J. Mogul, H Frystyk, T. Berners, "Hypertext [3] R. Fielding, J. Gettys, J. Mogul, H Frystyk, T. Berners, "Hypertext
Transfer Protocol -- HTTP/1.1", January 1997 Transfer Protocol -- HTTP/1.1", January 1997
J. Postel, J. Reynolds, "File Transfer Protocol(FTP)", October 1985 [4] J. Postel, J. Reynolds, "File Transfer Protocol(FTP)", October 1985
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/