draft-ietf-bmwg-secperf-06.txt   draft-ietf-bmwg-secperf-07.txt 
Network Working Group D. Newman Network Working Group D. Newman
INTERNET-DRAFT Data Communications INTERNET-DRAFT Data Communications
Expires in November 1999 May 1999 Expires in November 1999 May 1999
Benchmarking Terminology for Firewall Performance Benchmarking Terminology for Firewall Performance
<draft-ietf-bmwg-secperf-06.txt> <draft-ietf-bmwg-secperf-07.txt>
Status of This Memo Status of This Memo
This document is an Internet-Draft and is in full conformance with This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026. all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet- other groups may also distribute working documents as Internet-
Drafts. Drafts.
skipping to change at line 134 skipping to change at line 134
it difficult to do direct performance comparisons. Finally, more and it difficult to do direct performance comparisons. Finally, more and
more organizations are deploying firewalls on internal networks more organizations are deploying firewalls on internal networks
operating at relatively high speeds, while most firewall operating at relatively high speeds, while most firewall
implementations remain optimized for use over relatively low-speed implementations remain optimized for use over relatively low-speed
wide-area connections. As a result, users are often unsure whether wide-area connections. As a result, users are often unsure whether
the products they buy will stand up to relatively heavy loads. the products they buy will stand up to relatively heavy loads.
2. Existing definitions 2. Existing definitions
This document uses the conceptual framework established in RFCs 1242 This document uses the conceptual framework established in RFCs 1242
and 1944 (for routers) and RFC 2285 (for switches). The router and and 2544 (for routers) and RFC 2285 (for switches). The router and
switch documents contain discussions of several terms relevant to switch documents contain discussions of several terms relevant to
benchmarking the performance of firewalls. Readers should consult the benchmarking the performance of firewalls. Readers should consult the
router and switch documents before making use of this document. router and switch documents before making use of this document.
This document uses the definition format described in RFC 1242, This document uses the definition format described in RFC 1242,
Section 2. The sections in each definition are: definition, Section 2. The sections in each definition are: definition,
discussion, measurement units (optional), issues (optional), and discussion, measurement units (optional), issues (optional), and
cross-references. cross-references.
3. Term definitions 3. Term definitions
3.1 Allowed traffic 3.1 Allowed traffic
Definition: Definition:
Packets forwarded as a result of the rule set of the device under Packets forwarded as a result of the rule set of the device under
test/system under test (DUT/SUT). test/system under test (DUT/SUT).
Discussion: Discussion:
Firewalls typically are configured to forward only those packets Firewalls typically are configured to forward only those packets
explicitly permitted in the rule set. Forwarded packets MUST be explicitly permitted in the rule set. Forwarded packets must be
included in calculating the bit forwarding rate or maximum bit included in calculating the bit forwarding rate or maximum bit
forwarding rate of the DUT/SUT. All other packets MUST NOT be forwarding rate of the DUT/SUT. All other packets must not be
included in bit forwarding rate calculations. included in bit forwarding rate calculations.
This document assumes 1:1 correspondence of allowed traffic offered This document assumes 1:1 correspondence of allowed traffic offered
to the DUT/SUT and forwarded by the DUT/SUT. There are cases where to the DUT/SUT and forwarded by the DUT/SUT. There are cases where
the DUT/SUT may forward more traffic than it is offered; for example, the DUT/SUT may forward more traffic than it is offered; for example,
the DUT/SUT may act as a mail exploder or a multicast server. Any the DUT/SUT may act as a mail exploder or a multicast server. Any
Newman Page [3] Newman Page [3]
attempt to benchmark forwarding rates of such traffic MUST include a attempt to benchmark forwarding rates of such traffic must include a
description of how much traffic the tester expects to be forwarded. description of how much traffic the tester expects to be forwarded.
Unit of measurement: Unit of measurement:
not applicable not applicable
Issues: Issues:
See also: See also:
policy policy
rule set rule set
skipping to change at line 263 skipping to change at line 263
and section 3.6.1 of RFC 2285. and section 3.6.1 of RFC 2285.
Unlike both RFCs 1242 and 2285, this definition introduces the notion Unlike both RFCs 1242 and 2285, this definition introduces the notion
of different classes of traffic: allowed, illegal, and rejected (see of different classes of traffic: allowed, illegal, and rejected (see
definitions for each term). For benchmarking purposes, it is assumed definitions for each term). For benchmarking purposes, it is assumed
that bit forwarding rate measurements include only allowed traffic. that bit forwarding rate measurements include only allowed traffic.
Unlike RFC 1242, there is no reference to lost or retransmitted data. Unlike RFC 1242, there is no reference to lost or retransmitted data.
Forwarding rate is assumed to be a goodput measurement, in that only Forwarding rate is assumed to be a goodput measurement, in that only
data successfully forwarded to the destination interface is measured. data successfully forwarded to the destination interface is measured.
Bit forwarding rate MUST be measured in relation to the offered load. Bit forwarding rate must be measured in relation to the offered load.
Bit forwarding rate MAY be measured with differed load levels, Bit forwarding rate MAY be measured with differed load levels,
traffic orientation, and traffic distribution. traffic orientation, and traffic distribution.
Unlike RFC 2285, this measurement counts bits per second rather than Unlike RFC 2285, this measurement counts bits per second rather than
frames per second. Testers interested in frame (or frame-like) frames per second. Testers interested in frame (or frame-like)
measurements should use units of transfer. measurements should use units of transfer.
Unit of measurement: Unit of measurement:
bits per second bits per second
skipping to change at line 400 skipping to change at line 400
relevant, since all firewalls perform some form of connection relevant, since all firewalls perform some form of connection
maintenance; at the very least, all check connection requests maintenance; at the very least, all check connection requests
against their rule sets. against their rule sets.
Further, note that connection is not an atomic unit of measurement in Further, note that connection is not an atomic unit of measurement in
that it does not describe the various steps involved in connection that it does not describe the various steps involved in connection
setup, maintenance, and teardown. Testers may wish to take separate setup, maintenance, and teardown. Testers may wish to take separate
measurements of each of these components. measurements of each of these components.
When benchmarking firewall performance, it's important to identify When benchmarking firewall performance, it's important to identify
the connection establishment and teardown procedures, as these MUST the connection establishment and teardown procedures, as these must
NOT be included when measuring steady-state forwarding rates. NOT be included when measuring steady-state forwarding rates.
Further, forwarding rates MUST be measured only after any security Further, forwarding rates must be measured only after any security
associations have been established. associations have been established.
Though it seems paradoxical, connectionless protocols such as UDP may Though it seems paradoxical, connectionless protocols such as UDP may
also involve connections, at least for the purposes of firewall also involve connections, at least for the purposes of firewall
performance measurement. For example, one host may send UDP packets performance measurement. For example, one host may send UDP packets
to another across a firewall. If the destination host is listening on to another across a firewall. If the destination host is listening on
the correct UDP port, it receives the UDP packets. For the purposes the correct UDP port, it receives the UDP packets. For the purposes
of firewall performance measurement, this is considered a connection. of firewall performance measurement, this is considered a connection.
Unit of measurement: Unit of measurement:
skipping to change at line 705 skipping to change at line 705
Discussion: Discussion:
Firewalls are generally insensitive to packet loss in the network. As Firewalls are generally insensitive to packet loss in the network. As
such, measurements of gross bit forwarding rates are not meaningful such, measurements of gross bit forwarding rates are not meaningful
since (in the case of proxy-based and stateful packet filtering since (in the case of proxy-based and stateful packet filtering
firewalls) a receiving endpoint directly attached to a DUT/SUT would firewalls) a receiving endpoint directly attached to a DUT/SUT would
not receive any data dropped by the DUT/SUT. not receive any data dropped by the DUT/SUT.
The type of traffic lost or retransmitted is protocol-dependent. TCP The type of traffic lost or retransmitted is protocol-dependent. TCP
and ATM, for example, request different types of retransmissions. and ATM, for example, request different types of retransmissions.
Testers MUST observe retransmitted data for the protocol in use, and Testers must observe retransmitted data for the protocol in use, and
subtract this quantity from measurements of gross bit forwarding subtract this quantity from measurements of gross bit forwarding
rate. rate.
Unit of measurement: Unit of measurement:
bits per second bits per second
Issues: Issues:
allowed vs. rejected traffic allowed vs. rejected traffic
See also: See also:
skipping to change at line 949 skipping to change at line 949
3.26 Rejected traffic 3.26 Rejected traffic
Definition: Definition:
Packets dropped as a result of the rule set of the DUT/SUT. Packets dropped as a result of the rule set of the DUT/SUT.
Newman Page [17] Newman Page [17]
Discussion: Discussion:
For purposes of benchmarking firewall performance, it is expected For purposes of benchmarking firewall performance, it is expected
that firewalls will reject all traffic not explicitly permitted in that firewalls will reject all traffic not explicitly permitted in
the rule set. Dropped packets MUST NOT be included in calculating the the rule set. Dropped packets must not be included in calculating the
bit forwarding rate or maximum bit forwarding rate of the DUT/SUT. bit forwarding rate or maximum bit forwarding rate of the DUT/SUT.
Unit of measurement: Unit of measurement:
not applicable not applicable
Issues: Issues:
See also: See also:
allowed traffic allowed traffic
illegal traffic illegal traffic
skipping to change at line 976 skipping to change at line 976
The collection of access control rules that determines which packets The collection of access control rules that determines which packets
the DUT/SUT will forward and which it will reject. the DUT/SUT will forward and which it will reject.
Discussion: Discussion:
Rule sets control access to and from the network interfaces of the Rule sets control access to and from the network interfaces of the
DUT/SUT. By definition, rule sets do not apply equally to all network DUT/SUT. By definition, rule sets do not apply equally to all network
interfaces; otherwise there would be no need for the firewall. For interfaces; otherwise there would be no need for the firewall. For
benchmarking purposes, a specific rule set is typically applied to benchmarking purposes, a specific rule set is typically applied to
each network interface in the DUT/SUT. each network interface in the DUT/SUT.
The tester MUST describe the complete contents of the rule set of The tester must describe the complete contents of the rule set of
each DUT/SUT. each DUT/SUT.
To ensure measurements reflect only traffic forwarded by the DUT/SUT, To ensure measurements reflect only traffic forwarded by the DUT/SUT,
testers are encouraged to include a rule denying all access except testers are encouraged to include a rule denying all access except
for those packets allowed by the rule set. for those packets allowed by the rule set.
Unit of measurement: Unit of measurement:
not applicable not applicable
Issues: Issues:
skipping to change at line 1012 skipping to change at line 1012
The set of security information relating to a given network The set of security information relating to a given network
connection or set of connections. connection or set of connections.
Discussion: Discussion:
This definition covers the relationship between policy and This definition covers the relationship between policy and
connections. Security associations (SAs) are typically set up during connections. Security associations (SAs) are typically set up during
connection establishment, and they may be reiterated or revoked connection establishment, and they may be reiterated or revoked
during a connection. during a connection.
For purposes of benchmarking firewall performance, measurements of For purposes of benchmarking firewall performance, measurements of
bit forwarding rate or UOTs per second MUST be taken after all bit forwarding rate or UOTs per second must be taken after all
security associations have been established. security associations have been established.
Unit of measurement: Unit of measurement:
not applicable not applicable
See also: See also:
connection connection
connection establishment connection establishment
policy policy
rule set rule set
skipping to change at line 1096 skipping to change at line 1096
Discussion: Discussion:
This metric is intended for use in describing steady-state forwarding This metric is intended for use in describing steady-state forwarding
rate of the DUT/SUT. rate of the DUT/SUT.
The unit of transfer (UOT) definition is deliberately left open to The unit of transfer (UOT) definition is deliberately left open to
interpretation, allowing the broadest possible application. Examples interpretation, allowing the broadest possible application. Examples
of UOTs include TCP segments, IP packets, Ethernet frames, and ATM of UOTs include TCP segments, IP packets, Ethernet frames, and ATM
cells. cells.
While the definition is deliberately broad, its interpretation must While the definition is deliberately broad, its interpretation must
not be. The tester MUST describe what type of UOT will be offered to not be. The tester must describe what type of UOT will be offered to
the DUT/SUT, and MUST offer these UOTs at a consistent rate. Traffic the DUT/SUT, and must offer these UOTs at a consistent rate. Traffic
measurement MUST begin after all connection establishment routines measurement must begin after all connection establishment routines
complete and before any connection completion routine begins. complete and before any connection completion routine begins.
Further, measurements MUST begin after any security associations Further, measurements must begin after any security associations
(SAs) are established and before any SA is revoked. (SAs) are established and before any SA is revoked.
Testers also MUST compare only like UOTs. It is not appropriate, for Testers also must compare only like UOTs. It is not appropriate, for
example, to compare forwarding rates by offering 1,500-byte Ethernet example, to compare forwarding rates by offering 1,500-byte Ethernet
UOTs to one DUT/SUT and 53-byte ATM cells to another. UOTs to one DUT/SUT and 53-byte ATM cells to another.
Unit of measurement: Unit of measurement:
Units of transfer Units of transfer
Units of transfer per second Units of transfer per second
Issues: Issues:
Newman Page [20] Newman Page [20]
skipping to change at line 1160 skipping to change at line 1160
Discussion: Discussion:
"User" is a problematic term in the context of firewall performance "User" is a problematic term in the context of firewall performance
testing, for several reasons. First, a user may in fact be a process testing, for several reasons. First, a user may in fact be a process
or processes requesting services through the DUT/SUT. Second, or processes requesting services through the DUT/SUT. Second,
different "user" requests may require radically different amounts of different "user" requests may require radically different amounts of
DUT/SUT resources. Third, traffic profiles vary widely from one DUT/SUT resources. Third, traffic profiles vary widely from one
organization to another, making it difficult to characterize the load organization to another, making it difficult to characterize the load
offered by a typical user. offered by a typical user.
For these reasons, testers SHOULD NOT attempt to measure DUT/SUT For these reasons, testers should not attempt to measure DUT/SUT
performance in terms of users supported. Instead, testers SHOULD performance in terms of users supported. Instead, testers should
describe performance in terms of maximum bit forwarding rate and describe performance in terms of maximum bit forwarding rate and
maximum number of connections sustained. Further, testers SHOULD use maximum number of connections sustained. Further, testers should use
the term "data source" rather than user to describe traffic the term "data source" rather than user to describe traffic
generator(s). generator(s).
Unit of measurement: Unit of measurement:
Newman Page [21] Newman Page [21]
not applicable not applicable
Issues: Issues:
skipping to change at line 1191 skipping to change at line 1191
that there is some overlap between performance and security issues. that there is some overlap between performance and security issues.
Specifically, the optimal configuration for firewall performance may Specifically, the optimal configuration for firewall performance may
not be the most secure, and vice-versa. not be the most secure, and vice-versa.
Further, certain forms of attack may degrade performance. One common Further, certain forms of attack may degrade performance. One common
form of denial-of-service (DoS) attack bombards a firewall with so form of denial-of-service (DoS) attack bombards a firewall with so
much rejected traffic that it cannot forward allowed traffic. DoS much rejected traffic that it cannot forward allowed traffic. DoS
attacks do not always involve heavy loads; by definition, DoS attacks do not always involve heavy loads; by definition, DoS
describes any state in which a firewall is offered rejected traffic describes any state in which a firewall is offered rejected traffic
that prohibits it from forwarding some or all allowed traffic. Even a that prohibits it from forwarding some or all allowed traffic. Even a
small amount of traffic--such as the recent Teardrop2 attack small amount of traffic may significantly degrade firewall
involving a few packet fragments--may significantly degrade firewall performance,
performance, or stop the firewall altogether. Further, the safeguards or stop the firewall altogether. Further, the safeguards in firewalls
in firewalls to guard against such attacks may have have a to guard against such attacks may have a significant negative impact
significant negative impact on performance. on performance.
Since the library of attacks is constantly expanding, no attempt is Since the library of attacks is constantly expanding, no attempt is
made here to define specific attacks that may affect performance. made here to define specific attacks that may affect performance.
Nonetheless, any reasonable performance benchmark should take into Nonetheless, any reasonable performance benchmark should take into
consideration safeguards against such attacks. Specifically, the same consideration safeguards against such attacks. Specifically, the same
safeguards should be in place when comparing performance of different safeguards should be in place when comparing performance of different
firewall implementations. firewall implementations.
5. References 5. References
Bradner, S., editor. "Benchmarking Terminology for Network Bradner, S., editor. "Benchmarking Terminology for Network
Interconnection Devices." RFC 1242. Interconnection Devices." RFC 1242.
Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network
Interconnect Devices." RFC 1944. Interconnect Devices." RFC 2544.
Mandeville, R. "Benchmarking Terminology for LAN Switching Devices." Mandeville, R. "Benchmarking Terminology for LAN Switching Devices."
RFC 2285. RFC 2285.
Rekhter, Y., et al. "Address Allocation for Private Internets." RFC Rekhter, Y., et al. "Address Allocation for Private Internets." RFC
1918. 1918.
Newman Page [22] Newman Page [22]
6. Acknowledgments 6. Acknowledgments
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/