draft-ietf-bmwg-secperf-02.txt   draft-ietf-bmwg-secperf-03.txt 
Network Working Group D. Newman Network Working Group D. Newman
INTERNET-DRAFT Data Communications INTERNET-DRAFT Data Communications
Expires in September 1998 H. Holzbaur, J. Hurd, and S. Platt Expires in January 1999 July 1998
National Software Testing Laboratories
Benchmarking Terminology for Firewall Performance Benchmarking Terminology for Firewall Performance
<draft-ietf-bmwg-secperf-02.txt> <draft-ietf-bmwg-secperf-03.txt>
Status of This Memo Status of This Memo
This document is an Internet-Draft. Internet-Drafts are working This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas, documents of the Internet Engineering Task Force (IETF), its areas,
and its working groups. Note that other groups may also distribute and its working groups. Note that other groups may also distribute
working documents as Internet-Drafts. working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
To view the entire list of current Internet-Drafts, please check the To view the entire list of current Internet-Drafts, please check the
"1id-abstracts.txt" listing contained in the Internet-Drafts Shadow "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow
Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe), Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or
ftp.isi.edu (US West Coast). ftp.isi.edu (US West Coast).
1. Introduction ....................................................2 1. Introduction ....................................................2
2. Existing definitions ............................................3 2. Existing definitions ............................................3
3. Term definitions ................................................3 3. Term definitions ................................................3
3.1 Allowed traffic ..............................................3 3.1 Allowed traffic ..............................................3
3.2 Authentication ...............................................3
3.3 Connection ...................................................4
3.4 Data source ..................................................5
3.5 Demilitarized zone (DMZ) .....................................5
3.6 Dynamic proxy ................................................5
3.7 Firewall .....................................................6
3.8 Forwarding rate ..............................................6
3.9 Goodput ......................................................7
3.10 Homed .......................................................7
3.11 Logging .....................................................8
3.12 Network address translation (NAT) ...........................8
3.13 Packet filtering ............................................9
3.14 Perimeter network ...........................................9
3.15 Policy .....................................................10
3.16 Protected network ..........................................10
3.17 Proxy ......................................................11
3.18 Rejected traffic ...........................................11
3.19 Rule set ...................................................11
3.20 Session ....................................................12
3.21 Stateful packet filtering ..................................13
3.22 Tri-homed ..................................................13
3.23 Unprotected network ........................................14
3.24 User .......................................................14
4. Security considerations ...........................................15
5. References ........................................................15
6. Acknowledgments ...................................................15 3.2 Application proxy.............................................3
7. Contact information ...............................................16
3.3 Authentication ...............................................4
3.4 Bit forwarding rate ..........................................4
3.5 Circuit proxy ................................................5
3.6 Concurrent connections .......................................6
3.7 Connection ...................................................6
3.8 Connection establishment rate ................................7
3.9 Conection overhead ...........................................8
3.10 Data source .................................................8
3.11 Demilitarized zone (DMZ) ....................................9
3.12 Firewall ....................................................9
3.13 Goodput ....................................................10
3.14 Homed ......................................................10
3.15 Illegal traffic.............................................11
3.16 Logging ....................................................11
3.17 Network address translation (NAT) ..........................12
3.18 Packet filtering ...........................................12
3.19 Perimeter network ..........................................13
3.20 Policy .....................................................13
3.21 Protected network ..........................................13
3.22 Proxy ......................................................14
3.23 Rejected traffic ...........................................14
3.24 Rule set ...................................................15
3.25 Stateful packet filtering ..................................15
3.26 Tri-homed ..................................................16
3.27 Unprotected network ........................................16
3.28 User .......................................................17
4. Security considerations ...........................................17
5. References ........................................................18
6. Acknowledgments ...................................................18
7. Contact information ...............................................19
1. Introduction 1. Introduction
This document defines terms used in measuring the performance of This document defines terms used in measuring the performance of
firewalls. It extends the terminology already used for benchmarking firewalls. It extends the terminology already used for benchmarking
routers and switches and adds terminology specific to firewalls. The routers and switches and adds terminology specific to firewalls. The
primary metrics defined in this document are maximum forwarding rate primary metrics used in this document are connections and bit
and maximum number of connections. forwarding rate.
Why are firewall performance measurements needed? First, despite the There are several reasons why are firewall performance measurements
rapid rise in firewall deployment, there is no standard means of are needed. First, despite the rapid rise in firewall deployment,
performance measurement. Second, implementations vary widely, making there is no standard means of performance measurement. Second,
it difficult to do direct performance comparisons. Finally, more and implementations vary widely, making it difficult to do direct
more organizations are deploying firewalls on internal networks performance comparisons. Finally, more and more organizations are
operating at relatively high speeds, while most firewall deploying firewalls on internal networks operating at relatively high
implementations remain optimized for use over low-speed wide-area speeds, while most firewall implementations remain optimized for use
connections. As a result, users are often unsure whether the products over low-speed wide-area connections. As a result, users are often
they buy will stand up to relatively heavy loads.
We may also create additional terminology and methodology documents Newman Page [2]
to define other types of network security products such as virtual unsure whether the products they buy will stand up to relatively
private network (VPN) and encryption devices. This document, however, heavy loads.
focuses solely on firewall terminology.
2. Existing definitions 2. Existing definitions
This document uses the conceptual framework established in RFCs 1242 This document uses the conceptual framework established in RFCs 1242
and 1944 (for routers) and RFC 2285 (for switches). The router and and 1944 (for routers) and RFC 2285 (for switches). The router and
switch documents contain discussions of several terms relevant to switch documents contain discussions of several terms relevant to
benchmarking the performance of firewalls. Readers should consult the benchmarking the performance of firewalls. Readers should consult the
router and switch documents before making use of this document. router and switch documents before making use of this document.
This document uses the definition format described in RFC 1242, This document uses the definition format described in RFC 1242,
Section 2. The sections in each definition are: definition, Section 2. The sections in each definition are: definition,
discussion, measurement units (optional), issues (optional), and discussion, measurement units (optional), issues (optional), and
cross-references. cross-references.
3. Term definitions 3. Term definitions
3.1 Allowed traffic 3.1 Allowed traffic
Definition: Definition:
Packets forwarded as a result of the rule set of the DUT/SUT. Packets forwarded as a result of the rule set of the device under
test/system under test (DUT/SUT).
Discussion: Discussion:
Firewalls typically are configured to forward only those packets Firewalls typically are configured to forward only those packets
explicitly permitted in the rule set. Forwarded packets MUST be included explicitly permitted in the rule set. Forwarded packets MUST be included
in calculating the forwarding rate or maximum forwarding rate of the in calculating the bit forwarding rate or maximum bit forwarding rate of
DUT/SUT. All other packets MUST NOT be included in forwarding rate the DUT/SUT. All other packets MUST NOT be included in bit forwarding
calculations. rate calculations.
Measurement units: Measurement units:
not applicable not applicable
Newman et al. Page [2]
Issues: Issues:
See also: See also:
policy policy
rule set rule set
3.2 Authentication 3.2 Application proxy
Definition:
A proxy service that is set up and torn down in response to a client
request, rather than existing on a static basis.
Discussion:
Circuit proxies always forward packets containing a given port number if
that port number is permitted by the rule set. Application proxies, in
contrast, forward packets only once a connection has been established
using some known protocol. When the connection closes, a firewall using
dynamic proxies rejects individual packets, even if they contain port
numbers allowed by a rule set.
Measurement units:
Newman Page [3]
not applicable
Issues:
circuit proxy
rule sets
See also:
allowed traffic
circuit proxy
proxy
rejected traffic
rule set
3.3 Authentication
Definition: Definition:
The process of verifying that a user requesting a network resource is The process of verifying that a user requesting a network resource is
who he, she, or it claims to be, and vice versa. who he, she, or it claims to be, and vice versa.
Discussion: Discussion:
Trust is a critical concept in network security. Any network resource Trust is a critical concept in network security. Any network resource
(such as a file server or printer) with restricted access MUST require (such as a file server or printer) with restricted access MUST require
authentication before granting access. authentication before granting access.
skipping to change at line 157 skipping to change at line 219
introduce additional latency to the overall SUT. introduce additional latency to the overall SUT.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
user user
3.3 Connection 3.4 Bit forwarding rate
Definition: Definition:
A logical path established between two hosts, or between a host and the The number of bits per second of allowed traffic a DUT/SUT can be
DUT/SUT.
Newman Page [4]
observed to transmit to the correct destination interface(s) in response
to a specified offered load.
Discussion:
This definition differs substantially from section 3.17 of RFC 1242 and
section 3.6.1 of RFC 2285.
Unlike both RFCs 1242 and 2285, this definition introduces the notion of
different classes of traffic: allowed, illegal, and rejected (see
definitions for each term). Any bit forwarding rate measurement MUST
include only allowed traffic.
Unlike RFC 1242, there is no reference to lost or retransmitted data.
Forwarding rate is assumed to be a goodput measurement, in that only
data successfully forwarded to the destination interface is measured.
Bit forwarding rate MUST be measured in relation to the offered load.
Bit forwarding rate MAY be measured with differed load levels, traffic
orientation, and traffic distribution.
Unlike RFC 2285, this measurement counts bits per second rather than
frames per second. Per-frame metrics are not meaningful in the context
of a flow of application data between endpoints.
Units of measurement:
bits per second
Issues:
Allowed traffic vs. rejected traffic
See also:
allowed traffic
goodput
illegal traffic
rejected traffic
3.5 Circuit proxy
Definition:
A proxy service that statically defines which traffic will be forwarded
using a criterion like port number.
Discussion:
The key distinction with circuit proxies is that they are static and
thus will always set up a connection if the DUT/SUT's rule set allows
it. For example, if a firewall's rule set permits ftp connections, a
circuit proxy will forward traffic on TCP port 20 (ftp default data)
even if no control connection was first established on TCP port 21 (ftp
control).
Measurement units:
not applicable
Issues:
application proxy
Newman Page [5]
rule sets
See also:
allowed traffic
application proxy
proxy
rejected traffic
rule set
3.6 Concurrent connections
Definition:
The aggregate number of simultaneous connections between hosts across
the DUT/SUT, or between hosts and the DUT/SUT.
Discussion: Discussion:
The number of concurrent connections a firewall can support is just as The number of concurrent connections a firewall can support is just as
important a metric for some users as maximum forwarding rate. important a metric for some users as maximum bit forwarding rate.
Connections MAY be TCP sessions, but they don't have to be. Users of While "connection" describes only a state and not necessarily the
transfer of data, concurrency assumes that all existing connections are
in fact capable of transferring data. If a data cannot be sent over a
connection, that connection should not be counted toward the number of
concurrent connections.
Newman et al. Page [3] Measurement units:
Concurrent connections
Maximum number of concurrent connections
other connection-oriented protocols such as ATM may wish to use other Issues:
definitions of a connection, either instead of or in addition to TCP
connections. See also:
connections
connection establishment rate
connection overhead
3.7 Connection
Definition:
A state in which two hosts, or a host and the DUT/SUT, agree to exchange
data using a known protocol.
Discussion:
A connection is an abstraction describing an agreement between two
nodes: One agrees to send data and the other agrees to receive it.
Connections MAY be TCP sessions, but they don't have to be. Other
connection-oriented protocols such as ATM also may be used, either
instead of or in addition to TCP connections.
What constitutes a connection depends on the application. For a "native What constitutes a connection depends on the application. For a "native
ATM" application like a video stream, connections and VCs can be ATM" application like a video stream, connections and virtual circuits
synonymous. For TCP/IP applications on ATM networks (where multiple TCP can be synonymous. For TCP/IP applications on ATM networks (where
sockets may ride over a single ATM virtual circuit), TCP sockets and multiple TCP sessions may ride over a single ATM virtual circuit), the
connections are synonymous.
Newman Page [6]
number of TCP connections is probably the most important consideration.
Additionally, in some cases firewalls may handle a mixture of native TCP Additionally, in some cases firewalls may handle a mixture of native TCP
and native ATM connections. In this situation, the wrappers around user and native ATM connections. In this situation, the wrappers around user
data will differ. The most meaningful metric describes what an end-user data will differ. The most meaningful metric describes what an end-user
will see. will see.
Data connections describe state, not data transfer. The existence of a Data connections describe state, not data transfer. The existence of a
connection does NOT imply that data travels on that connection at any connection does not imply that data travels on that connection at any
given time. given time, although if data cannot be forwarded on a previously
established connection that connection should not be considered in any
aggregrate connection count (see concurrent connections).
A firewall's architecture dictates where a connection is terminated. In A firewall's architecture dictates where a connection is terminated. In
the case of proxy-based systems, a connection by definition terminates the case of proxy-based systems, a connection terminates at the DUT/SUT.
at the DUT/SUT. But firewalls using packet filtering or stateful packet But firewalls using packet filtering or stateful packet filtering
filtering designs act only as passthrough devices, in that they reside designs act only as passthrough devices, in that they reside between two
between two connection endpoints. Regardless of firewall architecture, connection endpoints. Regardless of firewall architecture, the number of
the number of data connections is still relevant, since all firewalls data connections is still relevant, since all firewalls perform some
perform some form of connection maintenance; at the very least, all form of connection maintenance; at the very least, all check connection
check connection requests against their rule sets. requests against their rule sets.
Though it seems paradoxical, connectionless protocols such as UDP may
also involve connections, at least for the purposes of firewall
performance measurement. For example, one host may send UDP packets to
another across a firewall. If the destination host is listening on the
correct UDP port, it receives the UDP packets. For the purposes of
firewall performance measurement, this is considered a connection.
Indeed, some firewall implementations dynamically alter their rule sets
to allow such connections.
Measurement units: Measurement units:
Maximum number of connections Connection establishment rate
Concurrent connections
Maximum number of concurrent connections
Issues: Issues:
proxy-based vs. stateful packet filtering proxy-based vs. stateful packet filtering
TCP/IP vs. ATM TCP/IP vs. ATM
connection-oriented vs. connectionless
See also: See also:
data source data source
session concurrent connections
connection establishment rate
3.4 Data source 3.8 Connection establishment rate
Definition:
The length of time needed for two hosts, or a host and the DUT/SUT, to
agree to set up a data exchange using a known protocol.
Discussion:
Each connection-oriented protocol has its own defined mechanisms for
Newman Page [7]
setting up a connection. For purposes of benchmarking firewall
performance, this shall be the interval between receipt of the first
octet of the packet carrying a connection establishment request on a
DUT/SUT interface until transmission of the last octet of the last
packet of the connection setup traffic headed in the opposite direction.
This definition applies only to connection-oriented protocols such as
TCP. For connectionless protocols such as UDP, the notion of connection
setup time is not meaningful.
Metric
Connection establishment rate
Issues:
See also:
concurrent connections
connection
connection overhead
3.9 Connection overhead
Definition:
The degradation in bit forwarding rate, if any, observed as a result of
the addition of one connection between two hosts through the DUT/SUT, or
the addition of one connection from a host to the DUT/SUT.
Discussion:
The memory cost of connection establishment and maintenance is highly
implementation-specific. This metric is intended to describe that cost
in a method visible outside the firewall.
It may also be desirable to invert this metric to show the performance
improvement as a result of tearing down one connection.
Measurement units:
bit forwarding rate
Issues:
3.10 Data source
Definition: Definition:
A station capable of generating traffic to the DUT/SUT. A station capable of generating traffic to the DUT/SUT.
Discussion: Discussion:
One data source MAY emulate multiple users or stations. In addition, one One data source MAY emulate multiple users or stations. In addition, one
data source MAY offer traffic to multiple network interfaces on the data source MAY offer traffic to multiple network interfaces on the
DUT/SUT. DUT/SUT.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
The term "data source" is deliberately independent of any number of
Newman et al. Page [4] Newman Page [8]
The term "data source" is deliberately independent of any number of
users. It is useful to think of data sources simply as traffic users. It is useful to think of data sources simply as traffic
generators, without any correlation to any given number of users. generators, without any correlation to any given number of users.
See also: See also:
connection connection
3.5 Demilitarized zone (DMZ) 3.11 Demilitarized zone (DMZ)
Definition: Definition:
A network segment or segments located between protected and unprotected A network segment or segments located between protected and unprotected
networks. DMZ networks are sometimes called perimeter networks. networks. DMZ networks are sometimes called perimeter networks.
Discussion: Discussion:
As an extra security measure, networks are often designed such that As an extra security measure, networks are often designed such that
protected and unprotected segments are never directly connected. protected and unprotected segments are never directly connected.
Instead, firewalls (and possibly public resources such as WWW or FTP Instead, firewalls (and possibly public resources such as WWW or FTP
servers) often reside on the so-called DMZ network. To connect servers) often reside on the so-called DMZ network. To connect
skipping to change at line 263 skipping to change at line 490
not applicable not applicable
Issues: Issues:
Homed Homed
See also: See also:
unprotected network unprotected network
perimeter network perimeter network
protected network protected network
3.6 Dynamic proxy 3.12 Firewall
Definition:
A proxy service that is set up and torn down in response to a client
request, rather than existing on a static basis.
Discussion:
Proxy services typically "listen" on a given TCP port number for client
requests. With static proxies, a firewall always forwards packets
containing a given TCP port number if that port number is permitted by
the rule set. Dynamic proxies, in contrast, forward TCP packets only
once an authenticated connection has been established. When the
connection closes, a firewall using dynamic proxies rejects individual
packets, even if they contain port numbers allowed by a rule set.
Measurement units:
not applicable
Issues:
Newman et al. Page [5]
rule sets
See also:
allowed traffic
proxy
rejected traffic
rule set
3.7 Firewall
Definition: Definition:
A device or group of devices that enforces an access control policy A device or group of devices that enforces an access control policy
between networks. between networks.
Discussion: Discussion:
While there are many different ways to accomplish it, all firewalls do While there are many different ways to accomplish it, all firewalls do
the same thing: control access between networks. the same thing: control access between networks.
The most common configuration involves a firewall connecting two The most common configuration involves a firewall connecting two
segments (one protected and one unprotected), but this is not the only segments (one protected and one unprotected), but this is not the only
possible configuration. Many firewalls support tri-homing, allowing use possible configuration. Many firewalls support tri-homing, allowing use
of a DMZ network. It is possible for a firewall to accommodate more than of a DMZ network. It is possible for a firewall to accommodate more than
three interfaces, each attached to a different network segment. three interfaces, each attached to a different network segment.
The criteria by which access is controlled is deliberately not specified The criteria by which access is controlled is deliberately not specified
here. Typically this has been done using network- or transport-layer here. Typically this has been done using network- or transport-layer
Newman Page [9]
criteria (such as IP subnet or TCP port number), but there is no reason criteria (such as IP subnet or TCP port number), but there is no reason
this must always be so. A growing number of firewalls are controlling this must always be so. A growing number of firewalls are controlling
access at the application layer, using user identification as the access at the application layer, using user identification as the
criterion. And firewalls for ATM networks may control access based on criterion. And firewalls for ATM networks may control access based on
data link-layer criteria. data link-layer criteria.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
DMZ DMZ
tri-homed tri-homed
user user
3.8 Forwarding rate 3.13 Goodput
Definition:
The number of bits per second that a firewall can be observed to
transmit successfully to the correct destination interface in response
to a specified offered load.
Discussion:
This definition differs substantially from section 3.17 of RFC 1242 and
section 3.6.1 of RFC 2285. Unlike RFC 1242, there is no reference to
lost or retransmitted data. Forwarding rate is assumed to be a goodput
measurement, in that only data successfully forwarded to the destination
Newman et al. Page [6]
interface is measured. Forwarding rate MUST be measured in relation to
the offered load. Forwarding rate MAY be measured with differed load
levels, traffic orientation, and traffic distribution.
Unlike RFC 2285, this measurement counts bits per second rather than
frames per second. Per-frame metrics are not meaningful in the context
of a flow of application data between endpoints.
Units of measurement:
bits per second
Issues:
Allowed traffic vs. rejected traffic
See also:
allowed traffic
goodput
rejected traffic
3.9 Goodput
Definition: Definition:
The number of bits per unit of time forwarded to the correct destination The number of bits per unit of time forwarded to the correct destination
interface of the DUT/SUT, minus any bits lost or retransmitted. interface of the DUT/SUT, minus any bits lost or retransmitted.
Discussion: Discussion:
Firewalls are generally insensitive to packet loss in the network. As Firewalls are generally insensitive to packet loss in the network. As
such, measurements of gross forwarding rates are not meaningful since such, measurements of gross bit forwarding rates are not meaningful
(in the case of proxy-based and stateful packet filtering firewalls) a since (in the case of proxy-based and stateful packet filtering
receiving endpoint directly attached to a DUT/SUT would not receive any firewalls) a receiving endpoint directly attached to a DUT/SUT would not
data dropped by the DUT/SUT. receive any data dropped by the DUT/SUT.
The type of traffic lost or retransmitted is protocol-dependent. TCP and The type of traffic lost or retransmitted is protocol-dependent. TCP and
ATM, for example, request different types of retransmissions. Testers ATM, for example, request different types of retransmissions. Testers
MUST observe retransmitted data for the protocol in use, and subtract MUST observe retransmitted data for the protocol in use, and subtract
this quantity from measurements of gross forwarding rate. this quantity from measurements of gross bit forwarding rate.
Unit of measurement: Unit of measurement:
bits per second bits per second
Issues: Issues:
allowed vs. rejected traffic allowed vs. rejected traffic
See also: See also:
allowed traffic allowed traffic
forwarding rate bit forwarding rate
rejected traffic rejected traffic
3.10 Homed 3.14 Homed
Definition: Definition:
The number of logical interfaces a DUT/SUT contains. The number of logical interfaces a DUT/SUT contains.
Discussion: Discussion:
Newman et al. Page [7]
Firewalls MUST contain at least two logical interfaces. In network Firewalls MUST contain at least two logical interfaces. In network
topologies where a DMZ is used, the firewall contains at least three topologies where a DMZ is used, the firewall contains at least three
interfaces and is said to be tri-homed. Additional interfaces would make interfaces and is said to be tri-homed. Additional interfaces would make
Newman Page [10]
a firewall quad-homed, quint-homed, and so on. a firewall quad-homed, quint-homed, and so on.
It is theoretically possible for a firewall to contain one physical It is theoretically possible for a firewall to contain one physical
interface and multiple logical interfaces. This configuration is interface and multiple logical interfaces. This configuration is
strongly discouraged for testing purposes because of the difficulty in strongly discouraged for testing purposes because of the difficulty in
verifying that no leakage occurs between protected and unprotected verifying that no leakage occurs between protected and unprotected
segments. segments.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
tri-homed tri-homed
3.11 Logging 3.15 Illegal traffic
Definition:
Packets specified for rejection in the rule set of the DUT/SUT.
Discussion:
A buggy or misconfigured firewall may forward packets even though its
rule set specifies that these packets be dropped. Illegal traffic
differs from rejected traffic in that it describes all traffic specified
for rejection by the rule set, while rejected traffic specifies only
those packets actually dropped by the DUT/SUT.
Measurement units:
not applicable
Issues:
See also:
accepted traffic
policy
rejected traffic
rule set
3.16 Logging
Definition: Definition:
The recording of user requests made to the firewall. The recording of user requests made to the firewall.
Discussion: Discussion:
Firewalls SHOULD log all requests they handle, both allowed and Firewalls MUST log all requests they handle, both allowed and rejected.
rejected. For many firewall designs, logging requires a significant For many firewall designs, logging requires a significant amount of
amount of processing overhead, especially when complex rule sets are in processing overhead, especially when complex rule sets are in use.
use.
The type and amount of data logged varies by implementation. Testers The type and amount of data logged varies by implementation. Testers
SHOULD attempt to log equivalent data when comparing different DUT/SUTs. SHOULD attempt to log equivalent data when comparing different DUT/SUTs.
Logging MAY take place on systems other than the DUT/SUT. Logging MAY take place on systems other than the DUT/SUT.
Newman Page [11]
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
rule sets rule sets
See also: See also:
allowed traffic allowed traffic
connection connection
rejected traffic rejected traffic
session session
3.12 Network address translation (NAT) 3.17 Network address translation (NAT)
Definition: Definition:
A method of mapping one or more private, reserved IP addresses to one or A method of mapping one or more private, reserved IP addresses to one or
more public IP addresses. more public IP addresses.
Discussion: Discussion:
Newman et al. Page [8]
In the interest of conserving the IPv4 address space, RFC 1918 proposed In the interest of conserving the IPv4 address space, RFC 1918 proposed
the use of certain private (reserved) blocks of IP addresses. the use of certain private (reserved) blocks of IP addresses.
Connections to public networks are made by use of a device that Connections to public networks are made by use of a device that
translates one or more RFC 1918 addresses to one or more public translates one or more RFC 1918 addresses to one or more public
addresses--a network address translator (NAT). addresses--a network address translator (NAT).
The use of private addressing also introduces a security benefit in that The use of private addressing also introduces a security benefit in that
RFC 1918 addresses are not visible to hosts on the public Internet. RFC 1918 addresses are not visible to hosts on the public Internet.
Some NAT implementations are computationally intensive, and may affect Some NAT implementations are computationally intensive, and may affect
forwarding rate. bit forwarding rate.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
3.13 Packet filtering 3.18 Packet filtering
Definition: Definition:
The process of controlling access by examining packets based on packet The process of controlling access by examining packets based on packet
header content. header content.
Discussion: Discussion:
Packet-filtering devices forward or deny packets based on information in Packet-filtering devices forward or deny packets based on information in
each packet's header, such as IP address or TCP port number. A packet- each packet's header, such as IP address or TCP port number. A packet-
filtering firewall uses a rule set to determine which traffic should be filtering firewall uses a rule set to determine which traffic should be
forwarded and which should be blocked. forwarded and which should be blocked.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
Newman Page [12]
static versus stateful packet filtering static versus stateful packet filtering
See also: See also:
dynamic proxy application proxy
circuit proxy
proxy proxy
rule set rule set
stateful packet filtering stateful packet filtering
3.14 Perimeter network 3.19 Perimeter network
Definition: Definition:
A network segment or segments located between protected and unprotected A network segment or segments located between protected and unprotected
networks. Perimeter networks are often called DMZ networks. networks. Perimeter networks are often called DMZ networks.
Discussion: Discussion:
See the definition of DMZ for a discussion. See the definition of DMZ for a discussion.
Measurement units: Measurement units:
not applicable not applicable
Newman et al. Page [9]
Issues: Issues:
Tri-homed Tri-homed
See also: See also:
demilitarized zone (DMZ) demilitarized zone (DMZ)
unprotected network unprotected network
protected network protected network
3.15 Policy 3.20 Policy
Definition: Definition:
A document defining acceptable access to protected, DMZ, and unprotected A document defining acceptable access to protected, DMZ, and unprotected
networks. networks.
Discussion: Discussion:
Security policies generally do not spell out specific configurations for Security policies generally do not spell out specific configurations for
firewalls; rather, they set general guidelines for what is and is not firewalls; rather, they set general guidelines for what is and is not
acceptable network access. acceptable network access.
skipping to change at line 543 skipping to change at line 732
implemented in the DUT/SUT. implemented in the DUT/SUT.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
rule set rule set
3.16 Protected network 3.21 Protected network
Definition: Definition:
Newman Page [13]
A network segment or segments to which access is controlled by the A network segment or segments to which access is controlled by the
DUT/SUT. DUT/SUT.
Discussion: Discussion:
Firewalls are intended to prevent unauthorized access either to or from Firewalls are intended to prevent unauthorized access either to or from
the protected network. Depending on the configuration specified by the the protected network. Depending on the configuration specified by the
policy and rule set, the DUT/SUT may allow stations on the protected policy and rule set, the DUT/SUT may allow stations on the protected
segment to act as clients for servers on either the DMZ or the segment to act as clients for servers on either the DMZ or the
unprotected network, or both. unprotected network, or both.
Protected networks are often called "internal networks." That term is Protected networks are often called "internal networks." That term is
not used here because firewalls increasingly are deployed within an not used here because firewalls increasingly are deployed within an
organization, where all segments are by definition internal. organization, where all segments are by definition internal.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
Newman et al. Page [10]
demilitarized zone (DMZ) demilitarized zone (DMZ)
unprotected network unprotected network
policy policy
rule set rule set
unprotected network unprotected network
3.17 Proxy 3.22 Proxy
Definition: Definition:
A request for a connection made on behalf of a host. A request for a connection made on behalf of a host.
Discussion: Discussion:
Proxy-based firewalls do not allow direct connections between hosts. Proxy-based firewalls do not allow direct connections between hosts.
Instead, two connections are established: one between the client host Instead, two connections are established: one between the client host
and the DUT/SUT, and another between the DUT/SUT and server host. and the DUT/SUT, and another between the DUT/SUT and server host.
As with packet-filtering firewalls, proxy-based devices use a rule set As with packet-filtering firewalls, proxy-based devices use a rule set
to determine which traffic should be forwarded and which should be to determine which traffic should be forwarded and which should be
rejected. rejected.
Proxies are generally application-specific. There are two types of proxies: application proxies and circuit proxies.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
application application
See also: See also:
dynamic proxy application proxy
circuit proxy
packet filtering packet filtering
stateful packet filtering stateful packet filtering
3.18 Rejected traffic Newman Page [14]
3.23 Rejected traffic
Definition: Definition:
Packets dropped as a result of the rule set of the DUT/SUT. Packets dropped as a result of the rule set of the DUT/SUT.
Discussion: Discussion:
Firewalls MUST reject any traffic not explicitly permitted in the rule Firewalls MUST reject any traffic not explicitly permitted in the rule
set. Dropped packets MUST NOT be included in calculating the forwarding set. Dropped packets MUST NOT be included in calculating the bit
rate or maximum forwarding rate of the DUT/SUT. forwarding rate or maximum bit forwarding rate of the DUT/SUT.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
policy policy
rule set rule set
3.19 Rule set 3.24 Rule set
Newman et al. Page [11]
Definition: Definition:
The collection of access control rules that determines which packets the The collection of access control rules that determines which packets the
DUT/SUT will forward and which it will reject. DUT/SUT will forward and which it will reject.
Discussion: Discussion:
Rule sets control access to and from the network interfaces of the Rule sets control access to and from the network interfaces of the
DUT/SUT. By definition, rule sets MUST NOT apply equally to all network DUT/SUT. By definition, rule sets MUST NOT apply equally to all network
interfaces; otherwise there would be no need for the firewall. interfaces; otherwise there would be no need for the firewall.
Therefore, a specific rule set MUST be applied to each network interface Therefore, a specific rule set MUST be applied to each network interface
skipping to change at line 657 skipping to change at line 847
Issues: Issues:
See also: See also:
demilitarized zone (DMZ) demilitarized zone (DMZ)
policy policy
protected network protected network
rejected traffic rejected traffic
unprotected network unprotected network
3.20 Session 3.25 Stateful packet filtering
Definition:
Data flowing through a previously established connection established
between two stations using a known protocol.
Discussion:
Because of the application-layer focus of many firewalls, sessions are a
more useful metric than the packet-based measurements used in
benchmarking routers and switches. Although firewall rule sets generally
work on a per-packet basis, it is ultimately sessions that a firewall
must handle. For example, the number of file transfer protocol (ftp)
sessions a DUT/SUT can handle concurrently is a more meaningful
measurement in benchmarking performance than the number of ftp "open"
packets it can reject. Further, a stateful packet filtering firewall
will not forward individual packets if those packets' headers conflict
with state information maintained by the firewall.
For purposes of this document, a session MUST be established using a
known protocol such as TCP. A traffic pattern is not considered a
session until it successfully completes the establishment procedures
defined by that protocol.
Newman et al. Page [12]
Also for purposes of this document, a session constitutes the logical
connection between two end-stations and not the intermediate connections
that proxy-based firewalls may use.
Issues:
TCP/IP vs. ATM
See also:
connection
policy
proxy
rule set
stateful packet filtering
3.21 Stateful packet filtering Newman Page [15]
Definition: Definition:
The process of forwarding or rejecting traffic based on the contents of The process of forwarding or rejecting traffic based on the contents of
a state table maintained by a firewall. a state table maintained by a firewall.
Discussion: Discussion:
Packet filtering and proxy firewalls are essentially static, in that Packet filtering and proxy firewalls are essentially static, in that
they always forward or reject packets based on the contents of the rule they always forward or reject packets based on the contents of the rule
set. set.
In contrast, devices using stateful packet filtering will only forward In contrast, devices using stateful packet filtering will only forward
packets if they correspond with state information maintained by the packets if they correspond with state information maintained by the
device about each session. For example, a stateful packet filtering device about each session. For example, a stateful packet filtering
device will reject a packet on port 20 (ftp-data) if no session has been device will reject a packet on port 20 (ftp-data) if no session has been
established over the ftp control port (usually port 21). established over the ftp control port (usually port 21).
Measurement units: Measurement units:
not applicable not applicable
skipping to change at line 724 skipping to change at line 876
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
dynamic proxy dynamic proxy
packet filter packet filter
proxy proxy
3.22 Tri-homed 3.26 Tri-homed
Definition: Definition:
A firewall with three network interfaces. A firewall with three network interfaces.
Discussion: Discussion:
Tri-homed firewalls connect three network segments with different Tri-homed firewalls connect three network segments with different
network addresses. Typically, these would be protected, DMZ, and network addresses. Typically, these would be protected, DMZ, and
unprotected segments. unprotected segments.
A tri-homed firewall may offer some security advantages over firewalls A tri-homed firewall may offer some security advantages over firewalls
with two interfaces. An attacker on an unprotected network may with two interfaces. An attacker on an unprotected network may
Newman et al. Page [13]
compromise hosts on the DMZ but still not reach any hosts on the compromise hosts on the DMZ but still not reach any hosts on the
protected network. protected network.
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
Usually the differentiator between one segment and another is its IP Usually the differentiator between one segment and another is its IP
address. However, firewalls may connect different networks of other address. However, firewalls may connect different networks of other
types, such as ATM or Netware segments. types, such as ATM or Netware segments.
See also: See also:
homed homed
3.23 Unprotected network 3.27 Unprotected network
Newman Page [16]
Definition: Definition:
A network segment or segments to which access is not controlled by the A network segment or segments to which access is not controlled by the
DUT/SUT. DUT/SUT.
Discussion: Discussion:
Firewalls are deployed between protected and unprotected segments. The Firewalls are deployed between protected and unprotected segments. The
unprotected network is not protected by the DUT/SUT. unprotected network is not protected by the DUT/SUT.
Note that a DUT/SUT's policy MAY specify hosts on an unprotected Note that a DUT/SUT's policy MAY specify hosts on an unprotected
skipping to change at line 779 skipping to change at line 930
not applicable not applicable
Issues: Issues:
See also: See also:
demilitarized zone (DMZ) demilitarized zone (DMZ)
policy policy
protected network protected network
rule set rule set
3.24 User 3.28 User
Definition: Definition:
A person or process requesting access to resources protected by the A person or process requesting access to resources protected by the
DUT/SUT. DUT/SUT.
Discussion: Discussion:
"User" is a problematic term in the context of firewall performance "User" is a problematic term in the context of firewall performance
testing, for several reasons. First, a user may in fact be a process or testing, for several reasons. First, a user may in fact be a process or
processes requesting services through the DUT/SUT. Second, different processes requesting services through the DUT/SUT. Second, different
"user" requests may require radically different amounts of DUT/SUT "user" requests may require radically different amounts of DUT/SUT
resources. Third, traffic profiles vary widely from one organization to resources. Third, traffic profiles vary widely from one organization to
another, making it difficult to characterize the load offered by a another, making it difficult to characterize the load offered by a
typical user. typical user.
Newman et al. Page [14]
For these reasons, we prefer not to measure DUT/SUT performance in terms For these reasons, we prefer not to measure DUT/SUT performance in terms
of users supported. Instead, we describe performance in terms of maximum of users supported. Instead, we describe performance in terms of maximum
forwarding rate and maximum number of sessions sustained. Further, we bit forwarding rate and maximum number of sessions sustained. Further,
use the term "data source" rather than user to describe the traffic we use the term "data source" rather than user to describe the traffic
generator(s). generator(s).
Measurement units: Measurement units:
not applicable not applicable
Issues: Issues:
See also: See also:
data source data source
4. Security considerations 4. Security considerations
The primary goal of this memo is to describe terms used in measuring Newman Page [17]
The primary goal of this memo is to describe terms used in benchmarking
firewall performance. However, readers should be aware that there is firewall performance. However, readers should be aware that there is
some overlap between performance and security issues. Readers should be some overlap between performance and security issues. Specifically, the
aware that the optimal configuration for firewall performance may not be optimal configuration for firewall performance may not be the most
the most secure, and vice-versa. secure, and vice-versa.
Further, certain forms of attack may degrade performance. One common Further, certain forms of attack may degrade performance. One common
form of denial-of-service (DoS) attack bombards a firewall with so much form of denial-of-service (DoS) attack bombards a firewall with so much
rejected traffic that it cannot forward allowed traffic. DoS attacks do rejected traffic that it cannot forward allowed traffic. DoS attacks do
not always involve heavy loads; by definition, DoS describes any state not always involve heavy loads; by definition, DoS describes any state
in which a firewall is offered rejected traffic that prohibits it from in which a firewall is offered rejected traffic that prohibits it from
forwarding some or all allowed traffic. Even a small amount of traffic-- forwarding some or all allowed traffic. Even a small amount of traffic--
such as the recent Teardrop2 attack involving a few packet fragments-- such as the recent Teardrop2 attack involving a few packet fragments--
may significantly degrade firewall performance, or stop the firewall may significantly degrade firewall performance, or stop the firewall
altogether. altogether. Further, the safeguards in firewalls to guard against such
attacks may have have a significant negative impact on performance.
Since the library of attacks is constantly expanding, no attempt is made
here to define specific attacks that may affect performance.
Nonetheless, any reasonable performance benchmark must take safeguards
against such attacks into consideration. Specifically, the same
safeguards must be in place when comparing performance of different
firewall implementations.
5. References 5. References
Bradner, S., editor. "Benchmarking Terminology for Network Bradner, S., editor. "Benchmarking Terminology for Network
Interconnection Devices." RFC 1242. Interconnection Devices." RFC 1242.
Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network
Interconnect Devices." RFC 1944. Interconnect Devices." RFC 1944.
Mandeville, R. "Benchmarking Terminology for LAN Switching Devices." RFC Mandeville, R. "Benchmarking Terminology for LAN Switching Devices." RFC
2285. 2285.
Rekhter, Y., et al. "Address Allocation for Private Internets." RFC Rekhter, Y., et al. "Address Allocation for Private Internets." RFC
1918. 1918.
6. Acknowledgments 6. Acknowledgments
The authors wish to thank the IETF Benchmarking Working Group for The author wishes to thank the IETF Benchmarking Working Group for
agreeing to review this document. Several other persons offered valuable agreeing to review this document. Several other persons offered valuable
contributions and critiques during this project: Ted Doty (Internet contributions and critiques during this project: Ted Doty (Internet
Security Systems), Shlomo Kramer (Check Point Software Technologies), Security Systems), Kevin Dubray (Bay Networks), Helen Holzbaur (NSTL),
Robert Mandeville (European Network Laboratories), Brent Melson Jim Hurd (NSTL), Robert Mandeville (European Network Laboratories),
Brent Melson (NSTL), Steve Platt (NSTL), Marcus Ranum (Network Flight
Newman et al. Page [15]
(National Software Testing Laboratories), Marcus Ranum (Network Flight
Recorder Inc.), Greg Shannon (Ascend Communications), Christoph Schuba Recorder Inc.), Greg Shannon (Ascend Communications), Christoph Schuba
(Sun Microsystems), Rick Siebenaler (Cyberguard), and Greg Smith (Check (Sun Microsystems), Rick Siebenaler (Cyberguard), and Greg Smith (Check
Point Software Technologies). Point Software Technologies).
Newman Page [18]
7. Contact information 7. Contact information
David Newman David Newman
Data Communications magazine Data Communications magazine
1221 Avenue of the Americas, 41st Floor 1221 Avenue of the Americas, 41st Floor
New York, NY 10020 New York, NY 10020
USA USA
212-512-6182 voice 212-512-6182 voice
212-512-6833 fax 212-512-6833 fax
dnewman@data.com dnewman@data.com
Helen Holzbaur Newman Page [19]
National Software Testing Laboratories Inc.
625 Ridge Pike
Conshohocken, PA 19428
USA
helen@nstl.com
Jim Hurd
National Software Testing Laboratories Inc.
625 Ridge Pike
Conshohocken, PA 19428
USA
jimh@nstl.com
Steven Platt
National Software Testing Laboratories Inc.
625 Ridge Pike
Conshohocken, PA 19428
USA
steve@nstl.com
Newman et al. Page [16]
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/