draft-ietf-bmwg-secperf-00.txt   draft-ietf-bmwg-secperf-01.txt 
Benchmarking Working Group D. Newman Network Working Group D. Newman
INTERNET-DRAFT Data Communications INTERNET-DRAFT Data Communications
Expires in January 1998 H. Holzbaur, J. Hurd, and S. Platt Expires in May 1998 H. Holzbaur, J. Hurd, and S. Platt
National Software Testing Laboratories National Software Testing Laboratories
Benchmarking Terminology for Network Security Devices Benchmarking Terminology for Firewall Performance
<draft-ietf-bmwg-secperf-00.txt> <draft-ietf-bmwg-secperf-01.txt>
Status of This Memo Status of This Memo
This document is an Internet-Draft. Internet-Drafts are working This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its documents of the Internet Engineering Task Force (IETF), its areas,
areas, and its working groups. Note that other groups may also and its working groups. Note that other groups may also distribute
distribute working documents as Internet-Drafts. working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six Internet-Drafts are draft documents valid for a maximum of six months
months and may be updated, replaced, or obsoleted by other and may be updated, replaced, or obsoleted by other documents at any
documents at any time. It is inappropriate to use Internet- time. It is inappropriate to use Internet- Drafts as reference
Drafts as reference material or to cite them other than as "work material or to cite them other than as "work in progress."
in progress."
To view the entire list of current Internet-Drafts, please check To view the entire list of current Internet-Drafts, please check the
the "1id-abstracts.txt" listing contained in the Internet-Drafts "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow
Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe),
(Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
Coast), or ftp.isi.edu (US West Coast). ftp.isi.edu (US West Coast).
This memo provides information for the Internet community. This 1. Introduction .......................................................2
memo does not specify an Internet standard of any kind. 2. Existing definitions ...............................................2
Distribution of this memo is unlimited. 3. Term definitions ...................................................2
3.1 Allowed traffic ..............................................2
3.2 Authentication ...............................................3
3.3 Data source ..................................................3
3.4 Data connection ..............................................4
3.5 Demilitarized zone (DMZ) .....................................4
3.6 Dual-homed ...................................................5
3.7 Dynamic proxy ................................................5
3.8 External network .............................................6
3.9 Homed ........................................................6
3.10 Packet filtering ............................................6
3.11 Perimeter network ...........................................7
3.12 Policy ......................................................7
3.13 Protected network ...........................................8
3.14 Proxy .......................................................8
3.15 Rejected traffic ............................................9
3.16 Rule set ....................................................9
3.17 Session .....................................................9
3.18 Stateful inspection ........................................10
3.19 Tri-homed ..................................................11
3.20 User .......................................................11
4. Security considerations ..........................................11
5. References ........................................................12
6. Acknowledgments ...................................................12
7. Contact Information ...............................................12
1. Introduction 1. Introduction
Despite the rapid rise in deployment of network security devices
such as firewalls and authentication/encryption products, there is
no standard method for evaluating the performance of these
devices.
The lack of a standard is troubling for two reasons. First,
hardware and software implementations vary widely, making it
difficult to do direct performance comparisons. Second, a growing
number of organizations are deploying these devices on internal
networks that operate at relatively high data rates, while many
network security devices are optimized for use over relatively
low-speed wide-area connections. As a result, users are often
unsure whether the products they buy will stand up to the
relatively heavy loads found on internal networks.
This document defines terms used in measuring the performance of This document defines terms used in measuring the performance of
network security devices. It extends the terminology already used firewalls. It extends the terminology already used for benchmarking
for benchmarking routers and switches to network security devices. routers and switches and adds terminology specific to firewalls. The
The primary metrics defined in this document are maximum primary metrics defined in this document are maximum forwarding rate
forwarding rate and maximum number of connections. and maximum number of connections.
Depending on the outcome of discussions within the BMWG, we may Why are firewall performance measurements needed? First, despite the
also attempt to classify devices using various architectural rapid rise in deployment of firewalls, there is no standard method
considerations (proxy, packet filter) or offered load levels for benchmarking their performance. Second, implementations vary
widely, making it difficult to do direct performance comparisons.
Finally, more and more organizations are deploying firewalls on
internal networks operating at relatively high speeds, while most
firewall implementations remain optimized for use over low-speed
wide-area connections. As a result, users are often unsure whether
the products they buy will stand up to relatively heavy loads.
Newman et al [Page 1] We may also create additional terminology and methodology documents
(high, medium, low) as criteria. Additionally, new metrics may to define other types of network security products such as virtual
need to be defined to evaluate application-level issues. private network (VPN) and encryption devices. This document, however,
focuses solely on firewall terminology.
2. Existing definitions 2. Existing definitions
This document uses the conceptual framework established in RFCs This document uses the conceptual framework established in RFCs 1242
1242 and 1944 and draft-ietf-bmwg-lanswitch-05.txt, which and 1944 (for routers) and draft-ietf-bmwg-lanswitch-07.txt (for
describes benchmarking of LAN switches). The router and switch documents contain discussions of
switch performance. In addition to defining basic practices, these several terms relevant to benchmarking the performance of firewalls.
documents Readers should consult the router and switch documents before making
contain discussions of several terms relevant to benchmarking use of this document.
performance of network security devices. This document uses the
definition format described in RFC 1242, Section 2. Readers should This document uses the definition format described in RFC 1242,
consult these documents before making use of this document. Section 2. The sections in each definition are: definition,
discussion, measurement units (optional), issues (optional), and
cross-references.
3. Term definitions 3. Term definitions
3.1 Authentication 3.1 Allowed traffic
Definition: Definition:
The process of verifying that a client user or machine requesting Packets forwarded as a result of the rule set of the DUT/SUT.
a network resource is who he, she, or it claims to be, and vice
versa.
Discussion: Discussion:
Trust is a critical concept in network security. Obviously, any Firewalls typically are configured to forward only those packets
network resource (such as a file server or printer) with explicitly permitted in the rule set. Forwarded packets MUST be included
restricted access MUST require authentication before granting in calculating the forwarding rate or maximum forwarding rate of the
access. DUT/SUT. All other packets MUST NOT be included in forwarding rate
calculations.
Authentication takes many forms, including but not limited to IP
addresses; TCP or UDP port numbers; passwords; external token
authentication cards; and pattern matching based on human
characteristics such as signature, speech, or retina patterns.
Authentication MAY work either by client machine (for example, by
proving that a given IP source address really is that address, and
not a rogue machine spoofing that address) or by user (by proving
that the user really is who he or she claims to be). Servers
SHOULD also authenticate themselves to clients.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
Newman et al. Page [2]
See also: See also:
forwarding rate (3.9) policy (3.12)
user (3.25) rule set (3.15)
virtual client (3.26)
3.2 Bidirectional traffic 3.2 Authentication
Definition: Definition:
The process of verifying that a client user or machine requesting a
Newman et al [Page 2] network resource is who he, she, or it claims to be, and vice versa.
Packets presented to a DUT/SUT such that the network interfaces of
the DUT/SUT both receive and transmit traffic.
Discussion: Discussion:
Traffic patterns offered to the DUT/SUT MUST be bidirectional or Trust is a critical concept in network security. Any network resource
fully meshed. See forwarding rate (3.9) for a more complete (such as a file server or printer) with restricted access MUST require
discussion of issues with traffic patterns. authentication before granting access.
Authentication takes many forms, including but not limited to IP
addresses; TCP or UDP port numbers; passwords; external token
authentication cards; and biometric identification such as signature,
speech, or retina recognition systems.
Authentication MAY work either by client machine (for example, by
proving that a given IP source address really is that address, and not a
rogue machine spoofing that address) or by user (by proving that the
user really is who he or she claims to be). Servers SHOULD also
authenticate themselves to clients.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
truncated binary exponential back-off algorithm Testers should be aware that in an increasingly mobile society,
authentication based on machine-specific criteria such as an IP address
or port number is not equivalent to verifying that a given individual is
making an access request. At this writing systems that verify the
identity of persons are typically external to the firewall, and may
introduce additional latency to the overall SUT.
See Also: See also:
forwarding rate (3.9) user (3.20)
fully meshed traffic (3.10)
unidirectional traffic (3.24)
3.3 Data source 3.3 Data source
Definition: Definition:
A station capable of generating traffic to the DUT/SUT. A station capable of generating traffic to the DUT/SUT.
Discussion: Discussion:
One data source MAY emulate multiple users or stations. In One data source MAY emulate multiple users or stations. In addition, one
addition, one data source MAY offer traffic to multiple network data source MAY offer traffic to multiple network interfaces on the
interfaces on the DUT/SUT. However, each virtual client MUST offer DUT/SUT.
traffic to only one interface.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: Newman et al. Page [3]
user (3.25)
virtual client (3.26)
3.4 Demilitarized zone (DMZ)
Definition:
A network segment or segments located between protected and
external networks. DMZ networks are sometimes called perimeter
networks.
Discussion:
As an extra security measure, networks are often designed such
that protected and external segments are never directly connected.
Instead, security devices (and possibly other public resources
such as WWW or FTP servers) often reside in the so-called DMZ
network. To connect protected, DMZ, and external networks with one
device, the device MUST have at least three network interfaces.
Newman et al [Page 3]
Multiple devices MAY constitute the DMZ, in which case the devices
connected the protected network with the DMZ and the DMZ with the
external network MUST have two network interfaces.
Measurement units:
Not applicable
Issues: The term "data source" is deliberately independent of any number of
Dual-homed users. It is useful to think of data sources simply as traffic
Multihomed generators, and not as a given number of users.
See also: See also:
external network (3.8) data connection (3.4)
perimeter network (3.15)
protected network (3.17)
3.5 Device under test (DUT) 3.4 Data connection
Definition: Definition:
The network security device to which traffic is offered and A logical link established between two hosts, or between a host and the
response measured. DUT/SUT.
Discussion: Discussion:
A single station, generally equipped with at least two network The number of concurrent data connections a firewall can field may be
interfaces. just as important a metric for some users as the rate at which it can
forward traffic. Data connections MAY be TCP sessions, but they don't
have to be. Users of other connection-oriented protocols such as ATM may
wish to measure these, either instead of or in addition to TCP
connections.
Measurement units: Measurement units:
Not applicable Number of connections
Issues: Issues:
A firewall's architecture dictates where the connection is terminated.
In the case of proxy-based systems, a connection by definition
terminates at the DUT/SUT. But firewalls using packet filtering or
stateful inspection designs act only as passthrough devices, in that
they reside between two connection endpoints. Regardless of firewall
architecture, the number of data connections is still relevant, since
all firewalls perform some form of connection maintenance; at the very
least, all check connection requests against their rule sets.
See also: See also:
system under test (SUT) (3.23) data source (3.3)
3.6 Dual-homed 3.5 Demilitarized zone (DMZ)
Definition: Definition:
A station with at least two network interfaces. A network segment or segments located between protected and external
networks. DMZ networks are sometimes called perimeter networks.
Discussion: Discussion:
Dual-homed network security devices connect two segments with As an extra security measure, networks are often designed such that
different network-layer addresses. protected and external segments are never directly connected. Instead,
firewalls (and possibly public resources such as WWW or FTP servers)
Measurement units: often reside on the so-called DMZ network. To connect protected, DMZ,
Not applicable and external networks with one device, the device MUST have at least
three network interfaces.
Issues:
See also:
multihomed (3.12)
3.7 Dynamic proxy
Definition:
Newman et al [Page 4] Multiple firewalls MAY bound the DMZ. In this case, the firewalls
A proxy service that is set up and torn down in response to a connecting the protected network with the DMZ and the DMZ with the
client request, rather than existing on a static basis. external network MUST each have at least two network interfaces.
Discussion: Newman et al. Page [4]
Proxy services (see section 3.18) typically are configured to
"listen" on a given port number for client requests. However, some
devices set up a proxy service only when a client requests the
service.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
rule sets Dual-homed
Homed
See also: See also:
proxy (3.18) external network (3.8)
rule sets (3.20) perimeter network (3.11)
protected network (3.13)
3.8 External network 3.6 Dual-homed
Definition: Definition:
The segment or segments not protected by the network security A firewall with at least two network interfaces.
DUT/SUT.
Discussion: Discussion:
Network security devices are deployed between protected and Dual-homed firewalls connect two segments with different network
unprotected segments. The external network is not protected by the addresses.
DUT/SUT.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
Typically the differentiator between one segment and another is its IP
address. However, firewalls may connect different networks of other
types, such as ATM or Netware segments.
See also: See also:
demilitarized zone (DMZ) (3.4) Homed (3.9)
protected network (3.17) Tri-homed (3.19)
3.9 Forwarding rate
Definition: The number of bits per second a DUT/SUT can transmit
to the correct destination network interface in response to a
specified offered load.
Discussion:
Network security devices are by definition session-oriented: They
will only grant access to a desired resource once authentication
occurs and a session has been established.
Because application-layer sessions are always involved,
unidirectional packet-per-second metrics are not meaningful in the
context of testing network security devices. Instead, this
Newman et al [Page 5]
definition MUST measure application-layer performance once a
session has been established.
Forwarding rate refers to the number of bits per second observed
on the output side of the network interface under test. Forwarding
rate can be measured with different traffic orientations and
distributions. When multiple network interfaces are measured,
measurements MUST be observed from the interface with the highest
forwarding rate.
Measurement units:
bits per second (bit/s)
kilobits per second (kbit/s)
Megabits per second (Mbit/s)
Issues:
truncated binary exponential back-off algorithm
unidirectional vs. bidirectional
See Also:
authentication (3.1)
maximum forwarding rate (3.11)
offered load (3.13)
unidirectional traffic (3.24)
3.10 Fully meshed traffic 3.7 Dynamic proxy
Definition: Definition:
Packets forwarded simultaneously among all of a designated number A proxy service that is set up and torn down in response to a client
of network interfaces of a DUT/SUT such that each of the request, rather than existing on a static basis.
interfaces under test will both forward packets to and receive
packets from all of the other interfaces.
Discussion: Discussion:
Fully meshed traffic is the most thorough method of exercising the Proxy services (see section 3.14) typically "listen" on a given TCP port
transmitting and receiving capabilities of the DUT/SUT. number for client requests. With static proxies, a firewall always
forwards packets containing a given TCP port number if that port number
Unlike past definitions for router or switch testing, it should be is permitted by the rule set. Dynamic proxies, in contrast, forward TCP
noted that fully meshed traffic in this context is not necessarily packets only once an authenticated connection has been established. When
symmetrical. While all of a designated group of network interfaces the connection closes, a firewall using dynamic proxies rejects
MUST simultaneously send and receive traffic, the type and amount individual packets, even if they contain port numbers allowed by a rule
of traffic offered MAY differ on each interface. For example, a set.
network security device may see more traffic from the protected
network bound for the external network than the opposite (although
the inverse could be true during an attack on the DUT/SUT).
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
Half duplex
Full duplex
See also:
Newman et al [Page 6]
bidirectional traffic (3.2)
unidirectional traffic (3.24)
3.11 Maximum forwarding rate
Definition: Newman et al. Page [5]
The highest forwarding rate of a network security device taken
from a set of iterative measurements.
Discussion:
Maximum forwarding rate may degrade before maximum load is
offered.
Unlike benchmarks for evaluating router and switch performance,
this definition MUST involve measurement of application-layer
performance rather than network-layer packet-per-second metrics.
Measurement units:
Megabits per second
kbytes per second
bytes per second
Issues: rule sets
full duplex vs. half duplex
truncated binary exponential back-off algorithm
See also: See also:
bidirectional traffic (3.2) allowed traffic (3.1)
partially meshed traffic (3.24) proxy (3.14)
unidirectional traffic rejected traffic (3.15)
rule set (3.16)
3.12 Multihomed 3.8 External network
Definition: Definition:
A network security device with more than two network interfaces. The segment or segments not protected by the network security DUT/SUT.
Discussion: Discussion:
Multihoming is a way to connect three or more networks-protected, Firewalls are deployed between protected and unprotected segments. The
DMZ, and external-with a single network security device. However, external network is not protected by the DUT/SUT.
this configuration is not mandatory if multiple network security
devices are used. For example, one device could secure the
connection between an external and DMZ network, while another
could secure the connection between a DMZ and protected network;
the two stations collectively form the SUT.
Because of the differences in traffic patterns between dual-homed
and multihomed devices, direct performance comparisons should be
avoided. However, it is acceptable to compare results between a
dual-homed device and a DUT/SUT in which only two network
interfaces are used.
Measurement units: Measurement units:
Not applicable Not applicable
Newman et al [Page 7]
Issues: Issues:
truncated binary exponential back-off algorithm
See also: See also:
bidirectional traffic (3.2) demilitarized zone (DMZ) (3.5)
dual-homed (3.6) protected network (3.13)
fully meshed traffic (3.10)
3.13 Offered load 3.9 Homed
Definition: Definition:
The number of bits per second that an external source can transmit The number of logical interfaces a DUT/SUT contains.
to a DUT/SUT for forwarding to a specified network interface or
interfaces.
Discussion: Discussion:
The load that an external source actually applies to a DUT/SUT may Firewalls MUST contain at least two interfaces, using a dual-homed
be lower than the external source attempts to apply because of configuration. In network topologies where a DMZ is used, the firewall
collisions on the wire. The transmission capabilities of the contains at least three interfaces and is said to be tri-homed.
external source SHOULD be verified without the DUT/SUT by Additional interfaces would make a firewall quad-homed, quint-homed, and
transmitting unidirectional traffic. so on.
Measurement units:
bits per second
kilobits per second (kbit/s)
Megabits per second (Mbit/s)
Issues: Issues:
truncated binary exponential back-off algorithm It is theoretically possible for a firewall to contain one physical
interface and multiple logical interfaces. This configuration is
strongly discouraged for testing purposes because of the possibility of
leakage between protected and unprotected segments.
See also: See also:
forwarding rate (3.9) Dual-homed (3.6)
maximum forwarding rate (3.11) Tri-homed (3.19)
3.14 Packet filtering 3.10 Packet filtering
Definition: Definition:
The process of controlling access by examining packets based on The process of controlling access by examining packets based on packet
network-layer or transport-layer criteria. header content.
Newman et al. Page [6]
Discussion: Discussion:
Packet-filtering devices forward or deny packets based on Packet-filtering devices forward or deny packets based on information in
information in each packet's header. A packet-filtering network each packet's header, such as IP address or TCP port number. A packet-
security device uses a rule set (see section 3.20) to determine filtering firewall uses a rule set (see section 3.16) to determine which
which traffic should be forwarded and which should be blocked. traffic should be forwarded and which should be blocked.
Packet filtering may be used in a dual-homed or multihomed device.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
dynamic proxy (3.7) dynamic proxy (3.7)
proxy (3.14)
rule set (3.16)
stateful inspection (3.18)
Newman et al [Page 8] 3.11 Perimeter network
proxy (3.18)
rule set (3.20)
stateful inspection (3.22)
3.15 Perimeter network
Definition: Definition:
A network segment or segments located between protected and A network segment or segments located between protected and external
external networks. Perimeter networks are often called DMZ networks. Perimeter networks are often called DMZ networks.
networks.
Discussion: Discussion:
See the definition of DMZ (which see) for a discussion. See the definition of DMZ for a discussion.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
Dual-homed Dual-homed
Multihomed Tri-homed
See also: See also:
Demilitarized zone (DMZ) (3.4) Demilitarized zone (DMZ) (3.5)
external network (3.8) external network (3.8)
protected network (3.17) protected network (3.13)
3.16 Policy 3.12 Policy
Definition: Definition:
A document defining acceptable use of protected, DMZ, and external A document defining acceptable use of protected, DMZ, and external
networks. networks.
Discussion: Discussion:
Security policies generally do not spell out specific Security policies generally do not spell out specific configurations for
configurations for network security devices; rather, they set firewalls; rather, they set general guidelines for what it and is not
general guidelines for what it and is not acceptable network acceptable network behavior.
behavior.
The actual mechanism for controlling access is usually the rule The actual mechanism for controlling access is usually the rule set (see
set (see section 3.20) implemented in the DUT/SUT. section 3.16) implemented in the DUT/SUT.
Measurement units: Measurement units:
Not applicable Not applicable
Newman et al. Page [7]
Issues: Issues:
See also: See also:
Rule set (3.20) Rule set (3.16)
3.17 Protected network 3.13 Protected network
Definition: Definition:
A network segment or segments to which access is controlled by the A network segment or segments to which access is controlled by the
DUT/SUT. DUT/SUT.
Newman et al [Page 9]
Discussion: Discussion:
Network security devices are intended to prevent unauthorized Firewalls are intended to prevent unauthorized access either to or from
access either to or from the protected network. Depending on the the protected network. Depending on the configuration specified by the
configuration specified by the policy and rule set, the DUT/SUT policy and rule set, the DUT/SUT may allow stations on the protected
may allow stations on the protected segment to act as clients for segment to act as clients for servers on either the DMZ or the external
servers on either the DMZ or the external network, or both. network, or both.
Protected networks are often called "internal networks." That term Protected networks are often called "internal networks." That term is
is not used here because network security devices increasingly are not used here because firewalls increasingly are deployed within an
deployed within an organization, where all segments are by organization, where all segments are by definition internal.
definition internal.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
Demilitarized zone (DMZ) (3.4) Demilitarized zone (DMZ) (3.5)
external network (3.8) external network (3.8)
policy (3.16) policy (3.12)
rule set (3.20) rule set (3.16)
3.18 Proxy 3.14 Proxy
Definition: Definition:
The process of requesting sessions with servers on behalf of A request for a connection made on behalf of a host.
clients.
Discussion: Discussion:
Proxy-based network security devices never involve direct Proxy-based firewalls never allow direct connections between hosts.
connections between client and server. Instead, two sessions are Instead, two connections are established: one between the client host
established: one between the client and the DUT/SUT, and another and the DUT/SUT, and another between the DUT/SUT and server host.
between the DUT/SUT and server.
As with packet-filtering network security devices, proxy-based As with packet-filtering firewalls, proxy-based devices use a rule set
devices use a rule set (which see) to determine which traffic to determine which traffic should be forwarded and which should be
should be forwarded and which should be blocked. rejected.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
Newman et al. Page [8]
dynamic proxy (3.7) dynamic proxy (3.7)
packet filtering (3.14) packet filtering (3.10)
stateful inspection (3.22) stateful inspection (3.18)
3.19 Rejected traffic 3.15 Rejected traffic
Definition: Definition:
Packets dropped as a result of the rule set of the DUT/SUT. Packets dropped as a result of the rule set of the DUT/SUT.
Newman et al [Page 10]
Discussion: Discussion:
Network security devices typically are configured to drop any Firewalls MUST reject any traffic not explicitly permitted in the rule
traffic not explicitly permitted in the rule set (which see). set. Dropped packets MUST NOT be included in calculating the forwarding
Dropped packets MUST NOT be included in calculating the forwarding
rate or maximum forwarding rate of the DUT/SUT. rate or maximum forwarding rate of the DUT/SUT.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
forwarding rate (3.9) policy (3.12)
maximum forwarding rate (3.11) rule set (3.16)
policy (3.16)
rule set (3.20)
3.20 Rule set 3.16 Rule set
Definition: Definition:
The collection of definitions that determines which packets the The collection of access control rules that determines which packets the
DUT/SUT will forward and which it will reject. DUT/SUT will forward and which it will reject.
Discussion: Discussion:
Rule sets control access to and from the network interfaces of the Rule sets control access to and from the network interfaces of the
DUT/SUT. By definition, rule sets MUST NOT apply equally to all DUT/SUT. By definition, rule sets MUST NOT apply equally to all network
network interfaces; otherwise there would be no need for the interfaces; otherwise there would be no need for the firewall.
network security device. Therefore, a specific rule set MUST be Therefore, a specific rule set MUST be applied to each network interface
applied to each network device used in the DUT/SUT. in the DUT/SUT.
The order of rules within the rule set is critical. Network The order of rules within the rule set is critical. Firewalls generally
security devices generally scan rule sets in a "top down" fashion, scan rule sets in a "top down" fashion, which is to say that the device
which is to say that the device compares each packet received with compares each packet received with each rule in the rule set until it
each rule in the rule set until it finds a rule that applies to finds a rule that applies to the packet. Once the device finds an
the packet. Once the device finds an applicable rule, it applies applicable rule, it applies the actions defined in that rule (such as
the actions defined in that rule (such as forwarding or rejecting forwarding or rejecting the packet) and ignores all subsequent rules.
the packet) and ignores all subsequent rules. For purposes of this For testing purposes, the rule set MUST conclude with a rule denying all
document, the rule set MUST conclude with a rule denying all access.
access except that which is permitted in the rule set.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
Demilitarized zone (DMZ) (3.4) Demilitarized zone (DMZ) (3.5)
external network (3.8) external network (3.8)
policy (3.17) policy (3.12)
protected network (3.18)
rejected traffic (3.19)
3.21 Session Newman et al. Page [9]
protected network (3.13)
rejected traffic (3.15)
3.17 Session
Newman et al [Page 11]
Definition: Definition:
A logical connection established between two stations using a A logical connection established between two stations using a known
known protocol. For purposes of this document, a session MUST be protocol.
conducted over either TCP (RFC 793) or UDP (RFC 768).
Discussion: Discussion:
Because of the application-layer focus of many network security Because of the application-layer focus of many firewalls, sessions are a
devices, sessions are a more useful metric than the packet-based more useful metric than the packet-based measurements used in
measurements used in benchmarking routers and switches. Although benchmarking routers and switches. Although firewall rule sets generally
network security device rule sets generally work on a per-packet work on a per-packet basis, it is ultimately sessions that a firewall
basis, it is ultimately sessions that a network security device must handle. For example, the number of file transfer protocol (ftp)
must handle. For example, the number of file transfer protocol sessions a DUT/SUT can handle concurrently is a more meaningful
(ftp) sessions a DUT/SUT can handle concurrently is a more measurement in benchmarking performance than the number of ftp "open"
meaningful measurement in benchmarking performance than the number packets it can reject. Further, a stateful inspection firewall will not
of ftp "open" packets it can reject. Further, a stateful forward individual packets if those packets' headers conflict with state
inspection device (which see) will not forward individual packets information maintained by the firewall.
if those packets' headers conflict with state information
maintained in the device's rule set.
For purposes of this document, a session MUST be established using For purposes of this document, a session MUST be established using a
a known protocol. A traffic pattern is not considered a session known protocol such as TCP. A traffic pattern is not considered a
until it successfully completes the establishment procedures session until it successfully completes the establishment procedures
defined by that protocol. defined by that protocol.
Also for purposes of this document, a session constitutes the Also for purposes of this document, a session constitutes the logical
logical connection between two end-stations and not the connection between two end-stations and not the intermediate connections
intermediate connections that proxy-based network security devices that proxy-based firewalls may use.
may use.
Issues: Issues:
See also: See also:
policy (3.16) policy (3.12)
proxy (3.18) proxy (3.14)
rule set (3.20) rule set (3.16)
stateful inspection (3.22) stateful inspection (3.18)
3.22 Stateful inspection 3.18 Stateful inspection
Definition: Definition:
The process of forwarding or rejecting traffic based on the The process of forwarding or rejecting traffic based on the contents of
contents of a state table maintained by the network security a state table maintained by a firewall.
device.
Discussion: Discussion:
Packet filtering and proxy devices are essentially static, in that Packet filtering and proxy firewalls are essentially static, in that
they always forward or reject traffic based on the contents of the they always forward or reject packets based on the contents of the rule
rule set. Devices using stateful inspection, in contrast, will set.
only forward traffic if it corresponds with state information
maintained by the device about each session. For example, a
stateful inspection device will reject a packet on TCP port 21
(ftp DATA) if no ftp session has been established.
Newman et al [Page 12]
Measurement units:
Not applicable
Issues:
See also:
dynamic proxy (3.7)
packet filter (3.14)
proxy (3.18)
3.23 System under test (SUT)
Definition: In contrast, devices using stateful inspection will only forward packets
The collective set of network security devices to which traffic is if they correspond with state information maintained by the device about
offered as a single entity and response measured. each session. For example, a stateful inspection device will reject a
packet on port 20 (ftp-data) if no session has been established over the
ftp control port (usually port 21).
Discussion: Newman et al. Page [10]
A system under test may comprise multiple network security
devices. A typical configuration involves two or more devices,
with at least one located between the protected network and DMZ
and at least one other located between the DMZ and external
network. Some devices may be active, such as firewalls or
authentication products; other devices, such as systems for
logging, may be passive.
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
demilitarized zone (DMZ) (3.4) dynamic proxy (3.7)
device under test (3.5) packet filter (3.10)
external network (3.8) proxy (3.14)
protected network (3.17)
3.24 Unidirectional traffic 3.19 Tri-homed
Definition: Definition:
Packets offered to the DUT/SUT such that the sending and receiving A firewall with three network interfaces.
network interface or interfaces are mutually exclusive.
Discussion: Discussion:
This definition is included mainly for purposes of completeness; Tri-homed firewalls connect three network segments with different
it is not particularly meaningful in the context of network network addresses. Typically, these would be protected, DMZ, and
security device performance. As noted in the discussion of external segments.
forwarding rate (see section 3.9), network security devices almost
invariably involve sessions with bidirectional traffic flow.
However, unidirectional traffic is appropriate for evaluating the
maximum forwarding rate of data sources (absent the DUT/SUT), and
for evaluating the maximum forwarding rate of certain
connectionless protocols.
Newman et al [Page 13]
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
half duplex vs. full duplex Usually the differentiator between one segment and another is its IP
address. However, firewalls may connect different networks of other
types, such as ATM or Netware segments.
See also: See also:
bidirectional traffic (3.2) Dual-homed (3.6)
forwarding rate (3.9) Homed (3.9)
maximum forwarding rate (3.11)
3.25 User 3.20 User
Definition: Definition:
The person or machine requesting access to resources protected by The person or machine requesting access to resources protected by the
the DUT/SUT. DUT/SUT.
Discussion: Discussion:
"User" is a problematic term in the context of security device "User" is a problematic term in the context of firewall performance
performance testing, for several reasons. First, a user may in testing, for several reasons. First, a user may in fact be a machine or
fact be a machine or machines requesting services through the machines requesting services through the DUT/SUT. Second, different
DUT/SUT. Second, different "user" requests may require radically "user" requests may require radically different amounts of DUT/SUT
different amounts of DUT/SUT resources. Third, traffic profiles resources. Third, traffic profiles vary widely from one organization to
vary widely from one organization to another, making it difficult another, making it difficult to characterize the load offered by a
to characterize the load offered by a typical users. For these typical users.
reasons, we prefer not to measure DUT/SUT performance in terms of
users supported. Instead, we describe performance in terms of
maximum forwarding rate and maximum number of sessions sustained.
Measurement units:
Not applicable
Issues:
See also:
data source (3.3)
virtual client (3.26)
3.26 Virtual client
Definition:
A subset of a data source that represents one individual user.
Discussion: For these reasons, we prefer not to measure DUT/SUT performance in terms
In offering traffic to the DUT/SUT it may be useful for one data of users supported. Instead, we describe performance in terms of maximum
source to emulate multiple users, machines, or networks. For forwarding rate and maximum number of sessions sustained. Further, we
purposes of this document, each emulated user should be considered use the term "data source" rather than user to describe the traffic
a virtual client. generator(s).
One data source MAY offer traffic from multiple virtual clients to Newman et al. Page [11]
multiple network interfaces on the DUT/SUT. However, each virtual
client MUST offer traffic to just one network interface.
Newman et al [Page 14]
Measurement units: Measurement units:
Not applicable Not applicable
Issues: Issues:
See also: See also:
data source (3.3) data source (3.3)
user (3.25)
4. Security considerations 4. Security considerations
Security considerations are explicitly excluded from this memo. Security considerations are explicitly excluded from this memo. The
The authors plan to address security and management concerns in a authors plan to address security and management concerns in a separate
separate proposal brought to the IETF's security directorate. proposal brought to the IETF's security directorate.
5. References 5. References
Bradner, S., editor. "Benchmarking Terminology for Network Bradner, S., editor. "Benchmarking Terminology for Network
Interconnection Devices." RFC 1242. Interconnection Devices." RFC 1242.
Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network
Interconnect Devices." RFC 1944. Interconnect Devices." RFC 1944.
Mandeville, B. "Benchmarking Terminology for LAN Switching Mandeville, B. "Benchmarking Terminology for LAN Switching Devices."
Devices." ftp://ietf.org/internet-drafts/draft-ietf-bmwg- ftp://ietf.org/internet-drafts/draft-ietf-bmwg-lanswitch-07.txt
lanswitch-05.txt
Newman, D., and Melson, B. "Can Firewalls Take the Heat?" Data Newman, D., and Melson, B. "Can Firewalls Take the Heat?" Data
Communications, November 21, 1995. Communications, November 21, 1995.
http://www.data.com/Lab_Tests/Firewalls.html http://www.data.com/Lab_Tests/Firewalls.html
Newman, D., Holzbaur, H., and Bishop, K. "Firewalls: Don't Get Newman, D., Holzbaur, H., and Bishop, K. "Firewalls: Don't Get Burned,"
Burned," Data Communications, March 21, 1997. Data Communications, March 21, 1997.
http://www.data.com/lab_tests/firewalls97.html http://www.data.com/lab_tests/firewalls97.html
Ranum, M. "Firewall Performance Measurement Techniques: A Ranum, M. "Firewall Performance Measurement Techniques: A Scientific
Scientific Approach." Approach." http://www.clark.net/pub/mjr/pubs/fwperf/intro.htm
http://www.clark.net/pub/mjr/pubs/fwperf/intro.htm
Shannon, G. "Profile of Corporate Internet Application Traffic." Shannon, G. "Profile of Corporate Internet Application Traffic."
http://www.milkyway.com/libr/prof.html http://www.milkyway.com/libr/prof.html
6. Acknowledgments 6. Acknowledgments
The authors wish to thank the IETF Benchmarking Working Group for The authors wish to thank the IETF Benchmarking Working Group for
agreeing to review this document. Ted Doty (Network Systems), agreeing to review this document. Ted Doty (Internet Security Systems),
Shlomo Kramer (Check Point Software Technologies), Bob Mandeville Shlomo Kramer (Check Point Software Technologies), Bob Mandeville
(European Network Laboratories), Brent Melson (National Software (European Network Laboratories), Brent Melson (National Software Testing
Testing Laboratories), Marcus Ranum (Network Flight Recorder Laboratories), Marcus Ranum (Network Flight Recorder Inc.), Greg Shannon
Inc.), Greg Shannon (Milkyway Networks), Rick Siebenaler (Ascend Communications), Rick Siebenaler (Cyberguard), and Greg Smith
(Cyberguard), and Greg Smith (Check Point Software Technologies) (Check Point Software Technologies) offered valuable contributions and
offered valuable contributions and critiques during this project. critiques during this project.
7. Contact Information 7. Contact Information
David Newman David Newman
Data Communications magazine Data Communications magazine
Newman et al [Page 15]
1221 Avenue of the Americas, 41st Floor 1221 Avenue of the Americas, 41st Floor
New York, NY 10020 New York, NY 10020
USA USA
Newman et al. Page [12]
212-512-6182 voice 212-512-6182 voice
212-512-6833 fax 212-512-6833 fax
dnewman@data.com dnewman@data.com
Helen Holzbaur Helen Holzbaur
National Software Testing Laboratories Inc. National Software Testing Laboratories Inc.
625 Ridge Pike 625 Ridge Pike
Conshohocken, PA 19428 Conshohocken, PA 19428
USA USA
helen@nstl.com helen@nstl.com
Jim Hurd Jim Hurd
National Software Testing Laboratories Inc. National Software Testing Laboratories Inc.
625 Ridge Pike 625 Ridge Pike
Conshohocken, PA 19428 Conshohocken, PA 19428
USA USA
jimh@nstl.com jimh@nstl.com
Steven Platt, PhD. Steven Platt
National Software Testing Laboratories Inc. National Software Testing Laboratories Inc.
625 Ridge Pike 625 Ridge Pike
Conshohocken, PA 19428 Conshohocken, PA 19428
USA USA
steve@nstl.com steve@nstl.com
Newman et al [Page 16] Newman et al. Page [13]
 End of changes. 

This html diff was produced by rfcdiff 1.23, available from http://www.levkowetz.com/ietf/tools/rfcdiff/