draft-ietf-bmwg-vswitch-opnfv-03.txt   draft-ietf-bmwg-vswitch-opnfv-04.txt 
Network Working Group M. Tahhan Network Working Group M. Tahhan
Internet-Draft B. O'Mahony Internet-Draft B. O'Mahony
Intended status: Informational Intel Intended status: Informational Intel
Expires: November 2, 2017 A. Morton Expires: December 10, 2017 A. Morton
AT&T Labs AT&T Labs
May 1, 2017 June 8, 2017
Benchmarking Virtual Switches in OPNFV Benchmarking Virtual Switches in OPNFV
draft-ietf-bmwg-vswitch-opnfv-03 draft-ietf-bmwg-vswitch-opnfv-04
Abstract Abstract
This memo describes the progress of the Open Platform for NFV (OPNFV) This memo describes the contributions of the Open Platform for NFV
project on virtual switch performance "VSPERF". This project intends (OPNFV) project on virtual switch performance "VSPERF", particularly
to build on the current and completed work of the Benchmarking in the areas of test set-ups and configuration parameters for the
Methodology Working Group in IETF, by referencing existing system under test. This project has extended the current and
literature. The Benchmarking Methodology Working Group has completed work of the Benchmarking Methodology Working Group in IETF,
traditionally conducted laboratory characterization of dedicated and references existing literature. The Benchmarking Methodology
physical implementations of internetworking functions. Therefore, Working Group has traditionally conducted laboratory characterization
this memo begins to describe the additional considerations when of dedicated physical implementations of internetworking functions.
Therefore, this memo describes the additional considerations when
virtual switches are implemented in general-purpose hardware. The virtual switches are implemented in general-purpose hardware. The
expanded tests and benchmarks are also influenced by the OPNFV expanded tests and benchmarks are also influenced by the OPNFV
mission to support virtualization of the "telco" infrastructure. mission to support virtualization of the "telco" infrastructure.
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
skipping to change at page 1, line 48 skipping to change at page 1, line 49
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 2, 2017. This Internet-Draft will expire on December 10, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 28 skipping to change at page 2, line 28
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 3
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5
3.1. Comparison with Physical Network Functions . . . . . . . 5 3.1. Comparison with Physical Network Functions . . . . . . . 5
3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 5 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 5
3.3. New Configuration Parameters . . . . . . . . . . . . . . 5 3.3. New Configuration Parameters . . . . . . . . . . . . . . 6
3.4. Flow classification . . . . . . . . . . . . . . . . . . . 7 3.4. Flow classification . . . . . . . . . . . . . . . . . . . 8
3.5. Benchmarks using Baselines with Resource Isolation . . . 8 3.5. Benchmarks using Baselines with Resource Isolation . . . 8
4. VSPERF Specification Summary . . . . . . . . . . . . . . . . 9 4. VSPERF Specification Summary . . . . . . . . . . . . . . . . 10
5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 17 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 18
5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 18 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 19
5.2. Accuracy of Activation section . . . . . . . . . . . . . 18 5.2. Accuracy of Activation section . . . . . . . . . . . . . 19
5.3. Reliability of Activation . . . . . . . . . . . . . . . . 18 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 19
5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 18 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 19
5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 18 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 19
5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 18 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 19
5.7. Reliability of Operation . . . . . . . . . . . . . . . . 18 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 20
5.8. Scalability of Operation . . . . . . . . . . . . . . . . 19 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 20
5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 20
6. Security Considerations . . . . . . . . . . . . . . . . . . . 19 6. Security Considerations . . . . . . . . . . . . . . . . . . . 20
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 21
9.1. Normative References . . . . . . . . . . . . . . . . . . 20 9.1. Normative References . . . . . . . . . . . . . . . . . . 21
9.2. Informative References . . . . . . . . . . . . . . . . . 21 9.2. Informative References . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23
1. Introduction 1. Introduction
Benchmarking Methodology Working Group (BMWG) has traditionally Benchmarking Methodology Working Group (BMWG) has traditionally
conducted laboratory characterization of dedicated physical conducted laboratory characterization of dedicated physical
implementations of internetworking functions. The Black-box implementations of internetworking functions. The Black-box
Benchmarks of Throughput, Latency, Forwarding Rates and others have Benchmarks of Throughput, Latency, Forwarding Rates and others have
served our industry for many years. Now, Network Function served our industry for many years. Now, Network Function
Virtualization (NFV) has the goal to transform how internetwork Virtualization (NFV) has the goal to transform how internetwork
functions are implemented, and therefore has garnered much attention. functions are implemented, and therefore has garnered much attention.
This memo summarizes the progress of the Open Platform for NFV A virtual switch (vswitch) is an important aspect of the NFV
infrastructure; it provides connectivity between and among physical
network functions and virtual network functions. As a result, there
are many vswitch benchmarking efforts, but few specifications to
guide the many new test design choices. This is a complex problem
and an industry-wide work-in-progress. In future, several of BMWG's
fundamental specifications will likely be updated as more testing
experience helps to form consensus around new methodologies, and BMWG
should continue to collaborate with all organizations who share the
same goal.
This memo describes the contributions of the Open Platform for NFV
(OPNFV) project on virtual switch performance characterization, (OPNFV) project on virtual switch performance characterization,
"VSPERF", through the Brahmaputra (second) release [BrahRel]. This "VSPERF", through the Danube 3.0 (fourth) release [DanubeRel] to the
project intends to build on the current and completed work of the chartered work of the BMWG (with stable references to their test
Benchmarking Methodology Working Group in IETF, by referencing descriptions). This project has extended the current and completed
existing literature. For example, currently the most often work of the BMWG in IETF, and references existing literature. For
referenced RFC is [RFC2544] (which depends on [RFC1242]) and example, the most often referenced RFC is [RFC2544] (which depends on
foundation of the benchmarking work in OPNFV is common and strong. [RFC1242]), so the foundation of the benchmarking work in OPNFV is
common and strong. The recommended extensions are specifically in
the areas of test set-ups and configuration parameters for the system
under test.
See [VSPERFhome] for more background, and the OPNFV website for See [VSPERFhome] for more background, and the OPNFV website for
general information [OPNFV]. general information [OPNFV].
The authors note that OPNFV distinguishes itself from other open The authors note that OPNFV distinguishes itself from other open
source compute and networking projects through its emphasis on source compute and networking projects through its emphasis on
existing "telco" services as opposed to cloud-computing. There are existing "telco" services as opposed to cloud-computing. There are
many ways in which telco requirements have different emphasis on many ways in which telco requirements have different emphasis on
performance dimensions when compared to cloud computing: support for performance dimensions when compared to cloud computing: support for
and transfer of isochronous media streams is one example. and transfer of isochronous media streams is one example.
Note also that the move to NFV Infrastructure has resulted in many
new benchmarking initiatives across the industry. The authors are
currently doing their best to maintain alignment with many other
projects, and this Internet Draft is one part of the efforts. We
acknowledge the early work in
[I-D.huang-bmwg-virtual-network-performance], and useful discussion
with the authors.
1.1. Abbreviations 1.1. Abbreviations
For the purposes of this document, the following abbreviations apply: For the purposes of this document, the following abbreviations apply:
ACK Acknowledge ACK Acknowledge
ACPI Advanced Configuration and Power Interface ACPI Advanced Configuration and Power Interface
BIOS Basic Input Output System BIOS Basic Input Output System
BMWG Benchmarking Methodology Working Group BMWG Benchmarking Methodology Working Group
CPDP Control Plane Data Plane CPDP Control Plane Data Plane
CPU Central Processing Unit CPU Central Processing Unit
skipping to change at page 4, line 40 skipping to change at page 4, line 40
SW Software SW Software
TCP Transmission control Protocol TCP Transmission control Protocol
TSO TCP Segment Offload TSO TCP Segment Offload
UDP User Datagram Protocol UDP User Datagram Protocol
VM Virtual Machine VM Virtual Machine
VNF Virtualised Network Function VNF Virtualised Network Function
VSPERF OPNFV vSwitch Performance Project VSPERF OPNFV vSwitch Performance Project
2. Scope 2. Scope
The primary purpose and scope of the memo is to inform the industry The primary purpose and scope of the memo is to describe key aspects
of work-in-progress that builds on the body of extensive BMWG of vswitch benchmarking, particularly in the areas of test set-ups
literature and experience, and describe the extensions needed for and configuration parameters for the system under test, and extend
benchmarking virtual switches. Initial feedback indicates that many the body of extensive BMWG literature and experience. Initial
of these extensions may be applicable beyond the current scope (to feedback indicates that many of these extensions may be applicable
hardware switches in the NFV Infrastructure and to virtual routers, beyond this memo's current scope (to hardware switches in the NFV
for example). Additionally, this memo serves as a vehicle to include Infrastructure and to virtual routers, for example). Additionally,
more detail and commentary from BMWG and other Open Source this memo serves as a vehicle to include more detail and relevant
communities, under BMWG's chartered work to characterize the NFV commentary from BMWG and other Open Source communities, under BMWG's
Infrastructure. A virtual switch (vswitch) is an important aspect of chartered work to characterize the NFV Infrastructure.
the NFV infrastructure; it provides connectivity between and among
physical network functions and virtual network functions.
The benchmarking covered in this memo should be applicable to many The benchmarking covered in this memo should be applicable to many
types of vswitches, and remain vswitch-agnostic to great degree. types of vswitches, and remain vswitch-agnostic to great degree.
There has been no attempt to track and test all features of any There has been no attempt to track and test all features of any
specific vswitch implementation. specific vswitch implementation.
3. Benchmarking Considerations 3. Benchmarking Considerations
This section highlights some specific considerations (from This section highlights some specific considerations (from
[I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual
switches. The OPNFV project is sharing its present view on these switches. The OPNFV project is sharing its present view on these
areas, as they develop their specifications in the Level Test Design areas, as they develop their specifications in the Level Test Design
(LTD) document. (LTD) document.
skipping to change at page 5, line 23 skipping to change at page 5, line 21
[I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual
switches. The OPNFV project is sharing its present view on these switches. The OPNFV project is sharing its present view on these
areas, as they develop their specifications in the Level Test Design areas, as they develop their specifications in the Level Test Design
(LTD) document. (LTD) document.
3.1. Comparison with Physical Network Functions 3.1. Comparison with Physical Network Functions
To compare the performance of virtual designs and implementations To compare the performance of virtual designs and implementations
with their physical counterparts, identical benchmarks are needed. with their physical counterparts, identical benchmarks are needed.
BMWG has developed specifications for many physical network BMWG has developed specifications for many physical network
functions. The OPNFV LTD document re-uses existing benchmarks and functions. The BMWG has recommended to re-use existing benchmarks
methods through references, and expands on them for development of and methods in [I-D.ietf-bmwg-virtual-net], and the OPNFV LTD expands
new methods. A key configuration aspect for vswitches is the number on them as described here. A key configuration aspect for vswitches
of parallel CPU cores required to achieve comparable performance with is the number of parallel CPU cores required to achieve comparable
a given physical device, or whether some limit of scale will be performance with a given physical device, or whether some limit of
reached before the vswitch can achieve the comparable performance scale will be reached before the vswitch can achieve the comparable
level. performance level.
It's unlikely that the virtual switch will be the only application It's unlikely that the virtual switch will be the only application
running on the System Under Test (SUT), so CPU utilization, Cache running on the System Under Test (SUT), so CPU utilization, Cache
utilization, and Memory footprint should also be recorded for the utilization, and Memory footprint should also be recorded for the
virtual implementations of internetworking functions. virtual implementations of internetworking functions. However,
internally-measured metrics such as these are not benchmarks; they
may be useful for the audience (operations) to know, and may also be
useful if there is a problem encountered during testing.
Benchmark Comparability between virtual and physical/hardware
implementations of equivalent functions will likely place more
detailed and exact requirements on the *testing systems* (in terms of
stream generation, algorithms to search for max values, and their
configurations of course). This is another area for standards
development to appreciate. However, the is a topic for a future
draft.
3.2. Continued Emphasis on Black-Box Benchmarks 3.2. Continued Emphasis on Black-Box Benchmarks
External observations remain essential as the basis for Benchmarks. External observations remain essential as the basis for Benchmarks.
Internal observations with fixed specification and interpretation Internal observations with fixed specification and interpretation
will be provided in parallel to assist the development of operations will be provided in parallel to assist the development of operations
procedures when the technology is deployed. procedures when the technology is deployed.
3.3. New Configuration Parameters 3.3. New Configuration Parameters
A key consideration when conducting any sort of benchmark is trying A key consideration when conducting any sort of benchmark is trying
to ensure the consistency and repeatability of test results. When to ensure the consistency and repeatability of test results. When
benchmarking the performance of a vswitch there are many factors that benchmarking the performance of a vswitch there are many factors that
can affect the consistency of results, one key factor is matching the can affect the consistency of results, one key factor is matching the
various hardware and software details of the SUT. This section lists various hardware and software details of the SUT. This section lists
some of the many new parameters which this project believes are some of the many new parameters which this project believes are
critical to report in order to achieve repeatability. critical to report in order to achieve repeatability.
It has been the goal of the project to produce repeatable results,
and a large set of the parameters believed to be critical is provided
so that the benchmarking community can better appreciate the increase
in configuration complexity inherent in this work. The parameter set
below is assumed sufficient for the infrastructure in use by the
VSPERF project to obtain repeatable results from test-to-test.
Hardware details (platform, processor, memory, and network) Hardware details (platform, processor, memory, and network)
including: including:
o BIOS version, release date and any configurations that were o BIOS version, release date and any configurations that were
modified modified
o Power management at all levels (ACPI sleep states, processor o Power management at all levels (ACPI sleep states, processor
package, OS...) package, OS...)
o CPU microcode level o CPU microcode level
skipping to change at page 7, line 30 skipping to change at page 7, line 50
itself) on the host itself) on the host
o Details of Resource isolation, such as CPUs designated for Host/ o Details of Resource isolation, such as CPUs designated for Host/
Kernel (isolcpu) and CPUs designated for specific processes Kernel (isolcpu) and CPUs designated for specific processes
(taskset). - Test duration. - Number of flows. (taskset). - Test duration. - Number of flows.
Test Traffic Information: Test Traffic Information:
o Traffic type - UDP, TCP, others. o Traffic type - UDP, TCP, others.
o Packet Sizes - fixed or IMIX [RFC6985] o Frame Sizes - fixed or IMIX [RFC6985](with [IEEE802.1ac], frames
may be longer than 1500 bytes, and up to 2000 bytes)
o Deployment Scenario o Deployment Scenario - defines the communications path in the SUT
3.4. Flow classification 3.4. Flow classification
Virtual switches group packets into flows by processing and matching Virtual switches group packets into flows by processing and matching
particular packet or frame header information, or by matching packets particular packet or frame header information, or by matching packets
based on the input ports. Thus a flow can be thought of a sequence based on the input ports. Thus a flow can be thought of a sequence
of packets that have the same set of header field values, or have of packets that have the same set of header field values, or have
arrived on the same physical or logical port. Performance results arrived on the same physical or logical port. Performance results
can vary based on the parameters the vswitch uses to match for a can vary based on the parameters the vswitch uses to match for a
flow. The recommended flow classification parameters for any vswitch flow. The recommended flow classification parameters for any vswitch
performance tests are: the input port (physical or logical), the performance tests are: the input port (physical or logical), the
source IP address, the destination IP address and the Ethernet source MAC address, the destination MAC address, the source IP
protocol type field. It is essential to increase the flow timeout address, the destination IP address and the Ethernet protocol type
time on a vswitch before conducting any performance tests that do not field (although classification may take place on other fields, such
intend to measure the flow setup time. Normally the first packet of as source and destination transport port numbers). It is essential
a particular stream will install the flow in the virtual switch which to increase the flow timeout time on a vswitch before conducting any
performance tests that do not intend to measure the flow setup time,
see Section 3 of [RFC2889]. Normally the first packet of a
particular stream will install the flow in the virtual switch which
adds an additional latency, subsequent packets of the same flow are adds an additional latency, subsequent packets of the same flow are
not subject to this latency if the flow is already installed on the not subject to this latency if the flow is already installed on the
vswitch. vswitch.
3.5. Benchmarks using Baselines with Resource Isolation 3.5. Benchmarks using Baselines with Resource Isolation
This outline describes measurement of baseline with isolated This outline describes measurement of baseline with isolated
resources at a high level, which is the intended approach at this resources at a high level, which is the intended approach at this
time. time.
1. Baselines: 1. Baselines:
* Optional: Benchmark platform forwarding capability without a * Optional: Benchmark platform forwarding capability without a
vswitch or VNF for at least 72 hours (serves as a means of vswitch or VNF for at least 72 hours (serves as a means of
platform validation and a means to obtain the base performance platform validation and a means to obtain the base performance
for the platform in terms of its maximum forwarding rate and for the platform in terms of its maximum forwarding rate and
latency). latency).
Benchmark platform forwarding capability Figure 1 Benchmark platform forwarding capability
__ __
+--------------------------------------------------+ | +--------------------------------------------------+ |
| +------------------------------------------+ | | | +------------------------------------------+ | |
| | | | | | | | | |
| | Simple Forwarding App | | Host | | Simple Forwarding App | | Host
| | | | | | | | | |
| +------------------------------------------+ | | | +------------------------------------------+ | |
| | NIC | | | | | NIC | | |
+---+------------------------------------------+---+ __| +---+------------------------------------------+---+ __|
^ : ^ :
| | | |
: v : v
+--------------------------------------------------+ +--------------------------------------------------+
| | | |
| traffic generator | | traffic generator |
| | | |
+--------------------------------------------------+ +--------------------------------------------------+
* Benchmark VNF forwarding capability with direct connectivity * Benchmark VNF forwarding capability with direct connectivity
(vswitch bypass, e.g., SR/IOV) for at least 72 hours (serves (vswitch bypass, e.g., SR/IOV) for at least 72 hours (serves
as a means of VNF validation and a means to obtain the base as a means of VNF validation and a means to obtain the base
performance for the VNF in terms of its maximum forwarding performance for the VNF in terms of its maximum forwarding
rate and latency). The metrics gathered from this test will rate and latency). The metrics gathered from this test will
serve as a key comparison point for vswitch bypass serve as a key comparison point for vswitch bypass
technologies performance and vswitch performance. technologies performance and vswitch performance.
Benchmark VNF forwarding capability Figure 2 Benchmark VNF forwarding capability
__ __
+--------------------------------------------------+ | +--------------------------------------------------+ |
| +------------------------------------------+ | | | +------------------------------------------+ | |
| | | | | | | | | |
| | VNF | | | | | VNF | | |
| | | | | | | | | |
| +------------------------------------------+ | | | +------------------------------------------+ | |
| | Passthrough/SR-IOV | | Host | | Passthrough/SR-IOV | | Host
| +------------------------------------------+ | | | +------------------------------------------+ | |
skipping to change at page 10, line 24 skipping to change at page 11, line 24
Some of the above/newer RFCs are being applied in benchmarking for Some of the above/newer RFCs are being applied in benchmarking for
the first time, and represent a development challenge for test the first time, and represent a development challenge for test
equipment developers. Fortunately, many members of the testing equipment developers. Fortunately, many members of the testing
system community have engaged on the VSPERF project, including an system community have engaged on the VSPERF project, including an
open source test system. open source test system.
In addition to this, the LTD also re-uses the terminology defined by: In addition to this, the LTD also re-uses the terminology defined by:
o [RFC2285] Benchmarking Terminology for LAN Switching Devices o [RFC2285] Benchmarking Terminology for LAN Switching Devices
o [RFC5481] Packet Delay Variation Applicability Statement It is recommended that these references are included in future
benchmarking specifications:
Specifications to be included in future updates of the LTD include:
o [RFC3918] Methodology for IP Multicast Benchmarking o [RFC3918] Methodology for IP Multicast Benchmarking
o [RFC4737] Packet Reordering Metrics o [RFC4737] Packet Reordering Metrics
As one might expect, the most fundamental internetworking As one might expect, the most fundamental internetworking
characteristics of Throughput and Latency remain important when the characteristics of Throughput and Latency remain important when the
switch is virtualized, and these benchmarks figure prominently in the switch is virtualized, and these benchmarks figure prominently in the
specification. specification.
When considering characteristics important to "telco" network When considering characteristics important to "telco" network
functions, we must begin to consider additional performance metrics. functions, additional performance metrics are needed. In this case,
In this case, the project specifications have referenced metrics from the project specifications have referenced metrics from the IETF IP
the IETF IP Performance Metrics (IPPM) literature. This means that Performance Metrics (IPPM) literature. This means that the [RFC2544]
the [RFC2544] test of Latency is replaced by measurement of a metric test of Latency is replaced by measurement of a metric derived from
derived from IPPM's [RFC2679], where a set of statistical summaries IPPM's [RFC2679], where a set of statistical summaries will be
will be provided (mean, max, min, etc.). Further metrics planned to provided (mean, max, min, and percentiles). Further metrics planned
be benchmarked include packet delay variation as defined by [RFC5481] to be benchmarked include packet delay variation as defined by
, reordering, burst behaviour, DUT availability, DUT capacity and [RFC5481] , reordering, burst behaviour, DUT availability, DUT
packet loss in long term testing at Throughput level, where some low- capacity and packet loss in long term testing at Throughput level,
level of background loss may be present and characterized. where some low-level of background loss may be present and
characterized.
Tests have been (or will be) designed to collect the metrics below: Tests have been designed to collect the metrics below:
o Throughput Tests to measure the maximum forwarding rate (in frames o Throughput Tests to measure the maximum forwarding rate (in frames
per second or fps) and bit rate (in Mbps) for a constant load (as per second or fps) and bit rate (in Mbps) for a constant load (as
defined by [RFC1242]) without traffic loss. defined by [RFC1242]) without traffic loss.
o Packet and Frame Delay Distribution Tests to measure average, min o Packet and Frame Delay Distribution Tests to measure average, min
and max packet and frame delay for constant loads. and max packet and frame delay for constant loads.
o Packet Delay Tests to understand latency distribution for o Packet Delay Tests to understand latency distribution for
different packet sizes and over an extended test run to uncover different packet sizes and over an extended test run to uncover
skipping to change at page 11, line 25 skipping to change at page 12, line 25
o Scalability Tests to understand how the virtual switch performs o Scalability Tests to understand how the virtual switch performs
with increasing number of flows, number of active ports, with increasing number of flows, number of active ports,
configuration complexity of the forwarding logic, etc. configuration complexity of the forwarding logic, etc.
o Stream Performance Tests (TCP, UDP) to measure bulk data transfer o Stream Performance Tests (TCP, UDP) to measure bulk data transfer
performance, i.e. how fast systems can send and receive data performance, i.e. how fast systems can send and receive data
through the switch. through the switch.
o Control Path and Datapath Coupling Tests, to understand how o Control Path and Datapath Coupling Tests, to understand how
closely coupled the datapath and the control path are as well as closely the datapath and the control path are coupled, as well as
the effect of this coupling on the performance of the DUT the effect of this coupling on the performance of the DUT
(example: delay of the initial packet of a flow). (example: delay of the initial packet of a flow).
o CPU and Memory Consumption Tests to understand the virtual o CPU and Memory Consumption Tests to understand the virtual
switch's footprint on the system, usually conducted as auxiliary switch's footprint on the system, conducted as auxiliary
measurements with benchmarks above. They include: CPU measurements with benchmarks above. They include: CPU
utilization, Cache utilization and Memory footprint. utilization, Cache utilization and Memory footprint.
o The so-called "Soak" tests, where the selected test is conducted o The so-called "Soak" tests, where the selected test is conducted
over a long period of time (with an ideal duration of 24 hours, over a long period of time (with an ideal duration of 24 hours,
but only long enough to determine that stability issues exist when but only long enough to determine that stability issues exist when
found; there is no requirement to continue a test when a DUT found; there is no requirement to continue a test when a DUT
exhibits instability over time). The key performance exhibits instability over time). The key performance
characteristics and benchmarks for a DUT are determined (using characteristics and benchmarks for a DUT are determined (using
short duration tests) prior to conducting soak tests. The purpose short duration tests) prior to conducting soak tests. The purpose
of soak tests is to capture transient changes in performance which of soak tests is to capture transient changes in performance which
may occur due to infrequent processes, memory leaks, or the low may occur due to infrequent processes, memory leaks, or the low
probability coincidence of two or more processes. The stability probability coincidence of two or more processes. The stability
of the DUT is the paramount consideration, so performance must be of the DUT is the paramount consideration, so performance must be
evaluated periodically during continuous testing, and this results evaluated periodically during continuous testing, and this results
in use of [RFC2889] Frame Rate metrics instead of [RFC2544] in use of [RFC2889] Frame Rate metrics instead of [RFC2544]
Throughput (which requires stopping traffic to allow time for all Throughput (which requires stopping traffic to allow time for all
traffic to exit internal queues), for example. traffic to exit internal queues), for example.
Future/planned test specs include: Additional test specification development should include:
o Request/Response Performance Tests (TCP, UDP) which measure the o Request/Response Performance Tests (TCP, UDP) which measure the
transaction rate through the switch. transaction rate through the switch.
o Noisy Neighbour Tests, to understand the effects of resource o Noisy Neighbour Tests, to understand the effects of resource
sharing on the performance of a virtual switch. sharing on the performance of a virtual switch.
o Tests derived from examination of ETSI NFV Draft GS IFA003 o Tests derived from examination of ETSI NFV Draft GS IFA003
requirements [IFA003] on characterization of acceleration requirements [IFA003] on characterization of acceleration
technologies applied to vswitches. technologies applied to vswitches.
The flexibility of deployment of a virtual switch within a network The flexibility of deployment of a virtual switch within a network
means that the BMWG IETF existing literature needs to be used to means that it is necessary to characterize the performance of a
characterize the performance of a vswitch in various deployment vswitch in various deployment scenarios. The deployment scenarios
scenarios. The deployment scenarios under consideration include: under consideration include:
Physical port to virtual switch to physical port Figure 3 Physical port to virtual switch to physical port
__ __
+--------------------------------------------------+ | +--------------------------------------------------+ |
| +--------------------+ | | | +--------------------+ | |
| | | | | | | | | |
| | v | | Host | | v | | Host
| +--------------+ +--------------+ | | | +--------------+ +--------------+ | |
| | phy port | vswitch | phy port | | | | | phy port | vswitch | phy port | | |
+---+--------------+------------+--------------+---+ __| +---+--------------+------------+--------------+---+ __|
^ : ^ :
| | | |
: v : v
+--------------------------------------------------+ +--------------------------------------------------+
| | | |
| traffic generator | | traffic generator |
| | | |
+--------------------------------------------------+ +--------------------------------------------------+
Physical port to virtual switch to VNF to virtual switch to physical Figure 4 Physical port to virtual switch to VNF to virtual switch to
port physical port
__ __
+---------------------------------------------------+ | +---------------------------------------------------+ |
| | | | | |
| +-------------------------------------------+ | | | +-------------------------------------------+ | |
| | Application | | | | | Application | | |
| +-------------------------------------------+ | | | +-------------------------------------------+ | |
| ^ : | | | ^ : | |
| | | | | Guest | | | | | Guest
| : v | | | : v | |
skipping to change at page 14, line 5 skipping to change at page 15, line 5
+---+--------------+------------+--------------+---+ __| +---+--------------+------------+--------------+---+ __|
^ : ^ :
| | | |
: v : v
+--------------------------------------------------+ +--------------------------------------------------+
| | | |
| traffic generator | | traffic generator |
| | | |
+--------------------------------------------------+ +--------------------------------------------------+
Physical port to virtual switch to VNF to virtual switch to VNF to Figure 5 Physical port to virtual switch to VNF to virtual switch to
virtual switch to physical port VNF to virtual switch to physical port
__ __
+----------------------+ +----------------------+ | +----------------------+ +----------------------+ |
| Guest 1 | | Guest 2 | | | Guest 1 | | Guest 2 | |
| +---------------+ | | +---------------+ | | | +---------------+ | | +---------------+ | |
| | Application | | | | Application | | | | | Application | | | | Application | | |
| +---------------+ | | +---------------+ | | | +---------------+ | | +---------------+ | |
| ^ | | | ^ | | | | ^ | | | ^ | | |
| | v | | | v | | Guests | | v | | | v | | Guests
| +---------------+ | | +---------------+ | | | +---------------+ | | +---------------+ | |
skipping to change at page 15, line 5 skipping to change at page 16, line 5
+---+--------------+----------+--------------+---+_| +---+--------------+----------+--------------+---+_|
^ : ^ :
| | | |
: v : v
+--------------------------------------------------+ +--------------------------------------------------+
| | | |
| traffic generator | | traffic generator |
| | | |
+--------------------------------------------------+ +--------------------------------------------------+
Physical port to virtual switch to VNF Figure 6 Physical port to virtual switch to VNF
__ __
+---------------------------------------------------+ | +---------------------------------------------------+ |
| | | | | |
| +-------------------------------------------+ | | | +-------------------------------------------+ | |
| | Application | | | | | Application | | |
| +-------------------------------------------+ | | | +-------------------------------------------+ | |
| ^ | | | ^ | |
| | | | Guest | | | | Guest
| : | | | : | |
skipping to change at page 16, line 5 skipping to change at page 17, line 5
+---+--------------+------------ -------------- ---+ __| +---+--------------+------------ -------------- ---+ __|
^ ^
| |
: :
+--------------------------------------------------+ +--------------------------------------------------+
| | | |
| traffic generator | | traffic generator |
| | | |
+--------------------------------------------------+ +--------------------------------------------------+
VNF to virtual switch to physical port Figure 7 VNF to virtual switch to physical port
__ __
+---------------------------------------------------+ | +---------------------------------------------------+ |
| | | | | |
| +-------------------------------------------+ | | | +-------------------------------------------+ | |
| | Application | | | | | Application | | |
| +-------------------------------------------+ | | | +-------------------------------------------+ | |
| : | | | : | |
| | | | Guest | | | | Guest
| v | | | v | |
skipping to change at page 17, line 5 skipping to change at page 18, line 5
+-------------------------------+--------------+---+ __| +-------------------------------+--------------+---+ __|
: :
| |
v v
+--------------------------------------------------+ +--------------------------------------------------+
| | | |
| traffic generator | | traffic generator |
| | | |
+--------------------------------------------------+ +--------------------------------------------------+
VNF to virtual switch to VNF Figure 8 VNF to virtual switch to VNF
__ __
+----------------------+ +----------------------+ | +----------------------+ +----------------------+ |
| Guest 1 | | Guest 2 | | | Guest 1 | | Guest 2 | |
| +---------------+ | | +---------------+ | | | +---------------+ | | +---------------+ | |
| | Application | | | | Application | | | | | Application | | | | Application | | |
| +---------------+ | | +---------------+ | | | +---------------+ | | +---------------+ | |
| | | | ^ | | | | | | ^ | |
| v | | | | | Guests | v | | | | | Guests
| +---------------+ | | +---------------+ | | | +---------------+ | | +---------------+ | |
skipping to change at page 18, line 29 skipping to change at page 19, line 29
o Throughput.RFC2544.ResetTime o Throughput.RFC2544.ResetTime
5.4. Scale of Activation 5.4. Scale of Activation
o Activation.RFC2889.AddressCachingCapacity o Activation.RFC2889.AddressCachingCapacity
5.5. Speed of Operation 5.5. Speed of Operation
o Throughput.RFC2544.PacketLossRate o Throughput.RFC2544.PacketLossRate
o CPU.RFC2544.0PacketLoss o Stress.RFC2544.0PacketLoss
o Throughput.RFC2544.PacketLossRateFrameModification o Throughput.RFC2544.PacketLossRateFrameModification
o Throughput.RFC2544.BackToBackFrames o Throughput.RFC2544.BackToBackFrames
o Throughput.RFC2889.MaxForwardingRate o Throughput.RFC2889.MaxForwardingRate
o Throughput.RFC2889.ForwardPressure o Throughput.RFC2889.ForwardPressure
o Throughput.RFC2889.BroadcastFrameForwarding o Throughput.RFC2889.BroadcastFrameForwarding
o Throughput.RFC2544.WorstN-BestN
o Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
5.6. Accuracy of Operation 5.6. Accuracy of Operation
o Throughput.RFC2889.ErrorFramesFiltering o Throughput.RFC2889.ErrorFramesFiltering
o Throughput.RFC2544.Profile o Throughput.RFC2544.Profile
5.7. Reliability of Operation 5.7. Reliability of Operation
o Throughput.RFC2889.Soak o Throughput.RFC2889.Soak
skipping to change at page 19, line 4 skipping to change at page 20, line 10
o Throughput.RFC2889.ErrorFramesFiltering o Throughput.RFC2889.ErrorFramesFiltering
o Throughput.RFC2544.Profile o Throughput.RFC2544.Profile
5.7. Reliability of Operation 5.7. Reliability of Operation
o Throughput.RFC2889.Soak o Throughput.RFC2889.Soak
o Throughput.RFC2889.SoakFrameModification o Throughput.RFC2889.SoakFrameModification
o PacketDelayVariation.RFC3393.Soak o PacketDelayVariation.RFC3393.Soak
5.8. Scalability of Operation 5.8. Scalability of Operation
o Scalability.RFC2544.0PacketLoss o Scalability.RFC2544.0PacketLoss
o MemoryBandwidth.RFC2544.0PacketLoss.Scalability o MemoryBandwidth.RFC2544.0PacketLoss.Scalability
o Scalability.VNF.RFC2544.PacketLossProfile
o Scalability.VNF.RFC2544.PacketLossRatio
5.9. Summary 5.9. Summary
|------------------------------------------------------------------------| |------------------------------------------------------------------------|
| | | | | | | | | | | |
| | SPEED | ACCURACY | RELIABILITY | SCALE | | | SPEED | ACCURACY | RELIABILITY | SCALE |
| | | | | | | | | | | |
|------------------------------------------------------------------------| |------------------------------------------------------------------------|
| | | | | | | | | | | |
| Activation | X | X | X | X | | Activation | X | X | X | X |
| | | | | | | | | | | |
skipping to change at page 20, line 15 skipping to change at page 21, line 25
7. IANA Considerations 7. IANA Considerations
No IANA Action is requested at this time. No IANA Action is requested at this time.
8. Acknowledgements 8. Acknowledgements
The authors appreciate and acknowledge comments from Scott Bradner, The authors appreciate and acknowledge comments from Scott Bradner,
Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik,
Christian Trautman, and others for their reviews. Christian Trautman, and others for their reviews.
We also acknowledge the early work in
[I-D.huang-bmwg-virtual-network-performance], and useful discussion
with the authors.
9. References 9. References
9.1. Normative References 9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. <http://www.rfc-editor.org/info/rfc2119>.
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
skipping to change at page 21, line 17 skipping to change at page 22, line 31
DOI 10.17487/RFC6201, March 2011, DOI 10.17487/RFC6201, March 2011,
<http://www.rfc-editor.org/info/rfc6201>. <http://www.rfc-editor.org/info/rfc6201>.
[RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet
Sizes for Additional Testing", RFC 6985, Sizes for Additional Testing", RFC 6985,
DOI 10.17487/RFC6985, July 2013, DOI 10.17487/RFC6985, July 2013,
<http://www.rfc-editor.org/info/rfc6985>. <http://www.rfc-editor.org/info/rfc6985>.
9.2. Informative References 9.2. Informative References
[BrahRel] "Brahmaputra, Second OPNFV Release [DanubeRel]
https://wiki.opnfv.org/display/SWREL/Brahmaputra". "Danube, Fourth OPNFV Release
https://wiki.opnfv.org/display/SWREL/Danube".
[I-D.huang-bmwg-virtual-network-performance] [I-D.huang-bmwg-virtual-network-performance]
Huang, L., Rong, G., Mandeville, B., and B. Hickman, Huang, L., Rong, G., Mandeville, B., and B. Hickman,
"Benchmarking Methodology for Virtualization Network "Benchmarking Methodology for Virtualization Network
Performance", draft-huang-bmwg-virtual-network- Performance", draft-huang-bmwg-virtual-network-
performance-02 (work in progress), March 2017. performance-02 (work in progress), March 2017.
[I-D.ietf-bmwg-virtual-net] [I-D.ietf-bmwg-virtual-net]
Morton, A., "Considerations for Benchmarking Virtual Morton, A., "Considerations for Benchmarking Virtual
Network Functions and Their Infrastructure", draft-ietf- Network Functions and Their Infrastructure", draft-ietf-
bmwg-virtual-net-05 (work in progress), March 2017. bmwg-virtual-net-05 (work in progress), March 2017.
[IEEE802.1ac]
https://standards.ieee.org/findstds/standard/802.1AC-
2016.html, "802.1AC-2016 - IEEE Standard for Local and
metropolitan area networks -- Media Access Control (MAC)
Service Definition", 2016.
[IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/ [IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/
IFA003_Acceleration_-_vSwitch_Spec/". IFA003_Acceleration_-_vSwitch_Spec/".
[LTD] "LTD Test Specification [LTD] Note: if the Danube Release LTD is available in artifacts
http://artifacts.opnfv.org/vswitchperf/brahmaputra/docs/ at publication, we will use that URL instead., "LTD Test S
requirements/index.html". pecificationhttp://artifacts.opnfv.org/vswitchperf/colorad
o/docs/requirements/vswitchperf_ltd.html".
[LTDoverV] [LTDoverV]
"LTD Test Spec Overview "LTD Test Spec Overview
https://wiki.opnfv.org/display/vsperf/ https://wiki.opnfv.org/display/vsperf/
LTD+Test+Spec+Overview". LTD+Test+Spec+Overview".
[OPNFV] "OPNFV Home https://www.opnfv.org/". [OPNFV] "OPNFV Home https://www.opnfv.org/".
[RFC1242] Bradner, S., "Benchmarking Terminology for Network [RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
 End of changes. 42 change blocks. 
127 lines changed or deleted 176 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/