draft-ietf-bmwg-virtual-net-05.txt   rfc8172.txt 
Network Working Group A. Morton Internet Engineering Task Force (IETF) A. Morton
Internet-Draft AT&T Labs Request for Comments: 8172 AT&T Labs
Intended status: Informational March 16, 2017 Category: Informational July 2017
Expires: September 17, 2017 ISSN: 2070-1721
Considerations for Benchmarking Virtual Network Functions and Their Considerations for Benchmarking Virtual Network Functions
Infrastructure and Their Infrastructure
draft-ietf-bmwg-virtual-net-05
Abstract Abstract
The Benchmarking Methodology Working Group has traditionally The Benchmarking Methodology Working Group has traditionally
conducted laboratory characterization of dedicated physical conducted laboratory characterization of dedicated physical
implementations of internetworking functions. This memo investigates implementations of internetworking functions. This memo investigates
additional considerations when network functions are virtualized and additional considerations when network functions are virtualized and
performed in general purpose hardware. performed in general-purpose hardware.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This document is not an Internet Standards Track specification; it is
provisions of BCP 78 and BCP 79. published for informational purposes.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months This document is a product of the Internet Engineering Task Force
and may be updated, replaced, or obsoleted by other documents at any (IETF). It represents the consensus of the IETF community. It has
time. It is inappropriate to use Internet-Drafts as reference received public review and has been approved for publication by the
material or to cite them other than as "work in progress." Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 7841.
This Internet-Draft will expire on September 17, 2017. Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc8172.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Considerations for Hardware and Testing . . . . . . . . . . . 4 3. Considerations for Hardware and Testing . . . . . . . . . . . 4
3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4 3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4
3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5 3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5
3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6 3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6
3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7 3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 7 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 8
4.1. Comparison with Physical Network Functions . . . . . . . 7 4.1. Comparison with Physical Network Functions . . . . . . . 8
4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8 4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8
4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 8 4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 9
4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 9 4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 10
4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12 4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12
5. Security Considerations . . . . . . . . . . . . . . . . . . . 12 5. Security Considerations . . . . . . . . . . . . . . . . . . . 12
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
8. Version history . . . . . . . . . . . . . . . . . . . . . . . 13 7.1. Normative References . . . . . . . . . . . . . . . . . . 13
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 7.2. Informative References . . . . . . . . . . . . . . . . . 14
9.1. Normative References . . . . . . . . . . . . . . . . . . 14 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 15
9.2. Informative References . . . . . . . . . . . . . . . . . 14
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15
1. Introduction 1. Introduction
The Benchmarking Methodology Working Group (BMWG) has traditionally The Benchmarking Methodology Working Group (BMWG) has traditionally
conducted laboratory characterization of dedicated physical conducted laboratory characterization of dedicated physical
implementations of internetworking functions (or physical network implementations of internetworking functions (or physical network
functions, PNFs). The Black-box Benchmarks of Throughput, Latency, functions (PNFs)). The black-box benchmarks of throughput, latency,
Forwarding Rates and others have served our industry for many years. forwarding rates, and others have served our industry for many years.
[RFC1242] and [RFC2544] are the cornerstones of the work. [RFC1242] and [RFC2544] are the cornerstones of the work.
An emerging set of service provider and vendor development goals is A set of service provider and vendor development goals has emerged:
to reduce costs while increasing flexibility of network devices, and reduce costs while increasing flexibility of network devices and
drastically accelerate their deployment. Network Function drastically reduce deployment time. Network Function Virtualization
Virtualization (NFV) has the promise to achieve these goals, and (NFV) has the promise to achieve these goals and therefore has
therefore has garnered much attention. It now seems certain that garnered much attention. It now seems certain that some network
some network functions will be virtualized following the success of functions will be virtualized following the success of cloud
cloud computing and virtual desktops supported by sufficient network computing and virtual desktops supported by sufficient network path
path capacity, performance, and widespread deployment; many of the capacity, performance, and widespread deployment; many of the same
same techniques will help achieve NFV. techniques will help achieve NFV.
In the context of Virtualized Network Functions (VNF), the supporting In the context of Virtual Network Functions (VNFs), the supporting
Infrastructure requires general-purpose computing systems, storage Infrastructure requires general-purpose computing systems, storage
systems, networking systems, virtualization support systems (such as systems, networking systems, virtualization support systems (such as
hypervisors), and management systems for the virtual and physical hypervisors), and management systems for the virtual and physical
resources. There will be many potential suppliers of Infrastructure resources. There will be many potential suppliers of Infrastructure
systems and significant flexibility in configuring the systems for systems and significant flexibility in configuring the systems for
best performance. There are also many potential suppliers of VNFs, best performance. There are also many potential suppliers of VNFs,
adding to the combinations possible in this environment. The adding to the combinations possible in this environment. The
separation of hardware and software suppliers has a profound separation of hardware and software suppliers has a profound
implication on benchmarking activities: much more of the internal implication on benchmarking activities: much more of the internal
configuration of the black-box device under test (DUT) must now be configuration of the black-box Device Under Test (DUT) must now be
specified and reported with the results, to foster both repeatability specified and reported with the results, to foster both repeatability
and comparison testing at a later time. and comparison testing at a later time.
Consider the following User Story as further background and Consider the following user story as further background and
motivation: motivation:
"I'm designing and building my NFV Infrastructure platform. The I'm designing and building my NFV Infrastructure platform. The
first steps were easy because I had a small number of categories of first steps were easy because I had a small number of categories
VNFs to support and the VNF vendor gave HW recommendations that I of VNFs to support and the VNF vendor gave hardware
followed. Now I need to deploy more VNFs from new vendors, and there recommendations that I followed. Now I need to deploy more VNFs
are different hardware recommendations. How well will the new VNFs from new vendors, and there are different hardware
perform on my existing hardware? Which among several new VNFs in a recommendations. How well will the new VNFs perform on my
given category are most efficient in terms of capacity they deliver? existing hardware? Which among several new VNFs in a given
And, when I operate multiple categories of VNFs (and PNFs) category are most efficient in terms of capacity they deliver?
*concurrently* on a hardware platform such that they share resources, And, when I operate multiple categories of VNFs (and PNFs)
what are the new performance limits, and what are the software design *concurrently* on a hardware platform such that they share
choices I can make to optimize my chosen hardware platform? resources, what are the new performance limits, and what are the
Conversely, what hardware platform upgrades should I pursue to software design choices I can make to optimize my chosen hardware
increase the capacity of these concurrently operating VNFs?" platform? Conversely, what hardware platform upgrades should I
pursue to increase the capacity of these concurrently operating
VNFs?
See http://www.etsi.org/technologies-clusters/technologies/nfv for See <http://www.etsi.org/technologies-clusters/technologies/nfv> for
more background, for example, the white papers there may be a useful more background; the white papers there may be a useful starting
starting place. The Performance and Portability Best Practices place. The "NFV Performance & Portability Best Practices" document
[NFV.PER001] are particularly relevant to BMWG. There are documents [NFV.PER001] is particularly relevant to BMWG. There are also
available in the Open Area http://docbox.etsi.org/ISG/NFV/Open/ documents available among the Approved ETSI NFV Specifications
Latest_Drafts/ including drafts describing Infrastructure aspects and [Approved_ETSI_NFV], including documents describing Infrastructure
service quality. performance aspects and service quality metrics, and drafts in the
ETSI NFV Open Area [Draft_ETSI_NFV], which may also have relevance to
benchmarking.
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2. Scope 2. Scope
At the time of this writing, BMWG is considering the new topic of At the time of this writing, BMWG is considering the new topic of
Virtual Network Functions and related Infrastructure to ensure that Virtual Network Functions and related Infrastructure to ensure that
common issues are recognized from the start, using background common issues are recognized from the start; background materials
materials from industry and SDOs (e.g., IETF, ETSI NFV). from respective standards development organizations and Open Source
development projects (e.g., IETF, ETSI NFV, and the Open Platform for
Network Function Virtualization (OPNFV)) are being used.
This memo investigates additional methodological considerations This memo investigates additional methodological considerations
necessary when benchmarking VNFs instantiated and hosted in general- necessary when benchmarking VNFs instantiated and hosted in general-
purpose hardware, using bare metal hypervisors [BareMetal] or other purpose hardware, using bare metal hypervisors [BareMetal] or other
isolation environments such as Linux containers. An essential isolation environments such as Linux containers. An essential
consideration is benchmarking physical and virtual network functions consideration is benchmarking physical and Virtual Network Functions
in the same way when possible, thereby allowing direct comparison. in the same way when possible, thereby allowing direct comparison.
Benchmarking combinations of physical and virtual devices and Benchmarking combinations of physical and virtual devices and
functions in a System Under Test is another topic of keen interest. functions in a System Under Test (SUT) is another topic of keen
interest.
A clearly related goal: the benchmarks for the capacity of a general- A clearly related goal is investigating benchmarks for the capacity
purpose platform to host a plurality of VNF instances should be of a general-purpose platform to host a plurality of VNF instances.
investigated. Existing networking technology benchmarks will also be Existing networking technology benchmarks will also be considered for
considered for adaptation to NFV and closely associated technologies. adaptation to NFV and closely associated technologies.
A non-goal is any overlap with traditional computer benchmark A non-goal is any overlap with traditional computer benchmark
development and their specific metrics (SPECmark suites such as development and their specific metrics (e.g., SPECmark suites such as
SPECCPU). SPEC CPU).
A continued non-goal is any form of architecture development related A continued non-goal is any form of architecture development related
to NFV and associated technologies in BMWG, consistent with all to NFV and associated technologies in BMWG, consistent with all
chartered work since BMWG began in 1989. chartered work since BMWG began in 1989.
3. Considerations for Hardware and Testing 3. Considerations for Hardware and Testing
This section lists the new considerations which must be addressed to This section lists the new considerations that must be addressed to
benchmark VNF(s) and their supporting infrastructure. The System benchmark VNF(s) and their supporting Infrastructure. The SUT is
Under Test (SUT) is composed of the hardware platform components, the composed of the hardware platform components, the VNFs installed, and
VNFs installed, and many other supporting systems. It is critical to many other supporting systems. It is critical to document all
document all aspects of the SUT to foster repeatability. aspects of the SUT to foster repeatability.
3.1. Hardware Components 3.1. Hardware Components
New Hardware components will become part of the test set-up. The following new hardware components will become part of the test
setup:
1. High volume server platforms (general-purpose, possibly with 1. High-volume server platforms (general-purpose, possibly with
virtual technology enhancements). virtual technology enhancements)
2. Storage systems with large capacity, high speed, and high 2. Storage systems with large capacity, high speed, and high
reliability. reliability
3. Network Interface ports specially designed for efficient service 3. Network interface ports specially designed for efficient service
of many virtual NICs. of many virtual Network Interface Cards (NICs)
4. High capacity Ethernet Switches. 4. High-capacity Ethernet switches
The components above are subjects for development of specialized The components above are subjects for development of specialized
benchmarks which are focused on the special demands of network benchmarks that focus on the special demands of network function
function deployment. deployment.
Labs conducting comparisons of different VNFs may be able to use the Labs conducting comparisons of different VNFs may be able to use the
same hardware platform over many studies, until the steady march of same hardware platform over many studies, until the steady march of
innovations overtakes their capabilities (as happens with the lab's innovations overtakes their capabilities (as happens with the lab's
traffic generation and testing devices today). traffic generation and testing devices today).
3.2. Configuration Parameters 3.2. Configuration Parameters
It will be necessary to configure and document the settings for the It will be necessary to configure and document the settings for the
entire general-purpose platform to ensure repeatability and foster entire general-purpose platform to ensure repeatability and foster
future comparisons, including but clearly not limited-to the future comparisons, including, but clearly not limited to, the
following: following:
o number of server blades (shelf occupation) o number of server blades (shelf occupation)
o CPUs o CPUs
o caches o caches
o memory o memory
o storage system o storage system
o I/O o I/O
as well as configurations that support the devices which host the VNF as well as configurations that support the devices that host the VNF
itself: itself:
o Hypervisor (or other forms of virtual function hosting) o Hypervisor (or other forms of virtual function hosting)
o Virtual Machine (VM) o Virtual Machine (VM)
o Infrastructure Virtual Network (which interconnects Virtual o Infrastructure virtual network (which interconnects virtual
Machines with physical network interfaces, or with each other machines with physical network interfaces or with each other
through virtual switches, for example) through virtual switches, for example)
and finally, the VNF itself, with items such as: and finally, the VNF itself, with items such as:
o specific function being implemented in VNF o specific function being implemented in VNF
o reserved resources for each function (e.g., CPU pinning and Non- o reserved resources for each function (e.g., CPU pinning and Non-
Uniform Memory Access, NUMA node assignment) Uniform Memory Access (NUMA) node assignment)
o number of VNFs (or sub-VNF components, each with its own VM) in o number of VNFs (or sub-VNF components, each with its own VM) in
the service function chain (see section 1.1 of [RFC7498] for a the service function chain (see Section 1.1 of [RFC7498] for a
definition of service function chain) definition of service function chain)
o number of physical interfaces and links transited in the service o number of physical interfaces and links transited in the service
function chain function chain
In the physical device benchmarking context, most of the In the physical device benchmarking context, most of the
corresponding infrastructure configuration choices were determined by corresponding Infrastructure configuration choices were determined by
the vendor. Although the platform itself is now one of the the vendor. Although the platform itself is now one of the
configuration variables, it is important to maintain emphasis on the configuration variables, it is important to maintain emphasis on the
networking benchmarks and capture the platform variables as input networking benchmarks and capture the platform variables as input
factors. factors.
3.3. Testing Strategies 3.3. Testing Strategies
The concept of characterizing performance at capacity limits may The concept of characterizing performance at capacity limits may
change. For example: change. For example:
1. It may be more representative of system capacity to characterize 1. It may be more representative of system capacity to characterize
the case where Virtual Machines (VM, hosting the VNF) are the case where the VMs hosting the VNFs are operating at 50%
operating at 50% Utilization, and therefore sharing the "real" utilization and therefore sharing the "real" processing power
processing power across many VMs. across many VMs.
2. Another important case stems from the need for partitioning 2. Another important test case stems from the need to partition (or
functions. A noisy neighbor (VM hosting a VNF in an infinite isolate) network functions. A noisy neighbor (VM hosting a VNF
loop) would ideally be isolated and the performance of other VMs in an infinite loop) would ideally be isolated; the performance
would continue according to their specifications. of other VMs would continue according to their specifications,
and tests would evaluate the degree of isolation.
3. System errors will likely occur as transients, implying a 3. System errors will likely occur as transients, implying a
distribution of performance characteristics with a long tail distribution of performance characteristics with a long tail
(like latency), leading to the need for longer-term tests of each (like latency) and leading to the need for longer-term tests of
set of configuration and test parameters. each set of configuration and test parameters.
4. The desire for elasticity and flexibility among network functions 4. The desire for elasticity and flexibility among network functions
will include tests where there is constant flux in the number of will include tests where there is constant flux in the number of
VM instances, the resources the VMs require, and the set-up/tear- VM instances, the resources the VMs require, and the setup/
down of network paths that support VM connectivity. Requests for teardown of network paths that support VM connectivity. Requests
and instantiation of new VMs, along with Releases for VMs hosting for and instantiation of new VMs, along with releases for VMs
VNFs that are no longer needed would be an normal operational hosting VNFs that are no longer needed, would be a normal
condition. In other words, benchmarking should include scenarios operational condition. In other words, benchmarking should
with production life cycle management of VMs and their VNFs and include scenarios with production life-cycle management of VMs
network connectivity in-progress, including VNF scaling up/down and their VNFs and network connectivity in progress, including
operations, as well as static configurations. VNF scaling up/down operations, as well as static configurations.
5. All physical things can fail, and benchmarking efforts can also 5. All physical things can fail, and benchmarking efforts can also
examine recovery aided by the virtual architecture with different examine recovery aided by the virtual architecture with different
approaches to resiliency. approaches to resiliency.
6. The sheer number of test conditionas and configuration 6. The sheer number of test conditions and configuration
combinations encourage increased efficiency, including automated combinations encourage increased efficiency, including automated
testing arrangements, combination sub-sampling through an testing arrangements, combination sub-sampling through an
understanding of inter-relationships, and machine-readable test understanding of inter-relationships, and machine-readable test
results. results.
3.4. Attention to Shared Resources 3.4. Attention to Shared Resources
Since many components of the new NFV Infrastructure are virtual, test Since many components of the new NFV Infrastructure are virtual, test
set-up design must have prior knowledge of inter-actions/dependencies setup design must have prior knowledge of interactions/dependencies
within the various resource domains in the System Under Test (SUT). within the various resource domains in the SUT. For example, a
For example, a virtual machine performing the role of a traditional virtual machine performing the role of a traditional tester function,
tester function such as generating and/or receiving traffic should such as generating and/or receiving traffic, should avoid sharing any
avoid sharing any SUT resources with the Device Under Test DUT. SUT resources with the DUT. Otherwise, the results will have
Otherwise, the results will have unexpected dependencies not unexpected dependencies not encountered in physical device
encountered in physical device benchmarking. benchmarking.
Note: The term "tester" has traditionally referred to devices Note that the term "tester" has traditionally referred to devices
dedicated to testing in BMWG literature. In this new context, dedicated to testing in BMWG literature. In this new context,
"tester" additionally refers to functions dedicated to testing, which "tester" additionally refers to functions dedicated to testing, which
may be either virtual or physical. "Tester" has never referred to may be either virtual or physical. "Tester" has never referred to
the individuals performing the tests. the individuals performing the tests.
The shared-resource aspect of test design remains one of the critical The possibility to use shared resources in test design while
challenges to overcome in a way to produce useful results. producing useful results remains one of the critical challenges to
Benchmarking set-ups may designate isolated resources for the DUT and overcome. Benchmarking setups may designate isolated resources for
other critical support components (such as the host/kernel) as the the DUT and other critical support components (such as the host/
first baseline step, and add other loading processes. The added kernel) as the first baseline step and add other loading processes.
complexity of each set-up leads to shared-resource testing scenarios, The added complexity of each setup leads to shared-resource testing
where the characteristics of the competing load (in terms of memory, scenarios, where the characteristics of the competing load (in terms
storage, and CPU utilization) will directly affect the benchmarking of memory, storage, and CPU utilization) will directly affect the
results (and variability of the results), but the results should benchmarking results (and variability of the results), but the
reconcile with the baseline. results should reconcile with the baseline.
The physical test device remains a solid foundation to compare with The physical test device remains a solid foundation to compare with
results using combinations of physical and virtual test functions, or results using combinations of physical and virtual test functions or
results using only virtual testers when necessary to assess virtual results using only virtual testers when necessary to assess virtual
interfaces and other virtual functions. interfaces and other virtual functions.
4. Benchmarking Considerations 4. Benchmarking Considerations
This section discusses considerations related to Benchmarks This section discusses considerations related to benchmarks
applicable to VNFs and their associated technologies. applicable to VNFs and their associated technologies.
4.1. Comparison with Physical Network Functions 4.1. Comparison with Physical Network Functions
In order to compare the performance of VNFs and system In order to compare the performance of VNFs and system
implementations with their physical counterparts, identical implementations with their physical counterparts, identical
benchmarks must be used. Since BMWG has already developed benchmarks must be used. Since BMWG has already developed
specifications for many network functions, there will be re-use of specifications for many network functions, there will be re-use of
existing benchmarks through references, while allowing for the existing benchmarks through references, while allowing for the
possibility of benchmark curation during development of new possibility of benchmark curation during development of new
methodologies. Consideration should be given to quantifying the methodologies. Consideration should be given to quantifying the
number of parallel VNFs required to achieve comparable scale/capacity number of parallel VNFs required to achieve comparable scale/capacity
with a given physical device, or whether some limit of scale was with a given physical device or whether some limit of scale was
reached before the VNFs could achieve the comparable level. Again, reached before the VNFs could achieve the comparable level. Again,
implementation based-on different hypervisors or other virtual implementation based on different hypervisors or other virtual
function hosting remain as critical factors in performance function hosting remain as critical factors in performance
assessment. assessment.
4.2. Continued Emphasis on Black-Box Benchmarks 4.2. Continued Emphasis on Black-Box Benchmarks
When the network functions under test are based on Open Source code, When the network functions under test are based on open-source code,
there may be a tendency to rely on internal measurements to some there may be a tendency to rely on internal measurements to some
extent, especially when the externally-observable phenomena only extent, especially when the externally observable phenomena only
support an inference of internal events (such as routing protocol support an inference of internal events (such as routing protocol
convergence observed in the dataplane). Examples include CPU/Core convergence observed in the data plane). Examples include CPU/Core
utilization, Network utilization, Storage utilization, and Memory utilization, network utilization, storage utilization, and memory
Comitted/used. These "white-box" metrics provide one view of the committed/used. These "white-box" metrics provide one view of the
resource footprint of a VNF. Note: The resource utilization metrics resource footprint of a VNF. Note that the resource utilization
do not easily match the 3x4 Matrix, described in Section 4.4 below. metrics do not easily match the 3x4 Matrix, described in Section 4.4.
However, external observations remain essential as the basis for However, external observations remain essential as the basis for
Benchmarks. Internal observations with fixed specification and benchmarks. Internal observations with fixed specification and
interpretation may be provided in parallel (as auxilliary metrics), interpretation may be provided in parallel (as auxiliary metrics), to
to assist the development of operations procedures when the assist the development of operations procedures when the technology
technology is deployed, for example. Internal metrics and is deployed, for example. Internal metrics and measurements from
measurements from Open Source implementations may be the only direct open-source implementations may be the only direct source of
source of performance results in a desired dimension, but performance results in a desired dimension, but corroborating
corroborating external observations are still required to assure the external observations are still required to assure the integrity of
integrity of measurement discipline was maintained for all reported measurement discipline was maintained for all reported results.
results.
A related aspect of benchmark development is where the scope includes A related aspect of benchmark development is where the scope includes
multiple approaches to a common function under the same benchmark. multiple approaches to a common function under the same benchmark.
For example, there are many ways to arrange for activation of a For example, there are many ways to arrange for activation of a
network path between interface points and the activation times can be network path between interface points, and the activation times can
compared if the start-to-stop activation interval has a generic and be compared if the start-to-stop activation interval has a generic
unambiguous definition. Thus, generic benchmark definitions are and unambiguous definition. Thus, generic benchmark definitions are
preferred over technology/protocol specific definitions where preferred over technology/protocol-specific definitions where
possible. possible.
4.3. New Benchmarks and Related Metrics 4.3. New Benchmarks and Related Metrics
There will be new classes of benchmarks needed for network design and There will be new classes of benchmarks needed for network design and
assistance when developing operational practices (possibly automated assistance when developing operational practices (possibly automated
management and orchestration of deployment scale). Examples follow management and orchestration of deployment scale). Examples follow
in the paragraphs below, many of which are prompted by the goals of in the paragraphs below, many of which are prompted by the goals of
increased elasticity and flexibility of the network functions, along increased elasticity and flexibility of the network functions, along
with accelerated deployment times. with reduced deployment times.
o Time to deploy VNFs: In cases where the general-purpose hardware o Time to deploy VNFs: In cases where the general-purpose hardware
is already deployed and ready for service, it is valuable to know is already deployed and ready for service, it is valuable to know
the response time when a management system is tasked with the response time when a management system is tasked with
"standing-up" 100's of virtual machines and the VNFs they will "standing up" 100s of virtual machines and the VNFs they will
host. host.
o Time to migrate VNFs: In cases where a rack or shelf of hardware o Time to migrate VNFs: In cases where a rack or shelf of hardware
must be removed from active service, it is valuable to know the must be removed from active service, it is valuable to know the
response time when a management system is tasked with "migrating" response time when a management system is tasked with "migrating"
some number of virtual machines and the VNFs they currently host some number of virtual machines and the VNFs they currently host
to alternate hardware that will remain in-service. to alternate hardware that will remain in service.
o Time to create a virtual network in the general-purpose o Time to create a virtual network in the general-purpose
infrastructure: This is a somewhat simplified version of existing Infrastructure: This is a somewhat simplified version of existing
benchmarks for convergence time, in that the process is initiated benchmarks for convergence time, in that the process is initiated
by a request from (centralized or distributed) control, rather by a request from (centralized or distributed) control, rather
than inferred from network events (link failure). The successful than inferred from network events (link failure). The successful
response time would remain dependent on dataplane observations to response time would remain dependent on data-plane observations to
confirm that the network is ready to perform. confirm that the network is ready to perform.
o Effect of verification measurements on performance: A complete o Effect of verification measurements on performance: A complete
VNF, or something as simple as a new policy to implement in a VNF, VNF, or something as simple as a new policy to implement in a VNF,
is implemented. The action to verify instantiation of the VNF or is implemented. The action to verify instantiation of the VNF or
policy could affect performance during normal operation. policy could affect performance during normal operation.
Also, it appears to be valuable to measure traditional packet Also, it appears to be valuable to measure traditional packet
transfer performance metrics during the assessment of traditional and transfer performance metrics during the assessment of traditional and
new benchmarks, including metrics that may be used to support service new benchmarks, including metrics that may be used to support service
engineering such as the Spatial Composition metrics found in engineering such as the spatial composition metrics found in
[RFC6049]. Examples include Mean one-way delay in section 4.1 of [RFC6049]. Examples include mean one-way delay in Section 4.1 of
[RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet [RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet
Reordering [RFC4737] [RFC4689]. Reordering [RFC4737] [RFC4689].
4.4. Assessment of Benchmark Coverage 4.4. Assessment of Benchmark Coverage
It can be useful to organize benchmarks according to their applicable It can be useful to organize benchmarks according to their applicable
life cycle stage and the performance criteria they were designed to life-cycle stage and the performance criteria they were designed to
assess. The table below (derived from [X3.102]) provides a way to assess. The table below (derived from [X3.102]) provides a way to
organize benchmarks such that there is a clear indication of coverage organize benchmarks such that there is a clear indication of coverage
for the intersection of life cycle stages and performance criteria. for the intersection of life-cycle stages and performance criteria.
|----------------------------------------------------------| |----------------------------------------------------------|
| | | | | | | | | |
| | SPEED | ACCURACY | RELIABILITY | | | SPEED | ACCURACY | RELIABILITY |
| | | | | | | | | |
|----------------------------------------------------------| |----------------------------------------------------------|
| | | | | | | | | |
| Activation | | | | | Activation | | | |
| | | | | | | | | |
|----------------------------------------------------------| |----------------------------------------------------------|
skipping to change at page 10, line 28 skipping to change at page 10, line 36
| | | | | | | | | |
| De-activation | | | | | De-activation | | | |
| | | | | | | | | |
|----------------------------------------------------------| |----------------------------------------------------------|
For example, the "Time to deploy VNFs" benchmark described above For example, the "Time to deploy VNFs" benchmark described above
would be placed in the intersection of Activation and Speed, making would be placed in the intersection of Activation and Speed, making
it clear that there are other potential performance criteria to it clear that there are other potential performance criteria to
benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups" benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups"
in a set of 100 attempts. This example emphasizes that the in a set of 100 attempts. This example emphasizes that the
Activation and De-activation life cycle stages are key areas for NFV Activation and De-activation life-cycle stages are key areas for NFV
and related infrastructure, and encourage expansion beyond and related Infrastructure and encourages expansion beyond
traditional benchmarks for normal operation. Thus, reviewing the traditional benchmarks for normal operation. Thus, reviewing the
benchmark coverage using this table (sometimes called the 3x3 matrix) benchmark coverage using this table (sometimes called the 3x3 Matrix)
can be a worthwhile exercise in BMWG. can be a worthwhile exercise in BMWG.
In one of the first applications of the 3x3 matrix in BMWG In one of the first applications of the 3x3 Matrix in BMWG
[I-D.ietf-bmwg-sdn-controller-benchmark-meth], we discovered that [SDN-BENCHMARK], we discovered that metrics on measured size,
metrics on measured size, capacity, or scale do not easily match one capacity, or scale do not easily match one of the three columns
of the three columns above. Following discussion, this was resolved above. Following discussion, this was resolved in two ways:
in two ways:
o Add a column, Scale, for use when categorizing and assessing the o Add a column, Scale, for use when categorizing and assessing the
coverage of benchmarks (without measured results). Examples of coverage of benchmarks (without measured results). An example of
this use are found in this use is found in [OPNFV-BENCHMARK] (and a variation may be
[I-D.ietf-bmwg-sdn-controller-benchmark-meth] and found in [SDN-BENCHMARK]). This is the 3x4 Matrix.
[I-D.vsperf-bmwg-vswitch-opnfv]. This is the 3x4 Matrix.
o If using the matrix to report results in an organized way, keep o If using the matrix to report results in an organized way, keep
size, capacity, and scale metrics separate from the 3x3 matrix and size, capacity, and scale metrics separate from the 3x3 Matrix and
incorporate them in the report with other qualifications of the incorporate them in the report with other qualifications of the
results. results.
Note: The resource utilization (e.g., CPU) metrics do not fit in the Note that the resource utilization (e.g., CPU) metrics do not fit in
Matrix. They are not benchmarks, and omitting them confirms their the matrix. They are not benchmarks, and omitting them confirms
status as auxilliary metrics. Resource assignments are configuration their status as auxiliary metrics. Resource assignments are
parameters, and these are reported seperately. configuration parameters, and these are reported separately.
This approach encourages use of the 3x3 matrix to organize reports of This approach encourages use of the 3x3 Matrix to organize reports of
results, where the capacity at which the various metrics were results, where the capacity at which the various metrics were
measured could be included in the title of the matrix (and results measured could be included in the title of the matrix (and results
for multiple capacities would result in separate 3x3 matrices, if for multiple capacities would result in separate 3x3 Matrices, if
there were sufficient measurements/results to organize in that way). there were sufficient measurements/results to organize in that way).
For example, results for each VM and VNF could appear in the 3x3 For example, results for each VM and VNF could appear in the 3x3
matrix, organized to illustrate resource occupation (CPU Cores) in a Matrix, organized to illustrate resource occupation (CPU Cores) in a
particular physical computing system, as shown below. particular physical computing system, as shown below.
VNF#1 VNF#1
.-----------. .-----------.
|__|__|__|__| |__|__|__|__|
Core 1 |__|__|__|__| Core 1 |__|__|__|__|
|__|__|__|__| |__|__|__|__|
| | | | | | | | | |
'-----------' '-----------'
VNF#2 VNF#2
skipping to change at page 11, line 48 skipping to change at page 12, line 7
VNF#6 VNF#6
.-----------. .-----------.
|__|__|__|__| |__|__|__|__|
Core 7 |__|__|__|__| Core 7 |__|__|__|__|
|__|__|__|__| |__|__|__|__|
| | | | | | | | | |
'-----------' '-----------'
The combination of tables above could be built incrementally, The combination of tables above could be built incrementally,
beginning with VNF#1 and one Core, then adding VNFs according to beginning with VNF#1 and one Core, then adding VNFs according to
their supporting core assignments. X-Y plots of critical benchmarks their supporting Core assignments. X-Y plots of critical benchmarks
would also provide insight to the effect of increased HW utilization. would also provide insight to the effect of increased hardware
All VNFs might be of the same type, or to match a production utilization. All VNFs might be of the same type, or to match a
environment there could be VNFs of multiple types and categories. In production environment, there could be VNFs of multiple types and
this figure, VNFs #3-#5 are assumed to require small CPU resources, categories. In this figure, VNFs #3-#5 are assumed to require small
while VNF#2 requires 4 cores to perform its function. CPU resources, while VNF#2 requires four Cores to perform its
function.
4.5. Power Consumption 4.5. Power Consumption
Although there is incomplete work to benchmark physical network Although there is incomplete work to benchmark physical network
function power consumption in a meaningful way, the desire to measure function power consumption in a meaningful way, the desire to measure
the physical infrastructure supporting the virtual functions only the physical Infrastructure supporting the virtual functions only
adds to the need. Both maximum power consumption and dynamic power adds to the need. Both maximum power consumption and dynamic power
consumption (with varying load) would be useful. The IPMI standard consumption (with varying load) would be useful. The Intelligent
[IPMI2.0] has been implemented by many manufacturers, and supports Platform Management Interface (IPMI) standard [IPMI2.0] has been
measurement of instantaneous energy consumption. implemented by many manufacturers and supports measurement of
instantaneous energy consumption.
To assess the instantaneous energy consumption of virtual resources, To assess the instantaneous energy consumption of virtual resources,
it may be possible to estimate the value using an overall metric it may be possible to estimate the value using an overall metric
based on utilization readings, according to based on utilization readings, according to [NFVIaas-FRAMEWORK].
[I-D.krishnan-nfvrg-policy-based-rm-nfviaas].
5. Security Considerations 5. Security Considerations
Benchmarking activities as described in this memo are limited to Benchmarking activities as described in this memo are limited to
technology characterization of a Device Under Test/System Under Test technology characterization of a DUT/SUT using controlled stimuli in
(DUT/SUT) using controlled stimuli in a laboratory environment, with a laboratory environment, with dedicated address space and the
dedicated address space and the constraints specified in the sections constraints specified in the sections above.
above.
The benchmarking network topology will be an independent test setup The benchmarking network topology will be an independent test setup
and MUST NOT be connected to devices that may forward the test and MUST NOT be connected to devices that may forward the test
traffic into a production network, or misroute traffic to the test traffic into a production network or misroute traffic to the test
management network. management network.
Further, benchmarking is performed on a "black-box" basis, relying Further, benchmarking is performed on a "black-box" basis, relying
solely on measurements observable external to the DUT/SUT. solely on measurements observable external to the DUT/SUT.
Special capabilities SHOULD NOT exist in the DUT/SUT specifically for Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
benchmarking purposes. Any implications for network security arising benchmarking purposes. Any implications for network security arising
from the DUT/SUT SHOULD be identical in the lab and in production from the DUT/SUT SHOULD be identical in the lab and in production
networks. networks.
6. IANA Considerations 6. IANA Considerations
No IANA Action is requested at this time. This document does not require any IANA actions.
7. Acknowledgements
The author acknowledges an encouraging conversation on this topic
with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhavani
Parise and Ilya Varlashkin have provided useful suggestions to expand
these considerations. Bhuvaneswaran Vengainathan has already tried
the 3x3 matrix with SDN controller draft, and contributed to many
discussions. Scott Bradner quickly pointed out shared resource
dependencies in an early vSwitch measurement proposal, and the topic
was included here as a key consideration. Further development was
encouraged by Barry Constantine's comments following the IETF-92 BMWG
session: the session itself was an affirmation for this memo. There
have been many interesting contributions from Maryam Tahhan, Marius
Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others.
8. Version history
(This section should be removed by the RFC Editor.)
version 05: Address IESG & Last Call Comments (editorial)
Version 03 & 04: address mininal comments and few WGLC comments
Version 02:
New version history section.
Added Memory in section 3.2, configuration.
Updated ACKs and References.
Version 01:
Addressed Ramki Krishnan's comments on section 4.5, power, see that
section (7/27 message to the list). Addressed Saurabh
Chattopadhyay's 7/24 comments on VNF resources and other resource
conditions and their effect on benchmarking, see section 3.4.
Addressed Marius Georgescu's 7/17 comments on the list (sections 4.3
and 4.4).
AND, comments from the extended discussion during IETF-93 BMWG
session:
Section 4.2: VNF footprint and auxilliary metrics (Maryam Tahhan), 7. References
Section 4.3: Verification affect metrics (Ramki Krishnan);
Section 4.4: Auxilliary metrics in the Matrix (Maryam Tahhan, Scott
Bradner, others)
9. References 7.1. Normative References
9.1. Normative References
[NFV.PER001] [NFV.PER001]
"Network Function Virtualization: Performance and ETSI, "Network Function Virtualization: Performance &
Portability Best Practices", Group Specification ETSI GS Portability Best Practices", ETSI GS NFV-PER 001, V1.1.2,
NFV-PER 001 V1.1.1 (2014-06), June 2014. December 2014.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. <http://www.rfc-editor.org/info/rfc2119>.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, Network Interconnect Devices", RFC 2544,
DOI 10.17487/RFC2544, March 1999, DOI 10.17487/RFC2544, March 1999,
<http://www.rfc-editor.org/info/rfc2544>. <http://www.rfc-editor.org/info/rfc2544>.
skipping to change at page 14, line 36 skipping to change at page 13, line 43
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
S., and J. Perser, "Packet Reordering Metrics", RFC 4737, S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
DOI 10.17487/RFC4737, November 2006, DOI 10.17487/RFC4737, November 2006,
<http://www.rfc-editor.org/info/rfc4737>. <http://www.rfc-editor.org/info/rfc4737>.
[RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for
Service Function Chaining", RFC 7498, Service Function Chaining", RFC 7498,
DOI 10.17487/RFC7498, April 2015, DOI 10.17487/RFC7498, April 2015,
<http://www.rfc-editor.org/info/rfc7498>. <http://www.rfc-editor.org/info/rfc7498>.
9.2. Informative References [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <http://www.rfc-editor.org/info/rfc8174>.
7.2. Informative References
[Approved_ETSI_NFV]
ETSI, Network Functions Virtualisation Technical
Committee, "ETSI NFV",
<http://www.etsi.org/standards-search>.
[BareMetal] [BareMetal]
Popek, Gerald J.; Goldberg, Robert P. , , "Formal Popek, G. and R. Goldberg, "Formal requirements for
requirements for virtualizable third generation virtualizable third generation architectures",
architectures". Communications of the ACM. 17 (7): Communications of the ACM, Volume 17, Issue 7, Pages
412-421. doi:10.1145/361011.361073.", 1974. 412-421, DOI 10.1145/361011.361073, July 1974.
[I-D.ietf-bmwg-sdn-controller-benchmark-meth] [Draft_ETSI_NFV]
Vengainathan, B., Basil, A., Tassinari, M., Manral, V., ETSI, "Network Functions Virtualisation: Specifications",
and S. Banks, "Benchmarking Methodology for SDN Controller <http://www.etsi.org/technologies-clusters/technologies/
Performance", draft-ietf-bmwg-sdn-controller-benchmark- nfv>.
meth-03 (work in progress), January 2017.
[I-D.krishnan-nfvrg-policy-based-rm-nfviaas] [IPMI2.0] Intel Corporation, Hewlett-Packard Company, NEC
Corporation, and Dell Inc., "Intelligent Platform
Management Interface Specification v2.0 (with latest
errata)", April 2015,
<http://www.intel.com/content/dam/www/public/us/en/
documents/specification-updates/ipmi-intelligent-platform-
mgt-interface-spec-2nd-gen-v2-0-spec-update.pdf>.
[NFVIaas-FRAMEWORK]
Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D.,
Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra, Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra,
"NFVIaaS Architectural Framework for Policy Based Resource "NFVIaaS Architectural Framework for Policy Based Resource
Placement and Scheduling", draft-krishnan-nfvrg-policy- Placement and Scheduling", Work in Progress,
based-rm-nfviaas-06 (work in progress), March 2016. draft-krishnan-nfvrg-policy-based-rm-nfviaas-06, March
2016.
[I-D.vsperf-bmwg-vswitch-opnfv] [OPNFV-BENCHMARK]
Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking
Virtual Switches in OPNFV", draft-vsperf-bmwg-vswitch- Virtual Switches in OPNFV", Work in Progress,
opnfv-02 (work in progress), March 2016. draft-ietf-bmwg-vswitch-opnfv-04, June 2017.
[IPMI2.0] "Intelligent Platform Management Interface, v2.0 with
latest Errata",
http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-
intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec-
update.html, April 2015.
[RFC1242] Bradner, S., "Benchmarking Terminology for Network [RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
July 1991, <http://www.rfc-editor.org/info/rfc1242>. July 1991, <http://www.rfc-editor.org/info/rfc1242>.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
March 2009, <http://www.rfc-editor.org/info/rfc5481>. March 2009, <http://www.rfc-editor.org/info/rfc5481>.
[RFC6049] Morton, A. and E. Stephan, "Spatial Composition of [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of
Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011,
<http://www.rfc-editor.org/info/rfc6049>. <http://www.rfc-editor.org/info/rfc6049>.
[X3.102] ANSI X3.102, , "ANSI Standard on Data Communications, [SDN-BENCHMARK]
User-Oriented Data Communications Framework", 1983. Vengainathan, B., Basil, A., Tassinari, M., Manral, V.,
and S. Banks, "Terminology for Benchmarking SDN Controller
Performance", Work in Progress, draft-ietf-bmwg-sdn-
controller-benchmark-term-04, June 2017.
[X3.102] ANSI, "Information Systems - Data Communication Systems
and Services - User-Oriented Performance Parameters
Communications Framework", ANSI X3.102, 1983.
Acknowledgements
The author acknowledges an encouraging conversation on this topic
with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhavani
Parise and Ilya Varlashkin have provided useful suggestions to expand
these considerations. Bhuvaneswaran Vengainathan has already tried
the 3x3 Matrix with the SDN controller document and contributed to
many discussions. Scott Bradner quickly pointed out shared resource
dependencies in an early vSwitch measurement proposal, and the topic
was included here as a key consideration. Further development was
encouraged by Barry Constantine's comments following the BMWG session
at IETF 92: the session itself was an affirmation for this memo.
There have been many interesting contributions from Maryam Tahhan,
Marius Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others.
Author's Address Author's Address
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown,, NJ 07748 Middletown, NJ 07748
USA United States of America
Phone: +1 732 420 1571 Phone: +1 732 420 1571
Fax: +1 732 368 1192 Fax: +1 732 368 1192
Email: acmorton@att.com Email: acmorton@att.com
URI: http://home.comcast.net/~acmacm/
 End of changes. 89 change blocks. 
287 lines changed or deleted 279 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/