draft-ietf-bmwg-virtual-net-04.txt   draft-ietf-bmwg-virtual-net-05.txt 
Network Working Group A. Morton Network Working Group A. Morton
Internet-Draft AT&T Labs Internet-Draft AT&T Labs
Intended status: Informational August 14, 2016 Intended status: Informational March 16, 2017
Expires: February 15, 2017 Expires: September 17, 2017
Considerations for Benchmarking Virtual Network Functions and Their Considerations for Benchmarking Virtual Network Functions and Their
Infrastructure Infrastructure
draft-ietf-bmwg-virtual-net-04 draft-ietf-bmwg-virtual-net-05
Abstract Abstract
Benchmarking Methodology Working Group has traditionally conducted The Benchmarking Methodology Working Group has traditionally
laboratory characterization of dedicated physical implementations of conducted laboratory characterization of dedicated physical
internetworking functions. This memo investigates additional implementations of internetworking functions. This memo investigates
considerations when network functions are virtualized and performed additional considerations when network functions are virtualized and
in general purpose hardware. performed in general purpose hardware.
See the new version history section for updates.
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 1, line 43 skipping to change at page 1, line 41
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 15, 2017. This Internet-Draft will expire on September 17, 2017.
Copyright Notice Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Considerations for Hardware and Testing . . . . . . . . . . . 4 3. Considerations for Hardware and Testing . . . . . . . . . . . 4
3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4 3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4
3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5 3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5
3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6 3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6
3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7 3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 7 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 7
4.1. Comparison with Physical Network Functions . . . . . . . 8 4.1. Comparison with Physical Network Functions . . . . . . . 7
4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8 4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8
4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 9 4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 8
4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 9 4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 9
4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12 4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12
5. Security Considerations . . . . . . . . . . . . . . . . . . . 12 5. Security Considerations . . . . . . . . . . . . . . . . . . . 12
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12
8. Version history . . . . . . . . . . . . . . . . . . . . . . . 13 8. Version history . . . . . . . . . . . . . . . . . . . . . . . 13
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 14 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
9.1. Normative References . . . . . . . . . . . . . . . . . . 14 9.1. Normative References . . . . . . . . . . . . . . . . . . 14
9.2. Informative References . . . . . . . . . . . . . . . . . 14 9.2. Informative References . . . . . . . . . . . . . . . . . 14
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15
1. Introduction 1. Introduction
Benchmarking Methodology Working Group (BMWG) has traditionally The Benchmarking Methodology Working Group (BMWG) has traditionally
conducted laboratory characterization of dedicated physical conducted laboratory characterization of dedicated physical
implementations of internetworking functions (or physical network implementations of internetworking functions (or physical network
functions, PNFs). The Black-box Benchmarks of Throughput, Latency, functions, PNFs). The Black-box Benchmarks of Throughput, Latency,
Forwarding Rates and others have served our industry for many years. Forwarding Rates and others have served our industry for many years.
[RFC1242] and [RFC2544] are the cornerstones of the work. [RFC1242] and [RFC2544] are the cornerstones of the work.
An emerging set of service provider and vendor development goals is An emerging set of service provider and vendor development goals is
to reduce costs while increasing flexibility of network devices, and to reduce costs while increasing flexibility of network devices, and
drastically accelerate their deployment. Network Function drastically accelerate their deployment. Network Function
Virtualization (NFV) has the promise to achieve these goals, and Virtualization (NFV) has the promise to achieve these goals, and
skipping to change at page 4, line 7 skipping to change at page 3, line 48
See http://www.etsi.org/technologies-clusters/technologies/nfv for See http://www.etsi.org/technologies-clusters/technologies/nfv for
more background, for example, the white papers there may be a useful more background, for example, the white papers there may be a useful
starting place. The Performance and Portability Best Practices starting place. The Performance and Portability Best Practices
[NFV.PER001] are particularly relevant to BMWG. There are documents [NFV.PER001] are particularly relevant to BMWG. There are documents
available in the Open Area http://docbox.etsi.org/ISG/NFV/Open/ available in the Open Area http://docbox.etsi.org/ISG/NFV/Open/
Latest_Drafts/ including drafts describing Infrastructure aspects and Latest_Drafts/ including drafts describing Infrastructure aspects and
service quality. service quality.
2. Scope 2. Scope
BMWG will consider the new topic of Virtual Network Functions and At the time of this writing, BMWG is considering the new topic of
related Infrastructure to ensure that common issues are recognized Virtual Network Functions and related Infrastructure to ensure that
from the start, using background materials from industry and SDOs common issues are recognized from the start, using background
(e.g., IETF, ETSI NFV). materials from industry and SDOs (e.g., IETF, ETSI NFV).
This memo investigates additional methodological considerations This memo investigates additional methodological considerations
necessary when benchmarking VNFs instantiated and hosted in general- necessary when benchmarking VNFs instantiated and hosted in general-
purpose hardware, using bare-metal hypervisors or other isolation purpose hardware, using bare metal hypervisors [BareMetal] or other
environments such as Linux containers. An essential consideration is isolation environments such as Linux containers. An essential
benchmarking physical and virtual network functions in the same way consideration is benchmarking physical and virtual network functions
when possible, thereby allowing direct comparison. Also, in the same way when possible, thereby allowing direct comparison.
benchmarking combinations of physical and virtual devices and Benchmarking combinations of physical and virtual devices and
functions in a System Under Test. functions in a System Under Test is another topic of keen interest.
A clearly related goal: the benchmarks for the capacity of a general- A clearly related goal: the benchmarks for the capacity of a general-
purpose platform to host a plurality of VNF instances should be purpose platform to host a plurality of VNF instances should be
investigated. Existing networking technology benchmarks will also be investigated. Existing networking technology benchmarks will also be
considered for adaptation to NFV and closely associated technologies. considered for adaptation to NFV and closely associated technologies.
A non-goal is any overlap with traditional computer benchmark A non-goal is any overlap with traditional computer benchmark
development and their specific metrics (SPECmark suites such as development and their specific metrics (SPECmark suites such as
SPECCPU). SPECCPU).
skipping to change at page 8, line 31 skipping to change at page 8, line 20
4.2. Continued Emphasis on Black-Box Benchmarks 4.2. Continued Emphasis on Black-Box Benchmarks
When the network functions under test are based on Open Source code, When the network functions under test are based on Open Source code,
there may be a tendency to rely on internal measurements to some there may be a tendency to rely on internal measurements to some
extent, especially when the externally-observable phenomena only extent, especially when the externally-observable phenomena only
support an inference of internal events (such as routing protocol support an inference of internal events (such as routing protocol
convergence observed in the dataplane). Examples include CPU/Core convergence observed in the dataplane). Examples include CPU/Core
utilization, Network utilization, Storage utilization, and Memory utilization, Network utilization, Storage utilization, and Memory
Comitted/used. These "white-box" metrics provide one view of the Comitted/used. These "white-box" metrics provide one view of the
resource footprint of a VNF. Note: The resource utilization metrics resource footprint of a VNF. Note: The resource utilization metrics
do not easily match the 3x4 Matrix. do not easily match the 3x4 Matrix, described in Section 4.4 below.
However, external observations remain essential as the basis for However, external observations remain essential as the basis for
Benchmarks. Internal observations with fixed specification and Benchmarks. Internal observations with fixed specification and
interpretation may be provided in parallel (as auxilliary metrics), interpretation may be provided in parallel (as auxilliary metrics),
to assist the development of operations procedures when the to assist the development of operations procedures when the
technology is deployed, for example. Internal metrics and technology is deployed, for example. Internal metrics and
measurements from Open Source implementations may be the only direct measurements from Open Source implementations may be the only direct
source of performance results in a desired dimension, but source of performance results in a desired dimension, but
corroborating external observations are still required to assure the corroborating external observations are still required to assure the
integrity of measurement discipline was maintained for all reported integrity of measurement discipline was maintained for all reported
skipping to change at page 9, line 35 skipping to change at page 9, line 26
o Time to create a virtual network in the general-purpose o Time to create a virtual network in the general-purpose
infrastructure: This is a somewhat simplified version of existing infrastructure: This is a somewhat simplified version of existing
benchmarks for convergence time, in that the process is initiated benchmarks for convergence time, in that the process is initiated
by a request from (centralized or distributed) control, rather by a request from (centralized or distributed) control, rather
than inferred from network events (link failure). The successful than inferred from network events (link failure). The successful
response time would remain dependent on dataplane observations to response time would remain dependent on dataplane observations to
confirm that the network is ready to perform. confirm that the network is ready to perform.
o Effect of verification measurements on performance: A complete o Effect of verification measurements on performance: A complete
VNF, or something as simple as a new poicy to implement in a VNF, VNF, or something as simple as a new policy to implement in a VNF,
is implemented. The action to verify instantiation of the VNF or is implemented. The action to verify instantiation of the VNF or
policy could affect performance during normal operation. policy could affect performance during normal operation.
Also, it appears to be valuable to measure traditional packet Also, it appears to be valuable to measure traditional packet
transfer performance metrics during the assessment of traditional and transfer performance metrics during the assessment of traditional and
new benchmarks, including metrics that may be used to support service new benchmarks, including metrics that may be used to support service
engineering such as the Spatial Composition metrics found in engineering such as the Spatial Composition metrics found in
[RFC6049]. Examples include Mean one-way delay in section 4.1 of [RFC6049]. Examples include Mean one-way delay in section 4.1 of
[RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet [RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet
Reordering [RFC4737] [RFC4689]. Reordering [RFC4737] [RFC4689].
4.4. Assessment of Benchmark Coverage 4.4. Assessment of Benchmark Coverage
It can be useful to organize benchmarks according to their applicable It can be useful to organize benchmarks according to their applicable
life cycle stage and the performance criteria they were designed to life cycle stage and the performance criteria they were designed to
assess. The table below provides a way to organize benchmarks such assess. The table below (derived from [X3.102]) provides a way to
that there is a clear indication of coverage for the intersection of organize benchmarks such that there is a clear indication of coverage
life cycle stages and performance criteria. for the intersection of life cycle stages and performance criteria.
|----------------------------------------------------------| |----------------------------------------------------------|
| | | | | | | | | |
| | SPEED | ACCURACY | RELIABILITY | | | SPEED | ACCURACY | RELIABILITY |
| | | | | | | | | |
|----------------------------------------------------------| |----------------------------------------------------------|
| | | | | | | | | |
| Activation | | | | | Activation | | | |
| | | | | | | | | |
|----------------------------------------------------------| |----------------------------------------------------------|
skipping to change at page 13, line 24 skipping to change at page 13, line 18
was included here as a key consideration. Further development was was included here as a key consideration. Further development was
encouraged by Barry Constantine's comments following the IETF-92 BMWG encouraged by Barry Constantine's comments following the IETF-92 BMWG
session: the session itself was an affirmation for this memo. There session: the session itself was an affirmation for this memo. There
have been many interesting contributions from Maryam Tahhan, Marius have been many interesting contributions from Maryam Tahhan, Marius
Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others. Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others.
8. Version history 8. Version history
(This section should be removed by the RFC Editor.) (This section should be removed by the RFC Editor.)
version 05: Address IESG & Last Call Comments (editorial)
Version 03 & 04: address mininal comments and few WGLC comments
Version 02: Version 02:
New version history section. New version history section.
Added Memory in section 3.2, configuration. Added Memory in section 3.2, configuration.
Updated ACKs and References. Updated ACKs and References.
Version 01: Version 01:
skipping to change at page 14, line 41 skipping to change at page 14, line 38
DOI 10.17487/RFC4737, November 2006, DOI 10.17487/RFC4737, November 2006,
<http://www.rfc-editor.org/info/rfc4737>. <http://www.rfc-editor.org/info/rfc4737>.
[RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for
Service Function Chaining", RFC 7498, Service Function Chaining", RFC 7498,
DOI 10.17487/RFC7498, April 2015, DOI 10.17487/RFC7498, April 2015,
<http://www.rfc-editor.org/info/rfc7498>. <http://www.rfc-editor.org/info/rfc7498>.
9.2. Informative References 9.2. Informative References
[BareMetal]
Popek, Gerald J.; Goldberg, Robert P. , , "Formal
requirements for virtualizable third generation
architectures". Communications of the ACM. 17 (7):
412-421. doi:10.1145/361011.361073.", 1974.
[I-D.ietf-bmwg-sdn-controller-benchmark-meth] [I-D.ietf-bmwg-sdn-controller-benchmark-meth]
Vengainathan, B., Basil, A., Tassinari, M., Manral, V., Vengainathan, B., Basil, A., Tassinari, M., Manral, V.,
and S. Banks, "Benchmarking Methodology for SDN Controller and S. Banks, "Benchmarking Methodology for SDN Controller
Performance", draft-ietf-bmwg-sdn-controller-benchmark- Performance", draft-ietf-bmwg-sdn-controller-benchmark-
meth-02 (work in progress), July 2016. meth-03 (work in progress), January 2017.
[I-D.krishnan-nfvrg-policy-based-rm-nfviaas] [I-D.krishnan-nfvrg-policy-based-rm-nfviaas]
Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D.,
Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra, Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra,
"NFVIaaS Architectural Framework for Policy Based Resource "NFVIaaS Architectural Framework for Policy Based Resource
Placement and Scheduling", draft-krishnan-nfvrg-policy- Placement and Scheduling", draft-krishnan-nfvrg-policy-
based-rm-nfviaas-06 (work in progress), March 2016. based-rm-nfviaas-06 (work in progress), March 2016.
[I-D.vsperf-bmwg-vswitch-opnfv] [I-D.vsperf-bmwg-vswitch-opnfv]
Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking
skipping to change at page 15, line 28 skipping to change at page 15, line 35
July 1991, <http://www.rfc-editor.org/info/rfc1242>. July 1991, <http://www.rfc-editor.org/info/rfc1242>.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
March 2009, <http://www.rfc-editor.org/info/rfc5481>. March 2009, <http://www.rfc-editor.org/info/rfc5481>.
[RFC6049] Morton, A. and E. Stephan, "Spatial Composition of [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of
Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011,
<http://www.rfc-editor.org/info/rfc6049>. <http://www.rfc-editor.org/info/rfc6049>.
[X3.102] ANSI X3.102, , "ANSI Standard on Data Communications,
User-Oriented Data Communications Framework", 1983.
Author's Address Author's Address
Al Morton Al Morton
AT&T Labs AT&T Labs
200 Laurel Avenue South 200 Laurel Avenue South
Middletown,, NJ 07748 Middletown,, NJ 07748
USA USA
Phone: +1 732 420 1571 Phone: +1 732 420 1571
Fax: +1 732 368 1192 Fax: +1 732 368 1192
 End of changes. 20 change blocks. 
34 lines changed or deleted 45 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/