draft-ietf-bmwg-virtual-net-01.txt   draft-ietf-bmwg-virtual-net-02.txt 
Network Working Group A. Morton Network Working Group A. Morton
Internet-Draft AT&T Labs Internet-Draft AT&T Labs
Intended status: Informational September 23, 2015 Intended status: Informational March 21, 2016
Expires: March 26, 2016 Expires: September 22, 2016
Considerations for Benchmarking Virtual Network Functions and Their Considerations for Benchmarking Virtual Network Functions and Their
Infrastructure Infrastructure
draft-ietf-bmwg-virtual-net-01 draft-ietf-bmwg-virtual-net-02
Abstract Abstract
Benchmarking Methodology Working Group has traditionally conducted Benchmarking Methodology Working Group has traditionally conducted
laboratory characterization of dedicated physical implementations of laboratory characterization of dedicated physical implementations of
internetworking functions. This memo investigates additional internetworking functions. This memo investigates additional
considerations when network functions are virtualized and performed considerations when network functions are virtualized and performed
in commodity off-the-shelf hardware. in general purpose hardware.
Version NOTES:
Addressed Ramki Krishnan's comments on section 4.5, power, see that
section (7/27 message to the list). Addressed Saurabh
Chattopadhyay's 7/24 comments on VNF resources and other resource
conditions and their effect on benchmarking, see section 3.4.
Addressed Marius Georgescu's 7/17 comments on the list (sections 4.3
and 4.4).
AND, comments from the extended discussion during IETF-93 BMWG
session:
Section 4.2: VNF footprint and auxilliary metrics (Maryam Tahhan), See the new version history section for updates.
Section 4.3: Verification affect metrics (Ramki Krishnan);
Section 4.4: Auxilliary metrics in the Matrix (Maryam Tahhan, Scott
Bradner, others)
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 2, line 12 skipping to change at page 1, line 43
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on March 26, 2016. This Internet-Draft will expire on September 22, 2016.
Copyright Notice Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Considerations for Hardware and Testing . . . . . . . . . . . 4 3. Considerations for Hardware and Testing . . . . . . . . . . . 4
3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 5 3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 4
3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5 3.2. Configuration Parameters . . . . . . . . . . . . . . . . 5
3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6 3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 6
3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7 3.4. Attention to Shared Resources . . . . . . . . . . . . . . 7
4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 7 4. Benchmarking Considerations . . . . . . . . . . . . . . . . . 7
4.1. Comparison with Physical Network Functions . . . . . . . 8 4.1. Comparison with Physical Network Functions . . . . . . . 7
4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8 4.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 8
4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 9 4.3. New Benchmarks and Related Metrics . . . . . . . . . . . 8
4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 9 4.4. Assessment of Benchmark Coverage . . . . . . . . . . . . 9
4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12 4.5. Power Consumption . . . . . . . . . . . . . . . . . . . . 12
5. Security Considerations . . . . . . . . . . . . . . . . . . . 12 5. Security Considerations . . . . . . . . . . . . . . . . . . . 12
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 8. Version history . . . . . . . . . . . . . . . . . . . . . . . 13
8.1. Normative References . . . . . . . . . . . . . . . . . . 13 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.2. Informative References . . . . . . . . . . . . . . . . . 14 9.1. Normative References . . . . . . . . . . . . . . . . . . 13
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 15 9.2. Informative References . . . . . . . . . . . . . . . . . 15
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 16
1. Introduction 1. Introduction
Benchmarking Methodology Working Group (BMWG) has traditionally Benchmarking Methodology Working Group (BMWG) has traditionally
conducted laboratory characterization of dedicated physical conducted laboratory characterization of dedicated physical
implementations of internetworking functions (or physical network implementations of internetworking functions (or physical network
functions, PNFs). The Black-box Benchmarks of Throughput, Latency, functions, PNFs). The Black-box Benchmarks of Throughput, Latency,
Forwarding Rates and others have served our industry for many years. Forwarding Rates and others have served our industry for many years.
[RFC1242] and [RFC2544] are the cornerstones of the work. [RFC1242] and [RFC2544] are the cornerstones of the work.
skipping to change at page 5, line 38 skipping to change at page 5, line 28
entire general-purpose platform to ensure repeatability and foster entire general-purpose platform to ensure repeatability and foster
future comparisons, including but clearly not limited-to the future comparisons, including but clearly not limited-to the
following: following:
o number of server blades (shelf occupation) o number of server blades (shelf occupation)
o CPUs o CPUs
o caches o caches
o memory
o storage system o storage system
o I/O o I/O
as well as configurations that support the devices which host the VNF as well as configurations that support the devices which host the VNF
itself: itself:
o Hypervisor (or other forms of virtual function hosting) o Hypervisor (or other forms of virtual function hosting)
o Virtual Machine (VM) o Virtual Machine (VM)
skipping to change at page 10, line 37 skipping to change at page 10, line 35
it clear that there are other potential performance criteria to it clear that there are other potential performance criteria to
benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups" benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups"
in a set of 100 attempts. This example emphasizes that the in a set of 100 attempts. This example emphasizes that the
Activation and De-activation life cycle stages are key areas for NFV Activation and De-activation life cycle stages are key areas for NFV
and related infrastructure, and encourage expansion beyond and related infrastructure, and encourage expansion beyond
traditional benchmarks for normal operation. Thus, reviewing the traditional benchmarks for normal operation. Thus, reviewing the
benchmark coverage using this table (sometimes called the 3x3 matrix) benchmark coverage using this table (sometimes called the 3x3 matrix)
can be a worthwhile exercise in BMWG. can be a worthwhile exercise in BMWG.
In one of the first applications of the 3x3 matrix in BMWG In one of the first applications of the 3x3 matrix in BMWG
[I-D.bhuvan-bmwg-sdn-controller-benchmark-meth], we discovered that [I-D.ietf-bmwg-sdn-controller-benchmark-meth], we discovered that
metrics on measured size, capacity, or scale do not easily match one metrics on measured size, capacity, or scale do not easily match one
of the three columns above. Following discussion, this was resolved of the three columns above. Following discussion, this was resolved
in two ways: in two ways:
o Add a column, Scale, for use when categorizing and assessing the o Add a column, Scale, for use when categorizing and assessing the
coverage of benchmarks (without measured results). Examples of coverage of benchmarks (without measured results). Examples of
this use are found in this use are found in
[I-D.bhuvan-bmwg-sdn-controller-benchmark-meth] and [I-D.ietf-bmwg-sdn-controller-benchmark-meth] and
[I-D.vsperf-bmwg-vswitch-opnfv]. This is the 3x4 Matrix. [I-D.vsperf-bmwg-vswitch-opnfv]. This is the 3x4 Matrix.
o If using the matrix to report results in an organized way, keep o If using the matrix to report results in an organized way, keep
size, capacity, and scale metrics separate from the 3x3 matrix and size, capacity, and scale metrics separate from the 3x3 matrix and
incorporate them in the report with other qualifications of the incorporate them in the report with other qualifications of the
results. results.
Note: The resource utilization (e.g., CPU) metrics do not fit in the Note: The resource utilization (e.g., CPU) metrics do not fit in the
Matrix. They are not benchmarks, and omitting them confirms their Matrix. They are not benchmarks, and omitting them confirms their
status as auxilliary metrics. Resource assignments are configuration status as auxilliary metrics. Resource assignments are configuration
skipping to change at page 13, line 16 skipping to change at page 13, line 10
The author acknowledges an encouraging conversation on this topic The author acknowledges an encouraging conversation on this topic
with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhavani with Mukhtiar Shaikh and Ramki Krishnan in November 2013. Bhavani
Parise and Ilya Varlashkin have provided useful suggestions to expand Parise and Ilya Varlashkin have provided useful suggestions to expand
these considerations. Bhuvaneswaran Vengainathan has already tried these considerations. Bhuvaneswaran Vengainathan has already tried
the 3x3 matrix with SDN controller draft, and contributed to many the 3x3 matrix with SDN controller draft, and contributed to many
discussions. Scott Bradner quickly pointed out shared resource discussions. Scott Bradner quickly pointed out shared resource
dependencies in an early vSwitch measurement proposal, and the topic dependencies in an early vSwitch measurement proposal, and the topic
was included here as a key consideration. Further development was was included here as a key consideration. Further development was
encouraged by Barry Constantine's comments following the IETF-92 BMWG encouraged by Barry Constantine's comments following the IETF-92 BMWG
session: the session itself was an affirmation for this memo with session: the session itself was an affirmation for this memo. There
many interested inputs from Scott, Ramki, Barry, Bhuvan, Jacob Rapp, have been many interesting contributions from Maryam Tahhan, Marius
and others. Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others.
8. References 8. Version history
8.1. Normative References (This section should be removed by the RFC Editor.)
Version 02:
New version history section.
Added Memory in section 3.2, configuration.
Updated ACKs and References.
Version 01:
Addressed Ramki Krishnan's comments on section 4.5, power, see that
section (7/27 message to the list). Addressed Saurabh
Chattopadhyay's 7/24 comments on VNF resources and other resource
conditions and their effect on benchmarking, see section 3.4.
Addressed Marius Georgescu's 7/17 comments on the list (sections 4.3
and 4.4).
AND, comments from the extended discussion during IETF-93 BMWG
session:
Section 4.2: VNF footprint and auxilliary metrics (Maryam Tahhan),
Section 4.3: Verification affect metrics (Ramki Krishnan);
Section 4.4: Auxilliary metrics in the Matrix (Maryam Tahhan, Scott
Bradner, others)
9. References
9.1. Normative References
[NFV.PER001] [NFV.PER001]
"Network Function Virtualization: Performance and "Network Function Virtualization: Performance and
Portability Best Practices", Group Specification ETSI GS Portability Best Practices", Group Specification ETSI GS
NFV-PER 001 V1.1.1 (2014-06), June 2014. NFV-PER 001 V1.1.1 (2014-06), June 2014.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. <http://www.rfc-editor.org/info/rfc2119>.
skipping to change at page 14, line 44 skipping to change at page 15, line 20
[RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
"Network Time Protocol Version 4: Protocol and Algorithms "Network Time Protocol Version 4: Protocol and Algorithms
Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
<http://www.rfc-editor.org/info/rfc5905>. <http://www.rfc-editor.org/info/rfc5905>.
[RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for
Service Function Chaining", RFC 7498, Service Function Chaining", RFC 7498,
DOI 10.17487/RFC7498, April 2015, DOI 10.17487/RFC7498, April 2015,
<http://www.rfc-editor.org/info/rfc7498>. <http://www.rfc-editor.org/info/rfc7498>.
8.2. Informative References 9.2. Informative References
[I-D.bhuvan-bmwg-sdn-controller-benchmark-meth] [I-D.ietf-bmwg-sdn-controller-benchmark-meth]
Vengainathan, B., Basil, A., Tassinari, M., Manral, V., Vengainathan, B., Basil, A., Tassinari, M., Manral, V.,
and S. Banks, "Benchmarking Methodology for SDN Controller and S. Banks, "Benchmarking Methodology for SDN Controller
Performance", draft-bhuvan-bmwg-sdn-controller-benchmark- Performance", draft-ietf-bmwg-sdn-controller-benchmark-
meth-01 (work in progress), July 2015. meth-01 (work in progress), March 2016.
[I-D.krishnan-nfvrg-policy-based-rm-nfviaas] [I-D.krishnan-nfvrg-policy-based-rm-nfviaas]
Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D.,
Wright, S., Hinrichs, T., and R. Krishnaswamy, "NFVIaaS Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra,
Architectural Framework for Policy Based Resource "NFVIaaS Architectural Framework for Policy Based Resource
Placement and Scheduling", draft-krishnan-nfvrg-policy- Placement and Scheduling", draft-krishnan-nfvrg-policy-
based-rm-nfviaas-05 (work in progress), September 2015. based-rm-nfviaas-06 (work in progress), March 2016.
[I-D.vsperf-bmwg-vswitch-opnfv] [I-D.vsperf-bmwg-vswitch-opnfv]
Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking
Virtual Switches in OPNFV", draft-vsperf-bmwg-vswitch- Virtual Switches in OPNFV", draft-vsperf-bmwg-vswitch-
opnfv-00 (work in progress), July 2015. opnfv-01 (work in progress), October 2015.
[IPMI2.0] "Intelligent Platform Management Interface, v2.0 with [IPMI2.0] "Intelligent Platform Management Interface, v2.0 with
latest Errata", latest Errata",
http://www.intel.com/content/www/us/en/servers/ipmi/ipmi- http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-
intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec- intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec-
update.html, April 2015. update.html, April 2015.
[RFC1242] Bradner, S., "Benchmarking Terminology for Network [RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
July 1991, <http://www.rfc-editor.org/info/rfc1242>. July 1991, <http://www.rfc-editor.org/info/rfc1242>.
 End of changes. 23 change blocks. 
46 lines changed or deleted 63 lines changed or added

This html diff was produced by rfcdiff 1.44. The latest version is available from http://tools.ietf.org/tools/rfcdiff/