< draft-dcn-bmwg-containerized-infra-00.txt   draft-dcn-bmwg-containerized-infra-01.txt >
Benchmarking Methodology Working Group K. Sun Benchmarking Methodology Working Group K. Sun
Internet-Draft H. Yang Internet-Draft H. Yang
Intended status: Informational Y. Park Intended status: Informational Y. Park
Expires: September 8, 2019 Y. Kim Expires: January 9, 2020 Y. Kim
Soongsil University Soongsil University
W. Lee W. Lee
ETRI ETRI
March 7, 2019 July 8, 2019
Considerations for Benchmarking Network Performance in Containerized Considerations for Benchmarking Network Performance in Containerized
Infrastructures Infrastructures
draft-dcn-bmwg-containerized-infra-00 draft-dcn-bmwg-containerized-infra-01
Abstract Abstract
This draft describes benchmarking considerations for a containerized This draft describes benchmarking considerations for the
infrastructure. In a containerized infrastructure, Virtualized containerized infrastructure. In the containerized infrastructure,
Network Functions(VNFs) are deployed on operating-system-level Virtualized Network Functions(VNFs) are deployed on operating-system-
virtualization platform by abstracting the user namespace as opposed level virtualization platform by abstracting the user namespace as
to virtualization using a hypervisor. Leveraging this, the system opposed to virtualization using a hypervisor. Leveraging this, the
configurations and networking scenarios for VNF benchmarking will be system configurations and networking scenarios for benchmarking will
partially changed by way of resource allocation and network port be partially changed by the way in which the resource allocation and
binding between a physical host and VNFs. In this draft we compare network technologies speicifed for containerized VNFs. In this draft
the state of the art in container networking architecture with we compare the state of the art in a container networking
networking on VM-based virtualized systems, and provide several test architecture with networking on VM-based virtualized systems, and
scenarios for network performance in containerized infrastructure. provide several test scenarios in the containerized infrastructure.
Status of This Memo Status of This Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 8, 2019. This Internet-Draft will expire on January 9, 2020.
Copyright Notice Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the Copyright (c) 2019 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of (https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Benchmarking Consideration . . . . . . . . . . . . . . . . . 3 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 3
3.1. Comparison with VM based Infrastructure . . . . . . . . . 3 3.1. Comparison with the VM-based Infrastructure . . . . . . . 3
3.2. Additional Considerations for Container Networking . . . 5 3.2. Container Networking Classification . . . . . . . . . . . 5
4. Test Scenarios . . . . . . . . . . . . . . . . . . . . . . . 7 3.3. Resource Considerations . . . . . . . . . . . . . . . . . 8
5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 4. Benchmarking Scenarios for the Containerized Infrastructure . 9
6. Informative References . . . . . . . . . . . . . . . . . . . 7 5. Additional Considerations . . . . . . . . . . . . . . . . . . 12
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 6. Security Considerations . . . . . . . . . . . . . . . . . . . 13
7. Acknkowledgement . . . . . . . . . . . . . . . . . . . . . . 13
8. Informative References . . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14
1. Introduction 1. Introduction
The Benchmarking Methodology Working Group(BMWG) has recently The Benchmarking Methodology Working Group(BMWG) has recently
expanded its benchmarking scope from Physical Network Function(PNF) expanded its benchmarking scope from Physical Network Function(PNF)
running on dedicated hardware system to Network Function running on dedicated hardware system to Network Function
Virtualization(NFV) infrastructure and Virtualized Network Virtualization(NFV) infrastructure and Virtualized Network
Function(VNF). [RFC8172] described considerations for configuring Function(VNF). [RFC8172] described considerations for configuring
NFV infrastructure and benchmarking metrics, and [RFC8204] gives NFV infrastructure and benchmarking metrics, and [RFC8204] gives
guidelines for benchmarking virtual switch which connects VNFs in guidelines for benchmarking virtual switch which connects VNFs in
Open Platform for NFV(OPNFV). Open Platform for NFV(OPNFV).
Recently NFV infrastructure has evolved to include a lightweight Recently NFV infrastructure has evolved to include a lightweight
virtualized platform called the containerized infrastructure, where virtualized platform called the containerized infrastructure, where
VNFs share the same host Operating System(OS) and they are logically VNFs share the same host Operating System(OS) and they are logically
isolated by using a different namespace. While previous NFV isolated by using a different namespace. While previous NFV
infrastructure uses a hypervisor to allocate resources for Virtual infrastructure uses a hypervisor to allocate resources for Virtual
Machine(VMs) and instantiate VNFs on it, the containerized Machine(VMs) and instantiate VNFs on it, the containerized
infrastructure virtualizes resources without a hypervisor, therefore infrastructure virtualizes resources without a hypervisor, therefore
making containers very lightweight and more efficient in making containers very lightweight and more efficient in
infrastructure resource utilization compared to a VM based NFV infrastructure resource utilization compared to the VM-based NFV
infrastructure. When we consider benchmarking for VNFs in the infrastructure. When we consider benchmarking for VNFs in the
containerized infrastructure, it may have a different Device Under containerized infrastructure, it may have a different System Under
Test(DUT) configuration compared with both black-box benchmarking and Test(SUT) and Device Under Test(DUT) configuration compared with both
VM-based NFV infrastructure as described in [RFC8172]. Accordingly, black-box benchmarking and VM-based NFV infrastructure as described
additional configuration parameters and testing strategies may be in [RFC8172]. Accordingly, additional configuration parameters and
required. testing strategies may be required.
In the containerized infrastructure, a VNF network is implemented by In the containerized infrastructure, a VNF network is implemented by
running both switch and router functions in the host system. For running both switch and router functions in the host system. For
example, the internal communication between VNFs in the same host example, the internal communication between VNFs in the same host
uses the L2 bridge function, while communication with external uses the L2 bridge function, while communication with external
node(s) uses the L3 router function. For container networking, the node(s) uses the L3 router function. For container networking, the
host system may use a virtual switch(vSwitch), but other options host system may use a virtual switch(vSwitch), but other options
exist. In the [ETSI-TST-009], they describe differences in exist. In the [ETSI-TST-009], they describe differences in
networking structure between VM-based and container-based networking structure between the VM-based and the containerized
infrastructure. Occasioned by these differences, deployment infrastructure. Occasioned by these differences, deployment
scenarios for testing network performance described in [RFC8204] may scenarios for testing network performance described in [RFC8204] may
be partially applied to the containerized infrastructure, but other be partially applied to the containerized infrastructure, but other
scenarios may be required. scenarios may be required.
In this draft, we describe differences and additional considerations In this draft, we describe differences and additional considerations
for benchmarking containerized infrastructure based on [RFC8172] and for benchmarking containerized infrastructure based on [RFC8172] and
[RFC8204]. In particular, we focus on differences in system [RFC8204]. In particular, we focus on differences in system
configuration parameters and networking configurations of the configuration parameters and networking configurations of the
containerized infrastructure compared with VM-based NFV containerized infrastructure compared with VM-based NFV
skipping to change at page 3, line 42 skipping to change at page 3, line 45
metrics or methodologies is out of scope. metrics or methodologies is out of scope.
2. Terminology 2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document is to be interpreted as described in [RFC2119]. This document is to be interpreted as described in [RFC2119]. This
document uses the terminology described in [RFC8172], [RFC8204], document uses the terminology described in [RFC8172], [RFC8204],
[ETSI-TST-009]. [ETSI-TST-009].
3. Benchmarking Consideration 3. Benchmarking Considerations
3.1. Comparison with VM based Infrastructure 3.1. Comparison with the VM-based Infrastructure
For benchmarking of containerized infrastructure, as mentioned in For the benchmarking of the containerized infrastructure, as
[RFC8172], the basic approach is to reuse existing benchmarks mentioned in [RFC8172], the basic approach is to reuse existing
developed within the BMWG. Various network function specifications benchmarking methods developed within the BMWG. Various network
already defined in BMWG should still be applied to containerized VNFs function specifications defined in BMWG should still be applied to
for performance comparison with physical network functions and VM containerized VNF(C-VNF)s for the performance comparison with
based VNFs. physical network functions and VM-based VNFs.
+---------------------------------+ +--------------------------------+ +---------------------------------+ +--------------------------------+
|+--------------+ +--------------+| |+------------+ +------------+| |+--------------+ +--------------+| |+------------+ +------------+|
|| Guest VM | | Guest VM || || Container | | Container || || Guest VM | | Guest VM || || Container | | Container ||
||+------------+| |+------------+|| ||+----------+| |+----------+|| ||+------------+| |+------------+|| ||+----------+| |+----------+||
||| APP || || APP ||| ||| APP || || APP ||| ||| APP || || APP ||| ||| APP || || APP |||
||+------------+| |+------------+|| ||+----------+| |+----------+|| ||+------------+| |+------------+|| ||+----------+| |+----------+||
||+------------+| |+------------+|| ||+----------+| |+----------+|| ||+------------+| |+------------+|| ||+----------+| |+----------+||
|||Guest Kernel|| ||Guest Kernel||| ||| Bin/Libs || || Bin/Libs ||| |||Guest Kernel|| ||Guest Kernel||| ||| Bin/Libs || || Bin/Libs |||
||+------------+| |+------------+|| ||+----------+| |+----------+|| ||+------------+| |+------------+|| ||+----------+| |+----------+||
skipping to change at page 4, line 29 skipping to change at page 4, line 32
|| | Host OS Kernel | || || | Host OS Kernel | || || | Host OS Kernel | || || | Host OS Kernel | ||
|+------|-----------------|-----+|| |+-----|------------------|-----+| |+------|-----------------|-----+|| |+-----|------------------|-----+|
| +--v-----------------v--+ | | +---v------------------v---+ | | +--v-----------------v--+ | | +---v------------------v---+ |
+----| physical network |----+ +--| physical network |--+ +----| physical network |----+ +--| physical network |--+
+-----------------------+ +--------------------------+ +-----------------------+ +--------------------------+
(a) VM-Based Infrastructure (b) Containerized Infrastructure (a) VM-Based Infrastructure (b) Containerized Infrastructure
Figure 1: Comparison of NFV Infrastructures Figure 1: Comparison of NFV Infrastructures
In Figure 1, we describe two different NFV architectures: VM-based In Figure 1, we describe two different NFV architectures: VM-based
and Containerized. A major distinction between containerized and Containerized. A major distinction between the containerized and
infrastructure and VM based infrastructure is that with the former, the VM-based infrastructure is that with the former, all VNFs share
all VNFs share the same host resources including but not limited to same host resources including but not limited to computing, storage
computing, storage and networking resources, as well as the host and networking resources, as well as the host Operating System(OS),
Operating System(OS), kernel and libraries. The absence of the guest kernel and libraries. The absence of the guest OS and the
OS and the hypervisor, necessitates the following considerations that hypervisor, necessitates the following considerations that occur in
occur in the test environment: the test environment:
o Concerning hardware for containerized infrastructure, all o When we consider hardware configurations for the containerized
components described in [RFC8172] can be part of the test setup. infrastructure, all components described in [RFC8172] can be part of
While the capabilities of servers and storage should meet the minimum the test setup. While capabilities of servers and storages should
requirements for testing, it is possible to deploy a test environment meet the minimum requirements for testing, it is possible to deploy a
with less capabilities than in a VM based infrastructure. test environment with less capabilities than in the VM-based
infrastructure.
o About configuration parameters, containerized infrastructure needs o About configuration parameters, the containerized infrastructure
specified management system instead of hypervisor(e.g. Linux needs specified management system instead of a hypervisor(e.g. Linux
Container, Docker Engine). Container, Docker Engine).
o In the VM based infrastructure, each VM has packet processing in o In the VM-based infrastructure, each VM manipulates packets in the
the kernel of the guest OS through its own CPU threads, virtualized kernel of the guest OS through its own CPU threads, virtualized and
and assigned by hypervisor. On the other hand, containerized VNFs assigned by the hypervisor. On the other hand, C-VNFs use the host
use the host CPU without virtualization. Different CPU resource CPU without virtualization. Different CPU resource assignment
assignment methods may have different CPU utilization perspectives methods may have different CPU utilization perspectives for the
for VNF performance benchmarking. performance benchmarking.
o From a Memory Management Unit(MMU) point of view, there is a o From a Memory Management Unit(MMU) point of view, there are
difference in how the paging process is conducted between two differences in how the paging process is conducted between two
environments. The main difference lies in the isolated nature of the environments. The main difference lies in the isolated nature of the
OS for VM-based VNFs. In the containerized infrastructure, memory OS for VM-based VNFs. In the containerized infrastructure, memory
paging which processes conversion between physical address and paging which processes conversion between physical address and
virtual address is affected by the host resource directly. Thus, virtual address is affected by the host resource directly. Thus,
memory usage of each VNFs is more dependent on the host resource memory usage of each C-VNFs is more dependent on the host resource
capabilities than in VM-based VNFs. capabilities than in VM-based VNFs.
o Some network drivers may have varying dependencies for each 3.2. Container Networking Classification
environment. For example, a vhost-net driver used in a guest OS
cannot be used for a container; on the other hand, a veth driver can
be only applicable within a containerized infrastructure.
3.2. Additional Considerations for Container Networking Container networking services are provided as network plugins.
Basically, using them, network services are deployed by using
isolation environment from container runtime through the host
namespace, creating virtual interface, allocating interface and IP
address to C-VNF. Since the containerized infrastructure has
defferent network architecture depending on its using plugins, it is
necessary to specify the plugin used in the infrastructure. There
are two proposed models for configuring network interfaces for
containers as belows;
In the containerized infrastructure, there are various network o CNM(Container Networking Model) proposed by Docker, using
architectures depending on the deployment environment and models. libnetwork which provides an interface between the Docker daemon and
Since container networking typically involves using virtual switch network drivers.
functions, base network configuration parameters for container
networking benchmarks are mostly similar with VM based VNF networking
described in [RFC8204]. Additional considerations for container
networking are described as follows:
o Networking depends on deployment models: Containerized VNFs have o CNI(Conainer Network Interface) proposed by CoreOS, describing
several deployment models. Containerized VNFs can be deployed as a network configuration files in JSON format and plugins are
cluster called POD by Kubernetes, otherwise each VNF can be deployed instantiated as new namespaces. Kubernetes uses CNI for providing
separately using Docker. In former case, there is only one external network service.
network interface for a POD which contains more than one VNF. An
alternative deployment model considers a scenario in which Regardless of both CNM and CNI, container network model can be
containerized VNFs or PODs are running on VM-based infrastructure. classified into kernel space network model and user space network
Figure 2 shows briefly differences of network architectures based on model according to the location of network service creation. In case
deployment models. [ETSI-TST-009] describes in more detail the of kernel-based network model, network interfaces are created in
differences between them. Other deployment models are classified kernel space so that data packets should be processed in network
bases on whether containerized VNFs are deployed on baremetal or stack of host kernel before transferring packets to the C-VNF running
inside of the VM. in user space. On the other hand, using user-based network model,
data packets from physical network port are bypassed kernel
processing and delivered directly to user space. Specific
technologies for each network model and example of network
architecture are written as follows:
o Kernel space network model: Docker Network[Docker-network], Flannel
Network[Flannel], Calico[Calico], OVS(OpenvSwitch)[OVS], OVN(Open
Virtual Network)[OVN], eBPF[eBPF]
+------------------------------------------------------------------+
| User Space |
| +-----------+ +-----------+ |
| | Container | | Container | |
| | +-------+ | | +-------+ | |
| +-| eth |-+ +-| eth |-+ |
| +--^----+ +----^--+ |
| | +------------------------------------------+ | |
| | | vSwitch | | |
| | | +--------------------------------------+ | | |
| | | | +--v---v---v--+ | | | |
| | | |bridge | tag[n] | | | | |
| | | | +--^-------^--+ | | | |
| | | +--^-------------|-------|-----------^-+ | | |
| | | | +---+ +---+ | | | |
| | | | +------ v-----+ +-------v----+ | | | |
| | | | |tunnel bridge| | flat bridge | | | | |
| | | | +------^------+ +-------^-----+ | | | |
| | +--- |--------|----------------|-------|---+ | |
---------|------ |--------|----------------|-------|------|---------
| +----|-------|--------|----------------|-------|------|----+ |
| | +--v-------v--+ | | +--v------v--+ | |
| | | veth | | | | veth | | |
| | +---^---------+ | | +---^--------+ | |
| | Kernel Datapath | | | |
| +---------------------|----------------|-------------------+ |
| | | |
| Kernel Space +--v----------------v--+ |
+----------------------| NIC |--------------------+
+----------------------+
Figure 2: Examples of Kernel Space Network Model
o User space network model - Device pass-through model: SR-
IOV[SR-IOV]
+------------------------------------------------------------------+
| User Space |
| +-----------------+ +-----------------+ |
| | Container | | Container | |
| | +-------------+ | | +-------------+ | |
| +-| vf driver |-+ +-| vf driver |-+ |
| +-----^-------+ +------^------+ |
| | | |
-------------|---------------------------------------|--------------
| +---------+ +---------+ |
| +------|-------------------|------+ |
| | +----v-----+ +-----v----+ | |
| | | virtual | | virtual | | |
| | | function | | function | | |
| Kernel Space | +----^-----+ NIC +-----^----+ | |
+---------------| | | |----------------+
| +----v-------------------v----+ |
| | Classify and Queue | |
| +-----------------------------+ |
+---------------------------------+
Figure 3: Examples of User Space Netowrk Model - Device Pass-through
- vSwitch model: ovs-dpdk[ovs-dpdk], vpp[vpp], netmap[netmap]
+------------------------------------------------------------------+
| User Space |
| +-----------------+ +-----------------+ |
| | Container | | Container | |
| | +-------------+ | | +-------------+ | |
| +-| virtio-user |-+ +-| virtio-user |-+ |
| +-----^-------+ +-------^-----+ |
| | | |
| +---------+ +---------+ |
| +-----------------|--------------------|-----------------+ |
| | vSwitch | | | |
| | +-------v-----+ +-----v-------+ | |
| | | virtio-user | | virtio-user | | |
| | +-------^-----+ +-----^-------+ | |
| | +------------|--------------------|-------------+ | |
| | | +--v--------------------v---+ | | |
| | |bridge | tag[n] | | | |
| | | +------------^--------------+ | | |
| | +----------------------|------------------------+ | |
| | +-------v--------+ | |
| | | dpdk0 bridge | | |
| | +-------^--------+ | |
| +---------------------------|----------------------------+ |
| +-------v--------+ |
| | DPDK PMD | |
| +-------^--------+ |
---------------------------------|----------------------------------
| Kernel Space +-----v------+ |
+--------------------------| NIC |--------------------------+
+------------+
Figure 4: Examples of User Space Netowrk Model - vSwitch Model using
DPDK
3.3. Resource Considerations
In the containerized infrastructure, resource utilization and
isolation may have different charateristics compared with the VM-
based infrastructure. Some details are listed as follows:
o Hugepage: When using Cent OS or RedHat OS in the VM-based
infrastructure, Hugepage should be set to at least 1G byte. In the
containerized infrastructure, container is isolated in the
application level so that administrators can set Hugepage more
granular level(e.g 2M, 4M, ...). In addition, since the increase of
the Hugepage can affect the Translation Lookaside Buffer (TLB) miss,
the value of the Hugepage should be taken into account in the
performance measurement. Moreover, benchmarking results may vary
according to Hugepage set value of kernel space model and user space
model in the containerized infrastructure so that Hugepage values
should be considered when we configure test environment.
o NUMA: NUMA technology can be used both in the VM-based and
containerized infrastructure. However, the containerized
infrastructure provides more variable options than the VM-based
infrastructure such as kernel memory, user memory, and CPU setting.
Instantiation of C-VNFs is somewhat non-deterministic and apparently
NUMA-Node agnostic, which is one way of saying that performance will
likely vary whenever this instantiation is performed. So, when we
use NUMA in the containerized infratructure, repeated instantiation
and testing to quantify the performance variation is required.
o RX/TX Multiple-Queue: RX/TX Multiple-Queue technology[Multique],
which enables packet sending/receiving processing to scale with
number of available vcpus of guest VM, may be used to enhance network
performance in the VM-based infrastructure. However, RX/TX Multiple-
Queue technology is not supported in the containerized infrastructure
yet.
4. Benchmarking Scenarios for the Containerized Infrastructure
Figure 5 shows briefly differences of network architectures based on
deployment models. Basically, on baremetal, C-VNFs can be deployed
as a cluster called POD by Kubernetes, otherwise each C-VNF can be
deployed separately using Docker. In former case, there is only one
external network interface even a POD contains more than one C-VNF.
An additional deployment model considers a scenario in which C-VNFs
or PODs are running on VM. In our draft, we define new
terminologies; BMP which is Pod on baremetal and VMP which is Pod on
VM.
+---------------------------------------------------------------------+ +---------------------------------------------------------------------+
| Baremetal Node | | Baremetal Node |
| | | |
| +--------------+ +--------------+ +-------------- + +-------------+ | | +--------------+ +--------------+ +-------------- + +-------------+ |
| | | | POD | | VM | | VM | | | | | | POD | | VM | | VM | |
| | | |+------------+| |+-------------+| | +------+ | | | | | |+------------+| |+-------------+| | +-------+ | |
| | Container | || Container || ||Container VNF|| | | PODs | | | | | C-VNF(A) | || C-VNFs(B) || || C-VNFs(C) || | |PODs(D)| | |
| | VNF | || VNFs || |+-----^-------+| | +---^--+ | | | | | |+------------+| |+-----^-------+| | +---^---+ | |
| | | |+------------+| | | | | | | | | | | | | | | | | | | |
| | +------+ | | +------+ | | +--v---+ | | +---v--+ | | | | +------+ | | +------+ | | +--v---+ | | +---v--+ | |
| +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ | | +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ |
| +--^---+ +---^--+ +--^---+ +---^--+ | | +--^---+ +---^--+ +--^---+ +---^--+ |
| | | | | | | | | | | |
| | | +--v---+ +---v--+ | | | | +--v---+ +---v--+ |
| +------|-----------------|------------|vhost |---------|vhost |---+ | | +------|-----------------|------------|vhost |---------|vhost |---+ |
| | | | +--^---+ +---^--+ | | | | | | +--^---+ +---^--+ | |
| | | | | | | | | | | | | | | |
| | +--v---+ +---v--+ +--v---+ +---v--+ | | | | +--v---+ +---v--+ +--v---+ +---v--+ | |
| | +-| veth |---------| veth |---------| Tap |---------| Tap |-+ | | | | +-| veth |---------| veth |---------| Tap |---------| Tap |-+ | |
skipping to change at page 6, line 39 skipping to change at page 10, line 39
| | | +---------+ | +--|-----------------|---+ | | | | | +---------+ | +--|-----------------|---+ | |
| | | |Container| | | | Hypervisor | | | | | | | |Container| | | | Hypervisor | | | |
| | | | Engine | | | | | | | | | | | | Engine | | | | | | | |
| | | +---------+ | +--|-----------------|---+ | | | | | +---------+ | +--|-----------------|---+ | |
| | | | Host Kernel | | | | | | | | Host Kernel | | | |
| +------|-----------------|---------------|-----------------|------+ | | +------|-----------------|---------------|-----------------|------+ |
| +--v-----------------v---------------v-----------------v--+ | | +--v-----------------v---------------v-----------------v--+ |
+-----| physical network |-----+ +-----| physical network |-----+
+---------------------------------------------------------+ +---------------------------------------------------------+
Figure 2: Examples of Networking Architecture based on Deployment Figure 5: Examples of Networking Architecture based on Deployment
Models Models - (A)C-VNF on Baremetal (B)Pod on Baremetal(BMP) (C)C-VNF on
VM (D)Pod on VM(VMP)
o Network Plug-ins: In the containerized infrastructure, specific In [ETSI-TST-009], they described data plane test scenarios in a
networking functions can be supported by attaching various plug-ins. single host. In that document, there are two scenario for
Container Network Model(CNM) and Container Network Interface(CNI) are containerized infrastucture; Container2Container which is internal
currently the most popular network plug-ins. According each network communication between two containers in the same Pod, and Pod2Pod
plug-in, they have different runtime structure or accessibilities to model which is communication between two containers runnung in
namespace. Actual testing results may vary depending on plug-in different Pods. According to our new terminologies, we can call
types and its supporting drivers. Pod2Pod model as BMP2BMP scenario. When we consider container
running on VM as an additional deployment option, there can be more
single host test scenarios as follows;
o BMP2VMP scenario
o Network Types: To enhance forwarding capabilities, similar to the +---------------------------------------------------------------------+
VM based infrastructure, the containerized infrastructure can also | HOST +-----------------------------+ |
employ use of specific networking technologies such as SR-IOV. | |VM +-------------------+ | |
| | | C-VNF | | |
| +--------------------+ | | +--------------+ | | |
| | C-VNF | | | | Logical Port | | | |
| | +--------------+ | | +-+--^-------^---+--+ | |
| | | Logical Port | | | +----|-------|---+ | |
| +-+--^-------^---+---+ | | Logical Port | | |
| | | +---+----^-------^---+--------+ |
| | | | | |
| +----v-------|----------------------------|-------v-------------+ |
| | l----------------------------l | |
| | Data Plane Networking | |
| | (Kernel or User space) | |
| +----^--------------------------------------------^-------------+ |
| | | |
| +----v------+ +----v------+ |
| | Phy Port | | Phy Port | |
| +-----------+ +-----------+
+-------^--------------------------------------------^----------------+
| |
+-------v--------------------------------------------v----------------+
| |
| Traffic Generator |
| |
+---------------------------------------------------------------------+
4. Test Scenarios Figure 6: Single Host Test Scenario - BMP2VMP
TBD o VMP2VMP scenario
5. Security Considerations +---------------------------------------------------------------------+
| HOST |
| +-----------------------------+ +-----------------------------+ |
| |VM +-------------------+ | |VM +-------------------+ | |
| | | C-VNF | | | | C-VNF | | |
| | | +--------------+ | | | | +--------------+ | | |
| | | | Logical Port | | | | | | Logical Port | | | |
| | +-+--^-------^---+--+ | | +-+--^-------^---+--+ | |
| | +----|-------|---+ | | +----|-------|---+ | |
| | | Logical Port | | | | Logical Port | | |
| +---+----^-------^---+--------+ +---+----^-------^---+--------+ |
| | | | | |
| +--------v-------v------------------------|-------v-------------+ |
| | l------------------------l | |
| | Data Plane Networking | |
| | (Kernel or User space) | |
| +----^--------------------------------------------^-------------+ |
| | | |
| +----v------+ +----v------+ |
| | Phy Port | | Phy Port | |
| +-----------+ +-----------+ |
+-------^--------------------------------------------^----------------+
| |
+-------v--------------------------------------------v----------------+
| |
| Traffic Generator |
| |
+---------------------------------------------------------------------+
Figure 7: Single Host Test Scenario - VMP2VMP
5. Additional Considerations
When we consider benchmarking for not only containerized but also VM-
based infrastucture and network functions, benchmarking scenarios may
contain various operational use cases. Traditional black-box
benchmarking is focused to measure in-out performance of packet from
physical network ports, since hardware is tightly coupled with its
function and only single function is running on its dedicated
hardware. However, in the NFV environment, the physical network port
commonly will be connected to multiple VNFs(i.e. Multiple PVP test
setup architecture was described in [ETSI-TST-009]) rather than
dedicated to a single VNF. Therefore, benchmarking scenarios should
reflect operational considerations such as number of VNFs or network
services defined by a set of VNFs in a single host.
[service-density], which proposed a way for measuring performance of
multiple NFV service instances at a varied service density on a
single host, is one example of these operational benchmarking
aspects.
6. Security Considerations
TBD TBD
6. Informative References 7. Acknkowledgement
We would like to thanks people Al, Maciek and Luis who reviewed and
gave comments of previous draft.
8. Informative References
[Calico] "Project Calico", July 2019,
<https://docs.projectcalico.org/>.
[Docker-network]
"Docker, Libnetwork design", July 2019,
<https://github.com/docker/libnetwork/>.
[eBPF] "eBPF, extended Berkeley Packet Filter", July 2019,
<https://www.iovisor.org/technology/ebpf>.
[ETSI-TST-009] [ETSI-TST-009]
"Network Functions Virtualisation (NFV) Release 3; "Network Functions Virtualisation (NFV) Release 3;
Testing; Specification of Networking Benchmarks and Testing; Specification of Networking Benchmarks and
Measurement Methods for NFVI", October 2018. Measurement Methods for NFVI", October 2018.
[Flannel] "flannel 0.10.0 Documentation", July 2019,
<https://coreos.com/flannel/>.
[Multique]
"Multiqueue virtio-net", July 2019,
<https://www.linux-kvm.org/page/Multiqueue>.
[netmap] "Netmap: a framework for fast packet I/O", July 2019,
<https://github.com/luigirizzo/netmap>.
[OVN] "How to use Open Virtual Networking with Kubernetes", July
2019, <https://github.com/ovn-org/ovn-kubernetes>.
[OVS] "Open Virtual Switch", July 2019,
<https://www.openvswitch.org/>.
[ovs-dpdk]
"Open vSwitch with DPDK", July 2019,
<http://docs.openvswitch.org/en/latest/intro/install/
dpdk/>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997. Requirement Levels", RFC 2119, March 1997.
[RFC8172] Morton, A., "Considerations for Benchmarking Virtual [RFC8172] Morton, A., "Considerations for Benchmarking Virtual
Network Functions and Their Infrastructure", RFC 8172, Network Functions and Their Infrastructure", RFC 8172,
July 2017. July 2017.
[RFC8204] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking [RFC8204] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking
Virtual Switches in the Open Platform for NFV (OPNFV)", Virtual Switches in the Open Platform for NFV (OPNFV)",
RFC 8204, September 2017. RFC 8204, September 2017.
[service-density]
Konstantynowicz, M. and P. Mikus, "NFV Service Density
Benchmarking", March 2019, <https://tools.ietf.org/html/
draft-mkonstan-nf-service-density-00>.
[SR-IOV] "SRIOV for Container-networking", July 2019,
<https://github.com/intel/sriov-cni>.
[vpp] "VPP with Containers", July 2019, <https://fdio-
vpp.readthedocs.io/en/latest/usecases/containers.html>.
Authors' Addresses Authors' Addresses
Kyoungjae Sun Kyoungjae Sun
School of Electronic Engineering School of Electronic Engineering
Soongsil University Soongsil University
369, Sangdo-ro, Dongjak-gu 369, Sangdo-ro, Dongjak-gu
Seoul, Seoul 06978 Seoul, Seoul 06978
Republic of Korea Republic of Korea
Phone: +82 10 3643 5627 Phone: +82 10 3643 5627
 End of changes. 32 change blocks. 
104 lines changed or deleted 382 lines changed or added

This html diff was produced by rfcdiff 1.47. The latest version is available from http://tools.ietf.org/tools/rfcdiff/