[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00

CORE WG                                                         F. Zheng
INTERNET-DRAFT                                                     B. Fu
Intended Status: Informational                              Z. Cao (Ed.)
Expires: Feb 3 , 2016                                        Huawei Tech

                                                            July 4, 2016


                        CoAP Latency Evaluation
              draft-zheng-core-coap-lantency-evaluation-00


Abstract


   This document presents the evaluation results of CoAP in terms of
   various latency metrics over UDP/TCP under different network
   environments.  We conduct experiments via both GPRS and WiFi.  We
   also evaluate how the latency metrics are impacted by the background
   traffic.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html


Copyright and License Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.




                         Expires <Expiry Date>                  [Page 1]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.



Table of Contents

   1  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . .  3
     2.1 General topology . . . . . . . . . . . . . . . . . . . . . .  3
     2.2 CoAP Client  . . . . . . . . . . . . . . . . . . . . . . . .  3
     2.3 CoAP Server  . . . . . . . . . . . . . . . . . . . . . . . .  3
     2.4 Bare Latency . . . . . . . . . . . . . . . . . . . . . . . .  4
     2.5 Packet loss rate . . . . . . . . . . . . . . . . . . . . . .  4
     2.6 Tested flows . . . . . . . . . . . . . . . . . . . . . . . .  4
   3 Metrics  . . . . . . . . . . . . . . . . . . . . . . . . . . . .  4
   4 Results  . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5
     4.1 How much CoCoA helps?  . . . . . . . . . . . . . . . . . . .  5
     4.2 How much impact of the competing flows under GPRS and WiFi .  6
     4.4 How much difference of CoAP-TCP and CoAP-over-UDP  . . . . .  7
   5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . .  7
   6  Security Considerations . . . . . . . . . . . . . . . . . . . .  8
   7  IANA Considerations . . . . . . . . . . . . . . . . . . . . . .  8
   8 Acknowledgement  . . . . . . . . . . . . . . . . . . . . . . . .  8
   9  References  . . . . . . . . . . . . . . . . . . . . . . . . . .  8
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . .  9


















                         Expires <Expiry Date>                  [Page 2]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


1  Introduction

   CoAP [RFC7252] is a light-weight application protocol for constrained
   networks. CoAP has been widely used in many Internet of Things
   scenarios and deployed by many products as well.

   In order to know how CoAP performs under various network environment,
   we conduct the experimental study and present the results in this
   document.  This work is a complementary work of the previous
   evaluation work presented in [COAPCCCOMP] and [COAPCC].  In
   particular, (a) we intentionally measure the latency related metrics;
   (b) we introduce competing flows while testing, which reveals the
   difference of Bufferbloat impact under GPRS and WiFi; (c) we conduct
   study in both cellular networks (GPRS) and WiFi network.

   This document presents the evaluation setup and results, and intents
   to provide insights for further protocol design and implementation.

2 Experiment Setup

   This section describes the evaluation setup.

2.1 General topology

   The topology is depicted in Fig.1


              ---GPRS--+     ________
             |         |    (        )
CoAP Client-{          +-  + Internet +----+CoAP Server
             |         |    (________)
              - Wi-Fi--+

                           Figure 1: Topology

2.2 CoAP Client

We use an Android phone as the CoAP client, which has both cellular and
Wi-Fi interfaces. This device is physically located at our lab.

libcoap is ported to this device with the extension defined in CoCoA
algorithm [CoCoA] and CoAP over TCP [COT].

2.3 CoAP Server

The CoAP server is a rented Linux server from some public cloud company
in China.   We also use libcoap and our extensions on the server side.




                         Expires <Expiry Date>                  [Page 3]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


2.4 Bare Latency

The ICMP RTT between the client and server is as below.


    |      | GPRS      |   Wi-Fi   |
    +------+-----------+-----------+
    | RTT  | ~572ms    |   ~19ms   |
    +------+-----------+-----------+


2.5 Packet loss rate

We conduct these tests with the native packet loss rate, as well as
increased packet loss rate by using Linux TC command at the server (+2%
and +5% separately). We use 0% to denote the native packet loss rate.

2.6 Tested flows

The client sends 100 CoAP requests to the server, with interval of 100
ms. The size of the CoAP request is 9 Bytes, and the response payload
size is 23 Bytes.  We believe this reflects a certain IoT scenario,
e.g., the reporting of data or status from device to server. For TCP
flows, long-lived connection has been established before the test, so
that TCP three-way handshake overhead has not been taken into account.

We also introduce background traffic as the competing flow while
measuring the latency. The background traffic is a FTP downloading
flow.

3 Metrics

We intend to evaluate the latency of the CoAP flow with the following
metrics:



     1. Flow completing time (FCT): defined as the time elapsed from the
     sending of first request until the receiving of the last response.

     2. CoAP RTT (C-RTT): defined as the average elapsed time between
     the sending of the original CoAP request and receiving of the CoAP
     response.  Note: if the original request has been lost and then
     retransmitted when RTO is triggered, it will certainly increase the
     CoAP-RTT.

     3. Retransmission ratio (RTR): defined as the percentage of CoAP-
     layer retransmission.  The maximum number of retransmission is



                         Expires <Expiry Date>                  [Page 4]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


     defined as FOUR in [RFC7252] for UDP.


4 Results

4.1 How much CoCoA helps?

     Table.1 summarizes the CoAP latency performance with and without
     CoCoA.  The results clearly indicate that CoCoA effectively reduces
     the latency, especially when the packet loss rate is high, due to
     its dynamic calculation and update of RTO.  The trade-off of CoCoA
     is its payment of more retransmission, alike other congestion
     control algorithm.


 GPRS  | CoAP-RAW      |   CoAP-CoCoA   |
+------+---------------+----------------+
| FCT  |+0%* | 10173   |+0%  |   10175  |
|(ms)  +---------------+----------------+
|      |+2%  | 10868   |+2%  |   10170  |
|      +---------------+----------------+
|      |+5%  | 10697   |+5%  |   10176  |
+------+---------------+----------------+
|C-RTT |+0%  |  178    |+0%  |   172    |
|(ms)  +---------------+----------------+
|      |+2%  |  243    |+2%  |   182    |
|      +---------------+----------------+
|      |+5%  |  332    |+5%  |   203    |
+------+---------------+----------------+
| RTR  |+0%  |    0    |+0%  |     7    |
| (% ) +---------------+----------------+
|      |+2%  |   2.7   |+2%  |    18    |
|      +---------------+----------------+
|      |+5%  |   5.3   |+5%  |     9    |
+++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++
 WiFi  | CoAP-RAW      |   CoAP-CoCoA   |
+------+---------------+----------------+
| FCT  |+0%  |  9988   |+0%  |   10175  |
|(ms)  +---------------+----------------+
|      |+2%  | 10818   |+2%  |   10170  |
|      +---------------+----------------+
|      |+5%  | 10889   |+5%  |   10176  |
+------+---------------+----------------+
|C-RTT |+0%  |   18    |+0%  |    18    |
|(ms)  +---------------+----------------+
|      |+2%  |  105    |+2%  |    18    |
|      +---------------+----------------+



                         Expires <Expiry Date>                  [Page 5]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


|      |+5%  |  180    |+5%  |    18    |
+------+---------------+----------------+
| RTR  |+0%  |    0    |+0%  |   3.5    |
| (% ) +---------------+----------------+
|      |+2%  |   3.3   |+2%  |   5.3    |
|      +---------------+----------------+
|      |+5%  |   6.3   |+5%  |   6.8    |
+------+---------------+----------------+
* 0% indicates the native packet loss rate, without
any manually introduced packet loss rate.
        Table.1 CoAP latency performance with and without CoCoA

4.2 How much impact of the competing flows under GPRS and WiFi

Table.2 summarizes the results.  With competing flows under GRPS, both
FCT and C-RTT deteriorate heavily, and especially C-RTT has been
increased by around seven times.  Comparatively, in Wi-Fi case,
existence of the competing flows does not deteriorates the two latency
factors much.

The latency bloat given the existence of competing flows most likely
stems from the famous buffer-bloat problem. The GPRS and WiFi differ
from each other because of their queue management deployed in the
middle.  This result we present here about GPRS is consistent with what
have been reported in [BufbloatLTE].  In addition, we find the WiFi
suffers less from Bufferbloat than GPRS.


 GPRS  | CoAP-RAW      |   CoAP-CoCoA   |
+------+---------------+----------------+
| FCT  |wo   | 10173   |wo   |   10175  |
|(ms)  +---------------+----------------+
|      |w    | 11270   |w    |   11288  |
+------+---------------+----------------+
|C-RTT |wo   |  178    |wo   |   172    |
|(ms)  +---------------+----------------+
|      |w    |  1344   |w    |   1315   |
+------+---------------+----------------+

 WiFi  | CoAP-RAW      |   CoAP-CoCoA   |
+------+---------------+----------------+
| FCT  |wo   |  9988   |wo   |    9985  |
|(ms)  +---------------+----------------+
|      |w    |  9969   |w    |    9969  |
+------+---------------+----------------+
|C-RTT |wo   |   18    |wo   |    18.3  |
|(ms)  +---------------+----------------+
|      |w    |  30.9   |w    |    25.4  |



                         Expires <Expiry Date>                  [Page 6]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


+------+---------------+----------------+

        Table.2 CoAP latency performance under competing flows
 (wo=without competing flow, w=with competing flow)

4.4 How much difference of CoAP-TCP and CoAP-over-UDP

 We implement the CoAP-TCP per [COAP-T], which is still working-in-
 progress. The C-RTT comparison of the two algorithm is shown in
 Table.3.  In the CoAP-TCP implementation, we used the default Linux
 congestion control algorithm -Cubic.  In the test, we first establish
 the TCP connection and then run the CoAP flows, which means that the
 handshake overhead has not been included into the results.  CoAP-TCP
 bears a longer latency possibly because of (a) TCP header overhead (20
 bytes as composed to 8 bytes, and TCP/UDP header constitutes a larger
 portion of the packet due to the small CoAP payload; (b) GPRS low data
 rate, which makes this effect more obvious in GPRS than in WiFi. (d)
 under WiFi, retransmission contributes more to latency than packet
 length.


 GPRS  | CoAP-CoCoA    |   CoAP-TCP     |
+------+---------------+----------------+
|C-RTT |+0%  |  172    |+0%  |   450    |
|(ms)  +---------------+----------------+
|      |+2%  |  182    |+2%  |   421    |
|      +---------------+----------------+
|      |+5%  |  203    |+5%  |   454    |
+------+---------------+----------------+
 WiFi  | CoAP-CoCoA    |   CoAP-TCP     |
+------+---------------+----------------+
|C-RTT |+0%  |  18     |+0%  |     18   |
|(ms)  +---------------+----------------+
|      |+2%  |  18     |+2%  |     31   |
|      +---------------+----------------+
|      |+5%  |  18     |+5%  |     40   |
+------+---------------+----------------+


      Table.3 CoAP latency performance of UDP-CoCoA and CoAP-TCP

5. Summary

 As a summary, we find:


     1. CoCoA algorithm helps shorten the CoAP message layer latency
     effectively for our tested cases.



                         Expires <Expiry Date>                  [Page 7]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


     2. With competing flows under GRPS, both FCT and C-RTT deteriorate
     heavily.  Comparatively, WiFi suffers less from competing flows
     than GPRS, because of deployment of active queue management to
     fight with Bufferbloat.

     3. COAP-TCP (with Cubic) in general bears a longer latency than
     that of CoCoA in UDP. Many factors contribute to this fact as
     analyzed in Section.4.4.


     This document intents to provide insights for further protocol
     design and implementation.  We will be glad to generate  more
     results if the IETF audience finds this document helpful.



6  Security Considerations

     We have not evaluate the CoAP with DTLS/TLS yet.  Security tunnels
     definitely add to the overall latency.




7  IANA Considerations

     Not applicable to this document.


8 Acknowledgement

     This document was inspired by related work presented in [COAPCC]
     [COAPCCCOMP].  We also appreciate all contributors to CoAP
     Congestion Control.  We also thanks Dr. Fanzhao Wang for discussion
     and reasoning about the results.

9  References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, DOI
              10.17487/RFC2119, March 1997.

   [RFC7252]  Shelby, Z., Hartke, K., and C. Bormann, "Constrained
              Application Protocol (CoAP)", RFC 7252, June 2014.

   [CoCoA] C. Bormann, A. Betzler, C. Gomez, I. Demirkol, "CoAP Simple
              Congestion Control/Advanced", draft-bormann-core-cocoa-03,
              Oct., 2015.



                         Expires <Expiry Date>                  [Page 8]


INTERNET DRAFT  draft-zheng-core-coap-latency-evaluation    <Issue Date>


   [COAP-T] C. Bormann, S. Lemay, and H. Tschofenig, "A TCP and TLS
              Transport for the Constrained Application Protocol
              (CoAP)", draft-ietf-core-coap-tcp-tls-03, May 2016.

   [COAPCCCOMP] Ilpo Jaervinen, Laila Daniel, Markku Kojo, "Experimental
              Evaluation of Alternative Congestion Control Algorithms
              for Constrained Application Protocol (CoAP)",  Internet of
              Things (WF-IoT), 2015.

   [COAPCC] August Betzler,  Carles Gomez,  Ilker Demirkol,  Josep
              Paradells, "CoAP Congestion Control for the Internet of
              Things", IEEE Networks Magazine, 2016.

   [BufbloatLTE] Stefan Alfredsson, Giacomo Del Giudice, Johan Garcia,
              Anna Brunstrom, Luca De Cicco, Saverio Mascolo, "Impact of
              TCP congestion control on bufferbloat in cellular
              networks", IEEE WOWMOM 2013.



Authors' Addresses



   Fei Zheng
   Huawei Tech,
   Shenzhen, China

   EMail: zhengfei10@huawei.com

   Baicheng Fu
   Huawei Tech,
   Shenzhen, China

   EMail: fubaicheng@huawei.com


   Zhen Cao (Ed.)
   Huawei Tech,
   Beijing, China

   EMail: zhencao.ietf@gmail.com









                         Expires <Expiry Date>                  [Page 9]


Html markup produced by rfcmarkup 1.129c, available from https://tools.ietf.org/tools/rfcmarkup/