--- 1/draft-ietf-mops-ar-use-case-03.txt 2022-03-06 11:13:15.861655667 -0800 +++ 2/draft-ietf-mops-ar-use-case-04.txt 2022-03-06 11:13:15.889656363 -0800 @@ -1,110 +1,135 @@ MOPS R. Krishna Internet-Draft InterDigital Europe Limited Intended status: Informational A. Rahman -Expires: April 28, 2022 InterDigital Communications, LLC - October 25, 2021 +Expires: 7 September 2022 InterDigital Communications, LLC + 6 March 2022 Media Operations Use Case for an Augmented Reality Application on Edge Computing Infrastructure - draft-ietf-mops-ar-use-case-03 + draft-ietf-mops-ar-use-case-04 Abstract - A use case describing transmission of an application on the Internet - that has several unique characteristics of Augmented Reality (AR) - applications is presented for the consideration of the Media - Operations (MOPS) Working Group. One key requirement identified is - that the Adaptive-Bit-Rate (ABR) algorithms' current usage of - policies based on heuristics and models is inadequate for AR - applications running on the Edge Computing infrastructure. + This document explores the issues involved in the use of Edge + Computing resources to operationalize media use cases that involve + Extended Reality (XR) applications. In particular, we discuss those + applications that run on devices having different form factors and + need Edge computing resources to mitigate the effect of problems such + as a need to support interactive communication requiring low latency, + limited battery power, and heat dissipation from those devices. The + intended audience for this document are network operators who are + interested in providing edge computing resources to operationalize + the requirements of such applications. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." - This Internet-Draft will expire on April 28, 2022. + This Internet-Draft will expire on 7 September 2022. Copyright Notice - Copyright (c) 2021 IETF Trust and the persons identified as the + Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal - Provisions Relating to IETF Documents - (https://trustee.ietf.org/license-info) in effect on the date of - publication of this document. Please review these documents - carefully, as they describe your rights and restrictions with respect - to this document. Code Components extracted from this document must - include Simplified BSD License text as described in Section 4.e of - the Trust Legal Provisions and are provided without warranty as - described in the Simplified BSD License. + Provisions Relating to IETF Documents (https://trustee.ietf.org/ + license-info) in effect on the date of publication of this document. + Please review these documents carefully, as they describe your rights + and restrictions with respect to this document. Code Components + extracted from this document must include Revised BSD License text as + described in Section 4.e of the Trust Legal Provisions and are + provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Conventions used in this document . . . . . . . . . . . . . . 3 3. Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . 3 - 3.1. Processing of Scenes . . . . . . . . . . . . . . . . . . 3 - 3.2. Generation of Images . . . . . . . . . . . . . . . . . . 4 - 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4 - 5. AR Network Traffic and Interaction with TCP . . . . . . . . . 6 - 6. Informative References . . . . . . . . . . . . . . . . . . . 7 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 + 3.1. Processing of Scenes . . . . . . . . . . . . . . . . . . 4 + 3.2. Generation of Images . . . . . . . . . . . . . . . . . . 5 + 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 5 + 5. AR Network Traffic and Interaction with TCP . . . . . . . . . 8 + 6. Informative References . . . . . . . . . . . . . . . . . . . 8 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction - The MOPS draft, [I-D.ietf-mops-streaming-opcons], provides an - overview of operational networking issues that pertain to Quality of - Experience (QoE) in delivery of video and other high-bitrate media - over the Internet. However, as it does not cover the increasingly - large number of applications with Augmented Reality (AR) - characteristics and their requirements on ABR algorithms, the - discussion in this draft compliments the overview presented in that - draft [I-D.ietf-mops-streaming-opcons]. + Extended Reality (XR) is a term that includes Augmented Realty (AR), + Virtual Reality (VR) and Mixed Realty (MR) [XR]. AR combines the + real and virtual, is interactive and is aligned to the physical world + of the user [AUGMENTED_2]. On the other hand, VR places the user + inside a virtual environment generated by a computer [AUGMENTED].MR + merges the real and virtual world along a continuum that connects + completely real environment at one end to a completely virtual + environment at the other end. In this continuum, all combinations of + the real and virtual are captured [AUGMENTED]. - Future AR applications will bring several requirements for the - Internet and the mobile devices running these applications. AR - applications require a real-time processing of video streams to + XR applications will bring several requirements for the network and + the mobile devices running these applications. Some XR applications + such as AR require a real-time processing of video streams to recognize specific objects. This is then used to overlay information - on the video being displayed to the user. In addition some AR - applications will also require generation of new video frames to be - played to the user. Both the real-time processing of video streams - and the generation of overlay information are computationally - intensive tasks that generate heat [DEV_HEAT_1], [DEV_HEAT_2] and - drain battery power [BATT_DRAIN] on the AR mobile device. - Consequently, in order to run future applications with AR - characteristics on mobile devices, computationally intensive tasks - need to be offloaded to resources provided by Edge Computing. + on the video being displayed to the user. In addition XR + applications such as AR and VR will also require generation of new + video frames to be played to the user. Both the real-time processing + of video streams and the generation of overlay information are + computationally intensive tasks that generate heat [DEV_HEAT_1], + [DEV_HEAT_2] and drain battery power [BATT_DRAIN] on the mobile + device running the XR application. Consequently, in order to run + applications with XR characteristics on mobile devices, + computationally intensive tasks need to be offloaded to resources + provided by Edge Computing. Edge Computing is an emerging paradigm where computing resources and storage are made available in close network proximity at the edge of - the Internet to mobile devices and sensors [EDGE_1], [EDGE_2]. + the Internet to mobile devices and sensors [EDGE_1], [EDGE_2]. These + edge computing devices use cloud technologies that enable them to + support offloaded XR applications. In particular, the edge devices + deploy cloud computing implementation techniques such as + disaggregation (breaking vertically integrated systems into + independent components with open interfaces using SDN), + virtualization (being able to run multiple independent copies of + those components such as SDN Controller apps, Virtual Network + Functions on a common hardware platform) and commoditization ( being + able to elastically scale those virtual components across commodity + hardware as the workload dictates) [EDGE_3]. Such techniques enable + XR applications requiring low-latency and high bandwidth to be + delivered by mini-clouds running on proximate edge devices - Adaptive-Bit-Rate (ABR) algorithms currently base their policy for - bit-rate selection on heuristics or models of the deployment - environment that do not account for the environment's dynamic nature - in use cases such as the one we present in this document. - Consequently, the ABR algorithms perform sub-optimally in such - deployments [ABR_1]. + In this document, we discuss the issues involved when edge computing + resources are offered by network operators to operationalize the + requirements of XR applications running on devices with various form + factors. Examples of such form factors include Head Mounted Displays + (HMD) such as Optical-see through HMDs and video-see-through HMDs and + Hand-held displays. Smart phones with video cameras and GPS are + another example of such devices. These devices have limited battery + capacity and dissipate heat when running. Besides as the user of + these devices moves around as they run the XR application, the + wireless latency and bandwidth available to the devices fluctuates + and the communication link itself might fail. As a result algorithms + such as those based on adaptive-bit-rate techniques that base their + policy on heuristics or models of deployment perform sub-optimally in + such dynamic environments.[ABR_1]. We motivate these issues with a + use-case that we present in the following sections. 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 3. Use Case We now describe a use case that involves an application with AR @@ -238,89 +263,94 @@ transmission times.In addition, edge devices and communication links may fail and logical communication relationships between various software components change frequently as the user moves around with their AR device [UBICOMP]. Thus, once the offloaded computationally intensive processing is completed on the Edge Computing, the video is streamed to the user with the help of an ABR algorithm which needs to meet the following requirements [ABR_1]: - o Dynamically changing ABR parameters: The ABR algorithm must be + * Dynamically changing ABR parameters: The ABR algorithm must be able to dynamically change parameters given the heavy-tailed nature of network throughput. This, for example, may be accomplished by AI/ML processing on the Edge Computing on a per client or global basis. - o Handling conflicting QoE requirements: QoE goals often require + * Handling conflicting QoE requirements: QoE goals often require high bit-rates, and low frequency of buffer refills. However in practice, this can lead to a conflict between those goals. For example, increasing the bit-rate might result in the need to fill up the buffer more frequently as the buffer capacity might be limited on the AR device. The ABR algorithm must be able to handle this situation. - o Handling side effects of deciding a specific bit rate: For + * Handling side effects of deciding a specific bit rate: For example, selecting a bit rate of a particular value might result in the ABR algorithm not changing to a different rate so as to ensure a non-fluctuating bit-rate and the resultant smoothness of video quality . The ABR algorithm must be able to handle this situation. 5. AR Network Traffic and Interaction with TCP In addition to the requirements for ABR algorithms, there are other operational issues that need to be considered for AR use cases such as the one descibed above. In a study [AR_TRAFFIC] conducted to characterize multi-user AR over cellular networks, the following issues were identified: - o The uploading of data from an AR device to a remote server for + * The uploading of data from an AR device to a remote server for processing dominates the end-to-end latency. - o A lack of visual features in the grid environment can cause + * A lack of visual features in the grid environment can cause increased latencies as the AR device uploads additional visual data for processing to the remote server. - o AR applications tend to have large bursts that are separated by + * AR applications tend to have large bursts that are separated by significant time gaps. As a result, the TCP congestion window enters slow start before the large bursts of data arrive increasing the perceived user latency. The study [AR_TRAFFIC] shows that segmentation latency at 4G LTE (Long Term Evolution)'s RAN (Radio Access Network)'s RLC (Radio Link Control) layer impacts TCP's performance during slow-start. 6. Informative References [ABR_1] Mao, H., Netravali, R., and M. Alizadeh, "Neural Adaptive Video Streaming with Pensieve", In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197-210, 2017. [ABR_2] Yan, F., Ayers, H., Zhu, C., Fouladi, S., Hong, J., Zhang, K., Levis, P., and K. Winstein, "Learning in situ: a - randomized experiment in video streaming", In 17th - USENIX Symposium on Networked Systems Design and - Implementation (NSDI 20), pp. 495-511, 2020. + randomized experiment in video streaming", In 17th USENIX + Symposium on Networked Systems Design and Implementation + (NSDI 20), pp. 495-511, 2020. [AR_TRAFFIC] Apicharttrisorn, K., Balasubramanian, B., Chen, J., Sivaraj, R., Tsai, Y., Jana, R., Krishnamurthy, S., Tran, T., and Y. Zhou, "Characterization of Multi-User Augmented Reality over Cellular Networks", In 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 1-9. IEEE, 2020. [AUGMENTED] - Schmalstieg, D. and T. Hollerer, "Augmented + Schmalstieg, D. S. and T.H. Hollerer, "Augmented Reality", Addison Wesley, 2016. + [AUGMENTED_2] + Azuma, R. T., "A Survey of Augmented + Reality.", Presence:Teleoperators and Virtual + Environments 6.4, pp. 355-385., 1997. + [BATT_DRAIN] Seneviratne, S., Hu, Y., Nguyen, T., Lan, G., Khalifa, S., Thilakarathna, K., Hassan, M., and A. Seneviratne, "A survey of wearable devices and challenges.", In IEEE Communication Surveys and Tutorials, 19(4), p.2573-2620., 2017. [BLUR] Kan, P. and H. Kaufmann, "Physically-Based Depth of Field in Augmented Reality.", In Eurographics (Short Papers), pp. 89-92., 2012. @@ -343,20 +373,24 @@ In Sensors, 20(5), p.1446., 2020. [EDGE_1] Satyanarayanan, M., "The Emergence of Edge Computing", In Computer 50(1) pp. 30-39, 2017. [EDGE_2] Satyanarayanan, M., Klas, G., Silva, M., and S. Mangiante, "The Seminal Role of Edge-Native Applications", In IEEE International Conference on Edge Computing (EDGE) pp. 33-40, 2019. + [EDGE_3] Peterson, L. and O. Sunay, "5G mobile networks: A systems + approach.", In Synthesis Lectures on Network Systems., + 2020. + [GLB_ILLUM_1] Kan, P. and H. Kaufmann, "Differential irradiance caching for fast high-quality light transport between virtual and real worlds.", In IEEE International Symposium on Mixed and Augmented Reality (ISMAR),pp. 133-141, 2013. [GLB_ILLUM_2] Franke, T., "Delta voxel cone tracing.", In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 39-44, 2014. @@ -365,70 +399,72 @@ Crovella, M. and B. Krishnamurthy, "Internet measurement: infrastructure, traffic and applications", John Wiley and Sons Inc., 2006. [HEAVY_TAIL_2] Taleb, N., "The Statistical Consequences of Fat Tails", STEM Academic Press, 2020. [I-D.ietf-mops-streaming-opcons] Holland, J., Begen, A., and S. Dawkins, "Operational - Considerations for Streaming Media", draft-ietf-mops- - streaming-opcons-07 (work in progress), September 2021. + Considerations for Streaming Media", Work in Progress, + Internet-Draft, draft-ietf-mops-streaming-opcons-09, 1 + March 2022, . [LENS_DIST] Fuhrmann, A. and D. Schmalstieg, "Practical calibration procedures for augmented reality.", In Virtual Environments 2000, pp. 3-12. Springer, Vienna, 2000. - [NOISE] Fischer, J., Bartz, D., and W. Strasser, "Enhanced visual + [NOISE] Fischer, J., Bartz, D., and W. Straßer, "Enhanced visual realism by incorporating camera image effects.", - In IEEE/ACM International Symposium on Mixed and - Augmented Reality, pp. 205-208., 2006. + In IEEE/ACM International Symposium on Mixed and Augmented + Reality, pp. 205-208., 2006. - [OCCL_1] Breen, D., Whitaker, R., and M. Tuceryan, "Interactive + [OCCL_1] Breen, D.E., Whitaker, R.T., and M. Tuceryan, "Interactive Occlusion and automatic object placementfor augmented - reality", In Computer Graphics Forum, vol. 15, no. 3 , - pp. 229-238,Edinburgh, UK: Blackwell Science Ltd, 1996. + reality", In Computer Graphics Forum, vol. 15, no. 3 , pp. + 229-238,Edinburgh, UK: Blackwell Science Ltd, 1996. [OCCL_2] Zheng, F., Schmalstieg, D., and G. Welch, "Pixel-wise closed-loop registration in video-based augmented reality", In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 135-143, 2014. [OCCL_3] Lang, B., "Oculus Shares 5 Key Ingredients for Presence in Virtual Reality.", https://www.roadtovr.com/oculus- shares-5-key-ingredients-for-presence-in-virtual-reality/, 2014. [PER_SENSE] - Mania, K., Adelstein, B., Ellis, S., and M. Hill, + Mania, K., Adelstein, B.D., Ellis, S.R., and M.I. Hill, "Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity.", In Proceedings of the 1st Symposium on Applied perception in graphics and visualization pp. 39-47., 2004. [PHOTO_REG] Liu, Y. and X. Granier, "Online tracking of outdoor lighting variations for augmented reality with moving cameras", In IEEE Transactions on visualization and computer graphics, 18(4), pp.573-580, 2012. - [PREDICT] Buker, T., Vincenzi, D., and J. Deaton, "The effect of - apparent latency on simulator sickness while using a see- - through helmet-mounted display: Reducing apparent latency - with predictive compensation..", In Human factors 54.2, - pp. 235-249., 2012. + [PREDICT] Buker, T. J., Vincenzi, D.A., and J.E. Deaton, "The effect + of apparent latency on simulator sickness while using a + see-through helmet-mounted display: Reducing apparent + latency with predictive compensation..", In Human factors + 54.2, pp. 235-249., 2012. - [REG] Holloway, R., "Registration error analysis for augmented - reality.", In Presence:Teleoperators and Virtual + [REG] Holloway, R. L., "Registration error analysis for + augmented reality.", In Presence:Teleoperators and Virtual Environments 6.4, pp. 413-432., 1997. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [SLAM_1] Ventura, J., Arth, C., Reitmayr, G., and D. Schmalstieg, "A minimal solution to the generalized pose-and-scale problem", In Proceedings of the IEEE Conference on @@ -440,27 +476,27 @@ problem", In European Conference on Computer Vision, pp. 16-31, 2014. [SLAM_3] Gauglitz, S., Sweeny, C., Ventura, J., Turk, M., and T. Hollerer, "Model estimation and selection towards unconstrained real-time tracking and mapping", In IEEE transactions on visualization and computer graphics, 20(6), pp. 825-838, 2013. [SLAM_4] Pirchheim, C., Schmalstieg, D., and G. Reitmayr, "Handling - pure camera rotation in keyframe-based SLAM", In 2013 - IEEE international symposium on mixed and augmented - reality (ISMAR), pp. 229-238, 2013. + pure camera rotation in keyframe-based SLAM", In 2013 IEEE + international symposium on mixed and augmented reality + (ISMAR), pp. 229-238, 2013. [UBICOMP] Bardram, J. and A. Friday, "Ubiquitous Computing Systems", - In Ubiquitous Computing Fundamentals pp. 37-94. CRC - Press, 2009. + In Ubiquitous Computing Fundamentals pp. 37-94. CRC Press, + 2009. [URLLC] 3GPP, "3GPP TR 23.725: Study on enhancement of Ultra- Reliable Low-Latency Communication (URLLC) support in the 5G Core network (5GC).", https://portal.3gpp.org/desktopmodules/Specifications/ SpecificationDetails.aspx?specificationId=3453, 2019. [VIS_INTERFERE] Kalkofen, D., Mendez, E., and D. Schmalstieg, "Interactive focus and context visualization for augmented reality.", @@ -465,25 +501,25 @@ Kalkofen, D., Mendez, E., and D. Schmalstieg, "Interactive focus and context visualization for augmented reality.", In 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 191-201., 2007. [XR] 3GPP, "3GPP TR 26.928: Extended Reality (XR) in 5G.", https://portal.3gpp.org/desktopmodules/Specifications/ SpecificationDetails.aspx?specificationId=3534, 2020. Authors' Addresses + Renan Krishna InterDigital Europe Limited 64, Great Eastern Street - London EC2A 3QR + London + EC2A 3QR United Kingdom - Email: renan.krishna@interdigital.com Akbar Rahman InterDigital Communications, LLC 1000 Sherbrooke Street West Montreal H3A 3G4 Canada - Email: Akbar.Rahman@InterDigital.com