draft-ietf-mops-ar-use-case-02.txt		draft-ietf-mops-ar-use-case-03.txt
	MOPS R. Krishna		MOPS R. Krishna
	Internet-Draft InterDigital Europe Limited		Internet-Draft InterDigital Europe Limited
	Intended status: Informational A. Rahman		Intended status: Informational A. Rahman
	Expires: January 29, 2022 InterDigital Communications, LLC		Expires: April 28, 2022 InterDigital Communications, LLC
	July 28, 2021		October 25, 2021
	Media Operations Use Case for an Augmented Reality Application on Edge		Media Operations Use Case for an Augmented Reality Application on Edge
	Computing Infrastructure		Computing Infrastructure
	draft-ietf-mops-ar-use-case-02		draft-ietf-mops-ar-use-case-03
	Abstract		Abstract
	A use case describing transmission of an application on the Internet		A use case describing transmission of an application on the Internet
	that has several unique characteristics of Augmented Reality (AR)		that has several unique characteristics of Augmented Reality (AR)
	applications is presented for the consideration of the Media		applications is presented for the consideration of the Media
	Operations (MOPS) Working Group. One key requirement identified is		Operations (MOPS) Working Group. One key requirement identified is
	that the Adaptive-Bit-Rate (ABR) algorithms' current usage of		that the Adaptive-Bit-Rate (ABR) algorithms' current usage of
	policies based on heuristics and models is inadequate for AR		policies based on heuristics and models is inadequate for AR
	applications running on the Edge Computing infrastructure.		applications running on the Edge Computing infrastructure.
	skipping to change at page 1, line 38 ¶		skipping to change at page 1, line 38 ¶
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at https://datatracker.ietf.org/drafts/current/.		Drafts is at https://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on January 29, 2022.		This Internet-Draft will expire on April 28, 2022.
	Copyright Notice		Copyright Notice
	Copyright (c) 2021 IETF Trust and the persons identified as the		Copyright (c) 2021 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(https://trustee.ietf.org/license-info) in effect on the date of		(https://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	skipping to change at page 2, line 17 ¶		skipping to change at page 2, line 17 ¶
	described in the Simplified BSD License.		described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
	2. Conventions used in this document . . . . . . . . . . . . . . 3		2. Conventions used in this document . . . . . . . . . . . . . . 3
	3. Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . 3		3. Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . 3
	3.1. Processing of Scenes . . . . . . . . . . . . . . . . . . 3		3.1. Processing of Scenes . . . . . . . . . . . . . . . . . . 3
	3.2. Generation of Images . . . . . . . . . . . . . . . . . . 4		3.2. Generation of Images . . . . . . . . . . . . . . . . . . 4
	4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4		4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4
	5. Informative References . . . . . . . . . . . . . . . . . . . 6		5. AR Network Traffic and Interaction with TCP . . . . . . . . . 6
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9		6. Informative References . . . . . . . . . . . . . . . . . . . 7
			Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10
	1. Introduction		1. Introduction
	The MOPS draft, [I-D.ietf-mops-streaming-opcons], provides an		The MOPS draft, [I-D.ietf-mops-streaming-opcons], provides an
	overview of operational networking issues that pertain to Quality of		overview of operational networking issues that pertain to Quality of
	Experience (QoE) in delivery of video and other high-bitrate media		Experience (QoE) in delivery of video and other high-bitrate media
	over the Internet. However, as it does not cover the increasingly		over the Internet. However, as it does not cover the increasingly
	large number of applications with Augmented Reality (AR)		large number of applications with Augmented Reality (AR)
	characteristics and their requirements on ABR algorithms, the		characteristics and their requirements on ABR algorithms, the
	discussion in this draft compliments the overview presented in that		discussion in this draft compliments the overview presented in that
	skipping to change at page 3, line 16 ¶		skipping to change at page 3, line 17 ¶
	deployments [ABR_1].		deployments [ABR_1].
	2. Conventions used in this document		2. Conventions used in this document
	The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",		The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
	"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this		"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
	document are to be interpreted as described in [RFC2119].		document are to be interpreted as described in [RFC2119].
	3. Use Case		3. Use Case
	We now descibe a use case that involves an application with AR		We now describe a use case that involves an application with AR
	systems' characteristics. Consider a group of tourists who are being		systems' characteristics. Consider a group of tourists who are being
	conducted in a tour around the historical site of the Tower of		conducted in a tour around the historical site of the Tower of
	London. As they move around the site and within the historical		London. As they move around the site and within the historical
	buildings, they can watch and listen to historical scenes in 3D that		buildings, they can watch and listen to historical scenes in 3D that
	are generated by the AR application and then overlaid by their AR		are generated by the AR application and then overlaid by their AR
	headsets onto their real-world view. The headset then continuously		headsets onto their real-world view. The headset then continuously
	updates their view as they move around.		updates their view as they move around.
	The AR application first processes the scene that the walking tourist		The AR application first processes the scene that the walking tourist
	is watching in real-time and identifies objects that will be targeted		is watching in real-time and identifies objects that will be targeted
	for overlay of high resolution videos. It then generates high		for overlay of high resolution videos. It then generates high
	resolution 3D images of historical scenes related to the perspective		resolution 3D images of historical scenes related to the perspective
	of the tourist in real-time. These generated video images are then		of the tourist in real-time. These generated video images are then
	overlaid on the view of the real-world as seen by the tourist.		overlaid on the view of the real-world as seen by the tourist.
	We now discuss this processsing of scenes and generation of high		We now discuss this processing of scenes and generation of high
	resolution images in greater detail.		resolution images in greater detail.
	3.1. Processing of Scenes		3.1. Processing of Scenes
	The AR application that runs on the mobile device needs to first		The task of processing a scene can be broken down into a pipeline of
	track the pose (coordinates and orientation) of the user's head, eyes		three consecutive subtasks namely tracking, followed by an
	and the objects that are in view.This requires tracking natural		acquisition of a model of the real world, and finally registration
	features and developing an annotated point cloud based model that is		[AUGMENTED].
	then stored in a database.To ensure that this database can be scaled
	up,techniques such as combining a client side simultaneous tracking
	and mapping and a server-side localization are used[SLAM_1],
	[SLAM_2], [SLAM_3], [SLAM_4]. Once the natural features are tracked,
	virtual objects are geometrically aligned with those features.This is
	followed by resolving occlusion that can occur between virtual and
	the real objects [OCCL_1], [OCCL_2].
	The next step for the AR apllication is to apply photometric		Tracking: This includes tracking of the three dimensional coordinates
	registration [PHOTO_REG]. This requires aligning the brightness and		and six dimensional pose (coordinates and orientation) of objects in
	color between the virtual and real objects.Additionally, algorithms		the real world[AUGMENTED]. The AR application that runs on the
	that calculate global illumination of both the virtual and real		mobile device needs to track the pose of the user's head, eyes and
	objects [GLB_ILLUM_1], [GLB_ILLUM_2] are executed.Various algorithms		the objects that are in view.This requires tracking natural features
	to deal with artifacts generated by lens distortion [LENS_DIST], blur		that are then used in the next stage of the pipeline.
	[BLUR], noise [NOISE] etc are also required.
			Acquisition of a model of the real world: The tracked natural
			features are used to develop an annotated point cloud based model
			that is then stored in a database.To ensure that this database can be
			scaled up,techniques such as combining a client side simultaneous
			tracking and mapping and a server-side localization are used[SLAM_1],
			[SLAM_2], [SLAM_3], [SLAM_4].
			Registration: The coordinate systems, brightness, and color of
			virtual and real objects need to be aligned in a process called
			registration [REG]. Once the natural features are tracked as
			discussed above, virtual objects are geometrically aligned with those
			features by geometric registration .This is followed by resolving
			occlusion that can occur between virtual and the real objects
			[OCCL_1], [OCCL_2]. The AR application also applies photometric
			registration [PHOTO_REG] by aligning the brightness and color between
			the virtual and real objects.Additionally, algorithms that calculate
			global illumination of both the virtual and real objects
			[GLB_ILLUM_1], [GLB_ILLUM_2] are executed.Various algorithms to deal
			with artifacts generated by lens distortion [LENS_DIST], blur [BLUR],
			noise [NOISE] etc are also required.
	3.2. Generation of Images		3.2. Generation of Images
	The AR application must generate a high-quality video that has the		The AR application must generate a high-quality video that has the
	properties descibed in the previous step and overlay the video on the		properties described in the previous step and overlay the video on
	AR device's display- a step called situated visualization. This		the AR device's display- a step called situated visualization. This
	entails dealing with registration errors that may arise, esuring that		entails dealing with registration errors that may arise, ensuring
	there is no visual interference [VIS_INTERFERE], and finally		that there is no visual interference [VIS_INTERFERE], and finally
	maintaining temporal coherence by adapting to the movement of user's		maintaining temporal coherence by adapting to the movement of user's
	eyes and head.		eyes and head.
	4. Requirements		4. Requirements
	The components of AR applications perform tasks such as real-time		The components of AR applications perform tasks such as real-time
	generation and processing of high-quality video content that are		generation and processing of high-quality video content that are
	computationally intensive. As a result,on AR devices such as AR		computationally intensive. As a result,on AR devices such as AR
	glasses excessive heat is generated by the chip-sets that are		glasses excessive heat is generated by the chip-sets that are
	involved in the computation [DEV_HEAT_1], [DEV_HEAT_2].		involved in the computation [DEV_HEAT_1], [DEV_HEAT_2].
	Additionally, the battery on such devices discharges quickly when		Additionally, the battery on such devices discharges quickly when
	running such applications [BATT_DRAIN].		running such applications [BATT_DRAIN].
	A solution to the heat dissipation and battery drainge problem is to		A solution to the heat dissipation and battery drainage problem is to
	offload the processing and video generation tasks to the remote		offload the processing and video generation tasks to the remote
	cloud.However, running such tasks on the cloud is not feasible as the		cloud.However, running such tasks on the cloud is not feasible as the
	end-to-end delays must be within the order of a few milliseconds.		end-to-end delays must be within the order of a few milliseconds.
	Additionally,such applications require high bandwidth and low jitter		Additionally,such applications require high bandwidth and low jitter
	to provide a high QoE to the user.In order to achieve such hard		to provide a high QoE to the user.In order to achieve such hard
	timing constraints, computationally intensive tasks can be offloaded		timing constraints, computationally intensive tasks can be offloaded
	to Edge devices.		to Edge devices.
	Another requirement for our use case and similar applications such as		Another requirement for our use case and similar applications such as
	360 degree streaming is that the display on the AR/VR device should		360 degree streaming is that the display on the AR/VR device should
	skipping to change at page 6, line 20 ¶		skipping to change at page 6, line 33 ¶
	limited on the AR device. The ABR algorithm must be able to		limited on the AR device. The ABR algorithm must be able to
	handle this situation.		handle this situation.
	o Handling side effects of deciding a specific bit rate: For		o Handling side effects of deciding a specific bit rate: For
	example, selecting a bit rate of a particular value might result		example, selecting a bit rate of a particular value might result
	in the ABR algorithm not changing to a different rate so as to		in the ABR algorithm not changing to a different rate so as to
	ensure a non-fluctuating bit-rate and the resultant smoothness of		ensure a non-fluctuating bit-rate and the resultant smoothness of
	video quality . The ABR algorithm must be able to handle this		video quality . The ABR algorithm must be able to handle this
	situation.		situation.
	5. Informative References		5. AR Network Traffic and Interaction with TCP
			In addition to the requirements for ABR algorithms, there are other
			operational issues that need to be considered for AR use cases such
			as the one descibed above. In a study [AR_TRAFFIC] conducted to
			characterize multi-user AR over cellular networks, the following
			issues were identified:
			o The uploading of data from an AR device to a remote server for
			processing dominates the end-to-end latency.
			o A lack of visual features in the grid environment can cause
			increased latencies as the AR device uploads additional visual
			data for processing to the remote server.
			o AR applications tend to have large bursts that are separated by
			significant time gaps. As a result, the TCP congestion window
			enters slow start before the large bursts of data arrive
			increasing the perceived user latency. The study [AR_TRAFFIC]
			shows that segmentation latency at 4G LTE (Long Term Evolution)'s
			RAN (Radio Access Network)'s RLC (Radio Link Control) layer
			impacts TCP's performance during slow-start.
			6. Informative References
	[ABR_1] Mao, H., Netravali, R., and M. Alizadeh, "Neural Adaptive		[ABR_1] Mao, H., Netravali, R., and M. Alizadeh, "Neural Adaptive
	Video Streaming with Pensieve", In Proceedings of the		Video Streaming with Pensieve", In Proceedings of the
	Conference of the ACM Special Interest Group on Data		Conference of the ACM Special Interest Group on Data
	Communication, pp. 197-210, 2017.		Communication, pp. 197-210, 2017.
	[ABR_2] Yan, F., Ayers, H., Zhu, C., Fouladi, S., Hong, J., Zhang,		[ABR_2] Yan, F., Ayers, H., Zhu, C., Fouladi, S., Hong, J., Zhang,
	K., Levis, P., and K. Winstein, "Learning in situ: a		K., Levis, P., and K. Winstein, "Learning in situ: a
	randomized experiment in video streaming", In 17th		randomized experiment in video streaming", In 17th
	USENIX Symposium on Networked Systems Design and		USENIX Symposium on Networked Systems Design and
	Implementation (NSDI 20), pp. 495-511, 2020.		Implementation (NSDI 20), pp. 495-511, 2020.
			[AR_TRAFFIC]
			Apicharttrisorn, K., Balasubramanian, B., Chen, J.,
			Sivaraj, R., Tsai, Y., Jana, R., Krishnamurthy, S., Tran,
			T., and Y. Zhou, "Characterization of Multi-User Augmented
			Reality over Cellular Networks", In 17th Annual IEEE
			International Conference on Sensing, Communication, and
			Networking (SECON), pp. 1-9. IEEE, 2020.
			[AUGMENTED]
			Schmalstieg, D. and T. Hollerer, "Augmented
			Reality", Addison Wesley, 2016.
	[BATT_DRAIN]		[BATT_DRAIN]
	Seneviratne, S., Hu, Y., Nguyen, T., Lan, G., Khalifa, S.,		Seneviratne, S., Hu, Y., Nguyen, T., Lan, G., Khalifa, S.,
	Thilakarathna, K., Hassan, M., and A. Seneviratne, "A		Thilakarathna, K., Hassan, M., and A. Seneviratne, "A
	survey of wearable devices and challenges.", In IEEE		survey of wearable devices and challenges.", In IEEE
	Communication Surveys and Tutorials, 19(4), p.2573-2620.,		Communication Surveys and Tutorials, 19(4), p.2573-2620.,
	2017.		2017.
	[BLUR] Kan, P. and H. Kaufmann, "Physically-Based Depth of Field		[BLUR] Kan, P. and H. Kaufmann, "Physically-Based Depth of Field
	in Augmented Reality.", In Eurographics (Short Papers),		in Augmented Reality.", In Eurographics (Short Papers),
	pp. 89-92., 2012.		pp. 89-92., 2012.
	skipping to change at page 7, line 47 ¶		skipping to change at page 8, line 47 ¶
	infrastructure, traffic and applications", John Wiley and		infrastructure, traffic and applications", John Wiley and
	Sons Inc., 2006.		Sons Inc., 2006.
	[HEAVY_TAIL_2]		[HEAVY_TAIL_2]
	Taleb, N., "The Statistical Consequences of Fat Tails",		Taleb, N., "The Statistical Consequences of Fat Tails",
	STEM Academic Press, 2020.		STEM Academic Press, 2020.
	[I-D.ietf-mops-streaming-opcons]		[I-D.ietf-mops-streaming-opcons]
	Holland, J., Begen, A., and S. Dawkins, "Operational		Holland, J., Begen, A., and S. Dawkins, "Operational
	Considerations for Streaming Media", draft-ietf-mops-		Considerations for Streaming Media", draft-ietf-mops-
	streaming-opcons-06 (work in progress), July 2021.		streaming-opcons-07 (work in progress), September 2021.
	[LENS_DIST]		[LENS_DIST]
	Fuhrmann, A. and D. Schmalstieg, "Practical calibration		Fuhrmann, A. and D. Schmalstieg, "Practical calibration
	procedures for augmented reality.", In Virtual		procedures for augmented reality.", In Virtual
	Environments 2000, pp. 3-12. Springer, Vienna, 2000.		Environments 2000, pp. 3-12. Springer, Vienna, 2000.
	[NOISE] Fischer, J., Bartz, D., and W. Strasser, "Enhanced visual		[NOISE] Fischer, J., Bartz, D., and W. Strasser, "Enhanced visual
	realism by incorporating camera image effects.",		realism by incorporating camera image effects.",
	In IEEE/ACM International Symposium on Mixed and		In IEEE/ACM International Symposium on Mixed and
	Augmented Reality, pp. 205-208., 2006.		Augmented Reality, pp. 205-208., 2006.
	skipping to change at page 8, line 45 ¶		skipping to change at page 9, line 45 ¶
	lighting variations for augmented reality with moving		lighting variations for augmented reality with moving
	cameras", In IEEE Transactions on visualization and		cameras", In IEEE Transactions on visualization and
	computer graphics, 18(4), pp.573-580, 2012.		computer graphics, 18(4), pp.573-580, 2012.
	[PREDICT] Buker, T., Vincenzi, D., and J. Deaton, "The effect of		[PREDICT] Buker, T., Vincenzi, D., and J. Deaton, "The effect of
	apparent latency on simulator sickness while using a see-		apparent latency on simulator sickness while using a see-
	through helmet-mounted display: Reducing apparent latency		through helmet-mounted display: Reducing apparent latency
	with predictive compensation..", In Human factors 54.2,		with predictive compensation..", In Human factors 54.2,
	pp. 235-249., 2012.		pp. 235-249., 2012.
			[REG] Holloway, R., "Registration error analysis for augmented
			reality.", In Presence:Teleoperators and Virtual
			Environments 6.4, pp. 413-432., 1997.
	[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate		[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
	Requirement Levels", BCP 14, RFC 2119,		Requirement Levels", BCP 14, RFC 2119,
	DOI 10.17487/RFC2119, March 1997,		DOI 10.17487/RFC2119, March 1997,
	<https://www.rfc-editor.org/info/rfc2119>.		<https://www.rfc-editor.org/info/rfc2119>.
	[SLAM_1] Ventura, J., Arth, C., Reitmayr, G., and D. Schmalstieg,		[SLAM_1] Ventura, J., Arth, C., Reitmayr, G., and D. Schmalstieg,
	"A minimal solution to the generalized pose-and-scale		"A minimal solution to the generalized pose-and-scale
	problem", In Proceedings of the IEEE Conference on		problem", In Proceedings of the IEEE Conference on
	Computer Vision and Pattern Recognition, pp. 422-429,		Computer Vision and Pattern Recognition, pp. 422-429,
	2014.		2014.
	skipping to change at page 10, line 18 ¶		skipping to change at page 11, line 18 ¶
	United Kingdom		United Kingdom
	Email: renan.krishna@interdigital.com		Email: renan.krishna@interdigital.com
	Akbar Rahman		Akbar Rahman
	InterDigital Communications, LLC		InterDigital Communications, LLC
	1000 Sherbrooke Street West		1000 Sherbrooke Street West
	Montreal H3A 3G4		Montreal H3A 3G4
	Canada		Canada
	Email: rahmansakbar@yahoo.com		Email: Akbar.Rahman@InterDigital.com
End of changes. 15 change blocks.
	33 lines changed or deleted		86 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/