Title
Generative RGB-D face completion for head-mounted display removal
Author
Numan, N.
Haar, F.T.
Cesar, P.
Publication year
2021
Abstract
Head-mounted displays (HMDs) are an essential display device for the observation of virtual reality (VR) environments. However, HMDs obstruct external capturing methods from recording the user’s upper face. This severely impacts social VR applications, such as teleconferencing, which commonly rely on external RGB-D sensors to capture a volumetric representation of the user. In this paper, we introduce an HMD removal framework based on generative adversarial networks (GANs), capable of jointly filling in missing color and depth data in RGB-D face images. Our framework includes an RGB based identity loss function for identity preservation and several components aimed at surface reproduction. Our results demonstrate that our framework is able to remove HMDs from synthetic RGB-D face images while preserving the subject’s identity
Subject
Artificial intelligence
Computer vision
Computing methodologies
Human computer interaction (HCI)
Interaction paradigms
Reconstruction
Human-centered computing
Virtual reality
Abstracting
Cell proliferation
Street traffic control
User interfaces
Virtual reality
Adversarial networks
Face images
Head mounted displays
Loss functions
Rgb-d sensors
Surface reproduction
Volumetric representation
VR applications
Helmet mounted displays
To reference this document use:
http://resolver.tudelft.nl/uuid:ef072136-bb97-45de-8d38-866f8332dc3b
DOI
https://doi.org/10.1109/vrw52623.2021.00028
TNO identifier
967972
Publisher
Institute of Electrical and Electronics Engineers Inc.
ISBN
9780738113678
Source
Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021, 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021, 27 March 2021 through 3 April 2021, 109-116
Document type
conference paper