VP9 bitstream-based Tiled Multipoint Control Unit: Scaling simultaneous RGBD user streams in an immersive 3D communication system

Video conference applications that allow group communication through video and audio over distance have become commonplace and mainstream. To scale the number of participants in video conferencing systems, the usual practice is to deploy centralized streaming components such as Multipoint Control Units (MCU) or Selective Forwarding Units (SFU). Using these components can become problematic due to significant resource overhead in the server or client. In this paper, we propose a Tile-aware Multipoint Control Unit (T-MCU) that is capable of combining and forwarding streams on a bitstream level. In our solution, multiple VP9 user video streams are combined into a single bitstream, requiring only a single video transmission (single receiving socket, single video decoder, and single rendering texture). Multiple changes (relating to dynamic header and motion vectors) had to be made to the reference VP9 encoder to allow for combining bitstream tiles. These changes are available as open source and are described in this paper, including their impact on encoding performance. Furthermore, all changes are fully compatible with existing VP9 decoders. The paper also presents an experimental design study to evaluate our new T-MCU under different streaming conditions (including the impact on the receiving client) within an immersive 3D communication system utilizing RGBD video data. Our approach significantly reduces the performance requirements of the client (10-20%) and server (∼80%) at the cost of increased bandwidth (∼16%). Ultimately, the T-MCU allows immersive 3D communication applications to support at least 16+ simultaneous users. (C) 2024 ACM.