Viewport: A Fully Distributed Immersive Teleconferencing System with Infrared Dot Pattern

In this paper, we present Viewport, a fully distributed system for immersive teleconferencing. The participant at each site is captured by a camera rig, which contains multiple color and infrared cameras and multiple infrared projectors. The projected infrared dot pattern is used to help reconstruct a 3D representation in real-time, which will be rendered in a shared virtual environment for immersive telecommunication. We conduct a multimodal fusion scheme for multiview stereo and 3D reconstruction, which combines the knowledge of the infrared dot patterns from the projectors and the observations of IR and RGB cameras. One novel aspect of our approach is the use of sparse point cloud instead of dense multiview stereo for geometry reconstruction, which leads to significant speed up in 3D reconstruction and rendering. In addition, we introduce the scheme of virtual seating, where the point clouds are positioned to maintain the same seating geometry as face-to-face meetings after careful calibration, such that the mutual gaze between participants are faithfully maintained.