Mobile Virtual Reality for Head-mounted Displays With Interactive Streaming Video and Likelihood-based Foveation

Immersive virtual reality has long been an aspiration
of many. For a truly immersive VR experience, three
properties are essential: quality, responsiveness and mobility.
By quality, we mean that realistic and life-like
visual portrayals in a virtual environment heighten our
sense of immersion. By responsiveness, we mean that
any motion, especially of the head, must be reflected
as quickly as possible in visual feedback because ocular
proprioception sensitivity is very high. By mobility, we
mean that we ought to be able to move untethered in
physical space, free to explore our virtual world. Unfortunately
mobile devices like phones are tablets are
power-constrained and cannot produce acceptable framerate.
While cloud offloading allows them to deliver
high-quality and framerate it places high bandwidth requirements
and introduces an unacceptable amount of
latency.
This demo introduces Matia (formerly Irides), a stereo HMD system
that simultaneously attains quality, responsiveness and
mobility. Matia achieves this by offloading rendering
work to a high-end GPU across the WAN. The HMD
client is mobile and only requires a low-end GPU, yet
receives high quality imagery. To overcome WAN latencies,
Matia borrows speculative execution techniques
inspired by previous work Outatime [1]. However, speculative
execution alone is insufficient. This is because
HMD sensitivities are substantially more stringent along
several key dimensions: responsiveness and quality.These
factors suggest that we ought to greatly benefit from
maximally utilizing all available bandwidth between client
and server to deliver the highest resolution imagery possible.
At the same time, any latency or bandwidth
changes must be handled very responsively.