Earlier in the year, you heard about how foveated rendering is the future of VR. Now, Facebook has taken the technology a step further with its DeepFovea AI.
In September 2018, Facebook-owned virtual reality company, Oculus announced a couple of additions to its Oculus Go VR headset. These included YouTube VR support as well as new visual rendering features to improve optical quality.
However, the headset’s new ability to perform fixed foveated rendering may have been the most exciting update at the time.
What is Foveated Rendering
It’s a technique that enables the VR headset to render sharp details for the eye’s visual sweet spot – the fovea. Meanwhile, the peripheral vision gets a simpler, blurrier version of the image.
Here’s how it works.
Cameras in a VR headset can precisely and quickly track the position of a user’s pupils. This allows the GPU to know where it needs to focus its rendering resources and where to skimp.
Compared with standard rendering, foveated rendering saves more processing power. In turn, the headset can conserve energy or use the extra processing power to increase detail within the area viewed by the pupil.
As impressive foveated rendering is, the engineers at Facebook Reality Labs have made it even better. They developed DeepFovea, an AI-assisted alternative that does what the researchers are calling “foveated reconstruction.”
Instead of rendering accurate peripheral imagery, DeepFovea AI reconstructs pixels for the peripheral vision.
How DeepFovea AI Foveated Rendering Works
DeepFovea AI uses a trained generative adversarial neural network to reconstruct each frame from a tiny sample in the peripheral area. Then the stream’s temporal and spatial content fills in the rest of the details.
The resulting pixels may not be as sharp or detailed as the reference image, but they retain more of the original shapes and colors. By comparison, a traditional foveated rendering system would depict those pixels as low-resolution flat-shaded blocks.
Why does this matter?
Along with being more power efficient, DeepFovea AI also produces a more superior image quality than standard foveated rendering. What’s more, it achieves over 14 times compression on RGB video with no significant degradation in user-perceived quality.
As a result, users can watch real-time, low-latency videos that depend on gaze detection. That’s a fundamental step towards developing a headset that’ll display high-resolution graphics rendered initially in the cloud.
Facebook is releasing a sample version of DeepFovea architecture for researchers, graphics engineers, and VR engineers.
Comments (0)
Most Recent