Technology 3 min read

Facebook's DeepFovea AI Offers Sharper VR Foveated Rendering

Andrush /

Andrush /

Earlier in the year, you heard about how foveated rendering is the future of VR. Now, Facebook has taken the technology a step further with its DeepFovea AI.

In September 2018, Facebook-owned virtual reality company, Oculus announced a couple of additions to its Oculus Go VR headset. These included YouTube VR support as well as new visual rendering features to improve optical quality.

However, the headset’s new ability to perform fixed foveated rendering may have been the most exciting update at the time.

What is Foveated Rendering

It’s a technique that enables the VR headset to render sharp details for the eye’s visual sweet spot – the fovea. Meanwhile, the peripheral vision gets a simpler, blurrier version of the image.

Here’s how it works.

Cameras in a VR headset can precisely and quickly track the position of a user’s pupils. This allows the GPU to know where it needs to focus its rendering resources and where to skimp.

Compared with standard rendering, foveated rendering saves more processing power. In turn, the headset can conserve energy or use the extra processing power to increase detail within the area viewed by the pupil.

As impressive foveated rendering is, the engineers at Facebook Reality Labs have made it even better. They developed DeepFovea, an AI-assisted alternative that does what the researchers are calling “foveated reconstruction.”

Instead of rendering accurate peripheral imagery, DeepFovea AI reconstructs pixels for the peripheral vision.

How DeepFovea AI Foveated Rendering Works

DeepFovea AI uses a trained generative adversarial neural network to reconstruct each frame from a tiny sample in the peripheral area. Then the stream’s temporal and spatial content fills in the rest of the details.

The resulting pixels may not be as sharp or detailed as the reference image, but they retain more of the original shapes and colors. By comparison, a traditional foveated rendering system would depict those pixels as low-resolution flat-shaded blocks.

Why does this matter?

Along with being more power efficient, DeepFovea AI also produces a more superior image quality than standard foveated rendering. What’s more, it achieves over 14 times compression on RGB video with no significant degradation in user-perceived quality.

As a result, users can watch real-time, low-latency videos that depend on gaze detection. That’s a fundamental step towards developing a headset that’ll display high-resolution graphics rendered initially in the cloud.

Facebook is releasing a sample version of DeepFovea architecture for researchers, graphics engineers, and VR engineers.

Read More: Negative Impact of Using VR Documentaries for Charities

First AI Web Content Optimization Platform Just for Writers

Found this article interesting?

Let Sumbo Bello know how much you appreciate this article by clicking the heart icon and by sharing this article on social media.

Profile Image

Sumbo Bello

Sumbo Bello is a creative writer who enjoys creating data-driven content for news sites. In his spare time, he plays basketball and listens to Coldplay.

Comments (0)
Most Recent most recent
share Scroll to top

Link Copied Successfully

Sign in

Sign in to access your personalized homepage, follow authors and topics you love, and clap for stories that matter to you.

Sign in with Google Sign in with Facebook

By using our site you agree to our privacy policy.