Researchers from NVIDIA developed an AI imaging system that can restore severely damaged photos.
In a study published by NVIDIA’s research and development team on arXiv.org, a new AI imaging technique is described to have fixed corrupted images with holes or missing pixels.
Aside from editing and restoring images, the team’s new method can also remove content and fill the resulting gap with a new one. Dubbed as “image inpainting,” the technique could be applied in photo editing software to take off the unwanted part of an image and replace it with a realistic computer-generated substitute.
“Our model can robustly handle holes of any shape, size location, or distance from the image borders. Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing. Further, our model gracefully handles holes of increasing size,” the researchers wrote in their paper.
While this is not the first time that such AI imaging technique has been used to fill in missing gaps in a corrupted photo, previous attempts have been limited to rectangular sections only. Aside from that, the focus of the holes was mostly in the middle of the picture and they have insufficient scaling sizes.
In a blog post by NVIDIA, the researchers reportedly used NVIDIA Tesla V100 GPUs and cuDNN-accelerated PyTorch deep learning framework in training their neural network. They do this by applying the “generated masks to images from the ImageNet, Places2 and CelebA-HQ datasets.”
“During the training phase, holes or missing parts are introduced into complete training images from the above datasets, to enable the network to learn to reconstruct the missing pixels,” the post read.
However, during the testing phase, the team introduced different holes or missing parts not applied during the training into the test images in the dataset. This was done to perform unbiased validation of reconstruction accuracy.
To fix issues like color discrepancy and blurriness in the images during inpainting, the team utilized a method that ensures the output for missing pixels will not depend on the input value produced by those same pixels. The method relies on a partial convolution layer that “renormalizes each output depending on the validity of its corresponding receptive field.”
According to the team, the renormalization ensures that the value of the output is not dependent on the values of the missing pixels in each “receptive” field.
“The model is built from a UNet architecture implemented with these partial convolutions. A set of loss functions, matching feature losses with a VGG model, as well as style losses, were used to train the model to produce realistic outputs. Because of this, the model outperforms previous methods,” the team said.
Comments (0)
Most Recent