The need

Generative Adversarial Networks (GANs) are powerful neural networks used for image generation.  While they can execute image completion tasks for small regions, GANs fail when asked to generate large-scale missing regions of an image.

The idea

GAN development has taken divergent approaches that have enabled a variety of image completion tasks. But we argue that GANs need greater generative capability to successfully fill in large missing regions than current methods provide.

The solution

We have developed a new co-modulated GAN, or CoModGAN, architecture that generates realistic imagery based on small amounts of visual information and does so better than previous models.

Technical details for CoModGAN

Generative Adversarial Networks execute image completion tasks by pitting two neural networks—a generator and discriminator—against each other such that this competitive relationship helps train both networks. While the generator is tasked with creating or completing an image, the discriminator analyzes how realistic the output image is relative to a dataset of real images.

More recently, we have seen the development of two different image completion algorithms. Image-conditional GANs can fill in small missing areas of an image. And unconditional GANs can generate completely new images. But if you ask either one to fill in a large region of an image, they fail.

We developed a co-modulated GAN, or CoModGAN, that bridges the generative capability successes of unconditional generators with the image-completion successes of image-conditional generators.

CoModGAN, like other GANs, is trained with discriminator losses, but unlike others it can get smarter at filling in large missing regions of an image. At the same time, we propose a new Paired/Unpaired Inception Discriminative Score metric to robustly measure the realness of generated images compared to the real ones.

We hope that CoModGAN can help create AI that can more naturally and successfully execute image completion tasks.

Projects related to CoModGAN

Browse more innovation sandbox stories

Explore the possibilities of AI

Jumpstart your own AI innovations with learning resources and development solutions from Microsoft AI.