Structured Prediction using cGANs with Fusion Discriminator

Mahmood F, Xu Wenhao, Durr NJ, Johnson JW, Yuille A
ICLR Workshop Paper 2019
https://arxiv.org/abs/1808.07528arXiv

Abstract

We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks. This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data. This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree. The proposed method is conceptually simpler than the joint convolutional neural network – conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials. Experimental results demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation.

Keywords: Generative Adversarial Networks, GANs, conditional GANs, Discriminator, Fusion

TL;DR: We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.