ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing - Robotics Institute Carnegie Mellon University

ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing

Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey
Conference Paper, Proceedings of (CVPR) Computer Vision and Pattern Recognition, pp. 9455 - 9464, June, 2018

Abstract

We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image. To achieve this, we propose a novel Generative Adversarial Network (GAN) architecture that utilizes Spatial Transformer Networks (STNs) as the generator, which we call Spatial Transformer GANs (ST-GANs). ST-GANs seek image realism by operating in the geometric warp parameter space. In particular, we exploit an iterative STN warping scheme and propose a sequential training strategy that achieves better results compared to naive training of a single generator. One of the key advantages of ST-GAN is its applicability to high-resolution images indirectly since the predicted warp
parameters are transferable between reference frames. We demonstrate our approach in two applications: (1) visualizing how indoor furniture (e.g. from product images) might be perceived in a room, (2) hallucinating how accessories like glasses would look when matched with real portraits.

BibTeX

@conference{Lin-2018-121024,
author = {Chen-Hsuan Lin and Ersin Yumer and Oliver Wang and Eli Shechtman and Simon Lucey},
title = {ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},
booktitle = {Proceedings of (CVPR) Computer Vision and Pattern Recognition},
year = {2018},
month = {June},
pages = {9455 - 9464},
}