Swapping Autoencoder for Deep Image Manipulation - Robotics Institute Carnegie Mellon University

Swapping Autoencoder for Deep Image Manipulation

Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang
Conference Paper, Proceedings of (NeurIPS) Neural Information Processing Systems, December, 2020

Abstract

Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging. We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation, rather than random sampling. The key idea is to encode an image into two independent components and enforce that any swapped combination maps to a realistic image. In particular, we encourage the components to represent structure and texture, by enforcing one component to encode co-occurrent patch statistics across different parts of the image. As our method is trained with an encoder, finding the latent codes for a new input image becomes trivial, rather than cumbersome. As a result, our method enables us to manipulate real input images in various ways, including texture swapping, local and global editing, and latent code vector arithmetic. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.

BibTeX

@conference{Park-2020-125666,
author = {Taesung Park and Jun-Yan Zhu and Oliver Wang and Jingwan Lu and Eli Shechtman and Alexei A. Efros and Richard Zhang},
title = {Swapping Autoencoder for Deep Image Manipulation},
booktitle = {Proceedings of (NeurIPS) Neural Information Processing Systems},
year = {2020},
month = {December},
}