Image Synthesis with Appearance Decomposition

Abstract

Our visual world is compositional and its appearance can be decomposed into various components. Leveraging these components can be beneficial for challenging image synthesis tasks. To this end, this thesis focuses on studying how appearance decomposition can improve image synthesis methods using two examples.

(1) Structural decomposition: we introduce a periodicity-aware single image framework to synthesize a scene of near-periodic patterns (NPP). In particular, the appearance of an NPP scene is decomposed into motifs and their corresponding periodicities (i.e., arrangement), which are injected into the proposed framework as a prior to synthesize the NPP scene. The proposed method can interpolate and extrapolate NPP images, in-paint large and arbitrarily shaped regions, recover blurry regions when images are remapped, segment periodic and non-periodic regions, in planar and multi-planar scenes.

(2) Intrinsic decomposition: we propose a novel approach to decompose a single panorama of an empty indoor environment into four appearance components: specular, direct sunlight, diffuse and diffuse ambient without direct sunlight. This appearance decomposition enables multiple image synthesis applications including sun direction estimation, virtual furniture insertion, floor material replacement, and sun direction change.

We conduct extensive experiments to demonstrate the effectiveness of both methods.