A Robust Subspace Approach to Extracting Layers from Image Sequences

Abstract

A layer is a 2D sub-image inside which pixels share common appar- ent motion of some 3D scene plane. Representing videos with such layers has many important applications, such as video compression, 3D scene and motion analysis, object detection and tracking, and vehicle naviga- tion. Extracting layers from videos involves solving three subproblems: 1) segment the image into sub-regions (layers); 2) estimate the 2D mo- tion of each layer; and 3) determine the number of layers. These three subproblems are highly intertwined, making the layer extraction problem very challenging. Existing approaches to layer extraction are limited by 1) requiring good initial segmentation, 2) strong assumptions about the scene, 3) unable to fully and simultaneously utilize the spatial and tem- poral constraints in video, and 4) unstable clustering in high dimensional space. This thesis presents a subspace approach to layer extraction which does not have the above limitations. We ?st show that the homographies induced by the planar patches in the scene form a linear subspace whose dimension is as low as two or three in many applications. We then for- mulate the layer extraction problem as clustering in such low dimensional subspace. Each layer in the input images will form a well-de?ed clus- ter in the subspace, and a simple mean shift based clustering algorithm can reliably identify the clusters thus the layers. A proof is presented to show that the subspace approach is guaranteed to increase signi?antly the layer discriminability, due to its ability to simultaneously utilize spa- tial and temporal constraints in the video. We present the detailed robust algorithm for layer extraction using subspace, as well as experimental re- sults on a variety of real image sequences.