Volumetric Correspondence Networks for Stereo Matching and Optical Flow

Abstract

Many classic tasks in vision, such as the estimation of optical flow or stereo disparities, can be cast as dense correspondence matching. Well-known techniques for doing so make use of a cost volume, which is typically a 3D/4D tensor of match costs between all pixels in a 2D image and their potential matches in a 1D/2D search window. However, it typically requires significant amounts of memory and compute, leading to various limitations in practice. In this thesis, we investigate efficient and effective ways of incorporating cost volume processing into deep neural networks for correspondence tasks. In particular, we show that 1) for stereo matching on high-resolution images, 3D cost volumes can be efficiently filtered in a coarse-to-fine manner, and 2) for optical flow estimation, "true" 4D volumetric processing can be effectively utilized to improve model's accuracy as well as generalization ability. As a result, our stereo algorithm achieves state-of-the-art performance on the high-resolution Middlebury benchmark, and our optical flow algorithm is the leading entry among two-frames methods on KITTI as well as MPI Sintel.