Space/Frequency Shape Inference and Segmentation of Textured 3D Surfaces

Abstract

Image texture is useful for segmentation and for computing surface orientations of uniformly textured objects. If texture is ignored, it can cause failure for stereo and gray-scale segmentation algorithms. In the past, mathematical representations of image texture have been applied to only specific texture problems, and no consideration has been given to the models' generality across different computer vision tasks and different image phenomea. We advocate the space/frequency representation, which shows the local spatial frequency content of every point in the image. From several different methods of computing the representation, we pick the spectrogram. The spectrogram elucidates many disparate image phenomena including texture boundaries, texture in perspective, aliasing, zoom, and blur. Many past shape-from-texture algorithms require potentially unreliable feature detections and "magic numbers" (arbitrary parameters), and none of them were developed in the context of a more general texture-understanding system. Toward this end, we show that the spatial frequency shifts caused by perspective can be approximated by an affine transformation which is a function of the textured surface's surface normal. We use this relationship in three different shape-from-texture algorithms. Two of them require no feature-finding and work on the raw spectrogram, giving a high-level scene parameter directly from low-level image data. The first algorithm includes an analytical sensitivity analysis. The third algorithm works with just the peak frequencies and gives a fast way of computing local surface normals from periodic texture. The algorithms need only a few magic numbers. On real textures, the average error in computed surface normal is only about four degrees. We use the third algorithm to solve a long-standing problem in image texture analysis: segmenting images of textured, 3D surfaces. Past work in texture segmentation and shape-from-texture is based on assumptions that make this problem impossible to solve. 3D perspective effects warp the otherwise uniform textures so segmentation fails. We develop a region-growing algorithm that accounts for the surface normals by unwarping the frequency distribution. It uses a minimum description length merge criterion. Our algorithm successfully segments images of real texture. We conclude by showing how the space/frequency representation has the potential for unifying several different computer vision algorithms.