Reconstruction, Registration, and Modeling of Deformable Object Shapes

Abstract

Biological or biomedical objects, such as expressive human faces and growing brain tumors, and dynamic scenes, such as cars running on the roads, generally vary their shapes as linear combinations of a number of shape bases. With the expeditious development of computer and imaging technologies, the problems of reconstruction, registration, and modeling of such deformable shapes from image measurements has shown enormous importance for applications such as biomedical image interpretation, human computer interaction, and robot navigation. Since the image measurements are generated by coupling two factors: non-rigid deformations and rigid similarity transformations between the shapes and the measurement systems, the essence of the three problems is to factorize the shape measurements and compute the deformable shapes (reconstruction), the rigid transformations (registration), and the shape bases (modeling). This thesis presents novel factorization algorithms respectively for reconstruction, registration, and modeling of deformable shapes. We also demonstrate an application of the proposed algorithms for modeling and fitting of expressive human faces. Firstly, we study the problem of reconstructing 3D deformable shapes from 2D images, assuming the weak-perspective camera model and non-degenerate cases. The previous methods usually enforce the constraints on the orthonormality of the rigid rotation transformations (rotation constraints) alone to solve this problem. The thesis quantitatively shows that enforcing the rotation constraints alone is inherently insufficient and yields ambiguous and invalid solutions. We point out that the ambiguity stems from the non-uniqueness of the shape bases, and introduce the basis constraints that resolves the ambiguity by implicitly specifying a unique set of bases. The thesis proves that, enforcing the basis constraints, together with the rotation constraints, results in a linear closed-form solution to factorizing the image measurements and simultaneously reconstructing the rigid transformations, the deformable shapes, and the underlying shape bases. We also develop methods for reconstruction of degenerate or planar deformations that are described by rank-2 or rank-1 shape bases. Specifically, we quantitatively show that enforcing the basis and rotation constraints still achieves a linear closed-form solution when all the degenerate bases are of rank 1, but results in an ambiguous solution space when rank-2 bases are involved. We then develop an alternating linear optimization method for reconstruction of rank-2 deformations. Secondly, we develop a 2-step factorization algorithm for reconstruction of 3D deformable shapes from uncalibrated images under the full perspective camera model. In the first step, we recover the projective depths using the sub-space constraints embedded in the deformable shape measurements. In the second step, we first scale the image measurements by the reconstructed projective depths. The scaled mea-surements are then factorized by an extension of the linear closed-form algorithm for weak-perspective reconstruction. This factorization step simultaneously recovers the rigid transformations, the deformable shapes, the underlying shape bases, and the varying camera parameters such as focal lengths. Thirdly, we address the problem of registering deformable shapes measured in respective local systems into a common coordinate system. In the literature, Generalized Procrustes Analysis has been widely used to resolve this problem. Our analysis shows that, because Generalized Procrustes Analysis treats the deformations as Gaussian noise and does not take into account the shape deformations during the registration process, it yields biased registrations when the deformations are asymmetric and significant. We then develop a factorization-based registration method, similar to the weak-perspective reconstruction algorithm. It again enforces the basis constraints together with the rotation constraints and achieves a linear closed-form solution to simultaneously registering the deformable shapes and computing the underlying shape bases. Finally, we apply the proposed algorithms to extracting the 3D Morphable Model of expressive human faces from monocular image sequences. Combining the benefits of the efficient fitting of the 2D Active Appearance Model and the explicit 3D parameterization of the 3D Morphable Model, we present a novel face model that describes the variations of both 2D and 3D face shapes and facial appearances. We then develop a real-time algorithm ( 60fps) that recovers the 2D and 3D face shapes, the 3D face poses, and the facial appearances by fitting the new model to images.