Virtualized Reality: Digitizing a 3D Time-Varying Event As Is and in Real Time

Abstract

I will present the CMU Virtualized Reality project. Digital imaging of two-dimensional pictures is common today. Capturing an entire 3D scene or even a time-varying event into a computer as a 3D form, however, is very difficult and rarely done. Imagine a few players playing basketball on a court. Can we digitize the whole scene into a computer as a "3D event", not as a collection of pictures, but as its three-dimensional, time-varying, and volumetric/surface representation? If we could do so, we can use the representation for various purposes. For example, we can think of a "soft" camera - creating images from any arbitrary viewpoints and angles at which there were not cameras originally. With a soft camera, one can see the basketball game from any view point independent of physical limitations or other viewers' interest: from inside of the court, from the referee's point of view, or even from the ball's eye point of view. Image rendering, however, is not the only application. We can archive, manipulate, combine, and alter real events - a whole new notion of "event archiving and manipulation" or "Virtualized Reality".

Since 1993, we have been developing Virtualized Reality technologies with the 3D Room - a fully digital room that can capture events occurring in it by many (at this moment 50) video cameras. I will describe the theory, facility, computation, and results of the project.