Home/Takeo Kanade

Takeo Kanade

U.A. and Helen Whitaker
University Professor

Email: kanade@andrew.cmu.edu
Phone: (412) 268-3016
Personal Homepage
Administrative Assistant: Yukiko Kano

My research interests are in the areas of computer vision, visual and multi-media technology, and robotics. Common themes that my students and I emphasize in performing research are the formulation of sound theories which use the physical, geometrical, and semantic properties involved in perceptual and control processes in order to create intelligent machines, and the demonstration of the working systems based on these theories.

My current projects include basic research and system development in computer vision (motion, stereo and object recognition), recognition of facial expressions, virtual(ized) reality, content-based video and image retrieval, VLSI-based computational sensors, medical robotics, and an autonomous helicopter.

Computer vision

Within the Image Understanding (IU) project, my students and I are conducting basic research in interpretation and sensing for computer vision. My major thrust is the “science of computer vision.” Traditionally, many computer vision algorithms were derived heuristically either by introspection or biological analogy. In contrast, my approach to vision is to transform the physical, geometrical, optical and statistical processes, which underlie vision, into mathematical and computational models. This approach results in algorithms that are far more powerful and revealing than traditional ad hoc methods based solely on heuristic knowledge. With this approach we have developed a new class of algorithms for color, stereo, motion, and texture.

The two most successful examples of this approach are the factorization method and the multi-baseline stereo method. The factorization method is for the robust recovering of shape and motion from an image sequence. Based on this theory we have been developing a system for “modeling by video taping”; a user takes a video tape of a scene or an object by either moving a camera or moving the object, and then from the video a three-dimensional model of the scene or the object is created. The multi-baseline stereo method, the second example, is a new stereo theory that uses multi-image fusion for creating a dense depth map of a natural scene. Based on this theory, a video-rate stereo machine has been developed, which can produce a 200×200 depth image at 30 frames/sec, aligned with an intensity image; in other words, a real 3D camera!!

Currently, we are working on a rapidly trainable object recognition method, a system for modeling-by-video-taping, and a multi-camera 3D object copying/reconstruction method.

Visual media technology for human-computer interaction

A combination of computer vision and computer graphics technology presents an opportunity for a new exciting visual media. We have been developing a new visual medium, named “virtualized reality.” In the existing visual medium, the view of the scene is determined at the transcription time, independent of the viewer. In contrast, the virtualized reality delays the selection of the viewing angle till view time, using techniques from computer vision and computer graphics. The visual event is captured using many cameras that cover the action from all sides. The 3D structure of the event, aligned with the pixels of the image, is computed for a few selected directions using the multi-baseline stereo technique. Triangulation and texture mapping enable the placement of a soft-camera to reconstruct the event from any new viewpoint. The viewer, wearing a stereo-viewing system, can freely move about in the world and observe it from a viewpoint chosen dynamically at view time. We have built a 3D Virtualized Studio using a hemispherical dome, 5 meters in diameter, currently with 51 cameras attached at its nodes.

There are many applications of virtualized reality. Virtualized reality starts with a real world, rather than creating an artificial model of it. So, training can become safer, more real and more effective. A surgery, recorded in a virtualized reality studio, could be revisited by medical students repeatedly, viewing it from positions of their choice. Or, an entirely new generation of entertainment media can be developed – “Let’s watch NBA in the court”: basketball enthusiasts could watch a game from inside the court, from a referee’s point of view, or even from the “ball’s eye” point of view.

A Virtualized Reality application, CBS’s Eye Vision, was demonstrated during SuperBowl XXXV.

Also, I am interested in and currently working on vision techniques for recognizing facial expression, gaze, and hand-finger gestures. Such techniques will provide natural non-intrusive means for human-computer interface by replacing current clumsy mechanical devices, such as datagloves.

Informedia Project

With the growth and popularity of multimedia computing technologies, video is gaining importance and broadening its uses in libraries. Digital video libraries open up great potentials for education, training and entertainment; but to achieve this potential, the information embedded within the digital video library must be easy to locate, manage and use. Searches within a large data set or lengthy video would take a user through vast amounts of material irrelevant to the search topic. The typical database, which searches by keywords (e.g. title) – where images are only referenced and not directly searched for – is not appropriate or useful for the digital video library, since it does not provide the user a way to know the contents of the image, short of viewing it. New techniques are needed to organize these vast video collections so that users can effectively retrieve and browse their holdings based on their content. The Informedia Digital Video Library, funded by NSF, ARPA, and NASA, is developing intelligent, automatic mechanisms to populate the video library and allow for a full-content knowledge-based search, retrieval and presentation of video. The distinguishing feature of Informedia’s approach is the integrated application of speech, language and image understanding technologies.

Computational Sensor

While significant advancements have been made over the last 30 years of computer vision research, the consistent paradigm has been that a “camera” sees the world and a computer “algorithm” recognizes the object. I have been undertaking a project with Dr. Vladimir Brajovic that breaks away from this traditional paradigm by integrating sensing and processing into a single VLSI chip a computational sensor. The first successful example was an ultra fast range sensor which can produce approximately 1000 frames of range images per second an improvement of two orders of magnitude over the state of the art. A few new sensors are being developed including a sorting sensor chip, a 2D salient feature detector (2D winner-take-all circuits), and others.

Medical Robotics and Computer Assisted Surgery

The emerging field of Medical Robotics and Computer Assisted Surgery strives to develop smart tools to perform medical procedures better than either a physician or machine could alone. Robotic and computer-based systems are now being applied in specialties that range from neurosurgery and laparoscopy to opthalmology and family practice. Robots are able to perform precise and repeatable tasks that would be impossible for any human. The physician provides these systems with the decision making skills and adaptable dexterity that are well beyond current technology. The potential combination of robots and physicians has created a new worldwide interest in the area of medical robotics.

We have developed a new computer assisted surgical systems for total hip replacement. The work is based on biomechanics-based surgical simulations and less invasive and more accurate vision-based techniques for determining the position of the patient anatomy during a robot surgery. The developed system, HipNav, has been already test -used in clinical setting.

Vision-based Autonomous Helicopter

An unmanned helicopter can take maximum advantage of the high maneuverability of helicopters in dangerous support tasks, such as search and rescue, and fire fighting, since it does not place a human pilot in danger. The CMU Vision-Guided Helicopter Project (with Dr. Omead Amidi) has been developing the basic technologies for an unmanned autonomous helicopter including robust control methods, vision algorithms for real-time object detection and tracking, integration of GPS, motion sensors, vision output for robust positioning, and high-speed real-time hardware. After having tested various control algorithms and real-time vision algorithms using an electric helicopter on an indoor teststand, we have developed a computer controlled helicopter (4 m long), which carries two CCD cameras, GPS, gyros and accelerometers together with a multiprocessor computing system. Autonomous outdoor free flight has been demonstrated with such capabilities as following prescribed trajectory, detecting an object, and tracking or picking it from the air.


Takeo Kanade, the U.A. and Helen Whitaker University Professor of Robotics and Computer Science at Carnegie Mellon University, received the prestigious 2016 Kyoto Prize for Advanced Technology, Nov. 10 in a ceremony in Kyoto, Japan.

The international award is presented by the Inamori Foundation to individuals such as Kanade who have contributed significantly to the scientific, cultural and spiritual betterment of humankind. Kanade’s prize recognizes his pioneering contributions to computer vision and robotics. The prize includes a gold medal and a cash award of 50 million yen (about $480,000). (read more)


Learning Scene-Specific Pedestrian Detectors without Real Data
Hironori Hattori, Yasodekshna Vishnu Naresh Boddeti, Kris M. Kitani and Takeo Kanade

CVPR June, 2015

An Implementation of Camera Geometry Correction Capability in a Video-Rate Stereo Machine
Hiroshi Kano, Shigeru Kimura, Masaya Tanaka and Takeo Kanade

Journal of the Robotics Society of Japan January, 1998

When is the Shape of a Scene Unique Given its Light-Field: A Fundamental Theorem of 3D Vision?
Simon Baker, Terence Sim and Takeo Kanade

IEEE Transaction on Pattern Analysis and Machine Intelligence January, 2003

A Robust Subspace Approach to Layer Extraction
Qifa Ke and Takeo Kanade

IEEE Workshop on Motion and Video Computing (Motion'2002), Orlando, Florida, Dec. 2002. December, 2002

Limits on Super-Resolution and How to Break Them
Simon Baker and Takeo Kanade

IEEE Transactions on Pattern Analysis and Machine Intelligence September, 2002

Detection, tracking, and classification of subtle changes in facial expression
Jenn-Jier James Lien, Takeo Kanade, Jeffrey Cohn and C. Li

Journal of Robotics and Autonomous Systems January, 2000

Spatio-Temporal View Interpolation
Sundar Vedula, Simon Baker and Takeo Kanade

Proceedings of the 13th ACM Eurographics Workshop on Rendering June, 2002

Object Detection Using the Statistics of Parts
Henry Schneiderman and Takeo Kanade

International Journal of Computer Vision January, 2002

Gauge Fixing for Accurate 3D Estimation
Daniel D. Morris, Kenichi Kanatani and Takeo Kanade

Computer Vision and Pattern Recognition December, 2001

Bayesian Color Constancy for Outdoor Object Recognition
Yanghai Tsin, Robert Collins, Visvanathan Ramesh and Takeo Kanade

IEEE Conference on Computer Vision and Pattern Recognition (CVPR'01) December, 2001

Classification-Driven Pathological Neuroimage Retrieval Using Statistical Asymmetry Measures
Yanxi Liu, Frank Dellaert, William E. Rothfus andrew Moore, Jeff Schneider and Takeo Kanade

Proceedings of the 2001 Medical Imaging Computing and Computer Assisted Intervention Conference (MICCAI '01) October, 2001

Algorithms for cooperative multisensor surveillance
Robert Collins, Alan Lipton, Hironobu Fujiyoshi and Takeo Kanade

Proceedings of the IEEE October, 2001

Multiple motion scene reconstruction from uncalibrated views
Mei Han and Takeo Kanade

Proceedings of the Eighth IEEE International Conference on Computer Vision (ICCV '01) July, 2001

Precision 3-D Modeling for Autonomous Helicopter Flight
James Ryan Miller , Omead Amidi, Chuck Thorpe and Takeo Kanade

Proceedings of International Symposium of Robotics Research (ISRR) January, 1999

Creating 3D Models with Uncalibrated Cameras
Mei Han and Takeo Kanade

proceeding of IEEE Computer Society Workshop on the Application of Computer Vision (WACV2000) December, 2000

Reconstructing specimens using DIC microscope images
Farhana Kagalwala and Takeo Kanade

Proceedings of the 2000 IEEE International Symposium on Bio-Informatics and Biomedical Engineering November, 2000

Image-Consistent Surface Triangulation
Daniel D. Morris and Takeo Kanade

Computer Vision and Pattern Recognition (CVPR 2000) June, 2000

Shape and Motion Carving in 6D
Sundar Vedula, Simon Baker, Steven Seitz and Takeo Kanade

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '00) June, 2000

Appearance-Based Virtual-View Generation for Fly Through in a Real Dynamic Scene
Shigeyuki Baba, Hideo Saito, Sundar Vedula, Kong Man Cheung and Takeo Kanade

VisSym '00 (Joint Eurographics - IEEE TCVG Symposium on Visualization) May, 2000

Comprehensive Database for Facial Expression Analysis
Takeo Kanade, Jeffrey Cohn and Ying-Li Tian

Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (FG'00) March, 2000

Hallucinating Faces
Simon Baker and Takeo Kanade

Fourth International Conference on Automatic Face and Gesture Recognition March, 2000

System Identification of a Model-Scale Helicopter
Bernard Mettler, Mark Tischler and Takeo Kanade

CMU-RI-TR-00-03 January, 2000

Recognizing Action Units for Facial Expression Analysis
Ying-Li Tian, Takeo Kanade and Jeffrey Cohn

CMU-RI-TR-99-40 December, 1999

Virtualized Reality: Digitizing a 3D Time-Varying Event As Is and in Real Time
Takeo Kanade, Peter Rander, Sundar Vedula and Hideo Saito

Mixed Reality, Merging Real and Virtual Worlds January, 1999

Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding
Jeffrey Cohn, Adena Zlochower, Jenn-Jier James Lien and Takeo Kanade

Psychophysiology January, 1999

A Cooperative Algorithm for Stereo Matching and Occlusion Detection
Charles Zitnick and Takeo Kanade

CMU-RI-TR-99-35 October, 1999

A Tracker for Broken and Closely Spaced Lines
N. Chiba and Takeo Kanade

Proceedings of the 1996 International Society for Photogrammetry and Remote Sensing Conference (ISPRS '98) January, 1998

Name-It: Association of Face and Name Video
Shin'ichi Satoh and Takeo Kanade

CMU-CS-96-205 January, 1996

An image overlay system for medical data visualization
Mike Blackwell, Constantinos Nikou, Anthony M. Digioia and Takeo Kanade

Proceedings of the 1998 Medical Imaging Computing and Computer Assisted Intervention Conference (MICCAI '98) January, 1998

System Identification of Small-Size Unmanned Helicopter Dynamics
Bernard Mettler, Mark B. Tischler and Takeo Kanade

Presented at the American Helicopter Society 55th Forum May, 1999

A VLSI Sorting Image Sensor: Global Massively Parallel Intensity-to-Time Processing for Low-Latency, Adaptive Vision
Vladimir Brajovic and Takeo Kanade

IEEE Transactions on Robotics and Automation February, 1999

The 3D Room: Digitizing Time-Varying 3D Events by Synchronized Multiple Video Streams
Takeo Kanade, Hideo Saito and Sundar Vedula

CMU-RI-TR-98-34 December, 1998

Temporal Photoreception for Adaptive Dynamic Range Image Sensing and Encoding
Vladimir Brajovic, Ryohei Miyagawa and Takeo Kanade

Neural Networks (1998 Special Issue) October, 1998

Computational Sensor for Visual Tracking with Attention
Vladimir Brajovic and Takeo Kanade

IEEE Journal of Solid State Circuits August, 1998

Autonomous Helicopter Research at Carnegie Mellon Robotics Institute
Omead Amidi, Takeo Kanade and James Ryan Miller

Proceedings of Heli Japan `98 April, 1998

Video OCR: Indexing Digital News Libraries by Recognition of Superimposed Caption
Toshio Sato, Takeo Kanade, Ellen Hughes, Michael Smith and Shin-ichi Satoh

ACM Multimedia Systems Special Issue on Video Libraries February, 1998

Constructing Virtual Worlds Using Dense Stereo
P J. Narayanan, Peter Rander and Takeo Kanade

Proceedings of the Sixth IEEE International Conference on ComputerVision (ICCV'98) January, 1998

Neural Network-Based Face Detection
Henry Rowley, Shumeet Baluja and Takeo Kanade

IEEE Transactions on Pattern Analysis and Machine Intelligence January, 1998

A Sequential Factorization Method for Recovering Shape and Motion From Image Streams
Toshihiko Morita and Takeo Kanade

IEEE Transactions on Pattern and Analysis and Machine Intelligence August, 1997

Intelligent Access to Digital Video: The Informedia Project
Howard Wactlar , Takeo Kanade, Michael Smith and Scott Stevens

IEEE Computer May, 1996

Development of a Video-Rate Stereo Machine
Takeo Kanade, H. Kato, S. Kimura, A. Yoshida and K. Oda

Proc. of International Robotics and Systems Conference (IROS '95), Human Robot Interaction and Cooperative Robots August, 1995

Development of a Video-Rate Stereo Machine
Takeo Kanade, H. Kato, S. Kimura, A. Yoshida and K. Oda

Proc. of International Robotics and Systems Conference (IROS '95), Human Robot Interaction and Cooperative Robots August, 1995

Video Skimming for Quick Browsing based on Audio and Image Characterization
Michael Smith and Takeo Kanade

CMU-CS-95-186 July, 1995

A Multi-body Factorization Method for Motion Analysis
J. Costeira and Takeo Kanade

Proceedings of the Fifth International Conference on Computer Vision (ICCV '95) June, 1995

A Paraperspective Factorization Method for Shape and Motion Recovery
Conrad Poelman and Takeo Kanade

CMU-CS-93-219 December, 1993

First Results in Robot Road-Following
R. Wallace, Anthony (Tony) Stentz , Chuck Thorpe, Hans Moravec, William (Red) L. Whittaker and Takeo Kanade

Proceedings of the International Joint Conference on Artificial Intelligence January, 1985

Picture Processing System by Computer Complex and Recognition of Human Faces
Takeo Kanade

doctoral dissertation, Kyoto University November, 1973

Mailing Address:

Carnegie Mellon University
Robotics Institute
5000 Forbes Avenue
Pittsburgh, PA 15213