PhD Thesis Defense
Carnegie Mellon University
Learning and Inference in Factor Graphs with Applications to Tactile Perception
Abstract: Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in hand [...]
Carnegie Mellon University
Unified Simulation, Perception, and Generation of Human Behavior
Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and [...]
Carnegie Mellon University
Search Algorithms and Search Spaces for Neural Architecture Search
Abstract: Neural architecture search (NAS) is recently proposed to automate the process of designing network architectures. Instead of manually designing network architectures, NAS automatically finds the optimal architecture in a data-driven way. Despite its impressive progress, NAS is still far from being widely adopted as a common paradigm for architecture design in practice. This thesis [...]
Carnegie Mellon University
Self-Improving 3D Scene Representations
Abstract: Most computer vision models in deployment today are not continually learning. Instead, they are in a “test” mode, where they will behave the same way perpetually, until they are replaced by newer models. This is a problem, because it means the models may perform poorly as soon as their “test” environment diverges from their [...]
Carnegie Mellon University
Direct-drive Hands: Making Robot Hands Transparent and Reactive to Contacts
Abstract: Industrial manipulators and end-effectors are a vital driver of the automation revolution. These robot hands, designed to reject disturbances with stiffness and strength, are inferior to their human counterparts. Human hands are dexterous and nimble effectors capable of a variety of interactions with the environment. Through this thesis we wish to answer a question: [...]
Carnegie Mellon University
Resource-Constrained Learning and Inference for Visual Perception
Abstract: We have witnessed rapid advancement across major computer vision benchmarks over the past years. However, the top solutions' hidden computation cost prevents them from being practically deployable. For example, training large models until convergence may be prohibitively expensive in practice, and autonomous driving or augmented reality may require a reaction time that rivals that [...]
Carnegie Mellon University
Physical Interaction and Manipulation of the Environment using Aerial Robots
Abstract: The physical interaction of aerial robots with their environment has countless potential applications and is an emerging area with many open challenges. Fully-actuated multirotors have been introduced to tackle some of these challenges. They provide complete control over position and orientation and eliminate the need for attaching a multi-DoF manipulation arm to the robot. [...]
Carnegie Mellon University
Visual Representation and Recognition without Human Supervision
Abstract: The advent of deep learning based artificial perception models has revolutionized the field of computer vision. These methods take advantage of the ever growing computational capacity of machines and the abundance of human-annotated data to build supervised learners for a wide-range of visual tasks. However, the reliance on human-annotated is also a bottleneck for [...]
Carnegie Mellon University
Learning Multi-Modal Navigation in Unstructured Environments
Abstract: A robot that operates efficiently in a team with humans in an unstructured outdoor environment must translate commands into actions from a modality intuitive to its operator. The robot must be able to perceive the world as humans do so that the actions taken by the robot reflect the nuances of natural language and [...]
Carnegie Mellon University
Towards Modular and Differentiable Autonomous Driving
Abstract: The classical "modular and cascaded" autonomy stack (object detection, tracking, trajectory prediction, then planning and control) has been widely used for interactive autonomous systems such as self-driving cars due to its interpretability and fast development cycle. In this thesis, we advocate the use of such a modular stack but improve its accuracy and robustness [...]
Carnegie Mellon University
Control Input and Natural Gaze for Goal Prediction in Shared Control
Abstract: Teleoperated systems are used widely in deployed robots today, for such tasks as space exploration, disaster recovery, or assisted manipulation. However, teleoperated systems are difficult to control, especially when performing high-dimensional, contact-rich tasks like manipulation. One approach to ease teleoperated manipulation is shared control; this strategy combines the user's direct control input with an [...]
Carnegie Mellon University
Liquid Metal Actuators
Abstract: This thesis contributes to the field of soft actuators by introducing a generalized framework of actuators from liquid metals. The evolution of robotic actuators has enabled robots to achieve a diversity of motions. Like natural muscles, which converts chemical energy into mechanical work in response to electrical stimuli from the nervous system, actuators are [...]
Carnegie Mellon University
Learning Structured World Model for Deformable Object Manipulation
Abstract: Manipulation of deformable objects challenges common assumptions in robotic manipulation, such as low-dimension state representation, known dynamics, and minimal occlusion. Deformable objects have high intrinsic state representation, complex dynamics with high degrees of freedom, and severe self-occlusion. These properties make them difficult for state estimation and planning. In this thesis, we introduce benchmarks and [...]