PhD Thesis Defense
Carnegie Mellon University
Provably Optimal Design of a Brain-Computer Interface
Abstract: Brain-computer interfaces are in the process of moving from the laboratory to the clinic. These devices act by reading neural activity and using it to directly control a device, such as a cursor on a computer screen. Over the past two decades, much attention has been devoted to the decoding problem: how should recorded [...]
Carnegie Mellon University
Extensions of the Principal Fiber Bundle Model for Locomoting Robots
Abstract: Our goal is to establish a rigorous formulation for modeling the locomotion of a broad class of robotic systems. Recent research has identifi ed a number of systems with the structure of a principal fiber bundle. This framework has led to a number of tools for analysis and motion planning applicable to various robotic [...]
Carnegie Mellon University
Safe Data Gathering in Physical Spaces
Abstract: Reliable and efficient acquisition of data from physical spaces has widespread applications in industry, policy, defense, and humanitarian work. Unmanned Aerial Vehicles (UAVs) are an excellent choice for data gathering applications, due to their capability of gaining information at multiple scales. A robust data gathering system needs to reason about multi-resolution nature of information [...]
Carnegie Mellon University
Learning to learn from simulation: Using simulations to learn faster on robots
Abstract: Learning for control is capable of acquiring controllers in novel task scenarios, paving the path to autonomous robots. However, typical learning approaches can be prohibitively expensive in terms of robot experiments, and policies learned in simulation do not transfer directly due to modelling inaccuracies. This encourages learning information from simulation that has a higher [...]
Carnegie Mellon University
Learning with Clusters
Abstract: Clustering, the problem of grouping similar data, has been extensively studied since at least the 1950's. As machine learning becomes more prominent, clustering has evolved from primarily a data analysis tool into an integrated component of complex robotic and machine learning systems, including those involving dimensionality reduction, anomaly detection, network analysis, image segmentation and [...]
Carnegie Mellon University
Sensing, Measuring, and Modeling Social Signals in Nonverbal Communication
Abstract: Humans convey their thoughts, emotions, and intentions through a concert of social displays: voice, facial expressions, hand gestures, and body posture, collectively referred to as social signals. Despite advances in machine perception, machines are unable to discern the subtle and momentary nuances that carry so much of the information and context of human communication. [...]
Carnegie Mellon University
Optimal control of compliant bipedal gaits and their implementation on robot hardware
Abstract: Legged animals exhibit diverse locomotion patterns known as gaits, which are capable of robustly traversing terrains of variable grade, roughness, and compliance. Despite the success of legs in nature, wheeled solutions still dominate the field of robotics. State-of-the-art humanoid robots have not yet demonstrated locomotion behaviors that are as robust or varied as their [...]
Carnegie Mellon University
Design and Evaluation of Robust Control Methods for Robotic Transfemoral Prostheses
Abstract: Amputees face a number of gait deficits due to a lack of control and power from their mechanically-passive prostheses. Of crucial importance among these deficits are those related to balance, as falls and a fear of falling can cause an avoidance of activity that leads to further debilitation. In this thesis, we investigate the [...]
Carnegie Mellon University
Intra-Robot Replanning and Learning for Multi-Robot Teams in Complex Dynamic Domains
Abstract: In complex dynamic multi-robot domains, there is a set of individual robots that must coordinate together through a centralized planner that inevitably makes assumptions based on a model of the environment and the actions of the individual. Eventually, the individuals may encounter failures, because the centralized planner’s models of the states and actions are [...]
Carnegie Mellon University
Light Sheet Depth Imaging
Abstract: Once confined to industrial manufacturing facilities and research labs, robots are increasingly entering everyday life. As specialized robots are developed for tasks such as autonomous driving, package delivery, and aerial videography, there is a growing need for affordable depth sensing technology. Robots use sensors like scanning LIDAR, depth cameras, and passive stereo cameras to [...]
Carnegie Mellon University
Towards Generalization and Efficiency in Reinforcement Learning
Abstract: In classic supervised machine learning, a learning agent behaves as a passive observer: it receives examples from some external environment which it has no control over and then makes predictions. Reinforcement Learning (RL), on the other hand, is fundamentally interactive: an autonomous agent must learn how to behave in an unknown and possibly hostile [...]
Carnegie Mellon University
Planning under Uncertainty with Multiple Heuristics
Abstract: Many robotic tasks, such as mobile manipulation, often require interaction with unstructured environments and are subject to imperfect sensing and actuation. This brings substantial uncertainty into the problems. Reasoning under this uncertainty can provide higher level of robustness but is computationally significantly more challenging. More specifically, sequential decision making under motion and sensing uncertainty [...]
Carnegie Mellon University
Analysis of Spatio-Temporally Varying Features in Optical Coherence Tomographic (OCT) and Ultrasound (US) Image Sequences
Abstract: Optical Coherence Tomography (OCT) and Ultrasound (US) are non-ionizing and non-invasive imaging modalities that are clinically used to visualize anatomical structures in the body. OCT has been widely adopted in clinical ophthalmology due to its micron-scale resolution to visualize in-vivo structures of the eye. Ultra-High Frequency Ultrasound (UHFUS) captures images of tissue at a [...]
Carnegie Mellon University
Spatiotemporal Understanding of People Using Scenes, Objects, and Poses
Abstract: Humans are arguably one of the most important entities that AI systems would need to understand to be useful and ubiquitous. From autonomous cars observing pedestrians to assistive robots helping the elderly, a large part of this understanding is focused on recognizing human actions, and potentially, their intentions. Humans themselves are quite good at [...]
Carnegie Mellon University
Deep Non-Rigid Structure from Motion
Abstract: Non-Rigid Structure from Motion (NRSfM) refers to the problem of reconstructing cameras and the 3D point cloud of a non-rigid object from a sequence of images with 2D correspondences. Current NRSfM algorithms are mainly limited within two perspectives: (i) the number of images, and (ii) the type of shape variability they can handle. These [...]
Carnegie Mellon University
Data Centric Robot Learning
Abstract: While robotics has made tremendous progress over the last few decades, most success stories are still limited to carefully engineered and precisely modeled environments. Getting these robots to work in the complex and diverse world that we live in has proven to be a difficult challenge. Interestingly, one of the most significant successes in [...]
Carnegie Mellon University
Exploiting Point Motion, Shape Deformation, and Semantic Priors for Dynamic 3D Reconstruction in the Wild
Abstract: With the advent of affordable and high-quality smartphone cameras, any significant events will be massively captured both actively and passively from multiple perspectives. This opens up exciting opportunities for low-cost high-end VFX effects and large scale media analytics. However, automatically organizing large scale visual data and creating a comprehensive 3D scene model is still [...]
Carnegie Mellon University
Learning and Reasoning with Visual Correspondence in Time
Abstract: There is a famous tale in computer vision: Once, a graduate student asked the famous computer vision scientist Takeo Kanade: "What are the three most important problems in computer vision?" Takeo replied: "Correspondence, correspondence, correspondence!" Indeed, even for the most commonly applied Convolutional Neural Networks (ConvNets), they are internally learning representations that lead to [...]
Carnegie Mellon University
Forecasting and Controlling Behavior by Learning from Visual Data
Abstract: Achieving a precise predictive understanding of the future is difficult, yet widely studied in the natural sciences. Significant research activity has been dedicated to building testable models of cause and effect. From a certain view, a perfect predictive model of the universe is the “holy grail”; the ultimate goal of science. If we had [...]
Carnegie Mellon University
Underwater Localization and Mapping with Imaging Sonar
Abstract: Acoustic imaging sonars have been used for a variety of tasks intended to increase the autonomous capabilities of underwater vehicles. Among the most critical tasks of any autonomous vehicle are localization and mapping, which are the focus of this work. The difficulties presented by the imaging sonar sensor have led many previous attempts at [...]
Carnegie Mellon University
Personalized and weakly supervised learning for Parkinson’s disease symptom detection
Abstract: Parkinson's Disease (PD) is a neurodegenerative disorder that affects approximately one million Americans. Medications exist to manage the symptoms, but doctors must periodically adjust dosage level and frequency as a patient's disease progresses. These adjustments are typically based on observations made during short clinic visits, which provide an incomplete picture of a patient's daily [...]
Carnegie Mellon University
Self-Supervised Learning on Mobile Robots Using Acoustics, Vibration, and Visual Models to Build Rich Semantic Terrain Maps
Abstract: Humans and robots would benefit from having rich semantic maps of the terrain in which they operate. Mobile robots equipped with sensors and perception software could build such maps as they navigate through a new environment. This information could then be used by humans or robots for better localization and path planning, as well [...]
Carnegie Mellon University
Combining Multiple Heuristics: Studies on Neighborhood-base Heuristics and Sampling-based Heuristics
Abstract: This thesis centers on the topic of how to automatically combine multiple heuristics. For most computationally challenging problems, there exist multiple heuristics, and it is generally the case that any such heuristic exploits only a limited number of aspects among all the possible problem characteristics that we can think of, and by definition, is [...]
Carnegie Mellon University
Multi-hypothesis iSAM2 for Ambiguity-aware Passive and Active SLAM
Archived video Abstract Simultaneous localization and mapping (SLAM) is the problem of estimating the state of a moving agent with sensors on it while simultaneously reconstructing a map of its surrounding environment, which has been a popular research field due to its wide applications. As many state-of-the-art SLAM algorithms can already achieve high accuracy in [...]
Carnegie Mellon University
Terrain Relative Navigation for Lunar Polar Roving: Exploiting Geometry, Shadows, and Planning
Archived Zoom Video Abstract Water ice at the lunar poles could be the most valuable resource beyond planet Earth. However, that value is not foregone, and can only be determined by rovers that evaluate the distributions of abundance, concentration, and characteristics of this ice. The near-term explorations will be solar and unlikely to endure night, [...]
Carnegie Mellon University
Resource-Constrained State Estimation with Multi-Modal Sensing
Zoom Link Accurate and reliable state estimation is essential for safe mobile robot operation in real-world environments because ego-motion estimates are required by many critical autonomy functions such as control, planning, and mapping. Computing accurate state estimates depends on the physical characteristics of the environment, the selection of suitable sensors to capture that information, and [...]
Carnegie Mellon University
Hybrid Soft Sensing in Robotic Systems
Zoom Link Abstract: The desire to operate robots in unstructured environments, side-by-side with humans, has created a demand for safe and robust sensing skins. Largely inspired by human skin, the ultimate goal of electronic skins is to measure diverse sensory information, conform to surfaces, and avoid interfering with the natural mechanics of the host or [...]
Carnegie Mellon University
Vision with Small Baselines
Zoom Link Abstract: 3D sensing with portable imaging systems is becoming more and more popular in computer vision applications such as autonomous driving, virtual reality, robotics manipulation and surveillance, due to the decreasing expense and size of RGB cameras. Despite the compactness and portability of the small baseline vision systems, it is well-known that the [...]
Carnegie Mellon University
Humans In Their Natural Habitat: Training AI to Understand People
Zoom Link Abstract: Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. First, we need to give [...]
Carnegie Mellon University
Automated Action Selection and Embodied Simulation for Socially Assistive Robots using Standardized Interactions
Zoom Link Abstract: Robots have the tremendous potential of assisting people in their lives, allowing them to achieve goals that they would not be able to achieve by themselves. In particular, socially assistive robots provide assistance primarily through social interaction, in healthcare, therapy, and education contexts. Despite their potential, current socially assistive robots still lack [...]
Carnegie Mellon University
Robot Deep Reinforcement Learning: Tensor State-Action Spaces and Auxiliary Task Learning with Multiple State Representations
Zoom Link Abstract: A long standing goal of robotics research is to create algorithms that can automatically learn complex control strategies from scratch. Part of the challenge of applying such algorithms to robots is the choice of representation. Reinforcement Learning (RL) algorithms have been successfully applied to many different robotic tasks such as the Ball-in-a-Cup [...]
Carnegie Mellon University
Online Inference of Joint Occupancy using Forward Sensor Models and Trajectory Posteriors for Deliberate Robot Navigation
Zoom Link Abstract: Robotic navigation algorithms for real-world robots require dense and accurate probabilistic volumetric representations of the environment in order to traverse efficiently. Sensor data in a Simultaneous Localisation And Mapping (SLAM) context, however, always has associated acquisition noise and pose uncertainty, and encoding this within the map representation while still maintaining computational tractability [...]
Carnegie Mellon University
Machine Learning Parallelism Could Be Adaptive, Composable and Automated
Zoom Link Abstract: In recent years, researchers in SysML have created algorithms and systems that parallelize ML training over multiple devices or computational nodes. As ML models become more structurally complex, many systems have struggled to provide all-round performance on a variety of models. Particularly, ML scale-up is usually underestimated in terms of the amount [...]
Carnegie Mellon University
Data-Driven Robotic Grasping in the Wild
Zoom Link Abstract: Humans can effortlessly grasp a wide variety of objects in diverse environments. On the other hand, robotic grasping has been extremely challenging in practice and is far from matching human dexterity. Despite recent progress in the community, most research is still largely focused on constrained environments like picking individual objects on a [...]
Carnegie Mellon University
Routing for Persistent Exploration in Dynamic Environments with Teams of Energy-Constrained Robots
Abstract: Disaster relief scenarios require rapid and persistent situational awareness to inform first-responders of safe and viable routes through a constantly shifting environment. Knowing what roads have become flooded or are suddenly obstructed by debris can significantly improve response time and ease the distribution of resources. In a sufficiently large environment, deploying and maintaining fixed [...]
Carnegie Mellon University
Coordinated online multi-robot planning
Abstract: Multi-robot applications frequently seek to employ human operators to direct robot actions online because fully automated planners struggle to encode human expertise or handle the extenuating circumstances that occur during real world operations. However, it is extremely challenging for a human to direct multi-robot teams, especially online, i.e., in real-time. From entertainment to defense, [...]
Sensor Planning for Large Numbers of Robots
Abstract: In the wake of a natural disaster, locating and extracting victims quickly is critical because mortality rises rapidly after the first forty-eight hours. In order to assist search and rescue teams and improve response times, teams of aerial robots equipped with sensors and cameras can engage in sensing tasks such as mapping buildings, assessing [...]
Carnegie Mellon University
Constraint-Based Coverage Path Planning: A Novel Approach to Achieving Energy-Efficient Coverage
Abstract: Despite substantial technological progress that has driven the proliferation of robots across various industries and aspects of our lives, the lack of a decisive breakthrough in energy storage capabilities has restrained this trend, particularly with respect to mobile robots designed for use in unstructured and unknown field environments. The fact that these domains are [...]
Carnegie Mellon University
Unsupervised Learning of the 4D Audio-Visual World from Sparse Unconstrained Real-World Samples
Abstract: We, humans, can easily observe, explore, and analyze the world we live in. We, however, struggle to share our observation, exploration, and analysis with others. This thesis introduce Computational Studio, computational machinery that can understand, explore, and create the four-dimensional audio-visual world. This allows: (1) humans to communicate with other humans without any loss [...]
Carnegie Mellon University
Expressive Real-time Intersection Scheduling: New Methods for Adaptive Traffic Signal Control
Abstract: Traffic congestion is a widespread problem throughout global metropolitan areas. In this thesis, we consider methods to optimize the performance of traffic signals to reduce congestion. We begin by presenting Expressive Real-time Intersection Scheduling (ERIS), a schedule-driven intersection control strategy that runs independently on each intersection in a traffic network. For each intersection, ERIS [...]
Carnegie Mellon University
Robust Manipulation with Active Compliance
Abstract: Human manipulation skills are filled with creative use of physical contacts to move the object about the hand and in the environment. However, it is difficult for robot manipulators to enjoy this dexterity since contacts may cause the manipulation task to fail by introducing huge forces or unexpected change of constraints, especially when modeling [...]
Carnegie Mellon University
Open-world Object Detection and Tracking
Abstract: Computer vision today excels at recognizing narrow slices of the real world: our models seem to accurately detect objects like cats, cars, or chairs in benchmark datasets. However, deploying models requires that they work in the open world, which includes arbitrary objects in diverse settings. Current methods struggle on both axes: they recognize only [...]
Carnegie Mellon University
Auto-generated Manipulation Primitives
Abstract: The central theme in robotic manipulation is that of the robot interacting with the world through physical contact. We tend to describe that physical contact using specific words that capture the nature of the contact and the action, such as grasp, roll, pivot, push, pull, tilt, close, open etc. We refer to these situation-specific [...]
Carnegie Mellon University
Learning 3D Registration and Reconstruction from the Visual World
Abstract: Humans learn to develop strong senses for 3D geometry by looking around in the visual world. Through pure visual perception, not only can we recover a mental 3D representation of what we are looking at, but meanwhile we can also recognize where we are looking at the scene from. Finding the 3D scene representation [...]
Carnegie Mellon University
Active Vision: Autonomous Aerial Cinematography with Learned Artistic Decision-Making
Abstract: Aerial cinematography is revolutionizing industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. Fundamentally, it is a tool with immense potential to improve human creativity, expressiveness, and sharing of experiences. However, safely piloting a drone while filming a moving target in the presence of obstacles is immensely taxing, often [...]
Carnegie Mellon University
Understanding and Mitigating Biases in Evaluation
Abstract: There are many problems in real life that involve collecting and aggregating evaluation from people, such as hiring, peer grading and conference peer review. In this thesis, we focus on three sources of biases that arise in such problems, and propose methods to mitigate them. First, we study human bias, that is, the bias [...]
Carnegie Mellon University
Towards Safe and Resilient Autonomy in Multi-Robot Systems
Abstract: Autonomous systems such as robotic systems are envisioned to co-exist with humans in our daily lives, from household service to large-scale warehouse logistics, agricultural monitoring, and smart city. Reliable interactions among robots and humans require provably correct guarantees about safety and performance when designing robot behaviors. While traditional approaches for safety and performance analysis [...]
Carnegie Mellon University
Provably Constant-Time Motion Planning
Abstract: In many robotic applications, including logistics and manufacturing, robots often operate in semi-structured environments and perform highly repetitive manipulation tasks. Additionally, large parts of these environments are static most of the time. Fast and reliable motion planning is one of the key elements that ensure efficient operations in such environments. A very common example [...]
Carnegie Mellon University
Planning to Optimize and Learn Reward in Navigation Tasks in Structured Environments with Time Constraints
Abstract: Planning problems in which an agent must perform tasks for reward by navigating its environment while constrained by time and location have a wide variety of applications in robotics. Many real-world environments in which such planning problems apply, such as office buildings or city streets, are very structured. They consist of passages with notable [...]
Carnegie Mellon University
Robust and Scalable Perception For Autonomy
Abstract: Autonomous mobile robots have the potential to drastically improve the quality of our daily life. For example, self-driving vehicles could make transportation safer and more affordable. To navigate complex environments, such robots need a perception system that translates raw sensory data to high-level understanding. This thesis focuses on two fundamental challenges in learning such [...]
Carnegie Mellon University
Semantic Mapping for Autonomous Navigation and Exploration
Abstract: The last two decades have seen enormous progress in the sensors and algorithms for 3D perception, giving robots the means to build accurate spatial maps and localize themselves in them in real time. The geometric information in these maps is invaluable for navigation while avoiding obstacles, but insufficient, by itself, for robots to robustly [...]
Carnegie Mellon University
Efficient Robot Decision-Making for Achieving Multiple Independent Tasks
Abstract: We focus on robotics applications where a robot is required to accomplish a set of tasks that are partially observable and evolve independently of each other according to their dynamics. One such domain that we target in this work is decision-making for a robot waiter waiting tables at a restaurant. The robot waiter should [...]
Carnegie Mellon University
Decentralized Navigation of Quadrotor Teams in Uncertain Workspaces
Abstract: A fundamental requirement for realizing scalable and responsive real-world multi-robot systems for time-sensitive critical applications such as search and rescue or building clearance is a motion-planning and coordination framework that exhibits three essential properties. The first property is safety that encompasses aspects relating to kinodynamic feasibility and collision-avoidance. The second property is reliability that [...]
Carnegie Mellon University
Bayesian Models for Science-Driven Robotic Exploration
Abstract Planetary rovers allow for science investigations at remote locations. They have traversed many kilometers and made major scientific discoveries. However, rovers spend a considerable amount of time awaiting instructions from mission control. The reason is that they are designed for highly supervised data collection, not for autonomous exploration. The exploration of farther worlds will [...]
Carnegie Mellon University
Heuristics for routing and scheduling of Spatio-temporal type problems in industrial environments
Abstract: Spatio-temporal problems are fairly common in industrial environments. In practice, these problems come with different characteristics and are often very hard to solve optimally. So, practitioners prefer to develop heuristics that exploit mathematical structure specific to the problem for obtaining good performance. In this thesis, we will present work on heuristics for 3 different [...]
Carnegie Mellon University
Relationships in instance segmentation and anomaly detection
Abstract: This thesis primarily covers work on two different tasks in computer vision: (1) anomaly detection and (2) instance segmentation. Anomaly detection is an underexplored unsupervised problem that has existed in many fields. On the other hand, instance (and panoptic) segmentation is a supervised problem that can leverage the powerful data and key developments from [...]
Carnegie Mellon University
Dynamical Model Learning and Inversion for Aggressive Quadrotor Flight
Abstract: Quadrotor applications have seen a surge recently and many tasks require precise and accurate controls. Flying fast is critical in many applications and the limited onboard power source makes completing tasks quickly even more important. Staying on a desired course while traveling at high speeds and high accelerations is difficult due to complex and [...]
Carnegie Mellon University
Person Transfers Between Multiple Service Robots
Abstract: As more service robots are deployed in the world, human-robot interaction will not be limited to one-to-one interactions between users and robots. Instead, users will likely have to interact with multiple robots, simultaneously or sequentially, throughout their day to receive services and complete different tasks. In this thesis, I describe work in which my [...]
Carnegie Mellon University
Understanding, Exploiting and Improving Inter-view Relationships
Abstract: Multi-view machine learning has garnered substantial attention in various applications over recent years. Many such applications involve learning on data obtained from multiple heterogeneous sources of information, for example, in multi-sensor systems such as self-driving cars, or monitoring intensive care patient vital signs at their bed-side. Learning models for such applications can often benefit [...]
Carnegie Mellon University
Human-in-the-loop Control of Mobile Robots
Abstract: Human-in-the-loop control for mobile robots is an important aspect of robot operation, especially for navigation in unstructured environments or in the case of unexpected events. However, traditional paradigms of human-in-the-loop control have relied heavily on the human to provide precise and accurate control inputs to the robot, or reduced the role of the human [...]