PhD Thesis Proposal
Measuring and Modeling Kinesic Signals in Social Communication
Abstract: Humans use subtle and elaborate body signals to convey their thoughts, emotions, and intentions. "Kinesics" is a term that refers to the study of such body movements used in social communication, including facial expressions and hand gestures. Understanding kinesic signals is fundamental to understanding human communication; it is among the key technical barriers to [...]
Probabilistic Approaches for Pose Estimation
Abstract: Pose estimation is central to several robotics applications such as registration, manipulation, SLAM, etc. In this thesis, we develop probabilistic approaches for fast and accurate pose estimation. A fundamental contribution of this thesis is formulating pose estimation in a parameter space in which the problem is truly linear and thus globally optimal solutions can [...]
Ada J. Zhang: Personalized Human Motion Classification
Abstract: Algorithms for human motion understanding have a wide variety of applications, including health monitoring, performance assessment, and user interfaces. However, differences between individual styles make it difficult to achieve robust performance, particularly for individuals who were not in the training population. We believe that adapting algorithms to individual behaviors is essential for effective human [...]
Carnegie Mellon University
Learning to learn from simulation: Using simulations to expedite learning on robots
Abstract: Robot controllers, including locomotion controllers, often consist of expert-designed heuristics. These heuristics can be hard to tune, particularly in higher dimensions. It is typical to use simulation to tune or learn these parameters and test on hardware. However, controllers learned in simulation often don't transfer to hardware due to model mismatch. This necessitates controller [...]
Carnegie Mellon University
Visual Learning without Exhaustive Supervision
Abstract Machine learning models have led to remarkable progress in visual recognition. A key driving factor for this progress is the abundance of labeled data. Over the years, researchers have spent a lot of effort curating visual data and carefully labeling it. However, moving forward, it seems impossible to annotate the vast amounts of visual [...]
Carnegie Mellon University
Learning with Clusters
Abstract As machine learning becomes more ubiquitous, clustering has evolved from primarily a data analysis tool into an integrated component of complex machine learning systems, including those involving dimensionality reduction, anomaly detection, network analysis, image segmentation and classifying groups of data. With this integration into multi-stage systems comes a need to better understand interactions between [...]
Carnegie Mellon University
Intra-Robot Replanning and Learning for Multi-Robot Teams in Complex Dynamic Domains
Abstract: In complex dynamic multi-robot domains, we have a set of individual robots that must coordinate together through a team planner that inevitably makes assumptions based on probabilities about the state of world and the actions of the individuals. Eventually, the individuals may encounter failures, because the team planner’s models of the states and actions [...]
Carnegie Mellon University
Automated Collaborations Among Neighborhood-based Search Heuristics
Abstract: For this thesis, we propose to study how to automatically combine multiple neighborhood-based heuristics. For most computationally challenging problems, there exists multiple heuristics, and it is generally the case that any such heuristic exploits only a limited number of aspects among all the possible problem characteristics that we can think of. As a result, [...]
Carnegie Mellon University
Computational Design Tools for Accessible Robotics
Abstract: A grand vision in robotics is that of a future wherein robots are integrated in daily human life just as smart phones are today. Such pervasive integration of robots would greatly benefit from faster design and manufacturing of robots that cater to individual needs. However, robots of today often take years to be created [...]
Carnegie Mellon University
Soft-Matter Robotic Materials
Abstract: Soft machines and electronics are key components for emerging applications in wearable biomonitoring, human-machine interaction, and soft robotics. In contrast to conventional machines and electronics, soft-matter technologies provide a method for replicating these traditionally rigid devices using intrinsically soft materials that exhibit properties similar to soft biological tissue. This provides a path forward for [...]
Carnegie Mellon University
Exploiting Redundancy for Learning Visual Representations
Abstract: Our visual world is highly structured and the visual data is highly redundant. In recent years, the computer vision field has been transformed by the success of Convolutional Neural Networks (ConvNets). However, the structure and redundancy in visual data has not been well explored in deep learning. The benefits of exploring data redundancy are [...]
Carnegie Mellon University
Towards Generalization and Efficiency of Reinforcement Learning
Abstract In classic supervised machine learning, a learning agent behaves as a passive observer: it receives examples from some external environment which it has no control over and then makes predictions. The predictions the agent made will not affect any future examples it will see (i.e., examples are identically and independently sampled from some unknown [...]
Carnegie Mellon University
Depth Imaging for Navigation in Challenging Environments
Abstract: Depth sensors for robust navigation must measure scenes in darkness, bright light, and in scattering media. Scanning LIDAR devices are the most robust to these conditions, but capture sparse measurements, are slow, and expensive. Consumer depth cameras, on the other hand, are inexpensive and produce dense, high rate depth measurements, but fail in bright [...]
Carnegie Mellon University
Observing Humans In Their Natural Habitat: Data, Algorithms, and Analysis
Abstract: Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. First, we need to give computers insight [...]
Carnegie Mellon University
Analysis of Spatio-Temporally Varying Features in Optical Coherence Tomographic (OCT) and Ultrasound (US) Image Sequences
Abstract: Optical Coherence Tomography (OCT) and Ultrasound (US) are non-ionizing and non-invasive imaging modalities that are clinically used to visualize anatomical structures in the body. OCT has been widely adopted in clinical practice due to its micron-scale resolution to visualize in-vivo structures of the eye. Ultra-High Frequency Ultrasound (UHFUS) can capture images at a depth [...]
Carnegie Mellon University
Planning for Energy-Efficient Coverage and Exploratory Deviation by Robots in Rivers
Abstract: Manual collection of environmental data over a large area can be a time-consuming, costly, and even dangerous process, making it a perfect candidate for automation with mobile robots. Despite this clear suitability and numerous advances in robotics resulting in decreased costs, improved reliability, and increased ease of use, the problem of powering autonomous robots [...]
Carnegie Mellon University
Learning to Forecast Egocentric and Allocentric Behavior in Diverse Domains
Abstract: Reasoning about the future is fundamental to intelligence. In this work, I consider the problem of reasoning about the future actions of an intelligent agent. This poses two key questions. How can we build learning-based systems to forecast the behavior of observed agents (third-person, "allocentric forecasting")? More challenging is the question: how should we [...]
Carnegie Mellon University
Search-based Robust Motion Planning under Uncertainty Guided by Multiple Heuristics
Abstract: Motion planning has achieved a great success in many robotic applications but still suffers in the real world under ample uncertainty. For example, manipulation involves interaction with unstructured and stochastic environments, which results in motion uncertainty. Perception that provides understanding of the environment is also not perfect, which in turn leads to sensing uncertainty. [...]
Learning Multi-Modal Navigation for Unmanned Ground Vehicles
Abstract: A robot that operates efficiently in a team with a human in an unstructured outdoor environment must be able to translate commands from a modality that is intuitive to its operator into actions. This capability is especially important as robots become ubiquitous and interact with untrained users. For this to happen, the robot must [...]
Carnegie Mellon University
Scaling up Self-Supervision for Robot Learning
Abstract: A general purpose robot will need to interact with objects in cluttered environments with minimal supervision. Machine learning provides methods that can deal with these complex tasks without explicitly modelling the environment. More recently, deep learning techniques combined with large scale data has revolutionized the fields of computer vision, language processing and reinforcement learning. [...]
Carnegie Mellon University
MRFMaps: A Representation for Multi-Hypothesis Dense Volumetric SLAM
Abstract: Robust robotic flight requires tightly coupled perception and control. Conventional approaches employ a SLAM algorithm to infer the most likely trajectory and then generate an occupancy grid map using dense sensor data for planning purposes. In such approaches all the robustness and accuracy costs are offset to the SLAM algorithm; if there are any [...]
Carnegie Mellon University
Sparse and Dense Methods for Underwater Localization and Mapping with Imaging Sonar
Abstract: Imaging sonars have been used for a variety of tasks geared towards increasing autonomy of underwater vehicles: image registration and mosaicing, vehicle localization, object recognition, mapping, and path planning, to name a few. However, the complexity of the image formation has led many algorithms to make the restrictive assumption that the scene geometry is [...]
Carnegie Mellon University
Deep Interpretable Non-rigid Structure from Motion
Abstract: Current non-rigid structure from motion (NRSfM) algorithms are limited with respect to: (i) the number of images, and (ii) the type of shape variability they can handle. This has hampered the practical utility of NRSfM for many applications within vision. Deep Neural Networks (DNNs) are an obvious candidate to help with such issue. However, [...]
Carnegie Mellon University
Vision with Small Baselines
Abstract: Portable camera sensor systems are becoming more and more popular in computer vision applications such as autonomous driving, virtual reality, robotics manipulation and surveillance, due to the decreasing expense and size of RGB camera. Despite the compactness and portability of the small baseline vision systems, it is well-known that the uncertainty in range finding [...]
Carnegie Mellon University
Machine Imagination: Data-driven User Controllable Visual Content Creation
Abstract: Humans have the remarkable ability to create visual worlds far beyond what could be seen by human eye, including inferring the state of unobserved, imagining the unknown, and thinking about diverse possibilities about what lies in the future. Machines lack this inquisitive ability despite the current revolution in machine learning and computer vision. We [...]
Carnegie Mellon University
Spatiotemporal Understanding of People Using Scenes, Objects, and Poses
Abstract: Humans are arguably one of the most important entities that AI systems would need to understand to be useful and ubiquitous. From autonomous cars observing pedestrians to assistive robots helping the elderly, a large part of this understanding is focused on recognizing human actions, and potentially, their intentions. Humans themselves are quite good at [...]
Carnegie Mellon University
Robust Active SLAM for Real-time Large-scale Indoor Dense 3D Reconstruction
Abstract: Real-time dense 3D reconstruction of indoor environments has been a popular research field due to its wide application, such as inspection, virtual / augmented reality (VR/AR), and service robotics. While many state-of-the-art algorithms are capable of reconstructing accurate 3D dense models in general indoor scenes, robustness is still an unsolved problem for all of [...]
Carnegie Mellon University
Terrain Relative Navigation for Lunar Polar Roving: Exploiting Geometry, Shadows, and Planning
Abstract: Water ice at the lunar poles could be the most valuable resource beyond planet Earth. However, that value is not foregone, and can only be determined by rovers that evaluate the distributions of abundance, concentration, and characteristics of this ice. The near-term explorations will be solar and unlikely to endure night, and hence are [...]
Carnegie Mellon University
Hybrid Soft Sensing in Robotic Systems
Abstract: The increasing prevalence of wearable technology in our daily lives has created a demand for safe and robust sensing skins. Largely inspired by human skin, the ultimate goal of electronic skins is to measure diverse sensory information, conform to surfaces, and avoid interfering with the natural mechanics of the host or user. These demands [...]
Carnegie Mellon University
Deep Reinforcement Learning Representations for Robotics
Abstract: A long standing goal of robotics research is to create algorithms that can automatically learn complex control strategies from scratch. Part of the challenge of applying such algorithms to robots is the choice of representation. While RL algorithms have been successfully applied to many robotics tasks such as Ball-in-a-Cup and various RoboCup soccer domains, [...]
Carnegie Mellon University
Learning multi-robot behaviors for online control
Abstract: Finding dynamically feasible and safe global plans for multi-agent teams in real world applications is enormously difficult because the decision branching factor, when considering all possible interactions across agents and an environment, is usually intractable. Humans, however, have great success in the multi-agent planning domain by using behaviors: practiced, coordinated responses for groups of [...]
Carnegie Mellon University
Routing for Persistent Exploration in Dynamic Environments with Teams of Energy-Constrained Robots
Abstract: In domains requiring effective situational awareness with limited resources, prioritizing focus is critical. Search and rescue tasks require fast identification of safe avenues for rescuers to traverse the area. Inspection tasks must realize trends over long durations to identify issues caused by the confluence of high-stress modes that compound into catastrophic failure. Deploying robots [...]
Carnegie Mellon University
Enabling Role-Reversible Human-Robot Interaction by Leveraging Standardized Tools for Provider-Receiver Interactions
Abstract: Developing 'social intelligence' for assistive robots to seamlessly interact with humans remains an open research challenge. However, socially assistive robots typically engage in types of interactions that already exist between humans, which makes models of human-human interactions useful to inform the design of robot social behaviors. In particular, in applications such as healthcare, therapy [...]
Carnegie Mellon University
Towards Understanding and Mitigating Biases
Abstract: There are many problems in real life that involve collecting and aggregating evaluation from people, such as conference peer review and peer grading. In this thesis, we consider multiple sources of biases that may arise in this process: (1) human bias -- the data collected from people are noisy and reflect people's calibration criteria [...]
Sensor Planning for Large Numbers of Robots
Abstract: Teams of aerial sensing robots can provide valuable situational awareness to first responders during disasters and emergencies by locating victims, mapping environments, identifying hazards, and locating gas sources. While individual sensing tasks may feature a variety of goals, many tasks will be time-sensitive such as after a widespread disaster or due to imminent danger [...]
Carnegie Mellon University
Resource-Constrained State Estimation with Multi-Modal Sensing
Abstract: Accurate and reliable state estimation is essential for safe mobile robot operation in real-world environments because ego-motion estimates are required by many critical autonomy functions such as control, planning, and mapping. Computing accurate state estimates depends on the physical characteristics of the environment, the selection of suitable sensors to capture that information, and the [...]
Carnegie Mellon University
Open-world Object Detection and Tracking
Abstract: Computer vision today excels at recognition in narrow slices of the real world. Our systems seem to accurately detect cats, cars, or chairs, but largely ignore the vast diversity of objects in the world that are absent from our training datasets. Perception in the open world, however, requires detecting and tracking any object, regardless [...]
Carnegie Mellon University
‘Unboxing’ anomaly detection and panoptic segmentation
Abstract: Panoptic segmentation is a recent problem in computer vision that attempts to classify each pixel in an image according to its semantic and instance label (accomplishing both semantic segmentation and instance segmentation respectively). Most existing panoptic and instance segmentation methods run a detection-first pipeline, where a bounding box is placed around an object and [...]
Carnegie Mellon University
Transfers Between Multiple Service Robots
Abstract: With the deployment of more robots, human-robot interaction will no longer be limited to a one-to-one interaction between a user and a robot. Instead, users will likely have to interact with multiple robots, simultaneously or sequentially, throughout their day to receive services and complete different tasks. In this thesis proposal, I am proposing joint [...]
Carnegie Mellon University
Machine Learning Parallelism Could Be Adaptive, Composable and Automated
Abstract: In recent years, the pace of innovations in the fields of machine learning has accelerated. To cope with the sheer computational complexity of training large ML models on large datasets, researchers in SysML have created algorithms and systems that parallelize ML training and inference over multiple CPUs or GPUs, or even multiple computing nodes [...]
Carnegie Mellon University
Expressive Real-time Intersection Scheduling: New Methods for Adaptive Traffic Signal Control
Abstract: Traffic congestion is a widespread problem throughout global metropolitan areas. In this thesis, we consider methods to optimize the performance of traffic signals to reduce congestion. We begin by presenting Expressive Real-time Intersection Scheduling (ERIS), a schedule-driven intersection control strategy that runs independently on each intersection in a traffic network. For each intersection, ERIS [...]
Adaptive Planning and Control of Wheeled Mobile Robots in Challenging Environments
Abstract: Over the last two decades, we have seen driverless cars conquer the Mojave desert, drive on mars and operate on our streets and warehouses. One of the most fundamental requirements of such robots is their ability to navigate their environment with minimal human oversight. As more robots graduate from the confines of laboratories to [...]
Carnegie Mellon University
Learning Dense 3D Object Reconstruction without Geometric Supervision
Abstract: Geometric alignment across visual data has been the fundamental issue for effective and efficient computer vision algorithms. The established pixel correspondences between images indirectly infer the underlying 3D geometry, physically or semantically. While this builds the foundation of classical multi-view 3D reconstruction algorithms such as Structure from Motion (SfM) and Simultaneous Localization and Mapping [...]
Carnegie Mellon University
Eye Gaze for Assistive Manipulation
Abstract: Full robot autonomy is the traditional goal of robotics research. To work in a human-inhabited world, however, robots will often need to collaborate with humans. Many scenarios require human users to teleoperate robots to perform tasks, a paradigm that appears everywhere from space exploration, to disaster recovery, to assistive robotics. This collaboration enables tasks [...]
Carnegie Mellon University
Stability-Centric Mechanics for Rigid Body Manipulation
Abstract: The repertoire of human manipulation is filled with creative use of contacts to move the object about the hand and the environment. It’s the combination of these skills that makes human manipulation dexterous. However, in most robotic applications the robot just fix all contact points on the object and do grasping. Reliable robot manipulation [...]
Carnegie Mellon University
Robotic Grasping in the Wild
Zoom Link Abstract Robotics and artificial intelligence have witnessed tremendous progress in the past decade. Yet, we are still far from building the general purpose robot butler that can autonomously operate in homes and help with manipulation tasks like household chores. Grasping is an important action primitive for manipulation and needs to generalize to unstructured [...]
Carnegie Mellon University
Active Vision: Autonomous Aerial Cinematography with Learned Artistic Decision-Making
Zoom Link Abstract: Aerial cinematography is revolutionizing industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. Fundamentally, it is a tool with immense potential to improve human creativity, expressiveness, and sharing of experiences. However, safely piloting a drone while filming a moving target in the presence of obstacles is immensely [...]
Carnegie Mellon University
Efficient Robot Decision-Making for Achieving Multiple Independent Tasks
Zoom Link Abstract: We focus on robotics applications where a robot is required to accomplish a set of tasks that are partially observable and evolve independently of each other according to their dynamics. One such domain that we target in this work is decision-making for a robot waiter waiting tables at a restaurant. The robot [...]
Carnegie Mellon University
Provably Constant-time Motion Planning
Zoom Link Abstract: In manufacturing and warehouse scenarios, robots often perform recurring manipulation tasks in structured environments. Fast and reliable motion planning is one of the key elements that ensure efficient operations in such environments. A very common example scenario is of manipulators working at conveyor belts, where they have limited time to pick moving [...]
Carnegie Mellon University
Safe and Resilient Multi-Robot Systems: Heterogeneity and Human Presence
Zoom Link Abstract: In the mission of a multi-robot team, the large number of robots behave like a system that relies on networking to enable smooth information propagation and inter-robot interaction as the mission evolves in a collective fashion. Key to the success of mission operation demands for safe and reliable robot interactions within the [...]
Carnegie Mellon University
Heuristics for routing and scheduling of Spatio-Temporal type problems in industrial environments
Zoom Link Abstract: Spatio-temporal problems are fairly common in industrial environments. In practice, these problems come with different characteristics and are often very hard to solve optimally. So practitioners prefer to develop heuristics that exploit mathematical structure specific to the problem for obtaining good performance. In this proposal, I will present work on heuristics for [...]
Carnegie Mellon University
Understanding, Exploiting and Improving Inter-view Relationships
Zoom Link Abstract: Multi-view machine learning has received substantial attention in various applications over recent years. These applications typically involve learning on data obtained from multiple sources of information, such as, for example, in multi-sensor systems such as self-driving cars and patient bed-side monitoring. Learning models for such applications can often benefit from leveraging not [...]
Carnegie Mellon University
Computational Contact Modes for Robotics
Zoom Link Abstract: A central theme in robotics is that of robots interacting with the world through physical contact. Whether it is a walking robot or robotic manipulator picking up an object, such as a spoon, we desire robots that physically interact with their environments. One significant challenge in physical robot interactions involves dealing with [...]
Carnegie Mellon University
Planning and Execution using Inaccurate Models with Provable Guarantees on Task Completeness
Abstract: Modern planning methods are effective in computing feasible and optimal plans for robotic tasks when given access to accurate dynamical models. However, robots operating in the real world often face situations that cannot be modeled perfectly before execution. Thus, we only have access to simplified but potentially inaccurate models. This imperfect modeling can lead [...]
Carnegie Mellon University
Physical Interaction and Manipulation of the Environment using Aerial Robots
Abstract: There has been an increasing demand for applications that include aerial robots' physical interactions with their environment, such as contact inspection, package pickup, and drilling. The demand has pushed the research groups towards new robot architectures and methods, but only limited research has been done to enable real-world applications. Fully-actuated multirotors were developed to [...]
Carnegie Mellon University
Visual Recognition Towards Autonomy
Abstract: Perception for autonomy presents a collection of compelling challenges for visual recognition. We focus on three key challenges in this thesis. The first key challenge is learning representations for 2D data such as RGB images. 2D sensing brings unique challenges in scale variance and occlusion. Intuitively, the cues for recognizing a 3px tall object [...]
Carnegie Mellon University
Rich Models and Maps in Factor Graphs with Applications to Tactile Sensing
Abstract: Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception. Typically these methods rely on simple models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional observations. They must in turn infer states that are used downstream by planning and [...]
Carnegie Mellon University
Distributed Navigation of Quadrotor Teams in Uncertain 3D Workspaces
Abstract: A fundamental requirement for realizing scalable and responsive real-world multi-robot systems for time-sensitive critical applications such as search and rescue or building clearance is a motion-planning and coordination framework that exhibits two essential properties. The first property is safety which encompasses aspects relating to kinodynamic feasibility and collision-avoidance. The second property is reliability which [...]
Carnegie Mellon University
Bayesian Models for Science-Driven Robotic Exploration
Abstract: Planetary rovers have traversed many kilometers and made major scientific discoveries. However, they spend a considerable amount of time awaiting instructions from ground operators. The reason is that they are designed for automated science data collection, not for autonomous exploration. The exploration of more distant worlds with stronger communication constraints will require a new [...]
Verification and Accreditation of Artificial Intelligence
Abstract: This work involves formally verifying a trained model's adherence to important design specifications for the purpose of model accreditation. Accreditation of a trained model requires enumeration of the explicit operational conditions under which the model is certified to meet all necessary specifications. By verifying model adherence to specifications set by developers, we increase the [...]
Carnegie Mellon University
Direct-drive Hands: Making Robot Hands Transparent and Reactive to Contacts
Abstract: Industrial manipulators and end-effectors are a vital driver of the automation revolution. These robot hands, designed to reject disturbances with stiffness and strength, are inferior to their human counterparts. Human hands are dexterous and nimble effectors capable of a variety of interactions with the environment. Through this thesis we wish to answer a question: [...]
Carnegie Mellon University
Teleoperation via Intuition: Safe and Intent Oriented Navigation
Abstract: This thesis aims to enable seamless teleoperation of a mobile robot by a human operator, such that the robot navigates in unstructured environments following the operator’s intent intuitively, safely, and efficiently. The roles of the human and robot are disproportionate in traditional teleoperation: The human is responsible for most of the autonomy of the [...]
Carnegie Mellon University
Towards the Automated Design of Neural Networks
Abstract: Neural architecture search (NAS) is recently proposed to automate the process of designing network architectures. Instead of manually designing network architectures, NAS automatically finds the optimal architecture in a data-driven way. Despite its impressive progress, NAS is still far from being widely adopted as a common paradigm for architecture design in practice. This thesis [...]
Carnegie Mellon University
Dynamical Model Learning and Inversion for Aggressive Quadrotor Flight
Quadrotor applications have seen a surge recently and many tasks require precise and accurate controls. Flying fast is critical in many applications and the limited onboard power source makes completing tasks quickly even more important. Staying on a desired course while traveling at high speeds and high accelerations is difficult due to complex and stochastic [...]
Carnegie Mellon University
Development of an Agile and Dexterous Balancing Mobile Manipulator Robot
Abstract: The proposed thesis work focuses on the design and control of a new unique agile and dexterous mobile manipulator, the Carnegie Mellon University (CMU) ballbot. The CMU ballbot is a human-sized dynamically stable mobile robot that balances on a single ball. We present the development and integration of a new pair of seven-degree-of-freedom (7-DOF) [...]
Carnegie Mellon University
Accelerating Numerical Methods for Optimal Control
Abstract: Many modern control methods, such as model-predictive control, rely heavily on solving optimization problems in real time. In particular, the ability to efficiently solve optimal control problems has enabled many of the recent breakthroughs in achieving highly dynamic behaviors for complex robotic systems. The high computational requirements of these algorithms demand novel algorithms tailor-suited [...]
Carnegie Mellon University
Robust Object Representations for Robot Manipulation
Abstract: As robots become more common in our daily lives, they will need to interact with many different environments and countless types of objects. While we, as humans, can easily understand an object after seeing it only once, this task is not trivial for robots. Researchers have, for the most part, been left with two [...]
Carnegie Mellon University
Visual Representation and Recognition without Human Supervision
Abstract: Visual recognition models have seen great advancements by relying on large-scale, carefully curated datasets with human annotations. Most computer vision models leverage human supervision to either construct strong initial representations (e.g. using the ImageNet dataset) or for modeling the visual concepts relevant for downstream tasks (e.g. MS-COCO for object detection). In this thesis, we [...]
Carnegie Mellon University
Heuristic Search Based Planning by Minimizing Anticipated Search Efforts
Abstract: Robot planning problems in dynamic environments, such as navigation among pedestrians, driving at high-speed on densely populated roads, and manipulation for collaborative tasks alongside humans, necessitate efficient planning. Bounded-suboptimal heuristic search algorithms are a popular alternative to optimal heuristic search algorithms that compromise solution quality for computation speed. Specifically, these searches aim to find [...]
Carnegie Mellon University
Liquid Metal Actuators
Abstract: Bioinspired robotic actuators arise from the advances in soft materials and activation methods to achieve desired performance. Because of their intrinsic compliance, actuators built from soft materials and liquids can achieve elastic resilience and adaptability similar to their biological counterparts. Liquid metals provide great opportunities for creating an artificial muscle that generates forces at [...]
Carnegie Mellon University
Self-Learning of Structured Visual Representations
Abstract: Most computer vision models in deployment today are not learning. Instead, they are in a "test" mode, where they will behave the same way perpetually, until they are replaced by newer models. This is a problem, because it means the models may perform poorly as soon as their "test" environment becomes different from their [...]
Carnegie Mellon University
Resource-Constrained Learning and Inference for Visual Perception
Abstract: Real-world applications usually require computer vision algorithms to meet certain resource constraints. In this talk, I will present evaluation methods and principled solutions for both training and testing. First, I will talk about a formal setting for studying training under the non-asymptotic, resource-constrained regime, i.e., budgeted training. We analyze the following problem: "given a [...]
Social Navigation with Pedestrian Groups
Abstract: Autonomous navigation in human crowds (i.e., social navigation) presents several challenges: The robot often needs to rely on its noisy sensors to identify and localize the pedestrians in human crowds; The robot needs plan efficient paths to reach its goals; The robot needs to do so in a safe and socially appropriate manner. In [...]
Control of Robots with Nonstationary Dynamics
Abstract: Robots can be constructed with fewer resources and less strict constraints than is generally believed. Soft robots can be constructed with very few parts and from a wide variety of materials. This makes them a potentially appealing choice for applications where there are resource constraints on system fabrication. However, soft robot dynamics are difficult [...]
Meshlet Primitives for Dense RGB-D SLAM in Dynamic Environments
Abstract: Dense RGB-D SLAM has been well established as a method for achieving robust localization while providing high quality dense surface reconstruction. However, despite significant progress, dense RGB-D SLAM has remained difficult to achieve on computationally constrained platforms, such as those used on autonomous aerial vehicles. A significant limiting factor in the current state of [...]
Computational Light Transport with Interferometry
Abstract: Optical interferometry is the measurement of small, sub-wavelength distances by exploiting the wave nature of light. Due to its capability to resolve micron-scale displacements, it has found widespread applications in biomedical imaging, industrial fabrication, physics, and astrophysics. In this thesis, we introduce a set of techniques we call computational interferometry, that bring the benefits [...]
Carnegie Mellon University
3D Reconstruction using Differential Imaging
Abstract: 3D reconstruction has been at the core of many computer vision applications, including autonomous driving, visual inspection in manufacturing, and augmented and virtual reality (AR/VR). Despite the tremendous progress made over the years, there remain challenging open-research problems. This thesis addresses three such problems in 3D reconstruction. First, we address the problem of defocus [...]
Carnegie Mellon University
Beyond rigid objects: Data-driven Methods for Manipulation of Deformable Objects
Abstract: Manipulation of deformable objects challenges common assumptions made for rigid objects. Deformable objects have high intrinsic state representation and complex dynamics with high degrees of freedom, making it difficult for state estimation and planning. The completed work can be divided into two parts. In the first part, we explore reinforcement learning (RL) as a [...]
Carnegie Mellon University
Simulation, Perception, and Generation of Human Behavior
Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout this thesis, we show how the three aspects are deeply connected and [...]
Carnegie Mellon University
Structured Learning for Robust Robot Manipulation
Abstract: Robust and generalizable robots that can autonomously manipulate objects in semi-structured environments can bring material benefits to society. Data-driven learning approaches are crucial for enabling such systems by identifying and exploiting patterns in semi-structured environments, allowing robots to adapt to novel scenarios with minimal human supervision. However, despite significant prior work in learning for [...]
Simulation-based Planning for Pick-and-Place in Heavy Clutter using Non-prehensile Manipulation
Abstract: Robot manipulation in domestic households, industrial manufacturing and warehouses might require contact-rich interactions with objects in the environment. For pick-and-place style grasping tasks in cluttered scenes, it can be more economical for the robot to rely on non-prehensile actions vis-à-vis deliberate prehensile rearrangement. Non-prehensile actions also let the robot manipulate large and bulky objects [...]
Carnegie Mellon University
Learning with Diverse Forms of Imperfect and Indirect Supervision
Abstract: High capacity Machine Learning (ML) models trained on large, annotated datasets have driven impressive advances in several fields including natural language processing and computer vision, in turn leading to impactful applications of ML in areas such as healthcare, e-commerce, and predictive maintenance. However, obtaining annotated datasets at the scale required for training such models [...]
Carnegie Mellon University
3D Representation Learning for Perception and Prediction: A Modular Yet Highly Integrated Approach
Abstract: Modularized and cascaded autonomy stacks (object detection, then tracking and then trajectory prediction) have been widely adopted in many autonomous systems such as self-driving cars due to its interpretability. In this talk, I advocate the use of such a modular approach but improve its accuracy and robustness by developing different 3D representations for each [...]
Carnegie Mellon University
Understanding Unbalanced Datasets Through Simple Models and Dataset Exploration
Abstract: Computer vision models have proven to be tremendously capable of recognizing and detecting several classes and objects. They succeed in classes widely ranging in type and scale from humans to cans to pens. However, the best performing classes have abundant examples in large-scale datasets today. In unbalanced datasets, where some categories are seen in [...]
Carnegie Mellon University
Self-Supervising Occlusions for Vision
Abstract: Virtually every scene has occlusions. Even a scene with a single object exhibits self-occlusions - a camera can only view one side of an object (left or right, front or back), or part of the object is outside the field of view. More complex occlusions occur when one or more objects block part(s) of [...]
Carnegie Mellon University
Massively Parallelized Lazy Planning Algorithms
Abstract: Search-based planning algorithms enable autonomous agents like robots to come up with well-reasoned long horizon plans to achieve a given task objective. They do so by optimizing a task-specific cost function while respecting the constraints on either the agent (e.g. motion constraints) or the environment (e.g. obstacles). In robotics, such as in motion planning [...]
Run-Time Optimization in the Deep Learning Age
Abstract: In a recovery task one seeks to obtain an estimate of an unknown signal from a set of incomplete measurements. These problems arise in a number of computer vision applications, from image based tasks such as super-resolution and in-painting to 3D reconstruction tasks such as Non-Rigid Structure from Motion and scene flow estimation. Early [...]
Carnegie Mellon University
System Identification and Control of Multiagent Systems Through Interactions
Abstract: This thesis investigates the problem of identifying dynamics models of individual agents of a multiagent system (MAS) and exploiting these models to shape their behavior using robots extrinsic to the MAS. While task-based control of a MAS using onboard controllers of its agents is well studied, we investigate (a) how easy it is for [...]
Carnegie Mellon University
Driving Reconfigurable Unmanned Vehicle Design for Mobility Performance
Abstract: Unmanned ground vehicles are being deployed in increasingly diverse and complex environments. Advances in the field of robotics, including perception technology, computing power, and machine learning, have brought robots from the lab to the real world. Remote and autonomous vehicles are now used to explore volcanoes, caves, pipes, war zones, disaster sites, and even [...]
Towards Large-scale and Long-term Neural Map Representations
Abstract: We address the problem of large-scale and long-term neural map representations. Maps, as our prior understanding toward the environment, provide valuable information for modern robotics applications such as autonomous driving and AR/VR. The size of maps largely affects the end task performance: usually a more detailed map can support better performance, but would cost [...]
Manipulating Objects with Challenging Visual and Geometric Properties
Abstract: Object manipulation is a well-studied domain in robotics, yet manipulation remains difficult for objects with visually and geometrically challenging properties. Visually challenging properties, such as transparency and specularity, break assumptions of Lambertian reflectance that existing methods rely on for grasp estimation. On the other hand, deformable objects such as cloth pose both visual and [...]
Large Scale Dense 3D Reconstruction via Sparse Representations
Abstract: Scene reconstruction systems take in (3D) videos as input, and output 3D models with associated poses for inputs. With the demand of 3D content generation, the technique has been drastically evolving in recent years. For professionals equipped with depth sensors, efficient dense reconstruction systems have become available to efficiently recover scene geometry. For ordinary [...]
Teaching Agent Reward Functions via Demonstrations for Human Inverse Reinforcement Learning
Abstract: For intelligent agents (e.g. robots) to be seamlessly integrated into human society, humans must be able to understand their decision making. For example, the decision making of autonomous cars must be clear to the engineers certifying their safety, passengers riding them, and nearby drivers negotiating the road simultaneously. As an agent's decision making can [...]
Policy Decomposition: Approximate Optimal Control with Suboptimality Estimates
Abstract: Optimal Control is a formulation for designing controllers for dynamical systems by posing it as an optimization problem, whereby the desired long-term behavior of the system is expressed using a cost function. The objective is to compute a policy, i.e. a mapping from the state of the system to its control inputs, that minimizes [...]
Audience-Aware Legibility for Social Navigation
Abstract: Robots often need to communicate their goals to humans when navigating in a shared space to assist observers in anticipating the robot’s future actions. These human observers are often scattered throughout the environment, and each observer only has a partial view of the robot and its movements. A path that non-verbally communicates with multiple [...]
On Sample-Efficient Reinforcement Learning for Nuclear Fusion
Abstract: In many practical applications of reinforcement learning (RL), it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, determining the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or [...]
Carnegie Mellon University
Towards reconstructing non-rigidity from single camera
Abstract: In this proposal, we study how to infer 3D from images captured by a single camera, without assuming the target scenes / objects being static. The non-static setting makes our problem ill-posed and challenging to solve, but is vital in practical applications where target-of-interest is non-static. To solve ill-posed problems, the current trend in [...]
Efficient 3D Representations: Algebraic Surfaces for Differentiable Rendering
Abstract: In this proposal, we show how some classic computer vision tasks can robustly be solved via optimization techniques by using an object representation that is compact and interpretable. Specifically, we explore the applications and benefits of representing 3D objects with an analytical, algebraic function by building an approximate, ray-based differentiable renderer. Our approximate formulation [...]
Continual Robot Learning: Benchmarks and Modular Methods
Zoom Meeting Passcode: 841755 Abstract: The earliest reinforcement learning models were designed to learn one task, specified up-front. However, an agent operating freely in the real world will not in general be granted this luxury, as the demands placed on the agent may change as environments or goals change. We refer to this ever-shifting scenario [...]
Improving Robotic Exploration with Self-Supervision and Diverse Data
Abstract: Reinforcement learning (RL) holds great promise for improving robotics, as it allows systems to move beyond passive learning and interact with the world while learning from these interactions. A key aspect of this interaction is exploration: which actions should an RL agent take to best learn about the world? Prior work on exploration is typically [...]
Combining Offline Reinforcement Learning with Stochastic Multi-Agent Planning for Autonomous Driving
Abstract: Fully autonomous vehicles have the potential to greatly reduce vehicular accidents and revolutionize how people travel and how we transport goods. Many of the major challenges for autonomous driving systems emerge from the numerous traffic situations that require complex interactions with other agents. For the foreseeable future, autonomous vehicles will have to share the [...]
Causal Robot Learning for Manipulation
Abstract: Two decades into the third age of AI, the rise of deep learning has yielded two seemingly disparate realities. In one, massive accomplishments have been achieved in deep reinforcement learning, protein folding, and large language models. Yet, in the other, the promises of deep learning to empower robots that operate robustly in real-world environments [...]
Dense Reconstruction of Dynamic Structures from Monocular RGB Videos
Abstract: We study the problem of 3D reconstruction of {\em generic} and {\em deformable} objects and scenes from {\em casually-taken} RGB videos, to create a system for capturing the dynamic 3D world. Being able to reconstruct dynamic structures from casual videos allows one to create avatars and motion references for arbitrary objects without specialized devices, [...]
Learning via Visual-Tactile Interaction
Abstract: Humans learn by interacting with their surroundings using all of their senses. The first of these senses to develop is touch, and it is the first way that young humans explore their environment, learn about objects, and tune their cost functions (via pain or treats). Yet, robots are often denied this highly informative and [...]
Tactile SLAM: perception for dexterity via vision-based touch
Abstract: Touch provides a direct window into robot-object interaction, free from occlusion and aliasing faced by visual sensing. Collated tactile perception can facilitate contact-rich tasks---like in-hand manipulation, sliding, and grasping. Here, online estimates of object geometry and pose are crucial for downstream planning and control. With significant advances in tactile sensing, like vision-based touch, a [...]
Resource Allocation for Learning in Robotics
Abstract: Robots operating in the real world need fast and intelligent decision making systems. While these systems have traditionally consisted of human-engineered behaviors and world models, there has been a lot of interest in integrating them with data-driven components to achieve faster execution and reduce hand-engineering. Unfortunately, these learning-based methods require large amounts of training [...]
Planning with Dynamics by Interleaving Search and Trajectory Optimization
Abstract: Search-based planning algorithms enable autonomous agents like robots to come up with well-reasoned long-horizon plans to achieve a given task objective. They do so by searching over the graph that results from discretizing the state and action space. However, in robotics, several dynamically rich tasks require high-dimensional planning in the continuous space. For such [...]
Utilizing Panoptic Segmentation and a Locally-Conditioned Neural Representation to Build Richer 3D Maps
Abstract: Advances in deep-learning based perception and maturation of volumetric RGB-D mapping algorithms have allowed autonomous robots to be deployed in increasingly complex environments. For robust operation in open-world conditions however, perceptual capabilities are still lacking. Limitations of commodity depth sensors mean that complex geometries and textures cannot be reconstructed accurately. Semantic understanding is still [...]
Multi-Human 3D Reconstruction from Monocular RGB Videos
Abstract: We study the problem of multi-human 3D reconstruction from RGB videos captured in the wild. Humans have dynamic motion, and reconstructing them in arbitrary settings is key to building immersive social telepresence, assistive humanoid robots, and augmented reality systems. However, creating such a system requires addressing fundamental issues with previous works regarding the data [...]
Learning and Translating Temporal Abstractions across Humans and Robots
Abstract: Humans possess a remarkable ability to learn to perform tasks from a variety of different sources-from language, instructions, demonstration, etc. In each case, they are able to easily extract the high-level strategy to solve the task, such as the recipe of cooking a dish, whilst ignoring irrelevant details, such as the precise shape of [...]
Predicting The Future and Linking the Past: Learning and Constructing Structured Models for Robotic Manipulation
Abstract: Intelligent robotic agents need to reason about the dynamics of their surrounding world, and use such dynamics reasoning to make future predictions for efficient task planning. In addition, it is also desirable for robots to associate past experience in their memories to their current observation, and conduct analogical reasoning to complete tasks at their [...]
Perception for High-Speed Off-Road Driving
Abstract: On-road autonomous driving has seen rapid progress in recent years with driverless vehicles being tested in various cities worldwide. However, this progress is limited to cities with well-established infrastructure and has yet to transfer to off-road regimes with unstructured environments and few paved roads. Advances in high-speed and reliable autonomous off-road driving can unlock [...]
Continual Learning of Compositional Skills for Robust Robot Manipulation
Abstract: Real world robots need to continuously learn new manipulation tasks in a lifelong learning manner. These new tasks often share sub-structures (in the form of sub-tasks, controllers) with previously learned tasks. To utilize these shared sub-structures, we explore a compositional and object-centric approach to learn manipulation tasks. While compositionality in robot manipulation can manifest [...]
Equivalent Policy Sets for Learning Aligned Models and Abstractions
Abstract: Recent successes in model-based reinforcement learning (MBRL) have demonstrated the enormous value that learned representations of environmental dynamics (i.e., models) can impart to autonomous decision making. While a learned model can never perfectly represent the dynamics of complex environments, models that are accurate in the "right” ways may still be highly useful for decision [...]
Adaptive Robotic Assistance through Observations of Human Behavior
Abstract: Assistive robots should take actions that support people's goals. This is especially true as robots enter into environments where personal agency is paramount, such as a person's home. Home environments have a wide variety of "optimal' solutions that depend on personal preference, making it difficult for a robot to know the goal it should [...]
Beyond Pick-and-Place: Towards Dynamic and Contact-rich Motor Skills with Reinforcement Learning
Abstract: Interactions with the physical world are at the core of robotics. However, robotics research, especially in manipulation, has been mainly focused on tasks with limited interactions with the physical world such as pick-and-place or pushing objects on the table top. These interactions are often quasi-static, have predefined or limited sequence of contact events and [...]
Adaptive-Anytime Planning and Mapping for Multi-Robot Exploration in Large Environments
Abstract: Robotic systems are being leveraged to explore environments too hazardous for humans to enter. Robot sensing, compute, and kinodynamic (SCK) capabilities are inextricably tied to the size, weight, and power (SWaP) constraints of the vehicle. When designing a robot team for exploration, the diversity and types of robots used must be carefully considered because [...]
Enabling Data-Efficient Real-World Model-Based Manipulation by Estimating Preconditions for Inaccurate Models
Abstract: This thesis explores estimating and reasoning about model deviation in robot learning for manipulation to improve data efficiency and reliability to enable real-robot manipulation in a world where models are inaccurate but still useful. Existing strategies are presented for improving planning robustness with low amounts of real-world data by an empirically estimated model precondition to guide [...]
Robust Adaptive Reinforcement Learning for Safety Critical Applications via Curricular Learning
Abstract: Reinforcement Learning (RL) presents great promises for autonomous agents. However, when using robots in a safety critical domain, a system has to be robust enough to be deployed in real life. For example, the robot should be able to perform across different scenarios it will encounter. The robot should avoid entering undesirable and irreversible [...]
Towards Photorealistic Dynamic Capture and Animation of Human Hair and Head
Abstract: Realistic human avatars play a key role in immersive virtual telepresence. To reach a high level of realism, a human avatar needs to faithfully reflect human appearance. A human avatar should also be drivable and express natural motions. Existing works have made significant progress on building drivable realistic face avatars, but they rarely include [...]
Eye Gaze for Intelligent Driving
Abstract: Intelligent vehicles have been proposed as one path to increasing vehicular safety and reduce on-road crashes. Driving intelligence has taken many forms, ranging from simple blind spot occupancy or forward collision warnings to lane keeping and all the way to full driving autonomy in certain situations. Primarily, these methods are outward-facing and operate on [...]
Passive Coupling in Robot Swarms
Abstract: In unstructured environments, ant colonies demonstrate remarkable abilities to adaptively form functional structures in response to various obstacles, such as stairs, gaps, and holes. Drawing inspiration from these creatures, robot swarms can collectively exhibit complex behaviors and achieve tasks that individual robots cannot accomplish. Existing modular robot platforms that employ dynamic coupling and decoupling [...]
Learning to Perceive and Predict Everyday Interactions
Abstract: This thesis aims to develop a computer vision system that can understand everyday human interactions with rich spatial information. Such systems can benefit VR/AR to perceive the reality and modify its virtual twin, and robotics to learn manipulation by watching human. Previous methods have been limited to constrained lab environment or pre-selected objects with [...]
Active Vision for Manipulation
Abstract: Decades of research on computer vision has highlighted the importance of active sensing -- where the agent actively controls parameters of the sensor to improve perception. Research on active perception the context of robotic manipulation has demonstrated many novel and robust sensing strategies involving a multitude of sensors like RGB and RGBD cameras, a [...]
Design Iteration of Dexterous Compliant Robotic Manipulators
Abstract: One goal of personal robotics is to have robots in homes performing everyday tasks efficiently to improve our quality of life. Towards this end, manipulators are needed which are low cost, safe around humans, and approach human-level dexterity. However, existing off-the-shelf manipulators are expensive both in cost and manufacturing time, difficult to repair, and [...]
Whisker Sensors for Unstructured Environments
Abstract: As robot applications expand from controllable factory settings to unknown environments, the robots will need a larger breadth of sensors to perceive these complex environments. In this thesis, I focus on developing whisker sensors for robot perception. The inspiration for whisker sensors comes from the biological world, where whiskers serve as tactile and flow [...]
Sparse-view 3D in the Wild
Abstract: Reconstructing 3D scenes and objects from images alone has been a long-standing goal in computer vision. We have seen tremendous progress in recent years, capable of producing near photo-realistic renderings from any viewpoint. However, existing approaches generally rely on a large number of input images (typically 50-100) in order to compute camera poses and [...]
Carnegie Mellon University
Spectral Mapping using Simple Sensors for Micro-Explorers
Abstract: Spectral mapping is an essential task in exploration as it expands our understanding of material composition in an explored region. Although imaging spectrometers are ideal for obtaining spectra to construct spectral maps, their large size, high power consumption, and operational complexity make them impractical for small rovers and limited missions. In contrast, RGB cameras [...]
Simulation-driven vision-based tactile sensor design using Physics Based Rendering
Abstract: Touch is an essential sensing modality for making autonomous robots more dexterous and works collaboratively with humans. With the advent of vision-based tactile sensors, roboticists have tried to incorporate tactile sensors in various robot structures for various robotic manipulation tasks to increase robustness, precision, and reliability. However, the design of vision-based tactile sensors is [...]
Efficient Interactive Learning with Unobserved Confounders
Abstract: Interactive learning systems like self-driving cars, recommender systems, and large language model chatbots are becoming increasingly ubiquitous in everyday life. From a machine learning perspective, the key technical challenge underlying such systems is that rather than simple prediction on i.i.d. data, an interactive learner influences the distribution of inputs it sees via the choices [...]
Learning to Manipulate Using Diverse Datasets
Abstract: Manipulation is a key challenge in the robotic fields that impedes the deployment of robots in real-world scenarios. While notable advancements have been made in solving high/mid level planning problems, such as decomposing tasks (e.g. "bring me a bottle") into primitives (e.g. "pick up bottle"), the acquisition of fundamental manipulation primitives remains a difficult [...]
Unified Control for Over and Fully-Actuated Aerial Vehicles
Abstract: The growing domain of aerial robotics necessitates advancements in the control strategies and robustness of over-actuated and fully-actuated aerial vehicles. This thesis proposal makes contributions to this endeavor by providing in-depth analysis and methodologies concerning these vehicles, control allocation strategies during actuator failures, high-fidelity simulations, and a unified control framework. Our completed work has [...]
Personalized Context-aware Affective Nonverbal Robot Feedback
Abstract: We first consider the problem of estimating context, specifically key features of the human state. We predict engagement-related events in an educational activity before the end of that activity, which could allow the robot to provide feedback early enough to improve the human's experience. We then explore generating nonverbal affective robot behavior by correlating [...]