PhD Thesis Proposal
Adaptive Robotic Assistance through Observations of Human Behavior
Abstract: Assistive robots should take actions that support people's goals. This is especially true as robots enter into environments where personal agency is paramount, such as a person's home. Home environments have a wide variety of "optimal' solutions that depend on personal preference, making it difficult for a robot to know the goal it should [...]
Beyond Pick-and-Place: Towards Dynamic and Contact-rich Motor Skills with Reinforcement Learning
Abstract: Interactions with the physical world are at the core of robotics. However, robotics research, especially in manipulation, has been mainly focused on tasks with limited interactions with the physical world such as pick-and-place or pushing objects on the table top. These interactions are often quasi-static, have predefined or limited sequence of contact events and [...]
Adaptive-Anytime Planning and Mapping for Multi-Robot Exploration in Large Environments
Abstract: Robotic systems are being leveraged to explore environments too hazardous for humans to enter. Robot sensing, compute, and kinodynamic (SCK) capabilities are inextricably tied to the size, weight, and power (SWaP) constraints of the vehicle. When designing a robot team for exploration, the diversity and types of robots used must be carefully considered because [...]
Enabling Data-Efficient Real-World Model-Based Manipulation by Estimating Preconditions for Inaccurate Models
Abstract: This thesis explores estimating and reasoning about model deviation in robot learning for manipulation to improve data efficiency and reliability to enable real-robot manipulation in a world where models are inaccurate but still useful. Existing strategies are presented for improving planning robustness with low amounts of real-world data by an empirically estimated model precondition to guide [...]
Robust Adaptive Reinforcement Learning for Safety Critical Applications via Curricular Learning
Abstract: Reinforcement Learning (RL) presents great promises for autonomous agents. However, when using robots in a safety critical domain, a system has to be robust enough to be deployed in real life. For example, the robot should be able to perform across different scenarios it will encounter. The robot should avoid entering undesirable and irreversible [...]
Towards Photorealistic Dynamic Capture and Animation of Human Hair and Head
Abstract: Realistic human avatars play a key role in immersive virtual telepresence. To reach a high level of realism, a human avatar needs to faithfully reflect human appearance. A human avatar should also be drivable and express natural motions. Existing works have made significant progress on building drivable realistic face avatars, but they rarely include [...]
Eye Gaze for Intelligent Driving
Abstract: Intelligent vehicles have been proposed as one path to increasing vehicular safety and reduce on-road crashes. Driving intelligence has taken many forms, ranging from simple blind spot occupancy or forward collision warnings to lane keeping and all the way to full driving autonomy in certain situations. Primarily, these methods are outward-facing and operate on [...]
Passive Coupling in Robot Swarms
Abstract: In unstructured environments, ant colonies demonstrate remarkable abilities to adaptively form functional structures in response to various obstacles, such as stairs, gaps, and holes. Drawing inspiration from these creatures, robot swarms can collectively exhibit complex behaviors and achieve tasks that individual robots cannot accomplish. Existing modular robot platforms that employ dynamic coupling and decoupling [...]
Learning to Perceive and Predict Everyday Interactions
Abstract: This thesis aims to develop a computer vision system that can understand everyday human interactions with rich spatial information. Such systems can benefit VR/AR to perceive the reality and modify its virtual twin, and robotics to learn manipulation by watching human. Previous methods have been limited to constrained lab environment or pre-selected objects with [...]
Active Vision for Manipulation
Abstract: Decades of research on computer vision has highlighted the importance of active sensing -- where the agent actively controls parameters of the sensor to improve perception. Research on active perception the context of robotic manipulation has demonstrated many novel and robust sensing strategies involving a multitude of sensors like RGB and RGBD cameras, a [...]
Design Iteration of Dexterous Compliant Robotic Manipulators
Abstract: One goal of personal robotics is to have robots in homes performing everyday tasks efficiently to improve our quality of life. Towards this end, manipulators are needed which are low cost, safe around humans, and approach human-level dexterity. However, existing off-the-shelf manipulators are expensive both in cost and manufacturing time, difficult to repair, and [...]
Whisker Sensors for Unstructured Environments
Abstract: As robot applications expand from controllable factory settings to unknown environments, the robots will need a larger breadth of sensors to perceive these complex environments. In this thesis, I focus on developing whisker sensors for robot perception. The inspiration for whisker sensors comes from the biological world, where whiskers serve as tactile and flow [...]
Sparse-view 3D in the Wild
Abstract: Reconstructing 3D scenes and objects from images alone has been a long-standing goal in computer vision. We have seen tremendous progress in recent years, capable of producing near photo-realistic renderings from any viewpoint. However, existing approaches generally rely on a large number of input images (typically 50-100) in order to compute camera poses and [...]
Carnegie Mellon University
Spectral Mapping using Simple Sensors for Micro-Explorers
Abstract: Spectral mapping is an essential task in exploration as it expands our understanding of material composition in an explored region. Although imaging spectrometers are ideal for obtaining spectra to construct spectral maps, their large size, high power consumption, and operational complexity make them impractical for small rovers and limited missions. In contrast, RGB cameras [...]
Simulation-driven vision-based tactile sensor design using Physics Based Rendering
Abstract: Touch is an essential sensing modality for making autonomous robots more dexterous and works collaboratively with humans. With the advent of vision-based tactile sensors, roboticists have tried to incorporate tactile sensors in various robot structures for various robotic manipulation tasks to increase robustness, precision, and reliability. However, the design of vision-based tactile sensors is [...]
Efficient Interactive Learning with Unobserved Confounders
Abstract: Interactive learning systems like self-driving cars, recommender systems, and large language model chatbots are becoming increasingly ubiquitous in everyday life. From a machine learning perspective, the key technical challenge underlying such systems is that rather than simple prediction on i.i.d. data, an interactive learner influences the distribution of inputs it sees via the choices [...]
Learning to Manipulate Using Diverse Datasets
Abstract: Manipulation is a key challenge in the robotic fields that impedes the deployment of robots in real-world scenarios. While notable advancements have been made in solving high/mid level planning problems, such as decomposing tasks (e.g. "bring me a bottle") into primitives (e.g. "pick up bottle"), the acquisition of fundamental manipulation primitives remains a difficult [...]
Unified Control for Over and Fully-Actuated Aerial Vehicles
Abstract: The growing domain of aerial robotics necessitates advancements in the control strategies and robustness of over-actuated and fully-actuated aerial vehicles. This thesis proposal makes contributions to this endeavor by providing in-depth analysis and methodologies concerning these vehicles, control allocation strategies during actuator failures, high-fidelity simulations, and a unified control framework. Our completed work has [...]
Personalized Context-aware Affective Nonverbal Robot Feedback
Abstract: We first consider the problem of estimating context, specifically key features of the human state. We predict engagement-related events in an educational activity before the end of that activity, which could allow the robot to provide feedback early enough to improve the human's experience. We then explore generating nonverbal affective robot behavior by correlating [...]
Watch, Practice, Improve: Towards In-the-wild Manipulation
Abstract: The longstanding dream of many roboticists is to see robots perform diverse tasks in diverse environments. To build such a robot that can operate anywhere, many methods train on robotic interaction data. While these approaches have led to significant advances, they rely on heavily engineered setups or high amounts of supervision, neither of which [...]
Preference Based Optimization of Multi-Objective Robot Performance
Abstract: Robotic systems often require that tradeoffs be made--for example, between performance and robustness, power and longevity, or efficiency and safety. While roboticists can design cost functions with hand-picked weights for different metrics, it is not always a straightforward task, particularly when some aspects of performance are not easily quantified. This can occur especially when [...]
Ensuring safety for uncertain high-dimensional robotic systems
Abstract: Two major obstacles for safe control and planning are (1) scaling to high-dimensional systems and (2) handling uncertain systems. This is problematic because such systems are ubiquitous in practice: e.g. drones with unknown drag, manipulators carrying unknown packages. In this proposal, we aim to address both challenges. At the control level, we have synthesized [...]
Trustworthy Learning using Uncertain Interpretation of Data
Abstract: Non-parametric models are popular in real-world applications of machine learning. However, many modern ML methods that ensure that models are pragmatic, safe, robust, fair, and otherwise trustworthy in increasingly critical applications, assume parametric, differentiable models. We show that, by interpreting data as locally uncertain, we can achieve many of these without being limited to [...]
Allocation, Planning, and Control in Off-road Automated Convoy Operations
Abstract: The lack of structure in off-road terrains makes off-road operations of automated platforms difficult. The difficulty arises from uncertainty in the optimality and safety of the actions (e.g., planning and control) taken by the automated platform. When multiple automated platforms are required to act in a coordinated manner (e.g., a convoy) in complex cluttered [...]
Learning Generalizable Robot Skills for Dynamic and Interactive Tasks
Abstract: Enabling robots to perform complex dynamic tasks such as picking up an object in one sweeping motion or pushing off a wall to quickly turn a corner is a challenging problem. The dynamic interactions implicit in these tasks are critical for successful task execution. Furthermore, given the interactive nature of such tasks, safety, in [...]
Low-Cost Multimodal Sensing and Dexterity for Deformable Object Manipulation
Abstract: To integrate robots seamlessly into daily life, they must be able to handle a variety of tasks in diverse environments, like assisting in hospitals or cooking in kitchens. Many of the items in these environments are deformable such as bedding in hospitals or vegetables in kitchens, and a certain level of dexterity is necessary [...]
Learning Safe Human-Robot Interactions for a Seamlessly Shared Airspace
Abstract: The growing need for fully autonomous aerial operations in shared spaces, necessitates the development of reliable agents capable of navigating safely and seamlessly alongside uncertain human agents. In response, we advocate endowing autonomous agents with the ability to predict human actions, comprehend and ground abstract rules in the action space, and embrace the uncertainty [...]
Informative Path Planning Toward Autonomous Real-World Applications
Abstract: Gathering information from the physical world plays a crucial role in many applications—whether it be scientific research, environmental monitoring, search and rescue, defense, or disaster response. The utilization of robots for information gathering allows for the leveraging of intelligent algorithms to efficiently collect data, providing critical insights and facilitating informed decision-making. These autonomous robots [...]
Efficient Sensor Coverage in Complex Environments
Abstract: This thesis develops sensor coverage algorithms for mobile robots that are scalable to large and complex environments. The core challenge is computing the shortest paths that can direct one or more robots to sweep onboard sensors over all accessible surfaces within an environment. This problem resembles the watchman route problem that is known to [...]
Combining Physics-Based Light Transport and Neural Fields for Robust Inverse Rendering
Abstract: Inverse rendering — the process of recovering shape, material, and/or lighting of an object or environment from a set of images — is essential for applications in robotics and elsewhere, from AR/VR to perception on self-driving vehicles. While it is possible to perform inverse rendering from color images alone, it is often far easier [...]
Robotic Climbing for Extreme Terrain Exploration
Abstract: Climbing robots can operate in steep and unstructured environments that are inaccessible to other ground robots, with applications ranging from the inspection of artificial structures on Earth to the exploration of natural terrain features throughout the solar system. Climbing robots for planetary exploration face many challenges to deployment, including mass restrictions, irregular surface features, [...]
Exploration for Continually Improving Robots
Abstract: General purpose robots should be able to perform arbitrary manipulation tasks, and get better at performing new ones as they obtain more experience. The current paradigm in robot learning involves imitation or simulation. Scaling these approaches to learn from more data for various tasks is bottle-necked by human labor required either in collecting demonstrations [...]
Deep 3D Geometric Reasoning for Robot Manipulation
Abstract: To solve general manipulation tasks in real-world environments, robots must be able to perceive and condition their manipulation policies on the 3D world. These agents will need to understand various common-sense spatial/geometric concepts about manipulation tasks: that local geometry can suggest potential manipulation strategies, that policies should be invariant across choice of reference frame, [...]
Towards diverse zero-shot manipulation via actualizing visual plans
Abstract: In this thesis, we seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation — interacting with unseen objects in novel scenes without test-time adaptation. Robots that can be reliably deployed out-of-the-box in new scenarios have the potential for helping humans in everyday tasks. Not requiring any test-time training through demonstrations or [...]
Deep Learning for Sensors: Development to Deployment
Abstract: Robots rely heavily on sensing to reason about physical interactions, and recent advancements in rapid prototyping, MEMS sensing, and machine learning have led to a plethora of sensing alternatives. However, few of these sensors have gained widespread use among roboticists. This thesis proposes a framework for incorporating sensors into a robot learning paradigm, from [...]
Towards Influence-Aware Safe Human-Robot Interaction
Abstract: In recent years, we have seen through recommender systems on social media how influential (and potentially harmful) algorithms can be in our lives, sometimes creating polarization and conspiracies that lead to unsafe behavior. Now that robots are also growing more common in the real world, we must be very careful to ensure that they [...]
Design Principles for Robotics Systems that Support Human-Human Collaborative Learning
Abstract: Robots possess unique affordances granted by combining software and hardware. Most existing research focuses on the impact of these affordances on human-robot collaboration, but the theory of how robots can facilitate human-human collaboration is underdeveloped. Such theory would be beneficial in education. An educational device can afford collaboration in both assembly and use. This [...]
Robust Incremental Distributed Collaborative Simultaneous Localization and Mapping
Abstract: Multi-robot teams show exceptional promise across applications like Search-and-Rescue, disaster-response, agriculture, forestry, and scientific exploration due to their ability to go where humans cannot, parallelize activity, operate robustly to failures, and expand capabilities beyond that of an individual robot. Collaborative Simultaneous Localization and Mapping (C-SLAM) is a fundamental capability for these multi-robot teams as [...]
A Unified Control Framework for Robust Aerial Manipulation
Abstract: Aerial robots are now widely employed in diverse applications, such as delivery, environmental monitoring, and especially aerial manipulation—the focus of this thesis. Aerial manipulation involves integrating robotic arms with drones to perform physical tasks remotely. This capability is particularly crucial for operations that are either too dangerous or inaccessible for humans, such as high-altitude [...]