PhD Speaking Qualifier
Carnegie Mellon University
Ergodic Trajectory Optimization for Information Gathering
Abstract: Planetary robots currently rely on significant guidance from expert human operators. Science autonomy adds algorithms and methods for autonomous scientific exploration to improve efficiency of discovery and overcome limited communication bandwidth and delay bottlenecks. This research focuses on planning trajectories for information gathering and choosing sampling locations that have the most informative samples. We [...]
Carnegie Mellon University
Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis
Abstract: Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the [...]
Studying the Evolution of Pedestrian Group Space
Abstract: Imagine walking along a busy sidewalk, do you track the movement of every single individual? Or do you simply group pedestrians with similar moving patterns and then track the movement of this group? Grouping is a common behavior in pedestrian navigation and it is typically inappropriate for a robot to cut through the social [...]
Carnegie Mellon University
Soft actuators by electrochemical oxidation of liquid metal surfaces
Abstract: Soft robotic systems typically operate through the use of soft actuators constructed from highly deformable materials or liquids. Because of their intrinsic compliance, these actuators can achieve elastic resilience and adaptability similar to their biological counterparts. One challenge with engineering these artificial muscles is the selection of soft materials and activation methods while maintaining [...]
A Graph-Based Method for Joint Instance Segmentation of Point Clouds and Image Sequences
Abstract: While learning-based semantic instance segmentation methods have achieved impressive progress, their use is limited in robotics applications due to reliance on expensive training data annotations and assumptions of single sensor modality or known object classes. We propose a novel graph-based instance segmentation approach that combines information from a 2D image sequence and a 3D [...]
Continual Reinforcement Learning using Self-Activating Neural Ensembles
Abstract: The ability for an agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of generally intelligent agents. Most methods devised to address this problem depend heavily on well-defined task boundaries which simplify the problem considerably. Our task-agnostic method, Self-Activating Neural Ensembles (SANE), uses a hierarchical [...]
Carnegie Mellon University
Unsupervised 2D-3D Lifting with Deep Structure Priors
Abstract: Learning to estimate non-rigid 3D structures from 2D imaged observations is bottle-necked by the availability of abundant 3D annotated data. Learning methods that reduce the amount of required annotation is of high practical value. In this regard, Non-Rigid Structure from Motion (NRSfM) methods offer the opportunity to infer 3D structures solely from 2D annotations. [...]
Model Adaptation for Compliant Parallel Robot with Nonstationary Dynamics
Abstract: Soft robots can be constructed with few parts and from a wide variety of materials. This makes them a potentially appealing choice for applications where there are resource constraints on system fabrication. However, soft robot dynamics are difficult to accurately model analytically, due to a multiphysics coupling between shape, forces, temperature, and history of [...]
Adaptive Safety Margins for Safe Replanning under Time-Varying Disturbances
Abstract: Safe real-time navigation is a considerable challenge because engineers often need to work with uncertain vehicle dynamics, variable external disturbances, and imperfect controllers. A common strategy used to address safety is to employ hand-defined margins for obstacle inflation. However, arbitrary static margins often fail in more dynamic scenarios, and using worst-case assumptions proves to [...]
HyperDynamics: Generating Expert Dynamics Models by Observation
Abstract: We propose HyperDynamics, a framework that conditions on an agent’s interactions with the environment and optionally its visual observations, and generates the parameters of neural dynamics models based on inferred properties of the dynamical system. Physical and visual properties of the environment that are not part of the low-dimensional state yet affect its temporal [...]
Direct Fitting of Mixture Models
Abstract: There exist many choices of 3D shape representation. Some recent work has advocated for the use of Gaussian Mixture Models as a compact representation for 3D shapes and scenes. These models are typically fit to point clouds, even when the shapes were obtained as 3D meshes. Here we present a formulation for fitting Gaussian [...]
Carnegie Mellon University
Terrain Perception using Structured Light for Micro-Rovers
Abstract: With continuing advancement in technology, the future of planetary exploration is likely to be dominated by robotic missions. Yet rovers capable of science investigations are slow and bulky with very limited computing which prohibits demonstrating full autonomy. These rovers are also risk averse due to their huge mission cost. However there is a new [...]
Carnegie Mellon University
Analysis of Deadlock in Multirobot Systems
Abstract: Collision avoidance for multirobot systems is a well-studied problem. Recently, control barrier functions (CBFs) have been proposed for synthesizing controllers that guarantee safety while simultaneously encouraging goal stabilization for multiple robots. However, it has been noted that reactive control synthesis methods (such as CBFs) are prone to deadlock, an equilibrium of system dynamics that [...]
Interleaving Graph Search and Trajectory Optimization for Aggressive Quadrotor Flight
Abstract: Quadrotors can achieve aggressive flight by tracking complex maneuvers and rapidly changing directions. Planning for aggressive flight with trajectory optimization could be incredibly fast, even in higher dimensions, and can account for dynamics of the quadrotor, however, only provides a locally optimal solution. On the other hand, planning with discrete graph search can handle [...]
See, Hear, Explore: Curiosity via Audio-Visual Association
Abstract: Exploration is one of the core challenges in reinforcement learning. A common formulation of curiosity-driven exploration uses the difference between the real future and the future predicted by a learned model. However, predicting the future is an inherently difficult task which can be ill-posed in the face of stochasticity. In this work, we introduce [...]
MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
Abstract: We present a method to capture temporally coherent dynamic clothing deformation from a monocular RGB video input. In contrast to the existing literature, our method does not require a pre-scanned personalized mesh template, and thus can be applied to in-the-wild videos. To constrain the output to a valid deformation space, we build statistical deformation [...]
Policy Decomposition : Approximate Optimal Control with Suboptimality Estimates
Abstract: Owing to the curse of dimensionality, numerically computing global policies to optimal control problems for complex dynamical systems quickly becomes intractable. In consequence, a number of approximation methods have been developed. However, none of the current methods can quantify by how much the resulting control underperforms the elusive globally optimal solution. We propose Policy [...]
Inverse Reinforcement Learning with Explicit Policy Estimates
Abstract: Various methods for solving the inverse reinforcement learning (IRL) problem have been developed independently in machine learning and economics. In particular, the method of Maximum Causal Entropy IRL is based on the perspective of entropy maximization, while related advances in the field of economics instead assume the existence of unobserved action shocks to explain [...]
Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation
Abstract: To perform manipulation tasks in the real world, robots need to operate on objects with various shapes, sizes and without access to geometric models. It is often infeasible to train monolithic neural network policies across such large variance in object properties. Towards this generalization challenge, we propose task-axis controllers, which are defined relative to [...]
Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policies
Abstract: Real-world environments, such as homes, hospitals, and restaurants, often contain many objects that a robot could possibly manipulate. However, for a given manipulation task, only a small number of objects and object properties may actually be relevant. This talk presents CREST (Causal Reasoning for Efficient Structure Transfer), our approach to learn the relevant state [...]
Grasping Transparent, Specular, and Deformable Objects
Abstract: A large body of research exists on grasping for objects with ideal properties like Lambertian reflectance and rigidity. On the other hand, real-world environments contain many objects for which such properties do not hold, such as transparent, specular, and deformable objects. For such objects, new approaches are required to achieve the same level of [...]
PoseIt: A Visual-Tactile Dataset of Holding Poses for Grasp Stability Analysis
Abstract: When humans grasp objects in the real world, we often move our arm to hold the object in a different pose where we can use it. In contrast, typical lab settings only study the stability of the grasp immediately after lifting, without any subsequent re-positioning of the arm. However, an object’s stability could vary [...]
Planning to Minimize Human and Robot Efforts Over Tasks
Abstract: It is not feasible to pre-program robots a priori for every possible task they may encounter in unstructured domains. Upon encountering a task that a robot can't solve, one common strategy is to teach it new skills via demonstrations. However, demonstrating a task can often be more cumbersome than performing the task directly. This [...]
Fine-Tuning Offline Reinforcement Learning with Model-Based Policy Optimization
Abstract: In offline reinforcement learning (RL), we attempt to learn a control policy from a fixed dataset of environment interactions. This setting has the potential benefit of allowing us to learn effective policies without needing to collect additional interactive data, which can be expensive or dangerous in real-world systems. However, traditional off-policy RL methods tend [...]
Modeling Coupled Human-Robot Motion for Provable Safety
Abstract: Guide robots that help users who are blind or low vision navigate through crowds and complex environments show promise for improving accessibility in public spaces. These robots must provide real-time safety guarantees for the users, which requires accurate modeling of their behavior in the context of closely coupled human-robot motion. This model must also [...]
Diminished Reality for Close Quarters Robotic Telemanipulation
Abstract: In robot telemanipulation tasks, the robot itself can sometimes occlude a target object from the user's view. We investigate the potential of diminished reality to address this problem. Our method uses an optical see-through head-mounted display to create a diminished reality illusion that the robot is transparent, allowing users to see occluded areas behind [...]
Learning Compositional Radiance Fields of Dynamic Human Heads
Meeting ID: 942 4671 0665 Passcode: jkhzoom Abstract: Photorealistic rendering of dynamic humans is an important capability for telepresence systems. Recently, neural rendering methods have been developed to create high-fidelity models of humans and objects. Some of these methods do not produce results with high-enough fidelity for driveable human models (Neural Volumes) whereas others have [...]
An Experimental Design Perspective on Model-Based Reinforcement Learning
Abstract: In many practical applications of RL, it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, computing the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or dollars of [...]
Learning Model Preconditions for Planning with Multiple Models
Abstract: Different models can provide differing levels of fidelity when a robot is planning. Analytical models are often fast to evaluate but only work in limited ranges of conditions. Meanwhile, physics simulators are effective at modeling complex interactions between objects but are typically more computationally expensive. Learning when to switch between the various models can [...]
Reconstructing common objects to interact with
Abstract: We humans are able to understand 3D shapes of common daily objects and interact with them from a wide range of categories. We understand cups are usually cylinder-like and we can easily predict the shape of one particular cup, both in isolation or even when it is held by a human. We aim to [...]
A causal framework to diagnose and fix issues with doors
Abstract: Many animals, such as ravens, (and a fortiori humans) exhibit a great deal of physical intelligence that allows them to solve complex multi-step physical puzzles. This ability indicates an understanding or a faculty to represent causality and mechanisms, understand when something goes wrong, and figure out how to deal with it. As a step [...]
Designing Whisker Sensors to Detect Multiple Mechanical Stimuli for Robotic Applications
Abstract: Many mammals, such as rats and seals, use their whiskers as versatile mechanical sensors to gain precise information about their surroundings. Whisker-inspired sensors on robotic platforms have shown their potential benefit, improving applications ranging from drone navigation to texture mapping. Despite this, there is a gap between the engineered sensors and many of the [...]
Towards Complex Robot Motions with Reinforcement Learning
Abstract: Reinforcement learning has shown to be a powerful tool for decision-making problems. In this talk, we present the opportunities and challenges of enabling increasingly complex robot behavior with reinforcement learning. First, we present a system that combines reinforcement learning and extrinsic dexterity to solve a novel task of “occluded grasping”. To reach an occluded [...]
Search-based Path Planning for a High Dimensional Manipulator in Cluttered Environments Using Optimization-based Primitives
Abstract: In this work we tackle the path planning problem for a 21-dimensional snake robot-like, navigating a cluttered gas turbine for the purposes of inspection. Heuristic search-based approaches are effective planning strategies for common manipulation domains. However, their performance on high-dimensional systems is heavily reliant on the effectiveness of the action space and the heuristics [...]
Vision-Based Tactile Sensor Design using Physics Based Rendering
Abstract: Tactile sensing has seen a rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and human-robot interaction. However, the simulation of tactile sensors is still a challenge. Simulation is a critical tool in the development of robotic systems. In [...]
Kernel Density Decision Trees
Abstract We propose kernel density decision trees (KDDTs), a novel fuzzy decision tree (FDT) formalism based on kernel density estimation that improves the robustness of decision trees and ensembles and offers additional utility. FDTs mitigate the sensitivity of decision trees to uncertainty by representing uncertainty through fuzzy partitions. However, compared to conventional, crisp decision trees, [...]
Energy-based Joint Pose Estimation for 3D Reconstruction
Abstract: In this talk, I will describe a data-driven method for inferring camera poses given a sparse collection of images of an arbitrary object. This task is a core component of classic geometric pipelines such as structure-from-motion (SFM), and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction. [...]
NeRF for Robotics
Abstract: In this talk I'll describe how recent advances in neural rendering and novel view synthesis - namely NeRF - can be leveraged by robotic agents to improve performance in manipulation tasks. Specifically, I'll argue that NeRF can enable robotic policies to: (1) generalize to new viewpoints; (2) perceive specular and reflective surfaces in a [...]
Robust Reinforcement Learning via Genetic Curriculum
Abstract: Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While [...]
Mouth Haptics in VR using a Headset Ultrasound Phased Array
Abstract: This talk is the same one I will be presenting at the ACM CHI Conference on Human Factors in Computing Systems on May 2nd. Paper abstract: Today’s consumer virtual reality (VR) systems offer limited haptic feedback via vibration motors in handheld controllers. Rendering haptics to other parts of the body is an open challenge, [...]
TIGRIS: An Informed Sampling-based Algorithm for Informative Path Planning
Abstract: In this talk I will present our sampling-based approach to informative path planning that allows us to tackle the challenges of large and high-dimensional search spaces. This is done by performing informed sampling in the high-dimensional continuous space and incorporating potential information gain along edges in the reward estimation. This method rapidly generates a [...]
Trajectory Optimization for Thermally-Actuated Soft Planar Robot Limbs
Abstract: Practical use of robotic manipulators made from soft materials requires generating and executing complex motions. We present the first approach for generating trajectories of a thermally-actuated soft robotic manipulator. Based on simplified approximations of the soft arm and its antagonistic shape-memory alloy actuator coils, we justify a dynamics model of a discretized rigid manipulator [...]
Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
Abstract: Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g., NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known to be an under-constrained problem, and so methods rely on data-driven priors for reconstructing dynamic content. We replace these [...]
Combining vision-based tactile, proximity, and global sensing for robotic manipulation
Abstract: I will begin by describing our work on visual servoing a manipulator and localizing objects using a robot-mounted suite of vision and vision-based tactile sensors, our results, algorithms used, and lessons learned. We show that by collocating tactile, and global (e.g. an RGB(D) camera) sensors, our setup can perform better than using each type [...]
Design, Modeling and Control for a Tilt-rotor VTOL UAV in the Presence of Actuator Failure
Abstract: Providing both the vertical take-off and landing capabilities and the ability to fly long distances to aircraft opens the door to a wide range of new real-world aircraft applications while improving many existing applications. Tiltrotor vertical take-off and landing (VTOL) unmanned aerial vehicles (UAVs) are a better choice than fixed-wing and multirotor aircraft for [...]
Lessons Learned from Creating Low-Cost Dexterous Soft Robot Hands
Abstract: Soft robot hands have shown promising results when it comes to dexterous grasping and manipulation. Compared to their rigid counterparts, soft hands can be manufactured for a fraction of the cost and offer robustness to uncertainty due to their inherent compliance. Unfortunately, the design and fabrication of soft robot hands is still a time-consuming [...]
Modern Trajectory Forecasting Methods Lack Social Awareness
Abstract: We present a thorough evaluation and analysis of state-of-the-art (SOTA) human trajectory forecasting methods with respect to metrics for safe and socially-aware prediction, e.g., collision rate, in addition to traditional displacement metrics, e.g., average displacement error. First, we introduce a system for trajectory classification which is used to evaluate the strengths and weaknesses of [...]
Learning to perform dynamic and interactive tasks using structural and algorithmic priors
Abstract: Everyday human tasks such as picking up an object in one smooth motion, pushing a heavy door using the momentum of our bodies or pushing off a wall to quickly turn a corner involve complex dynamic interactions between the human and the environment, as well as switching dynamics when the robot makes and breaks [...]
Simple Shape Descriptors for Retinal Surface Estimation using a Laser-Aiming Beam
Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]
Affective Robot Behavior Improves Learning in a Sorting Game
Abstract: Nonverbal communication in the field of education can allow teachers to emotionally support their students and improve educational experience and performance. Robot nonverbal movements have been shown to improve both subjective experiences and task performance, and this work investigates whether affective robot behavior can improve human learning. This is tested using an online sorting [...]
Learning Strategies to Solve Real-World Physics Puzzles
Abstract: In this talk, I focus on efficient online learning for solving real-world physics puzzles. I discuss challenges associated with learning in this domain and how those challenges inform certain design decisions. In particular, learning from scratch in the real world would be difficult. I present a practical mixture of experts framework for learning strategies [...]
Forecasting from LiDAR via Future Object Detection
Abstract: Object detection and forecasting are fundamental components of embodied perception. These two problems, however, are largely studied in isolation by the community. In this paper, we propose an end-to-end approach for detection and motion forecasting based on raw sensor measurement as opposed to ground truth tracks. Instead of predicting the current frame locations and [...]
Safe control under input limits with neural CBF
Abstract: In theory, control barrier functions (CBFs) provide a convenient means to construct provably safe controllers. However, a typical problem is that the constructed controller will exceed input limits, and merely clipping the inputs will break all safety guarantees. To address this practical flaw, we consider synthesizing a CBF that will respect input limits. We [...]