PhD Speaking Qualifier
When to use CNNs for Inverse Problems in Vision
Abstract: Reconstruction tasks in computer vision aim fundamentally to recover an undetermined signal from a set of noisy measurements. Examples include super-resolution, image denoising, and non-rigid structure from motion\cite{Kong_2019}, all of which have seen recent advancements through deep learning. However, earlier work made extensive use of sparse signal reconstruction frameworks (e.g. convolutional sparse coding). While [...]
Tendon Driven Foam Hands
Abstract: There has been great progress in soft robot design, manufacture, and control in recent years, and soft robots are a tool of choice for safe and robust handling of objects in conditions of uncertainty. Still, dexterous in-hand manipulation using soft robots remains a challenge. This talk introduces a novel class of soft robots in [...]
Carnegie Mellon University
Towards a Good Representation For Reinforcement Learning
Abstract: Deep reinforcement learning has achieved many successes over the recent years. However, its high sample complexity and the difficulty in specifying a reward function have limited its application. In this talk, I will take a representation learning perspective towards these issues. Is it possible to map from the raw observation, potentially in high dimension, [...]
Carnegie Mellon University
Resource-constrained learning and inference for visual perception
Zoom Link Abstract Real-world applications usually require computer vision algorithms to meet certain resource constraints. In this talk, I will present evaluation methods and principled solutions for both cases of training and testing. First, I will talk about a formal setting for studying training under the non-asymptotic, resource-constrained regime, i.e., budgeted training. We analyze the [...]
Carnegie Mellon University
Planning and Execution using Inaccurate Models with Provable Guarantees
Zoom Link Abstract: Models used in modern planning problems to simulate outcomes of real world action executions are becoming increasingly complex, ranging from simulators that do physics-based reasoning to precomputed analytical motion primitives. However, robots operating in the real world often face situations not modeled by these models before execution. This imperfect modeling can lead [...]
Carnegie Mellon University
The Effect of Locomotion Configuration on Discrete Obstacle Traversal for a Small Tracked Vehicle
Zoom Link Abstract: As mobile robots are being designed for increasingly rugged and unknown terrain, mechanical reconfigurability presents one possibility for improving vehicle efficiency and mobility. To validate this idea, we created an 18.5-kg modular tracked vehicle with adjustable track tension, track width, track length, and sprocket diameter. In this talk, I will explain the [...]
Carnegie Mellon University
Task-Driven Modular Networks for Zero-Shot Compositional Learning
Zoom Link Abstract: One of the hallmarks of human intelligence is the ability to compose learned knowledge into novel concepts which can be recognized without a single training example. In contrast, current state-of-the-art methods require hundreds of training examples for each possible category to build reliable and accurate classifiers. To alleviate this striking difference in [...]
Image to LiDAR Map Registration using Late Feature Projection
Zoom Link Abstract: Accurate localization is essential for autonomous operation in many problem domains. This is most often performed by comparing LiDAR scans collected in real-time to a HD point cloud based map. While this enables centimeter-level accuracy, it depends on an expensive LiDAR sensor at run time. Recently, efforts have been underway to reduce [...]
Carnegie Mellon University
A Theory of Fermat Paths for Non-line-of-sight Shape Reconstruction
Zoom Link Abstract: Traditionally, computer vision systems and algorithms, such as stereo vision, and shape from shading, have been developed to mimic human vision. As a consequence, a lot of these systems operate under constraints that we take for granted in human vision. An example of such a constraint is that the scene of interest [...]
Learning Contextual Actions for Heuristic Search-Based Motion Planning
Zoom Link Abstract: Heuristic search-based motion planning can be computationally costly in large state and action spaces. In this work we explore the use of generative models to learn contextual actions for successor generation in heuristic search. We focus on cases where the robot operates in similar environments, i.e. environments drawn from some underlying distribution. [...]
Carnegie Mellon University
Interactive Weak Supervision – Learning Useful Heuristics for Data Labeling
Zoom Link Abstract: Obtaining large annotated datasets is critical for training successful machine learning models and it is frequently a bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics. This process can scale to large amounts of data and [...]
Carnegie Mellon University
Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap
Zoom Link Abstract: Training robotic policies in simulation suffers from the sim-to-real gap, as simulated dynamics can be different from real-world dynamics. Past works tackled this problem through domain randomization and online system-identification. The former is sensitive to the manually-specified training distribution of dynamics parameters and can result in behaviors that are overly conservative. The [...]
Interferometric light transmission probing with coded mutual intensity
Zoom Link Abstract: We introduce a new interferometric imaging methodology that we term interferometry with coded mutual intensity, which allows selectively imaging photon paths based on attributes such as their length and endpoints. At the core of our methodology is a new technical result that shows that manipulating the spatial coherence properties of the light [...]
Sparse Spatial Hashing for Dense 3D Reconstruction
Abstract: Real-world 3D data is locally dense but globally sparse. Therefore, efficient sparse data structures are an essential component of dense 3D perception for computer vision and robotics. We manifest the power of spatial hashing by two typical tasks: dense scene reconstruction and global registration. In the first task, we accelerate volumetric integration and surface [...]
Carnegie Mellon University
3D Multi-Object Tracking for Autonomous Driving
Abstract: 3D multi-object tracking (MOT) is a key component of a perception system for autonomous driving. Due to recent progress in 3D object detection in the context of autonomous driving, recent work in 3D MOT primarily focuses on online tracking with the use of a tracking-by-detection pipeline. In this talk, we introduce a new 3D [...]
Carnegie Mellon University
Ergodic Trajectory Optimization for Information Gathering
Abstract: Planetary robots currently rely on significant guidance from expert human operators. Science autonomy adds algorithms and methods for autonomous scientific exploration to improve efficiency of discovery and overcome limited communication bandwidth and delay bottlenecks. This research focuses on planning trajectories for information gathering and choosing sampling locations that have the most informative samples. We [...]
Carnegie Mellon University
Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis
Abstract: Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the [...]
Studying the Evolution of Pedestrian Group Space
Abstract: Imagine walking along a busy sidewalk, do you track the movement of every single individual? Or do you simply group pedestrians with similar moving patterns and then track the movement of this group? Grouping is a common behavior in pedestrian navigation and it is typically inappropriate for a robot to cut through the social [...]
Carnegie Mellon University
Soft actuators by electrochemical oxidation of liquid metal surfaces
Abstract: Soft robotic systems typically operate through the use of soft actuators constructed from highly deformable materials or liquids. Because of their intrinsic compliance, these actuators can achieve elastic resilience and adaptability similar to their biological counterparts. One challenge with engineering these artificial muscles is the selection of soft materials and activation methods while maintaining [...]
A Graph-Based Method for Joint Instance Segmentation of Point Clouds and Image Sequences
Abstract: While learning-based semantic instance segmentation methods have achieved impressive progress, their use is limited in robotics applications due to reliance on expensive training data annotations and assumptions of single sensor modality or known object classes. We propose a novel graph-based instance segmentation approach that combines information from a 2D image sequence and a 3D [...]
Continual Reinforcement Learning using Self-Activating Neural Ensembles
Abstract: The ability for an agent to continuously learn new skills without catastrophically forgetting existing knowledge is of critical importance for the development of generally intelligent agents. Most methods devised to address this problem depend heavily on well-defined task boundaries which simplify the problem considerably. Our task-agnostic method, Self-Activating Neural Ensembles (SANE), uses a hierarchical [...]
Carnegie Mellon University
Unsupervised 2D-3D Lifting with Deep Structure Priors
Abstract: Learning to estimate non-rigid 3D structures from 2D imaged observations is bottle-necked by the availability of abundant 3D annotated data. Learning methods that reduce the amount of required annotation is of high practical value. In this regard, Non-Rigid Structure from Motion (NRSfM) methods offer the opportunity to infer 3D structures solely from 2D annotations. [...]
Model Adaptation for Compliant Parallel Robot with Nonstationary Dynamics
Abstract: Soft robots can be constructed with few parts and from a wide variety of materials. This makes them a potentially appealing choice for applications where there are resource constraints on system fabrication. However, soft robot dynamics are difficult to accurately model analytically, due to a multiphysics coupling between shape, forces, temperature, and history of [...]
Adaptive Safety Margins for Safe Replanning under Time-Varying Disturbances
Abstract: Safe real-time navigation is a considerable challenge because engineers often need to work with uncertain vehicle dynamics, variable external disturbances, and imperfect controllers. A common strategy used to address safety is to employ hand-defined margins for obstacle inflation. However, arbitrary static margins often fail in more dynamic scenarios, and using worst-case assumptions proves to [...]
HyperDynamics: Generating Expert Dynamics Models by Observation
Abstract: We propose HyperDynamics, a framework that conditions on an agent’s interactions with the environment and optionally its visual observations, and generates the parameters of neural dynamics models based on inferred properties of the dynamical system. Physical and visual properties of the environment that are not part of the low-dimensional state yet affect its temporal [...]
Direct Fitting of Mixture Models
Abstract: There exist many choices of 3D shape representation. Some recent work has advocated for the use of Gaussian Mixture Models as a compact representation for 3D shapes and scenes. These models are typically fit to point clouds, even when the shapes were obtained as 3D meshes. Here we present a formulation for fitting Gaussian [...]
Carnegie Mellon University
Terrain Perception using Structured Light for Micro-Rovers
Abstract: With continuing advancement in technology, the future of planetary exploration is likely to be dominated by robotic missions. Yet rovers capable of science investigations are slow and bulky with very limited computing which prohibits demonstrating full autonomy. These rovers are also risk averse due to their huge mission cost. However there is a new [...]
Carnegie Mellon University
Analysis of Deadlock in Multirobot Systems
Abstract: Collision avoidance for multirobot systems is a well-studied problem. Recently, control barrier functions (CBFs) have been proposed for synthesizing controllers that guarantee safety while simultaneously encouraging goal stabilization for multiple robots. However, it has been noted that reactive control synthesis methods (such as CBFs) are prone to deadlock, an equilibrium of system dynamics that [...]
Interleaving Graph Search and Trajectory Optimization for Aggressive Quadrotor Flight
Abstract: Quadrotors can achieve aggressive flight by tracking complex maneuvers and rapidly changing directions. Planning for aggressive flight with trajectory optimization could be incredibly fast, even in higher dimensions, and can account for dynamics of the quadrotor, however, only provides a locally optimal solution. On the other hand, planning with discrete graph search can handle [...]
See, Hear, Explore: Curiosity via Audio-Visual Association
Abstract: Exploration is one of the core challenges in reinforcement learning. A common formulation of curiosity-driven exploration uses the difference between the real future and the future predicted by a learned model. However, predicting the future is an inherently difficult task which can be ill-posed in the face of stochasticity. In this work, we introduce [...]
MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
Abstract: We present a method to capture temporally coherent dynamic clothing deformation from a monocular RGB video input. In contrast to the existing literature, our method does not require a pre-scanned personalized mesh template, and thus can be applied to in-the-wild videos. To constrain the output to a valid deformation space, we build statistical deformation [...]
Policy Decomposition : Approximate Optimal Control with Suboptimality Estimates
Abstract: Owing to the curse of dimensionality, numerically computing global policies to optimal control problems for complex dynamical systems quickly becomes intractable. In consequence, a number of approximation methods have been developed. However, none of the current methods can quantify by how much the resulting control underperforms the elusive globally optimal solution. We propose Policy [...]
Inverse Reinforcement Learning with Explicit Policy Estimates
Abstract: Various methods for solving the inverse reinforcement learning (IRL) problem have been developed independently in machine learning and economics. In particular, the method of Maximum Causal Entropy IRL is based on the perspective of entropy maximization, while related advances in the field of economics instead assume the existence of unobserved action shocks to explain [...]
Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation
Abstract: To perform manipulation tasks in the real world, robots need to operate on objects with various shapes, sizes and without access to geometric models. It is often infeasible to train monolithic neural network policies across such large variance in object properties. Towards this generalization challenge, we propose task-axis controllers, which are defined relative to [...]
Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policies
Abstract: Real-world environments, such as homes, hospitals, and restaurants, often contain many objects that a robot could possibly manipulate. However, for a given manipulation task, only a small number of objects and object properties may actually be relevant. This talk presents CREST (Causal Reasoning for Efficient Structure Transfer), our approach to learn the relevant state [...]
Grasping Transparent, Specular, and Deformable Objects
Abstract: A large body of research exists on grasping for objects with ideal properties like Lambertian reflectance and rigidity. On the other hand, real-world environments contain many objects for which such properties do not hold, such as transparent, specular, and deformable objects. For such objects, new approaches are required to achieve the same level of [...]
PoseIt: A Visual-Tactile Dataset of Holding Poses for Grasp Stability Analysis
Abstract: When humans grasp objects in the real world, we often move our arm to hold the object in a different pose where we can use it. In contrast, typical lab settings only study the stability of the grasp immediately after lifting, without any subsequent re-positioning of the arm. However, an object’s stability could vary [...]
Planning to Minimize Human and Robot Efforts Over Tasks
Abstract: It is not feasible to pre-program robots a priori for every possible task they may encounter in unstructured domains. Upon encountering a task that a robot can't solve, one common strategy is to teach it new skills via demonstrations. However, demonstrating a task can often be more cumbersome than performing the task directly. This [...]
Fine-Tuning Offline Reinforcement Learning with Model-Based Policy Optimization
Abstract: In offline reinforcement learning (RL), we attempt to learn a control policy from a fixed dataset of environment interactions. This setting has the potential benefit of allowing us to learn effective policies without needing to collect additional interactive data, which can be expensive or dangerous in real-world systems. However, traditional off-policy RL methods tend [...]
Modeling Coupled Human-Robot Motion for Provable Safety
Abstract: Guide robots that help users who are blind or low vision navigate through crowds and complex environments show promise for improving accessibility in public spaces. These robots must provide real-time safety guarantees for the users, which requires accurate modeling of their behavior in the context of closely coupled human-robot motion. This model must also [...]
Diminished Reality for Close Quarters Robotic Telemanipulation
Abstract: In robot telemanipulation tasks, the robot itself can sometimes occlude a target object from the user's view. We investigate the potential of diminished reality to address this problem. Our method uses an optical see-through head-mounted display to create a diminished reality illusion that the robot is transparent, allowing users to see occluded areas behind [...]
Learning Compositional Radiance Fields of Dynamic Human Heads
Meeting ID: 942 4671 0665 Passcode: jkhzoom Abstract: Photorealistic rendering of dynamic humans is an important capability for telepresence systems. Recently, neural rendering methods have been developed to create high-fidelity models of humans and objects. Some of these methods do not produce results with high-enough fidelity for driveable human models (Neural Volumes) whereas others have [...]
An Experimental Design Perspective on Model-Based Reinforcement Learning
Abstract: In many practical applications of RL, it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, computing the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or dollars of [...]
Learning Model Preconditions for Planning with Multiple Models
Abstract: Different models can provide differing levels of fidelity when a robot is planning. Analytical models are often fast to evaluate but only work in limited ranges of conditions. Meanwhile, physics simulators are effective at modeling complex interactions between objects but are typically more computationally expensive. Learning when to switch between the various models can [...]
Reconstructing common objects to interact with
Abstract: We humans are able to understand 3D shapes of common daily objects and interact with them from a wide range of categories. We understand cups are usually cylinder-like and we can easily predict the shape of one particular cup, both in isolation or even when it is held by a human. We aim to [...]
A causal framework to diagnose and fix issues with doors
Abstract: Many animals, such as ravens, (and a fortiori humans) exhibit a great deal of physical intelligence that allows them to solve complex multi-step physical puzzles. This ability indicates an understanding or a faculty to represent causality and mechanisms, understand when something goes wrong, and figure out how to deal with it. As a step [...]
Designing Whisker Sensors to Detect Multiple Mechanical Stimuli for Robotic Applications
Abstract: Many mammals, such as rats and seals, use their whiskers as versatile mechanical sensors to gain precise information about their surroundings. Whisker-inspired sensors on robotic platforms have shown their potential benefit, improving applications ranging from drone navigation to texture mapping. Despite this, there is a gap between the engineered sensors and many of the [...]
Towards Complex Robot Motions with Reinforcement Learning
Abstract: Reinforcement learning has shown to be a powerful tool for decision-making problems. In this talk, we present the opportunities and challenges of enabling increasingly complex robot behavior with reinforcement learning. First, we present a system that combines reinforcement learning and extrinsic dexterity to solve a novel task of “occluded grasping”. To reach an occluded [...]
Search-based Path Planning for a High Dimensional Manipulator in Cluttered Environments Using Optimization-based Primitives
Abstract: In this work we tackle the path planning problem for a 21-dimensional snake robot-like, navigating a cluttered gas turbine for the purposes of inspection. Heuristic search-based approaches are effective planning strategies for common manipulation domains. However, their performance on high-dimensional systems is heavily reliant on the effectiveness of the action space and the heuristics [...]
Vision-Based Tactile Sensor Design using Physics Based Rendering
Abstract: Tactile sensing has seen a rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and human-robot interaction. However, the simulation of tactile sensors is still a challenge. Simulation is a critical tool in the development of robotic systems. In [...]
Kernel Density Decision Trees
Abstract We propose kernel density decision trees (KDDTs), a novel fuzzy decision tree (FDT) formalism based on kernel density estimation that improves the robustness of decision trees and ensembles and offers additional utility. FDTs mitigate the sensitivity of decision trees to uncertainty by representing uncertainty through fuzzy partitions. However, compared to conventional, crisp decision trees, [...]
Energy-based Joint Pose Estimation for 3D Reconstruction
Abstract: In this talk, I will describe a data-driven method for inferring camera poses given a sparse collection of images of an arbitrary object. This task is a core component of classic geometric pipelines such as structure-from-motion (SFM), and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction. [...]
NeRF for Robotics
Abstract: In this talk I'll describe how recent advances in neural rendering and novel view synthesis - namely NeRF - can be leveraged by robotic agents to improve performance in manipulation tasks. Specifically, I'll argue that NeRF can enable robotic policies to: (1) generalize to new viewpoints; (2) perceive specular and reflective surfaces in a [...]
Robust Reinforcement Learning via Genetic Curriculum
Abstract: Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While [...]
Mouth Haptics in VR using a Headset Ultrasound Phased Array
Abstract: This talk is the same one I will be presenting at the ACM CHI Conference on Human Factors in Computing Systems on May 2nd. Paper abstract: Today’s consumer virtual reality (VR) systems offer limited haptic feedback via vibration motors in handheld controllers. Rendering haptics to other parts of the body is an open challenge, [...]
TIGRIS: An Informed Sampling-based Algorithm for Informative Path Planning
Abstract: In this talk I will present our sampling-based approach to informative path planning that allows us to tackle the challenges of large and high-dimensional search spaces. This is done by performing informed sampling in the high-dimensional continuous space and incorporating potential information gain along edges in the reward estimation. This method rapidly generates a [...]
Trajectory Optimization for Thermally-Actuated Soft Planar Robot Limbs
Abstract: Practical use of robotic manipulators made from soft materials requires generating and executing complex motions. We present the first approach for generating trajectories of a thermally-actuated soft robotic manipulator. Based on simplified approximations of the soft arm and its antagonistic shape-memory alloy actuator coils, we justify a dynamics model of a discretized rigid manipulator [...]
Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
Abstract: Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g., NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known to be an under-constrained problem, and so methods rely on data-driven priors for reconstructing dynamic content. We replace these [...]
Combining vision-based tactile, proximity, and global sensing for robotic manipulation
Abstract: I will begin by describing our work on visual servoing a manipulator and localizing objects using a robot-mounted suite of vision and vision-based tactile sensors, our results, algorithms used, and lessons learned. We show that by collocating tactile, and global (e.g. an RGB(D) camera) sensors, our setup can perform better than using each type [...]
Design, Modeling and Control for a Tilt-rotor VTOL UAV in the Presence of Actuator Failure
Abstract: Providing both the vertical take-off and landing capabilities and the ability to fly long distances to aircraft opens the door to a wide range of new real-world aircraft applications while improving many existing applications. Tiltrotor vertical take-off and landing (VTOL) unmanned aerial vehicles (UAVs) are a better choice than fixed-wing and multirotor aircraft for [...]
Lessons Learned from Creating Low-Cost Dexterous Soft Robot Hands
Abstract: Soft robot hands have shown promising results when it comes to dexterous grasping and manipulation. Compared to their rigid counterparts, soft hands can be manufactured for a fraction of the cost and offer robustness to uncertainty due to their inherent compliance. Unfortunately, the design and fabrication of soft robot hands is still a time-consuming [...]
Modern Trajectory Forecasting Methods Lack Social Awareness
Abstract: We present a thorough evaluation and analysis of state-of-the-art (SOTA) human trajectory forecasting methods with respect to metrics for safe and socially-aware prediction, e.g., collision rate, in addition to traditional displacement metrics, e.g., average displacement error. First, we introduce a system for trajectory classification which is used to evaluate the strengths and weaknesses of [...]
Learning to perform dynamic and interactive tasks using structural and algorithmic priors
Abstract: Everyday human tasks such as picking up an object in one smooth motion, pushing a heavy door using the momentum of our bodies or pushing off a wall to quickly turn a corner involve complex dynamic interactions between the human and the environment, as well as switching dynamics when the robot makes and breaks [...]
Simple Shape Descriptors for Retinal Surface Estimation using a Laser-Aiming Beam
Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]
Affective Robot Behavior Improves Learning in a Sorting Game
Abstract: Nonverbal communication in the field of education can allow teachers to emotionally support their students and improve educational experience and performance. Robot nonverbal movements have been shown to improve both subjective experiences and task performance, and this work investigates whether affective robot behavior can improve human learning. This is tested using an online sorting [...]
Learning Strategies to Solve Real-World Physics Puzzles
Abstract: In this talk, I focus on efficient online learning for solving real-world physics puzzles. I discuss challenges associated with learning in this domain and how those challenges inform certain design decisions. In particular, learning from scratch in the real world would be difficult. I present a practical mixture of experts framework for learning strategies [...]
Forecasting from LiDAR via Future Object Detection
Abstract: Object detection and forecasting are fundamental components of embodied perception. These two problems, however, are largely studied in isolation by the community. In this paper, we propose an end-to-end approach for detection and motion forecasting based on raw sensor measurement as opposed to ground truth tracks. Instead of predicting the current frame locations and [...]
Safe control under input limits with neural CBF
Abstract: In theory, control barrier functions (CBFs) provide a convenient means to construct provably safe controllers. However, a typical problem is that the constructed controller will exceed input limits, and merely clipping the inputs will break all safety guarantees. To address this practical flaw, we consider synthesizing a CBF that will respect input limits. We [...]
Thermal Management Considerations For Lunar Polar Micro-Rovers
Meeting ID: 940 0396 4889 Passcode: 906118 Abstract: This research addresses the significant and unprecedented challenge of thermal regulation for lunar polar micro-rovers. These are distinct from priors by way of very small size, mass, and power, but particularly for the extremes of ambient environment in which they must operate. On the lunar poles, rovers experience temperatures [...]
An Extension to Model Predictive Path Integral Control and Modeling Considerations for Off-road Autonomous Driving in Complex Environment
Abstract: The ability to traverse complex environments and terrains is critical to autonomously driving off-road in a fast and safe manner. Challenges such as terrain navigation and vehicle rollover prevention become imperative due to the off-road vehicle configuration and the operating environment itself. This talk will introduce some of these challenges and the different tools [...]
Human-to-Robot Imitation in the Wild
Abstract: In this talk, I approach the problem of learning by watching humans in the wild. While traditional approaches in Imitation and Reinforcement Learning are promising for learning in the real world, they are either sample inefficient or are constrained to lab settings. Meanwhile, there has been a lot of success in processing passive, unstructured human [...]
Differentiable Collision Detection
Abstract: Collision detection between objects is critical for simulation, control, and learning for robotic systems. However, existing collision detection routines are inherently non-differentiable, limiting their applications in gradient-based optimization tools. In this talk, I present DCOL: a fast and fully differentiable collision-detection framework that reasons about collisions between a set of composable and highly expressive [...]
On Interaction, Imitation, and Causation
Abstract: A standard critique of machine learning models (especially neural networks) is that they pick up on spurious correlations rather than causal relationships and are therefore brittle in the face of distribution shift. Solving this problem in full generality is impossible (i.e. there might be no good way to distinguish between the two). However, if [...]
Solving Constraint Tasks with Memory-Based Learning
Abstract: In constraint tasks, the current task state heavily limits what actions are available to an agent. Mechanical constraints exist in many common tasks such as construction, disassembly, and rearrangement and task space constraints exist in an even broader range of tasks. Deep reinforcement learning algorithms have typically struggled with constraint tasks for two main [...]
Head-Worn Assistive Teleoperation of Mobile Manipulators
Abstract: Mobile manipulators in the home can provide increased autonomy to individuals with severe motor impairments, who often cannot complete activities of daily living (ADLs) without the help of a caregiver. Teleoperation of an assistive mobile manipulator could enable an individual with motor impairments to independently perform self-care and household tasks, yet limited motor function [...]
Text Classification with Class Descriptions Only
Abstract: In this work, we introduce KeyClass, a weakly-supervised text classification framework that learns from class-label descriptions only, without the need to use any human-labeled documents. It leverages the linguistic domain knowledge stored within pre-trained language models and data programming to automatically label documents. We demonstrate its efficacy and flexibility by comparing it to state-of-the-art [...]
Multi-Object Tracking in the Crowd
Abstract: In this talk, I will focus on the problem of multi-object tracking in crowded scenes. Tracking within crowds is particularly challenging due to heavy occlusion and frequent crossover between tracking targets. The problem becomes more difficult when we only have noisy bounding boxes due to background and neighboring objects. Existing tracking methods try to [...]
Magnification-invariant retinal distance estimation using a laser aiming beam
Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]
Bridging Humans and Generative Models
Abstract: Deep generative models make visual content creation more accessible to novice and professional users alike by automating the synthesis of diverse, realistic content based on a collected dataset. People often use generative models as data-driven sources, making it challenging to personalize a model easily. Currently, personalizing a model requires careful data curation, which is [...]
Impulse considerations for reasoning about intermittent contacts
Abstract: Many of our interactions with the environment involve making and breaking contacts. However, it is not always obvious how one should reason about these intermittent contacts (sequence, timings, locations) in an online and adaptive way. This is particularly relevant in gait generation for legged locomotion control, where it is standard to simply predefine and [...]
Robust Incremental Smoothing and Mapping
Abstract: In this work we present a method for robust optimization for online incremental Simultaneous Localization and Mapping (SLAM). Due to the NP-Hardness of data association in the presence of perceptual aliasing, tractable (approximate) approaches to data association will produce erroneous measurements. We require SLAM back-ends that can converge to accurate solutions in the presence [...]
Robotic Interestingness via Human-Informed Few-Shot Object Detection
Abstract: Interestingness recognition is crucial for decision making in autonomous exploration for mobile robots. Previous methods proposed an unsupervised online learning approach that can adapt to environments and detect interesting scenes quickly, but lack the ability to adapt to human-informed interesting objects. To solve this problem, we introduce a human-interactive framework, AirInteraction, that can detect [...]
FRIDA: Supporting Artistic Communication in Real-World Image Synthesis Through Diverse Input Modalities
Abstract: FRIDA, a Framework and Robotics Initiative for Developing Arts, is a robot painting system designed to translate an artist's high-level intentions into real world paintings. FRIDA can paint from combinations of input images, text, style examples, sounds, and sketches. Planning is performed in a differentiable, simulated environment created using real data from the robot [...]
Robust and Context-Aware Real-Time Collaborative Robot Handling with Dynamic Gesture Commands
Abstract: Real-time collaborative robot (cobot) handling is a task where the cobot maneuvers an object under human dynamic gesture commands. Enabling dynamic gesture commands is useful when the human needs to avoid direct contact with the robot or the object handled by the robot. However, the key challenge lies in the heterogeneity in human behaviors [...]
Dynamic Route Guidance in Vehicle Networks by Simulating Future Traffic Patterns
Abstract: Roadway congestion leads to wasted time and money and environmental damage. Since adding more roadway capacity is often not possible in urban environments, it is becoming more important to use existing road networks more efficiently. Toward this goal, recent research in real-time, schedule-driven intersection control has shown an ability to significantly reduce the delays [...]
Controllable Visual-Tactile Synthesis
Abstract: Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual Try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. The main challenges for multi-modal synthesis lie in the significant scale discrepancy between vision [...]
Perceiving Particles Inside a Container using Dynamic Touch Sensing
Abstract: Dynamic touch sensing has shown potential for multiple tasks. In this talk, I will present how we utilize dynamic touch sensing to perceive particles inside a container with two tasks: classification of the particles inside a container and property estimation of the particles inside a container. First, we try to recognize what is inside [...]
Examining the Role of Adaptation in Human-Robot Collaboration
Abstract: Human and AI partners increasingly need to work together to perform tasks as a team. In order to act effectively as teammates, collaborative AI should reason about how their behaviors interplay with the strategies and skills of human team members as they coordinate on achieving joint goals. This talk will discuss a formalism for [...]
A Multi-view Synthetic and Real-world Human Activity Recognition Dataset
Abstract: Advancements in Human Activity Recognition (HAR) partially relies on the creation of datasets that cover a broad range of activities under various conditions. Unfortunately, obtaining and labeling datasets containing human activity is complex, laborious, and costly. One way to mitigate these difficulties with sufficient generality to provide robust activity recognition on unseen data is [...]
Dense 3D Representation Learning for Geometric Reasoning in Manipulation Tasks
Abstract: When solving a manipulation task like "put away the groceries" in real environments, robots must understand what *can* happen in these environments, as well as what *should* happen in order to accomplish the task. This knowledge can enable downstream robot policies to directly reason about which actions they should execute, and rule out behaviors [...]
Learning novel objects during robot exploration via human-informed few-shot detection
Abstract: Autonomous mobile robots exploring in unfamiliar environments often need to detect target objects during exploration. Most prevalent approach is to use conventional object detection models, by training the object detector on large abundant image-annotation dataset, with a fixed and predefined categories of objects, and in advance of robot deployment. However, it lacks the capability [...]
Continually Improving Robots
Abstract: General purpose robots should be able to perform arbitrary manipulation tasks, and get better at performing new ones as they obtain more experience. The current paradigm in robot learning involves training a policy, in simulation or directly in the real world, with engineered rewards or demonstrations. However, for robots that need to keep learning [...]
3D-aware Conditional Image Synthesis
Abstract: We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available posed [...]
Robotic Climbing for Extreme Terrain Exploration
Abstract: Climbing robots can investigate scientifically valuable sites that are inaccessible to conventional rovers due to steep terrain features. Robots equipped with microspine grippers are particularly well-suited to ascending rocky cliff faces, but existing designs are either large and slow, or limited to relatively flat surfaces such as buildings. We have developed a novel free-climbing [...]
Multi-Objective Ergodic Search for Dynamic Information Maps
Abstract: Robotic explorers are essential tools for gathering information about regions that are inaccessible to humans. For applications like planetary exploration or search and rescue, robots use prior knowledge about the area to guide their search. Ergodic search methods find trajectories that effectively balance exploring unknown regions and exploiting prior information. In many search based [...]
Observing Assistance Preferences via User-controlled Arbitration in Shared Control
Abstract: What factors influence people’s preferences for robot assistance during human-robot collaboration tasks? Answering this question can help roboticists formalize definitions of assistance that lead to higher user satisfaction and increased user acceptance of assistive technology. Often in human robot collaboration literature, we see assistance paradigms that aim to optimize task success metrics and/or measures [...]
Safely Influencing Humans in Human-Robot Interaction
Abstract: Robots are becoming more common in industrial manufacturing because of their speed and precision on repetitive tasks, but they lack the flexibility of human collaborators. In order to take advantage of both humans’ and robots’ abilities, we investigate how to improve the efficiency of human-robot collaborations by making sure that robots both 1. stay [...]
Inductive Biases for Learning Long-Horizon Manipulation Skills
Abstract: Enabling robots to execute temporally extended sequences of behaviors is a challenging problem for learned systems, due to the difficulty of learning both high-level task information and low-level control. In this talk, I will discuss three approaches that we have developed to address this problem. Each of these approaches centers on an inductive bias [...]
Analogy-Forming Transformers for Few-Shot 3D Parsing
Abstract: How do we build agents that can fast generalize to novel scenarios given only a single example? In this talk, I will present analogy-forming transformers, a semi-parametric model that segments 3D object scenes by retrieving related memories and predicting analogous part structures for the input. This enables a single neural network to continually learn [...]
Range-based Gaussian Process Maps for Mobile Exploration Robots
Abstract: Mobile robots exploring unknown, natural environments with limited communication must map their surroundings using onboard sensors. In this context, terrain mapping can rely on Gaussian process models to incorporate spatial correlations and provide uncertainty estimates when predicting ground height - however, these models fail to account for the oblique viewpoint of a sensor on [...]
Learning Exploration Strategies to Solve Real-World Marble Runs
Abstract: Tasks involving locally unstable or discontinuous dynamics (such as bifurcations and collisions) remain challenging in robotics, because small variations in the environment can have a significant impact on task outcomes. In this talk, we present a robot system that we developed to evaluate learning algorithms on real-world physical problem solving tasks which incorporate these [...]
Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery
Abstract: Underwater imagery often exhibits distorted coloration as a result of light-water interactions, which complicates the study of benthic environments in marine biology and geography. In this research, we propose an algorithm to restore the true color (albedo) in underwater imagery by jointly learning the effects of the medium and neural scene representations. Our approach [...]