PhD Speaking Qualifier
Analogy-Forming Transformers for Few-Shot 3D Parsing
Abstract: How do we build agents that can fast generalize to novel scenarios given only a single example? In this talk, I will present analogy-forming transformers, a semi-parametric model that segments 3D object scenes by retrieving related memories and predicting analogous part structures for the input. This enables a single neural network to continually learn [...]
Range-based Gaussian Process Maps for Mobile Exploration Robots
Abstract: Mobile robots exploring unknown, natural environments with limited communication must map their surroundings using onboard sensors. In this context, terrain mapping can rely on Gaussian process models to incorporate spatial correlations and provide uncertainty estimates when predicting ground height - however, these models fail to account for the oblique viewpoint of a sensor on [...]
Learning Exploration Strategies to Solve Real-World Marble Runs
Abstract: Tasks involving locally unstable or discontinuous dynamics (such as bifurcations and collisions) remain challenging in robotics, because small variations in the environment can have a significant impact on task outcomes. In this talk, we present a robot system that we developed to evaluate learning algorithms on real-world physical problem solving tasks which incorporate these [...]
Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery
Abstract: Underwater imagery often exhibits distorted coloration as a result of light-water interactions, which complicates the study of benthic environments in marine biology and geography. In this research, we propose an algorithm to restore the true color (albedo) in underwater imagery by jointly learning the effects of the medium and neural scene representations. Our approach [...]
Force-Torque Sensors – Calibration & Estimation
Abstract: Wrist force-torque sensors were among the first proprioception sensors to be developed when robotics emerged as a field. They are now a mature technology already used in structured industrial applications like sanding and drilling. While they provide essential feedback in many manipulation algorithms, they do not garner as much excitement as exteroception sensors like [...]
Optimized Tradeoffs for Differentially Private Majority Ensembling
Abstract: Inspired by the common subtask of ensembling or calibrating private models, we study the problem of computing an m*epsilon-differentially private majority of K epsilon-differentially private algorithms for m < K. We introduce a general framework to compute the private majority via Randomized Response (RRM) with a data-dependent noise function gamma that subsumes any non-trivial [...]
Incorporating Robustness into Learning-Based Aircraft Detection and Tracking Systems
Abstract: In the field of aviation, the Detect and Avoid (DAA) problem deals with incorporating collision avoidance capabilities into current autopilot navigation systems. In order to standardize DAA capabilities, ASTM has published performance requirements to define safe DAA operations of unmanned aircraft systems (UAS). However, the performance of DAA models are entirely dependent on the [...]
Differentiable Fluid-Structure Interaction for Robotics
Abstract: We present Aquarium, a differentiable fluid-structure interaction solver for robotics that offers stable simulation, accurately coupled fluid-robot physics in two dimensions, and full differentiability with respect to fluid and robot states and parameters. Aquarium achieves stable simulation with accurate flow physics by directly integrating over the incompressible Navier-Stokes equations using a fully implicit Crank-Nicolson [...]
An Effective Learning Framework for Active Perception and a Case Study on Liquid Property Estimation
Abstract: Active perception refers to a perception process where robot actions are taken to improve perception. To do this, the robot needs an observation model that knows what it will observe based on the actions it takes. However, existing approaches struggle to learn a good observation model since it needs to account for all possible [...]
Vision-based Proprioceptive and Tactile Sensing for Soft Robots
Abstract: Soft robotic manipulators present many unique advantages in difficult manipulation tasks. The inherent compliance of soft robots' constituent deformable material makes them safe and reliable in delicate tasks such as harvesting fruit and assisting in household work. To address challenges in proprioceptive and tactile sensing for soft robots, we present a family of vision-based [...]
Robot Learning for Assistive Dressing
Abstract: Robot-assisted dressing could benefit the lives of many people such as older adults and individuals with disabilities. In this talk, I will present two pieces of work that use robot learning for this assistive task. In the first half of the talk, I will present our work on developing a robot-assisted dressing system that [...]
Towards Robotic Tree Manipulation: Leveraging Graph Representations
Abstract: There is growing interest in automating agricultural tasks that require intricate and precise interaction with specialty crops, such as trees and vines. However, developing robotic solutions for crop manipulation remains a difficult challenge due to complexities involved in modeling their deformable behavior. In this study, we present a framework for learning the deformation behavior [...]
Tracking Any”Thing” in Videos
Abstract: Being able to track anything is one of the fundamental steps to parse and understand a video. In this talk, I will present two pieces of work that tackle this problem at different spatial granularities. In the first half of the talk, I will discuss tracking any video pixel or particle through time in [...]
Customizing Large-scale Text-to-Image Models
Abstract: Advancements in large-scale generative models represent a watershed moment. These models can generate a wide variety of objects and scenes with different styles and compositions. However, these models are trained on a fixed snapshot of available data and often contain copyrighted or private images. This assumption makes them lacking in two aspects – (a) [...]
How to Design Robotic Hands That Wield Tools
Abstract: Tool manipulation is an essential human skill. It extends our manipulation capability beyond the capability of the biological hand, and is a defining feature of many important jobs centered on physical interaction with the real world. Yet, wielding a tool is drastically different from generally grasping an object. The prime examples are pens and [...]
Learning Local Heuristics in Heuristic Search
Abstract: Motion planning is a fundamental problem in robotics; how can we move robots efficiently and safely? Motion planning can be solved using several paradigms with their own strengths and weaknesses. This talk dives into Heuristic Graph Search and its application to motion planning by converting it to a problem of finding a start-goal path [...]
Joint 2D and 3D Semi-Supervised Object Detection
Abstract: While numerous 3D detection works leverage the complementary relationship between RGB images and point clouds, developments in the broader framework of semi-supervised object recognition remain uninfluenced by multi-modal fusion. Current methods develop independent pipelines for 2D and 3D semi-supervised learning despite the availability of paired image and point cloud frames. Observing that the distinct [...]
Towards Agile Robotics: Creating Push-Off Skills for Dynamic Interactions
Abstract: Dynamic interactions play a fundamental role in human capabilities, enabling us to achieve a wide range of tasks such as moving heavy objects, manipulating our surroundings, and changing directions rapidly and safely. In contrast, most conventional robotic systems lack this level of agility and cannot perform dynamic interactions, limiting their potential in practical applications. [...]
Generative Evolutionary Search with Diffusion Models for Trajectory Optimization
Abstract: Diffusion models excel at modeling complex and multimodal trajectory distributions for decision-making and control. Reward-gradient guided denoising has been recently proposed to generate trajectories that maximize both a differentiable reward function and the likelihood under the data distribution captured by a diffusion model. Reward-gradient guided denoising requires a differentiable reward function fitted to both [...]
Tartancalib: Iterative Wide-Angle Lens Calibration
Abstract: Mobile vision systems greatly benefit from the large field-of-view enabled by wide-angle lenses. Accurate and robust intrinsic calibration is a critical prerequisite for leveraging this property. Calibrating wide-angle lenses with current state-of-the-art techniques yields poor results due to extreme distortion at the edge. In this work, we present TartanCalib, an accurate and robust method [...]
Zero-Shot Video Question Answering with Procedural Programs
Abstract: We propose to answer zero-shot questions about videos by generating short procedural programs that derive a final answer from solving a sequence of visual subtasks. We present Procedural Video Querying (ProViQ), which uses a large language model to generate such programs from an input question and an API of visual modules in the prompt, [...]
Robust Body Exposure (RoBE): A Graph-based Dynamics Modeling Approach to Manipulating Blankets over People
Abstract: Robotic caregivers could potentially improve the quality of life of many who require physical assistance. However, in order to assist individuals who are lying in bed, robots must be capable of dealing with a significant obstacle: the blanket or sheet that will almost always cover the person's body. We propose a method for targeted [...]
Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter
This talk has been postponed […]
Learning to Manipulate beyond Imitation
Abstract: Imitation learning has been a prevalent approach for teaching robots manipulation skills but still suffers from scalability and generalizability. In this talk, I'll argue for going beyond elementary behavioral imitation from human demonstrations. Instead, I'll present two key directions: 1) Creating Manipulation Controllers from Pre-Trained Representations, and 2) Representing Video Demonstrations with Parameterized Symbolic [...]
Leveraging Parallelism to Accelerate Quadratic Program Solvers for MPC
Abstract: Many problems in robotics can be formulated as quadratic programs (QPs). In particular, model-predictive control problems often involve repeatedly solving QPs at very high rates (up to kilohertz). However, while other areas of robotics like machine learning have achieved high performance by taking advantage of parallelism on modern computing hardware, state-of-the-art algorithms for solving [...]
Composing Generative and Discriminative Models for Better Generalization
Abstract: Computer Vision is Correspondence, correspondence, correspondence! Inspite of the singular definition of computer vision, we still have two broad categories of approaches in the literature. Generative Models, like Stable Diffusion, learn a correspondence between image and text modality, while learning a mapping from text to image. Discriminative Models, like CLIP, on the other hand [...]