Toward an ImageNet Moment for Synthetic Data
Abstract: Data, especially large-scale labeled data, has been a critical driver of progress in computer vision. However, many important tasks remain starved of high-quality data. Synthetic data from computer graphics is a promising solution to this challenge, but still remains in limited use. This talk will present our work on Infinigen, a procedural synthetic data [...]
Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
Abstract: We show that imitating shortest-path planners in simulation produces Stretch RE-1 robotic agents that, given language instructions, can proficiently navigate, explore, and manipulate objects in both simulation and in the real world using only RGB sensors (no depth maps or GPS coordinates). This surprising result is enabled by our end-to-end, transformer-based, SPOC architecture, powerful [...]
Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter
This talk has been postponed […]
Towards diverse zero-shot manipulation via actualizing visual plans
Abstract: In this thesis, we seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation — interacting with unseen objects in novel scenes without test-time adaptation. Robots that can be reliably deployed out-of-the-box in new scenarios have the potential for helping humans in everyday tasks. Not requiring any test-time training through demonstrations or [...]
Deep Learning for Sensors: Development to Deployment
Abstract: Robots rely heavily on sensing to reason about physical interactions, and recent advancements in rapid prototyping, MEMS sensing, and machine learning have led to a plethora of sensing alternatives. However, few of these sensors have gained widespread use among roboticists. This thesis proposes a framework for incorporating sensors into a robot learning paradigm, from [...]
Offline Learning for Stochastic Multi-Agent Planning in Autonomous Driving
Abstract: Fully autonomous vehicles have the potential to greatly reduce vehicular accidents and revolutionize how people travel and how we transport goods. Many of the major challenges for autonomous driving systems emerge from the numerous traffic situations that require complex interactions with other agents. For the foreseeable future, autonomous vehicles will have to share the [...]
Teruko Yata Memorial Lecture
Human-Centric Robots and How Learning Enables Generality Abstract: Humans have dreamt of robot helpers forever. What's new is that this dream is becoming real. New developments in AI, building on foundations of hardware and passive dynamics, enable vastly improved generality. Robots can step out of highly structured environments and become more human-centric: operating in human [...]
2024 Robotics Institute National Robotics Week Celebration Tours and Demos
April 12 1:00 - 4:00 pm: PUBLIC SPACE ROBOTS Open to the public TANK the roboceptionist Newell-Simon Hall 3rd floor entry area Meet Marion (Tank) LeFleur, Newell-Simon’s Roboceptionist. He’ll be glad to see you! The goal of the project is to produce a robot helpmate that is useful, exhibits social competence, and remains compelling to [...]
Creating robust deep learning models involves effectively managing nuisance variables
Abstract: Over the past decade, we have witnessed significant advances in capabilities of deep neural network models in vision and machine learning. However, issues related to bias, discrimination, and fairness in general, have received a great deal of negative attention (e.g., mistakes in surveillance and animal-human confusion of vision models). But bias in AI models [...]
Transfer Learning via Temporal Contrastive Learning Inbox
Abstract: This thesis introduces a novel transfer learning framework for deep reinforcement learning. The approach automatically combines goal-conditioned policies with temporal contrastive learning to discover meaningful sub-goals. The approach involves pre-training a goal-conditioned agent, finetuning it on the target domain, and using contrastive learning to construct a planning graph that guides the agent via sub-goals. Experiments [...]
Towards Influence-Aware Safe Human-Robot Interaction
Abstract: In recent years, we have seen through recommender systems on social media how influential (and potentially harmful) algorithms can be in our lives, sometimes creating polarization and conspiracies that lead to unsafe behavior. Now that robots are also growing more common in the real world, we must be very careful to ensure that they [...]
Learning to Manipulate beyond Imitation
Abstract: Imitation learning has been a prevalent approach for teaching robots manipulation skills but still suffers from scalability and generalizability. In this talk, I'll argue for going beyond elementary behavioral imitation from human demonstrations. Instead, I'll present two key directions: 1) Creating Manipulation Controllers from Pre-Trained Representations, and 2) Representing Video Demonstrations with Parameterized Symbolic [...]
Advanced robotics for manufacturing: challenges and opportunities
Abstract: Presenting projects with ARM Institute (including robot grinding, human-robot collaboration, and modularized manufacturing) and discussing some new opportunities in applying AI and robotics in manufacturing domain.
Improving Robot Capabilities Through Reconfigurability
Abstract: Advancements in robot capabilities are often achieved through integrating more hardware components. These hardware additions often lead to systems with high power consumption, fragility, and difficulties in control and maintenance. However, is this approach the only path to enhancing robot functionality? In this talk, I introduce the PuzzleBots, a modular multi-robot system with passive [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Reduced-Gravity Flights and Field Testing for Lunar and Planetary Rovers
Abstract: As humanity returns to the Moon and is developing outposts and related infrastructure, we need to understand how robots and work machines will behave in this harsh environment. It is challenging to find representative testing environments on Earth for Lunar and planetary rovers. To investigate the effects of reduced-gravity on interactions with granular terrains, [...]
Design Principles for Robotics Systems that Support Human-Human Collaborative Learning
Abstract: Robots possess unique affordances granted by combining software and hardware. Most existing research focuses on the impact of these affordances on human-robot collaboration, but the theory of how robots can facilitate human-human collaboration is underdeveloped. Such theory would be beneficial in education. An educational device can afford collaboration in both assembly and use. This [...]
Leveraging Parallelism to Accelerate Quadratic Program Solvers for MPC
Abstract: Many problems in robotics can be formulated as quadratic programs (QPs). In particular, model-predictive control problems often involve repeatedly solving QPs at very high rates (up to kilohertz). However, while other areas of robotics like machine learning have achieved high performance by taking advantage of parallelism on modern computing hardware, state-of-the-art algorithms for solving [...]
Shedding Light on 3D Cameras
Abstract: The advent (and commoditization) of low-cost 3D cameras is revolutionizing many application domains, including robotics, autonomous navigation, human computer interfaces, and recently even consumer devices such as cell-phones. Most modern 3D cameras (e.g., LiDAR) are active; they consist of a light source that emits coded light into the scene, i.e., its intensity is modulated over [...]
Robust Incremental Distributed Collaborative Simultaneous Localization and Mapping
Abstract: Multi-robot teams show exceptional promise across applications like Search-and-Rescue, disaster-response, agriculture, forestry, and scientific exploration due to their ability to go where humans cannot, parallelize activity, operate robustly to failures, and expand capabilities beyond that of an individual robot. Collaborative Simultaneous Localization and Mapping (C-SLAM) is a fundamental capability for these multi-robot teams as [...]
Towards Equitable Representation in Text-to-Image Generation
Abstract: Accurate representation in media is known to improve the well-being of the people who consume it. There is a growing concern about the increasing use of generative AI in media as the generative image models trained on large web-crawled datasets such as LAION are known to produce images with harmful stereotypes and misrepresentations of various groups, [...]
3D Inference from Unposed Sparse View Images
Abstract: We propose UpFusion, a system that can perform novel view synthesis and infer 3D representations for generic objects given a sparse set of reference images without corresponding pose information. Current sparse-view 3D inference methods typically rely on camera poses to geometrically aggregate information from input views, but are not robust in-the-wild when such information [...]
Tightly Coupled LIDAR-Inertial Odometry
Abstract: In the age of self-driving, LIDAR and IMU represent two of the most ubiqui- tous sensors in use. Kalman Filtering and loosely coupled approaches dominate industry techniques, while current research trends towards a more tightly coupled formulation involving a joint optimization of IMU and LIDAR measurements. After two years of experience working with and [...]