Towards editable indoor lighting estimation
Abstract: Combining virtual and real visual elements into a single, realistic image requires the accurate estimation of the lighting conditions of the real scene. In recent years, several approaches of increasing complexity---ranging from simple encoder-decoder architecture to more sophisticated volumetric neural rendering---have been proposed. While the quality of automatic estimates has increased, they have the unfortunate downside [...]
Causal Robot Learning for Manipulation
Abstract: Two decades into the third age of AI, the rise of deep learning has yielded two seemingly disparate realities. In one, massive accomplishments have been achieved in deep reinforcement learning, protein folding, and large language models. Yet, in the other, the promises of deep learning to empower robots that operate robustly in real-world environments [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Computational imaging with multiply scattered photons
Abstract: Computational imaging has advanced to a point where the next significant milestone is to image in the presence of multiply-scattered light. Though traditionally treated as noise, multiply-scattered light carries information that can enable previously impossible imaging capabilities, such as imaging around corners and deep inside tissue. The combinatorial complexity of multiply-scattered light transport makes [...]
Dense Reconstruction of Dynamic Structures from Monocular RGB Videos
Abstract: We study the problem of 3D reconstruction of {\em generic} and {\em deformable} objects and scenes from {\em casually-taken} RGB videos, to create a system for capturing the dynamic 3D world. Being able to reconstruct dynamic structures from casual videos allows one to create avatars and motion references for arbitrary objects without specialized devices, [...]
Differentiable Collision Detection
Abstract: Collision detection between objects is critical for simulation, control, and learning for robotic systems. However, existing collision detection routines are inherently non-differentiable, limiting their applications in gradient-based optimization tools. In this talk, I present DCOL: a fast and fully differentiable collision-detection framework that reasons about collisions between a set of composable and highly expressive [...]
Towards $1 robots
Abstract: Robots are pretty great -- they can make some hard tasks easy, some dangerous tasks safe, or some unthinkable tasks possible. And they're just plain fun to boot. But how many robots have you interacted with recently? And where do you think that puts you compared to the rest of the world's people? In [...]
Mental models for 3D modeling and generation
Abstract: Humans have extraordinary capabilities of comprehending and reasoning about our 3D visual world. One particular reason is that when looking at an object or a scene, not only can we see the visible surface, but we can also hallucinate the invisible parts - the amodal structure, appearance, affordance, etc. We have accumulated thousands of [...]
On Interaction, Imitation, and Causation
Abstract: A standard critique of machine learning models (especially neural networks) is that they pick up on spurious correlations rather than causal relationships and are therefore brittle in the face of distribution shift. Solving this problem in full generality is impossible (i.e. there might be no good way to distinguish between the two). However, if [...]
Learning via Visual-Tactile Interaction
Abstract: Humans learn by interacting with their surroundings using all of their senses. The first of these senses to develop is touch, and it is the first way that young humans explore their environment, learn about objects, and tune their cost functions (via pain or treats). Yet, robots are often denied this highly informative and [...]
Carnegie Mellon University
Accelerating Numerical Methods for Optimal Control
Abstract: Many modern control methods, such as model-predictive control, rely heavily on solving optimization problems in real time. In particular, the ability to efficiently solve optimal control problems has enabled many of the recent breakthroughs in achieving highly dynamic behaviors for complex robotic systems. The high computational requirements of these algorithms demand novel algorithms tailor-suited [...]
Tactile SLAM: perception for dexterity via vision-based touch
Abstract: Touch provides a direct window into robot-object interaction, free from occlusion and aliasing faced by visual sensing. Collated tactile perception can facilitate contact-rich tasks---like in-hand manipulation, sliding, and grasping. Here, online estimates of object geometry and pose are crucial for downstream planning and control. With significant advances in tactile sensing, like vision-based touch, a [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Resource Allocation for Learning in Robotics
Abstract: Robots operating in the real world need fast and intelligent decision making systems. While these systems have traditionally consisted of human-engineered behaviors and world models, there has been a lot of interest in integrating them with data-driven components to achieve faster execution and reduce hand-engineering. Unfortunately, these learning-based methods require large amounts of training [...]
What (else) can you do with a robotics degree?
Abstract: In 2004, half-way through my robotics Ph.D., I had a panic-inducing thought: What if I don’t want to build robots for the rest of my life? What can I do with this degree?! Nearly twenty years later, I have some answers: tackle climate change in Latin America, educate Congress about autonomous vehicles, improve how [...]
Complete Codec Telepresence
Abstract: Imagine two people, each of them within their own home, being able to communicate and interact virtually with each other as if they are both present in the same shared physical space. Enabling such an experience, i.e., building a telepresence system that is indistinguishable from reality, is one of the goals of Reality Labs [...]
R.I.P ohyay: experiences building online virtual experiences during the pandemic: what works, what hasn’t, and what we need in the future
Abstract: During the pandemic I helped design ohyay (https://ohyay.co), a creative tool for making and hosting highly customized video-based virtual events. Since Fall 2020 I have personally designed many online events: ranging from classroom activities (lectures, small group work, poster sessions, technical papers PC meetings), to conferences, to virtual offices, to holiday parties involving 100's [...]
Planning with Dynamics by Interleaving Search and Trajectory Optimization
Abstract: Search-based planning algorithms enable autonomous agents like robots to come up with well-reasoned long-horizon plans to achieve a given task objective. They do so by searching over the graph that results from discretizing the state and action space. However, in robotics, several dynamically rich tasks require high-dimensional planning in the continuous space. For such [...]
Physics-informed image translation
Abstract: Generative Adversarial Networks (GANs) have shown remarkable performances in image translation, being able to map source input images to target domains (e.g. from male to female, day to night, etc.). However, their performances may be limited by insufficient supervision, which may be challenging to obtain. In this talk, I will present our recent works [...]
Robots Should Reduce, Reuse, and Recycle
Abstract: Despite numerous successes in deep robotic learning over the past decade, the generalization and versatility of robots across environments and tasks has remained a major challenge. This is because much of reinforcement and imitation learning research trains agents from scratch in a single or a few environments, training special-purpose policies from special-purpose datasets. In [...]
Solving Constraint Tasks with Memory-Based Learning
Abstract: In constraint tasks, the current task state heavily limits what actions are available to an agent. Mechanical constraints exist in many common tasks such as construction, disassembly, and rearrangement and task space constraints exist in an even broader range of tasks. Deep reinforcement learning algorithms have typically struggled with constraint tasks for two main [...]
Weak Multi-modal Supervision for Object Detection and Persuasive Media
Abstract: The diversity of visual content available on the web presents new challenges and opportunities for computer vision models. In this talk, I present our work on learning object detection models from potentially noisy multi-modal data, retrieving complementary content across modalities, transferring reasoning models across dataset boundaries, and recognizing objects in non-photorealistic media. While the [...]
Head-Worn Assistive Teleoperation of Mobile Manipulators
Abstract: Mobile manipulators in the home can provide increased autonomy to individuals with severe motor impairments, who often cannot complete activities of daily living (ADLs) without the help of a caregiver. Teleoperation of an assistive mobile manipulator could enable an individual with motor impairments to independently perform self-care and household tasks, yet limited motor function [...]
Text Classification with Class Descriptions Only
Abstract: In this work, we introduce KeyClass, a weakly-supervised text classification framework that learns from class-label descriptions only, without the need to use any human-labeled documents. It leverages the linguistic domain knowledge stored within pre-trained language models and data programming to automatically label documents. We demonstrate its efficacy and flexibility by comparing it to state-of-the-art [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Multi-Object Tracking in the Crowd
Abstract: In this talk, I will focus on the problem of multi-object tracking in crowded scenes. Tracking within crowds is particularly challenging due to heavy occlusion and frequent crossover between tracking targets. The problem becomes more difficult when we only have noisy bounding boxes due to background and neighboring objects. Existing tracking methods try to [...]
Utilizing Panoptic Segmentation and a Locally-Conditioned Neural Representation to Build Richer 3D Maps
Abstract: Advances in deep-learning based perception and maturation of volumetric RGB-D mapping algorithms have allowed autonomous robots to be deployed in increasingly complex environments. For robust operation in open-world conditions however, perceptual capabilities are still lacking. Limitations of commodity depth sensors mean that complex geometries and textures cannot be reconstructed accurately. Semantic understanding is still [...]
NREC Study Group & Recent Projects
This talk will describe the NREC study process that has been developed as a lower cost of entry work product for potential partners. This is a process that is available for anyone on campus that wants to help their sponsors create viable system concepts and potential development costs before committing to a full development program. [...]
Machine Learning and Model Predictive Control for Adaptive Robotic Systems
Abstract: In this talk I will discuss several different ways in which ideas from machine learning and model predictive control (MPC) can be combined to build intelligent, adaptive robotic systems. I’ll begin by showing how to learn models for MPC that perform well on a given control task. Next, I’ll introduce an online learning perspective on [...]
Magnification-invariant retinal distance estimation using a laser aiming beam
Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]
Towards more effective remote execution of exploration operations using multimodal interfaces
Abstract: Remote robots enable humans to explore and interact with environments while keeping them safe from existing harsh conditions (e.g., in search and rescue, deep sea or planetary exploration scenarios). However, the gap between the control station and the remote robot presents several challenges (e.g., situation awareness, cognitive load, perception, latency) for effective teleoperation. Multimodal [...]
Bridging Humans and Generative Models
Abstract: Deep generative models make visual content creation more accessible to novice and professional users alike by automating the synthesis of diverse, realistic content based on a collected dataset. People often use generative models as data-driven sources, making it challenging to personalize a model easily. Currently, personalizing a model requires careful data curation, which is [...]