Magnification-invariant retinal distance estimation using a laser aiming beam
Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]
Towards more effective remote execution of exploration operations using multimodal interfaces
Abstract: Remote robots enable humans to explore and interact with environments while keeping them safe from existing harsh conditions (e.g., in search and rescue, deep sea or planetary exploration scenarios). However, the gap between the control station and the remote robot presents several challenges (e.g., situation awareness, cognitive load, perception, latency) for effective teleoperation. Multimodal [...]
Bridging Humans and Generative Models
Abstract: Deep generative models make visual content creation more accessible to novice and professional users alike by automating the synthesis of diverse, realistic content based on a collected dataset. People often use generative models as data-driven sources, making it challenging to personalize a model easily. Currently, personalizing a model requires careful data curation, which is [...]
Learning Visual, Audio, and Cross-Modal Correspondences
Abstract: Today's machine perception systems rely heavily on supervision provided by humans, such as labels and natural language. I will talk about our efforts to make systems that, instead, learn from two ubiquitous sources of unlabeled data: visual motion and cross-modal sensory associations. I will begin by discussing our work on creating unified models for [...]
Impulse considerations for reasoning about intermittent contacts
Abstract: Many of our interactions with the environment involve making and breaking contacts. However, it is not always obvious how one should reason about these intermittent contacts (sequence, timings, locations) in an online and adaptive way. This is particularly relevant in gait generation for legged locomotion control, where it is standard to simply predefine and [...]
Multi-Human 3D Reconstruction from Monocular RGB Videos
Abstract: We study the problem of multi-human 3D reconstruction from RGB videos captured in the wild. Humans have dynamic motion, and reconstructing them in arbitrary settings is key to building immersive social telepresence, assistive humanoid robots, and augmented reality systems. However, creating such a system requires addressing fundamental issues with previous works regarding the data [...]
Learning and Translating Temporal Abstractions across Humans and Robots
Abstract: Humans possess a remarkable ability to learn to perform tasks from a variety of different sources-from language, instructions, demonstration, etc. In each case, they are able to easily extract the high-level strategy to solve the task, such as the recipe of cooking a dish, whilst ignoring irrelevant details, such as the precise shape of [...]
Robust Incremental Smoothing and Mapping
Abstract: In this work we present a method for robust optimization for online incremental Simultaneous Localization and Mapping (SLAM). Due to the NP-Hardness of data association in the presence of perceptual aliasing, tractable (approximate) approaches to data association will produce erroneous measurements. We require SLAM back-ends that can converge to accurate solutions in the presence [...]
Carnegie Mellon University
3D Reconstruction using Differential Imaging
Abstract: 3D reconstruction has been at the core of many computer vision applications, including autonomous driving, visual inspection in manufacturing, and augmented and virtual reality (AR/VR). Because monocular 3D sensing is fundamentally ill-posed, many techniques aiming for accurate reconstruction use multiple captures to solve the inverse problem. Depending on the amount of change in these [...]
Learning with Structured Priors for Robust Robot Manipulation
Abstract: Robust and generalizable robots that can autonomously manipulate objects in semi-structured environments can bring material benefits to society. Data-driven learning approaches are crucial for enabling such systems by identifying and exploiting patterns in semi-structured environments, allowing robots to adapt to novel scenarios with minimal human supervision. However, despite significant prior work in learning for [...]
Learning Parameter-Efficient Quadrotor Dynamics Models
Abstract: Operation of quadrotors through high-speed, high-acceleration maneuvers remains a challenging problem due to the complex aerodynamics in this regime. While standard physical models suffice for control in near-hover conditions, the primary challenge in executing aggressive trajectories is obtaining a model for the quadrotor dynamics that adequately models the aerodynamic effects present, including lift, drag, [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Carnegie Mellon University
Self-Supervising Occlusions For Vision
Abstract: Virtually every scene has occlusions. Even a scene with a single object exhibits self-occlusions - a camera can only view one side of an object (left or right, front or back), or part of the object is outside the field of view. More complex occlusions occur when one or more objects block part(s) of [...]
Multi-Sensor Robot Navigation and Subterranean Exploration
Predicting The Future and Linking the Past: Learning and Constructing Structured Models for Robotic Manipulation
Abstract: Intelligent robotic agents need to reason about the dynamics of their surrounding world, and use such dynamics reasoning to make future predictions for efficient task planning. In addition, it is also desirable for robots to associate past experience in their memories to their current observation, and conduct analogical reasoning to complete tasks at their [...]
Carnegie Mellon University
MSR Thesis Talk: Tushar Kusnur
Title: Search-based Planning for Sensor-based Coverage Abstract: Robots are excellent candidates for the dull, dirty, and dangerous jobs we do not want humans to perform. Today, these include inspection of large areas or structures, post-disaster assessment, and surveillance. Assessing the aftermath of the recent Fern Hollow bridge collapse in Pittsburgh is one such example. Many [...]
Human-in-the-loop Model Creation
Abstract: Deep generative models make visual content creation more accessible to novice users by automating the synthesis of diverse, realistic content based on a collected dataset. However, the current machine learning approaches miss several elements of the creative process -- the ability to synthesize things that go far beyond the data distribution and everyday experience, [...]
Robotic Interestingness via Human-Informed Few-Shot Object Detection
Abstract: Interestingness recognition is crucial for decision making in autonomous exploration for mobile robots. Previous methods proposed an unsupervised online learning approach that can adapt to environments and detect interesting scenes quickly, but lack the ability to adapt to human-informed interesting objects. To solve this problem, we introduce a human-interactive framework, AirInteraction, that can detect [...]
Towards a formal theory of deep optimisation
Abstract: Precise understanding of the training of deep neural networks is largely restricted to architectures such as MLPs and cost functions such as the square cost, which is insufficient to cover many practical settings. In this talk, I will argue for the necessity of a formal theory of deep optimisation. I will describe such a [...]
Carnegie Mellon University
MSR Thesis Talk: Nikhil Angad Bakshi
Title: See But Don't Be Seen: Towards Stealthy Active Search in Heterogeneous Multi-Robot Systems Abstract: Robotic solutions for quick disaster response are essential to ensure minimal loss of life, especially when the search area is too dangerous or too vast for human rescuers. We model this problem as an asynchronous multi-agent active-search task where each robot aims [...]
Carnegie Mellon University
MSR Thesis Talk: Yves Georgy Daoud
Title: Spatial Tasking in Human-Robot Collaborative Exploration Abstract: This work develops a methodology for collaborative human-robot exploration that leverages implicit coordination. Most autonomous single- and multi-robot exploration systems require a remote operator to provide explicit guidance to the robot team. Few works consider how to integrate the human partner alongside robots to provide guidance in the [...]
Carnegie Mellon University
MSR Thesis Talk: Ambareesh Revanur
Title: Towards Video-based Physiology Estimation Abstract: RGB-video based human physiology estimation has a wide range of practical applications in telehealth, sports and deep fake detection. Therefore, researchers in the community have collected several video datasets and have advanced new methods over the years. In this dissertation, we study these methods extensively and aim to address the [...]
Carnegie Mellon University
MSR Thesis Talk: Raghavv Goel
Title: Automating Ultrasound Based Vascular Access Abstract: Timely care of trauma patients is important to prevent casualties in resource-limited regions such as the battlefield. In order to treat such trauma using point of care diagnosis, medical practitioners typically use an ultrasound for vascular access or detection of subcutaneous splinters for providing critical care. The problem here is two-fold: [...]
Carnegie Mellon University
MSR Thesis Talk: Mayank Singh
Title: Analogical Networks: Memory-Modulated In-Context 3D Parsing Abstract: Recent advances in the applications of deep neural networks to numerous visual perception tasks have shown excellent performance. However, this generally requires access to large amount of training samples and hence one persistent challenge is the setting of few-shot learning. In most existing works, a separate parametric neural [...]
Carnegie Mellon University
Learning with Diverse Forms of Imperfect and Indirect Supervision
Abstract: Powerful Machine Learning (ML) models trained on large, annotated datasets have driven impressive advances in fields including natural language processing and computer vision. In turn, such developments have led to impactful applications of ML in areas such as healthcare, e-commerce, and predictive maintenance. However, obtaining annotated datasets at the scale required for training high [...]
Carnegie Mellon University
MSR Thesis Talk: Yutian Lei
Title: ARC: AdveRsarial Calibration between Modalities Abstract: Advances in computer vision and machine learning techniques have led to flourishing success in RGB-input perception tasks, which has also opened unbounded possibilities for non-RGB-input perception tasks, such as object detection from wireless signals, point clouds, and infrared light. However, compared to the matured development pipeline of RGB-input [...]
FRIDA: Supporting Artistic Communication in Real-World Image Synthesis Through Diverse Input Modalities
Abstract: FRIDA, a Framework and Robotics Initiative for Developing Arts, is a robot painting system designed to translate an artist's high-level intentions into real world paintings. FRIDA can paint from combinations of input images, text, style examples, sounds, and sketches. Planning is performed in a differentiable, simulated environment created using real data from the robot [...]
Perception for High-Speed Off-Road Driving
Abstract: On-road autonomous driving has seen rapid progress in recent years with driverless vehicles being tested in various cities worldwide. However, this progress is limited to cities with well-established infrastructure and has yet to transfer to off-road regimes with unstructured environments and few paved roads. Advances in high-speed and reliable autonomous off-road driving can unlock [...]
Continual Learning of Compositional Skills for Robust Robot Manipulation
Abstract: Real world robots need to continuously learn new manipulation tasks in a lifelong learning manner. These new tasks often share sub-structures (in the form of sub-tasks, controllers) with previously learned tasks. To utilize these shared sub-structures, we explore a compositional and object-centric approach to learn manipulation tasks. While compositionality in robot manipulation can manifest [...]
Junior Faculty PhD Admissions Process Presentation
A presentation lead by David Wettergreen regarding the PhD Admission process.
Carnegie Mellon University
MSR Thesis Talk: Samuel Ong
Title: Data-Driven Slip Model for Improved Localization and Path Following applied to Lunar Micro-Rovers Abstract Micro-lunar rovers need to solve a slew of challenges on the Moon, with no human intervention. One such challenge is the need to know their location in order to navigate and build maps. However, localization is challenging on the moon due [...]
Computational Interferometric Imaging
Abstract: Imaging systems typically accumulate photons that, as they travel from a light source to a camera, follow multiple different paths and interact with several scene objects. This multi-path accumulation process confounds the information that is available in captured images about the scene and makes using these images to infer properties of scene objects, such [...]
Making AI trustworthy and understandable by clinicians
Abstract: Understandable-AI techniques facilitate to use of AI as a tool by human experts, giving humans insight into how AI decisions are made thereby helping experts discern which AI predictions should or shouldn’t be trusted. Understandable techniques may be especially useful for applications with insufficient validation data for regulatory approval, for which human experts must remain the final decision [...]
Towards Interactive Radiance Fields
Abstract: Over the last years, the fields of computer vision and computer graphics have increasingly converged. Using the exact same processes to model appearance during 3D reconstruction and rendering has shown tremendous benefits, especially when combined with machine learning techniques to model otherwise hard-to-capture or -simulate optical effects. In this talk, I will give an [...]
Robust and Context-Aware Real-Time Collaborative Robot Handling with Dynamic Gesture Commands
Abstract: Real-time collaborative robot (cobot) handling is a task where the cobot maneuvers an object under human dynamic gesture commands. Enabling dynamic gesture commands is useful when the human needs to avoid direct contact with the robot or the object handled by the robot. However, the key challenge lies in the heterogeneity in human behaviors [...]
Learning Representations for Interactive Robotics
In this talk, I will be discussing the role of learning representations for robots that interact with humans and robots that interactively learn from humans through a few different vignettes. I will first discuss how bounded rationality of humans guided us towards developing learned latent action spaces for shared autonomy. It turns out this “bounded rationality” is not a [...]
Motion Planning Around Obstacles with Graphs of Convex Sets
Abstract: In this talk, I'll describe a new approach to planning that strongly leverages both continuous and discrete/combinatorial optimization. The framework is fairly general, but I will focus on a particular application of the framework to planning continuous curves around obstacles. Traditionally, these sort of motion planning problems have either been solved by trajectory optimization [...]
RE2 Robotics: from RI spinout to Acquisition
Abstract: It was July 2001. Jorgen Pedersen founded RE2 Robotics. It was supposed to be a temporary venture while he figured out his next career move. But the journey took an unexpected course. RE2 became a leading developer of mobile manipulation systems. Fast forward to 2022, RE2 Robotics exited via an acquisition to Sarcos Technology and [...]
Equivalent Policy Sets for Learning Aligned Models and Abstractions
Abstract: Recent successes in model-based reinforcement learning (MBRL) have demonstrated the enormous value that learned representations of environmental dynamics (i.e., models) can impart to autonomous decision making. While a learned model can never perfectly represent the dynamics of complex environments, models that are accurate in the "right” ways may still be highly useful for decision [...]
Dynamic Route Guidance in Vehicle Networks by Simulating Future Traffic Patterns
Abstract: Roadway congestion leads to wasted time and money and environmental damage. Since adding more roadway capacity is often not possible in urban environments, it is becoming more important to use existing road networks more efficiently. Toward this goal, recent research in real-time, schedule-driven intersection control has shown an ability to significantly reduce the delays [...]
Enabling Self-sufficient Robot Learning
Abstract: Autonomous exploration and data-efficient learning are important ingredients for helping machine learning handle the complexity and variety of real-world interactions. In this talk, I will describe methods that provide these ingredients and serve as building blocks for enabling self-sufficient robot learning. First, I will outline a family of methods that facilitate active global exploration. [...]
Adaptive Robotic Assistance through Observations of Human Behavior
Abstract: Assistive robots should take actions that support people's goals. This is especially true as robots enter into environments where personal agency is paramount, such as a person's home. Home environments have a wide variety of "optimal' solutions that depend on personal preference, making it difficult for a robot to know the goal it should [...]
Perceiving Objects and Interactions in 3D
Abstract: We observe and interact with myriad of objects in our everyday lives, from cups and bottles to hammers and tennis rackets. In this talk, I will outline our group’s efforts towards understanding these objects and our everyday interactions with them in 3D. I will first focus on scaling 3D prediction for isolated objects across [...]
Understanding the Physical World from Images
If I show you a photo of a place you have never been to, you can easily imagine what you could do in that picture. Your understanding goes from the surfaces you see to the ones you know are there but cannot see, and can even include reasoning about how interaction would change the scene. [...]
Beyond Pick-and-Place: Towards Dynamic and Contact-rich Motor Skills with Reinforcement Learning
Abstract: Interactions with the physical world are at the core of robotics. However, robotics research, especially in manipulation, has been mainly focused on tasks with limited interactions with the physical world such as pick-and-place or pushing objects on the table top. These interactions are often quasi-static, have predefined or limited sequence of contact events and [...]
How Computer Vision Helps – from Research to Scale
Abstract: Vasudevan (Vasu) Sundarababu, SVP and Head of Digital Engineering, will cover the topic: ‘How Computer Vision Helps – from Research to Scale’. During his time, Vasu will explore how Computer Vision technology can be leveraged in-market today, the key projects he is currently leading that leverage CV, and the end-to-end lifecycle of a CV initiative - [...]
Adaptive-Anytime Planning and Mapping for Multi-Robot Exploration in Large Environments
Abstract: Robotic systems are being leveraged to explore environments too hazardous for humans to enter. Robot sensing, compute, and kinodynamic (SCK) capabilities are inextricably tied to the size, weight, and power (SWaP) constraints of the vehicle. When designing a robot team for exploration, the diversity and types of robots used must be carefully considered because [...]
Neural Radiance Fields with LiDAR Maps
Abstract: Maps, as our prior understanding of the environment, play an essential role for many modern robotic applications. The design of maps, in fact, is a non-trivial art of balance between storage and richness. In this thesis, we explored map compression for image-to-LiDAR registration, LiDAR-to-LiDAR map registration, and image-to-SfM map registration, and finally, inspired by [...]
Enabling Data-Efficient Real-World Model-Based Manipulation by Estimating Preconditions for Inaccurate Models
Abstract: This thesis explores estimating and reasoning about model deviation in robot learning for manipulation to improve data efficiency and reliability to enable real-robot manipulation in a world where models are inaccurate but still useful. Existing strategies are presented for improving planning robustness with low amounts of real-world data by an empirically estimated model precondition to guide [...]
Robust Adaptive Reinforcement Learning for Safety Critical Applications via Curricular Learning
Abstract: Reinforcement Learning (RL) presents great promises for autonomous agents. However, when using robots in a safety critical domain, a system has to be robust enough to be deployed in real life. For example, the robot should be able to perform across different scenarios it will encounter. The robot should avoid entering undesirable and irreversible [...]
Motion Matters in the Metaverse
Abstract: Abstract: In the early 1970s, Psychologists investigated biological motion perception by attaching point-lights to the joints of the human body, known as ‘point light walkers’. These early experiments showed biological motion perception to be an extreme example of sophisticated pattern analysis in the brain, capable of easily differentiating human motions with reduced motion cues. Further [...]
MSR Thesis Talk: Yichen Li
Title: Simulation-guided Design for Vision-based Tactile Sensing on a Soft Robot Finger Abstract: Soft pneumatic robot manipulators have garnered widespread interest due to their compliance and flexibility, which enable soft, non-destructive grasping and strong adaptability to complex working environments. Tactile sensing is crucial for these manipulators to provide real-time contact information for control and manipulation. [...]
Controllable Visual-Tactile Synthesis
Abstract: Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual Try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. The main challenges for multi-modal synthesis lie in the significant scale discrepancy between vision [...]
Geometry Processing and Differential Geometry
Abstract: Basic representations for three-dimensional geometry have a profound effect on what can be achieved downstream, in a variety of disciplines (physical simulation, computational design, geometric learning, etc.). In this talk I will discuss recent efforts in our group to revisit fundamental choices in the way we represent digital geometry, and solve geometric equations. The guiding [...]
Perceiving Particles Inside a Container using Dynamic Touch Sensing
Abstract: Dynamic touch sensing has shown potential for multiple tasks. In this talk, I will present how we utilize dynamic touch sensing to perceive particles inside a container with two tasks: classification of the particles inside a container and property estimation of the particles inside a container. First, we try to recognize what is inside [...]
Towards Photorealistic Dynamic Capture and Animation of Human Hair and Head
Abstract: Realistic human avatars play a key role in immersive virtual telepresence. To reach a high level of realism, a human avatar needs to faithfully reflect human appearance. A human avatar should also be drivable and express natural motions. Existing works have made significant progress on building drivable realistic face avatars, but they rarely include [...]
What do generative models know about geometry and illumination?
Abstract: Generative models can produce compelling pictures of realistic scenes. Objects are in sensible places, surfaces have rich textures, illumination effects appear accurate, and the models are controllable. These models, such as StyleGAN, can also generate semantically meaningful edits of scenes by modifying internal parameters. But do these models manipulate a purely abstract representation of the [...]
Life as a Professor Seminar
Have you ever wondered what life is like as a professor? What do professors do on a daily basis? What makes the faculty career challenging and rewarding? Maybe you have even thought about becoming a faculty member yourself? Join us on March 22nd from 2:00 - 3:30 PM, where a panel of CMU faculty will [...]
Carnegie Mellon University
System Identification and Control of Multiagent Systems Through Interactions
Abstract: This thesis investigates the problem of inferring the underlying dynamic model of individual agents of a multiagent system (MAS) and using these models to shape the MAS's behavior using robots extrinsic to the MAS. We investigate (a) how an observer can infer the latent task and inter-agent interaction constraints from the agents' motion and [...]
Examining the Role of Adaptation in Human-Robot Collaboration
Abstract: Human and AI partners increasingly need to work together to perform tasks as a team. In order to act effectively as teammates, collaborative AI should reason about how their behaviors interplay with the strategies and skills of human team members as they coordinate on achieving joint goals. This talk will discuss a formalism for [...]
A Multi-view Synthetic and Real-world Human Activity Recognition Dataset
Abstract: Advancements in Human Activity Recognition (HAR) partially relies on the creation of datasets that cover a broad range of activities under various conditions. Unfortunately, obtaining and labeling datasets containing human activity is complex, laborious, and costly. One way to mitigate these difficulties with sufficient generality to provide robust activity recognition on unseen data is [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
A Constructivist’s Guide to Robot Learning
Over the last decade, a variety of paradigms have sought to teach robots complex and dexterous behaviors in real-world environments. On one end of the spectrum we have nativist approaches that bake in fundamental human knowledge through physics models, simulators and knowledge graphs. While on the other end of the spectrum we have tabula-rasa approaches [...]
Robot Learning by Understanding Egocentric Videos
Abstract: True gains of machine learning in AI sub-fields such as computer vision and natural language processing have come about from the use of large-scale diverse datasets for learning. In this talk, I will discuss if and how we can leverage large-scale diverse data in the form of egocentric videos (first-person videos of humans conducting [...]
Eye Gaze for Intelligent Driving
Abstract: Intelligent vehicles have been proposed as one path to increasing vehicular safety and reduce on-road crashes. Driving intelligence has taken many forms, ranging from simple blind spot occupancy or forward collision warnings to lane keeping and all the way to full driving autonomy in certain situations. Primarily, these methods are outward-facing and operate on [...]
Dense 3D Representation Learning for Geometric Reasoning in Manipulation Tasks
Abstract: When solving a manipulation task like "put away the groceries" in real environments, robots must understand what *can* happen in these environments, as well as what *should* happen in order to accomplish the task. This knowledge can enable downstream robot policies to directly reason about which actions they should execute, and rule out behaviors [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Next-Generation Robot Perception: Hierarchical Representations, Certifiable Algorithms, and Self-Supervised Learning
Spatial perception —the robot’s ability to sense and understand the surrounding environment— is a key enabler for robot navigation, manipulation, and human-robot interaction. Recent advances in perception algorithms and systems have enabled robots to create large-scale geometric maps of unknown environments and detect objects of interest. Despite these advances, a large gap still separates robot [...]
Autonomous mobility in Mars exploration: recent achievements and future prospects
Abstract: This talk will summarize key recent advances in autonomous surface and aerial mobility for Mars exploration, then discuss potential future missions and technology needs for Mars and other planetary bodies. Among recent advances, the Perseverance rover that is now operating on Mars includes new autonomous navigation capability that dramatically increases its traverse speed over [...]
Passive Coupling in Robot Swarms
Abstract: In unstructured environments, ant colonies demonstrate remarkable abilities to adaptively form functional structures in response to various obstacles, such as stairs, gaps, and holes. Drawing inspiration from these creatures, robot swarms can collectively exhibit complex behaviors and achieve tasks that individual robots cannot accomplish. Existing modular robot platforms that employ dynamic coupling and decoupling [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Structures and Environments for Generalist Agents
Abstract: We are entering an era of highly general AI, enabled by supervised models of the Internet. However, it remains an open question how intelligence emerged in the first place, before there was an Internet to imitate. Understanding the emergence of skillful behavior, without expert data to imitate, has been a longstanding goal of reinforcement [...]
Learning novel objects during robot exploration via human-informed few-shot detection
Abstract: Autonomous mobile robots exploring in unfamiliar environments often need to detect target objects during exploration. Most prevalent approach is to use conventional object detection models, by training the object detector on large abundant image-annotation dataset, with a fixed and predefined categories of objects, and in advance of robot deployment. However, it lacks the capability [...]
Learning to Perceive and Predict Everyday Interactions
Abstract: This thesis aims to develop a computer vision system that can understand everyday human interactions with rich spatial information. Such systems can benefit VR/AR to perceive the reality and modify its virtual twin, and robotics to learn manipulation by watching human. Previous methods have been limited to constrained lab environment or pre-selected objects with [...]
Faculty Candidate: Wenshan Wang
Title: Towards General Autonomy: Learning from Simulation, Interaction, and Demonstration Abstract: Today's autonomous systems are still brittle in challenging environments or rely on designers to anticipate all possible scenarios to respond appropriately. On the other hand, leveraging machine learning techniques, robot systems are trained in simulation or the real world for various tasks. Due to [...]
From Videos to 4D Worlds and Beyond
Abstract: Abstract: The world underlying images and videos is 3-dimensional and dynamic, i.e. 4D, with people interacting with each other, objects, and the underlying scene. Even in videos of a static scene, there is always the camera moving about in the 4D world. Accurately recovering this information is essential for building systems that can reason [...]
Learning Models and Cost Functions from Unlabeled Data for Off-Road Driving
Abstract: Off-road driving is an important instance of navigation in unstructured environments, which is a key robotics problem with many applications, such as exploration, agriculture, disaster response and defense. The key challenge in off-road driving is to be able to take in high dimensional, multi-modal sensing data and use it to make intelligent decisions on [...]
Active Vision for Manipulation
Abstract: Decades of research on computer vision has highlighted the importance of active sensing -- where the agent actively controls parameters of the sensor to improve perception. Research on active perception the context of robotic manipulation has demonstrated many novel and robust sensing strategies involving a multitude of sensors like RGB and RGBD cameras, a [...]
Teruko Yata Memorial Lecture : Mobility and Manipulation Independence with Interface-Aware Robotics Intelligence
Dr. Brenna Argall is an associate professor of Mechanical Engineering, Electrical Engineering & Computer Science and Physical Medicine & Rehabilitation at Northwestern University. Her research lies at the intersection of robotics autonomy, machine learning and human rehabilitation. She is director of the assistive & rehabilitation robotics laboratory (argallab) at the Rehabilitation Institute of Chicago (RIC, [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Mars Robots and Robotics at NASA JPL
Abstract: In this seminar I’ll discuss Mars robots, the unprecedented results we’re seeing with the latest Mars mission, and how we got here. Perseverance’s manipulation and sampling systems have collected samples from unique locations at twice the rate of any prior mission. 88% of all driving has been autonomous. This has enabled the mission to [...]
Continually Improving Robots
Abstract: General purpose robots should be able to perform arbitrary manipulation tasks, and get better at performing new ones as they obtain more experience. The current paradigm in robot learning involves training a policy, in simulation or directly in the real world, with engineered rewards or demonstrations. However, for robots that need to keep learning [...]
Carnegie Mellon University
Parallelized Search on Graphs with Expensive-to-Compute Edges
Abstract: Search-based planning algorithms enable robots to come up with well-reasoned long-horizon plans to achieve a given task objective. They formulate the problem as a shortest path problem on a graph embedded in the state space of the domain. Much research has been dedicated to achieving greater planning speeds to enable robots to respond quickly [...]
Generative and Animatable Radiance Fields
Abstract: Generating and transforming content requires both creativity and skill. Creativity defines what is being created and why, while skill answers the question of how. While creativity is believed to be abundant, skill can often be a barrier to creativity. In our team, we aim to substantially reduce this barrier. Recent Generative AI methods have simplified the problem for 2D [...]
MSR Thesis Talk: Chonghyuk Song
Title: Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis Abstract: We explore the task of embodied view synthesis from monocular videos of deformable scenes. Given a minute-long RGBD video of people interacting with their pets, we render the scene from novel camera trajectories derived from in-scene motion of actors: (1) egocentric cameras that simulate the point [...]
Design Iteration of Dexterous Compliant Robotic Manipulators
Abstract: One goal of personal robotics is to have robots in homes performing everyday tasks efficiently to improve our quality of life. Towards this end, manipulators are needed which are low cost, safe around humans, and approach human-level dexterity. However, existing off-the-shelf manipulators are expensive both in cost and manufacturing time, difficult to repair, and [...]
MSR Thesis Talk: Shivam Duggal
Title: Learning Single Image 3D Reconstruction from Single-View Image Collections Abstract We present a framework for learning 3D object shapes and dense cross-object 3D correspondences from just an unaligned category-specific image collection. The 3D shapes are generated implicitly as deformations to a category-specific signed distance field and are learned in an unsupervised manner solely from unaligned [...]
Whisker Sensors for Unstructured Environments
Abstract: As robot applications expand from controllable factory settings to unknown environments, the robots will need a larger breadth of sensors to perceive these complex environments. In this thesis, I focus on developing whisker sensors for robot perception. The inspiration for whisker sensors comes from the biological world, where whiskers serve as tactile and flow [...]
MSR Thesis Talk: Himangi Mittal
Title: Audio-Visual State-Aware Representation Learning from Interaction-Rich Data Abstract In robotics and augmented reality, the input to the agent is a long stream of video from the first-person or egocentric point of view. Recently, there have been significant efforts to capture humans from their first-person/egocentric view interacting with their own environment as they go about [...]
MSR Thesis Talk: Ken Liu
Title: On Privacy and Personalization in Federated Learning: Analyses and Applications Abstract: Recent advances in machine learning often rely on large and centralized datasets. However, curating such data can be challenging when they hold private information, and policies/regulations may mandate that they remain distributed across data silos (e.g. mobile devices or hospitals). Federated learning (FL) [...]
Carnegie Mellon University
MSR Thesis Talk: Haolun Zhang
Title: Seeing in 3D: Towards Generalizable 3D Visual Representations for Robotic Manipulation Abstract: Despite the recent progress in computer vision and deep learning, robot perception remains a tremendous challenge due to the variations of the objects and the scenes in manipulation tasks. Ideally, a robot trying to manipulate a new object should be able to [...]
MSR Thesis Talk: Muyang Li
Title: Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models Abstract: During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a significant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general-purpose technique [...]
3D-aware Conditional Image Synthesis
Abstract: We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available posed [...]
Promoting Human Creativity with FRIDA: Framework and Robotics Initiative for Developing Arts
ABSTRACT: FRIDA, a reference to the vibrant painter Frida Kahlo, stands for a Framework and Robotics Initiative for Developing Arts to promote human creativity. FRIDA supports intuitive ways for people to collaboratively create artworks including natural language, images, and sounds. Because FRIDA is for real-world arts, our work is uniquely different from digital art tools [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Robotic Climbing for Extreme Terrain Exploration
Abstract: Climbing robots can investigate scientifically valuable sites that are inaccessible to conventional rovers due to steep terrain features. Robots equipped with microspine grippers are particularly well-suited to ascending rocky cliff faces, but existing designs are either large and slow, or limited to relatively flat surfaces such as buildings. We have developed a novel free-climbing [...]
MSR Thesis Talk: Rohan Zeng
Title: Spectral Unmixing and Mapping of Coral Reef Benthic Cover Abstract: Coral reefs are important to the global ecosystem and the local communities and wildlife that rely on the habitat they create. However, coral reefs are also in critical and rapid decline: reefs have degraded over recent decades and what remains is at increasing risk [...]
Generative modeling: from 3D scenes to fields and manifold
Abstract: In this keynote talk, we delve into some of our progress on generative models that are able to capture the distribution of intricate and realistic 3D scenes and fields. We explore a formulation of generative modeling that optimizes latent representations for disentangling radiance fields and camera poses, enabling both unconditional and conditional generation of 3D [...]
MSR Thesis Talk: Ashwin Misra
Title: Learn2Plan: Learning variable ordering heuristics for scalable task planning Abstract: Traditional approaches to planning attempt to transform a system into a goal state by applying specific actions in a specific order. In these methods, there is an exponential search space due to considering many possible actions at every decision point. Hierarchical Task Networks use incremental [...]
MSR Thesis Talk: Andrew Jong
Title: Robot Information Gathering for Dynamic Systems in Wildfire Scenarios Abstract: The monitoring of complex dynamic systems, such as those encountered in disaster response, search and rescue, wildlife conservation, and environmental monitoring, presents the fundamental challenge of how to track efficiently with limited resources and partial observability. This thesis presents algorithms and techniques for robotic [...]
Carnegie Mellon University
Visual Dataset Pipeline: From Curation to Long-Tail Learning
Abstract: Computer vision models have proven to be tremendously capable of recognizing and detecting several real-world objects: cars, people, pets. These models are only possible due to a meticulous pipeline where a task and application is first conceived followed by an appropriate dataset curation that collects and labels all necessary data. Commonly, studies are focused [...]
MSR Thesis Talk: Erin Wong
Title: Edge Detection by Centimeter Scale Low-Cost Mobile Robots Abstract: In Search and Rescue (SaR) efforts after natural disasters like earthquakes, the primary focus is to find and rescue people in building rubble. These rescue efforts could put first responders at risk and are slow due to the unstable nature of the environment. Robotic solutions [...]
Multi-Objective Ergodic Search for Dynamic Information Maps
Abstract: Robotic explorers are essential tools for gathering information about regions that are inaccessible to humans. For applications like planetary exploration or search and rescue, robots use prior knowledge about the area to guide their search. Ergodic search methods find trajectories that effectively balance exploring unknown regions and exploiting prior information. In many search based [...]
Observing Assistance Preferences via User-controlled Arbitration in Shared Control
Abstract: What factors influence people’s preferences for robot assistance during human-robot collaboration tasks? Answering this question can help roboticists formalize definitions of assistance that lead to higher user satisfaction and increased user acceptance of assistive technology. Often in human robot collaboration literature, we see assistance paradigms that aim to optimize task success metrics and/or measures [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.
Sparse-view 3D in the Wild
Abstract: Reconstructing 3D scenes and objects from images alone has been a long-standing goal in computer vision. We have seen tremendous progress in recent years, capable of producing near photo-realistic renderings from any viewpoint. However, existing approaches generally rely on a large number of input images (typically 50-100) in order to compute camera poses and [...]
Safely Influencing Humans in Human-Robot Interaction
Abstract: Robots are becoming more common in industrial manufacturing because of their speed and precision on repetitive tasks, but they lack the flexibility of human collaborators. In order to take advantage of both humans’ and robots’ abilities, we investigate how to improve the efficiency of human-robot collaborations by making sure that robots both 1. stay [...]
Inductive Biases for Learning Long-Horizon Manipulation Skills
Abstract: Enabling robots to execute temporally extended sequences of behaviors is a challenging problem for learned systems, due to the difficulty of learning both high-level task information and low-level control. In this talk, I will discuss three approaches that we have developed to address this problem. Each of these approaches centers on an inductive bias [...]