VASC Seminar
Tadas Baltrusaitis
Principal Scientist
Microsoft, Mixed Reality Cambridge

Fake It Till You Make It: Face analysis in the wild using synthetic data alone

Abstract: In this seminar I will demonstrate how synthetic data alone can be used to perform face-related computer vision in the wild. The community has long enjoyed the benefits of synthesizing training data with graphics, but the domain gap between real and synthetic data has remained a problem, especially for human faces. Researchers have tried [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Structured Learning for Robust Robot Manipulation

NSH 4305

Abstract: Robust and generalizable robots that can autonomously manipulate objects in semi-structured environments can bring material benefits to society. Data-driven learning approaches are crucial for enabling such systems by identifying and exploiting patterns in semi-structured environments, allowing robots to adapt to novel scenarios with minimal human supervision. However, despite significant prior work in learning for [...]

RI Seminar
Professor Emeritus
Robotics Institute,
Carnegie Mellon University

Robotics and Warehouse Automation at Berkshire Grey

1305 Newell Simon Hall

Abstract:  This talk tells the Berkshire Grey story, from its founding in 2013 to its IPO earlier this year — the first robotics IPO since iRobot over15 years ago.  Berkshire Grey produces automated systems for e-commerce order fulfillment, parcel sortation, store replenishment, and related operations in warehouses, distribution centers, and in the back ends of [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

An Experimental Design Perspective on Model-Based Reinforcement Learning

NSH 3305

Abstract: In many practical applications of RL, it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, computing the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or dollars of [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Learning Model Preconditions for Planning with Multiple Models

Abstract: Different models can provide differing levels of fidelity when a robot is planning. Analytical models are often fast to evaluate but only work in limited ranges of conditions. Meanwhile, physics simulators are effective at modeling complex interactions between objects but are typically more computationally expensive. Learning when to switch between the various models can [...]

VASC Seminar
Or Patashnik
Graduate Student
School of Computer Science at Tel-Aviv University

Leveraging StyleGAN for Image Editing and Manipulation

Abstract: StyleGAN has recently been established as the state-of-the-art unconditional generator, synthesizing images of phenomenal realism and fidelity, particularly for human faces. With its rich semantic space, many works have attempted to understand and control StyleGAN’s latent representations with the goal of performing image manipulations. To perform manipulations on real images, however, one must learn to [...]

RI Seminar
Sebastian Scherer & Matthew Travers
Robotics Institute, Carnegie Mellon University

Resilient Exploration in SubT Environments: Team Explorer’s Approach and Lessons Learned in the Final Event

1305 Newell Simon Hall

Abstract: Subterranean robot exploration is difficult with many mobility, communications, and navigation challenges that require an approach with a diverse set of systems, and reliable autonomy. While prior work has demonstrated partial successes in addressing the problem, here we convey a comprehensive approach to address the problem of subterranean exploration in a wide range of [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Simulation-based Planning for Pick-and-Place in Heavy Clutter using Non-prehensile Manipulation

NSH 3305

Abstract: Robot manipulation in domestic households, industrial manufacturing and warehouses might require contact-rich interactions with objects in the environment. For pick-and-place style grasping tasks in cluttered scenes, it can be more economical for the robot to rely on non-prehensile actions vis-à-vis deliberate prehensile rearrangement. Non-prehensile actions also let the robot manipulate large and bulky objects [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Relationships in instance segmentation and anomaly detection

GHC 4405

Abstract: This thesis primarily covers work on two different tasks in computer vision: (1) anomaly detection and (2) instance segmentation. Anomaly detection is an underexplored unsupervised problem that has existed in many fields. On the other hand, instance (and panoptic) segmentation is a supervised problem that can leverage the powerful data and key developments from [...]

VASC Seminar
Soumyadip Sengupta
Postdoctoral Research Associate
University of Washington

Next-Gen Video Communication

Abstract: Video communication connects our world. It is necessary in conducting business, educational and personal activities across different geographical locations. However, the quality of an average user’s video communication is dramatically worse than that of professionally created videos in news broadcasts, talk shows, and on YouTube. This is because professionally created videos are often captured with [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Learning with Diverse Forms of Imperfect and Indirect Supervision

Abstract: High capacity Machine Learning (ML) models trained on large, annotated datasets have driven impressive advances in several fields including natural language processing and computer vision, in turn leading to impactful applications of ML in areas such as healthcare, e-commerce, and predictive maintenance. However, obtaining annotated datasets at the scale required for training such models [...]

Student Talks

MRSD Annual Poster Presentation

Newell Simon Hall Atrium

Four student teams from the MRSD program will use posters, videos, and hardware to show their project work on robots for room disinfection, search & rescue, increasing human capability via a third arm, and increased-efficiency factory-floor obstacle avoidance.

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

3D Representation Learning for Perception and Prediction: A Modular Yet Highly Integrated Approach

Abstract: Modularized and cascaded autonomy stacks (object detection, then tracking and then trajectory prediction) have been widely adopted in many autonomous systems such as self-driving cars due to its interpretability. In this talk, I advocate the use of such a modular approach but improve its accuracy and robustness by developing different 3D representations for each [...]

MSR Thesis Defense
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk: Avi Rudich

1305 Newell Simon Hall

Title: Kinematic Analysis of 3D Printed Flexible Delta Robots   Abstract: Flexible Delta robots show significant promise for use in a wide array of manipulation tasks.  They are simple to design and manufacture, and they maintain a high level of repeatability and precision in open loop control.  This thesis analyzes the kinematic properties of flexible [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Reconstructing common objects to interact with

Abstract: We humans are able to understand 3D shapes of common daily objects and interact with them from a wide range of categories. We understand cups are usually cylinder-like and we can easily predict the shape of one particular cup, both in isolation or even when it is held by a human. We aim to [...]

VASC Seminar
Robert Collins
Associate Professor
Penn State University

Activity Understanding of Scripted Performances

Abstract: The PSU Taichi for Smart Health project has been doing a deep-dive into vision-based analysis of 24-form Yang-style Taichi (TaijiQuan). A key property of Taichi, shared by martial arts katas and prearranged form exercises in other sports, is practice of a scripted routine to build both mental and physical competence.  The scripted nature of routines [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Dynamical Model Learning and Inversion for Aggressive Quadrotor Flight

Abstract: Quadrotor applications have seen a surge recently and many tasks require precise and accurate controls. Flying fast is critical in many applications and the limited onboard power source makes completing tasks quickly even more important. Staying on a desired course while traveling at high speeds and high accelerations is difficult due to complex and [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Person Transfers Between Multiple Service Robots

NSH 3305

Abstract: As more service robots are deployed in the world, human-robot interaction will not be limited to one-to-one interactions between users and robots. Instead, users will likely have to interact with multiple robots, simultaneously or sequentially, throughout their day to receive services and complete different tasks. In this thesis, I describe work in which my [...]

PhD Speaking Qualifier
MSR Student
Robotics Institute,
Carnegie Mellon University

A causal framework to diagnose and fix issues with doors

Abstract: Many animals, such as ravens, (and a fortiori humans) exhibit a great deal of physical intelligence that allows them to solve complex multi-step physical puzzles. This ability indicates an understanding or a faculty to represent causality and mechanisms, understand when something goes wrong, and figure out how to deal with it. As a step [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Understanding Unbalanced Datasets Through Simple Models and Dataset Exploration

GHC 4405

Abstract: Computer vision models have proven to be tremendously capable of recognizing and detecting several classes and objects. They succeed in classes widely ranging in type and scale from humans to cans to pens. However, the best performing classes have abundant examples in large-scale datasets today. In unbalanced datasets, where some categories are seen in [...]

VASC Seminar
Vishal Patel
Associate Professor
Johns Hopkins University

Domain adaptive object detection

Abstract: Recent advances in deep learning have led to the development of accurate and efficient models for object detection. However, learning highly accurate models relies on the availability of large-scale annotated datasets. Due to this, model performance drops drastically when evaluated on label-scarce datasets having visually distinct images.  Domain adaptation tries to mitigate this degradation.  In [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Understanding, Exploiting and Improving Inter-view Relationships

NSH 3305

Abstract: Multi-view machine learning has garnered substantial attention in various applications over recent years. Many such applications involve learning on data obtained from multiple heterogeneous sources of information, for example, in multi-sensor systems such as self-driving cars, or monitoring intensive care patient vital signs at their bed-side. Learning models for such applications can often benefit [...]

RI Event
Project Scientist
Robotics Institute,
Carnegie Mellon University

Model-Centric Verification of Artificial Intelligence

Abstract: This work shows how provable guarantees can be used to supplement probabilistic estimates in the context of Artificial Intelligence (AI) systems. Statistical techniques measure the expected performance of a model, but low error rates say nothing about the ways in which errors manifest. Formal verification of model adherence to design specifications can yield certificates [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Designing Whisker Sensors to Detect Multiple Mechanical Stimuli for Robotic Applications

Abstract: Many mammals, such as rats and seals, use their whiskers as versatile mechanical sensors to gain precise information about their surroundings. Whisker-inspired sensors on robotic platforms have shown their potential benefit, improving applications ranging from drone navigation to texture mapping. Despite this, there is a gap between the engineered sensors and many of the [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Human-in-the-loop Control of Mobile Robots

Abstract: Human-in-the-loop control for mobile robots is an important aspect of robot operation, especially for navigation in unstructured environments or in the case of unexpected events. However, traditional paradigms of human-in-the-loop control have relied heavily on the human to provide precise and accurate control inputs to the robot, or reduced the role of the human [...]

VASC Seminar
Umberto Michieli
Postdoctoral Researcher and Adjunct Professor
University of Padua

Visual Understanding across Semantic Groups, Domains and Devices

Abstract: Deep neural networks often lack generalization capabilities to accommodate changes in the input/output domain distributions and, therefore, are inherently limited by the restricted visual and semantic information contained in the original training set. In this talk, we argue the importance of the versatility of deep neural architectures and we explore it from various perspectives.   [...]

RI Seminar
Stefanos Nikolaidis
Assistant Professor
Computer Science, University of Southern California

Towards Robust Human-Robot Interaction: A Quality Diversity Approach

Abstract: The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring the diverse scenarios of interaction between humans and robots in simulation can improve understanding of complex human-robot interaction systems and avoid potentially costly failures in real-world settings. [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Planning and Execution using Inaccurate Models with Provable Guarantees on Task Completeness

Abstract: Modern planning methods are effective in computing feasible and optimal plans for robotic tasks when given access to accurate dynamical models. However, robots operating in the real world often face situations that cannot be modeled perfectly before execution. Thus, we only have access to simplified but potentially inaccurate models. This imperfect modeling can lead [...]

VASC Seminar
Chao Chen
Assistant Professor
Stony Brook University

Topology-Driven Learning for Biomedical Imaging Informatics

Abstract: Thanks to decades of technology development, we are now able to visualize in high quality complex biomedical structures such as neurons, vessels, trabeculae and breast tissues. We need innovative methods to fully exploit these structures, which encode important information about underlying biological mechanisms. In this talk, we explain how topology, i.e., connected components, handles, loops, [...]

RI Seminar
Professor / Director of RI
Robotics Institute,
Carnegie Mellon University

Lessons from the Field: Deep Learning and Machine Perception for field robots

Abstract: Mobile robots now deliver vast amounts of sensor data from large unstructured environments. In attempting to process and interpret this data there are many unique challenges in bridging the gap between prerecorded data sets and the field. This talk will present recent work addressing the application of machine learning techniques to mobile robotic perception. [...]

VASC Seminar
Gianfranco Doretto
Associate Professor
West Virginia University

Learning generative representations for image distributions

Abstract: Autoencoder neural networks are an unsupervised technique for learning representations, which have been used effectively in many data domains. While capable of generating data, autoencoders have been inferior to other models like Generative Adversarial Networks (GAN’s) in their ability to generate image data. We will describe a general autoencoder architecture that addresses this limitation, and [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Self-Supervising Occlusions for Vision

Abstract: Virtually every scene has occlusions. Even a scene with a single object exhibits self-occlusions - a camera can only view one side of an object (left or right, front or back), or part of the object is outside the field of view. More complex occlusions occur when one or more objects block part(s) of [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Development of an Agile and Dexterous Balancing Mobile Manipulator Robot

Abstract: This thesis focuses on designing and controlling a dynamically stable shape-accelerating dual-arm mobile manipulator, the Carnegie Mellon University (CMU) ballbot. The CMU ballbot is a human-sized dynamically stable mobile robot that balances on a single spherical wheel. We describe the development of a pair of seven-degree-of-freedom (DOF) humanoid arms. The new 7-DOF arm pair [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Massively Parallelized Lazy Planning Algorithms

GHC 4405

Abstract: Search-based planning algorithms enable autonomous agents like robots to come up with well-reasoned long horizon plans to achieve a given task objective. They do so by optimizing a task-specific cost function while respecting the constraints on either the agent (e.g. motion constraints) or the environment (e.g. obstacles). In robotics, such as in motion planning [...]

VASC Seminar
Daniel McDuff
Principal Researcher
Microsoft Research

Building Intelligent and Visceral Machines: From Sensing to Application

Abstract: Humans have evolved to have highly adaptive behaviors that help us survive and thrive. As AI prompts a move from computing interfaces that are explicit and procedural to those that are implicit and intelligent, we are presented with extraordinary opportunities. In this talk, I will argue that understanding affective and behavioral signals presents many opportunities [...]

VASC Seminar
Arun Mallya
Senior Research Scientist
NVIDIA

GANcraft – an unsupervised 3D neural method for world-to-world translation

Abstract: Advances in 2D image-to-image translation methods, such as SPADE/GauGAN, have enabled users to paint photorealistic images by drawing simple sketches similar to those created in Microsoft Paint. Despite these innovations, creating a realistic 3D scene remains a painstaking task, out of the reach of most people. It requires years of expertise, professional software, a library [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Run-Time Optimization in the Deep Learning Age

Abstract: In a recovery task one seeks to obtain an estimate of an unknown signal from a set of incomplete measurements. These problems arise in a number of computer vision applications, from image based tasks such as super-resolution and in-painting to 3D reconstruction tasks such as Non-Rigid Structure from Motion and scene flow estimation. Early [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

System Identification and Control of Multiagent Systems Through Interactions

GHC 6501

Abstract: This thesis investigates the problem of identifying dynamics models of individual agents of a multiagent system (MAS) and exploiting these models to shape their behavior using robots extrinsic to the MAS. While task-based control of a MAS using onboard controllers of its agents is well studied, we investigate (a) how easy it is for [...]

Faculty Events
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Human-in-the-loop Model Creation

Newell-Simon Hall 4305

Abstract: Modern machine learning systems have made astonishing progress in automating labor-intensive tasks such as visual recognition and machine translation. While ML systems complete these tasks better and faster, humans are largely left behind. Indeed, most humans are entirely excluded from the creation process of machine learning models, except for tedious data annotation.   In [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Learning and Inference in Factor Graphs with Applications to Tactile Perception

Abstract: Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in hand [...]

VASC Seminar
Deqing Sun
Senior Research Scientist
Google

Learning Optical Flow: Model, Data, and Applications

Abstract: Optical flow provides important information about the dynamic world and is of fundamental importance to many tasks. In this talk, I will present my work on different aspects of learning optical flow. I will start with the background and talk about PWC-Net, a compact and effective model built using classical principles for optical flow. Next, [...]

Faculty Events
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Hands-On Interactions

Newell-Simon Hall 3305

Abstract: Our sense of touch is present in almost all our interactions with the world, from providing us with the feedback necessary to perceive and manipulate objects without having to look at them, to allowing our limbs to move and walk without us having to think about how to take the next step. We use [...]

RI Seminar
Leila Bridgeman
Assistant Professor of Mechanical Engineering & Materials Science
Duke University

Distributed Dissipativity: Applying Foundational Stability Theory to Modern Networked Control

Abstract: Despite its diverse areas of application, the desire to optimize performance and guarantee acceptable behaviour in the face of inevitable uncertainty is pervasive throughout control theory. This creates a fundamental challenge since the necessity of robustly stable control schemes often favors conservative designs, while the desire to optimize performance typically demands the opposite. While [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Towards Complex Robot Motions with Reinforcement Learning

Abstract: Reinforcement learning has shown to be a powerful tool for decision-making problems. In this talk, we present the opportunities and challenges of enabling increasingly complex robot behavior with reinforcement learning. First, we present a system that combines reinforcement learning and extrinsic dexterity to solve a novel task of “occluded grasping”. To reach an occluded [...]

RI Seminar
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Haptic Perspective-taking from Vision and Force

Abstract: Physically collaborative robots present an opportunity to positively impact society across many domains. However, robots currently lack the ability to infer how their actions physically affect people. This is especially true for robotic caregiving tasks that involve manipulating deformable cloth around the human body, such as dressing and bathing assistance. In this talk, I [...]

VASC Seminar
Chen Sun
Assistant Professor, Computer Science
Brown University

Do Vision-Language Pretrained Models Learn Spatiotemporal Primitive Concepts?

Abstract:  Vision-language models pretrained on web-scale data have revolutionized deep learning in the last few years. They have demonstrated strong transfer learning performance on a wide range of tasks, even under the "zero-shot" setup, where text "prompts" serve as a natural interface for humans to specify a task, as opposed to collecting labeled data. These models are [...]

Faculty Events

RI Council Meeting

Newell Simon Hall 4119

RI Council is a leadership group made up of the Director of RI, Academic Program Leads, Committee Chairs, and members at large as appointed by the Director. RI Council meets generally once a week to discuss department business.

RI Seminar
Jing Xiao
Professor and Department Head
Robotics Engineering Department, Worcester Polytechnic Institute (WPI)

Perception-Action Synergy in Uncertain Environments

Abstract: Many robotic applications require a robot to operate in an environment with unknowns or uncertainty, at least initially, before it gathers enough information about the environment. In such a case, a robot must rely on sensing and perception to feel its way around. Moreover, it has to couple sensing/perception and motion synergistically in real [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Driving Reconfigurable Unmanned Vehicle Design for Mobility Performance

Abstract: Unmanned ground vehicles are being deployed in increasingly diverse and complex environments. Advances in the field of robotics, including perception technology, computing power, and machine learning, have brought robots from the lab to the real world. Remote and autonomous vehicles are now used to explore volcanoes, caves, pipes, war zones, disaster sites, and even [...]

VASC Seminar
Dr. Randall Balestriero
Post-Doctorate Researcher
Meta AI

Max-Affine Spline Insights into Deep Learning

Abstract:  We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs) that provide a powerful portal through which we view and analyze their inner workings. For instance, [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Search-based Path Planning for a High Dimensional Manipulator in Cluttered Environments Using Optimization-based Primitives

Abstract: In this work we tackle the path planning problem for a 21-dimensional snake robot-like, navigating a cluttered gas turbine for the purposes of inspection. Heuristic search-based approaches are effective planning strategies for common manipulation domains. However, their performance on high-dimensional systems is heavily reliant on the effectiveness of the action space and the heuristics [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Vision-Based Tactile Sensor Design using Physics Based Rendering

GHC 8102

Abstract: Tactile sensing has seen a rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and human-robot interaction. However, the simulation of tactile sensors is still a challenge. Simulation is a critical tool in the development of robotic systems. In [...]

SCS Distinguished Lecture
Jeannette Bohg
Assistant Professor of Computer Science
Stanford University

Teruko Yata Memorial Lecture

Rashid Auditorium - 4401 Gates and Hillman Centers

Leveraging Language and Video Demonstrations for Learning Robot Manipulation Skills and Enabling Closed-Loop Task Planning Humans have gradually developed language, mastered complex motor skills, created and utilized sophisticated tools. The act of conceptualization is fundamental to these abilities because it allows humans to mentally represent, summarize and abstract diverse knowledge and skills. By means of [...]

Staff Events

Robotics Institute Staff Offices 12PM Early Dismissal

Dear RI Faculty and Staff, In observance of the coming holiday, institute staff offices will close Friday, April 15th at noon.  They will reopen at 8:30 on Monday, April 18th. Happy Holiday! Thank you – Debbie Z. =================================================== Deborah H. Zalewski, Senior Associate Business Manager | The Robotics Institute - Carnegie Mellon University | Newell-Simon [...]

Faculty Events

RI Hiring Meeting

Newell Simon Hall 4119

A faculty hiring meeting to discuss candidates for faculty position

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Unified Simulation, Perception, and Generation of Human Behavior

Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Kernel Density Decision Trees

Abstract We propose kernel density decision trees (KDDTs), a novel fuzzy decision tree (FDT) formalism based on kernel density estimation that improves the robustness of decision trees and ensembles and offers additional utility. FDTs mitigate the sensitivity of decision trees to uncertainty by representing uncertainty through fuzzy partitions. However, compared to conventional, crisp decision trees, [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Energy-based Joint Pose Estimation for 3D Reconstruction

Abstract: In this talk, I will describe a data-driven method for inferring camera poses given a sparse collection of images of an arbitrary object. This task is a core component of classic geometric pipelines such as structure-from-motion (SFM), and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction. [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

NeRF for Robotics

GHC 8102

Abstract: In this talk I'll describe how recent advances in neural rendering and novel view synthesis - namely NeRF - can be leveraged by robotic agents to improve performance in manipulation tasks. Specifically, I'll argue that NeRF can enable robotic policies to: (1) generalize to new viewpoints; (2) perceive specular and reflective surfaces in a [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Search Algorithms and Search Spaces for Neural Architecture Search

NSH 4305

Abstract: Neural architecture search (NAS) is recently proposed to automate the process of designing network architectures. Instead of manually designing network architectures, NAS automatically finds the optimal architecture in a data-driven way. Despite its impressive progress, NAS is still far from being widely adopted as a common paradigm for architecture design in practice. This thesis [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Evan Harber

Title: Stiffness Mapping of Deformable Objects Through Supervised Embedding and Gaussian Process Regression   Abstract: The stiffness map of a deformable object stores information about that object's surface compliance. Thus, through a stiffness map, we gain insight into the physical properties of that object. Depending on the object, an understanding of stiffness has applications ranging [...]

RI Seminar
Kirstin H. Petersen
Assistant Professor
Electrical & Computer Engineering, College of Engineering, Cornell University

Designing Robotic Systems with Collective Embodied Intelligence

Abstract: Natural swarms exhibit sophisticated colony-level behaviors with remarkable scalability and error tolerance. Their evolutionary success stems from more than just intelligent individuals, it hinges on their morphology, their physical interactions, and the way they shape and leverage their environment. Mound-building termites, for instance, are believed to use their own body as a template for [...]

MSR Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Gaurav Parmar

NSH 1109

Title: Spatially-Adaptive Multilayer GAN Inversion   Abstract: Existing GAN inversion and editing methods are well suited for only a target images that contain aligned objects with a clean background, such as portraits and animal faces, but often struggle for more difficult categories with complex scene layouts and object occlusions, such as cars, animals, and outdoor images. [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Robust Reinforcement Learning via Genetic Curriculum

GHC 6501

Abstract: Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Mouth Haptics in VR using a Headset Ultrasound Phased Array

GHC 7501

Abstract: This talk is the same one I will be presenting at the ACM CHI Conference on Human Factors in Computing Systems on May 2nd. Paper abstract: Today’s consumer virtual reality (VR) systems offer limited haptic feedback via vibration motors in handheld controllers. Rendering haptics to other parts of the body is an open challenge, [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Towards Large-scale and Long-term Neural Map Representations

Abstract: We address the problem of large-scale and long-term neural map representations. Maps, as our prior understanding toward the environment, provide valuable information for modern robotics applications such as autonomous driving and AR/VR. The size of maps largely affects the end task performance: usually a more detailed map can support better performance, but would cost [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Self-Improving 3D Scene Representations

GHC 6501

Abstract: Most computer vision models in deployment today are not continually learning. Instead, they are in a “test” mode, where they will behave the same way perpetually, until they are replaced by newer models. This is a problem, because it means the models may perform poorly as soon as their “test” environment diverges from their [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Manash Pratim Das

TBA

Title: Model-Accuracy Aware Anytime Planning with Simulation Verification for Navigating Complex Terrains Abstract: Off-road and unstructured environments often contain complex patches of various types of terrain, rough elevation changes, deformable objects, etc. An autonomous ground vehicle traversing such environments experiences physical interactions that are extremely hard to model at scale and thus very hard to [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Akshay Dharamavaram

NSH 4305

Title: Stabilizing the Training Dynamics of Generative Models using Self-Supervision   Abstract: Generative Models have been shown to be adept in mimicking the behavior of an unknown distribution solely from bootstrapped data. However, deep learning models have been shown to overfit in either the minimization or maximization stage of the two player min-max game, resulting [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Direct-drive Hands: Making Robot Hands Transparent and Reactive to Contacts

GHC 6501

Abstract: Industrial manipulators and end-effectors are a vital driver of the automation revolution. These robot hands, designed to reject disturbances with stiffness and strength, are inferior to their human counterparts. Human hands are dexterous and nimble effectors capable of a variety of interactions with the environment. Through this thesis we wish to answer a question: [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Vivek Roy

Newell-Simon Hall 3305

Title: Smartphone localization for Indoor Pedestrian Navigation Abstract: Global positioning system (GPS) interfacing with applications such as Google Maps has proven very useful for navigation in outdoor open settings. However in crowded metropolitan environments with high rise buildings or in indoor settings, GPS quickly becomes unreliable. Using sensors found on commodity smartphones to perform accurate [...]

VASC Seminar
David Fouhey
Assistant Professor
EECS Department , University of Michigan

Understanding 3D Scenes and Interacting Hands

Abstract:  Abstract: The long-term goal of my research is to help computers understand the physical world from images, including both 3D properties and how humans or robots could interact with things. This talk will summarize two recent directions aimed at enabling this goal.   I will begin with learning to reconstruct full 3D scenes, including [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Manipulating Objects with Challenging Visual and Geometric Properties

GHC 6501

Abstract: Object manipulation is a well-studied domain in robotics, yet manipulation remains difficult for objects with visually and geometrically challenging properties. Visually challenging properties, such as transparency and specularity, break assumptions of Lambertian reflectance that existing methods rely on for grasp estimation. On the other hand, deformable objects such as cloth pose both visual and [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

TIGRIS: An Informed Sampling-based Algorithm for Informative Path Planning

GHC 9115

Abstract: In this talk I will present our sampling-based approach to informative path planning that allows us to tackle the challenges of large and high-dimensional search spaces. This is done by performing informed sampling in the high-dimensional continuous space and incorporating potential information gain along edges in the reward estimation. This method rapidly generates a [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Zhe Huang

NSH 4305

Title: Distributed Reinforcement Learning for Autonomous Driving Abstract: Due to the complex and safety-critical nature of autonomous driving, recent works typically test their ideas on simulators designed for the very purpose of advancing self-driving research. Despite the convenience of modeling autonomous driving as a trajectory optimization problem, few of these methods resort to online reinforcement [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk- Xinjie Yao

NSH 4305

Title: Ride Comfort-Aware Visual Navigation via Self-Supervised Learning Abstract: Under shared autonomy, wheelchair users expect vehicles to provide safe and comfortable rides while following users’ high-level navigation plans. To find such a path, vehicles negotiate with different terrains and assess their traversal difficulty. Most prior works model surroundings either through geometric representations or semantic classifications, [...]

MSR Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

MS Thesis Talk – Shun Iwase

GHC 6501

Title: Fast 6D Object Pose Refinement via Deep Texture Rendering   Abstract: We present RePOSE, a fast iterative refinement method for 6D object pose estimation. Prior methods perform refinement by feeding zoomed-in input and rendered RGB images into a CNN and directly regressing an update of a refined pose. Their runtime is slow due to the [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Resource-Constrained Learning and Inference for Visual Perception

Abstract: We have witnessed rapid advancement across major computer vision benchmarks over the past years. However, the top solutions' hidden computation cost prevents them from being practically deployable. For example, training large models until convergence may be prohibitively expensive in practice, and autonomous driving or augmented reality may require a reaction time that rivals that [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Trajectory Optimization for Thermally-Actuated Soft Planar Robot Limbs

Abstract: Practical use of robotic manipulators made from soft materials requires generating and executing complex motions. We present the first approach for generating trajectories of a thermally-actuated soft robotic manipulator. Based on simplified approximations of the soft arm and its antagonistic shape-memory alloy actuator coils, we justify a dynamics model of a discretized rigid manipulator [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Physical Interaction and Manipulation of the Environment using Aerial Robots

Abstract: The physical interaction of aerial robots with their environment has countless potential applications and is an emerging area with many open challenges. Fully-actuated multirotors have been introduced to tackle some of these challenges. They provide complete control over position and orientation and eliminate the need for attaching a multi-DoF manipulation arm to the robot. [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis

NSH 3305

Abstract: Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g., NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known to be an under-constrained problem, and so methods rely on data-driven priors for reconstructing dynamic content. We replace these [...]

RI Seminar
Ross L. Hatton
Associate Professor
Robotics & Mechanical Engineering , Oregon State University

Snakes & Spiders, Robots & Geometry

Newell-Simon Hall 4305

Abstract: Locomotion and perception are a common thread between robotics and biology. Understanding these phenomena at a mechanical level involves nonlinear dynamics and the coordination of many degrees of freedom. In this talk, I will discuss geometric approaches to organizing this information in two problem domains: Undulatory locomotion of snakes and swimmers, and vibration propagation [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Combining vision-based tactile, proximity, and global sensing for robotic manipulation

Abstract: I will begin by describing our work on visual servoing a manipulator and localizing objects using a robot-mounted suite of vision and vision-based tactile sensors, our results, algorithms used, and lessons learned. We show that by collocating tactile, and global (e.g. an RGB(D) camera) sensors, our setup can perform better than using each type [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Visual Representation and Recognition without Human Supervision

NSH 4305

Abstract: The advent of deep learning based artificial perception models has revolutionized the field of computer vision. These methods take advantage of the ever growing computational capacity of machines and the abundance of human-annotated data to build supervised learners for a wide-range of visual tasks. However, the reliance on human-annotated is also a bottleneck for [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Design, Modeling and Control for a Tilt-rotor VTOL UAV in the Presence of Actuator Failure

Abstract: Providing both the vertical take-off and landing capabilities and the ability to fly long distances to aircraft opens the door to a wide range of new real-world aircraft applications while improving many existing applications. Tiltrotor vertical take-off and landing (VTOL) unmanned aerial vehicles (UAVs) are a better choice than fixed-wing and multirotor aircraft for [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Large Scale Dense 3D Reconstruction via Sparse Representations

Abstract: Scene reconstruction systems take in (3D) videos as input, and output 3D models with associated poses for inputs. With the demand of 3D content generation, the technique has been drastically evolving in recent years. For professionals equipped with depth sensors, efficient dense reconstruction systems have become available to efficiently recover scene geometry. For ordinary [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Learning Multi-Modal Navigation in Unstructured Environments

Abstract: A robot that operates efficiently in a team with humans in an unstructured outdoor environment must translate commands into actions from a modality intuitive to its operator. The robot must be able to perceive the world as humans do so that the actions taken by the robot reflect the nuances of natural language and [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Lessons Learned from Creating Low-Cost Dexterous Soft Robot Hands

NSH 4305

Abstract: Soft robot hands have shown promising results when it comes to dexterous grasping and manipulation. Compared to their rigid counterparts, soft hands can be manufactured for a fraction of the cost and offer robustness to uncertainty due to their inherent compliance. Unfortunately, the design and fabrication of soft robot hands is still a time-consuming [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Modern Trajectory Forecasting Methods Lack Social Awareness

NSH 4305

Abstract: We present a thorough evaluation and analysis of state-of-the-art (SOTA) human trajectory forecasting methods with respect to metrics for safe and socially-aware prediction, e.g., collision rate, in addition to traditional displacement metrics, e.g., average displacement error. First, we introduce a system for trajectory classification which is used to evaluate the strengths and weaknesses of [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

Vision-based Aircraft Detection and Tracking for Detect-and-Avoid

NSH 4305

Abstract: Detect-and-Avoid (DAA) capabilities are critical for autonomous operations of small unmanned aircraft systems (sUAS). Traditionally DAA systems for large aircraft have been ground and radar-based. Due to the size, weight, and power (SWaP) constraints of sUAS, current DAA systems rely mainly on vision-based sensors and ADS-B (Automatic Dependent Surveillance-Broadcast) transponders. However, not all flying [...]

PhD Thesis Proposal
Extern
Robotics Institute,
Carnegie Mellon University

Teaching Agent Reward Functions via Demonstrations for Human Inverse Reinforcement Learning

NSH 4305

Abstract: For intelligent agents (e.g. robots) to be seamlessly integrated into human society, humans must be able to understand their decision making. For example, the decision making of autonomous cars must be clear to the engineers certifying their safety, passengers riding them, and nearby drivers negotiating the road simultaneously. As an agent's decision making can [...]

Faculty Events

RI Council Meeting

Newell Simon Hall 4119

RI Council is a leadership group made up of the Director of RI, Academic Program Leads, Committee Chairs, and members at large as appointed by the Director. RI Council meets generally once a week to discuss department business.

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Learning to perform dynamic and interactive tasks using structural and algorithmic priors

NSH 3002

Abstract: Everyday human tasks such as picking up an object in one smooth motion, pushing a heavy door using the momentum of our bodies or pushing off a wall to quickly turn a corner involve complex dynamic interactions between the human and the environment, as well as switching dynamics when the robot makes and breaks [...]

Special Events

The Robotics Institute Semi-formal

All Robotics Institute faculty, students, visitors and staff are invited with to attend. One guest per person. RSVP required. Please check your emails for the e-vite and RSVP link. Please contact Debbie Tobin, dmz@cs.cmu.edu, with any questions.

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Simple Shape Descriptors for Retinal Surface Estimation using a Laser-Aiming Beam

Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Affective Robot Behavior Improves Learning in a Sorting Game

GHC 4405

Abstract: Nonverbal communication in the field of education can allow teachers to emotionally support their students and improve educational experience and performance. Robot nonverbal movements have been shown to improve both subjective experiences and task performance, and this work investigates whether affective robot behavior can improve human learning. This is tested using an online sorting [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Policy Decomposition: Approximate Optimal Control with Suboptimality Estimates

NSH 3305

Abstract: Optimal Control is a formulation for designing controllers for dynamical systems by posing it as an optimization problem, whereby the desired long-term behavior of the system is expressed using a cost function. The objective is to compute a policy, i.e. a mapping from the state of the system to its control inputs, that minimizes [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Audience-Aware Legibility for Social Navigation

Abstract: Robots often need to communicate their goals to humans when navigating in a shared space to assist observers in anticipating the robot’s future actions. These human observers are often scattered throughout the environment, and each observer only has a partial view of the robot and its movements. A path that non-verbally communicates with multiple [...]

Special Events

Commencement Celebration

The Pittsburgh Golf Club 5280 Northumberland Street, Pittsburgh, PA, United States

Special Events

CMU Community Picnic

As shared during President Jahanian’s recent town hall discussions, the CMU Community Picnic is returning on May 18 (11:30 am to 1:30 pm). The Office of Human Resources, in partnership with Staff Council and the Office of the President, sponsors and organizes this yearly celebration as a thank you for the hard work and contributions [...]

Faculty Events
Raj Reddy Assistant Professor in Robotics
Robotics Institute,
Carnegie Mellon University

Generalization for Robot Learning In The Wild

Newell-Simon Hall 4305

Abstract: How can we train a robot that can generalize to perform thousands of tasks in thousands of environments? This question underscores the holy grail of robot learning, more generally machine learning, research. Current AI systems are incredibly specific in that they only perform the tasks they are trained for and are miserable at generalization. [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

On Sample-Efficient Reinforcement Learning for Nuclear Fusion

NSH 4305

Abstract: In many practical applications of reinforcement learning (RL), it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, determining the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Learning Strategies to Solve Real-World Physics Puzzles

Abstract: In this talk, I focus on efficient online learning for solving real-world physics puzzles. I discuss challenges associated with learning in this domain and how those challenges inform certain design decisions. In particular, learning from scratch in the real world would be difficult. I present a practical mixture of experts framework for learning strategies [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Towards Modular and Differentiable Autonomous Driving

NSH 4305

Abstract: The classical "modular and cascaded" autonomy stack (object detection, tracking, trajectory prediction, then planning and control) has been widely used for interactive autonomous systems such as self-driving cars due to its interpretability and fast development cycle. In this thesis, we advocate the use of such a modular stack but improve its accuracy and robustness [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Towards reconstructing non-rigidity from single camera

Abstract: In this proposal, we study how to infer 3D from images captured by a single camera, without assuming the target scenes / objects being static. The non-static setting makes our problem ill-posed and challenging to solve, but is vital in practical applications where target-of-interest is non-static. To solve ill-posed problems, the current trend in [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Control Input and Natural Gaze for Goal Prediction in Shared Control

GHC 4405

Abstract: Teleoperated systems are used widely in deployed robots today, for such tasks as space exploration, disaster recovery, or assisted manipulation. However, teleoperated systems are difficult to control, especially when performing high-dimensional, contact-rich tasks like manipulation. One approach to ease teleoperated manipulation is shared control; this strategy combines the user's direct control input with an [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Forecasting from LiDAR via Future Object Detection

NSH 3305

Abstract: Object detection and forecasting are fundamental components of embodied perception. These two problems, however, are largely studied in isolation by the community. In this paper, we propose an end-to-end approach for detection and motion forecasting based on raw sensor measurement as opposed to ground truth tracks. Instead of predicting the current frame locations and [...]