VASC Seminar
Xavier Giro Nieto
Associate Professor
Universitat Politecnica de Catalunya

Open Challenges in Sign Language Translation & Production

Abstract: Machine translation and computer vision have greatly benefited of the advances in deep learning. The large and diverse amount of textual and visual data have been used to train neural networks whether in a supervised or self-supervised manner. Nevertheless, the convergence of the two field in sign language translation and production is still poses [...]

RI Seminar
Andrew E. Johnson
Principal Robotics Systems Engineer
NASA Jet Propulsion Laboratory, California Institute of Technology

The Search for Ancient Life on Mars Began with a Safe Landing

1305 Newell Simon Hall

Abstract: Prior mars rover missions have all landed in flat and smooth regions, but for the Mars 2020 mission, which is seeking signs of ancient life, this was no longer acceptable. To maximize the variety of rock samples that will eventually be returned to earth for analysis, the Perseverance rover needed to land in a [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Meshlet Primitives for Dense RGB-D SLAM in Dynamic Environments

Abstract: Dense RGB-D SLAM has been well established as a method for achieving robust localization while providing high quality dense surface reconstruction. However, despite significant progress, dense RGB-D SLAM has remained difficult to achieve on computationally constrained platforms, such as those used on autonomous aerial vehicles. A significant limiting factor in the current state of [...]

VASC Seminar
Ishan Misra
Research Scientist
Facebook AI Research

3D Recognition with self-supervised learning and generic architectures

Abstract: Supervised learning relies on manual labeling which scales poorly with the number of tasks and data. Manual labeling is especially cumbersome for 3D recognition tasks such as detection and segmentation and thus most 3D datasets are surprisingly small compared to image or video datasets. 3D recognition methods are also fragmented based on the type [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Heuristics for routing and scheduling of Spatio-temporal type problems in industrial environments

Abstract: Spatio-temporal problems are fairly common in industrial environments. In practice, these problems come with different characteristics and are often very hard to solve optimally. So, practitioners prefer to develop heuristics that exploit mathematical structure specific to the problem for obtaining good performance. In this thesis, we will present work on heuristics for 3 different [...]

PhD Thesis Proposal
Postdoctoral Fellow
Robotics Institute,
Carnegie Mellon University

Computational Light Transport with Interferometry

3305 Newell-Simon Hall

Abstract: Optical interferometry is the measurement of small, sub-wavelength distances by exploiting the wave nature of light. Due to its capability to resolve micron-scale displacements, it has found widespread applications in biomedical imaging, industrial fabrication, physics, and astrophysics. In this thesis, we introduce a set of techniques we call computational interferometry, that bring the benefits [...]

VASC Seminar
Deepak Pathak
Assistant Professor
Carnegie Mellon University

Rapid Adaptation for Robot Learning

Abstract: How can we train a robot to generalize to diverse environments? This question underscores the holy grail of robot learning research because it is difficult to supervise an agent for all possible situations it can encounter in the future. We posit that the only way to guarantee such a generalization is to continually learn and [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

3D Reconstruction using Differential Imaging

Abstract: 3D reconstruction has been at the core of many computer vision applications, including autonomous driving, visual inspection in manufacturing, and augmented and virtual reality (AR/VR). Despite the tremendous progress made over the years, there remain challenging open-research problems. This thesis addresses three such problems in 3D reconstruction. First, we address the problem of defocus [...]

RI Seminar
Systems Scientist
Robotics Institute,
Carnegie Mellon University

Robotic Cave Exploration for Search, Science, and Survey

1305 Newell Simon Hall

Abstract: Robotic cave exploration has the potential to create significant societal impact through facilitating search and rescue, in the fight against antibiotic resistance (science), and via mapping (survey). But many state-of-the-art approaches for active perception and autonomy in subterranean environments rely on disparate perceptual pipelines (e.g., pose estimation, occupancy modeling, hazard detection) that process the same underlying sensor data in [...]

VASC Seminar
Iasonas Kokkinos
Research Manager
Snap Inc, UCL

Humans, hands, and horses: 3D reconstruction of articulated object categories using strong, weak, and self-supervision

Abstract: Reconstructing 3D objects from a single 2D image is a task that humans perform effortlessly,  yet computer vision so far has only robustly solved 3D face reconstruction. In this talk we will see how we can extend the scope of monocular 3D reconstruction to more challenging, articulated categories such as human bodies, hands and [...]

RI Seminar
Thomas Howard
Assistant Professor of Electrical and Computer Engineering
Electrical & Computer Engineering, University of Rochester

Enabling Grounded Language Communication for Human-Robot Teaming

1305 Newell Simon Hall

Abstract:  The ability for robots to effectively understand natural language instructions and convey information about their observations and interactions with the physical world is highly dependent on the sophistication and fidelity of the robot’s representations of language, environment, and actions.  As we progress towards more intelligent systems that perform a wider range of tasks in a [...]

VASC Seminar
Alex Schwing
Assistant Professor
University of Illinois

Looking behind the Seen in Order to Anticipate

Abstract: Despite significant recent progress in computer vision and machine learning, personalized autonomous agents often still don’t participate robustly and safely across tasks in our environment. We think this is largely because they lack an ability to anticipate, which in turn is due to a missing understanding about what is happening behind the seen, i.e., [...]

RI Seminar
Matthew Walter
Assistant Professor
Robot Intelligence through Perception Lab (RIPL), Toyota Technological Institute at Chicago

Robots that Learn through Language

1305 Newell Simon Hall

Abstract: Advances in perception have been integral to transitioning robots from machines restricted to factory automation to autonomous agents that operate robustly in unstructured environments. As our surrogates, robots enable people to explore the deepest depths of the ocean and distant regions of space, making discoveries that would otherwise be impossible. The age of robots [...]

RI Seminar
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Towards Reconstructing Any Object in 3D

1305 Newell Simon Hall

Abstract: The world we live in is incredibly diverse, comprising of over 10k natural and man-made object categories. While the computer vision community has made impressive progress in classifying images from such diverse categories, the state-of-the-art 3D prediction systems are still limited to merely tens of object classes.  A key reason for this stark difference [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Beyond rigid objects: Data-driven Methods for Manipulation of Deformable Objects

Abstract: Manipulation of deformable objects challenges common assumptions made for rigid objects. Deformable objects have high intrinsic state representation and complex dynamics with high degrees of freedom, making it difficult for state estimation and planning. The completed work can be divided into two parts. In the first part, we explore reinforcement learning (RL) as a [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Simulation, Perception, and Generation of Human Behavior

Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout this thesis, we show how the three aspects are deeply connected and [...]

VASC Seminar
Serena Yeung
Assistant Professor
Stanford University

The Clinician’s AI Partner: Augmenting Clinician Capabilities Across the Spectrum of Healthcare

Abstract: Clinicians often work under highly demanding conditions to deliver complex care to patients. As our aging population grows and care becomes increasingly complex, physicians and nurses are now also experiencing feelings of burnout at unprecedented levels. In this talk, I will discuss possibilities for computer vision to function as a partner to clinicians, and to augment their capabilities, across [...]

RI Seminar
Siddharth Srivastava
Assistant Professor
School of Computing, Informatics, & Decision Systems Engineering, Arizona State University

The Unusual Effectiveness of Abstractions for Assistive AI

1305 Newell Simon Hall

Abstract: Can we balance efficiency and reliability while designing assistive AI systems? What would such AI systems need to provide? In this talk I will present some of our recent work addressing these questions. In particular, I will show that a few fundamental principles of abstraction are surprisingly effective in designing efficient and reliable AI [...]

VASC Seminar
Judy Hoffman
Assistant Professor
College of Computing, Georgia Tech

Reliable and Accessible Visual Recognition

Abstract: As visual recognition models are developed across diverse applications; we need the ability to reliably deploy our systems in a variety of environments. At the same time, visual models tend to be trained and evaluated on a static set of curated and annotated data which only represents a subset of the world. In this [...]

VASC Seminar
Tadas Baltrusaitis
Principal Scientist
Microsoft, Mixed Reality Cambridge

Fake It Till You Make It: Face analysis in the wild using synthetic data alone

Abstract: In this seminar I will demonstrate how synthetic data alone can be used to perform face-related computer vision in the wild. The community has long enjoyed the benefits of synthesizing training data with graphics, but the domain gap between real and synthetic data has remained a problem, especially for human faces. Researchers have tried [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Structured Learning for Robust Robot Manipulation

NSH 4305

Abstract: Robust and generalizable robots that can autonomously manipulate objects in semi-structured environments can bring material benefits to society. Data-driven learning approaches are crucial for enabling such systems by identifying and exploiting patterns in semi-structured environments, allowing robots to adapt to novel scenarios with minimal human supervision. However, despite significant prior work in learning for [...]

RI Seminar
Professor Emeritus
Robotics Institute,
Carnegie Mellon University

Robotics and Warehouse Automation at Berkshire Grey

1305 Newell Simon Hall

Abstract:  This talk tells the Berkshire Grey story, from its founding in 2013 to its IPO earlier this year — the first robotics IPO since iRobot over15 years ago.  Berkshire Grey produces automated systems for e-commerce order fulfillment, parcel sortation, store replenishment, and related operations in warehouses, distribution centers, and in the back ends of [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

An Experimental Design Perspective on Model-Based Reinforcement Learning

NSH 3305

Abstract: In many practical applications of RL, it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, computing the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or dollars of [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Learning Model Preconditions for Planning with Multiple Models

Abstract: Different models can provide differing levels of fidelity when a robot is planning. Analytical models are often fast to evaluate but only work in limited ranges of conditions. Meanwhile, physics simulators are effective at modeling complex interactions between objects but are typically more computationally expensive. Learning when to switch between the various models can [...]

VASC Seminar
Or Patashnik
Graduate Student
School of Computer Science at Tel-Aviv University

Leveraging StyleGAN for Image Editing and Manipulation

Abstract: StyleGAN has recently been established as the state-of-the-art unconditional generator, synthesizing images of phenomenal realism and fidelity, particularly for human faces. With its rich semantic space, many works have attempted to understand and control StyleGAN’s latent representations with the goal of performing image manipulations. To perform manipulations on real images, however, one must learn to [...]

RI Seminar
Sebastian Scherer & Matthew Travers
Robotics Institute, Carnegie Mellon University

Resilient Exploration in SubT Environments: Team Explorer’s Approach and Lessons Learned in the Final Event

1305 Newell Simon Hall

Abstract: Subterranean robot exploration is difficult with many mobility, communications, and navigation challenges that require an approach with a diverse set of systems, and reliable autonomy. While prior work has demonstrated partial successes in addressing the problem, here we convey a comprehensive approach to address the problem of subterranean exploration in a wide range of [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Simulation-based Planning for Pick-and-Place in Heavy Clutter using Non-prehensile Manipulation

NSH 3305

Abstract: Robot manipulation in domestic households, industrial manufacturing and warehouses might require contact-rich interactions with objects in the environment. For pick-and-place style grasping tasks in cluttered scenes, it can be more economical for the robot to rely on non-prehensile actions vis-à-vis deliberate prehensile rearrangement. Non-prehensile actions also let the robot manipulate large and bulky objects [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Relationships in instance segmentation and anomaly detection

GHC 4405

Abstract: This thesis primarily covers work on two different tasks in computer vision: (1) anomaly detection and (2) instance segmentation. Anomaly detection is an underexplored unsupervised problem that has existed in many fields. On the other hand, instance (and panoptic) segmentation is a supervised problem that can leverage the powerful data and key developments from [...]

VASC Seminar
Soumyadip Sengupta
Postdoctoral Research Associate
University of Washington

Next-Gen Video Communication

Abstract: Video communication connects our world. It is necessary in conducting business, educational and personal activities across different geographical locations. However, the quality of an average user’s video communication is dramatically worse than that of professionally created videos in news broadcasts, talk shows, and on YouTube. This is because professionally created videos are often captured with [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Learning with Diverse Forms of Imperfect and Indirect Supervision

Abstract: High capacity Machine Learning (ML) models trained on large, annotated datasets have driven impressive advances in several fields including natural language processing and computer vision, in turn leading to impactful applications of ML in areas such as healthcare, e-commerce, and predictive maintenance. However, obtaining annotated datasets at the scale required for training such models [...]

Student Talks

MRSD Annual Poster Presentation

Newell Simon Hall Atrium

Four student teams from the MRSD program will use posters, videos, and hardware to show their project work on robots for room disinfection, search & rescue, increasing human capability via a third arm, and increased-efficiency factory-floor obstacle avoidance.

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

3D Representation Learning for Perception and Prediction: A Modular Yet Highly Integrated Approach

Abstract: Modularized and cascaded autonomy stacks (object detection, then tracking and then trajectory prediction) have been widely adopted in many autonomous systems such as self-driving cars due to its interpretability. In this talk, I advocate the use of such a modular approach but improve its accuracy and robustness by developing different 3D representations for each [...]

MSR Thesis Defense
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk: Avi Rudich

1305 Newell Simon Hall

Title: Kinematic Analysis of 3D Printed Flexible Delta Robots   Abstract: Flexible Delta robots show significant promise for use in a wide array of manipulation tasks.  They are simple to design and manufacture, and they maintain a high level of repeatability and precision in open loop control.  This thesis analyzes the kinematic properties of flexible [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Reconstructing common objects to interact with

Abstract: We humans are able to understand 3D shapes of common daily objects and interact with them from a wide range of categories. We understand cups are usually cylinder-like and we can easily predict the shape of one particular cup, both in isolation or even when it is held by a human. We aim to [...]

VASC Seminar
Robert Collins
Associate Professor
Penn State University

Activity Understanding of Scripted Performances

Abstract: The PSU Taichi for Smart Health project has been doing a deep-dive into vision-based analysis of 24-form Yang-style Taichi (TaijiQuan). A key property of Taichi, shared by martial arts katas and prearranged form exercises in other sports, is practice of a scripted routine to build both mental and physical competence.  The scripted nature of routines [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Dynamical Model Learning and Inversion for Aggressive Quadrotor Flight

Abstract: Quadrotor applications have seen a surge recently and many tasks require precise and accurate controls. Flying fast is critical in many applications and the limited onboard power source makes completing tasks quickly even more important. Staying on a desired course while traveling at high speeds and high accelerations is difficult due to complex and [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Person Transfers Between Multiple Service Robots

NSH 3305

Abstract: As more service robots are deployed in the world, human-robot interaction will not be limited to one-to-one interactions between users and robots. Instead, users will likely have to interact with multiple robots, simultaneously or sequentially, throughout their day to receive services and complete different tasks. In this thesis, I describe work in which my [...]

PhD Speaking Qualifier
MSR Student
Robotics Institute,
Carnegie Mellon University

A causal framework to diagnose and fix issues with doors

Abstract: Many animals, such as ravens, (and a fortiori humans) exhibit a great deal of physical intelligence that allows them to solve complex multi-step physical puzzles. This ability indicates an understanding or a faculty to represent causality and mechanisms, understand when something goes wrong, and figure out how to deal with it. As a step [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Understanding Unbalanced Datasets Through Simple Models and Dataset Exploration

GHC 4405

Abstract: Computer vision models have proven to be tremendously capable of recognizing and detecting several classes and objects. They succeed in classes widely ranging in type and scale from humans to cans to pens. However, the best performing classes have abundant examples in large-scale datasets today. In unbalanced datasets, where some categories are seen in [...]

VASC Seminar
Vishal Patel
Associate Professor
Johns Hopkins University

Domain adaptive object detection

Abstract: Recent advances in deep learning have led to the development of accurate and efficient models for object detection. However, learning highly accurate models relies on the availability of large-scale annotated datasets. Due to this, model performance drops drastically when evaluated on label-scarce datasets having visually distinct images.  Domain adaptation tries to mitigate this degradation.  In [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Understanding, Exploiting and Improving Inter-view Relationships

NSH 3305

Abstract: Multi-view machine learning has garnered substantial attention in various applications over recent years. Many such applications involve learning on data obtained from multiple heterogeneous sources of information, for example, in multi-sensor systems such as self-driving cars, or monitoring intensive care patient vital signs at their bed-side. Learning models for such applications can often benefit [...]

RI Event
Project Scientist
Robotics Institute,
Carnegie Mellon University

Model-Centric Verification of Artificial Intelligence

Abstract: This work shows how provable guarantees can be used to supplement probabilistic estimates in the context of Artificial Intelligence (AI) systems. Statistical techniques measure the expected performance of a model, but low error rates say nothing about the ways in which errors manifest. Formal verification of model adherence to design specifications can yield certificates [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Designing Whisker Sensors to Detect Multiple Mechanical Stimuli for Robotic Applications

Abstract: Many mammals, such as rats and seals, use their whiskers as versatile mechanical sensors to gain precise information about their surroundings. Whisker-inspired sensors on robotic platforms have shown their potential benefit, improving applications ranging from drone navigation to texture mapping. Despite this, there is a gap between the engineered sensors and many of the [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Human-in-the-loop Control of Mobile Robots

Abstract: Human-in-the-loop control for mobile robots is an important aspect of robot operation, especially for navigation in unstructured environments or in the case of unexpected events. However, traditional paradigms of human-in-the-loop control have relied heavily on the human to provide precise and accurate control inputs to the robot, or reduced the role of the human [...]

VASC Seminar
Umberto Michieli
Postdoctoral Researcher and Adjunct Professor
University of Padua

Visual Understanding across Semantic Groups, Domains and Devices

Abstract: Deep neural networks often lack generalization capabilities to accommodate changes in the input/output domain distributions and, therefore, are inherently limited by the restricted visual and semantic information contained in the original training set. In this talk, we argue the importance of the versatility of deep neural architectures and we explore it from various perspectives.   [...]

RI Seminar
Stefanos Nikolaidis
Assistant Professor
Computer Science, University of Southern California

Towards Robust Human-Robot Interaction: A Quality Diversity Approach

Abstract: The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring the diverse scenarios of interaction between humans and robots in simulation can improve understanding of complex human-robot interaction systems and avoid potentially costly failures in real-world settings. [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Planning and Execution using Inaccurate Models with Provable Guarantees on Task Completeness

Abstract: Modern planning methods are effective in computing feasible and optimal plans for robotic tasks when given access to accurate dynamical models. However, robots operating in the real world often face situations that cannot be modeled perfectly before execution. Thus, we only have access to simplified but potentially inaccurate models. This imperfect modeling can lead [...]

VASC Seminar
Chao Chen
Assistant Professor
Stony Brook University

Topology-Driven Learning for Biomedical Imaging Informatics

Abstract: Thanks to decades of technology development, we are now able to visualize in high quality complex biomedical structures such as neurons, vessels, trabeculae and breast tissues. We need innovative methods to fully exploit these structures, which encode important information about underlying biological mechanisms. In this talk, we explain how topology, i.e., connected components, handles, loops, [...]

RI Seminar
Professor / Director of RI
Robotics Institute,
Carnegie Mellon University

Lessons from the Field: Deep Learning and Machine Perception for field robots

Abstract: Mobile robots now deliver vast amounts of sensor data from large unstructured environments. In attempting to process and interpret this data there are many unique challenges in bridging the gap between prerecorded data sets and the field. This talk will present recent work addressing the application of machine learning techniques to mobile robotic perception. [...]

VASC Seminar
Gianfranco Doretto
Associate Professor
West Virginia University

Learning generative representations for image distributions

Abstract: Autoencoder neural networks are an unsupervised technique for learning representations, which have been used effectively in many data domains. While capable of generating data, autoencoders have been inferior to other models like Generative Adversarial Networks (GAN’s) in their ability to generate image data. We will describe a general autoencoder architecture that addresses this limitation, and [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Self-Supervising Occlusions for Vision

Abstract: Virtually every scene has occlusions. Even a scene with a single object exhibits self-occlusions - a camera can only view one side of an object (left or right, front or back), or part of the object is outside the field of view. More complex occlusions occur when one or more objects block part(s) of [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Development of an Agile and Dexterous Balancing Mobile Manipulator Robot

Abstract: This thesis focuses on designing and controlling a dynamically stable shape-accelerating dual-arm mobile manipulator, the Carnegie Mellon University (CMU) ballbot. The CMU ballbot is a human-sized dynamically stable mobile robot that balances on a single spherical wheel. We describe the development of a pair of seven-degree-of-freedom (DOF) humanoid arms. The new 7-DOF arm pair [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Massively Parallelized Lazy Planning Algorithms

GHC 4405

Abstract: Search-based planning algorithms enable autonomous agents like robots to come up with well-reasoned long horizon plans to achieve a given task objective. They do so by optimizing a task-specific cost function while respecting the constraints on either the agent (e.g. motion constraints) or the environment (e.g. obstacles). In robotics, such as in motion planning [...]

VASC Seminar
Daniel McDuff
Principal Researcher
Microsoft Research

Building Intelligent and Visceral Machines: From Sensing to Application

Abstract: Humans have evolved to have highly adaptive behaviors that help us survive and thrive. As AI prompts a move from computing interfaces that are explicit and procedural to those that are implicit and intelligent, we are presented with extraordinary opportunities. In this talk, I will argue that understanding affective and behavioral signals presents many opportunities [...]

VASC Seminar
Arun Mallya
Senior Research Scientist
NVIDIA

GANcraft – an unsupervised 3D neural method for world-to-world translation

Abstract: Advances in 2D image-to-image translation methods, such as SPADE/GauGAN, have enabled users to paint photorealistic images by drawing simple sketches similar to those created in Microsoft Paint. Despite these innovations, creating a realistic 3D scene remains a painstaking task, out of the reach of most people. It requires years of expertise, professional software, a library [...]

PhD Thesis Proposal
PhD Student
Robotics Institute,
Carnegie Mellon University

Run-Time Optimization in the Deep Learning Age

Abstract: In a recovery task one seeks to obtain an estimate of an unknown signal from a set of incomplete measurements. These problems arise in a number of computer vision applications, from image based tasks such as super-resolution and in-painting to 3D reconstruction tasks such as Non-Rigid Structure from Motion and scene flow estimation. Early [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

System Identification and Control of Multiagent Systems Through Interactions

GHC 6501

Abstract: This thesis investigates the problem of identifying dynamics models of individual agents of a multiagent system (MAS) and exploiting these models to shape their behavior using robots extrinsic to the MAS. While task-based control of a MAS using onboard controllers of its agents is well studied, we investigate (a) how easy it is for [...]

Faculty Events
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Human-in-the-loop Model Creation

Newell-Simon Hall 4305

Abstract: Modern machine learning systems have made astonishing progress in automating labor-intensive tasks such as visual recognition and machine translation. While ML systems complete these tasks better and faster, humans are largely left behind. Indeed, most humans are entirely excluded from the creation process of machine learning models, except for tedious data annotation.   In [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Learning and Inference in Factor Graphs with Applications to Tactile Perception

Abstract: Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in hand [...]

VASC Seminar
Deqing Sun
Senior Research Scientist
Google

Learning Optical Flow: Model, Data, and Applications

Abstract: Optical flow provides important information about the dynamic world and is of fundamental importance to many tasks. In this talk, I will present my work on different aspects of learning optical flow. I will start with the background and talk about PWC-Net, a compact and effective model built using classical principles for optical flow. Next, [...]

Faculty Events
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Hands-On Interactions

Newell-Simon Hall 3305

Abstract: Our sense of touch is present in almost all our interactions with the world, from providing us with the feedback necessary to perceive and manipulate objects without having to look at them, to allowing our limbs to move and walk without us having to think about how to take the next step. We use [...]

RI Seminar
Leila Bridgeman
Assistant Professor of Mechanical Engineering & Materials Science
Duke University

Distributed Dissipativity: Applying Foundational Stability Theory to Modern Networked Control

Abstract: Despite its diverse areas of application, the desire to optimize performance and guarantee acceptable behaviour in the face of inevitable uncertainty is pervasive throughout control theory. This creates a fundamental challenge since the necessity of robustly stable control schemes often favors conservative designs, while the desire to optimize performance typically demands the opposite. While [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Towards Complex Robot Motions with Reinforcement Learning

Abstract: Reinforcement learning has shown to be a powerful tool for decision-making problems. In this talk, we present the opportunities and challenges of enabling increasingly complex robot behavior with reinforcement learning. First, we present a system that combines reinforcement learning and extrinsic dexterity to solve a novel task of “occluded grasping”. To reach an occluded [...]

RI Seminar
Assistant Professor
Robotics Institute,
Carnegie Mellon University

Haptic Perspective-taking from Vision and Force

Abstract: Physically collaborative robots present an opportunity to positively impact society across many domains. However, robots currently lack the ability to infer how their actions physically affect people. This is especially true for robotic caregiving tasks that involve manipulating deformable cloth around the human body, such as dressing and bathing assistance. In this talk, I [...]

VASC Seminar
Chen Sun
Assistant Professor, Computer Science
Brown University

Do Vision-Language Pretrained Models Learn Spatiotemporal Primitive Concepts?

Abstract:  Vision-language models pretrained on web-scale data have revolutionized deep learning in the last few years. They have demonstrated strong transfer learning performance on a wide range of tasks, even under the "zero-shot" setup, where text "prompts" serve as a natural interface for humans to specify a task, as opposed to collecting labeled data. These models are [...]

Faculty Events

RI Council Meeting

Newell Simon Hall 4119

RI Council is a leadership group made up of the Director of RI, Academic Program Leads, Committee Chairs, and members at large as appointed by the Director. RI Council meets generally once a week to discuss department business.

RI Seminar
Jing Xiao
Professor and Department Head
Robotics Engineering Department, Worcester Polytechnic Institute (WPI)

Perception-Action Synergy in Uncertain Environments

Abstract: Many robotic applications require a robot to operate in an environment with unknowns or uncertainty, at least initially, before it gathers enough information about the environment. In such a case, a robot must rely on sensing and perception to feel its way around. Moreover, it has to couple sensing/perception and motion synergistically in real [...]

PhD Thesis Proposal
Robotics Institute,
Carnegie Mellon University

Driving Reconfigurable Unmanned Vehicle Design for Mobility Performance

Abstract: Unmanned ground vehicles are being deployed in increasingly diverse and complex environments. Advances in the field of robotics, including perception technology, computing power, and machine learning, have brought robots from the lab to the real world. Remote and autonomous vehicles are now used to explore volcanoes, caves, pipes, war zones, disaster sites, and even [...]

VASC Seminar
Dr. Randall Balestriero
Post-Doctorate Researcher
Meta AI

Max-Affine Spline Insights into Deep Learning

Abstract:  We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs) that provide a powerful portal through which we view and analyze their inner workings. For instance, [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Search-based Path Planning for a High Dimensional Manipulator in Cluttered Environments Using Optimization-based Primitives

Abstract: In this work we tackle the path planning problem for a 21-dimensional snake robot-like, navigating a cluttered gas turbine for the purposes of inspection. Heuristic search-based approaches are effective planning strategies for common manipulation domains. However, their performance on high-dimensional systems is heavily reliant on the effectiveness of the action space and the heuristics [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Vision-Based Tactile Sensor Design using Physics Based Rendering

GHC 8102

Abstract: Tactile sensing has seen a rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and human-robot interaction. However, the simulation of tactile sensors is still a challenge. Simulation is a critical tool in the development of robotic systems. In [...]

SCS Distinguished Lecture
Jeannette Bohg
Assistant Professor of Computer Science
Stanford University

Teruko Yata Memorial Lecture

Rashid Auditorium - 4401 Gates and Hillman Centers

Leveraging Language and Video Demonstrations for Learning Robot Manipulation Skills and Enabling Closed-Loop Task Planning Humans have gradually developed language, mastered complex motor skills, created and utilized sophisticated tools. The act of conceptualization is fundamental to these abilities because it allows humans to mentally represent, summarize and abstract diverse knowledge and skills. By means of [...]

Staff Events

Robotics Institute Staff Offices 12PM Early Dismissal

Dear RI Faculty and Staff, In observance of the coming holiday, institute staff offices will close Friday, April 15th at noon.  They will reopen at 8:30 on Monday, April 18th. Happy Holiday! Thank you – Debbie Z. =================================================== Deborah H. Zalewski, Senior Associate Business Manager | The Robotics Institute - Carnegie Mellon University | Newell-Simon [...]

Faculty Events

RI Hiring Meeting

Newell Simon Hall 4119

A faculty hiring meeting to discuss candidates for faculty position

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Unified Simulation, Perception, and Generation of Human Behavior

Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Kernel Density Decision Trees

Abstract We propose kernel density decision trees (KDDTs), a novel fuzzy decision tree (FDT) formalism based on kernel density estimation that improves the robustness of decision trees and ensembles and offers additional utility. FDTs mitigate the sensitivity of decision trees to uncertainty by representing uncertainty through fuzzy partitions. However, compared to conventional, crisp decision trees, [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Energy-based Joint Pose Estimation for 3D Reconstruction

Abstract: In this talk, I will describe a data-driven method for inferring camera poses given a sparse collection of images of an arbitrary object. This task is a core component of classic geometric pipelines such as structure-from-motion (SFM), and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction. [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

NeRF for Robotics

GHC 8102

Abstract: In this talk I'll describe how recent advances in neural rendering and novel view synthesis - namely NeRF - can be leveraged by robotic agents to improve performance in manipulation tasks. Specifically, I'll argue that NeRF can enable robotic policies to: (1) generalize to new viewpoints; (2) perceive specular and reflective surfaces in a [...]

PhD Thesis Defense
Robotics Institute,
Carnegie Mellon University

Search Algorithms and Search Spaces for Neural Architecture Search

NSH 4305

Abstract: Neural architecture search (NAS) is recently proposed to automate the process of designing network architectures. Instead of manually designing network architectures, NAS automatically finds the optimal architecture in a data-driven way. Despite its impressive progress, NAS is still far from being widely adopted as a common paradigm for architecture design in practice. This thesis [...]

MSR Speaking Qualifier
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Evan Harber

Title: Stiffness Mapping of Deformable Objects Through Supervised Embedding and Gaussian Process Regression   Abstract: The stiffness map of a deformable object stores information about that object's surface compliance. Thus, through a stiffness map, we gain insight into the physical properties of that object. Depending on the object, an understanding of stiffness has applications ranging [...]

RI Seminar
Kirstin H. Petersen
Assistant Professor
Electrical & Computer Engineering, College of Engineering, Cornell University

Designing Robotic Systems with Collective Embodied Intelligence

Abstract: Natural swarms exhibit sophisticated colony-level behaviors with remarkable scalability and error tolerance. Their evolutionary success stems from more than just intelligent individuals, it hinges on their morphology, their physical interactions, and the way they shape and leverage their environment. Mound-building termites, for instance, are believed to use their own body as a template for [...]

MSR Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

MSR Thesis Talk – Gaurav Parmar

NSH 1109

Title: Spatially-Adaptive Multilayer GAN Inversion   Abstract: Existing GAN inversion and editing methods are well suited for only a target images that contain aligned objects with a clean background, such as portraits and animal faces, but often struggle for more difficult categories with complex scene layouts and object occlusions, such as cars, animals, and outdoor images. [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Robust Reinforcement Learning via Genetic Curriculum

GHC 6501

Abstract: Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While [...]

PhD Speaking Qualifier
PhD Student
Robotics Institute,
Carnegie Mellon University

Mouth Haptics in VR using a Headset Ultrasound Phased Array

GHC 7501

Abstract: This talk is the same one I will be presenting at the ACM CHI Conference on Human Factors in Computing Systems on May 2nd. Paper abstract: Today’s consumer virtual reality (VR) systems offer limited haptic feedback via vibration motors in handheld controllers. Rendering haptics to other parts of the body is an open challenge, [...]