Learning generative representations for image distributions
Abstract: Autoencoder neural networks are an unsupervised technique for learning representations, which have been used effectively in many data domains. While capable of generating data, autoencoders have been inferior to other models like Generative Adversarial Networks (GAN’s) in their ability to generate image data. We will describe a general autoencoder architecture that addresses this limitation, and [...]
Carnegie Mellon University
Self-Supervising Occlusions for Vision
Abstract: Virtually every scene has occlusions. Even a scene with a single object exhibits self-occlusions - a camera can only view one side of an object (left or right, front or back), or part of the object is outside the field of view. More complex occlusions occur when one or more objects block part(s) of [...]
Carnegie Mellon University
Development of an Agile and Dexterous Balancing Mobile Manipulator Robot
Abstract: This thesis focuses on designing and controlling a dynamically stable shape-accelerating dual-arm mobile manipulator, the Carnegie Mellon University (CMU) ballbot. The CMU ballbot is a human-sized dynamically stable mobile robot that balances on a single spherical wheel. We describe the development of a pair of seven-degree-of-freedom (DOF) humanoid arms. The new 7-DOF arm pair [...]
Carnegie Mellon University
Massively Parallelized Lazy Planning Algorithms
Abstract: Search-based planning algorithms enable autonomous agents like robots to come up with well-reasoned long horizon plans to achieve a given task objective. They do so by optimizing a task-specific cost function while respecting the constraints on either the agent (e.g. motion constraints) or the environment (e.g. obstacles). In robotics, such as in motion planning [...]
Building Intelligent and Visceral Machines: From Sensing to Application
Abstract: Humans have evolved to have highly adaptive behaviors that help us survive and thrive. As AI prompts a move from computing interfaces that are explicit and procedural to those that are implicit and intelligent, we are presented with extraordinary opportunities. In this talk, I will argue that understanding affective and behavioral signals presents many opportunities [...]
GANcraft – an unsupervised 3D neural method for world-to-world translation
Abstract: Advances in 2D image-to-image translation methods, such as SPADE/GauGAN, have enabled users to paint photorealistic images by drawing simple sketches similar to those created in Microsoft Paint. Despite these innovations, creating a realistic 3D scene remains a painstaking task, out of the reach of most people. It requires years of expertise, professional software, a library [...]
Run-Time Optimization in the Deep Learning Age
Abstract: In a recovery task one seeks to obtain an estimate of an unknown signal from a set of incomplete measurements. These problems arise in a number of computer vision applications, from image based tasks such as super-resolution and in-painting to 3D reconstruction tasks such as Non-Rigid Structure from Motion and scene flow estimation. Early [...]
Carnegie Mellon University
System Identification and Control of Multiagent Systems Through Interactions
Abstract: This thesis investigates the problem of identifying dynamics models of individual agents of a multiagent system (MAS) and exploiting these models to shape their behavior using robots extrinsic to the MAS. While task-based control of a MAS using onboard controllers of its agents is well studied, we investigate (a) how easy it is for [...]
Human-in-the-loop Model Creation
Abstract: Modern machine learning systems have made astonishing progress in automating labor-intensive tasks such as visual recognition and machine translation. While ML systems complete these tasks better and faster, humans are largely left behind. Indeed, most humans are entirely excluded from the creation process of machine learning models, except for tedious data annotation. In [...]
Carnegie Mellon University
Learning and Inference in Factor Graphs with Applications to Tactile Perception
Abstract: Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in hand [...]
Learning Optical Flow: Model, Data, and Applications
Abstract: Optical flow provides important information about the dynamic world and is of fundamental importance to many tasks. In this talk, I will present my work on different aspects of learning optical flow. I will start with the background and talk about PWC-Net, a compact and effective model built using classical principles for optical flow. Next, [...]
Hands-On Interactions
Abstract: Our sense of touch is present in almost all our interactions with the world, from providing us with the feedback necessary to perceive and manipulate objects without having to look at them, to allowing our limbs to move and walk without us having to think about how to take the next step. We use [...]
Distributed Dissipativity: Applying Foundational Stability Theory to Modern Networked Control
Abstract: Despite its diverse areas of application, the desire to optimize performance and guarantee acceptable behaviour in the face of inevitable uncertainty is pervasive throughout control theory. This creates a fundamental challenge since the necessity of robustly stable control schemes often favors conservative designs, while the desire to optimize performance typically demands the opposite. While [...]
Towards Complex Robot Motions with Reinforcement Learning
Abstract: Reinforcement learning has shown to be a powerful tool for decision-making problems. In this talk, we present the opportunities and challenges of enabling increasingly complex robot behavior with reinforcement learning. First, we present a system that combines reinforcement learning and extrinsic dexterity to solve a novel task of “occluded grasping”. To reach an occluded [...]
Haptic Perspective-taking from Vision and Force
Abstract: Physically collaborative robots present an opportunity to positively impact society across many domains. However, robots currently lack the ability to infer how their actions physically affect people. This is especially true for robotic caregiving tasks that involve manipulating deformable cloth around the human body, such as dressing and bathing assistance. In this talk, I [...]
Do Vision-Language Pretrained Models Learn Spatiotemporal Primitive Concepts?
Abstract: Vision-language models pretrained on web-scale data have revolutionized deep learning in the last few years. They have demonstrated strong transfer learning performance on a wide range of tasks, even under the "zero-shot" setup, where text "prompts" serve as a natural interface for humans to specify a task, as opposed to collecting labeled data. These models are [...]
RI Council Meeting
RI Council is a leadership group made up of the Director of RI, Academic Program Leads, Committee Chairs, and members at large as appointed by the Director. RI Council meets generally once a week to discuss department business.
Perception-Action Synergy in Uncertain Environments
Abstract: Many robotic applications require a robot to operate in an environment with unknowns or uncertainty, at least initially, before it gathers enough information about the environment. In such a case, a robot must rely on sensing and perception to feel its way around. Moreover, it has to couple sensing/perception and motion synergistically in real [...]
Carnegie Mellon University
Driving Reconfigurable Unmanned Vehicle Design for Mobility Performance
Abstract: Unmanned ground vehicles are being deployed in increasingly diverse and complex environments. Advances in the field of robotics, including perception technology, computing power, and machine learning, have brought robots from the lab to the real world. Remote and autonomous vehicles are now used to explore volcanoes, caves, pipes, war zones, disaster sites, and even [...]
Max-Affine Spline Insights into Deep Learning
Abstract: We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs) that provide a powerful portal through which we view and analyze their inner workings. For instance, [...]
Search-based Path Planning for a High Dimensional Manipulator in Cluttered Environments Using Optimization-based Primitives
Abstract: In this work we tackle the path planning problem for a 21-dimensional snake robot-like, navigating a cluttered gas turbine for the purposes of inspection. Heuristic search-based approaches are effective planning strategies for common manipulation domains. However, their performance on high-dimensional systems is heavily reliant on the effectiveness of the action space and the heuristics [...]
Vision-Based Tactile Sensor Design using Physics Based Rendering
Abstract: Tactile sensing has seen a rapid adoption with the advent of vision-based tactile sensors. Vision-based tactile sensors provide high resolution, compact and inexpensive data to perform precise in-hand manipulation and human-robot interaction. However, the simulation of tactile sensors is still a challenge. Simulation is a critical tool in the development of robotic systems. In [...]
Teruko Yata Memorial Lecture
Leveraging Language and Video Demonstrations for Learning Robot Manipulation Skills and Enabling Closed-Loop Task Planning Humans have gradually developed language, mastered complex motor skills, created and utilized sophisticated tools. The act of conceptualization is fundamental to these abilities because it allows humans to mentally represent, summarize and abstract diverse knowledge and skills. By means of [...]
Details to Follow . . .
Details to Follow . . .
Robotics Institute Staff Offices 12PM Early Dismissal
Dear RI Faculty and Staff, In observance of the coming holiday, institute staff offices will close Friday, April 15th at noon. They will reopen at 8:30 on Monday, April 18th. Happy Holiday! Thank you – Debbie Z. =================================================== Deborah H. Zalewski, Senior Associate Business Manager | The Robotics Institute - Carnegie Mellon University | Newell-Simon [...]
RI Hiring Meeting
A faculty hiring meeting to discuss candidates for faculty position
Carnegie Mellon University
Unified Simulation, Perception, and Generation of Human Behavior
Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and [...]
Kernel Density Decision Trees
Abstract We propose kernel density decision trees (KDDTs), a novel fuzzy decision tree (FDT) formalism based on kernel density estimation that improves the robustness of decision trees and ensembles and offers additional utility. FDTs mitigate the sensitivity of decision trees to uncertainty by representing uncertainty through fuzzy partitions. However, compared to conventional, crisp decision trees, [...]
Energy-based Joint Pose Estimation for 3D Reconstruction
Abstract: In this talk, I will describe a data-driven method for inferring camera poses given a sparse collection of images of an arbitrary object. This task is a core component of classic geometric pipelines such as structure-from-motion (SFM), and also serves as a vital pre-processing requirement for contemporary neural approaches (e.g. NeRF) to object reconstruction. [...]
NeRF for Robotics
Abstract: In this talk I'll describe how recent advances in neural rendering and novel view synthesis - namely NeRF - can be leveraged by robotic agents to improve performance in manipulation tasks. Specifically, I'll argue that NeRF can enable robotic policies to: (1) generalize to new viewpoints; (2) perceive specular and reflective surfaces in a [...]
Carnegie Mellon University
Search Algorithms and Search Spaces for Neural Architecture Search
Abstract: Neural architecture search (NAS) is recently proposed to automate the process of designing network architectures. Instead of manually designing network architectures, NAS automatically finds the optimal architecture in a data-driven way. Despite its impressive progress, NAS is still far from being widely adopted as a common paradigm for architecture design in practice. This thesis [...]
Carnegie Mellon University
MSR Thesis Talk – Evan Harber
Title: Stiffness Mapping of Deformable Objects Through Supervised Embedding and Gaussian Process Regression Abstract: The stiffness map of a deformable object stores information about that object's surface compliance. Thus, through a stiffness map, we gain insight into the physical properties of that object. Depending on the object, an understanding of stiffness has applications ranging [...]
Designing Robotic Systems with Collective Embodied Intelligence
Abstract: Natural swarms exhibit sophisticated colony-level behaviors with remarkable scalability and error tolerance. Their evolutionary success stems from more than just intelligent individuals, it hinges on their morphology, their physical interactions, and the way they shape and leverage their environment. Mound-building termites, for instance, are believed to use their own body as a template for [...]
MSR Thesis Talk – Gaurav Parmar
Title: Spatially-Adaptive Multilayer GAN Inversion Abstract: Existing GAN inversion and editing methods are well suited for only a target images that contain aligned objects with a clean background, such as portraits and animal faces, but often struggle for more difficult categories with complex scene layouts and object occlusions, such as cars, animals, and outdoor images. [...]
Robust Reinforcement Learning via Genetic Curriculum
Abstract: Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While [...]
Mouth Haptics in VR using a Headset Ultrasound Phased Array
Abstract: This talk is the same one I will be presenting at the ACM CHI Conference on Human Factors in Computing Systems on May 2nd. Paper abstract: Today’s consumer virtual reality (VR) systems offer limited haptic feedback via vibration motors in handheld controllers. Rendering haptics to other parts of the body is an open challenge, [...]
Towards Large-scale and Long-term Neural Map Representations
Abstract: We address the problem of large-scale and long-term neural map representations. Maps, as our prior understanding toward the environment, provide valuable information for modern robotics applications such as autonomous driving and AR/VR. The size of maps largely affects the end task performance: usually a more detailed map can support better performance, but would cost [...]
Carnegie Mellon University
Self-Improving 3D Scene Representations
Abstract: Most computer vision models in deployment today are not continually learning. Instead, they are in a “test” mode, where they will behave the same way perpetually, until they are replaced by newer models. This is a problem, because it means the models may perform poorly as soon as their “test” environment diverges from their [...]
Carnegie Mellon University
MSR Thesis Talk – Manash Pratim Das
Title: Model-Accuracy Aware Anytime Planning with Simulation Verification for Navigating Complex Terrains Abstract: Off-road and unstructured environments often contain complex patches of various types of terrain, rough elevation changes, deformable objects, etc. An autonomous ground vehicle traversing such environments experiences physical interactions that are extremely hard to model at scale and thus very hard to [...]
Carnegie Mellon University
MSR Thesis Talk – Akshay Dharamavaram
Title: Stabilizing the Training Dynamics of Generative Models using Self-Supervision Abstract: Generative Models have been shown to be adept in mimicking the behavior of an unknown distribution solely from bootstrapped data. However, deep learning models have been shown to overfit in either the minimization or maximization stage of the two player min-max game, resulting [...]
Carnegie Mellon University
Direct-drive Hands: Making Robot Hands Transparent and Reactive to Contacts
Abstract: Industrial manipulators and end-effectors are a vital driver of the automation revolution. These robot hands, designed to reject disturbances with stiffness and strength, are inferior to their human counterparts. Human hands are dexterous and nimble effectors capable of a variety of interactions with the environment. Through this thesis we wish to answer a question: [...]
Carnegie Mellon University
MSR Thesis Talk – Vivek Roy
Title: Smartphone localization for Indoor Pedestrian Navigation Abstract: Global positioning system (GPS) interfacing with applications such as Google Maps has proven very useful for navigation in outdoor open settings. However in crowded metropolitan environments with high rise buildings or in indoor settings, GPS quickly becomes unreliable. Using sensors found on commodity smartphones to perform accurate [...]
Understanding 3D Scenes and Interacting Hands
Abstract: Abstract: The long-term goal of my research is to help computers understand the physical world from images, including both 3D properties and how humans or robots could interact with things. This talk will summarize two recent directions aimed at enabling this goal. I will begin with learning to reconstruct full 3D scenes, including [...]
Manipulating Objects with Challenging Visual and Geometric Properties
Abstract: Object manipulation is a well-studied domain in robotics, yet manipulation remains difficult for objects with visually and geometrically challenging properties. Visually challenging properties, such as transparency and specularity, break assumptions of Lambertian reflectance that existing methods rely on for grasp estimation. On the other hand, deformable objects such as cloth pose both visual and [...]
TIGRIS: An Informed Sampling-based Algorithm for Informative Path Planning
Abstract: In this talk I will present our sampling-based approach to informative path planning that allows us to tackle the challenges of large and high-dimensional search spaces. This is done by performing informed sampling in the high-dimensional continuous space and incorporating potential information gain along edges in the reward estimation. This method rapidly generates a [...]
Carnegie Mellon University
MSR Thesis Talk – Zhe Huang
Title: Distributed Reinforcement Learning for Autonomous Driving Abstract: Due to the complex and safety-critical nature of autonomous driving, recent works typically test their ideas on simulators designed for the very purpose of advancing self-driving research. Despite the convenience of modeling autonomous driving as a trajectory optimization problem, few of these methods resort to online reinforcement [...]
Carnegie Mellon University
MSR Thesis Talk- Xinjie Yao
Title: Ride Comfort-Aware Visual Navigation via Self-Supervised Learning Abstract: Under shared autonomy, wheelchair users expect vehicles to provide safe and comfortable rides while following users’ high-level navigation plans. To find such a path, vehicles negotiate with different terrains and assess their traversal difficulty. Most prior works model surroundings either through geometric representations or semantic classifications, [...]
MS Thesis Talk – Shun Iwase
Title: Fast 6D Object Pose Refinement via Deep Texture Rendering Abstract: We present RePOSE, a fast iterative refinement method for 6D object pose estimation. Prior methods perform refinement by feeding zoomed-in input and rendered RGB images into a CNN and directly regressing an update of a refined pose. Their runtime is slow due to the [...]
Carnegie Mellon University
Resource-Constrained Learning and Inference for Visual Perception
Abstract: We have witnessed rapid advancement across major computer vision benchmarks over the past years. However, the top solutions' hidden computation cost prevents them from being practically deployable. For example, training large models until convergence may be prohibitively expensive in practice, and autonomous driving or augmented reality may require a reaction time that rivals that [...]
Trajectory Optimization for Thermally-Actuated Soft Planar Robot Limbs
Abstract: Practical use of robotic manipulators made from soft materials requires generating and executing complex motions. We present the first approach for generating trajectories of a thermally-actuated soft robotic manipulator. Based on simplified approximations of the soft arm and its antagonistic shape-memory alloy actuator coils, we justify a dynamics model of a discretized rigid manipulator [...]
Carnegie Mellon University
Physical Interaction and Manipulation of the Environment using Aerial Robots
Abstract: The physical interaction of aerial robots with their environment has countless potential applications and is an emerging area with many open challenges. Fully-actuated multirotors have been introduced to tackle some of these challenges. They provide complete control over position and orientation and eliminate the need for attaching a multi-DoF manipulation arm to the robot. [...]
Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
Abstract: Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g., NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known to be an under-constrained problem, and so methods rely on data-driven priors for reconstructing dynamic content. We replace these [...]
Snakes & Spiders, Robots & Geometry
Abstract: Locomotion and perception are a common thread between robotics and biology. Understanding these phenomena at a mechanical level involves nonlinear dynamics and the coordination of many degrees of freedom. In this talk, I will discuss geometric approaches to organizing this information in two problem domains: Undulatory locomotion of snakes and swimmers, and vibration propagation [...]
Combining vision-based tactile, proximity, and global sensing for robotic manipulation
Abstract: I will begin by describing our work on visual servoing a manipulator and localizing objects using a robot-mounted suite of vision and vision-based tactile sensors, our results, algorithms used, and lessons learned. We show that by collocating tactile, and global (e.g. an RGB(D) camera) sensors, our setup can perform better than using each type [...]
Carnegie Mellon University
Visual Representation and Recognition without Human Supervision
Abstract: The advent of deep learning based artificial perception models has revolutionized the field of computer vision. These methods take advantage of the ever growing computational capacity of machines and the abundance of human-annotated data to build supervised learners for a wide-range of visual tasks. However, the reliance on human-annotated is also a bottleneck for [...]
Design, Modeling and Control for a Tilt-rotor VTOL UAV in the Presence of Actuator Failure
Abstract: Providing both the vertical take-off and landing capabilities and the ability to fly long distances to aircraft opens the door to a wide range of new real-world aircraft applications while improving many existing applications. Tiltrotor vertical take-off and landing (VTOL) unmanned aerial vehicles (UAVs) are a better choice than fixed-wing and multirotor aircraft for [...]
Large Scale Dense 3D Reconstruction via Sparse Representations
Abstract: Scene reconstruction systems take in (3D) videos as input, and output 3D models with associated poses for inputs. With the demand of 3D content generation, the technique has been drastically evolving in recent years. For professionals equipped with depth sensors, efficient dense reconstruction systems have become available to efficiently recover scene geometry. For ordinary [...]
Carnegie Mellon University
Learning Multi-Modal Navigation in Unstructured Environments
Abstract: A robot that operates efficiently in a team with humans in an unstructured outdoor environment must translate commands into actions from a modality intuitive to its operator. The robot must be able to perceive the world as humans do so that the actions taken by the robot reflect the nuances of natural language and [...]
Lessons Learned from Creating Low-Cost Dexterous Soft Robot Hands
Abstract: Soft robot hands have shown promising results when it comes to dexterous grasping and manipulation. Compared to their rigid counterparts, soft hands can be manufactured for a fraction of the cost and offer robustness to uncertainty due to their inherent compliance. Unfortunately, the design and fabrication of soft robot hands is still a time-consuming [...]
Modern Trajectory Forecasting Methods Lack Social Awareness
Abstract: We present a thorough evaluation and analysis of state-of-the-art (SOTA) human trajectory forecasting methods with respect to metrics for safe and socially-aware prediction, e.g., collision rate, in addition to traditional displacement metrics, e.g., average displacement error. First, we introduce a system for trajectory classification which is used to evaluate the strengths and weaknesses of [...]
Carnegie Mellon University
Vision-based Aircraft Detection and Tracking for Detect-and-Avoid
Abstract: Detect-and-Avoid (DAA) capabilities are critical for autonomous operations of small unmanned aircraft systems (sUAS). Traditionally DAA systems for large aircraft have been ground and radar-based. Due to the size, weight, and power (SWaP) constraints of sUAS, current DAA systems rely mainly on vision-based sensors and ADS-B (Automatic Dependent Surveillance-Broadcast) transponders. However, not all flying [...]
Teaching Agent Reward Functions via Demonstrations for Human Inverse Reinforcement Learning
Abstract: For intelligent agents (e.g. robots) to be seamlessly integrated into human society, humans must be able to understand their decision making. For example, the decision making of autonomous cars must be clear to the engineers certifying their safety, passengers riding them, and nearby drivers negotiating the road simultaneously. As an agent's decision making can [...]
RI Council Meeting
RI Council is a leadership group made up of the Director of RI, Academic Program Leads, Committee Chairs, and members at large as appointed by the Director. RI Council meets generally once a week to discuss department business.
Learning to perform dynamic and interactive tasks using structural and algorithmic priors
Abstract: Everyday human tasks such as picking up an object in one smooth motion, pushing a heavy door using the momentum of our bodies or pushing off a wall to quickly turn a corner involve complex dynamic interactions between the human and the environment, as well as switching dynamics when the robot makes and breaks [...]
The Robotics Institute Semi-formal
All Robotics Institute faculty, students, visitors and staff are invited with to attend. One guest per person. RSVP required. Please check your emails for the e-vite and RSVP link. Please contact Debbie Tobin, dmz@cs.cmu.edu, with any questions.
Simple Shape Descriptors for Retinal Surface Estimation using a Laser-Aiming Beam
Abstract: Retinal surgery procedures like epiretinal membrane peeling and retinal vein cannulation require surgeons to manipulate very delicate structures in the eye with little room for error. Many robotic surgery systems have been developed to help surgeons and enforce safeguards during these demanding procedures. One essential piece of information that is required to create and [...]
Affective Robot Behavior Improves Learning in a Sorting Game
Abstract: Nonverbal communication in the field of education can allow teachers to emotionally support their students and improve educational experience and performance. Robot nonverbal movements have been shown to improve both subjective experiences and task performance, and this work investigates whether affective robot behavior can improve human learning. This is tested using an online sorting [...]
Policy Decomposition: Approximate Optimal Control with Suboptimality Estimates
Abstract: Optimal Control is a formulation for designing controllers for dynamical systems by posing it as an optimization problem, whereby the desired long-term behavior of the system is expressed using a cost function. The objective is to compute a policy, i.e. a mapping from the state of the system to its control inputs, that minimizes [...]
Audience-Aware Legibility for Social Navigation
Abstract: Robots often need to communicate their goals to humans when navigating in a shared space to assist observers in anticipating the robot’s future actions. These human observers are often scattered throughout the environment, and each observer only has a partial view of the robot and its movements. A path that non-verbally communicates with multiple [...]
Commencement Celebration
CMU Community Picnic
As shared during President Jahanian’s recent town hall discussions, the CMU Community Picnic is returning on May 18 (11:30 am to 1:30 pm). The Office of Human Resources, in partnership with Staff Council and the Office of the President, sponsors and organizes this yearly celebration as a thank you for the hard work and contributions [...]
Carnegie Mellon University
Generalization for Robot Learning In The Wild
Abstract: How can we train a robot that can generalize to perform thousands of tasks in thousands of environments? This question underscores the holy grail of robot learning, more generally machine learning, research. Current AI systems are incredibly specific in that they only perform the tasks they are trained for and are miserable at generalization. [...]
On Sample-Efficient Reinforcement Learning for Nuclear Fusion
Abstract: In many practical applications of reinforcement learning (RL), it is expensive to observe state transitions from the environment. For example, in the problem of plasma control for nuclear fusion, determining the next state for a given state-action pair requires querying an expensive transition function which can lead to many hours of computer simulation or [...]
Learning Strategies to Solve Real-World Physics Puzzles
Abstract: In this talk, I focus on efficient online learning for solving real-world physics puzzles. I discuss challenges associated with learning in this domain and how those challenges inform certain design decisions. In particular, learning from scratch in the real world would be difficult. I present a practical mixture of experts framework for learning strategies [...]
Carnegie Mellon University
Towards Modular and Differentiable Autonomous Driving
Abstract: The classical "modular and cascaded" autonomy stack (object detection, tracking, trajectory prediction, then planning and control) has been widely used for interactive autonomous systems such as self-driving cars due to its interpretability and fast development cycle. In this thesis, we advocate the use of such a modular stack but improve its accuracy and robustness [...]
Carnegie Mellon University
Towards reconstructing non-rigidity from single camera
Abstract: In this proposal, we study how to infer 3D from images captured by a single camera, without assuming the target scenes / objects being static. The non-static setting makes our problem ill-posed and challenging to solve, but is vital in practical applications where target-of-interest is non-static. To solve ill-posed problems, the current trend in [...]
Carnegie Mellon University
Control Input and Natural Gaze for Goal Prediction in Shared Control
Abstract: Teleoperated systems are used widely in deployed robots today, for such tasks as space exploration, disaster recovery, or assisted manipulation. However, teleoperated systems are difficult to control, especially when performing high-dimensional, contact-rich tasks like manipulation. One approach to ease teleoperated manipulation is shared control; this strategy combines the user's direct control input with an [...]
Forecasting from LiDAR via Future Object Detection
Abstract: Object detection and forecasting are fundamental components of embodied perception. These two problems, however, are largely studied in isolation by the community. In this paper, we propose an end-to-end approach for detection and motion forecasting based on raw sensor measurement as opposed to ground truth tracks. Instead of predicting the current frame locations and [...]
Details to Follow . . .
Details to Follow . . .
Efficient 3D Representations: Algebraic Surfaces for Differentiable Rendering
Abstract: In this proposal, we show how some classic computer vision tasks can robustly be solved via optimization techniques by using an object representation that is compact and interpretable. Specifically, we explore the applications and benefits of representing 3D objects with an analytical, algebraic function by building an approximate, ray-based differentiable renderer. Our approximate formulation [...]
Carnegie Mellon University
Liquid Metal Actuators
Abstract: This thesis contributes to the field of soft actuators by introducing a generalized framework of actuators from liquid metals. The evolution of robotic actuators has enabled robots to achieve a diversity of motions. Like natural muscles, which converts chemical energy into mechanical work in response to electrical stimuli from the nervous system, actuators are [...]
Continual Robot Learning: Benchmarks and Modular Methods
Zoom Meeting Passcode: 841755 Abstract: The earliest reinforcement learning models were designed to learn one task, specified up-front. However, an agent operating freely in the real world will not in general be granted this luxury, as the demands placed on the agent may change as environments or goals change. We refer to this ever-shifting scenario [...]
Carnegie Mellon University
MSR Thesis Talk: Yash Oza
Title: Preprocessing-based Methods for Robotic Manipulation Abstract: Robotic manipulation is a key problem for several applications such as welding, pick-and-place, and automated assembly. However, motion planning for manipulation can be computationally expensive as it requires planning in the high-dimensional configuration space of the manipulator. Additionally, task-specific constraints such as strict time limits or constraints on end-effector [...]
Five Traps for Robots in Human Environments….And How to Avoid Them
Abstract: Robotics today is moving beyond fixed environments and into human spaces like homes, restaurants, and hospitals. In these new spaces, robots will necessarily have to interact with people. In some sense, every recent robotics problem is partly a human-robot interaction problem. Thus, the field of HRI can offer insights to the broader robotics community [...]
MSR Thesis Talk: Ingrid Navarro Anaya
Title: Socially-Aware Trajectory Prediction Guided by Motion Patterns Abstract: As intelligent robots across domains start collaborating with humans in shared environments, e.g., urban settings and terminal airspace, algorithms that enable them to reason over human motion and intent are important to enable seamless and safe interplay. In our work, we study human intent by focusing on the [...]
Carnegie Mellon University
MSR Thesis Talk: Qichen Fu
Date: Tuesday, July 19, 2022 Time: 9:00 AM - 10:00 AM ET Location: Newell-Simon Hall (NSH) 3305 Title: Detect Active Object in a Sequential Voting Process Abstract: A key component of understanding hand-object interactions is the ability to identify the active object -- the object that is being manipulated by the human hand. In order to accurately localize the [...]
Carnegie Mellon University
MSR Thesis Talk: Ruohai Ge
Title: Real-Time Visual Localization System in Changing and Challenging Environments via Visual Place Recognition Abstract: Localization is one of the fundamental capabilities to guarantee reliable robot autonomy. Many excellent Visual-Inertial and LiDAR-based algorithms have been developed to solve the localization problem. However, deploying these methods on a real-time portable device is challenging due to high [...]
TechCrunch Sessions: Robotics 2022
Matt Johnson-Roberson will be one of the featured speakers in TC Sessions: Robotics 2022 on Thursday, July 21. You can register and stream for free! From the TechCrunch event site: Where Minds Meet Machines: The Future of Robotics Rapid advances in robotics technology continue to change the way businesses compete — and how people work [...]
Carnegie Mellon University
MSR Thesis Talk – Zixuan Huang
Title: Seeing the Unseen: Closed-loop Occlusion Reasoning for Cloth Manipulation Robotic manipulation of cloth remains challenging due to the complex dynamics of cloth, lack of a low-dimensional state representation, and self-occlusions. Particularly, self-occlusion makes it difficult to estimate the full state of the cloth, which poses significant challenges to policy learning and dynamics modeling. Ideally, [...]
Carnegie Mellon University
MSR Thesis Talk – Zhaoyuan Fang
Title: Features in Extra Dimensions: Spatial and Temporal Scene Representations Abstract: Computer vision models have made great progress in featurizing pixels of images. However, an image is only a projection of the actual 3D scene: occlusions and perspective distortions exist. To arrive at a better representation of the scene itself, extra dimensions are needed to [...]
Carnegie Mellon University
MSR Thesis Talk – Yunchu Zhang
Title: Library of behaviors and tools for robot manipulation Abstract: Learned policies often fail to generalize across environment variations, such as, different objects, object arrangements, or camera viewpoints. Moreover, most policies are trained and tested in simulation environments, and the sim2real gap remains large under weak visual representations that do not disentangle the scene from [...]
Carnegie Mellon University
Learning Structured World Model for Deformable Object Manipulation
Abstract: Manipulation of deformable objects challenges common assumptions in robotic manipulation, such as low-dimension state representation, known dynamics, and minimal occlusion. Deformable objects have high intrinsic state representation, complex dynamics with high degrees of freedom, and severe self-occlusion. These properties make them difficult for state estimation and planning. In this thesis, we introduce benchmarks and [...]
Safe control under input limits with neural CBF
Abstract: In theory, control barrier functions (CBFs) provide a convenient means to construct provably safe controllers. However, a typical problem is that the constructed controller will exceed input limits, and merely clipping the inputs will break all safety guarantees. To address this practical flaw, we consider synthesizing a CBF that will respect input limits. We [...]
Carnegie Mellon University
MSR Thesis Talk – Chi Yen Lee
Title: Enhancing Quadruped Locomotion Stability with Reaction Wheel Systems and Model Predictive Control Zoom: https://cmu.zoom.us/j/96808397411?pwd=YnFDaFk1WVVyZjc5UndlOTBZL0tjUT09 Abstract: The development of quadruped robots offers a mobility solution that allows robot agents to navigate complicated terrains, making them extremely versatile robots in a variety of environments. Today, there are a number of research challenges facing quadruped development. First, the [...]
MSR Thesis Talk – Chu Er Pan
Title: 6D Object Pose Estimation for Manipulation via Weak Supervision Abstract: 6D object pose estimation is essential for robotic manipulation tasks. Existing learning-based pose estimators often rely on training from labeled absolute poses with fixed object canonical frames, which (1) requires datasets with annotations of object absolute pose that are resource-intensive to collect; (2) is hard [...]
Carnegie Mellon University
MSR Thesis Talk: Ruoyang Xu
Title: Using 3D Imaging Radar for Indoor Localization and Mapping Zoom: https://cmu.zoom.us/j/95090884062?pwd=dVZDVHJDTGVUWW9iSlJLTWtidThBUT09 Meeting ID: 950 9088 4062 Passcode: 411959 Abstract: 3D Imaging Radars offer robust perception capability through visually demanding environments due to the unique penetrative and reflective properties of millimeter waves. However, the utilization of Imaging radar for robot navigation and mapping remains under-explored due [...]
Carnegie Mellon University
MSR Thesis Talk: Gaurav Pathak
Title: Programmable light curtains for Safety Envelopes, SLAM and Navigation Abstract: Conventional robot perception and navigation pipelines are built using traditional sensors such as RGB cameras, stereo depth sensors and LiDARs.These sensors scan the entire scene in a fixed and uniform way. In contrast, programmable light curtains are a recently-invented, resource-efficient sensor that measure the [...]
Carnegie Mellon University
MSR Thesis Talk: Andrew VanOsten
Title: Lidar-Visual-Inertial Odometry via Modifications and Improvements to Super Odometry Abstract: The main focus of this thesis involves improvements and extensions to Super Odometry, a preexisting method for lidar-inertial odometry. This was done in the context of the DARPA RACER program as a member of Carnegie Mellon's DEAD Fast team, aiming to provide reliable [...]
Carnegie Mellon University
MSR Thesis Talk – Bassam Bikdash
Title: Boundary-Aware Demons Algorithm with Applications in Electronic Waste Recycling Abstract Electronic waste (e-waste) refers to electronic devices that are nearing the end of their useful life, and are discarded, donated, or given away. Valuable metallic and plastic components in e-waste (gold, silver, platinum) is estimated to value upwards of $60 billion and although e-waste represents [...]
Carnegie Mellon University
MSR Thesis Talk – Mary Hatfalvi
Title: Introspective Perception through Identifying Blur, Light Direction, and Angle-of-View Abstract Robotic perception tasks have achieved great performance, especially in autonomous vehicles and robot assistance. However, we still often do not understand how and when perception tasks fail. Researchers have achieved some success in creating introspective perception systems that detect when perception tasks will fail, [...]
Carnegie Mellon University
MSR Thesis Talk: Bowei Chen
Title: Image Synthesis with Appearance Decomposition Abstract: Our visual world is compositional and its appearance can be decomposed into various components. Leveraging these components can be beneficial for challenging image synthesis tasks. To this end, this thesis focuses on studying how appearance decomposition can improve image synthesis methods using two examples. (1) Structural decomposition: we introduce [...]
MSR Thesis Talk: Ivan Cisneros
Title: A VPR-Based Technique for UAV Localization In Unseen Environments Abstract: Unmanned Aerial Vehicles (UAVs) primarily rely on GPS-assisted localization and navigation due to the accessibility and ubiquity of such systems. However, this presents a potentially catastrophic single point of failure that may prevent autonomous UAVs from becoming truly reliable, as GPS is prone to dropout, [...]
MSR Thesis Talk: Yehonathan Litman
Title: GPS-Denied Global Visual-Inertial Ground Vehicle State Estimation via Image Registration Abstract: Robotic systems such as unmanned ground vehicles (UGVs) often depend on GPS for navigation in outdoor environments. In GPS-denied environments, one approach to maintain a global state estimate is localizing based on preexisting georeferenced aerial or satellite imagery. However, this is inherently challenged [...]
Carnegie Mellon University
MSR Thesis Talk – George Cazenavette
Title: Learning to Distill Datasets by Matching Expert Training Trajectories Project Page: https://georgecazenavette.github.io/mtt-distillation/ Abstract: Dataset distillation is the task of synthesizing a small dataset such that a model trained on the synthetic set will match the test accuracy of the model trained on the full dataset. In this talk, we review 3 several of our recent [...]
Carnegie Mellon University
MSR Thesis Talk – Zongyue Zhao
Title: Coordinating Heterogeneous Teams for Urban Search and Rescue Abstract: The mission of Urban Search and Rescue (USAR) has drawn significant interest in robotics. Autonomous entities must be able to share knowledge efficiently to address visibility and collaboration challenges in a complex environment shortly after structural collapse catastrophes. In this thesis, we present methods to coordinate [...]
Carnegie Mellon University
MSR Thesis Talk – Jeff Hu
Title: Composition Learning in “Modular” Robot Systems Abstract: Modular robot and multi-robot systems share a concept in common: composition, i.e. the study of how parts can be combined so they can be used to achieve certain objectives. Our vision is to enable robotic systems to configure and reconfigure themselves during field deployment, either autonomously or [...]
Carnegie Mellon University
MSR Thesis Talk – Tom Bu
Title: Towards HD Map Updates With Crosswalk Change Detection From Vehicle-mounted Cameras Zoom: https://cmu.zoom.us/j/4452379705 Abstract: Many autonomous vehicles rely on high-definition maps that contain road layout and road semantics as priors for perception, planning and prediction. However, these maps can become stale over time as the road environment changes. This thesis develops a road monitoring framework [...]
MSR Thesis Talk: Zilin Si
Title: Taxim: An Example-based Simulation Model for GelSight Tactile Sensors and its Sim-to-Real Applications Location: NSH 4305 or Zoom https://cmu.zoom.us/j/91769761787?pwd=cGZ2RElKMVJaQ1NVNG5BdFQ0Ny9uQT09 Abstract: Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating a robot system efficiently and with high fidelity, from sensing, perception to manipulation, has been a long-standing challenge. Tactile sensing, as [...]
Carnegie Mellon University
MSR Thesis Talk – Benjamin Jensen
Title: A Low-Cost Attitude Determination and Control System and Hardware-in-the-Loop Testbed for CubeSats Zoom: https://cmu.zoom.us/j/92654622790?pwd=d0pYcTJ4K0xzdmYvUHFYWC9lMDBhQT09 Abstract: Since their initial development in the late 1990s, CubeSats have quickly grown popular due to their relatively low cost and short development period. However, CubeSat launches are prone to failure, with less than half of CubeSats completely fulfilling their [...]
Carnegie Mellon University
MSR Thesis Talk – Swapnil Pande
Title: Driving by Dreaming: Offline Model-Based Reinforcement Learning for Motion Planning for Autonomous Vehicles Abstract: While there has been significant progress in deploying autonomous vehicles (AVs) in urban driving settings, there remains a long-tail of challenging motion planning scenarios that must be addressed before truly driverless operation is possible. The current paradigm for motion planner [...]
Carnegie Mellon University
MSR Thesis Talk – Alvin Shek
Title: Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method Abstract: For robots to be effectively deployed in novel environments and tasks, they must be able to understand the feedback expressed by humans during intervention. This can either correct undesirable behavior or indicate additional preferences. Existing methods either require repeated episodes of interactions or [...]
Carnegie Mellon University
MSR Thesis Talk: Jiaqi Geng
Title: Dense Human Pose Estimation From WiFi Abstract: Advances in computer vision and machine learning techniques have led to significant development in 2D and 3D human pose estimation from RGB cameras, LiDAR, and radars. However, human pose estimation from images is adversely affected by occlusion and lighting, which are common in many scenarios of interest. [...]
Carnegie Mellon University
MSR Thesis Talk: Jianchun Chen
Title: An efficient approach for sequential shape human performance capture from monocular video Abstract: Human performance capture from RGB videos in unconstrained environments has become very popular for applications to generate virtual avatars or digital actors. Modern approaches rely on neural network algorithms to estimate geometry directly from images, resulting in a coarse representation of [...]
Thermal Management Considerations For Lunar Polar Micro-Rovers
Meeting ID: 940 0396 4889 Passcode: 906118 Abstract: This research addresses the significant and unprecedented challenge of thermal regulation for lunar polar micro-rovers. These are distinct from priors by way of very small size, mass, and power, but particularly for the extremes of ambient environment in which they must operate. On the lunar poles, rovers experience temperatures [...]
Carnegie Mellon University
MSR Thesis Talk: Zhihao Zhang
Title: Efficient Methods for Model Performance Inference Abstract: A key challenge in neural architecture search (NAS) is quickly inferring the predictive performance of a broad spectrum of neural networks to discover statistically accurate and computationally efficient ones. We refer to this task as model performance inference (MPI). The current practice for efficient MPI is gradient-based methods [...]
Carnegie Mellon University
MSR Thesis Talk: Chufan Gao
Title: Addressing Time-series Signal Quality in Healthcare Data Abstract: Healthcare data time-series signal quality assessment (SQA) plays a vital role in the accuracy and reliability of machine learning algorithms to analyze health metrics. However, these signals are often corrupted with different kinds of noises and artifacts, including Baseline Wander, Muscle Artifacts, Powerline Interference, and Equipment Failure. This [...]
Carnegie Mellon University
Object Pose Estimation without Direct Supervision
Abstract: Currently, robot manipulation is a special purpose tool, restricted to isolated environments with a fixed set of objects. In order to make robot manipulation more general, robots need to be able to perceive and interact with a large number of objects in cluttered scenes. Traditionally, object pose has been used as a representation to [...]
Multimodal Modeling: Learning Beyond Visual Knowledge
Abstract: The computer vision community has embraced the success of learning specialist models by training with a fixed set of predetermined object categories, such as ImageNet or COCO. However, learning only from visual knowledge might hinder the flexibility and generality of visual models, which requires additional labeled data to specify any other visual concept and [...]
Improving Robotic Exploration with Self-Supervision and Diverse Data
Abstract: Reinforcement learning (RL) holds great promise for improving robotics, as it allows systems to move beyond passive learning and interact with the world while learning from these interactions. A key aspect of this interaction is exploration: which actions should an RL agent take to best learn about the world? Prior work on exploration is typically [...]
RI Faculty Business Meeting
Meeting for RI Faculty. Discussions include various department topics, policies, and procedures. Generally meets weekly.