PhD Thesis Defense
Visual Utility – A Framework for Focusing Computer Vision Algorithms
Event Location: NSH 3305Abstract: The real world is a rich environment, fraught with complexity. To be robust in this complex environment, computer Vision algorithms that operate in Unstructured Environments (VUE) tend to use large amounts of data or complex modeling. Unfortunately, these algorithms also require significant computational resources. In this thesis, we examine a visual [...]
Organ-Mounted Robots for Minimally Invasive Beating-Heart Surgery
Event Location: NSH 1507Abstract: In the push to improve patient outcomes in cardiac interventions, minimally invasive beating-heart surgery is a major field of surgical research. However, interventions on a soft tissue organ under continuous motion through remote incisions pose a significant challenge. Endoscopic approaches eliminate the associated morbidity of median sternotomy, but they require either [...]
Subdimensional Expansion: A Framework for Computationally Tractable Multirobot Path Planning
Event Location: NSH 1507Abstract: Planning optimal paths for large numbers of robots is computationally expensive. In this thesis, we present a new framework for multirobot path planning called subdimensional expansion, which initially plans for each robot individually, and then coordinates motion among the robots as needed. More specifically, subdimensional expansion initially creates a one-dimensional search [...]
Evaluation of Decentralized Reactive Swing-Leg Controllers on Powered Robotic Legs
Event Location: NSH 3305Abstract: We present work to transfer decentralized neuromuscular control strategies of human locomotion to powered segmented robotic legs. State-of-the-art robotic locomotion control approaches, like centralized planning and tracking in fully robotic systems and predefined motion pattern replay in prosthetic systems, do not enable the dynamism and reactiveness of able-bodied humans. Animals largely [...]
Online Hierarchical Optimization for Humanoid Control
Event Location: NSH 3305Abstract: This thesis presents an online approach for controlling humanoid robots using hierarchical optimization. While our primary focus is to develop a fast and robust walking controller that is able to follow desired foot steps, full body manipulation capability is also achieved. The proposed hierarchical system consists of three levels: a high [...]
Using Planned View Trajectories to Build Good Models of Planetary Features under Transient Illumination
Event Location: GHC 2109Abstract: This research addresses the modeling of substantially 3D planetary terrain features, such as skylights, canyons, craters, rocks, and mesas, by a surface robot. The sun lights planetary features with transient, directional illumination. For concave features like skylight pits, craters, and canyons, this can lead to dark shadows. For all terrain features, [...]
Tracking and Calibrating Articulated Robots using SLAM Techniques
Event Location: NSH 1305Abstract: Robots still struggle with everyday manipulation tasks. An overriding problem with robotic manipulation is uncertainty in the robot's state and calibration parameters. Small amounts of uncertainty can lead to complete task failure. This thesis explores ways of tracking and calibrating noisy robot arms using computer vision, with an aim toward making [...]
Factoring Scenes into 3D Structure and Style
Event Location: GHC 4405Abstract: Given a single image of a scene, humans have few issues answering questions about its 3D structure like “is this facing upwards?” even though mathematically speaking this should be impossible. We have similarly have few issues accounting for this 3D structure in answering viewpoint independent questions like ``is this the same [...]
Direct Pose Estimation and Refinement
Event Location: NSH 1305Abstract: We study a fundamental question in pose estimation from vision-only video data: should the pose of a camera be determined from fixed and known correspondences? Or should correspondences be simultaneously estimated alongside the pose? Determining pose from fixed correspondences is known as feature-based, where well-established tools from projective geometry are utilized [...]
The Optical Coherence Tomography Microsurgical Augmented Reality System (OCT-MARS): a novel device for microsurgeries
Event Location: NSH 3305Abstract: We describe the development and testing of the Optical Coherence Tomography Microsurgical Augmented Reality System (OCT-MARS). This system allows surgeons to view real-time medical image data as an in-situ overlay within the surgical field. There are a number of clinical applications for which real time, in situ visualization of otherwise transparent [...]
Expressive Motion for Low Degree-of-Freedom Robots
Event Location: GHC 4405Abstract: As social and collaborative robots move into everyday life, the need for algorithms enabling their acceptance becomes critical. People parse non-verbal communications intuitively, even from machines that do not look like people, thus, expressive motion is a natural and efficient way to communicate with people. This work presents a computational Expressive [...]
Completing Manipulation Tasks Efficiently in Complex Environments
Event Location: NSH 3305Abstract: An effective autonomous robot performing dangerous or menial tasks will need to act under significant time and energy constraints. At task time, the amount of effort a robot spends planning its motion directly detracts from its total performance. Manipulation tasks, however, present challenges to efficient motion planning. Tightly coupled steps (e.g. [...]
Robust Manipulation via Contact Sensing
Event Location: NSH 3305Abstract: Humans effortlessly manipulate objects in cluttered and uncertain environments. In contrast, most robotic manipulators are limited to carefully engineered environments to circumvent the difficulty of manipulation under uncertainty. Contact sensors can provide robots with with the feedback vital to addressing this limitation. This thesis proposes a framework for using feedback from [...]
The Optical Coherence Tomography Microsurgical Augmented Reality System (OCT-MARS): a novel device for microsurgeries
Event Location: NSH 3305Abstract: We describe the development and testing of the Optical Coherence Tomography Microsurgical Augmented Reality System (OCT-MARS). This system allows surgeons to view real-time medical image data as an in-situ overlay within the surgical field. There are a number of clinical applications for which real time, in situ visualization of otherwise transparent [...]
Robust Rearrangement Planning using Nonprehensile Interaction
Event Location: GHC 6501Abstract: As we work to move robots out of factories and into human environments, we must empower robots to interact freely in unstructured, cluttered spaces. Humans do this easily, using diverse, whole-arm, nonprehensile actions such as pushing or pulling in everyday tasks. These interaction strategies make difficult tasks easier and impossible tasks [...]
Efficiently Sampling from Underlying Physical Models
Event Location: GHC 6501Abstract: The capability and mobility of exploration robots is increasing rapidly, yet missions will always be constrained by one main resource: time. Time limits the number of samples a robot can collect, sites it can analyze, and the availability of human oversight, so it is imperative the robot is able to make [...]
Navigation and Physical Interaction with Balancing Robots
Event Location: NSH 1507Abstract: This work describes methods for advancing the state of the art in mobile robot navigation and physical Human-Robot Interaction (pHRI). An enabling technology in this effort is the ballbot, a person-sized mobile robot that balances on a ball. This underactuated robot presents unique challenges in planning, navigation, and control; however, it [...]
Calibration and Characterization of Low-Cost Fine Particulate Monitors and their Effect on Individual Empowerment
Event Location: GHC 8102Abstract: Air quality has long been a major health concern for citizens around the world, and increased levels of exposure to fine particulate matter (PM2.5) has been definitively linked to serious health effects such as cardiovascular disease, respiratory illness, and increased mortality. PM2.5 is one of six attainment criteria pollutants used by [...]
Improving Prosody through Analysis by Synthesis
Event Location: GHC 6501Abstract: An iterative model-based method is proposed for improving linguistic structure, segmentation, and prosodic annotations that correspond to the delivery of each utterance as regularized across the data. For each iteration, the training utterances are resynthized according to the existing symbolic annotation. Values of various features and subgraph structures are "twiddled:" each [...]
Automatic Analysis of Facial Actions: Learning from Transductive, Supervised and Unsupervised Frameworks
Abstract Automatic analysis of facial actions (AFA) can reveal a person's emotion, intention, and physical state, and make possible a wide range of applications. To enable reliable, valid, and efficient AFA, this thesis investigates both supervised and unsupervised learning. Supervised learning for AFA is challenging, in part, because of individual differences among persons in face [...]
Measuring Human Motion in Social Interactions
Tomas Simon Carnegie Mellon University Abstract This thesis develops methods for social signal reconstruction---in particular, we measure human motion during social interactions. Compared to other work in this space, we aim to measure the entire body, from the overall body pose to subtle hand gestures and facial expressions. The key to achieving this without placing [...]
Online Lidar and Vision based Ego-motion Estimation and Mapping
Ji Zhang Carnegie Mellon University Abstract In many real-world applications, ego-motion estimation and mapping must be conducted online. In the robotics world, especially, real-time motion estimates are important for control of autonomous vehicles, while online generated maps are crucial for obstacle avoidance and path planning. Further, the complete map of a traversed environment can be [...]
Creative Robotic Systems for Talent-based Learning
Event Location: GHC 4405Abstract: In recent years, the U.S. educational system has fallen short in training the technology innovators of the future. To do so, we must give students the experience of designing and creating technological artifacts, rather than relegating students to the role of technology consumers, and must provide educators with opportunities and professional [...]
The theory, implementation, and evaluation of spring mass running on ATRIAS, a bipedal robot
Event Location: NSH 3305Abstract: We expect legged robots to be highly mobile. Human walking and running can execute quick changes in speed and direction, even on non-flat ground. Indeed, analysis of simplified models shows that these quantities can be tightly controlled by adjusting the leg placement between steps, and that leg placement can also compensate [...]
Improving Prediction of Traversability for Planetary Rovers Using Thermal Imaging
Event Location: GHC 4405Abstract: The most significant mobility challenges that planetary rovers encounter are compounded by loose, granular materials that cause slippage and sinkage on slopes or are deep enough to entrap a vehicle. The inability of current technology to detect loose terrain hazards has caused significant delays for rovers on both the Moon and [...]
Safe, Efficient, and Robust Predictive Control of Constrained Nonlinear Systems
Vishnu R. Desaraju Carnegie Mellon University April 12, 2017, 2:00 p.m., NSH 1305 Abstract As autonomous systems are deployed in increasingly complex and uncertain environments, safe, accurate, and robust feedback control techniques are required to ensure reliable operation. Accurate trajectory tracking is essential to complete a variety of tasks, but this may be difficult if [...]
Seungmoon Song: The Development, Evaluation and Applications of a Neuromechanical Control Model of Human Locomotion
Seungmoon Song Ph.D. Thesis Defense Abstract: The neural control of human locomotion is not fully understood. As current experimental techniques provide only partial and indirect access to the neural control network, our understanding remains fragmentary with large gaps between detectable neural circuits and measurable behavioral data. Neuromechanical simulation studies can help bridging these gaps. By [...]
Juan Pablo Mendoza: Regions of Inaccurate Modeling for Robot Anomaly Detection and Model Correction
Juan Pablo Mendoza Ph.D. Thesis Defense Abstract: To make intelligent decisions, robots often use models of the stochastic effects of their actions on the world. Unfortunately, in complex environments, it is often infeasible to create models that are accurate in every plausible situation, which can lead to suboptimal performance. This thesis enables robots to reason [...]
Reasoning About Spatial Patterns of Human Behavior During Group Conversations with Robots
Abstract: The goal of this dissertation is to develop computational models for robots to detect and sustain the spatial patterns of behavior that naturally emerge during free-standing group conversations with people. These capabilities have often been overlooked by the Human-Robot Interaction (HRI) community, but they are essential for robots to appropriately interact with and around [...]
Acting under Uncertainty for Information Gathering and Shared Autonomy
Abstract: Acting under uncertainty is a fundamental challenge for any decision maker in the real world. As uncertainty is often the culprit of failure, many prior works attempt to reduce the problem to one with a known state. However, this fails to account for a key property of acting under uncertainty: we can often gain [...]
Situational Awareness and Mixed Initiative Markup for Human-Robot Team Plans
Abstract: As robots become more reliable and user interfaces (UI) become more powerful, human-robot teams are being applied to more real world problems. Human-robot teams offer redundancy and heterogeneous capabilities desirable in scientific investigation, surveillance, disaster response, and search and rescue operations. Large teams are overwhelming for a human operator, so systems employ high level [...]
Discovering and Leveraging Visual Structure for Large-scale Recognition
Abstract: Our visual world is extraordinarily varied and complex, but despite its richness, the space of visual data may not be that astronomically large. We live in a well-structured, predictable world, where cars almost always drive on roads, sky is always above the ground, and so on. As humans, the ability to learn this structure [...]
Deliberative Perception
Abstract: A recurrent and elementary robot perception task is to identify and localize objects of interest in the physical world. In many real-world situations such as in automated warehouses and assembly lines, this task entails localizing specific object instances with known 3D models. Most modern-day methods for the 3D multi-object localization task employ scene-to-model feature [...]
Carnegie Mellon University
Compact Generative Models of Point Cloud Data for 3D Perception
Abstract: One of the most fundamental tasks for any robotics application is the ability to adequately assimilate and respond to incoming sensor data. In the case of 3D range sensing, modern-day sensors generate massive quantities of point cloud data that strain available computational resources. Dealing with large quantities of unevenly sampled 3D point data is [...]
Carnegie Mellon University
Mathematical Models of Adaptation in Human-Robot Collaboration
Abstract: While much work in human-robot interaction has focused on leader- follower teamwork models, the recent advancement of robotic systems that have access to vast amounts of information suggests the need for robots that take into account the quality of the human decision making and actively guide people towards better ways of doing their task. [...]
Carnegie Mellon University
Training Strategies for Time Series: Learning for Prediction, Filtering, and Reinforcement Learning
Abstract: Data driven approaches to modeling time-series are important in a variety of applications from market prediction in economics to the simulation of robotic systems. However, traditional supervised machine learning techniques designed for i.i.d. data often perform poorly on these sequential problems. This thesis proposes that time series and sequential prediction, whether for forecasting, filtering, [...]
Carnegie Mellon University
Planning for a Small Team of Heterogeneous Robots: from Collaborative Exploration to Collaborative Localization
Abstract: Robots have become increasingly adept at performing a wide variety of tasks in the world. However, many of these tasks can benefit tremendously from having more than a single robot simultaneously working on the problem. Multiple robots can aid in a search and rescue mission each scouting a subsection of the entire area in [...]
Carnegie Mellon University
Adaptive Motion Planning
Abstract: Mobile robots are increasingly being deployed in the real world in response to a heightened demand for applications such as transportation, delivery and inspection. The motion planning systems for these robots are expected to have consistent performance across the wide range of scenarios that they encounter. While state-of-the-art planners, with provable worst-case guarantees, can [...]
Carnegie Mellon University
Kernel and Moment based Prediction and Planning: Applications to Robotics and Natural Language Processing
Abstract This thesis focuses on moment and kernel-based methods for applications in Robotics and Natural Language Processing. Kernel and moment-based learning leverage information about correlated data that allow the design of compact representations and efficient learning algorithms. We explore kernel algorithms for planning by leveraging inherently continuous properties of reproducing kernel Hilbert spaces. We introduce [...]
Carnegie Mellon University
Harnessing Task Mechanics for Robotic Pushing and Grasping
Abstract: A high-fidelity and tractable mechanics model of physical interaction is essential for autonomous robotic manipulation in complex and uncertain environments. This thesis studies several aspects of harnessing task mechanics for robotic pushing and grasping operations: mechanics model learning, pose and model uncertainty reduction, and planning and control synthesis in the minimal coordinate space. We [...]
Learning Multi-Modal Navigation for Unmanned Ground Vehicles
The Event has been Postponed. Abstract: A robot that operates efficiently in a team with a human in an unstructured outdoor environment must be able to translate commands from a modality that is intuitive to its operator into actions. This capability is especially important as robots become ubiquitous and interact with untrained users. For this [...]
Carnegie Mellon University
Algorithms, Implementation, and Studies on Eating with a Shared Control Robot Arm
Abstract: People with upper extremity disabilities are gaining increased independence through the use of assisted devices such as wheelchair-mounted robotic arms. However, the increased capability and dexterity of these robotic arms also makes them challenging to control through accessible interfaces like joysticks, sip-and-puff, and buttons that are lower-dimensional than the control space of the robot. [...]
Carnegie Mellon University
Foraging, Prospecting, and Falsification – Improving Three Aspects of Autonomous Science
Abstract: Robots exploring the subsurface ocean of Europa, for example, may not have reliable communications with scientists on Earth. Robots exploring with unreliable communications must conduct scientific exploration autonomously. Approaches to deliberative and opportunistic science autonomy that work in the laboratory may not work in the field. This thesis presents three algorithms designed to improve [...]
Carnegie Mellon University
Data-Driven Visual Forecasting
Abstract: Understanding the temporal dimension of images is a fundamental part of computer vision. Humans are able to interpret how the entities in an image will change over time. However, it has only been relatively recently that researchers have focused on visual forecasting—getting machines to anticipate events in the visual world before the actually happen. [...]
Carnegie Mellon University
Planning for Sustained Lunar Polar Roving
Abstract: Lunar polar resources can accelerate deep space exploration by resupplying missions with oxygen, water, and propellent. Before lunar resupply can be established, the distribution and concentration of water ice and other volatiles abundant at the poles of the Moon must be verified and mapped. The need for affordable, scalable exploration of the lunar poles [...]
Carnegie Mellon University
Lidar Simulation for Robotic Application Development: Modeling and Evaluation
Abstract: Given the increase in scale and complexity of robotics, robot application development is challenging in the real world. It may be expensive, unsafe, or impractical to collect data, or test systems, in reality. Simulation provides an answer to these challenges. In simulation, data collection is relatively inexpensive, scenes can be procedurally generated, and state [...]
Carnegie Mellon University
Adapting to Context in Robot Perception
Abstract: The promised future filled with robots sensing and acting intelligently in the world is near fruition, thanks in part to continuous progress in robotic perception. However, a number of challenges remain before robots and their perception systems can be truly reliable. In particular, we must consider what happens when highly complex perception systems designed [...]
Carnegie Mellon University
Using Multiple Fidelity Models in Motion Planning
Abstract: Hospitals and warehouses use autonomous delivery robots to increase productivity. Robots must reliably navigate unstructured non-uniform environments which requires efficient long-term operation that robustly accounts for unforeseen circumstances. However, unreliable autonomous robots need continuous operator assistance, which decreases throughput and negates a robot's benefit. Planning with high fidelity models is more likely to lead [...]
Carnegie Mellon University
Probabilistic Approaches for Pose Estimation
Abstract: Virtually all robotics and computer vision applications require some form of pose estimation; such as registration, structure from motion, sensor calibration, etc. This problem is challenging because it is highly nonlinear and nonconvex. A fundamental contribution of this thesis is the development of fast and accurate pose estimation by formulating in a parameter space [...]
Carnegie Mellon University
Algorithms for Timing and Sequencing Behaviors in Robotic Swarms
Abstract: Robotic swarms are multi-robot systems whose global behavior emerges from local interactions between individual robots and spatially proximal neighboring robots. Each robot can be programmed with several local control laws that can be activated depending on an operator's choice of global swarm behavior (e.g. flocking, aggregation, formation control, area coverage). In contrast to other [...]
Carnegie Mellon University
Data-Driven Statistical Models of Robotic Manipulation
Abstract: Improving robotic manipulation is critical for robots to be actively useful in real-world factories and homes. While some success has been shown in simulation and controlled environments, robots are slow, clumsy, and not general or robust enough when interacting with their environment. By contrast, humans effortlessly manipulate objects. One possible reason for this discrepancy [...]
Carnegie Mellon University
Learning to Learn for Small Sample Visual Recognition
Abstract: Understanding how humans and machines recognize novel visual concepts from few examples remains a fundamental challenge. Humans are remarkably able to grasp a new concept and make meaningful generalization from just few examples. By contrast, state-of-the-art machine learning techniques and visual recognition systems typically require thousands of training examples and often break down if [...]
Carnegie Mellon University
Designing Interactive Systems for Community Citizen Science
Abstract: Citizen science forges partnerships between experts and citizens through collaboration and has become a trend in public participation in scientific research over the past decade. Besides this trend, public participation can also contribute to participatory democracy, which empowers citizens to advocate for their local problems. This strategy supports citizens to form a community, increase [...]
Carnegie Mellon University
Visual Learning with Minimal Human Supervision
Abstract: Machine learning models have led to remarkable progress in visual recognition. A key factor driving this progress is the abundance of labeled data. Unfortunately, this reliance on lots of labeled data is also a key limitation in the rapid development and deployment of vision systems. These visual recognition systems show poor performance on concepts [...]
Carnegie Mellon University
Robot Design for Everyone: Computational Tools that Democratize the Design of Robots
Abstract: A grand vision in robotics is that of a future wherein robots are integrated in daily human life just as smart phones and computers are today. Such pervasive integration of robots would require faster design and manufacturing of robots that cater to individual needs. For instance, people would be able to obtain customized smart [...]
Carnegie Mellon University
Robust Soft-Matter Robotic Materials
Abstract: Emerging applications in wearable computing, human-machine interaction, and soft robotics will increasingly rely on new soft-matter technologies. These soft-matter technologies are considered inherently safe as they are primarily composed of intrinsically soft materials---elastomers, gels, and fluids. These materials provide a method for creating soft-matter counterparts to traditionally rigid devices that exhibit the mechanical compliance [...]
Carnegie Mellon University
Provably Optimal Design of a Brain-Computer Interface
Abstract: Brain-computer interfaces are in the process of moving from the laboratory to the clinic. These devices act by reading neural activity and using it to directly control a device, such as a cursor on a computer screen. Over the past two decades, much attention has been devoted to the decoding problem: how should recorded [...]
Carnegie Mellon University
Extensions of the Principal Fiber Bundle Model for Locomoting Robots
Abstract: Our goal is to establish a rigorous formulation for modeling the locomotion of a broad class of robotic systems. Recent research has identifi ed a number of systems with the structure of a principal fiber bundle. This framework has led to a number of tools for analysis and motion planning applicable to various robotic [...]
Carnegie Mellon University
Safe Data Gathering in Physical Spaces
Abstract: Reliable and efficient acquisition of data from physical spaces has widespread applications in industry, policy, defense, and humanitarian work. Unmanned Aerial Vehicles (UAVs) are an excellent choice for data gathering applications, due to their capability of gaining information at multiple scales. A robust data gathering system needs to reason about multi-resolution nature of information [...]
Carnegie Mellon University
Learning to learn from simulation: Using simulations to learn faster on robots
Abstract: Learning for control is capable of acquiring controllers in novel task scenarios, paving the path to autonomous robots. However, typical learning approaches can be prohibitively expensive in terms of robot experiments, and policies learned in simulation do not transfer directly due to modelling inaccuracies. This encourages learning information from simulation that has a higher [...]
Carnegie Mellon University
Learning with Clusters
Abstract: Clustering, the problem of grouping similar data, has been extensively studied since at least the 1950's. As machine learning becomes more prominent, clustering has evolved from primarily a data analysis tool into an integrated component of complex robotic and machine learning systems, including those involving dimensionality reduction, anomaly detection, network analysis, image segmentation and [...]
Carnegie Mellon University
Sensing, Measuring, and Modeling Social Signals in Nonverbal Communication
Abstract: Humans convey their thoughts, emotions, and intentions through a concert of social displays: voice, facial expressions, hand gestures, and body posture, collectively referred to as social signals. Despite advances in machine perception, machines are unable to discern the subtle and momentary nuances that carry so much of the information and context of human communication. [...]
Carnegie Mellon University
Optimal control of compliant bipedal gaits and their implementation on robot hardware
Abstract: Legged animals exhibit diverse locomotion patterns known as gaits, which are capable of robustly traversing terrains of variable grade, roughness, and compliance. Despite the success of legs in nature, wheeled solutions still dominate the field of robotics. State-of-the-art humanoid robots have not yet demonstrated locomotion behaviors that are as robust or varied as their [...]
Carnegie Mellon University
Design and Evaluation of Robust Control Methods for Robotic Transfemoral Prostheses
Abstract: Amputees face a number of gait deficits due to a lack of control and power from their mechanically-passive prostheses. Of crucial importance among these deficits are those related to balance, as falls and a fear of falling can cause an avoidance of activity that leads to further debilitation. In this thesis, we investigate the [...]
Carnegie Mellon University
Intra-Robot Replanning and Learning for Multi-Robot Teams in Complex Dynamic Domains
Abstract: In complex dynamic multi-robot domains, there is a set of individual robots that must coordinate together through a centralized planner that inevitably makes assumptions based on a model of the environment and the actions of the individual. Eventually, the individuals may encounter failures, because the centralized planner’s models of the states and actions are [...]
Carnegie Mellon University
Light Sheet Depth Imaging
Abstract: Once confined to industrial manufacturing facilities and research labs, robots are increasingly entering everyday life. As specialized robots are developed for tasks such as autonomous driving, package delivery, and aerial videography, there is a growing need for affordable depth sensing technology. Robots use sensors like scanning LIDAR, depth cameras, and passive stereo cameras to [...]
Carnegie Mellon University
Towards Generalization and Efficiency in Reinforcement Learning
Abstract: In classic supervised machine learning, a learning agent behaves as a passive observer: it receives examples from some external environment which it has no control over and then makes predictions. Reinforcement Learning (RL), on the other hand, is fundamentally interactive: an autonomous agent must learn how to behave in an unknown and possibly hostile [...]
Carnegie Mellon University
Planning under Uncertainty with Multiple Heuristics
Abstract: Many robotic tasks, such as mobile manipulation, often require interaction with unstructured environments and are subject to imperfect sensing and actuation. This brings substantial uncertainty into the problems. Reasoning under this uncertainty can provide higher level of robustness but is computationally significantly more challenging. More specifically, sequential decision making under motion and sensing uncertainty [...]
Carnegie Mellon University
Analysis of Spatio-Temporally Varying Features in Optical Coherence Tomographic (OCT) and Ultrasound (US) Image Sequences
Abstract: Optical Coherence Tomography (OCT) and Ultrasound (US) are non-ionizing and non-invasive imaging modalities that are clinically used to visualize anatomical structures in the body. OCT has been widely adopted in clinical ophthalmology due to its micron-scale resolution to visualize in-vivo structures of the eye. Ultra-High Frequency Ultrasound (UHFUS) captures images of tissue at a [...]
Carnegie Mellon University
Spatiotemporal Understanding of People Using Scenes, Objects, and Poses
Abstract: Humans are arguably one of the most important entities that AI systems would need to understand to be useful and ubiquitous. From autonomous cars observing pedestrians to assistive robots helping the elderly, a large part of this understanding is focused on recognizing human actions, and potentially, their intentions. Humans themselves are quite good at [...]
Carnegie Mellon University
Deep Non-Rigid Structure from Motion
Abstract: Non-Rigid Structure from Motion (NRSfM) refers to the problem of reconstructing cameras and the 3D point cloud of a non-rigid object from a sequence of images with 2D correspondences. Current NRSfM algorithms are mainly limited within two perspectives: (i) the number of images, and (ii) the type of shape variability they can handle. These [...]
Carnegie Mellon University
Data Centric Robot Learning
Abstract: While robotics has made tremendous progress over the last few decades, most success stories are still limited to carefully engineered and precisely modeled environments. Getting these robots to work in the complex and diverse world that we live in has proven to be a difficult challenge. Interestingly, one of the most significant successes in [...]
Carnegie Mellon University
Exploiting Point Motion, Shape Deformation, and Semantic Priors for Dynamic 3D Reconstruction in the Wild
Abstract: With the advent of affordable and high-quality smartphone cameras, any significant events will be massively captured both actively and passively from multiple perspectives. This opens up exciting opportunities for low-cost high-end VFX effects and large scale media analytics. However, automatically organizing large scale visual data and creating a comprehensive 3D scene model is still [...]
Carnegie Mellon University
Learning and Reasoning with Visual Correspondence in Time
Abstract: There is a famous tale in computer vision: Once, a graduate student asked the famous computer vision scientist Takeo Kanade: "What are the three most important problems in computer vision?" Takeo replied: "Correspondence, correspondence, correspondence!" Indeed, even for the most commonly applied Convolutional Neural Networks (ConvNets), they are internally learning representations that lead to [...]
Carnegie Mellon University
Forecasting and Controlling Behavior by Learning from Visual Data
Abstract: Achieving a precise predictive understanding of the future is difficult, yet widely studied in the natural sciences. Significant research activity has been dedicated to building testable models of cause and effect. From a certain view, a perfect predictive model of the universe is the “holy grail”; the ultimate goal of science. If we had [...]
Carnegie Mellon University
Underwater Localization and Mapping with Imaging Sonar
Abstract: Acoustic imaging sonars have been used for a variety of tasks intended to increase the autonomous capabilities of underwater vehicles. Among the most critical tasks of any autonomous vehicle are localization and mapping, which are the focus of this work. The difficulties presented by the imaging sonar sensor have led many previous attempts at [...]
Carnegie Mellon University
Personalized and weakly supervised learning for Parkinson’s disease symptom detection
Abstract: Parkinson's Disease (PD) is a neurodegenerative disorder that affects approximately one million Americans. Medications exist to manage the symptoms, but doctors must periodically adjust dosage level and frequency as a patient's disease progresses. These adjustments are typically based on observations made during short clinic visits, which provide an incomplete picture of a patient's daily [...]
Carnegie Mellon University
Self-Supervised Learning on Mobile Robots Using Acoustics, Vibration, and Visual Models to Build Rich Semantic Terrain Maps
Abstract: Humans and robots would benefit from having rich semantic maps of the terrain in which they operate. Mobile robots equipped with sensors and perception software could build such maps as they navigate through a new environment. This information could then be used by humans or robots for better localization and path planning, as well [...]
Carnegie Mellon University
Combining Multiple Heuristics: Studies on Neighborhood-base Heuristics and Sampling-based Heuristics
Abstract: This thesis centers on the topic of how to automatically combine multiple heuristics. For most computationally challenging problems, there exist multiple heuristics, and it is generally the case that any such heuristic exploits only a limited number of aspects among all the possible problem characteristics that we can think of, and by definition, is [...]
Carnegie Mellon University
Multi-hypothesis iSAM2 for Ambiguity-aware Passive and Active SLAM
Archived video Abstract Simultaneous localization and mapping (SLAM) is the problem of estimating the state of a moving agent with sensors on it while simultaneously reconstructing a map of its surrounding environment, which has been a popular research field due to its wide applications. As many state-of-the-art SLAM algorithms can already achieve high accuracy in [...]
Carnegie Mellon University
Terrain Relative Navigation for Lunar Polar Roving: Exploiting Geometry, Shadows, and Planning
Archived Zoom Video Abstract Water ice at the lunar poles could be the most valuable resource beyond planet Earth. However, that value is not foregone, and can only be determined by rovers that evaluate the distributions of abundance, concentration, and characteristics of this ice. The near-term explorations will be solar and unlikely to endure night, [...]
Carnegie Mellon University
Resource-Constrained State Estimation with Multi-Modal Sensing
Zoom Link Accurate and reliable state estimation is essential for safe mobile robot operation in real-world environments because ego-motion estimates are required by many critical autonomy functions such as control, planning, and mapping. Computing accurate state estimates depends on the physical characteristics of the environment, the selection of suitable sensors to capture that information, and [...]
Carnegie Mellon University
Hybrid Soft Sensing in Robotic Systems
Zoom Link Abstract: The desire to operate robots in unstructured environments, side-by-side with humans, has created a demand for safe and robust sensing skins. Largely inspired by human skin, the ultimate goal of electronic skins is to measure diverse sensory information, conform to surfaces, and avoid interfering with the natural mechanics of the host or [...]
Carnegie Mellon University
Vision with Small Baselines
Zoom Link Abstract: 3D sensing with portable imaging systems is becoming more and more popular in computer vision applications such as autonomous driving, virtual reality, robotics manipulation and surveillance, due to the decreasing expense and size of RGB cameras. Despite the compactness and portability of the small baseline vision systems, it is well-known that the [...]
Carnegie Mellon University
Humans In Their Natural Habitat: Training AI to Understand People
Zoom Link Abstract: Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. First, we need to give [...]
Carnegie Mellon University
Automated Action Selection and Embodied Simulation for Socially Assistive Robots using Standardized Interactions
Zoom Link Abstract: Robots have the tremendous potential of assisting people in their lives, allowing them to achieve goals that they would not be able to achieve by themselves. In particular, socially assistive robots provide assistance primarily through social interaction, in healthcare, therapy, and education contexts. Despite their potential, current socially assistive robots still lack [...]
Carnegie Mellon University
Robot Deep Reinforcement Learning: Tensor State-Action Spaces and Auxiliary Task Learning with Multiple State Representations
Zoom Link Abstract: A long standing goal of robotics research is to create algorithms that can automatically learn complex control strategies from scratch. Part of the challenge of applying such algorithms to robots is the choice of representation. Reinforcement Learning (RL) algorithms have been successfully applied to many different robotic tasks such as the Ball-in-a-Cup [...]
Carnegie Mellon University
Online Inference of Joint Occupancy using Forward Sensor Models and Trajectory Posteriors for Deliberate Robot Navigation
Zoom Link Abstract: Robotic navigation algorithms for real-world robots require dense and accurate probabilistic volumetric representations of the environment in order to traverse efficiently. Sensor data in a Simultaneous Localisation And Mapping (SLAM) context, however, always has associated acquisition noise and pose uncertainty, and encoding this within the map representation while still maintaining computational tractability [...]
Carnegie Mellon University
Machine Learning Parallelism Could Be Adaptive, Composable and Automated
Zoom Link Abstract: In recent years, researchers in SysML have created algorithms and systems that parallelize ML training over multiple devices or computational nodes. As ML models become more structurally complex, many systems have struggled to provide all-round performance on a variety of models. Particularly, ML scale-up is usually underestimated in terms of the amount [...]
Carnegie Mellon University
Data-Driven Robotic Grasping in the Wild
Zoom Link Abstract: Humans can effortlessly grasp a wide variety of objects in diverse environments. On the other hand, robotic grasping has been extremely challenging in practice and is far from matching human dexterity. Despite recent progress in the community, most research is still largely focused on constrained environments like picking individual objects on a [...]
Carnegie Mellon University
Routing for Persistent Exploration in Dynamic Environments with Teams of Energy-Constrained Robots
Abstract: Disaster relief scenarios require rapid and persistent situational awareness to inform first-responders of safe and viable routes through a constantly shifting environment. Knowing what roads have become flooded or are suddenly obstructed by debris can significantly improve response time and ease the distribution of resources. In a sufficiently large environment, deploying and maintaining fixed [...]
Carnegie Mellon University
Coordinated online multi-robot planning
Abstract: Multi-robot applications frequently seek to employ human operators to direct robot actions online because fully automated planners struggle to encode human expertise or handle the extenuating circumstances that occur during real world operations. However, it is extremely challenging for a human to direct multi-robot teams, especially online, i.e., in real-time. From entertainment to defense, [...]
Sensor Planning for Large Numbers of Robots
Abstract: In the wake of a natural disaster, locating and extracting victims quickly is critical because mortality rises rapidly after the first forty-eight hours. In order to assist search and rescue teams and improve response times, teams of aerial robots equipped with sensors and cameras can engage in sensing tasks such as mapping buildings, assessing [...]
Carnegie Mellon University
Constraint-Based Coverage Path Planning: A Novel Approach to Achieving Energy-Efficient Coverage
Abstract: Despite substantial technological progress that has driven the proliferation of robots across various industries and aspects of our lives, the lack of a decisive breakthrough in energy storage capabilities has restrained this trend, particularly with respect to mobile robots designed for use in unstructured and unknown field environments. The fact that these domains are [...]
Carnegie Mellon University
Unsupervised Learning of the 4D Audio-Visual World from Sparse Unconstrained Real-World Samples
Abstract: We, humans, can easily observe, explore, and analyze the world we live in. We, however, struggle to share our observation, exploration, and analysis with others. This thesis introduce Computational Studio, computational machinery that can understand, explore, and create the four-dimensional audio-visual world. This allows: (1) humans to communicate with other humans without any loss [...]
Carnegie Mellon University
Expressive Real-time Intersection Scheduling: New Methods for Adaptive Traffic Signal Control
Abstract: Traffic congestion is a widespread problem throughout global metropolitan areas. In this thesis, we consider methods to optimize the performance of traffic signals to reduce congestion. We begin by presenting Expressive Real-time Intersection Scheduling (ERIS), a schedule-driven intersection control strategy that runs independently on each intersection in a traffic network. For each intersection, ERIS [...]
Carnegie Mellon University
Robust Manipulation with Active Compliance
Abstract: Human manipulation skills are filled with creative use of physical contacts to move the object about the hand and in the environment. However, it is difficult for robot manipulators to enjoy this dexterity since contacts may cause the manipulation task to fail by introducing huge forces or unexpected change of constraints, especially when modeling [...]
Carnegie Mellon University
Open-world Object Detection and Tracking
Abstract: Computer vision today excels at recognizing narrow slices of the real world: our models seem to accurately detect objects like cats, cars, or chairs in benchmark datasets. However, deploying models requires that they work in the open world, which includes arbitrary objects in diverse settings. Current methods struggle on both axes: they recognize only [...]
Carnegie Mellon University
Auto-generated Manipulation Primitives
Abstract: The central theme in robotic manipulation is that of the robot interacting with the world through physical contact. We tend to describe that physical contact using specific words that capture the nature of the contact and the action, such as grasp, roll, pivot, push, pull, tilt, close, open etc. We refer to these situation-specific [...]
Carnegie Mellon University
Learning 3D Registration and Reconstruction from the Visual World
Abstract: Humans learn to develop strong senses for 3D geometry by looking around in the visual world. Through pure visual perception, not only can we recover a mental 3D representation of what we are looking at, but meanwhile we can also recognize where we are looking at the scene from. Finding the 3D scene representation [...]
Carnegie Mellon University
Active Vision: Autonomous Aerial Cinematography with Learned Artistic Decision-Making
Abstract: Aerial cinematography is revolutionizing industries that require live and dynamic camera viewpoints such as entertainment, sports, and security. Fundamentally, it is a tool with immense potential to improve human creativity, expressiveness, and sharing of experiences. However, safely piloting a drone while filming a moving target in the presence of obstacles is immensely taxing, often [...]
Carnegie Mellon University
Understanding and Mitigating Biases in Evaluation
Abstract: There are many problems in real life that involve collecting and aggregating evaluation from people, such as hiring, peer grading and conference peer review. In this thesis, we focus on three sources of biases that arise in such problems, and propose methods to mitigate them. First, we study human bias, that is, the bias [...]
Carnegie Mellon University
Towards Safe and Resilient Autonomy in Multi-Robot Systems
Abstract: Autonomous systems such as robotic systems are envisioned to co-exist with humans in our daily lives, from household service to large-scale warehouse logistics, agricultural monitoring, and smart city. Reliable interactions among robots and humans require provably correct guarantees about safety and performance when designing robot behaviors. While traditional approaches for safety and performance analysis [...]
Carnegie Mellon University
Provably Constant-Time Motion Planning
Abstract: In many robotic applications, including logistics and manufacturing, robots often operate in semi-structured environments and perform highly repetitive manipulation tasks. Additionally, large parts of these environments are static most of the time. Fast and reliable motion planning is one of the key elements that ensure efficient operations in such environments. A very common example [...]
Carnegie Mellon University
Planning to Optimize and Learn Reward in Navigation Tasks in Structured Environments with Time Constraints
Abstract: Planning problems in which an agent must perform tasks for reward by navigating its environment while constrained by time and location have a wide variety of applications in robotics. Many real-world environments in which such planning problems apply, such as office buildings or city streets, are very structured. They consist of passages with notable [...]
Carnegie Mellon University
Robust and Scalable Perception For Autonomy
Abstract: Autonomous mobile robots have the potential to drastically improve the quality of our daily life. For example, self-driving vehicles could make transportation safer and more affordable. To navigate complex environments, such robots need a perception system that translates raw sensory data to high-level understanding. This thesis focuses on two fundamental challenges in learning such [...]
Carnegie Mellon University
Semantic Mapping for Autonomous Navigation and Exploration
Abstract: The last two decades have seen enormous progress in the sensors and algorithms for 3D perception, giving robots the means to build accurate spatial maps and localize themselves in them in real time. The geometric information in these maps is invaluable for navigation while avoiding obstacles, but insufficient, by itself, for robots to robustly [...]
Carnegie Mellon University
Efficient Robot Decision-Making for Achieving Multiple Independent Tasks
Abstract: We focus on robotics applications where a robot is required to accomplish a set of tasks that are partially observable and evolve independently of each other according to their dynamics. One such domain that we target in this work is decision-making for a robot waiter waiting tables at a restaurant. The robot waiter should [...]
Carnegie Mellon University
Decentralized Navigation of Quadrotor Teams in Uncertain Workspaces
Abstract: A fundamental requirement for realizing scalable and responsive real-world multi-robot systems for time-sensitive critical applications such as search and rescue or building clearance is a motion-planning and coordination framework that exhibits three essential properties. The first property is safety that encompasses aspects relating to kinodynamic feasibility and collision-avoidance. The second property is reliability that [...]
Carnegie Mellon University
Bayesian Models for Science-Driven Robotic Exploration
Abstract Planetary rovers allow for science investigations at remote locations. They have traversed many kilometers and made major scientific discoveries. However, rovers spend a considerable amount of time awaiting instructions from mission control. The reason is that they are designed for highly supervised data collection, not for autonomous exploration. The exploration of farther worlds will [...]
Carnegie Mellon University
Heuristics for routing and scheduling of Spatio-temporal type problems in industrial environments
Abstract: Spatio-temporal problems are fairly common in industrial environments. In practice, these problems come with different characteristics and are often very hard to solve optimally. So, practitioners prefer to develop heuristics that exploit mathematical structure specific to the problem for obtaining good performance. In this thesis, we will present work on heuristics for 3 different [...]
Carnegie Mellon University
Relationships in instance segmentation and anomaly detection
Abstract: This thesis primarily covers work on two different tasks in computer vision: (1) anomaly detection and (2) instance segmentation. Anomaly detection is an underexplored unsupervised problem that has existed in many fields. On the other hand, instance (and panoptic) segmentation is a supervised problem that can leverage the powerful data and key developments from [...]
Carnegie Mellon University
Dynamical Model Learning and Inversion for Aggressive Quadrotor Flight
Abstract: Quadrotor applications have seen a surge recently and many tasks require precise and accurate controls. Flying fast is critical in many applications and the limited onboard power source makes completing tasks quickly even more important. Staying on a desired course while traveling at high speeds and high accelerations is difficult due to complex and [...]
Carnegie Mellon University
Person Transfers Between Multiple Service Robots
Abstract: As more service robots are deployed in the world, human-robot interaction will not be limited to one-to-one interactions between users and robots. Instead, users will likely have to interact with multiple robots, simultaneously or sequentially, throughout their day to receive services and complete different tasks. In this thesis, I describe work in which my [...]
Carnegie Mellon University
Understanding, Exploiting and Improving Inter-view Relationships
Abstract: Multi-view machine learning has garnered substantial attention in various applications over recent years. Many such applications involve learning on data obtained from multiple heterogeneous sources of information, for example, in multi-sensor systems such as self-driving cars, or monitoring intensive care patient vital signs at their bed-side. Learning models for such applications can often benefit [...]
Carnegie Mellon University
Human-in-the-loop Control of Mobile Robots
Abstract: Human-in-the-loop control for mobile robots is an important aspect of robot operation, especially for navigation in unstructured environments or in the case of unexpected events. However, traditional paradigms of human-in-the-loop control have relied heavily on the human to provide precise and accurate control inputs to the robot, or reduced the role of the human [...]
Carnegie Mellon University
Planning and Execution using Inaccurate Models with Provable Guarantees on Task Completeness
Abstract: Modern planning methods are effective in computing feasible and optimal plans for robotic tasks when given access to accurate dynamical models. However, robots operating in the real world often face situations that cannot be modeled perfectly before execution. Thus, we only have access to simplified but potentially inaccurate models. This imperfect modeling can lead [...]
Carnegie Mellon University
Development of an Agile and Dexterous Balancing Mobile Manipulator Robot
Abstract: This thesis focuses on designing and controlling a dynamically stable shape-accelerating dual-arm mobile manipulator, the Carnegie Mellon University (CMU) ballbot. The CMU ballbot is a human-sized dynamically stable mobile robot that balances on a single spherical wheel. We describe the development of a pair of seven-degree-of-freedom (DOF) humanoid arms. The new 7-DOF arm pair [...]
Carnegie Mellon University
Learning and Inference in Factor Graphs with Applications to Tactile Perception
Abstract: Factor graphs offer a flexible and powerful framework for solving large-scale, nonlinear inference problems as encountered in robot perception and control. Typically, these methods rely on handcrafted models that are efficient to optimize. However, robots often perceive the world through complex, high-dimensional sensor observations. For instance, consider a robot manipulating an object in hand [...]
Carnegie Mellon University
Unified Simulation, Perception, and Generation of Human Behavior
Abstract: Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and [...]
Carnegie Mellon University
Search Algorithms and Search Spaces for Neural Architecture Search
Abstract: Neural architecture search (NAS) is recently proposed to automate the process of designing network architectures. Instead of manually designing network architectures, NAS automatically finds the optimal architecture in a data-driven way. Despite its impressive progress, NAS is still far from being widely adopted as a common paradigm for architecture design in practice. This thesis [...]
Carnegie Mellon University
Self-Improving 3D Scene Representations
Abstract: Most computer vision models in deployment today are not continually learning. Instead, they are in a “test” mode, where they will behave the same way perpetually, until they are replaced by newer models. This is a problem, because it means the models may perform poorly as soon as their “test” environment diverges from their [...]
Carnegie Mellon University
Direct-drive Hands: Making Robot Hands Transparent and Reactive to Contacts
Abstract: Industrial manipulators and end-effectors are a vital driver of the automation revolution. These robot hands, designed to reject disturbances with stiffness and strength, are inferior to their human counterparts. Human hands are dexterous and nimble effectors capable of a variety of interactions with the environment. Through this thesis we wish to answer a question: [...]
Carnegie Mellon University
Resource-Constrained Learning and Inference for Visual Perception
Abstract: We have witnessed rapid advancement across major computer vision benchmarks over the past years. However, the top solutions' hidden computation cost prevents them from being practically deployable. For example, training large models until convergence may be prohibitively expensive in practice, and autonomous driving or augmented reality may require a reaction time that rivals that [...]
Carnegie Mellon University
Physical Interaction and Manipulation of the Environment using Aerial Robots
Abstract: The physical interaction of aerial robots with their environment has countless potential applications and is an emerging area with many open challenges. Fully-actuated multirotors have been introduced to tackle some of these challenges. They provide complete control over position and orientation and eliminate the need for attaching a multi-DoF manipulation arm to the robot. [...]
Carnegie Mellon University
Visual Representation and Recognition without Human Supervision
Abstract: The advent of deep learning based artificial perception models has revolutionized the field of computer vision. These methods take advantage of the ever growing computational capacity of machines and the abundance of human-annotated data to build supervised learners for a wide-range of visual tasks. However, the reliance on human-annotated is also a bottleneck for [...]
Carnegie Mellon University
Learning Multi-Modal Navigation in Unstructured Environments
Abstract: A robot that operates efficiently in a team with humans in an unstructured outdoor environment must translate commands into actions from a modality intuitive to its operator. The robot must be able to perceive the world as humans do so that the actions taken by the robot reflect the nuances of natural language and [...]