MSR Thesis Defense
Reinforcement Learning with Spatial Reasoning for Dexterous Robotic Manipulation
Abstract: Robotic manipulation in unstructured environments requires adaptability and the ability to handle a wide variety of objects and tasks. This thesis presents novel approaches for learning robotic manipulation skills using reinforcement learning (RL) with spatially-grounded action spaces, addressing the challenges of high-dimensional, continuous action spaces and alleviating the need for extensive training data. Our [...]
Leveraging Vision, Force Sensing, and Language Feedback for Deformable Object Manipulation
Deformable object manipulation represents a significant challenge in robotics due to its complex dynamics, lack of low-dimensional state representations, and severe self-occlusions. This challenge is particularly critical in assistive tasks, where safe and effective manipulation of various deformable materials can significantly improve the quality of life for individuals with disabilities and address the growing needs [...]
CBGT-Net: A Neuromimetic Architecture for Robust Classification of Streaming Data
Abstract: This research introduces CBGT-Net, a neural network model inspired by the cortico-basal ganglia-thalamic (CBGT) circuits in mammalian brains, which are crucial for critical thinking and decision-making. Unlike traditional neural network models that generate an output for each input or after a fixed sequence of inputs, CBGT-Net learns to produce an output once sufficient evidence [...]
Enhancing Robot Perception and Interaction Through Structured Domain Knowledge
Abstract: Despite the advancements in deep learning driven by increased computational power and large datasets, significant challenges remain. These include difficulty in handling novel entities, limited mechanisms for human experts to update knowledge, and lack of interpretability, all of which are crucial for human-centric applications like assistive robotics. To address these issues, we propose leveraging [...]
Towards Universal Place Recognition
Title: Towards Universal Place Recognition Abstract: Place Recognition is essential for achieving robust robot localization. However, current state-of-art systems remain environment/domain-specific and fragile. By leveraging insights from vision foundation models, we present AnyLoc, a universal VPR solution that performs across diverse environments without retraining or fine-tuning, significantly outperforming supervised baselines. We further introduce MultiLoc, and enable [...]
GNSS-denied Ground Vehicle Localization for Off-road Environments with Bird’s-eye-view Synthesis
Abstract: Global localization is essential for the smooth navigation of autonomous vehicles. To obtain accurate vehicle states, on-board localization systems typically rely on Global Navigation Satellite System (GNSS) modules for consistent and reliable global positioning. However, GNSS signals can be obstructed by natural or artificial barriers, leading to temporary system failures and degraded state estimation. On the [...]
Scaling up Robot Skill Learning with Generative Simulation
Abstract: Generalist robots need to learn a wide variety of skills to perform diverse tasks across multiple environments. Current robot training pipelines rely on humans to either provide kinesthetic demonstrations or program simulation environments with manually-designed reward functions for reinforcement learning. Such human involvement is an important bottleneck towards scaling up robot learning across diverse [...]
Simulation as a Tool for Conspicuity Measurement
Abstract: The use of unmanned aerial vehicles (UAVs) for time critical tasks is becoming increasingly popular. Operators are expected to use information from these swarms to make real-time and informed decisions. Consequently, detecting and recognizing targets from video is extremely pivotal to the success of these systems. At greater altitudes or with more vehicles, this [...]
VP4D: View Planning for 3D and 4D Scene Understanding
Abstract: View planning plays a critical role by gathering views that optimize scene reconstruction. Such reconstruction has played an important part in virtual production and computer animation, where a 3D map of the film set and motion capture of actors lead to an immersive experience. Current methods use uncertainty estimation in neural rendering of view [...]
Automating Annotation Pipelines by leveraging Multi-Modal Data
Abstract: The era of vision-language models (VLMs) trained on large web-scale datasets challenges conventional formulations of “open-world" perception. In this work, we revisit the task of few-shot object detection (FSOD) in the context of recent foundational VLMs. First, we point out that zero-shot VLMs such as GroundingDINO significantly outperform state-of-the-art few-shot detectors (48 vs. 33 AP) [...]
Leveraging Affordances for Accelerating Online RL
Abstract: The inability to explore environments efficiently makes online RL sample-inefficient. Most existing works tackle this problem in a setting devoid of prior information. However, additional affordances may often be cheaply available at the time of training. These affordances include small quantities of demo data, simulators that can reset to arbitrary states and domain specific [...]
Safe, Robust and Adaptive Model Learning for Agile Robots: Autonomous Racing
Abstract: In recent years there has been a rapid development in agile robots capable of operating at their limits in dynamic environments. Autonomous racing and recent developments in it also spurred by competitions such as the Indy Autonomous Challenge, A2RL, and F1Tenth have shown how modern autonomous control algorithms are capable of operating racecars at [...]
Improving Lego Assembly with Vibro-Tactile Feedback
Abstract: Robotic manipulation is an important area of research to improve the level of efficiency and autonomy in manufacturing processes. Due to the high precision and repeatability of industrial robot arms, robotic manufacturing tasks are dominated by simple pick, place, and peg insertion actions performed in a highly structured environment. Lego blocks are an excellent [...]
DeltaWalker: A Soft, Linearly Actuated Delta Quadruped Robot
Abstract: Quadruped robots offer a versatile solution for navigating complex terrain, making them valuable for applications such as industrial automation or search and rescue. Although quadrupeds are more complex than bipeds, they are easier to balance and control and require fewer joints to actuate compared to hexapods. Traditional quadruped designs, however, often feature complex leg [...]
Propagative Distance Optimization for Constrained Inverse Kinematics
Abstract: This work investigates a constrained inverse kinematic (IK) problem that seeks a feasible configuration of an articulated robot under various constraints such as joint limits and obstacle collision avoidance. Due to the high-dimensionality and complex constraints, this problem is often solved numerically via iterative local optimization. Classic local optimization methods take joint angles as [...]
Advancing Legged Robot Agility: from Video Imitation to GPU Acceleration
Abstract: Achieving human and animal-level agility has been a long-standing goal in robotics research. Recent advancements in numerical optimization and machine learning have pushed legged systems to greater capabilities than ever before, enabling black flips, parkour, and manipulation of heavy objects. Despite these exciting developments, this thesis identifies two key limitations of current legged robot [...]
Model Predictive Control on Resource-Constrained Robots
Abstract: Model predictive control (MPC) is a powerful tool for controlling highly dynamic robotic systems subject to complex constraints. However, it is computationally expensive and often requires a large memory footprint. Larger robotic systems are capable of carrying and powering sophisticated computational hardware onboard. On the other hand, smaller robots typically have faster dynamics that [...]
Enhancing Bipedal Locomotion With Reaction Wheels
Abstract: Legged robot hardware has become more accessible in the last ten years. However, there is still a dearth of low-cost hardware platforms that are open-source and easy to build. With recent developments in accessible manufacturing methods, such as 3D printing, it has become possible to design and manufacture parts without relying on precision machining. [...]
Building Micron: The Next Handheld Manipulator for Microsurgery
Abstract: Robotic assistance is used today in a variety of surgeries as a means of precise, dexterous, and minimally-invasive manipulation. However, practical use in microsurgical environments such as vitreoretinal surgery remains a challenge for the most common mechanically-grounded robotic platforms. Microsurgery requires micron-level accuracy and the ability to manipulate with interaction forces in millinewtons. Vitreoretinal [...]
Towards Estimation, Modeling, and Control of Mixed Material Flows on Variable-Speed Conveyor Belt Systems with Applications in Recycling
Abstract: Whether it is in sorting defects from grain in an agricultural setting, ore from tailings in a mine, or letters in a postal system, the sorting of bulk material has long been a crucial aspect of human industry. Today, in the face of dwindling natural resource deposits and accelerating climate change, a particularly important [...]
Expressive Attentional Communication Learning using Graph Neural Networks
Abstract: Multi-agent reinforcement learning presents unique hurdles such as the non-stationary problem beyond single-agent reinforcement learning that makes learning effective decentralized cooperative policies using an agent's local state extremely challenging. Effective communication to share information and coordinate is vital for agents to work together and solve cooperative tasks, as the ubiquitous evidence of communication in [...]
Estimating Object Importance and Modeling Driver’s Situational Awareness for Intelligent Driving
Abstract: The ability to identify important objects in a complex and dynamic driving environment can help assistive driving systems alert drivers. These assistance systems also require a model of the drivers' situational awareness (SA) (what aspects of the scene they are already aware of) to avoid unnecessary alerts. This thesis builds towards such intelligent driving [...]
Learning for Perception and Strategy: Adaptive Omnidirectional Stereo Vision and Tactical Reinforcement Learning
Abstract: Multi-view stereo omnidirectional distance estimation usually needs to build a cost volume with many hypothetical distance candidates. The cost volume building process is often computationally heavy considering the limited resources a mobile robot has. We propose a new geometry-informed way of distance candidates selection method which enables the use of a very small number [...]