Data-Driven Robotic Grasping in the Wild

Abstract

Robotic grasping has seen tremendous advancements in recent years. Yet, the current paradigm of manipulation research is typically some form of table-top manipulation in constrained setups or in simulation. Building general purpose personal robots that can autonomously grasp unknown objects in unstructured environments like homes is an open problem. In this thesis, we explore important directions in scaling data-driven grasping to the diversity and constraints imposed by the real world.

We first discuss how we can go beyond picking individual objects in isolation to 6- DOF grasping in clutter. Most existing methods train policies on datasets collected in curated settings (in lab or simulation) and hence may not cope with the mismatch in data distribution when deployed in the wild. We build and open-source a low-cost mobile manipulator platform to parallelize data collection in challenging settings like homes and show that policies trained on this data generalize to novel objects in unseen homes. As a result, we also discuss ideas for scaling robot learning with several robots and transferring policies between different hardware. Yet, we hypothesize that visual perception alone is insufficient for robustness and present a self-supervised tactile-based re-grasping framework to close the loop on grasp execution. Lastly, we strive to go beyond robotic pick-and-place and generalize to diverse semantic manipulation tasks. We do so by scaling task-oriented grasping datasets with crowdsourcing and learning from semantic information like knowledge graphs.