Abstract:
To perform manipulation tasks in the real world, robots need to operate on objects with various shapes, sizes and without access to geometric models. It is often infeasible to train monolithic neural network policies across such large variance in object properties. Towards this generalization challenge, we propose task-axis controllers, which are defined relative to the objects being manipulated. We use reinforcement learning to learn task policies that hierarchically compose these task-axis controllers. Since task-axes controllers are parameterized by properties associated with underlying objects in the scene, we also infer these controller parameters directly from visual input using multi-view dense correspondence learning. Our overall approach provides a simple, modular and yet powerful framework for learning manipulation tasks. Experiments in both simulation and real world show how the proposed approach leads to improved sample efficiency, zero-shot generalization to novel test environments, and simulation-to-reality transfer without fine-tuning.

Committee:
Oliver Kroemer (Chair)
Abhinav Gupta
Yonatan Bisk
Senthil Purushwalkam