Seeing the Unseen: Closed-loop Occlusion Reasoning for Cloth Manipulation

Abstract

Robotic manipulation of cloth remains challenging due to the complex dynamics of cloth, lack of a low-dimensional state representation, and self-occlusions. Particularly, self-occlusion makes it difficult to estimate the full state of the cloth, which poses significant challenges to policy learning and dynamics modeling. Ideally, a robot trying to unfold a crumpled or folded cloth should be able to reason about the cloth's occluded regions. In this thesis, we aim to investigate different strategies of enabling the robot to reason about occlusion in a close-loop manner.

In the first part, we introduce a particle-based dynamics model on partial point cloud, and how we achieve implicit occlusion reasoning by graph imitation. Second, we propose to combine a neural network with self-supervised test-time fine tuning to explicitly reason about occlusion by cloth reconstruction. Third, we present a self-supervised method for cloth reconstruction in the real-world.