Abstract:
Object manipulation is a well-studied domain in robotics, yet manipulation remains difficult for objects with visually and geometrically challenging properties. Visually challenging properties, such as transparency and specularity, break assumptions of Lambertian reflectance that existing methods rely on for grasp estimation. On the other hand, deformable objects such as cloth pose both visual and geometric challenges due to self-occlusions and non-rigid motion. Objects with these challenging properties are ubiquitous, and robotic systems will have to manipulate them when operating in human environments to perform autonomous and assistive tasks.

The goal of this thesis is to develop approaches for manipulating visually and geometrically challenging objects. First, we propose multi-modal transfer learning to improve grasping for transparent and specular objects (RA-L+ICRA ’20). Then, we turn our focus toward manipulating cloth. In one work, we combine semantic segmentation with uncertainty estimation to precisely grasp cloth at hems and corners, as opposed to wrinkles and folds (IROS ’20). In another, we learn a goal-conditioned bimanual cloth manipulation policy (CoRL ’21). This policy leverages optical flow in several key ways to improve manipulation performance. Lastly, we depart from vision-only approaches to develop a tactile classifier for cloth grasping (in submission).

The proposed work builds on the completed work, with the aim to combine vision and tactile sensing for manipulating challenging objects. Vision and tactile sensing have complementary strengths, with vision providing global but noisy information, and tactile providing local, more accurate information. We propose developing visuo-tactile policies that aggregate multi-modal sensor observations, to manipulate either cloth or transparent and specular objects.

Thesis Committee Members:
Dave Held, Chair
Oliver Kroemer
Shubham Tulsiani
Alberto Rodriguez, MIT

More Information