Abstract:  In robotics, understanding the link between perception and action is pivotal. Typically, perception systems process sensory data into state representations like  segmentations and bounding boxes, which a planner uses to plan actions. However, this state estimation approach can fail in environments with partial observability, and in cases with challenging object properties like transparency and deformability.  [...]