Tactile sensing for Robot Learning: Development to Deployment

Abstract

The role of tactile sensing is widely acknowledged for robots interacting with the physical environment. However, few contemporary sensors have gained widespread use among roboticists. This thesis proposes a framework for incorporating tactile sensing into a robot learning paradigm, from development to deployment, through the lens of ReSkin – a versatile and scalable magnetic tactile sensor. By examining design, integration, policy learning and representation learning in the context of ReSkin, this thesis aims to provide guidance on the implementation of effective sensing systems for robot learning.

We begin by proposing ReSkin – a low-cost, compact, and diverse platform for tactile sensing. We develop a self-supervised learning technique that enables sensor replaceability by adapting learned models to generalize to new instances of the sensor. Next, we investigate the scalability of ReSkin in the context of dexterous manipulation: we introduce the D’Manus, an inexpensive, modular, and robust platform with integrated large-area ReSkin sensing, aimed at satisfying the large-scale data collection demands of robot learning.

Based on the learnings from the development of ReSkin and the D’Manus, we propose AnySkin – an upgraded sensor tailored for robot learning that further reduces variability in response across sensor instances. AnySkin is as easy to integrate as putting on a phone case, eliminates the need for adhesion and demonstrates enhanced signal consistency. We deploy AnySkin in a policy learning setting for precise manipulation, demonstrate improved task performance when augmenting camera information, and exhibit zero-shot transfer of learned policies across sensor instances.

Going beyond sensor design and deployment, we explore representation learning for sensors including but not limited to ReSkin. Sensory data is typically sequential and continuous; however, most research on existing sequential architectures like LSTMs and Transformers focuses primarily on discrete modalities such as text and DNA. To address this gap, we propose Hierarchical State Space (HiSS) models, a conceptually simple and novel technique for continuous sequence-to-sequence prediction (CSP). HiSS creates a temporal hierarchy by stacking structured statespace models on top of each other, and outperforms state-of-the-art sequence models such as causal Transformers, LSTMs, S4, and Mamba. Further, we introduce CSPBench, a new benchmark for CSP tasks from real-world sensory data. CSP-Bench aims to address the lack of real-world datasets available for CSP tasks, providing a valuable resource for researchers working in this area.

Finally, we conclude by summarizing our takeaways throughout the journey of ReSkin from development to deployment, and outline promising directions for bringing tactile sensing into the fold of mainstream robotics research.