Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating a robot system efficiently and with high fidelity, from sensing, perception to manipulation, has been a long-standing challenge. Tactile sensing, as one of the sensing modalities has shown its essential functionality in robotic applications such as manipulation, while its simulation is still under exploration. This thesis contributes an example-based simulation model for GelSight, a dense optical tactile sensor and its applications in robotic perception and manipulation with sim-to-real transfer.

First, we propose Taxim, a realistic and efficient simulation model for a vision-based tactile sensor, GelSight which measures the contact geometries from images and forces from marker motions: we simulate the optical response to the deformation of contact by mapping the contact geometries to pixel intensity sampled by the embedded camera. We also simulate the surface markers’ motion caused by the surface stretch of the elastomer under contact. To the best of our knowledge, our simulation framework is the first to incorporate marker motion field simulation that derives from elastomer deformation together with the optical simulation, creating a comprehensive and computationally efficient tactile simulation framework.

Second, we present ObjectFolder 2.0, a large-scale multisensory dataset of common household objects in the form of implicit neural representations from vision, audio and touch where touch representation is learned from virtual objects based on Taxim simulation model. We show that models learned from simulation in our dataset successfully transfer to their real-world counterparts in three perception tasks: object scale estimation, contact localization and shape-reconstruction.

In the end, we integrate simulation of robot dynamics and Taxim simulation for vision-based tactile sensors by modeling the physics of contact. This contact model uses simulated contact forces at the robot’s end-effector to inform the generation of realistic tactile outputs. We demonstrate the effectiveness of our system on a sim-to-real grasp stability prediction task on various objects. Experiments reveal the potential of applying our simulation framework on more complicated manipulation tasks.