PalmSight: an assistive technology helping the blind to locate and grasp objects

Abstract

In this technical report, we introduce a novel hand-mounted intelligent device called PalmSight that aims at helping the blind and visually impaired identify, locate, and grasp an object in their surroundings. The present implementation of PalmSight acquires stereo visual information through a binocular camera mounted on user’s palm and provides haptic feedback via five vibrators attached on the back of the same hand, which direct the user to grasp the target object. A standard laptop computer is used to receive and process the acquired visual information, and then generate corresponding control signals for the five vibrators. PalmSight uses computer vision methods to identify and locate the target in 3D, and subsequently to realize vision-to-touch substitution. This assistive technology is an extension of the previous research in our laboratory on a device called FingerSight, which translates visual information to haptics stimulation on an individual finger for use in localization, navigation, and control by the blind and visually impaired. This report is submitted in partial fulfillment of the requirements for the degree of Master of Science in Biomedical Engineering at Carnegie Mellon University for Zhixuan Yu.