Home/PalmSight: an assistive technology helping the blind to locate and grasp objects

PalmSight: an assistive technology helping the blind to locate and grasp objects

Zhixuan Yu, Samantha J. Horvath, Alexandra Delazio, Jihang Wang, Rebeka Almasi, Roberta Klatzky, John Galeotti, George D. Stetten
CMU-RI-TR-16-59, Robotics Institute, Carnegie Mellon University, December, 2016

Download: Adobe portable document format (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


In this technical report, we introduce a novel hand-mounted intelligent device called PalmSight that aims at helping the blind and visually impaired identify, locate, and grasp an object in their surroundings. The present implementation of PalmSight acquires stereo visual information through a binocular camera mounted on user’s palm and provides haptic feedback via five vibrators attached on the back of the same hand, which direct the user to grasp the target object. A standard laptop computer is used to receive and process the acquired visual information, and then generate corresponding control signals for the five vibrators. PalmSight uses computer vision methods to identify and locate the target in 3D, and subsequently to realize vision-to-touch substitution. This assistive technology is an extension of the previous research in our laboratory on a device called FingerSight, which translates visual information to haptics stimulation on an individual finger for use in localization, navigation, and control by the blind and visually impaired. This report is submitted in partial fulfillment of the requirements for the degree of Master of Science in Biomedical Engineering at Carnegie Mellon University for Zhixuan Yu.

BibTeX Reference
title = {PalmSight: an assistive technology helping the blind to locate and grasp objects},
author = {Zhixuan Yu and Samantha J. Horvath and Alexandra Delazio and Jihang Wang and Rebeka Almasi and Roberta Klatzky and John Galeotti and George D. Stetten},
grantID = {NIH 1R01EY021641, NSF GRFP 0946825, and Research to Prevent Blindness},
school = {Robotics Institute, Carnegie Mellon University},
month = {December},
year = {2016},
number = {CMU-RI-TR-16-59},
address = {Pittsburgh, PA},