Discovering Salient Regions on 3D Photo-textured Maps: Crowdsourcing Interaction Data from Multitouch Smartphones and Tablets

Abstract

We model human interest on 3D models using multi-touch interactions from tablets.We create a crowdsourcing system to gather interaction data from thousands of users.We propose using the view frustum and a Hidden Markov model for calculating saliency.We present results comparing the proposed interest model to traditional visual saliency.We report results demonstrating the proposed techniques on over 500,000 interactions. This paper presents a system for crowdsourcing saliency interest points for 3D photo-textured maps rendered on smartphones and tablets. An app was created that is capable of interactively rendering 3D reconstructions gathered with an Autonomous Underwater Vehicle. Through hundreds of thousands of logged user interactions with the models we attempt to data-mine salient interest points. To this end we propose two models for calculating saliency from human interaction with the data. The first uses the view frustum of the camera to track the amount of time points are on screen. The second uses the velocity of the camera as an indicator of saliency and uses a Hidden Markov model to learn the classification of salient and non-salient points. To provide a comparison to existing techniques several traditional visual saliency approaches are applied to orthographic views of the models' photo-texturing. The results of all approaches are validated with human attention ground truth gathered using a remote gaze-tracking system that recorded the locations of the person's attention while exploring the models.