SimSense - Gestural Interaction Design for Information Exchange between Large Public Displays and Personal Mobile Devices

Abstract

Large displays in public and semi-public spaces continuously permeate our everyday lives as the price of display hardware continues to drop. These displays act as sources of information, entertainment and advertisement in public environments such as airports, hotels, universities, retail stores, hospitals, and stadiums, amongst others. The information shown on these displays often varies in form that ranges from simple text to rich interactive content. However, most of this rich information remains in the displays and methods to effectively retrieve them to ones’ mobile devices without the need to explicitly manipulate them remains unexplored.

Sensing technologies were used to implement a use case, wherein a person can simply walk up to a public display, retrieve interesting content onto their personal device without having the need to take it out of their pockets or bags. For this purpose a novel system called SimSense, which is capable of automatically detecting and establishing a connection with mobile phones that come in close proximity with the public display was developed. This thesis presents two alternative mid-air hand gesture interaction techniques: ‘Grab & Pull’ and ‘Grab & Drop’ to retrieve content from the public display without explicitly operating the mobile device. The results of a laboratory experiment conducted to evaluate these interaction techniques and gather preliminary impressions on the overall concept, are also presented. The results indicate that participants found ‘Grab & Pull’ to be slightly easier, more confident, and requires less effort to perform in comparison with ‘Grab & Drop’. Participants found the overall concept to be seamless and a useful way to retrieve interesting content.