Mid-Air Gestural Interaction with Large Interactive Displays

Abstract

Gestures, a fundamental element of human communication, have become a natural way to interact with various multimedia displays. Depth-sensing computer vision technologies, which detect user gestures and body movements, eliminate the need for physical contact during interaction.

The present doctoral thesis investigated the capabilities of mid-air gestural interaction with large interactive displays, focusing on overall user experience, including usability, functionality, and satisfaction. Specifically, the research examined two primary interface actions: <i>translation</i>, which refers to moving entities while retaining their size and rotation, and <i>confirmation</i>, which refers to confirming a user selection, decision, or movement.

Study I explored the translation performance and subjective evaluation of different mid-air gestures (i.e. <i>fist-, palm-, pinch-based</i>, and <i>sideways</i>) with large projected screens. Participants (<i>N = 30</i>) used these gestures to repetitively translate 2D objects over short and long distances. The results showed that the palm-based gesture was the easiest and most accurate over the others for both distances. The gestures that were ranked as second-best varied with distance. Specifically, the fist-based gesture performed well for short distances, while the sideways gesture was better for long distances in terms of error rates and target re-entries.

Study II investigated direct gestural interaction with a large walk-through fogscreen display for the selection confirmation action. Participants (<i>N = 20</i>) repeatedly selected objects using <i>tapping</i> and <i>dwell-based</i> gestural techniques with audio-visual and audio-visual-haptic feedback. The results showed that tapping gestures with audio-visual feedback yielded approximately 60% higher throughput, was faster, and produced fewer reentries than dwell-based selections. While the inclusion of haptic feedback had minimal effect on the interaction performance of both gestures, users preferred it for dwell-based gesture.

Study III investigated the potential of mid-air gestural techniques specifically designed for children, focusing on full-body motion control with large projected displays. Two types of confirmation gestures were explored: active full-body gesturing (i.e., <i>jumps </i>and <i>jumps with hands up</i>,) and a freezing task (i.e., maintaining <i>predefined motionless postures</i> for a set time) in two experiments. In Experiment I, typically developing children (<i>N = 11</i>) evaluated the system’s functionality and robustness. The results showed that gesture recognition software was accurate and robust, recognizing successfully 92% of jumps, 95% of jumps with hands up, and it tracked consistently body parts during freezing tasks. Experiment II investigated the feasibility and usability of these gestures for confirmation action over six game sessions with children having attention deficits (<i>N = 18</i>). The results showed that the children interacted with consistent accuracy and responsiveness in both active gesturing and freezing scenarios. Jumps with hands up were found as feasible, preferable, and more reliable confirmation method compared to simple jumps. In the freezing task, considered a special case of dwell-based gestures, results showed that children successfully repeated predefined postures in 83% of tasks and remained motionless in 73% of tasks, finding it effective and suitable as a solution for confirmation actions.

Study IV was the first exploratory study to evaluate the applicability, performance, and user satisfaction of gestural interaction with a large fogscreen in game settings with children having attention deficits (<i>N = 18</i>). This study examined how participants used hand gestures to translate puzzle pieces on the fogscreen over six game sessions. The results showed that the proposed gestural interface effectively supported the gameplay. Participants typically needed two to three attempts to translate a puzzle piece precisely, with 42% of movements involving adjustments after an initial placement to achieve the desired alignment. The results also provided insights for improving visual feedback.

In conclusion, this thesis demonstrates the effectiveness and feasibility of mid-air gestural techniques for translation and confirmation actions with large interactive displays. The results support future studies in implementing effective mid-air gestural interfaces by identifying functional and usable gestures and their application in various interaction contexts. Overall, this thesis suggests functional solutions that address the specific needs of both, adults and children when aiming to improve midair gestural interaction performance, user engagement, and satisfaction.