Evaluating Interaction Modalities for Augmented Thermal Vision in Mixed Reality : A Comparative Study of Gaze and Hand-based Input

Abstract

Integrating live non-visible data streams, such as radiometric thermal imagery, into Mixed Reality (MR) creates opportunities for human augmentation but introduces a critical underexplored challenge: how users should interact with a live, dynamic sensory feed that is fundamentally different from the static virtual objects addressed by most Extended Reality (XR) interaction research. This thesis presents two primary contributions to address this challenge. First, a complete open-source thermal-MR platform is developed that integrates a Teledyne FLIR Hadron 640R (640×512 pixel) thermal sensor with a Varjo XR-3 head-mounted display via a low-latency ZeroMQ pipeline on an NVIDIA Jetson Nano, achieving a photon-to-photon latency of less than 50 ms. Second, a controlled empirical study with 24 participants compares two interaction modalities for operating this system across three thermal inspection tasks of increasing cognitive complexity: a gaze-based dwell-trigger technique (Gaze-Enlarge) and a gesture-based wrist-menu technique (Hand-Menu Zoom). A Linear Mixed-Effects Model on log-transformed completion times revealed no significant main effect of modality on task performance (p = .530), with performance differences dominated by task complexity rather than input method, and both modalities supported near-ceiling accuracy across all tasks. However, the two modalities imposed qualitatively different subjective costs: gaze interaction received substantially higher System Usability Scale scores (mean ≈70 vs. ≈42, Δ = 27.81, p < .001, Hedges’ g = 0.94), lower NASA Task Load Index workload ratings (p = .018), and significantly higher perceived naturalness ratings (p = .002). Interaction effort analysis further revealed a key adaptive strategy: under high cognitive load, participants shifted from continuous hand-slider adjustments to discrete button presses, prioritising interaction stability over fine grained control. These findings demonstrate that perceptual and motor effort in MR are co-regulated, support a gaze-to-select, hand-to-confirm design paradigm for augmented perception interfaces, and introduce the concept of Perceptual Equilibrium to describe users’ self-regulation of zoom behaviour toward a consistent perceptual information density regardless of the control modality employed. All software, hardware configurations, and 3D models are released openly to support reproducible research.