System Usability Prediction Based on Eye-Tracking Data Using Machine Learning: A Case of Text-Entry in Virtual Reality

Abstract

The vast development and adoption of Extended Reality (XR) technology requires improving user experience through testing and evaluation. The ability to separate cases where users potentially encounter issues would be a beneficial solution to complement efforts in enhancing user experience. Therefore, this thesis explores the potential use cases of classifying above and below-average usability based on System Usability Scale (SUS) scores using eye-tracking data and machine learning from a text-entry experiment in Virtual Reality (VR). Two text-entry methods were tested which were speech-to-text and virtual keyboard. The recorded eye-tracking data from the experiment were extracted using a sliding window approach. The extracted metrics served as features for machine learning models, with Random Forest as the chosen algorithm and K-Nearest Neighbors (KNN) as the baseline algorithm. The best-performing models were further analyzed using SHapley Additive exPlanations (SHAP) to explain features’ impact on the models’ output and the potential links between eye-tracking data, cognitive processes, and system usability.

The best accuracy of the symbiosis model (i.e., the model trained using data from the two text-entry methods) achieved an average accuracy of 71.46%. Meanwhile, the method-specific models (i.e., the models trained using data from each specific text-entry method) achieved an average accuracy of 78.00% for the speech-to-text method and 74.05% for the virtual keyboard method. The SHAP analysis revealed the variations of features’ impact on the classification output highlighting the similarity and distinctiveness of participants’ gaze behavior between the two text-entry methods. Furthermore, some eye-tracking metrics indicated a correlation between previous research on eye-tracking data as a psychophysiological parameter of cognitive processes such as cognitive load and mental effort which might influence users’ opinion on system usability.