Synchrony Between Facial Expressions and Heart Rate Variability During Game-Based Learning: Insights from Cross-Wavelet Transformation

Abstract

Game-based learning (GBL) environments are designed to foster emotional experiences conducive to learning; yet, there are mixed findings regarding their effectiveness. The inconsistent results may stem from challenges in measuring and modeling emotions as multi-dimensional constructs during GBL. Traditional approaches often use one data channel and conventional statistics to study emotions, which limit our understanding of the multi-componential interactions that underlie emotional states during GBL. In this study, we merged non-linear dynamical systems (NLDS) theory with the component process model of emotion to examine interactions and synchrony among two emotion signals during GBL, facial expressions and heart rate variability (HRV), and assessed its relation to knowledge and learning gain. Data were collected from 58 participants (n = 58) at a university in Central Finland while they learned about pathology with a tower defense game called Antidote COVID-19. Results showed a significant improvement in knowledge after GBL. A NLDS technique called cross-wavelet transformation showed there were varying degrees of synchrony between facial expressions and HRV. Neutral expressions showed the highest degree of synchrony with HRV, followed closely by happiness and anger with HRV. However, the synchrony between facial expressions and HRV did not affect knowledge and learning gain. This research contributes to the field by studying emotions as multidimensional systems during GLB.