Multimodal Analysis of Emotions in Gaming: Understanding Cultural Influences

Abstract

This study investigates the emotional dynamics from different cultural backgrounds using a multimodal approach that combines Facial Emotion Recognition and Heart Rate Variability (HRV) analysis. A total of 109 participants from Italy, Japan, and Korea (mean age = 24.5 years) played two casual games, namely Snake and Matching Pairs, to investigate cultural differences in emotional and physiological responses. The results revealed distinct cultural patterns in static emotional expression using generalised linear mixed models (GLMMs). The Italian cultural group showed higher levels of positive facial expressions (FE), particularly happiness; the Korean cultural group showed more frequent negative FE, while the Japanese cultural group showed restrained FE, particularly related to fear. Moreover, emotional transitions were analysed using a Markov-inspired continuousstate operator derived from probabilistic FE vectors, which characterised the temporal structure of emotional changes and uncovered systematic cross-cultural and task-dependent differences in emotional latency. These findings show that emotional transitions are shaped by cultural norms and the cognitive demands of the games. Furthermore, integrating FE and HRV features into GLMMs showed that autonomic indices predict performance and vary across game types. Overall, this study provides three key contributions. First, it indicates that FEs of emotion during gameplay differ significantly across cultures, in accordance with cultural display norms. Second, it demonstrates that emotional transitions are dynamic and influenced by game performance, with cultural background shaping these patterns. Third, it identifies cross-cultural differences in physiological responses, specifically bodily signals such as HRV. These findings enhance understanding of how games elicit and regulate emotional and physiological responses, suggesting applications beyond entertainment.