Electroencephalogram-Based Emotion Recognition Using a Transformer Model with CNN Feature Extraction

Abstract

Transformer neural networks have become extremely powerful in deep learning with their exceptional capacity to capture complex patterns in sequential data. They were initially developed for natural language processing but their adaptability has extended to neuroscience, where the Electroencephalogram~(EEG) signals demand advanced methods of computation. This thesis investigates the application of a hybrid deep learning architecture that combines the extraction of features from the Convolutional Neural Network~(CNN) with a Transformer self-attention module to the problem of EEG-based emotion recognition across nine discrete emotional categories: Amusement, Anger, Calmness, Disgust, Excitement, Fear, Happiness, Sadness, and Surprise.

In the study, a dataset was recorded in our research group under ethical approval from the Human Sciences Ethics Committee of the Universities in Satakunta, Finland~(approval ID: 16.05.2025). Sixteen participants were exposed to pre-annotated movie clips drawn from a stimulus set validated on over 300 subjects. EEG was recorded using an EMOTIV EPOC-X wireless headset~(14 channels, 128 Hz).

The extended EEG Conformer was evaluated and validated under two cross-validation~(CV) schemes.~Under five-fold cross-validation, the model achieved a mean accuracy of 90.99\% ± 0.84\%.~Under Leave-One-Subject-Out~(LOSO) cross-validation, zero-shot performance was 11.87\%± 5.56\%, reflecting the challenge of inter-subject distributional shift. A subject-specific post-calibration was applied using 10\% of target-subject data and accuracy was increased to 62.08\% ± 12.11\%, an uplift of +50.21\%. These results illustrate a strong baseline for nine-class EEG emotion recognition from consumer-level hardware and highlight the importance of subject-specific adaptation for real world affective computing systems.