Electroencephalogram-Based Emotion Recognition Using a Transformer Model with CNN Feature Extraction

Abstract

In this study the conformer architecture, combining Convolutional Neural Networks for feature extraction and the Transformer model for detecting long-range relations and for classification, is used to discriminate between emotional states evoked by pre-annotated movie clips. The study is based on the dataset of 16 subjects acquired recently in our lab. Two different cross-validation (CV) schemes are tested: stratified 5-fold CV and Leave-One-Subject-Out CV with calibration. The results indicate that the conformer architecture can efficiently discriminate between 9 distinct emotional states if 5-fold stratified CV is used (mean accuracy 90.99 %). LOSO CV combined with fine-tuning of the model with 10 % of the target subject's data also gave reasonably good results (mean accuracy 62.08 %) especially when considering the high number of distinct emotional states.