Unimodal vs. Audio-Visual Transformers : Feature Quality In Classification

Abstract

Deep learning, a branch of machine learning, has become increasingly influential across numerous areas of computer science. This thesis examines multimodal learning within transformer based architectures, with a particular focus on evaluating the performance and representational quality of such models.

The objective is to evaluate the quality of feature representations produced by transformer models across different modalities. Additionally, the research explores how unimodal versus integrated audio-visual representations impact classification performance. To conduct this investigation, we employ a framework that utilizes a linear probing task to assess the embeddings generated by pretrained transformer models.

Three distinct transformers are explored in this study: AST, Video SwinTransformer, and V-AURA, with V-AURA serving as the audio-visual model compared to the two unimodal models. The unimodal models set up a baseline for comparison. All three are open-source models, and they are evaluated on the VGGSound dataset.

The results indicate that V-AURA benefits in classification when using both audio and video modalities. However, when compared to the baseline unimodal models, V-AURA performs worse. Both unimodal models perform better on the defined probing task in this study than the audio-visual model V-AURA.