Enhancing Accessibility in Virtual Collaboration : An ASL-to-English Translator

Abstract

This thesis describes and evaluates the performance of three different deep learning architectures for real-time American Sign Language (ASL) recognition: PureLandmarkTemporal (PLT) - a Transformer encoder that uses skeletal landmark sequences; WithTunedMo-bileNetV2 (MN2) - a hybrid CNN-BiLSTM (Convolutional Neural Network Bidirectional Long Short-Term Memory) model that utilizes both skeletal image renderings and landmark sequences; and Fusion - a three branch model that incorporates CNN spatial feature extraction, Transformer-based temporal feature extraction, and LSTM-based micro-motion feature extraction. The training of all three models was conducted using a 40-class dataset that contained 158,157 samples. The samples included static fingerspelling letters (A–Z, del, space) and dynamic word signs. The training process involved a three-step curriculum training strategy, and a two-level weighted sampling strategy to compensate for the significant difference in size of the letter classes versus the word classes (98% vs. 2%), as well as the variability in the signers introduced by incorporating personal video recordings into public datasets. An ablation study was performed to assess the contribution of velocity features, geometric hand geometry hint features, domain randomization augmentation, and weighted sampling to the overall performance of each model. The results indicated that MN2 achieved the highest letter recognition accuracy (letter F1 = 0.987) and demonstrated superior performance in recognizing fine-grained fist signs (N: F1 = 0.96) and dynamic signs (J: F1 = 0.99). Fusion achieved the highest word recognition accuracy (word F1 = 0.920) and was the only model to resolve a catch-all failure in which the model incorrectly predicts the me sign for ambiguous inputs from other classes — a failure mode characterized by high recall but very low precision. Fusion achieved me class precision of 0.91, compared to 0.46 for MN2 and 0.18 for PLT. PLT exhibited the lowest inference latency (~30–50 ms). All three models were integrated into a custom-made Microsoft Teams application that transcribed real-time video feeds into text and displayed the output as live captions for users. In general, this study illustrates that it is possible to achieve real-time ASL recognition of both static fingerspelling letters and dynamic word signs on consumer GPU hardware, and that the optimal architecture will depend on whether the primary goal of the target application is to maximize letter recognition accuracy, word recognition accuracy, or minimize inference latency.