T2CT: Transformer-CTC Adaptive Framework for Continuous Sign Language Recognition
Aljubran, Murtadha Khan, Muhammad Haris
Aljubran, Murtadha
Khan, Muhammad Haris
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Continuous Sign Language Recognition (CSLR) is a challenging task due to the intricate linguistic structures of sign language and the scarcity of annotated datasets. Existing methods often rely on CTC-based frameworks that assume gloss independence or pretrained language models for linguistic modeling, limiting their ability to capture long-range dependencies and requiring more complex training pipelines. To address these challenges, we propose T2CT, a Transformer-CTC Adaptive Framework for CSLR, which integrates a Connectionist Temporal Classification (CTC) decoder and a Transformer decoder through an adaptive selection mechanism. This framework combines the CTC decoder's temporal alignment with the Transformer decoder's capacity to model global dependencies, offering an end-to-end trainable solution. Additionally, we leverage depth maps—a modality not previously utilized in CSLR—to enrich spatial representation and improve robustness to variations in signing styles and sequences. Experimental analyses, including cross-attention maps, reveal that the Transformer decoder effectively attends to semantically important glosses even when they appear further in the sequence, highlighting its ability to model global dependencies. Evaluated on the RWTH-PHOENIX-Weather 2014T dataset, T2CT achieves state-of-the-art results in a unimodal setting, with a Dev word error rate (WER) of 17.0% and a Test WER of 18.9%, underscoring its effectiveness for advancing CSLR.
Citation
Source
2025 International Joint Conference on Neural Networks (IJCNN)
Conference
2025 International Joint Conference on Neural Networks (IJCNN)
Keywords
Information and Computing Sciences, Language, Communication and Culture, Linguistics
Subjects
Source
2025 International Joint Conference on Neural Networks (IJCNN)
Publisher
IEEE