Statement-Tuning: Enabling Efficient Transfer and Generalization in Natural Language Understanding with Encoder-only Models
Elshabrawy, Ahmed
Elshabrawy, Ahmed
Author
Supervisor
Department
Natural Language Processing
Embargo End Date
30/05/2025
Type
Thesis
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large Language Models (LLMs) excel at zero and few-shot learning but are often inefficient. In contrast, encoder-only models like BERT and RoBERTa are more efficient but lack generalization due to task-specific fine-tuning. This thesis proposes Statement Tuning, a method that reformulates tasks as natural language statements, enabling encoder-only models to perform zero-shot inference without specialized layers. We extend this to multilingual tasks using translated prompts and diverse datasets. Results show that Statement Tuning enables encoder-only models to rival LLMs in zero-shot multilingual settings while remaining faster and more memory efficient.
Citation
Ahmed Elshabrawy, “Statement-Tuning: Enabling Efficient Transfer and Generalization in Natural Language Understanding with Encoder-only Models,” Master of Science thesis, Natural Language Processing, MBZUAI, 2025.
Source
Conference
Keywords
Data-efficient Training, Data Augmentation, NLP in Resource-constrained Settings, Cross-lingual Transfer, Less-resourced Languages