Exploring Language Model Generalization in Low-Resource Extractive QA
Sengupta, Saptarshi Yin, Wenpeng Nakov, Preslav Ghosh, Shreya Wang, Suhang
Sengupta, Saptarshi
Yin, Wenpeng
Nakov, Preslav
Ghosh, Shreya
Wang, Suhang
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize to domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to explain the performance gap empirically. Our findings suggest that: (a) LLMs struggle with dataset demands of closed domains such as retrieving long answer spans; (b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; (c) Scaling model parameters is not always effective for cross-domain generalization; and (d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.
Citation
S. Sengupta, W. Yin, P. Nakov, S. Ghosh, and S. Wang, “Exploring Language Model Generalization in Low-Resource Extractive QA,” Proceedings - International Conference on Computational Linguistics, COLING, vol. Part, pp. 7106–7126, Jan. 2025
Source
Proceedings - International Conference on Computational Linguistics, COLING
Conference
Keywords
Large Language Models (LLMs), Extractive Question Answering (EQA), Domain adaptation, Zero-shot learning, Closed-domain datasets
Subjects
Source
Publisher
Association for Computational Linguistics