Loading...
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Vashurin, Roman Fadeeva, Ekaterina Vazhentsev, Artem Rvanova, Lyudmila Vasilev, Daniil Tsvigun, Akim Petrakov, Sergey Xing, Rui Sadallah, Abdelrahman Grishchenkov, Kirill
Vashurin, Roman
Fadeeva, Ekaterina
Vazhentsev, Artem
Rvanova, Lyudmila
Vasilev, Daniil
Tsvigun, Akim
Petrakov, Sergey
Xing, Rui
Sadallah, Abdelrahman
Grishchenkov, Kirill
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The rapid proliferation of large language models (LLMs) has stimulated researchers to seek effective and efficient approaches to deal with LLM hallucinations and low-quality outputs. Uncertainty quantification (UQ) is a key element of machine learning applications in dealing with such challenges. However, research to date on UQ for LLMs has been fragmented in terms of techniques and evaluation methodologies. In this work, we address this issue by introducing a novel benchmark that implements a collection of state-of-the-art UQ baselines and offers an environment for controllable and consistent evaluation of novel UQ techniques over various text generation tasks. Our benchmark also supports the assessment of confidence normalization methods in terms of their ability to provide interpretable scores. Using our benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
Citation
R. Vashurin et al., “Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph,” Trans Assoc Comput Linguist, vol. 13, pp. 220–248, Apr. 2025, doi: 10.1162/TACL_A_00737.
Source
TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS
Conference
Keywords
Subjects
Source
Publisher
Association for Computational Linguistics