OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs
Wang, Yuxia Wang, Minghan Iqbal, Hasan Georgiev, Georgi N. Geng, Jiahui Gurevych, Iryna Nakov, Preslav
Wang, Yuxia
Wang, Minghan
Iqbal, Hasan
Georgiev, Georgi N.
Geng, Jiahui
Gurevych, Iryna
Nakov, Preslav
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the fac- tual accuracy of their outputs. Difficulties lie in assessing the factuality of free-form responses in open domains. Also, different pa- pers use disparate evaluation benchmarks and measurements, which renders them hard to compare and hampers future progress. To mitigate these issues, we propose OpenFactCheck, a unified framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. OpenFactCheck consists of three modules: (i) CUSTCHECKER allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims, (ii) LLMEVAL, a unified evaluation framework assesses LLM’s factuality ability from various perspectives fairly, and (iii) CHECKEREVAL is an extensible solution for gauging the reliability of automatic fact-checkers’ verification results using human-annotated datasets. Data and code are publicly available at https: //github.com/yuxiaw/openfactcheck.
Citation
Y. Wang et al., “OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs,” 2025. Accessed: Mar. 12, 2025. [Online]. Available: https://aclanthology.org/2025.coling-main.755/
Source
Proceedings of the 31st International Conference on Computational Linguistics, 2025
Conference
Keywords
OpenFactCheck framework, Customized fact-checking systems, Large Language Models (LLMs), Factual accuracy verification, Automatic fact-checkers
Subjects
Source
Publisher
Association for Computational Linguistics