What Makes Cryptic Crosswords Challenging for LLMs?
Sadallah, Abdelrahman Kotova, Daria Kochmar, Ekaterina
Sadallah, Abdelrahman
Kotova, Daria
Kochmar, Ekaterina
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels, dealing with various types of wordplay. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs). However, there is little to no research on the reasons for their poor performance on this task. In this paper, we establish the benchmark results for three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance on this task is still significantly below that of humans. We also investigate why these models struggle to achieve superior performance. We release our code and introduced datasets at https://github.com/bodasadallah/decrypting-crosswords.
Citation
A. Sadallah, D. Kotova, and E. Kochmar, “What Makes Cryptic Crosswords Challenging for LLMs?,” Proceedings - International Conference on Computational Linguistics, COLING, vol. Part, pp. 5102–5114, Jan. 2025.
Source
Proceedings - International Conference on Computational Linguistics, COLING
Conference
Keywords
Cryptic crosswords, Large Language Models (LLMs), Wordplay challenges, Natural Language Processing (NLP), Benchmarking
Subjects
Source
Publisher
Association for Computational Linguistics