Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating and Improving LLMs for Competitive Programming
Hadhoud, Sama Mahmoud Mohamed
Hadhoud, Sama Mahmoud Mohamed
Citations
Altmetric:
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Thesis
Date
2026
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic reasoning with code-level implementation. We argue that competitive programming is fundamentally a problem-solving task and propose centering natural-language editorials in both solution generation and evaluation. Generating an editorial prior to code improves solve rates for some LLMs, with substantially larger gains when using expertly written gold editorials. However, even with gold editorials, models continue to struggle with implementation, while the gap between generated and gold editorials reveals a persistent problem-solving bottleneck in specifying correct and complete algorithms. Beyond pass/fail metrics, we diagnose reasoning errors by comparing model-generated editorials to gold standards using expert annotations and validate an LLM-as-a-judge protocol for scalable evaluation. We introduce a dataset of 83 ICPC-style problems with gold editorials and test suites, and evaluate 19 LLMs, arguing that future benchmarks should explicitly separate problem solving from implementation.
To address the implementation gap, we decompose editorial-to-code generation into a multi-agent pipeline with specialized planning, critique, implementation, and debugging agents. We show that prompt optimisation via an evolutionary, blame-guided search with a difficulty-bucketed curriculum raises a 20B-parameter model's pass@1 from 47.0\% to 59.0\%, matching much larger frontier models. These results demonstrate that pipeline design and prompt optimisation are underexplored dimensions of model capability in competitive programming.
Citation
Hadhoud, Sama Mahmoud Mohamed, "Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating and Improving LLMs for Competitive Programming," M.S. Thesis, Natural Language Processing, MBZUAI, 2026.
Source
Conference
Keywords
Editorial-Based Evaluation, Competitive programming, Large language models, Reasoning–implementation disentanglement, Multi-Agent Systems, Prompt Optimization