More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives
Zhang, Xiaoqing Lv, Ang Liu, Yuhan Sung, Flood Liu, Wei Luan, Jian Shang, Shuo Chen, Xiuying Yan, Rui
Zhang, Xiaoqing
Lv, Ang
Liu, Yuhan
Sung, Flood
Liu, Wei
Luan, Jian
Shang, Shuo
Chen, Xiuying
Yan, Rui
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large language models (LLMs) excel at few-shot in-context learning (ICL) without requiring parameter updates. However, as ICL demonstrations increase from a few to many, performance tends to plateau and eventually decline. We identify two primary causes for this trend: the suboptimal negative log-likelihood (NLL) optimization objective and the incremental data noise. To address these issues, we introduce DrICL, a novel optimization method that enhances model performance through Differentiated and Reweighting objectives. Globally, DrICL utilizes differentiated learning to optimize the NLL objective, ensuring that many-shot performance surpasses zero-shot levels. Locally, it dynamically adjusts the weighting of many-shot demonstrations by leveraging cumulative advantages inspired by reinforcement learning, thereby mitigating the impact of noisy data. Recognizing the lack of multi-task datasets with diverse many-shot distributions, we develop the Many-Shot ICL Benchmark (ICL-50)-a large-scale benchmark of 50 tasks that cover shot numbers from 1 to 350 within sequences of up to 8,000 tokens-for both fine-tuning and evaluation purposes. Experimental results demonstrate that LLMs enhanced with DrICL achieve significant improvements in many-shot setups across various tasks, including both in-domain and out-of-domain scenarios. We release the code and dataset hoping to facilitate further research in many-shot ICL.
Citation
X. Zhang et al., “More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives,” vol. 1, pp. 30539–30552, Aug. 2025, doi: 10.18653/V1/2025.ACL-LONG.1475
Source
Proceedings of the Annual Meeting of the Association for Computational Linguistics
Conference
63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025
Keywords
Many-Shot In-Context Learning, Differentiated Objective, Reweighting Demonstrations, In-Context Learning Benchmark (ICL-50), Large Language Models, Data Noise Mitigation, Few-to-Many-Shot Scaling, Out-of-Domain Generalisation
Subjects
Source
63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025
Publisher
Association for Computational Linguistics