CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

CLaMP 3 is a unified framework developed to address challenges of cross-modal and cross-lingual generalization in music information retrieval. Using contrastive learning, it aligns all major music modalities–including sheet music, performance signals, and audio recordings–with multilingual text in a shared representation space, enabling retrieval across unaligned modalities with text as a bridge. It features a multilingual text encoder adaptable to unseen languages, exhibiting strong cross-lingual generalization. Leveraging retrieval-augmented generation, we curated M4-RAG, a web-scale dataset consisting of 2.31 million music-text pairs. This dataset is enriched with detailed metadata that represents a wide array of global musical traditions. To advance future research, we release WikiMT-X, a benchmark comprising 1,000 triplets of sheet music, audio, and richly varied text descriptions. Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple MIR tasks, significantly surpassing previous strong baselines and demonstrating excellent generalization in multimodal and multilingual music contexts.

Citation

S. Wu, G. Zhancheng, R. Yuan, J. Jiang, S. Doh, G. Xia, J. Nam, X. Li, F. Yu, M. Sun, "CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages," 2025, pp. 2605-2625.

Source

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Conference

Findings of the Association for Computational Linguistics: ACL 2025

Source

Findings of the Association for Computational Linguistics: ACL 2025

Publisher

Association for Computational Linguistics

DOI

10.18653/v1/2025.findings-acl.133

Additional links

https://aclanthology.org/2025.findings-acl.133/