MQM-Chat: Multidimensional Quality Metrics for Chat Translation
Li, Yunmeng Suzuki, Jun Morishita, Makoto Abe, Kaori Inui, Kentaro
Li, Yunmeng
Suzuki, Jun
Morishita, Makoto
Abe, Kaori
Inui, Kentaro
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The complexities of chats, such as the stylized contents specific to source segments and dialogue consistency, pose significant challenges for machine translation. Recognizing the need for a precise evaluation metric to address the issues associated with chat translation, this study introduces Multidimensional Quality Metrics for Chat Translation (MQM-Chat), which encompasses seven error types, including three specifically designed for chat translations: ambiguity and disambiguation, buzzword or loanword issues, and dialogue inconsistency. In this study, human annotations were applied to the translations of chat data generated by five translation models. Based on the error distribution of MQM-Chat and the performance of relabeling errors into chat-specific types, we concluded that MQM-Chat effectively classified the errors while highlighting chat-specific issues explicitly. The results demonstrate that MQM-Chat can qualify both the lexical accuracy and semantical accuracy of translation models in chat translation tasks.
Citation
Y. Li, J. Suzuki, M. Morishita, K. Abe, and K. Inui, “MQM-Chat: Multidimensional Quality Metrics for Chat Translation,” Proceedings - International Conference on Computational Linguistics, COLING, vol. Part, pp. 3283–3299, Jan. 2025, Accessed: Apr. 02, 2025. [Online]. Available: https://aclanthology.org/2025.coling-main.221/
Source
Proceedings - International Conference on Computational Linguistics, COLING
Conference
Keywords
Chat translation evaluation, Multidimensional Quality Metrics (MQM), Ambiguity, disambiguation, Buzzword or loanword issues, Dialogue inconsistency?
Subjects
Source
Publisher
Association for Computational Linguistics