KG4RecEval: Does Knowledge Graph Really Matter for Recommender Systems?
Zhang, Haonan Wang, Dongxia Sun, Zhu Li, Yanhui Sun, Youcheng Liang, Huizhi Wang, Wenhai
Zhang, Haonan
Wang, Dongxia
Sun, Zhu
Li, Yanhui
Sun, Youcheng
Liang, Huizhi
Wang, Wenhai
Author
Supervisor
Department
Computer Science
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Recommender systems (RSs) are designed to provide personalized recommendations to users. Recently, knowledge graphs (KGs) have been widely introduced in RSs to improve recommendation accuracy. In this study, however, we demonstrate that RSs do not necessarily perform worse even if the KG is downgraded to the user-item interaction graph only (or removed). We propose an evaluation framework KG4RecEval to systematically evaluate how much a KG contributes to the recommendation accuracy of a KG-based RS, using our defined metric KG utilization efficiency in recommendation (KGER). We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users. Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users. These findings inspire us to rethink how to better utilize knowledge from existing KGs, whereby we discuss and provide insights into what characteristics of datasets and KG-based RSs may help improve KG utilization efficiency. The code and supplementary material of this article are available at: https://github.com/HotBento/KG4RecEval.
Co-author(s)
H Zhang, D Wang, Z Sun, Y Li, Y Sun, H Liang, W Wang
Citation
ZhangHaonan et al., “KG4RecEval: Does Knowledge Graph Really Matter for Recommender Systems?,” ACM Trans Inf Syst, vol. 43, no. 3, pp. 1–36, Feb. 2025, doi: 10.1145/3713071.
Source
ACM Transactions on Information Systems
Conference
Keywords
Reproducibility, Knowledge Graph, Heterogeneous Information Network
Subjects
Source
Publisher
Association for Computing Machinery