A Survey on Spatio-Temporal Prediction: From Transformers to Foundation Models
Mao, Yingchi Zhou, Hongliang Chen, Ling Qi, Rongzhi Sun, Zhende Rong, Yi He, Xiaoming Chen, Mingkai Mumtaz, Shahid Frascolla, Valerio
Mao, Yingchi
Zhou, Hongliang
Chen, Ling
Qi, Rongzhi
Sun, Zhende
Rong, Yi
He, Xiaoming
Chen, Mingkai
Mumtaz, Shahid
Frascolla, Valerio
Supervisor
Department
Machine Learning
Embargo End Date
Type
Journal article
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Spatio-Temporal (ST) data is pervasive on the various aspects in our daily lives. By mining the ST information from the data, we are able to predict trends in numerous domains. The Transformer, and one of its more recent enhancements, foundation models, have achieved a remarkable success in such ST prediction. In this paper, we first survey the state of the art of Transformers-related work, then introduce the network architecture of the Transformer and summarize the improvements to adapt to the ST prediction Transformer and foundation models, including module enhancement and adjustment. Subsequently, we categorize the ST Transformer and foundation models in selected applications in some relevant domains, mainly urban transportation, climate monitoring, and motion prediction. Next, we propose an evaluation method in the ST prediction with Transformers and foundation models, list the most relevant open-source datasets, evaluation metrics and performance analysis. Finally, we discuss some future directions on the task of ST prediction with Transformer and foundation models. Relevant papers and open-source resources have been collated and are continuously updated at: https://github.com/cyhforlight/Spatio-Temporal-Prediction-Transformer-Review.
Citation
MaoYingchi et al., “A Survey on Spatio-Temporal Prediction: From Transformers to Foundation Models,” ACM Comput Surv, Sep. 2024, doi: 10.1145/3766546
Source
ACM Computing Surveys
Conference
Keywords
Subjects
Source
Publisher
Association for Computing Machinery