TERA: Time Series Prediction through Transformer-Enabled Retrieval Augmentation
Mikhailov, Ruslan
Mikhailov, Ruslan
Author
Supervisor
Department
Machine Learning
Embargo End Date
2024-01-01
Type
Thesis
Date
2024
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Time series prediction plays a crucial role in many fields, including finance and healthcare, where forecasting future events with high accuracy using historical data can result in significant breakthroughs. More efficient techniques are required because traditional methods frequently need to improve when dealing with the complexity of real-world data. Transformer-Enabled Retrieval Augmentation (TERA), a new approach that uses retrieval-augmented mechanisms within transformer-based encoder-decoder architectures to improve predictive accuracy, is introduced in this thesis. The central hypothesis of this study is that continuous re-parametrization of the knowledge base can help models perform better in forecasting tasks. The study conducts a three-dimensional analysis to test this hypothesis: it compares retrieval-augmented methods with one another, assesses their performance on various datasets, and investigates their effectiveness in different transformer configurations. TERA showed a significant improvement in error reduction with an average of 44% decrease in error compared to 23% for MQRetNN. Though TERA has many advantages, it also has drawbacks. For example, training time increases significantly, and complexity increases quadratically with the increase of context size. Additionally, TERA s performance is limited on datasets that exhibit strong stationarity. These difficulties point to significant directions for further study in enhancing the retrieval process and broadening the techniques to different neural network topologies. By setting new benchmarks for accuracy and dependability, the results of this thesis advance the field of time series prediction research and pave the way for the valuable implementation of retrieval-augmented techniques in various dynamic and heterogeneous settings. The source code of the project is available publicly on github: github.com/Dmmc123/time-wise-rat
Citation
R. Mikhailov, "TERA: Time Series Prediction through Transformer-Enabled Retrieval Augmentation", MS. Thesis, Machine Learning, MBZUAI, Abu Dhabi, UAE, 2024