Improvised Performance Following in Real Time for Automatic Accompaniment
Jiang, Junyan Maezawa, Akira Xia, Gus
Jiang, Junyan
Maezawa, Akira
Xia, Gus
Author
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Real-time performance tracking is the core component of automatic accompaniment systems. Previous methods typically require full information on the performance score (the human part) and also assume very limited improvisations off the score, otherwise, the tracker will get lost. In this paper, we aim to track a fully improvised performance for automatic accompaniment, in which the only reference is the accompaniment score (the machine part). The key idea is to incorporate semantic-level alignment to model the correspondence between the user performance and the accompaniment. Our system contains two real-time components: The first component is an accompaniment-conditioned quantization model using a Long Short-Term Memory (LSTM) layer. For each performance note onset, we first compute its rough projected score position using previously estimated performance-to-score mapping and then refine the mapping using the quantization results. The second component is a playback control system, which updates the performance-to-score mapping via the performance-quantized time pair (similar to the performance-score alignment pair in traditional automatic accompaniment systems). Experiments show the system achieves better tracking results on expressive piano performance than baselines, allowing score-free tracking on both texture and tempo variations.
Citation
J. Jiang, A. Maezawa and G. Xia, "Improvised Performance Following in Real Time for Automatic Accompaniment," ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 2025, pp. 1-5, doi: 10.1109/ICASSP49660.2025.10888267.
Source
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Conference
Keywords
Quantization (Signal), Computational Modeling, Speech Recognition, Streaming Media, Control Systems, Real-Time Systems, Human In The Loop, Acoustics, Speech Processing, Long Short Term Memory, Online Performance Tracking, Automatic Accompaniment, Human-Computer Ensemble
Subjects
Source
Publisher
IEEE