A Multi-Agent Diffusion Approach for MRI Anomaly Segmentation via Modality-Specific LoRA Specialization
Al Ghallabi, Wafa Zaheer, Muhammad Zaigham Thawkar, Ritesh Thawakar, Omkar Khan, Salman Khan, Fahad Shahbaz
Al Ghallabi, Wafa
Zaheer, Muhammad Zaigham
Thawkar, Ritesh
Thawakar, Omkar
Khan, Salman
Khan, Fahad Shahbaz
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Unsupervised anomaly segmentation in multi-sequence MRI is a promising way to scale lesion screening, but existing reconstruction-based methods face three persistent issues: they fail to generalize across modalities, they depend on hand-crafted masking or paired translations, and they often require separate models with high inference cost. In this work, we take a stepwise approach to address these limitations. In the first stage, we fully finetune a diffusion model on healthy brain MRI slices pooled across T1, T2, and FLAIR, which produces anatomically consistent reconstructions. To further improve, we introduce a lightweight second stage where modality-specific LoRA adapters are trained on top of the pretrained diffusion backbone. A simple router automatically selects the right adapter for each input, effectively turning the system into a modality-aware, agentic-like framework. To further stabilize reconstructions, we incorporate a learnable latent-frequency mask that suppresses non-informative spectral components and preserves structural detail. This design allows the model to emphasize healthy anatomy while efficiently capturing modality-dependent contrasts. This two-stage strategy boosts Dice to 88% on BraTS2021 (FLAIR), achieving state-of-the-art performance. Experiments on BraTS2021, ISLES, and ATLAS datasets confirm that the approach consistently improves Dice and SSIM across all modalities, outperforming diffusion, masking, and cycle-based baselines, and offering a practical balance between accuracy and efficiency for clinical MRI anomaly detection. Source code is available at https://github.com/wafaalghallabi/mri-router.
Citation
W. Al Ghallabi, M.Z. Zaheer, R. Thawkar, O. Thawakar, S. Khan, F.S. Khan, "A Multi-Agent Diffusion Approach for MRI Anomaly Segmentation via Modality-Specific LoRA Specialization," 2026, pp. 128-137.
Source
2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Conference
2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Keywords
46 Information and Computing Sciences, 4605 Data Management and Data Science
Subjects
Source
2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Publisher
IEEE