Medical Referring Image Segmentation: Addressing Multi-Lesion Reference and Annotation Uncertainty via Vision-Language Fusion
He, Yuanyang Feng, Chun-Mei Zhou, Yang Cheng, Lionel Tran, Anh Ooi, Gideon Thng, Choon Khan, Salman Zuo, Wangmeng Liu, Yong
He, Yuanyang
Feng, Chun-Mei
Zhou, Yang
Cheng, Lionel
Tran, Anh
Ooi, Gideon
Thng, Choon
Khan, Salman
Zuo, Wangmeng
Liu, Yong
Supervisor
Department
Computer Vision
Embargo End Date
Type
Conference proceeding
Date
2026
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
We introduce a novel method MedRIS for the challenging task of Medical Referring Image Segmentation (Medical RIS): segment the lesions described in a medical report. MedRIS addresses two major challenges: (I) a single text description may correspond to multiple lesions (one-to-many reference); (II) annotation may be uncertain due to complex appearance of lesions and subjectivity of annotators. To solve (I), mask self-augmentation separates the original mask into independent or combinations of lesions. To solve (II), ChatGPT and organ position matching are leveraged to reformulate textual descriptions into content-diverse sentences that are consistent with the augmented masks. Organ-related positional attention combined with real-world medical knowledge focuses the model on critical image areas. Exponential moving average (EMA) and local annotation correction are used to suppress the effect of noisy annotations during training. Extensive experiments on public and in house chest X-ray datasets show that MedRIS outperforms the state-of-the-art methods by a large margin.
Citation
Y. He et al., “Medical Referring Image Segmentation: Addressing Multi-Lesion Reference and Annotation Uncertainty via Vision-Language Fusion,” pp. 369–378, 2026, doi: 10.1007/978-3-032-09513-8_36
Source
Machine Learning in Medical Imaging
Conference
16th International Workshop, MLMI 2025, Held in Conjunction with MICCAI 2025
Keywords
Subjects
Source
16th International Workshop, MLMI 2025, Held in Conjunction with MICCAI 2025
Publisher
Springer Nature