In search of truth: evaluating concordance of AI-based anatomy segmentation models.
Giebeler, Lena Krishnaswamy, Deepa Clunie, David Wasserthal, Jakob Sundar, Lalith Kumar Shiyam Diaz-Pinto, Andres Maier-Hein, Klaus H Xu, Murong Menze, Bjoern Pieper, Steve
Giebeler, Lena
Krishnaswamy, Deepa
Clunie, David
Wasserthal, Jakob
Sundar, Lalith Kumar Shiyam
Diaz-Pinto, Andres
Maier-Hein, Klaus H
Xu, Murong
Menze, Bjoern
Pieper, Steve
Supervisor
Department
Computer Vision
Embargo End Date
Type
Journal article
Date
License
http://creativecommons.org/licenses/by/4.0/
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Purpose: Artificial intelligence based methods for anatomy segmentation can help automate characterization of large imaging datasets. The growing number of similar functionality models raises the challenge of evaluating them on datasets that do not contain ground truth annotations. We introduce a practical framework to assist in this task.
Approach: We harmonize the segmentation results into a standard, interoperable representation, which enables consistent, terminology-based labeling of the structures. We extend 3D Slicer to streamline loading and comparison of these harmonized segmentations and demonstrate how standard representation simplifies review of the results using interactive summary plots and browser-based visualization using the OHIF Viewer. To demonstrate the utility of the approach, we apply it to evaluating segmentation of 31 anatomical structures (lungs, vertebrae, ribs, and heart) by 6 open-source models-TotalSegmentator 1.5 and 2.6, Auto3DSeg, MOOSE, MultiTalent, and CADS-for a sample of computed tomography scans from the publicly available National Lung Screening Trial dataset.
Results: We demonstrate the utility of the framework in enabling automating loading, structure-wise inspection, and comparison across models. Preliminary results ascertain the practical utility of the approach in allowing quick detection and review of problematic results. The comparison shows excellent agreement segmenting some (e.g., lung) but not all structures (e.g., some models produce invalid vertebrae or rib segmentations).
Conclusions: The open-source resources developed include segmentation harmonization scripts, interactive summary plots, and visualization tools. These resources assist in segmentation model evaluation in the absence of ground truth, ultimately enabling informed model selection.
Citation
L. Giebeler, D. Krishnaswamy, D. Clunie, J. Wasserthal, L.K.S. Sundar, A. Diaz-Pinto , et al., "In search of truth: evaluating concordance of AI-based anatomy segmentation models.," Journal of Medical Imaging, vol. 13, no. 6, pp. 062204-062204, 2026, https://doi.org/10.1117/1.jmi.13.6.062204.
Source
Journal of Medical Imaging
Conference
Keywords
32 Biomedical and Clinical Sciences, 3202 Clinical Sciences, 40 Engineering, 4003 Biomedical Engineering
Subjects
Source
Publisher
SPIE, the international society for optics and photonics