Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
Giang Nguyen, Raghav Mehta, Emma A. M. Stanley, Tian Xia, Thi Hao Nguyen, Hieu Pham, Ben Glocker
Why It Matters
What makes this one worth your time
Understanding the robustness of foundation models in mammography under domain shifts is crucial for reliable clinical applications and improving diagnostic accuracy across diverse datasets.
The paper benchmarks foundation models for mammography under domain shifts, revealing the importance of dataset-level evaluations.
Summary
The paper benchmarks the robustness of 15 foundation models used for mammography under domain shifts by evaluating their performance across various datasets and conditions. It finds that mammography-specific vision-language models perform best on average, but robustness is not solely due to mammography exposure. The study highlights the importance of dataset-level out-of-distribution evaluation for assessing mammography representations.
Key contributions
- Benchmarking 15 foundation models for mammography under domain shifts.
- Highlighting the importance of dataset-level out-of-distribution evaluation.
- Providing a publicly available codebase for reproducibility.
Notable insights
- Mammography-specific vision-language models outperform others, but their robustness is not solely due to mammography exposure.
- Dataset-level out-of-distribution evaluation is crucial for assessing the effectiveness of mammography representations.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.10358v1 Announce Type: cross Abstract: Foundation models are increasingly used as image feature extractors for mammography, but their robustness under external domain shift remains unclear. We benchmark 15 foundation-model backbones across breast density, BI-RADS severity, and cancer status using a unified frozen-backbone linear-probe protocol, training on 3 source datasets and evaluating on 12 task-compatible out-of-distribution (OOD) datasets after label harmonization. Mammography-specific vision-language models (Mammo-FM and MaMA) provide the strongest mean OOD performance, but robustness is not explained by mammography exposure alone. DINOv3 remains a competitive vision-only baseline, and mammography-adapted pretraining does not consistently improve generalization. Dataset-level analysis further shows that even leading models show heterogeneous performance across datasets. Feature-space inspection reveals that useful representations can preserve clinical signal while retaining dataset and acquisition structure. These findings highlight dataset-level OOD evaluation as a central criterion for assessing mammography representations. Our code is publicly available: https://github.com/biomedia-mira/mammo-ood.