Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis
Jingyu Hu, Giuseppe Tripodi, Reed Naidoo, Sarah F. McGough, Tapabrata Chakraborti
Why It Matters
What makes this one worth your time
Understanding the effectiveness of foundation models in handling multimodal medical data under distribution shifts is crucial for improving clinical decision support systems.
Systematic evaluation of foundation models for multimodal cancer analysis reveals benefits of fusion strategies.
Summary
The paper evaluates foundation model representations for multimodal cancer analysis, focusing on whole-slide images and transcriptomic profiles. It benchmarks unimodal and multimodal fusion strategies across computational pathology tasks, assessing performance on out-of-distribution data and using conformal prediction to evaluate trustworthiness.
Key contributions
- Systematic evaluation of foundation models on multimodal cancer analysis tasks.
- Comparison of unimodal and multimodal fusion strategies for predictive performance.
- Assessment of trustworthiness using conformal prediction.
Notable insights
- Multimodal fusion strategies provide additional gains primarily when no single modality dominates the predictive signal.
- Conformal prediction can recover true diagnoses within the prediction set even when point predictions fail, highlighting the importance of uncertainty-aware inference.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.17115v1 Announce Type: cross Abstract: Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored. This work systematically evaluates FM-based representations on a suite of computational pathology tasks across two real-world commercial cohorts, IH-BC and IH-NSCLC, drawn from the licensed in-house (IH) oncology dataset. The analysis focuses on two modalities, whole-slide images and transcriptomic profiles, drawn from the IH multimodal data. We first benchmark unimodal probing performance across five FMs on eight downstream classification tasks, and find that image and omics representations carry complementary predictive signals. Then we investigate whether multimodal fusion can yield additional gains over unimodal baselines by comparing three image-omics fusion strategies built on paired representations. The trustworthiness of selected unimodal and multimodal pipelines is further assessed through conformal prediction. Our results show that FM representations achieve competitive performance on out-of-distribution data and that multimodal fusion helps mainly when no single modality dominates the signal. Conformal prediction reveals that in the majority of cases where a point prediction fails, the true diagnosis remains recoverable within the prediction set, reinforcing the value of uncertainty-aware inference for clinical support.