Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling
Zhemin Zhang, Weijie Chen, David Le, Amara Tariq, Alex Wallace, Matthew Stib, Juan Maria Farina, Chadi Ayoub, Reza Arsanjani, Imon Banerjee
Why It Matters
What makes this one worth your time
Improving time-to-event predictions in clinical settings can lead to better patient outcomes and more efficient healthcare delivery.
A framework for aligning multimodal clinical data to enhance time-to-event predictions.
Summary
The paper introduces a framework for cross-modal representation alignment between CT imaging and EHR data to improve time-to-event prediction in clinical settings, using foundation models and four fusion strategies to handle modality imbalance and distribution shift.
Key contributions
- Introduction of a foundation model-driven framework for cross-modal representation alignment.
- Systematic analysis of fusion strategies under modality imbalance in time-to-event prediction.
- Demonstration of improved concordance index using multimodal fusion over unimodal baselines.
Notable insights
- Contrastive multimodal fusion with CLMBR representations shows consistent improvements, especially for pulmonary embolism mortality prediction.
- Different fusion strategies excel in different scenarios, indicating the importance of task-specific alignment.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.15038v1 Announce Type: new Abstract: Accurate time-to-event (TTE) prediction from multimodal clinical data remains challenging due to modality imbalance and distribution shift. We introduce a foundation model-driven framework for cross-modal representation alignment between CT imaging and longitudinal EHR data, designed to generalize across tasks and institutions. CT and EHR modalities are encoded independently using domain-specific foundation models and aligned in a shared latent space through four principled fusion strategies: late fusion, contrastive alignment, cross-attention, and co-attention. We evaluate two clinically distinct TTE tasks: pulmonary embolism (PE) mortality and cardiovascular disease (CVD) outcomes, on large-scale multi-institutional cohorts (PE: N=3,099 train; 1,098 internal; 435 external; CVD: N=2,951 train; 837 internal; 682 external). Fusion consistently improves concordance index by 1.5-5.4% over unimodal baselines when modalities contribute comparably. Overall, contrastive multimodal fusion, particularly with CLMBR representations, provided the most consistent and statistically robust improvements, especially for PE mortality prediction. For MACE, cross-attention (one-hot) achieved the highest internal performance and image-guided co-attention achieved the best external performance. We therefore introduce a generalizable foundation model-based cross-modal alignment framework and provide the first systematic analysis of fusion behavior under modality imbalance in TTE prediction. Our results establish task-aware multimodal alignment as a necessary design principle for robust generalization and scalable clinical deployment.