Autoregressive EHR Foundation Models with Multimodal Inputs
Yuxuan Liu, Joshua Placidi, Jinpei Han, Alfred John Balston, Marek Rei, A. Aldo Faisal
Why It Matters
What makes this one worth your time
This research is relevant for AI engineers and researchers focused on improving clinical prediction models by effectively utilizing diverse data types from electronic health records.
A framework for integrating multimodal inputs into autoregressive EHR models to enhance clinical predictions.
Summary
The paper presents a framework for autoregressive foundation models that incorporate multimodal inputs from electronic health records, such as ECG waveforms, chest X-ray images, and clinical notes. It explores the impact of latent compression and pretrained encoder choice on model performance, demonstrating that careful design of fusion architecture is necessary for improved clinical prediction.
Key contributions
- A framework for conditioning autoregressive EHR models on multimodal inputs.
- Investigation of latent compression techniques for multimodal data.
- Analysis of the impact of pretrained encoder choice on model performance.
Notable insights
- Latent compression of long sequences before cross-attention can reduce computational overhead and potentially improve generalization.
- The choice of pretrained encoder significantly affects performance within each modality.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.22264v1 Announce Type: new Abstract: Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.