Multimodal Pretraining for Generalizable EEG Representation Learning
Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi
Why It Matters
What makes this one worth your time
This work is significant for AI researchers and engineers interested in EEG analysis, as it offers a generalizable model that can adapt to various seizure detection scenarios without requiring extensive labeled data.
A multimodal EEG foundation model achieves state-of-the-art seizure detection performance and generalizes well across datasets.
Summary
The paper presents a multimodal EEG foundation model that integrates a raw signal encoder, a Vision Transformer-style encoder for time-frequency data, and a lightweight text encoder within a shared embedding space. The model is pretrained using techniques like masked modeling and cross-view contrastive alignment to learn seizure-relevant representations without labeled data. It achieves state-of-the-art performance on the CHB-MIT seizure detection benchmark and demonstrates robust generalization across different datasets and evaluation protocols.
Key contributions
- Development of a multimodal EEG foundation model combining different encoders in a shared embedding space.
- Introduction of innovative pretraining techniques like masked modeling and cross-view contrastive alignment.
- Achievement of state-of-the-art performance on the CHB-MIT seizure detection benchmark.
Notable insights
- The integration of a Vision Transformer-style encoder for time-frequency data with a shared embedding space is a novel approach in EEG representation learning.
- The use of cross-view contrastive alignment and temporal consistency losses enhances the model's ability to learn seizure-relevant features without labeled data.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks. This limited approach can make it challenging to apply these models across different datasets or in various situations. However, recent studies in foundation models and self-supervised learning suggest that an adaptable EEG backbone could support a range of EEG related tasks. In this study, we have developed a multimodal EEG foundation model that combines a raw signal encoder based on the Mamba architecture, a Vision Transformer (ViT)-style encoder for time-frequency data, and a lightweight encoder for text, all within a shared embedding space. The pretraining process relies on several innovative techniques, such as masked modeling, cross-view contrastive alignment, and temporal consistency losses. These methods are designed to create rich, seizure-relevant representations without requiring labeled data. To assess the efficacy and generalization of our pretrained model, we fine-tuned it on the canonical CHB-MIT seizure detection benchmark and additional seizure detection datasets, and conducted extensive experiments comparing different model variants. On the standard CHB-MIT split, our best single model achieved an AUROC of 0.874, and an ensemble variant reached 0.878 AUROC, representing state-of-the-art performance on this benchmark. In addition to standard train-test splits, we evaluated performance under a leave-one-subject-out (LOSO) protocol, which is rarely reported in prior EEG seizure modeling work and highlights the difficulty of patient-independent seizure detection, with a mean LOSO balanced accuracy of 0.558 across 19 subjects. Across datasets and evaluation settings, our multimodal foundation model enabled robust seizure detection and straightforward adaptation to new seizure detection scenarios, while also supporting interpretable seizure localization.