Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
Hanning Guo, Hanwen Bi, Farah Abdellatif, Andrei Galbenus, Jon. N. Shah, Abigail Morrison, J\"urgen Dammers
Why It Matters
What makes this one worth your time
This work is significant for AI researchers and engineers interested in neuroscience as it offers a unified model for integrating multiple neuroimaging modalities, potentially leading to more comprehensive brain activity analysis.
Brain-OF unifies fMRI, EEG, and MEG data into a single foundation model for enhanced neuroimaging analysis.
Summary
The paper introduces Brain-OF, a foundation model for neuroimaging that integrates fMRI, EEG, and MEG data to handle both unimodal and multimodal inputs. It uses an Any-Resolution Neural Signal Sampler to unify diverse brain signals and employs DINT attention with a Sparse Mixture of Experts to manage semantic shifts. The model is pretrained on a large corpus and uses Masked Temporal-Frequency Modeling for dual-domain pretraining.
Key contributions
- Development of Brain-OF, a model capable of handling multimodal neuroimaging data.
- Introduction of the Any-Resolution Neural Signal Sampler for unifying diverse brain signals.
- Proposal of Masked Temporal-Frequency Modeling for dual-domain pretraining.
Notable insights
- The use of an Any-Resolution Neural Signal Sampler to project diverse brain signals into a shared semantic space.
- The integration of DINT attention with a Sparse Mixture of Experts to handle modality-specific and invariant representations.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2602.23410v3 Announce Type: replace-cross Abstract: Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited to a single functional modality, restricting their ability to exploit complementary spatiotemporal dynamics and the collective data scale across different neuroimaging techniques. This limitation largely arises from severe semantic heterogeneity and resolution discrepancies among modalities. To address these challenges, we propose Brain-OF, an omnifunctional brain foundation model jointly pretrained on fMRI, EEG and MEG, capable of handling both unimodal and multimodal inputs within a unified framework. To reconcile heterogeneous spatiotemporal resolutions, we introduce the Any-Resolution Neural Signal Sampler, which projects diverse brain signals into a shared semantic space. To further manage semantic shifts, the Brain-OF backbone integrates DINT attention with a Sparse Mixture of Experts, where shared experts capture modality-invariant representations and routed experts specialize in modality-specific semantics. Furthermore, to explicitly internalize the characteristics of neural activity through self-supervised learning, we propose Masked Temporal-Frequency Modeling, a dual-domain pretraining objective that jointly reconstructs brain signals in both the time and frequency domains. Brain-OF is pretrained on a large-scale corpus comprising around 40 datasets and demonstrates superior performance across diverse downstream tasks, highlighting the benefits of joint multimodal integration and dual-domain pretraining.