Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan
Why It Matters
What makes this one worth your time
This research could reduce the dependency on expensive image-text datasets, making it easier and more cost-effective to train multimodal models at scale.
A novel framework for aligning multimodal embeddings without large-scale paired data.
Summary
The paper addresses the modality gap in multimodal large language models by introducing a new framework that characterizes and aligns the geometric differences between visual and linguistic embeddings. It proposes the Fixed-frame Modality Gap Theory and a training-free alignment strategy called ReAlign, which uses unpaired data to align text and image representations. This approach is integrated into a scalable training paradigm named ReVision, which allows for efficient model scaling without relying on large-scale image-text pairs.
Key contributions
- Introduction of the Fixed-frame Modality Gap Theory.
- Development of ReAlign, a training-free modality alignment strategy.
- Proposal of ReVision, a scalable training paradigm for MLLMs using unpaired data.
Notable insights
- The use of unpaired data statistics to align modalities is a clever approach to reduce reliance on paired datasets.
- Decomposing the modality gap into stable biases and anisotropic residuals offers a nuanced understanding of the alignment problem.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2602.07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models~(MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.