PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
Zibo Shao, Baochen Xiong, Xiaoshan Yang, Yaguang Song, Qimeng Zhang, Haifeng Chen, Changsheng Xu
Why It Matters
What makes this one worth your time
This work addresses a significant gap in multimodal pre-training, potentially improving the performance and efficiency of large language models in diverse applications.
PivotMerge enhances multimodal model integration through innovative post-alignment merging techniques.
Summary
The paper introduces PivotMerge, a framework for post-alignment merging of multimodal large language models, addressing challenges in cross-modal alignment during pre-training by integrating complementary strengths from heterogeneous models.
Key contributions
- Development of the PivotMerge framework for integrating multimodal models.
- Introduction of Shared-space Decomposition and Filtering to manage parameter conflicts.
- Implementation of Alignment-guided Layer-wise Merging to optimize model integration.
Notable insights
- The introduction of post-alignment merging as a distinct task highlights the importance of cross-modal alignment during pre-training.
- The framework's focus on layer-wise merging weights based on alignment contributions is a novel approach to mitigating parameter interference.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2604.22823v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities. Model merging provides a cost-effective mechanism for integrating multiple expert MLLMs with complementary strengths into a unified model. However, existing model merging research mainly focuses on post-finetuning scenarios, leaving the pre-training stage largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from heterogeneous multimodal pre-training. This setting introduces two key challenges: cross-domain parameter interference, where parameter updates learned from different data distributions conflict during merging, and layer-wise alignment contribution disparity, where different layers and projectors contribute unevenly to cross-modal alignment. To address them, we propose \textbf{PivotMerge}, a post-alignment merging framework for cross-modal projectors. PivotMerge incorporates two key components: Shared-space Decomposition and Filtering, which disentangles shared alignment patterns from domain-specific variations and suppresses conflicting directions, and Alignment-guided Layer-wise Merging, which assigns layer-specific merging weights based on differing alignment contributions. We construct systematic CC12M-based post-alignment merging scenarios for evaluation. Extensive experiments on multiple multimodal benchmarks show that PivotMerge consistently outperforms existing baselines, demonstrating its effectiveness and generalization ability.