Scaling Native Multimodal Pre-Training From Scratch
Haoyuan Wu, Aoqi Wu, Hai Wang, Jiajia Wu, Jinxiang Ou, Bei Yu
Why It Matters
What makes this one worth your time
Understanding the scaling properties of multimodal models can lead to more efficient resource allocation and improved model performance, which is crucial for advancing AI capabilities in processing diverse data types.
The paper explores optimal scaling strategies for native multimodal pre-training to enhance cross-modal integration.
Summary
The paper investigates the scaling properties of native multimodal pre-training by examining optimal model size and token count for a transformer-based vision-language model under a fixed computational budget. It identifies distinct scaling behaviors for language and multimodal objectives and derives an efficiency frontier for model configurations.
Key contributions
- Characterization of scaling properties for native multimodal pre-training.
- Derivation of an efficiency frontier for model size, token count, and data mixture.
- Demonstration of positive cross-modal transfer enhancing spatial reasoning and multimodal learning.
Notable insights
- Language and multimodal objectives exhibit distinct scaling behaviors, with language learning being stable across data compositions.
- Text-heavy data mixtures require larger model scales for compute efficiency, influencing resource allocation strategies.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.22043v1 Announce Type: new Abstract: Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.