MMCORE: MultiModal COnnection with Representation Aligned Latent Embeddings
Zijie Li, Yichun Shi, Jingxiang Sun, Ye Wang, Yixuan Huang, Zhiyao Guo, Xiaochen Lian, Peihao Zhu, Yu Tian, Zhonghua Zhai, Peng Wang
Why It Matters
What makes this one worth your time
This work is relevant for AI researchers and engineers interested in efficient multimodal image generation and editing, as it promises reduced computational costs and improved synthesis quality.
MMCORE enhances multimodal image generation by integrating Vision-Language Models with diffusion models for efficient and high-fidelity synthesis.
Summary
The paper introduces MMCORE, a framework for multimodal image generation and editing that uses a pre-trained Vision-Language Model to generate semantic visual embeddings as conditioning signals for a diffusion model, reducing computational overhead while maintaining high-quality synthesis.
Key contributions
- Proposes a unified framework for multimodal image generation and editing.
- Integrates Vision-Language Models with diffusion models for efficient synthesis.
- Demonstrates superior performance over state-of-the-art baselines in text-to-image and image editing tasks.
Notable insights
- Utilizes learnable query tokens to predict semantic visual embeddings from a Vision-Language Model.
- Avoids deep fusion between autoregressive and diffusion models, reducing computational demands.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2604.19902v1 Announce Type: cross Abstract: We present MMCORE, a unified framework designed for multimodal image generation and editing. MMCORE leverages a pre-trained Vision-Language Model (VLM) to predict semantic visual embeddings via learnable query tokens, which subsequently serve as conditioning signals for a diffusion model. This streamlined design effectively transfers the rich understanding and reasoning capabilities of VLMs into the visual generation process. By obviating the need for deep fusion between autoregressive and diffusion models or training from scratch, MMCORE significantly reduces computational overhead while maintaining high-fidelity synthesis. MMCORE seamlessly integrates text-to-image synthesis with interleaved image generation, demonstrating robust multimodal comprehension in complex scenarios such as spatial reasoning and visual grounding. Comprehensive evaluations indicate that MMCORE consistently outperforms state-of-the-art baselines across a broad spectrum of text-to-image and single/multi-image editing benchmarks.