Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
Wei-Yao Wang, Zhao Wang, Helen Suzuki, Yoshiyuki Kobayashi
Why It Matters
What makes this one worth your time
This work addresses a critical challenge in multimodal AI, potentially improving the accuracy and relevance of model outputs in applications that integrate text and images.
A new attention mechanism enhances multimodal understanding in LLMs.
Summary
The paper introduces a novel modality-mutual attention (MMA) mechanism for multimodal large language models (MLLMs) that allows image tokens to attend to text tokens, addressing vision-language misalignment and achieving state-of-the-art performance on multiple benchmarks.
Key contributions
- Introduction of modality-mutual attention (MMA) for MLLMs.
- Demonstration of state-of-the-art performance improvements across 12 multimodal benchmarks.
- A generic design applicable to various multimodal scenarios.
Notable insights
- The proposed MMA mechanism allows for a more integrated approach to processing multimodal inputs, which could lead to better contextual understanding.
- The architecture change does not require additional parameters, making it resource-efficient.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2503.02597v3 Announce Type: replace-cross Abstract: Recent Multimodal Large Language Models (MLLMs) have demonstrated significant progress in perceiving and reasoning over multimodal inquiries, ushering in a new research era for foundation models. However, vision-language misalignment in MLLMs has emerged as a critical challenge, where the textual responses generated by these models are not factually aligned with the given text-image inputs. Existing efforts to address vision-language misalignment have focused on developing specialized vision-language connectors or leveraging visual instruction tuning from diverse domains. In this paper, we tackle this issue from a fundamental yet unexplored perspective by revisiting the core architecture of MLLMs. Most MLLMs are typically built on decoder-only LLMs consisting of a causal attention mechanism, which limits the ability of the earlier modalities (e.g., images) to incorporate information from the latter modalities (e.g., text). To address this problem a MLLM that unlocks causal attention into our proposed modality-mutual attention (MMA) to enable image tokens to attend to text tokens. This simple yet effective design allows MMA to achieve state-of-the-art performance in 12 multimodal understanding benchmarks (+6.2% on average across 3 LLMs backbones) without introducing additional parameters. Our MMA design is intended to be generic, allowing for applications across various modalities, and scalable to accommodate diverse multimodal scenarios.