Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
Dominick Reilly, Qiyu Wu, Hiromi Wakaki, Srijan Das, Yuki Mistufuji
Why It Matters
What makes this one worth your time
This research is relevant for AI engineers and researchers working with multimodal models, as it addresses the challenge of leveraging training-only modalities to improve model performance in real-world applications where all modalities may not be available at inference.
MoP framework enhances multimodal LLMs by effectively utilizing training-only modalities for improved inference.
Summary
The paper introduces Mixture of Probes (MoP), a framework for multimodal large language models that leverages privileged modalities available only during training. MoP uses a structured probing mechanism to disentangle modality-specific and modality-general signals, enhancing cross-modal learning. The proposed MoP Cross-modal Training strategy includes a probe disentanglement loss to prevent probe collapse. The framework is evaluated across multiple tasks and modalities, showing significant improvements over existing baselines.
Key contributions
- Development of the Mixture of Probes (MoP) framework for disentangling modality-specific and modality-general signals.
- Introduction of MoP Cross-modal Training strategy with a probe disentanglement loss.
- Comprehensive evaluation protocol tailored to the privileged modality setting.
Notable insights
- The use of a structured probing mechanism to extract and organize information from intermediate representations rather than relying solely on final-layer alignment.
- Introduction of a probe disentanglement loss to encourage cross-modal learning and prevent probe collapse.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference. However, many real-world settings violate this assumption, requiring models to operate under a privileged modality setting, where auxiliary modalities are available only during training. While these modalities contain valuable information, existing MLLMs largely fail to leverage them effectively, as they treat modalities as interchangeable inputs rather than sources of complementary supervision. We propose Mixture of Probes (MoP), a novel framework that disentangles modality-specific and modality-general signals within the MLLM, allowing the model to preserve modality-dependent structure while learning transferable representations across modalities. At its core, MoP achieves this through a structured probing mechanism that extracts and organizes information from intermediate representations of a shared modality encoder, rather than relying only on final-layer alignment as done in existing MLLMs. To support this disentanglement, we further introduce MoP Cross-modal Training (MoP-X), a training strategy for MoP centered around a probe disentanglement loss that prevents probe collapse and encourages cross-modal learning. We evaluate MoP across two domains spanning eight tasks and four modalities under a comprehensive evaluation protocol tailored to the privileged modality setting, where each modality is independently treated as the sole input at inference time. MoP consistently outperforms strong MLLM baselines, achieving up to 65% relative improvement, demonstrating that auxiliary modalities, even when unavailable at inference, can provide substantial gains when effectively leveraged during training. Code, model checkpoints, and evaluation protocols will be made available at https://github.com/Sony/MoP.