MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
Haoran Ma, Yinfeng Yu, Liejun Wang
Why It Matters
What makes this one worth your time
Understanding and improving the reliability of multimodal sentiment analysis is crucial for developing robust AI systems that can interpret human emotions accurately across diverse conditions.
MRUF enhances multimodal sentiment analysis by dynamically adjusting fusion weights based on modality uncertainty.
Summary
The paper introduces MRUF, a method for multimodal sentiment analysis that addresses the variability in modality quality by using a reliability-aware fusion approach. It incorporates multi-granularity routing and uncertainty-aware calibration to improve sentiment analysis performance by adjusting the fusion weights based on modality uncertainty.
Key contributions
- Introduction of a reliability-aware fusion method for multimodal sentiment analysis.
- Development of a multi-granularity routing mechanism that includes subspace- and modality-level routing.
- Implementation of uncertainty-aware calibration to adjust fusion weights dynamically.
Notable insights
- The use of leave-one-out error increases to estimate modality importance is a clever approach to dynamically assess the reliability of each modality.
- Inverse-variance reweighting for modality gates is an innovative method to refine fusion based on predicted uncertainty.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.10599v1 Announce Type: new Abstract: Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.