SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits
Adis Alihodzic, Selma Skopljakovic Hubljar
Why It Matters
What makes this one worth your time
This research provides insights into improving multimodal emotion recognition systems, which are crucial for applications in human-computer interaction and affective computing.
A new method for multimodal emotion recognition that leverages SHAP for expert fusion shows competitive performance.
Summary
The paper presents a novel approach to multimodal emotion and sentiment recognition using SHAP-weighted cross-modal expert fusion, demonstrating its effectiveness against traditional fusion methods on benchmark datasets.
Key contributions
- Introduction of SHAP-weighted adaptive fusion for multimodal emotion and sentiment recognition.
- Empirical analysis of how different SHAP reduction methods affect expert performance.
- Demonstration of the effectiveness of trimodal experts in enhancing recognition accuracy.
Notable insights
- The study highlights the impact of SHAP attribution reduction on the performance of cross-modal experts with unequal dimensionalities.
- The findings suggest that the choice of weight reduction method significantly influences the effectiveness of multimodal fusion.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.08573v1 Announce Type: new Abstract: Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors. Early fusion can be accurate but monolithic, while late fusion is modular but may lose cross-modal interactions. This paper revisits XAI-guided adaptive fusion (\xgaf), a tree-based mixture of unimodal and cross-modal experts whose sample-level weights are derived from TreeSHAP attribution magnitudes. We focus on the effect of SHAP attribution reduction when experts have unequal feature dimensionalities. In this setting, mean-abs and median-abs reductions can suppress high-dimensional cross-modal experts, whereas sum-abs reduction preserves total attribution mass. On MELD 7-class emotion recognition, sum-abs \xgaf{} nearly matches early fusion across three face-sequence aggregators; the Transformer variant reaches 0.5983 \wf{}, compared with 0.6018 for early fusion and 0.4598 for probability-average late fusion. McNemar testing shows no significant difference between sum-abs \xgaf{} and early fusion on MELD ($p=1.000$), while \xgaf{} remains significantly better than late fusion ($p<0.0001$). On CMU-MOSEI 3-class sentiment recognition, sum-abs \xgaf{} reaches 0.6519 \wf{}, slightly exceeding early fusion (0.6485) and late fusion (0.5696). Ablation studies show that the main gain comes from adding cross-modal experts, especially the trimodal expert, rather than from complex per-sample routing. Diagnostics further show that mean-abs and median-abs weights are nearly uniform, while sum-abs weights concentrate on the trimodal expert. Thus, the main contribution is a transparent empirical analysis of how SHAP reduction, expert dimensionality, and cross-modal expert design affect modular multimodal fusion.