AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification
Chuxiao Zuo, Yao Zhu, Minqiang Xu, Manhong Wang, Yunke Zhang, Fei Huang
Why It Matters
What makes this one worth your time
This research is significant for real-world applications in speaker identification, particularly in noisy environments and diverse language contexts, potentially improving systems used in security, customer service, and accessibility.
AMR enhances multimodal speaker identification by dynamically routing modality information for improved accuracy.
Summary
The paper presents an Adaptive Modality Routing (AMR) system for multimodal polyglot speaker identification, addressing challenges such as missing modalities and language mismatches by dynamically assessing input quality and integrating modality information to improve identification accuracy.
Key contributions
- Introduction of the Adaptive Modality Routing (AMR) framework for multimodal speaker identification.
- Development of a modality-aware training strategy that simulates diverse input conditions.
- Demonstration of significant performance improvements over existing baselines in a competitive evaluation setting.
Notable insights
- The use of modality adapters to process embeddings from both audio and visual sources allows for a more nuanced understanding of speaker identity under varying conditions.
- Employing KL divergence for explicit supervision in dynamic weight assignment is a clever approach to optimize the routing mechanism.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.29335v1 Announce Type: cross Abstract: Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions. In practical scenarios, background multi-speaker conversations, ambient noise, and overlapping speech further degrade identification accuracy. To address these challenges, we propose a multimodal polyglot speaker identification system for the POLY-SIM 2026 Grand Challenge. The system is fundamentally built upon Adaptive Modality Routing(AMR), a modality fusion module that dynamically assesses per-sample input quality and integrates modality information. Specifically, AMR employs two modality adapters to process the embeddings extracted from a linguistically robust audio encoder(W2V-BERT 2.0) and a large-scale pretrained face encoder(IResNet-18), producing modality-adapted embeddings. Based on these adapted embeddings, a trainable router estimates dynamic modality weights, which are subsequently applied to aggregate the modality-specific logits for the final prediction. To optimize this routing mechanism, we adopt a modality-aware training strategy that constructs four types of sample pairs to simulate diverse input conditions, with KL divergence serving as explicit supervision for weight assignment. Experimental results on the POLY-SIM 2026 evaluation set show that the proposed system achieves identification accuracy of 99.93%(English multimodal, P3), 100.00%(Urdu multimodal, P5), 97.50%(English audio-only, P4), and 98.83%(Urdu audio-only, P6). The average accuracy across all four protocols is 99.07%, surpassing the Fusion and Orthogonal Projection(FOP) baseline by 32.73%.