Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
Ryo Fukuda, Takatomo Kano, Siddhant Arora, Marc Delcroix, Naohiro Tawara, Atsunori Ogawa, Yuya Chiba, Atsushi Ando, William Chen, Shinji Watanabe
Why It Matters
What makes this one worth your time
Understanding how LLMs can effectively engage in turn-taking can enhance their application in real-time communication systems and collaborative tools.
LLMs show promise in predicting conversational dynamics despite limited training data.
Summary
The paper investigates turn-taking in multimodal multi-party conversations by evaluating large language models (LLMs) on tasks such as addressee detection, turn-change prediction, and next speaker prediction, demonstrating that LLMs can outperform supervised models and humans in specific tasks despite lacking training on the target domain and audio-visual data.
Key contributions
- Development of an evaluation framework for addressee detection, turn-change prediction, and next speaker prediction.
- Empirical comparison of LLMs with supervised models and human performance on the AMI corpus.
Notable insights
- Conversational context significantly influences next speaker prediction accuracy.
- The performance gap between MM-LLMs and human subjects highlights challenges in utilizing raw audio-visual signals.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.17542v1 Announce Type: new Abstract: We investigate turn-taking in multimodal multi-party conversations using large language models (LLMs). We construct an evaluation framework for three tasks: addressee detection, turn-change prediction, and next speaker prediction. We compare supervised models trained for these tasks, text-based LLMs, multimodal LLMs (MM-LLMs), and human subjects. Experiments on the AMI corpus showed that LLMs outperformed supervised models and humans in next speaker prediction, despite not being trained on the target domain and without access to audio or visual information. An MM-LLM performed better than text-based LLMs on addressee detection and turn-change prediction but remained below human performance, indicating difficulty leveraging raw audio-visual signals. Ablation analyses revealed that conversational context was critical, particularly for next speaker prediction. We observed that human and LLM prediction patterns were similar, and intervals with frequent turn changes were difficult for both.