SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation
Ruohan Liu, Shukang Yin, Tao Wang, Dong Zhang, Weiji Zhuang, Shuhuai Ren, Ran He, Caifeng Shan, Chaoyou Fu
Why It Matters
What makes this one worth your time
Understanding and improving paralinguistic cues in speech generation is crucial for developing more natural and effective human-computer interactions.
SpeechParaling-Bench offers a new benchmark for evaluating paralinguistic features in speech generation.
Summary
The paper introduces SpeechParaling-Bench, a benchmark designed to evaluate paralinguistic-aware speech generation in large audio-language models by expanding feature coverage and using a pairwise comparison pipeline to reduce subjectivity in assessments.
Key contributions
- Introduction of a benchmark with over 100 fine-grained paralinguistic features.
- Development of a pairwise comparison pipeline for evaluation.
- Identification of limitations in current LALMs regarding paralinguistic feature control.
Notable insights
- The use of a pairwise comparison pipeline to mitigate subjectivity in evaluating paralinguistic features.
- Highlighting the significant error rate in current models due to misinterpretation of paralinguistic cues.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2604.20842v1 Announce Type: cross Abstract: Paralinguistic cues are essential for natural human-computer interaction, yet their evaluation in Large Audio-Language Models (LALMs) remains limited by coarse feature coverage and the inherent subjectivity of assessment. To address these challenges, we introduce SpeechParaling-Bench, a comprehensive benchmark for paralinguistic-aware speech generation. It expands existing coverage from fewer than 50 to over 100 fine-grained features, supported by more than 1,000 English-Chinese parallel speech queries, and is organized into three progressively challenging tasks: fine-grained control, intra-utterance variation, and context-aware adaptation. To enable reliable evaluation, we further develop a pairwise comparison pipeline, in which candidate responses are evaluated against a fixed baseline by an LALM-based judge. By framing evaluation as relative preference rather than absolute scoring, this approach mitigates subjectivity and yields more stable and scalable assessments without costly human annotation. Extensive experiments reveal substantial limitations in current LALMs. Even leading proprietary models struggle with comprehensive static control and dynamic modulation of paralinguistic features, while failure to correctly interpret paralinguistic cues accounts for 43.3% of errors in situational dialogue. These findings underscore the need for more robust paralinguistic modeling toward human-aligned voice assistants.