Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
Parishruthi Ganesh, Gerry Dozier, Cheryl Seals
Why It Matters
What makes this one worth your time
This research provides valuable insights for practitioners in selecting appropriate language models for real-world applications, addressing common challenges in compute and latency.
A comprehensive evaluation of 41 language models for intent classification under various constraints.
Summary
The paper conducts a systematic evaluation of 41 open-weight language models for zero-shot intent classification, analyzing their performance across various datasets and metrics, including accuracy and robustness.
Key contributions
- A systematic evaluation framework for comparing open-weight language models in intent classification.
- Insights into the effects of instruction tuning on model performance and confidence calibration.
- Identification of benchmark saturation in current intent classification datasets.
Notable insights
- Instruction-tuned 3B models can outperform larger 7B base models, suggesting efficiency in model selection.
- The saturation of widely used benchmarks like SNIPS indicates a need for new evaluation metrics.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.27421v1 Announce Type: new Abstract: Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight language models under compute, latency, and robustness constraints. We present a systematic zero-shot evaluation of 41 open-weight language models spanning 15 families and the 135M--9B parameter range across eight English single-label intent-classification datasets. A ninth dataset, ATIS, uses five labeled demonstrations and is reported as an auxiliary five-shot result. The evaluation includes standard benchmarks, a large-scale voice-assistant corpus, and production-derived e-commerce datasets. Beyond exact-match accuracy, we analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and benchmark saturation. Our results show that instruction-tuned 3B models can outperform several evaluated 7B base models, that differences among leading models on MASSIVE are statistically indistinguishable under pairwise McNemar tests, and that widely used benchmarks such as SNIPS have become saturated and no longer meaningfully discriminate among current open-weight models. Instruction tuning's effect on confidence calibration is inconsistent rather than uniformly harmful. These findings provide practical guidance for selecting and evaluating open-weight language models for intent classification.