Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
Yongxi Zhou, Lai Yun Choi, Jiaxi Wen, Wenbo Ye
Why It Matters
What makes this one worth your time
Understanding the stability of LLMs in deterministic tasks is crucial for reliable deployment in real-world applications where consistent performance is required.
The paper highlights the importance of stability analysis in evaluating LLMs for deterministic tasks.
Summary
The paper investigates the relationship between accuracy and stability in large language models (LLMs) for deterministic programming tasks, introducing a repeated-run evaluation protocol. It evaluates 16 models on 100 programming problems, highlighting the discrepancy between run-level pass rate and retry-free coverage, which can affect model rankings.
Key contributions
- Introduced a repeated-run evaluation protocol for deterministic programming tasks.
- Evaluated 16 LLMs on a benchmark of 100 programming problems with repeated runs.
Notable insights
- Run-level pass rate can overstate retry-free coverage by up to 17.8 percentage points.
- Prompt effects on model performance are model-dependent, not uniformly beneficial.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.00920v1 Announce Type: cross Abstract: Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points -- and the gap is largest precisely for mid-performing systems. We investigate this accuracy--stability relationship in large language model (LLM) evaluation for deterministic text-conditioned generation, using programming tasks as a concrete testbed. Standard code-generation benchmarks emphasize single-run accuracy or eventual success under repeated sampling, but many deployment settings also require stability: consistent outcomes across repeated invocations under the same task description. We present a repeated-run evaluation protocol with metrics for run-level accuracy, retry-free coverage, and per-problem variability. On a recency-based benchmark of 100 LeetCode-style problems, we evaluate 16 models from five provider families under two prompt templates with five repeated runs per problem, yielding 16,000 evaluation instances. Although run-level pass rate and perfect stability rate are strongly correlated (r=0.985), pass rate consistently exceeds retry-free coverage -- a gap that reaches 17.8 percentage points and reverses model rankings even among closely matched systems. Prompt effects are model-dependent rather than uniformly beneficial. These results suggest that repeated-run stability analysis is a necessary complement to conventional accuracy reporting for deterministic text-conditioned generation tasks.