FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
Soham Dan
Why It Matters
What makes this one worth your time
This benchmark provides a structured way to assess the capabilities of language models in generating code for complex combinatorial tasks, which is crucial for advancing AI's practical applications in programming.
FindStatBench sets a new standard for evaluating language models in combinatorial code synthesis.
Summary
The paper introduces FindStatBench, a benchmark for evaluating large language models on combinatorial code synthesis, consisting of 2,329 tasks and scoring based on exact execution of Python solve functions.
Key contributions
- Development of a comprehensive benchmark with 2,329 tasks for evaluating combinatorial code synthesis.
- Insights into the performance of both open-source and closed-source models in this domain.
- Identification of specific failure modes related to output-budget mechanics in code generation.
Notable insights
- The performance of models can degrade with increased examples, suggesting that more data does not always lead to better outcomes in this context.
- The distinction in difficulty between statistic synthesis and map synthesis indicates varying levels of complexity in code generation tasks.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 collections and 5.52M hidden instances, covering statistic synthesis, which maps objects to integers, and map synthesis, which maps objects to objects. Each task gives a mathematical description and at most five public input-output examples; a model must emit one Python solve function with no retrieval, tool use, execution feedback, voting, or reranking. Submissions are scored by exact sandboxed execution on held-out combinatorial objects. We evaluate eleven systems: four closed-source production models and seven open-weight models served through one inference provider. FindStatBench reveals three main patterns. First, the strongest open- and closed-source systems converge within 1 pp instance accuracy, and both an oracle over all systems and five-way sampling from one mid-tier model yield only limited task-accuracy gains. Second, examples can hurt: several classical bijections are solved perfectly with zero examples but fail under five-example prompts. Third, some failures reflect output-budget mechanics, as reasoning can exhaust the visible response before code is emitted. Overall, statistic synthesis is much easier than map synthesis, some collections remain near-zero, long prompts cause a sharp accuracy cliff, and exact symbolic rule induction remains brittle.