Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, David Lo
Why It Matters
What makes this one worth your time
Understanding the limitations and capabilities of LLMs in generating complete software systems can guide future improvements in AI-driven software development.
CLI-Tool-Bench evaluates LLMs' ability to autonomously generate CLI tools from scratch.
Summary
The paper introduces CLI-Tool-Bench, a new benchmark for evaluating the ability of large language models to autonomously generate command-line interface tools from scratch, without relying on predefined structures or rigid testing frameworks. It uses a black-box differential testing approach to assess the generated software against human-written oracles, revealing that current models achieve a maximum success rate of 43.8%.
Key contributions
- Introduction of CLI-Tool-Bench, a structure-agnostic benchmark for 0-to-1 software generation.
- Use of a black-box differential testing framework to evaluate generated CLI tools.
- Extensive evaluation of seven state-of-the-art LLMs on the benchmark.
Notable insights
- Higher token consumption does not correlate with better performance in 0-to-1 software generation.
- LLMs tend to produce monolithic code structures when generating software from scratch.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2604.06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch. However, existing benchmarks fail to adequately assess this 0-to-1 generation capability due to two fundamental limitations. First, they rely on predefined structural scaffolds, which reduces the task to mere file-filling. Second, they depend on rigid white-box unit testing, which forces generated code to conform to specific internal implementations rather than validating end-to-end user-centric behavior. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure-agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by an automated black-box differential testing framework, the benchmark comprises 94 high-quality, real-world repositories spanning diverse programming languages and complexity levels. For each task, agents are provided with an empty workspace, forcing them to autonomously handle repository planning and dependencies. We evaluate the generated software by executing it in isolated sandboxes. The system-level side effects and terminal outputs are then compared against human-written oracles using a rigorous multi-tiered equivalence metric. Extensive evaluation of seven state-of-the-art LLMs reveals that the top-tier models achieve a maximum overall success rate of only 43.8%, highlighting that 0-to-1 software generation remains a highly challenging frontier. Furthermore, we discover that agents exhibit a strong tendency to generate monolithic code structures, and that higher token consumption does not necessarily yield better task performance.