WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics
Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, Tao Xie
Why It Matters
What makes this one worth your time
This benchmark addresses critical gaps in evaluating LLM-generated web applications, providing a structured approach that can enhance model development and optimization.
WebCoderBench sets a new standard for benchmarking web app generation in LLMs.
Summary
The paper introduces WebCoderBench, a benchmark for evaluating web application generation by large language models, featuring real user requirements and a comprehensive set of evaluation metrics.
Key contributions
- Introduction of a benchmark with 1,572 real user requirements for web app generation.
- Development of 24 fine-grained evaluation metrics across 9 perspectives.
- Implementation of a fully automated evaluation system that combines different evaluation paradigms.
Notable insights
- The combination of rule-based and LLM-as-a-judge paradigms for evaluation allows for a more nuanced assessment of model performance.
- The use of human-preference-aligned weights over metrics enhances the interpretability of overall scores.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2601.02430v3 Announce Type: replace-cross Abstract: Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.