UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
Pengyu Zhu, Lijun Li, Yaxing Lyu, Qianxin Luo, Jingyi Yang, Yi Liu, Tingfeng Hui, Xinyu Yuan, Li Sun, Sen Su, Jing Shao
Why It Matters
What makes this one worth your time
This framework enables more reliable assessments of LLMs as agents, which is crucial for their deployment in safety-critical applications and for advancing research in AI capabilities.
A comprehensive framework to evaluate LLM agentic capabilities while disentangling environmental effects.
Summary
The paper presents a unified framework for evaluating the agentic capabilities of large language models (LLMs), addressing issues with benchmark interpretation by integrating diverse benchmarks into a standardized format and analyzing the effects of framework and environment on model performance.
Key contributions
- A unified configuration system that standardizes diverse benchmarks for LLM evaluation.
- A methodology that separates framework effects from environment effects in performance assessments.
- Adaptation of 7 widely used benchmarks across various domains for comprehensive analysis.
Notable insights
- The framework's fixed ReAct-style architecture allows for controlled experimentation, isolating intrinsic model capabilities from external influences.
- The introduction of unified metrics for resource consumption provides a new dimension for evaluating LLM performance beyond task success.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2605.27898v3 Announce Type: replace Abstract: Agent benchmarks are increasingly used to compare large language models (LLMs) across domains, yet a reported score reflects a complete model--harness--environment configuration rather than the model alone. Benchmark packages couple native tasks with specific prompts, tool protocols, orchestration logic, and sometimes dynamic external resources, making cross-benchmark comparisons sensitive to implementation and resource conditions. We present UniACE, a unified framework for model-centric evaluation under an explicit, common execution condition. UniACE represents each benchmark as an instruction--tool--environment triplet, executes LLMs through a shared, task-agnostic harness in isolated per-task runtimes, and preserves native success criteria. For tasks that rely on dynamic resources, an optional offline mode replaces live access with fixed, pre-collected snapshots. Its evaluation protocol further standardizes efficiency measurement, execution records, and trace-based failure attribution. We migrate 7 benchmarks spanning 24 domains and evaluate 15 models in more than 400K rollouts consuming 5B tokens. Comparisons with source implementations show large bidirectional score changes and model-ranking reversals, while matched online and offline runs reveal substantial sensitivity to accessible evidence and its representation. Under the shared UniACE configuration, efficiency and failure profiles expose task-dependent model behaviors hidden by task-success scores alone. These findings motivate reporting agent benchmark outcomes as properties of an explicit evaluation configuration, enabling more interpretable and reproducible cross-benchmark comparisons. Codes and benchmarks at are available at https://github.com/whfeLingYu/A-Unified-Framework-for-the-Evaluation-of-LLM-Agentic-Capabilities, https://huggingface.co/datasets/whfeLingYu/Unified_Agent_Framework.