PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents
Hongliang Li, Yijin Liu, Zhiwei Zhang, Zihe Liu, Xinyue Lou, Jinan Xu, Fandong Meng, Kaiyu Huang
Why It Matters
What makes this one worth your time
Understanding the impact of multilinguality on LLM agents is crucial for developing more robust and versatile AI systems capable of handling real-world multilingual workflows.
PolyWorkBench evaluates LLM agents on multilingual long-horizon tasks, revealing performance challenges.
Summary
The paper introduces PolyWorkBench, a benchmark designed to evaluate large language model agents on multilingual long-horizon tasks across various domains. It highlights the challenges of multilinguality in agentic execution and proposes a hybrid evaluation framework to assess functional correctness and linguistic consistency.
Key contributions
- Introduction of PolyWorkBench, a benchmark for multilingual long-horizon tasks.
- Empirical analysis showing performance degradation in multilingual settings.
- Proposal of a hybrid evaluation framework for comprehensive assessment.
Notable insights
- Multilinguality introduces compounding effects on reasoning and execution in LLM agents.
- A hybrid evaluation framework combining structural grading, executable verification, and semantic assessment is proposed.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.06008v2 Announce Type: replace-cross Abstract: Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks implicitly assume a monolingual setting, where the entire execution process, including reasoning, tool invocation, and output generation, is conducted within a single language. In contrast, real-world applications often involve multilingual inputs and outputs within a unified workflow, yet the interaction between multilinguality and agentic execution remains underexplored. In this work, we introduce PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows. PolyWorkBench consists of 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing, where agents must process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. To enable comprehensive evaluation, we propose a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment. This design allows us to capture both functional correctness and linguistic consistency across complex workflows. Empirical results show that state-of-the-art LLM agents suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts. Our analysis suggests that multilinguality introduces compounding effects across reasoning and execution steps, highlighting the importance of jointly modeling language variation and procedural decision-making in agent evaluation.