EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks
Pau Arnal, Khaled Denfir, Danylo Smahliuk, Amrut Avhad, Marcus A. Castro
Why It Matters
What makes this one worth your time
Understanding the limitations of language models in complex, real-world decision-making tasks is crucial for their deployment in professional settings.
Frontier language models underperform on expert-level European executive decision tasks.
Summary
The paper evaluates six frontier language models on a new benchmark, EuroExec, which consists of 413 open-ended European executive decision tasks created by domain experts. The study finds that the best-performing model solves only 56.9% of tasks, significantly underperforming compared to human experts. The evaluation relies on human judgment and highlights the limitations of automatic measurements for such complex tasks.
Key contributions
- Introduction of the EuroExec benchmark for evaluating language models on executive decision tasks.
- Comprehensive human evaluation of language model performance on open-ended tasks.
Notable insights
- Human evaluators are essential for assessing model performance on subjective, real-world tasks.
- Automatic evaluation metrics may not capture the nuances of complex decision-making tasks.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2608.04549v2 Announce Type: replace Abstract: Frontier LLMs are increasingly put to use on open-ended complex questions, different in nature from the ones they are typically evaluated on. We dedicate more than 4,000 human expert hours to evaluate a selection of six frontier LLMs on a member of this class of problems: EuroExec, our introduced human expert-based benchmark composed of 413 open-ended long-form European executive tasks authored by 47 vetted domain experts, each question drawn from experience in a real case. Every response is manually evaluated through a multi-attribute rubric, an item-specific checklist of requirements, and a preference rank ordering, extracting an aggregate metric "Solve Rate". The strongest model solves only 56.9% of tasks, while expert-written reference answers judged blindly are solved at near-ceiling levels and are preferred over every model response in 74% of direct rankings, placing frontier generative systems well below the professional standard of work they are already used for. We see that the best way to extract this kind of conclusion is by employing human evaluators, carefully checking their consistency through rigorous statistical analysis, and observe that automatic measurements also fall short when evaluating on this case of real-world open-ended problems with a subjective ground truth.