What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities
Sukai Huang, Chenyuan Zhang, Fucai Ke, Zhixi Cai, Naim Rastgoo, Gholamreza Haffari, Hamid Rezatofighi
Why It Matters
What makes this one worth your time
Understanding distinct planning abilities in LLMs can guide the development of more effective models and improve task-specific performance evaluation.
The study identifies two distinct planning abilities in LLMs, offering a nuanced understanding of their planning performance.
Summary
The paper investigates the planning abilities of large language models (LLMs) by analyzing their performance on the ACPBench-Hard benchmark. It identifies two distinct planning competencies: operational reasoning and structural enumeration, and examines how these competencies are affected by model scaling and reasoning budgets.
Key contributions
- Identification of two distinct planning abilities in LLMs: operational reasoning and structural enumeration.
- Application of a multidimensional item response theory model to analyze LLM planning competencies.
Notable insights
- The paper uses a multidimensional item response theory model to uncover latent competencies in LLM planning.
- Operational reasoning improves with model scaling and longer reasoning traces, unlike structural enumeration.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.11197v1 Announce Type: new Abstract: When LLMs exhibit uneven performance across planning tasks, these gaps are often attributed to task difficulty. We argue that this explanation is incomplete, as task-level variation may reflect distinct latent planning competencies rather than differences along a single ability spectrum. We study this question on ACPBench-Hard by evaluating multiple LLM families under varying test-time reasoning budgets and applying a multidimensional item response theory model to uncover the latent competency structure underlying LLM planning. The analysis reveals two principal dimensions that shape planning performance: operational reasoning, the ability to evaluate local action applicability and immediate state transitions, and structural enumeration, the ability to reason about goal reachability and landmark structure. Operational reasoning improving under model scaling and longer reasoning traces, while structural enumeration remains comparatively insensitive. Our findings motivate competency-level evaluation of LLM planning, shifting the focus from whether models improve overall to which planning competencies improve, under what conditions, and why.