Lomekwi: Resource-Bounded Tool Discovery in LLM Agents
Roshan Klein-Seetharaman, Daniel Wang, Andrew Xu
Why It Matters
What makes this one worth your time
This research could inform the development of more effective LLM agents by addressing the complexities of tool discovery, which is crucial for advanced AI applications.
A new framework for understanding tool discovery in LLM agents reveals scaling challenges.
Summary
The paper introduces a framework for distinguishing tool discovery from tool use in LLM agents, focusing on curiosity, recognition, and efficiency, and presents evidence of inverse scaling of recognition with model size through combinatorial games and real-world task emulation.
Key contributions
- A framework that decomposes tool discovery into curiosity, recognition, and efficiency.
- Empirical evidence showing inverse scaling of recognition with model size.
- Analysis of combinatorial games to demonstrate the proposed framework.
Notable insights
- Recognition of tool creation processes may diminish as model size increases, suggesting a potential trade-off in scaling LLMs.
- The introduction of combinatorial games as a testing ground for tool discovery offers a novel approach to evaluate LLM capabilities.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.16961v1 Announce Type: new Abstract: Existing tool-use benchmarks report a single success rate for complex, multistep tasks. Inspired by ideas from cognitive science, we distinguish tool use from tool discovery and decompose the latter into curiosity (the model's ability to discover the parts needed to build the tool), recognition (the model's ability to discover the process of creating the tool), and efficiency (the model's use of the tool after creation). We show that this framework can be applied to existing discovery tasks, such as Voyager. In addition, we provide evidence that recognition inversely scales with model size, and we introduce and analyze a class of combinatorial games that demonstrates this. We further observe inverse scaling in a separate environment designed to emulate real-world tasks.