Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoning
Zach Studdiford, Gary Lupyan
Why It Matters
What makes this one worth your time
Understanding the similarities in reasoning errors between humans and LLMs can inform the development of more robust AI systems and improve our understanding of human cognition.
The paper suggests that both human and LLM reasoning may rely on pattern-matching rather than abstract world models.
Summary
The paper evaluates human participants and 25 large language models (LLMs) on common-sense reasoning tasks, finding similar error patterns between humans and LLMs. It identifies attention heads in LLMs that perform pattern-matching, suggesting that both human and LLM reasoning may rely more on pattern-matching than abstract world models.
Key contributions
- Comparison of reasoning errors between humans and LLMs.
- Identification of attention heads in LLMs that perform pattern-matching.
Notable insights
- Attention heads in LLMs can predict reasoning errors caused by irrelevant prompt details.
- Human reasoning errors may be more aligned with pattern-matching than previously thought.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.13607v1 Announce Type: new Abstract: When large language models (LLMs) fail to generalize or make haphazard errors in reasoning, it is often taken as evidence that LLMs are not truly reasoning, but rather performing a kind of pattern matching. The implication is that people's behavior does not exhibit the same types of failures because human reasoning uses principled and abstract world models. We evaluate human participants and 25 LLMs on their ability to engage in common-sense reasoning about a variety of everyday situations and observe similar patterns of errors in both people and models. We then identify the set of attention heads driving LLM responses and find that these heads implement a form of pattern-matching. These attention heads allow us to predict seemingly inexplicable reasoning errors in people caused by ostensibly irrelevant prompt details. Taken together, our results suggest that everyday causal reasoning in people and LLMs is more consistent with a form of pattern-matching than with abstract world models.