Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry
Hannah VanderHoeven, Abhijnan Nath, Nikhil Krishnaswamy
Why It Matters
What makes this one worth your time
Understanding LLMs' limitations in pragmatic reasoning is crucial for improving their performance in collaborative and agentic applications where information asymmetry is common.
The paper explores LLMs' challenges in cooperative communication under partial information conditions.
Summary
The paper investigates the pragmatic reasoning capabilities of large language models (LLMs) in collaborative tasks where information is asymmetrically distributed among participants. It formalizes the concept of collaborative epistemic asymmetry and evaluates LLMs' abilities to adhere to Grice's cooperative principle under these conditions, using both prompting and post-training strategies.
Key contributions
- Formalization of collaborative epistemic asymmetry in LLMs.
- Empirical assessment of LLMs' pragmatic reasoning under partial information conditions.
Notable insights
- The study connects task success with Grice's cooperative principle, highlighting specific failure modes in LLMs.
- The research employs both prompting and post-training strategies to enhance LLMs' pragmatic capabilities.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.11053v1 Announce Type: cross Abstract: Fruitful collaborations rely on cooperative communications, including of contextual cues to incorporate into reasoning. The increasing use of LLMs in collaborative and agentic pipelines raises questions about the extent to which they exhibit these pragmatic capabilities, especially in scenarios where they may not have access to the same information as their collaborators. In this paper, we perform a novel investigation into the pragmatic reasoning capabilities of LLMs in a multi-party collaborative task under partial information conditions. We formalize a notion of collaborative epistemic asymmetry that explicitly connects objective task success to Grice's cooperative principle and empirically assess various LLMs' abilities to act cooperatively as both speakers and listeners, including both prompting and post-training strategies. Our results show that while LLMs exhibit certain pragmatic capabilities in collaborative settings, and these can be elicited through prompting and post-training, they still face challenges in pragmatic communication with incomplete information, and that certain failure modes do correlate with floutings of Grice's maxims that go unrecognized.