Toward Measuring Structural Drift in LLM Communication Loops
Wael Hafez, Amir Nazeri, Chenan Wei
Why It Matters
What makes this one worth your time
Understanding and monitoring conversational consistency is crucial for improving the reliability of LLMs in real-world applications, where coherent interactions are essential for user trust and decision-making.
This research provides a novel approach to assess conversational drift in LLM interactions through token statistics.
Summary
The paper introduces a method to monitor conversational consistency in multi-turn interactions of large language models (LLMs) using token frequency statistics, formalizing a measure called Bipredictability and implementing it in an auxiliary architecture called Information Digital Twin.
Key contributions
- Formalization of Bipredictability as a measure of conversational structural consistency.
- Development of the Information Digital Twin architecture for monitoring token statistics.
- Empirical validation of the method across a substantial dataset of conversational turns.
Notable insights
- The introduction of Bipredictability as a metric for assessing conversational consistency is a novel approach that diverges from traditional semantic evaluations.
- The use of token frequency statistics to detect contradictions and topic shifts offers a lightweight alternative to more complex evaluation methods.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2604.13061v3 Announce Type: replace-cross Abstract: Large language models increasingly run in stateful pipelines that assemble each prompt from retrieval, memory, tools, and other agents. Such pipelines drift: information that should shape the next response is dropped, compressed, or misrouted while every component still reports success. Existing diagnostics miss this because they evaluate isolated prompts, responses, or task scores, whereas what decouples is the relation between a prompt and the response it draws. Here we show that treating the prompt to response to next prompt chain as the fundamental unit of analysis makes these relations measurable. We introduce structural communication coherence, quantified by two metrics: communication closure, which asks if what the pipeline returns at one turn matches what it faces next, and normalized conditional action contribution, which measures how much a sent message resolves the subsequent reply. Across 2,171 human to human, 58 human to LLM, and 8 LLM to LLM dialogues, these metrics reveal directional interaction structures; crucially, the measured contribution drops by 87 to 92% when a response is swapped for one from another turn, leaving surrounding prompts untouched. Because this approach requires no labels, healthy reference data, or predefined rules only the raw prompts and responses drift can be defined and measured directly from operational traffic, rather than inferred from eventual task failure. Establishing prospective detection performance is the next step.