A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents
Vasundra Srinivasan
Why It Matters
What makes this one worth your time
Understanding and optimizing the integration of stochastic LLM outputs with deterministic systems is crucial for improving the reliability and efficiency of production AI systems.
The paper proposes a new architectural framework for integrating stochastic LLM outputs with deterministic systems in production environments.
Summary
The paper introduces the concept of a stochastic-deterministic boundary (SDB) as a key architectural element in production LLM agents, proposing a methodology for selecting and composing runtime architecture patterns. It presents six runtime patterns and a five-step methodology for pattern selection, along with a diagnostic procedure for identifying pattern weaknesses and a new failure mode called replay divergence.
Key contributions
- Definition and exploration of the stochastic-deterministic boundary (SDB) in LLM agent architectures.
- A catalog of six runtime patterns for different agent types, linking them to distributed-systems concepts.
- A five-step methodology for selecting runtime patterns and a diagnostic procedure for identifying pattern weaknesses.
Notable insights
- The introduction of the stochastic-deterministic boundary (SDB) as a load-bearing primitive in production agent runtimes.
- Identification of a new failure mode, replay divergence, highlighting challenges in maintaining consistency across model versions and prompts.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2605.20173v1 Announce Type: new Abstract: Production LLM agents combine stochastic model outputs with deterministic software systems, yet the boundary between the two is rarely treated as a first-class architectural object. This paper names that boundary the stochastic-deterministic boundary (SDB): a four-part contract among a proposer, verifier, commit step, and reject signal that specifies how an LLM output becomes a system action. We argue that the SDB is the load-bearing primitive of production agent runtimes. Around this primitive, we organize agent runtime design into three concerns: Coordination, State, and Control. We present a catalog of six runtime patterns that compose the SDB differently across conversational, autonomous, and long-horizon agents: hierarchical delegation, scatter-gather plus saga, event-driven sequencing, shared state machine, supervisor plus gate, and human in the loop. For each pattern, we trace its lineage to distributed-systems concepts and identify what changes when the worker is stochastic. The paper contributes a five-step methodology for selecting runtime patterns, a diagnostic procedure that maps production failures to pattern weaknesses, and a failure mode called replay divergence, in which LLM-based consumers of a deterministic event log produce different downstream outputs under model-version or prompt changes. A stylized reliability decomposition separates per-call model variance from architectural momentum, motivating the claim that as model variance decreases, pattern choice and SDB strength become increasingly important levers for long-run reliability. We apply the methodology to five workloads and provide one runnable reference implementation for a 90-day contract-renewal agent.