When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents
Yin Li
Why It Matters
What makes this one worth your time
Understanding the limitations of schema validation is crucial for developers relying on LLMs for safe and accurate transaction processing.
OrderBench exposes the limitations of schema validation in LLM transaction outputs.
Summary
The paper introduces OrderBench, a benchmark for evaluating the semantic reliability of schema-constrained LLM ordering agents, revealing that schema-valid outputs can still exhibit significant semantic errors.
Key contributions
- Introduction of OrderBench as a deterministic benchmark for restaurant ordering agents.
- Empirical analysis demonstrating the discrepancy between schema validity and semantic success rates across different LLMs.
Notable insights
- Schema validity does not guarantee semantic correctness, highlighting a critical gap in LLM output evaluation.
- The benchmark's focus on separating various aspects of output validity provides a more nuanced understanding of LLM performance.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.18261v1 Announce Type: new Abstract: LLM agents are increasingly used as transaction compilers: a user states an intent in natural language, and the model emits a structured object that an API can execute. JSON Schema and provider-level structured-output modes are useful because they remove a large class of parse failures, but they do not by themselves decide whether the object is a safe, faithful transaction. We introduce OrderBench, a deterministic benchmark for restaurant ordering agents that separates syntactic validity, schema validity, status decisions, exact item semantics, constraint preservation, and unsafe acceptances. Across 2,400 Nebius Token Factory calls to four open models in prompt-only and JSON-schema modes, we find that schema-valid output can still have large semantic error rates. In the strongest model, both modes achieve 100% schema validity, yet semantic success remains near 80%; in weaker models, schema-valid unsafe acceptances occur in double digits. The result is a concrete engineering warning: structured output is a necessary interface layer, not a substitute for domain verification and fail-closed execution.