Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran
Why It Matters
What makes this one worth your time
This framework provides a structured approach to evaluate conversational agents at scale, addressing key challenges in governance and reproducibility, which are crucial for deploying reliable AI systems in production environments.
A scalable pipeline for evaluating retail conversational agents with a focus on governance and auditability.
Summary
The paper introduces GenAI Evaluation, a governed, configuration-driven pipeline designed for large-scale evaluation of retail conversational agents. It addresses challenges in governance, reproducibility, and reliability by processing chatbot logs through normalization, sharding, and schema-constrained LLM scoring. The framework evaluates multiple dimensions such as helpfulness and clarity, and supports auditability through schema locking and record-level provenance. It processes around 50,000 records daily and has been validated with human-labeled records, achieving a macro F1 score of 0.93.
Key contributions
- Development of a governed, configuration-driven evaluation pipeline for conversational agents.
- Implementation of schema-constrained LLM scoring for multi-dimensional evaluation.
- Integration of auditability features like schema locking and record-level provenance.
Notable insights
- The use of schema-constrained LLM scoring for evaluating conversational agents.
- Selective re-evaluation of only incomplete or malformed records to optimize resource use.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality. Although LLM-as-a-judge methods provide scalable alternatives to human evaluation, production deployment introduces challenges in governance, reproducibility, cost, schema consistency, traceability, and reliability. We present GenAI Evaluation, a governed, configuration-driven pipeline for large-scale evaluation of retail conversational systems. It processes production chatbot logs through normalization, sharding, asynchronous execution, and schema-constrained LLM scoring. The framework evaluates helpfulness, truthfulness, clarity, tone alignment, and translation-specific dimensions. Selective re-evaluation processes only incomplete, malformed, or schema-invalid records, while schema locking, versioned configurations, validation logs, and record-level provenance support auditability. The framework processes approximately 50,000 records daily and has evaluated more than two million interactions. Validation used 12,980 stratified-random human-labeled records from four trained annotators. Classification covered 14 intents, 156 sub-intents, 18 major domains, and 129 sub-domains. The pipeline achieved a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation.