Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System
Alyssa Unell, Miguel Fuentes, Brenna Li, Bridget Lin, Meena Jagadeesan, Sanmi Koyejo, Nigam Shah
Why It Matters
What makes this one worth your time
Understanding and predicting user rejection in clinical LLM systems can enhance system reliability and user trust, leading to more effective deployment in healthcare settings.
Predicting user rejection in clinical LLM systems using deployment-specific context.
Summary
The paper presents a deployment-centered evaluation of a large language model (LLM) system within a clinical setting, focusing on predicting the likelihood of user rejection of LLM responses based on query content and deployment-specific context. The study trains a pre-response classifier and evaluates its performance over 4.5 months, achieving an AUROC of 0.719. The research highlights the importance of using deployment-specific context to improve prediction accuracy.
Key contributions
- Development of a pre-response classifier to predict user rejection risk in clinical LLM systems.
- Prospective analysis of prediction model performance over 4.5 months of user feedback.
- Demonstration of the benefit of deployment-specific context in improving prediction accuracy.
Notable insights
- Utilizing deployment-specific context, such as provider type and department name, improves prediction of user rejection.
- The study demonstrates the feasibility of predicting user rejection in real-world clinical settings.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.12702v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets -- leading to major blind spots for evaluating clinical systems. In this work, we perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions. Specifically, we train a pre-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment-specific context available before generation. We conduct a prospective analysis of our model over 4.5 months of user feedback, finding that our prediction model achieves an AUROC of 0.719. Further, we estimate the benefit of such predictions in two downstream use cases (guardrail triggering and abstention). Our key conceptual insight is that making use of deployment-specific context (i.e., the provider type, department name, language model used for response), as opposed to only query content, improves the ability to predict whether the user will reject the system output. Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment-specific context, opening the door to targeted guardrails.