When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis
Aisha Najera, Alvin Moon, Vedant Srinivasan, Rajesh Veeraraghavan
Why It Matters
What makes this one worth your time
Understanding and addressing model disagreements can improve the reliability and interpretive accuracy of LLMs in policy-related applications, ensuring diverse perspectives are considered.
The paper introduces a disagreement-based evaluation method for LLMs in public comment analysis.
Summary
The paper proposes an Interpretive Audit Pipeline to evaluate large language models (LLMs) used for categorizing public comments, focusing on inter-model disagreement as a diagnostic tool for interpretive complexity. It analyzes 1,260 public comments using four LLMs and highlights that thematic divergence between models is greater than within-model prompt variation. The study also includes a two-stage labeling process involving LLMs and a human annotator, revealing that human revisions often introduce new framings absent from the models' outputs.
Key contributions
- Proposes an Interpretive Audit Pipeline for evaluating LLMs based on inter-model disagreement.
- Conducts a comparative analysis of LLMs and human annotators on public comment categorization.
Notable insights
- Inter-model thematic divergence is greater than within-model prompt variation, indicating a need for multi-model analysis.
- Human annotators can introduce new interpretive framings that LLMs may miss, suggesting the value of human oversight.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2605.29025v1 Announce Type: new Abstract: Federal agencies are deploying large language models (LLMs) to categorize public comment corpora, where the model's organization of the record shapes what policymakers see and which arguments register. Standard evaluation, anchored on stance accuracy against a small validated set, cannot detect when different models produce materially different categorizations of the same public input. We propose an Interpretive Audit Pipeline that treats multi-model disagreement as diagnostic of interpretive complexity and directs human review toward genuinely ambiguous public input. Analyzing 1,260 public comments on a federal USDA docket across four LLMs, we find that inter-model thematic divergence exceeds within-model prompt variation, and that an expert rubric suppresses deep interpretive disagreement without resolving it. In a two-stage labeling study on a stratified 40-comment subsample, four LLMs and a human annotator labeled independently and then revised after seeing the others' labels. Revision behavior varied across labelers, and the human annotator's revisions frequently introduced framings absent from the ensemble's collective output. We argue disagreement-based evaluation is a necessary complement to accuracy metrics for LLM-assisted interpretive coding.