When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses
Zihan Chen, Di Zhu, Lei Nico Zheng
Why It Matters
What makes this one worth your time
Understanding the limitations of LLMs in simulating human responses is crucial for researchers and practitioners relying on synthetic data for decision-making in product, policy, and market contexts.
The paper benchmarks LLMs as synthetic survey respondents, revealing significant limitations.
Summary
The paper evaluates the validity of using large language models (LLMs) as synthetic users for survey responses by benchmarking them against human data in two domains: U.S. general social attitudes and cross-cultural values. It identifies two consistent failures across models, highlighting the limitations of LLMs in accurately simulating human survey responses.
Key contributions
- A cross-domain benchmark for evaluating LLM-simulated survey responses.
- Identification of systematic failures in LLMs' ability to simulate human survey responses.
Notable insights
- LLMs tend to overestimate the predictive power of demographics on attitudes.
- Larger, more capable models do not necessarily overcome the identified failures.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.26348v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems. A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data: U.S. general social attitudes (General Social Survey) and cross-cultural values (World Values Survey). Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data. Under demographic prompting and the survey-simulation protocols we test, two failures replicate across both domains, all four models, and both families. First, at the individual level no LLM beats even the strongest baseline; on cross-cultural values every model falls well below it, and the gap survives distance-aware and proper scoring. Second, models systematically over-determine demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question-group combination and robust to a coding-invariant measure. Neither failure is remedied by a larger, more capable model. A decision-impact analysis shows why this matters in practice: on a segment-targeting task the models inflate between-segment gaps two to fourfold, would direct a team to the wrong segment in half of U.S. and most cross-cultural cases, and manufacture segment splits that do not exist in real people. We make the cross-domain benchmark and the evaluation framework available on request, so that teams can determine in advance when synthetic-user evidence is safe for decision support and when it is not.