Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation
Jonghyun Jee, Aaron Shaw
Why It Matters
What makes this one worth your time
Understanding LLMs' biases in evaluative hierarchies can inform their deployment in recommendation systems and cultural content analysis.
The study reveals LLMs' bias towards critically acclaimed films over commercially successful ones.
Summary
The paper investigates whether large language models (LLMs) reflect evaluative hierarchies in film preferences, focusing on critical acclaim versus commercial success. Using a 200-film benchmark and analyzing 20,000 pairwise comparisons per model, the study finds that LLMs consistently prefer critically acclaimed films over commercially successful ones, with this tendency increasing with model scale. The study also examines how evaluative orientation, public visibility, and popular reception influence these preferences.
Key contributions
- A benchmark of 200 films categorized by critical acclaim and commercial success.
- Analysis of LLMs' film preferences using Bradley--Terry estimation across multiple model families.
- Investigation of the impact of evaluative orientation, public visibility, and popular reception on LLMs' preferences.
Notable insights
- Adjusting for public visibility can reverse LLMs' preference for dual-legitimacy films over critical acclaim-only films.
- Evaluative and recommendation-oriented prompt framings produce divergent film rankings, indicating indirect manifestations of critical acclaim orientation.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2608.06955v1 Announce Type: new Abstract: Large language models (LLMs) are trained on corpora that contain expressions of human judgment about films, books, music, and more. Yet whether LLMs systematically reproduce evaluative hierarchies remains unclear. Prior research on cultural bias in LLMs suggests competing expectations: models may mirror the popularity signals of internet texts, or may reproduce forms of prestige embedded in critical discourse. We probe this question through a study of film evaluations with eight models from four families (Anthropic, OpenAI, Alibaba, and Mistral), using a 200-film benchmark partitioned into critically acclaimed, commercially successful, and dual-legitimacy (critical acclaim + commercial success) films. Across 20,000 pairwise forced-choice comparisons per model analyzed with Bradley--Terry estimation, we observe a consistent critical acclaim orientation with all models: critically acclaimed yet commercially obscure films are selected over commercially successful yet critically unrecognized ones. This pattern grows with model scale within each family. In addition, nested OLS regression analyses show that evaluative orientation, public visibility, and popular reception distinctly help explain preferences. Adjusting for public visibility reverses the models' preference for dual-legitimacy films over critical acclaim-only films, while additionally accounting for popular reception attenuates much of the disadvantage of films with commercial success only. Finally, evaluative and recommendation-oriented prompt framings produce divergent rankings, suggesting that critical acclaim orientation may manifest indirectly in real-world LLM deployments.