Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation
Prabal Shrestha, Bohan Jiang, Haoning Xue, Huan Liu, Xinyi Zhou
Why It Matters
What makes this one worth your time
Understanding the limitations of MLLMs in approximating human subjective responses can inform future research and development in AI-driven evaluations.
This study assesses the effectiveness of MLLMs in simulating human responses to video content.
Summary
The paper evaluates multimodal large language models (MLLMs) as synthetic participants in video-based studies, comparing their ratings of perceived sensory engagement with those of human participants using a specific framework.
Key contributions
- Empirical comparison of MLLM ratings against human participant ratings in video engagement studies.
- Identification of specific biases in MLLM responses, including mean-shift and central-tendency biases.
- Analysis of the impact of different prompting strategies on MLLM performance.
Notable insights
- The study reveals that leading MLLMs exhibit biases in rating distributions, which could affect their reliability as synthetic participants.
- Prompting strategies have varying effects on MLLM performance, indicating the need for careful design in AI evaluations.
Possible limitations
- The abstract does not address potential variations in video content that could influence results.
- Limited generalizability of findings due to the specific context of the study.
Abstract
arXiv:2606.07541v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have shown strong performance on objective tasks such as video understanding and reasoning. However, it remains unclear whether they can approximate subjective human responses, which depend not only on content comprehension but also on individuals' social contexts. To address this gap, we evaluate MLLMs as synthetic participants in an emerging task: assessing perceived sensory engagement with short videos. Grounded in the Perceived Message Sensation Value (PMSV) framework, we compare ratings from recruited human participants and profile-conditioned MLLM simulations (n=673) using a 17-item scale measuring emotional arousal, dramatic impact, and novelty. We find that even leading MLLMs (Gemini 3 Flash and Qwen 3 Omni) show limited agreement with human participants. The models exhibit distinct downward mean-shift and central-tendency biases in their rating distributions. They both introduce and flatten subgroup differences, while showing inconsistent sensitivity to participant profiles. Prompting strategies affect these metrics differently, modestly improving some aspects while worsening others. These results highlight both the challenges and opportunities of developing MLLMs as synthetic participants in video-based research. Data and code: https://github.com/MINDLab25/mllm-human-simulation-eval