Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
Yijiang Li, Huiqi Zou, Bingyang Wang, Ziang Xiao
Why It Matters
What makes this one worth your time
This research addresses the critical gap between benchmark evaluations and real-world applications, providing insights into model vulnerabilities that could inform future improvements in MLLMs.
CEDI offers a novel approach to evaluate MLLMs through interactive, contextualized assessments.
Summary
The paper introduces CEDI, a framework for evaluating multi-modal large language models through dynamic, multi-turn interactions, revealing more contextualized performance issues such as visual hallucinations compared to traditional static evaluations.
Key contributions
- Development of the CEDI framework for dynamic evaluation of MLLMs.
- Empirical demonstration of CEDI revealing more hallucinations than static evaluations.
- Identification of specific vulnerabilities in MLLMs related to premise rejection and refusal questions.
Notable insights
- The use of a graph-based representation for task guidance in evaluations is a clever methodological choice that enhances interaction dynamics.
- The finding that hallucinations accumulate over long contexts suggests a need for better handling of dialogue history in model training.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.14499v1 Announce Type: new Abstract: Multi-modal Large Language Models (MLLMs) have made substantial advances on benchmarks, yet their real-world effectiveness remains uncertain. This gap stems from the fundamental misalignment between benchmarks in controlled, static settings and the dynamic, interactive, and contextualized nature of real-world applications. To bridge this gap, we propose CEDI (Contextualized Evaluations of MLLMs through Dynamic, multi-round Interactions), a framework that recasts evaluation as a three-party interaction between an evaluatee model, an automated examiner, and a grader. The examiner conducts multi-turn, semi-structured conversation guided by a graph-based representation of the task. By navigating state-space transitions, CEDI deploys diverse strategies, from clarification requests to adversarial probes, to elicit performance evidence. We apply CEDI to visual hallucinations. Empirical results across multiple models, diverse settings, datasets, and domains show that contextualized, interactive evaluations reveal not only significantly more hallucinations than conventional static evaluation but also ones that more closely resemble those arising in practical use cases. We further show that hallucinations often accumulate over long contexts, through self-reinforcing dialogue history, and models are particularly vulnerable to questions requiring premise rejection or refusal. Together, these findings highlight CEDI as a step toward realistic, systematic, and ecologically valid assessments of MLLMs' capabilities. Code is available at github.com/williamium3000/cedi.