NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning
Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar
Why It Matters
What makes this one worth your time
This research addresses a critical gap in evaluating AI's normative competence in real-world scenarios, moving beyond simple action selection to deeper reasoning.
NoRA benchmarks the ability of AI to justify actions based on visual context.
Summary
The paper introduces NoRA, a benchmark for evaluating visual first-person normative action reasoning in AI systems, requiring models to generate actions and justify them using a support graph based on visible facts.
Key contributions
- Development of the NoRA benchmark with 1,420 annotated video clips.
- Introduction of a grounded reasonableness score that aggregates multiple evaluation criteria.
- Benchmarking of 12 multimodal systems under structured prompting regimes.
Notable insights
- The introduction of a support graph for justifying actions is a novel approach to assessing grounded reasonableness.
- The benchmark's evaluation metrics focus on action alignment and factual grounding, which are crucial for real-world applicability.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.04806v1 Announce Type: cross Abstract: LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior. However, existing approaches either assess normative judgment in text alone or reduce it to choosing among a fixed set of candidate actions. We argue both are insufficient. In practice, agents are never handed a menu of options; they must identify a reasonable action from scratch, grounded in visible facts and supported by inspectable reasons. We introduce NoRA, a visual first-person video benchmark that requires models to generate candidate next actions and justify each through an explicit fact-reason-action support graph. The benchmark comprises 1,420 annotated video clips, including HumanGold-190 and LLMSilver-1230 splits. Each instance is evaluated through action alignment, factual grounding, and support binding, aggregated into a single grounded reasonableness score. We benchmark 12 multimodal systems under direct, deliberate, and structured prompting regimes, finding that current VLMs frequently recover plausible actions and relevant scene facts, but consistently struggle to construct the full reasonable action space and bind selected actions to the correct local support. NoRA makes this gap measurable, shifting the evaluation question from whether a model can pick an action to whether it can justify an appropriate action for the right visible reasons.