EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards
Siddhant Panpatil, Arth Singh, Mijin Koo, Chaeyun Kim, Haon Park, Dasol Choi
Why It Matters
What makes this one worth your time
This research addresses the critical need for effective safety measures in embodied AI systems, which are increasingly deployed in real-world environments.
EgoSafetyBench provides a novel benchmark for assessing VLMs in safety-critical environments.
Summary
The paper introduces EgoSafetyBench, a benchmark designed to evaluate vision-language models (VLMs) as runtime safety guards for embodied agents, focusing on distinguishing between safe and unsafe scenarios in egocentric video data.
Key contributions
- Introduction of EgoSafetyBench, a comprehensive benchmark with 1,200 annotated scenarios.
- Evaluation of ten VLMs against specific safety criteria, revealing performance gaps.
- Identification of the impact of misleading in-scene text on safety judgments.
Notable insights
- The use of contrastive ladders to isolate specific cues in safety judgments is a clever methodological approach.
- The differentiation between situational and visual-channel tracks highlights the complexity of safety assessments in VLMs.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.00218v1 Announce Type: cross Abstract: Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genuinely unsafe situations while avoiding unnecessary intervention on routine but superficially alarming activity, a distinction that binary safety benchmarks obscure. We introduce EgoSafetyBench, an egocentric video benchmark of 1,200 robot-view scenarios annotated at half-second granularity, to evaluate VLMs as streaming guards across two tracks. The situational track (800 scenarios) spans four families, from routine and safe-but-suspicious scenes to obvious and contextual hazards. The visual-channel track (400 scenarios) targets in-scene text-a sign, sticker, or label visible in the scene-that can misrepresent the physical situation, pairing each misleading sign with a truthful version to test both whether a guard flags the text as misleading and whether the text corrupts its physical-safety judgment. Both tracks use contrastive ladders: near-identical scenarios differing only in a single visible deciding cue, so a correct call must hinge on that cue rather than the overall scene type. We evaluate ten open- and closed-source VLMs. We find that while guards reliably recognize videos containing hazards, they often miss specific hazardous moments, particularly contextual hazards. Furthermore, misleading in-scene signs degrade all tested guards: vulnerable models miss up to a third of hazards, while robust models over-intervene on safe content. Matched controls reveal that apparent safety robustness often reflects indiscriminate alarming rather than true physical reasoning.