DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving
Zilin Huang, Zihao Sheng, Zhengyang Wan, Yansong Qu, Junwei You, Sicong Jiang, Sikai Chen
Why It Matters
What makes this one worth your time
This research addresses critical safety challenges in autonomous driving by leveraging advanced semantic understanding, making it relevant for both practical applications and further research in the field.
DriveVLM-RL combines vision-language models with reinforcement learning for safer autonomous driving.
Summary
The paper presents DriveVLM-RL, a novel reinforcement learning framework that integrates vision-language models to enhance safe and deployable autonomous driving by employing a dual-pathway architecture for reward learning and risk assessment.
Key contributions
- Introduction of a neuroscience-inspired dual-pathway architecture for integrating vision-language models into reinforcement learning.
- Development of a hierarchical reward synthesis mechanism that combines semantic and vehicle state information.
- Demonstration of significant performance improvements in collision avoidance and task success in the CARLA simulator.
Notable insights
- The dual-pathway architecture effectively separates static safety assessments from dynamic risk reasoning, which is inspired by human cognitive processes.
- The asynchronous training pipeline allows for decoupling expensive inference from real-time vehicle control, enhancing deployment efficiency.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2603.18315v2 Announce Type: replace-cross Abstract: Traditional reinforcement learning (RL) methods rely on manually engineered rewards or sparse collision signals, which fail to capture the rich contextual understanding required for safe driving and make unsafe exploration unavoidable in real-world settings. Recent vision-language models (VLMs) offer promising semantic understanding capabilities; however, their high inference latency and susceptibility to hallucination hinder direct application to real-time vehicle control. To address these limitations, this paper proposes DriveVLM-RL, a neuroscience-inspired framework that integrates VLMs into RL through a dual-pathway architecture for safe and deployable autonomous driving. Inspired by the human brain's habitual and deliberative visual processing, DriveVLM-RL decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment via CLIP-based contrasting language goals, and a Dynamic Pathway for attention-gated multi-frame semantic risk reasoning via a lightweight detection model and large VLM (LVLM). A hierarchical reward synthesis mechanism fuses these signals with vehicle state information, while an asynchronous training pipeline decouples expensive LVLM inference from environment interaction. Critically, all VLM components operate exclusively during offline training and are completely removed at deployment, eliminating inference latency at test time. Extensive experiments in the CARLA simulator demonstrate that DriveVLM-RL significantly outperforms state-of-the-art baselines in collision avoidance and task success, attaining the highest success rate while reducing collision severity from 10.09 to 1.75 km/h relative to the strongest VLM-based baseline. The demo video, code, and model checkpoints are available at: https://zilin-huang.github.io/DriveVLM-RL-website/