CMU-Drive Benchmark
We introduce Cooperative Multi-agent Unified Driving with Reasoning (CMU-Drive), a closed-loop, end-to-end cooperative driving benchmark. CMU-Drive inherits the 44 types of safety-critical scenarios from the single-agent benchmark Bench2Drive, including pedestrian crossing, emergency vehicles running a red light. Each scenario is instantiated under five different weather and lighting conditions, resulting in a total of 220 evaluation routes. CMU-Drive extends each scenario to include 2 to 16 cooperative autonomous vehicles (CAVs).
Evaluation route samples in CMU-Drive across different types of safety-critical scenarios and varying numbers of connected autonomous vehicles (CAVs). Each solid arrow represents a feasible path from the configured starting location to the destination of a CAV. Each dashed arrow represents the traffic flow of the configured background traffic participants within each safety-critical driving scenario. Each scenario also includes additional non-cooperative background vehicles driving nearby.
Statistics of the evaluation routes in CMU-Drive, in comparison to other cooperative multi-agent, end-to-end, closed-loop driving benchmarks. # Routes: the number of evaluation routes. # CAVs: the number of connected autonomous vehicles per route. # GPUs: the number of GPUs required to run the evaluation per route.
V2V-VLA Model
We propose V2V-VLA, a new vehicle-to-vehicle vision-language-action model for cooperative autonomous driving, as the first baseline model with multi-agent cooperation in CMU-Drive, as shown in the following figure. In each frame of the closed-loop simulation, each cooperative driving agent uses its own V2V-VLA model to jointly generate driving actions, language reasoning, future waypoints, and an optional communication policy indicating whether another cooperative vehicle should be queried in the next timestep.
V2V-VLA model architecture. The input consists of a language prompt, target waypoints, and vision tokens derived from the ego vehicle's front-camera image and a merged occupancy map from the ego and the selected communicating vehicle. The output comprises language outputs for the action, reasoning, and communication, alongside the future waypoints.
Experimental Results
Simlingo is used as a baseline method that does not involve multi-agent cooperation. As shown in the following table, our proposed V2V-VLA achieves a better driving score DS and a success rate SR.
The following figure shows the average driving score of V2V-VLA in each of the 44 different scenario types in CMU-Drive, indicating that several safety-critical scenarios are still challenging, such as NonSignalizedJunctionLeftTurnEnterFlow, which requires more future research.
The following figure llustrates a sample frame from an evaluation route in CMU-Drive. The detection of the leading bicycle by the ego vehicle (CAV_0) is hindered by the long distance and rain-induced image blur. However, the bicycle is successfully identified by a leading connected autonomous vehicle (CAV_4), which is closer to the hazard. By utilizing vehicle-to-vehicle (V2V) communication, the ego vehicle (CAV_0) safely reduces its speed to prevent a collision.
Acknowledgment
The authors thank Ryo Hachiuma, Chien-Yi Wang, Yu-Chiang Frank Wang, Min-Hung Chen from NVIDIA and John Dolan, Deva Ramanan from Carnegie Mellon University Robotics Institute for valuable discussions and feedback.