Multimodal Agentic System for Highway Safety Monitoring
Almarzooqi, Abdulla Abderrazek, Abid Karray, Fakhri
Almarzooqi, Abdulla
Abderrazek, Abid
Karray, Fakhri
Supervisor
Department
Machine Learning
Embargo End Date
Type
Conference proceeding
Date
License
Language
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
This research introduces a multimodal agentic system designed to improve highway safety by automating vehicle scene monitoring and accident reporting, addressing the limitations of traditional human-based surveillance methods. Two primary systems were developed: one that integrates YOLOv11 object detection with multimodal large language models (MLLMs) for rapid and accurate driving scene descriptions and anomaly detection; another automating detailed accident reporting through specialized agents monitoring weather conditions, seatbelt usage, airbag deployment, and video data. Upon accident detection, this system synthesizes processed dashcam footage and contextual input to generate comprehensive accident reports autonomously. The evaluations confirmed the system’s effectiveness in both tasks. In particular, integrating YOLOv11 with MLLMs in the scene description system significantly improved performance, with GPT-4o achieving a top score of 95.6%. In the accident report generation system, the frame sampling technique yielded the best results, reaching a peak score of 89.6% with GPT-4o. Integrating MLLMs and computer vision tools within an adaptable, scalable agentic framework provides an effective solution to enhance traffic safety and supports advancements in intelligent transportation systems.
Citation
A. Almarzooqi, A. Abderrazek, F. Karray, "Multimodal Agentic System for Highway Safety Monitoring," 2026, pp. 291-297.
Source
Conference
2025 Computing, Communications and IoT Applications (ComComAp)
Keywords
35 Commerce, Management, Tourism and Services, 3509 Transportation, Logistics and Supply Chains, 40 Engineering, 4005 Civil Engineering, 46 Information and Computing Sciences, 3 Good Health and Well Being
Subjects
Source
2025 Computing, Communications and IoT Applications (ComComAp)
Publisher
IEEE