How Can Simulation-based Safety Testing Help Understand the Real-World Safety of Autonomous Driving Systems?


  • Fauzia Khan Dept. of Computer Science, University of Tartu
  • Laima Dalbina Dept. of Computer Science, University of Tartu
  • Hina Anwar Dept. of Computer Science, University of Tartu
  • Dietmar Pfahl Dept. of Computer Science, University of Tartu


Autonomous Driving System (ADS), Human Driven Vehicle (HDV), Safety Testing, Scenario Generation, CARLA Simulator


An Automated Driving System (ADS) requires exhaustive safety testing before receiving a road permit. Moreover, it is not clear what exactly constitutes sufficient safety for an ADS. One would assume that an ADS is safe enough if it is at least as safe as a Human Driven Vehicle (HDV). However, evaluating the safety of an ADS by comparing its behavior with that of a typical HDV in the real world is costly and risky. In this paper, we give an overview of our approach to compare the performance of ADS with HDV. While the overall approach is still in progress and ongoing, we provide a detailed approach utilizing established guidelines to systematically generate test scenarios specifically aimed at safety testing. Using our approach, various scenarios could be generated and tested, contributing to autonomous vehicles’ trustworthiness.


F. Duarte and C. Ratti, “The impact of autonomous vehicles on cities: A review,” Journal of Urban Technology, vol. 25, no. 4, 2018.

“Uk standards: Natural language description for abstract scenarios for automated driving systems,” Specification-BSI Flex 1889v1.0:202207,

Y. Zhu, J. Wang, F. Meng, and T. Liu, “Review on functional testing scenario library generation for connected and automated vehicles,” Sensors, vol. 22, no. 20, p. 7735, 2022.

W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on safety-critical driving scenario generation-a methodological perspective,” IEEE Transactions on Intelligent Transportation Systems, 2023.

H. Elrofai, D. Worm, and O. Op den Camp, “Scenario identification for validation of automated driving functions,” in Advanced Microsystems for Automotive Applications 2016: Smart Systems for the Automobile of the Future, Springer, 2016, pp. 153–163.

S. Geyer, M. Baltzer, B. Franz, S. Hakuli, M. Kauer, M. Kienle, S. Meier, T. Weißgerber, K. Bengler, R. Bruder, et al., “Concept and development of a unified ontology for generating test and usecase catalogues for assisted and automated vehicle guidance,” IET Intelligent Transport Systems, vol. 8, no. 3, pp. 183–189, 2014.

T. Menzel, G. Bagschik, and M. Maurer, “Scenarios for development, test and validation of automated vehicles,” in 2018 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2018, pp. 1821–1827.

P. Kaur, S. Taghavi, Z. Tian, and W. Shi, “A survey on simulators for testing self-driving cars,” in Fourth International Conference on Connected and Autonomous Driving (MetroCAD), IEEE, 2021.

S. Park, S. Park, H. Jeong, I. Yun, and J. So, “Scenario-mining for level 4 automated vehicle safety assessment from real accident situations in urban areas using a natural language process,” Sensors, vol. 21, no. 20, p. 6929, 2021.

M. O’Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi, “Scalable end-to-end autonomous vehicle testing via rare-event simulation,” Advances in neural information processing systems, 2018.

S. Jha, T. Tsai, S. Hari, M. Sullivan, Z. Kalbarczyk, S. W. Keckler, and R. K. Iyer, “Kayotee: A fault injection-based system to assess the safety and reliability of autonomous vehicles to faults and errors,” arXiv preprint arXiv:1907.01024, 2019.

R. Ben Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, “Testing advanced driver assistance systems using multi-objective search and neural networks,” in IEEE/ACM international conference on automated software engineering, 2016.

M. Wicker, X. Huang, and M. Kwiatkowska, “Feature-guided black box safety testing of deep neural networks,” in 24th International Conference, Held as Part of the European Joint Conferences on Theory and Practice of Software, Greece, Springer, 2018.