How Can Simulation-based Safety Testing Help Understand the Real-World Safety of Autonomous Driving Systems?

Authors

  • Fauzia Khan Dept. of Computer Science, University of Tartu
  • Laima Dalbina Dept. of Computer Science, University of Tartu
  • Hina Anwar Dept. of Computer Science, University of Tartu
  • Dietmar Pfahl Dept. of Computer Science, University of Tartu

Keywords:

Autonomous Driving System (ADS), Human Driven Vehicle (HDV), Safety Testing, Scenario Generation, CARLA Simulator

Abstract

An Automated Driving System (ADS) requires exhaustive safety testing before receiving a road permit. Moreover, it is not clear what exactly constitutes sufficient safety for an ADS. One would assume that an ADS is safe enough if it is at least as safe as a Human Driven Vehicle (HDV). However, evaluating the safety of an ADS by comparing its behavior with that of a typical HDV in the real world is costly and risky. In this paper, we give an overview of our approach to compare the performance of ADS with HDV. While the overall approach is still in progress and ongoing, we provide a detailed approach utilizing established guidelines to systematically generate test scenarios specifically aimed at safety testing. Using our approach, various scenarios could be generated and tested, contributing to autonomous vehicles’ trustworthiness.

References

F. Duarte and C. Ratti, “The impact of autonomous vehicles on cities: A review,” Journal of Urban Technology, vol. 25, no. 4, 2018.

“Uk standards: Natural language description for abstract scenarios for automated driving systems,” Specification-BSI Flex 1889v1.0:202207,

Y. Zhu, J. Wang, F. Meng, and T. Liu, “Review on functional testing scenario library generation for connected and automated vehicles,” Sensors, vol. 22, no. 20, p. 7735, 2022.

W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on safety-critical driving scenario generation-a methodological perspective,” IEEE Transactions on Intelligent Transportation Systems, 2023.

H. Elrofai, D. Worm, and O. Op den Camp, “Scenario identification for validation of automated driving functions,” in Advanced Microsystems for Automotive Applications 2016: Smart Systems for the Automobile of the Future, Springer, 2016, pp. 153–163.

S. Geyer, M. Baltzer, B. Franz, S. Hakuli, M. Kauer, M. Kienle, S. Meier, T. Weißgerber, K. Bengler, R. Bruder, et al., “Concept and development of a unified ontology for generating test and usecase catalogues for assisted and automated vehicle guidance,” IET Intelligent Transport Systems, vol. 8, no. 3, pp. 183–189, 2014.

T. Menzel, G. Bagschik, and M. Maurer, “Scenarios for development, test and validation of automated vehicles,” in 2018 IEEE Intelligent Vehicles Symposium (IV), IEEE, 2018, pp. 1821–1827.

P. Kaur, S. Taghavi, Z. Tian, and W. Shi, “A survey on simulators for testing self-driving cars,” in Fourth International Conference on Connected and Autonomous Driving (MetroCAD), IEEE, 2021.

S. Park, S. Park, H. Jeong, I. Yun, and J. So, “Scenario-mining for level 4 automated vehicle safety assessment from real accident situations in urban areas using a natural language process,” Sensors, vol. 21, no. 20, p. 6929, 2021.

M. O’Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi, “Scalable end-to-end autonomous vehicle testing via rare-event simulation,” Advances in neural information processing systems, 2018.

S. Jha, T. Tsai, S. Hari, M. Sullivan, Z. Kalbarczyk, S. W. Keckler, and R. K. Iyer, “Kayotee: A fault injection-based system to assess the safety and reliability of autonomous vehicles to faults and errors,” arXiv preprint arXiv:1907.01024, 2019.

R. Ben Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, “Testing advanced driver assistance systems using multi-objective search and neural networks,” in IEEE/ACM international conference on automated software engineering, 2016.

M. Wicker, X. Huang, and M. Kwiatkowska, “Feature-guided black box safety testing of deep neural networks,” in 24th International Conference, Held as Part of the European Joint Conferences on Theory and Practice of Software, Greece, Springer, 2018.

Downloads

Published

2023-09-11

Issue

Vol. 9 No. 2 (2023): WiPiEC Journal

Section

Articles

License

Copyright (c) 2023 Fauzia Khan, Laima Dalbina, Hina Anwar, Dietmar Pfahl

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

License Terms: 

Except where otherwise noted, content on this website is lincesed under a  Creative Commons Attribution Non-Commercial License (CC BY NC) 

Use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes, is permitted.

Copyright to any article published by WiPiEC retained by the author(s). Authors grant WiPiEC Journal a license to publish the article and identify itself as the original publisher. Authors also grant any third party the right to use the article freely as long as it is not used for commercial purposes and its original authors, citation details, and publisher are identified, in accordance with CC BY NC license. Fore more information on license terms, click here.