Comparing Approaches for Prioritizing and Selecting Scenarios in Simulation-based Safety Testing of Automated Driving Systems
Keywords:
Automated Driving System (ADS), Simulation- based Testing, Safety Testing, Scenario Selection, Scenario PrioritizationAbstract
Simulation-based safety testing provides a cost-effective method for testing Automated Driving Systems (ADS) across diverse scenarios. However, prioritizing or selecting test scenarios for simulation-based safety testing remains challenging due to the infinite variety of scenarios that ADS may encounter. In this study, we conducted a literature review to identify approaches for selecting or prioritizing scenarios for ADS safety testing. We compare the six identified approaches in a tabular form across various aspects. We discuss one approach in detail, illustrating how it could complement the other selected approaches through an example. Our ongoing work involves extending the comparative analysis to cover all approaches comprehensively.
References
C.-Y. Chan, “Advancements, prospects, and impacts of automated driving systems,” International journal of transportation science and technology, vol. 6, no. 3, pp. 208–216, 2017.
B. Schoettle and M. Sivak, “A survey of public opinion about autonomous and self-driving vehicles in the us, the uk, and australia,” tech. rep., University of Michigan, Ann Arbor, Transportation Research Institute, 2014.
R. Pfeffer, G. N. Basedow, N. R. Thiesen, M. Spadinger, A. Albers, and E. Sax, “Automated driving-challenges for the automotive industry in product development with focus on process models and organizational structure,” in 2019 IEEE International Systems Conference (SysCon), pp. 1–6, IEEE, 2019.
X. Zhang, J. Tao, K. Tan, M. Torngren, J. M. G. S¨ anchez, M. R. Ramli,´ X. Tao, M. Gyllenhammar, F. Wotawa, N. Mohan, et al., “Finding critical scenarios for automated driving systems: A systematic literature review,” arXiv preprint arXiv:2110.08664, 2021.
F. Khan, M. Falco, H. Anwar, and D. Pfahl, “Safety testing of automated driving systems: A literature review,” IEEE Access, 2023.
E. Zablocki, H. Ben-Younes, P. P´ erez, and M. Cord, “Explainability of´ deep vision-based autonomous driving systems: Review and challenges,” International Journal of Computer Vision, vol. 130, no. 10, pp. 2425– 2452, 2022.
J. M. Anderson, K. Nidhi, K. D. Stanley, P. Sorensen, C. Samaras, and O. A. Oluwatola, Autonomous vehicle technology: A guide for policymakers. Rand Corporation, 2014.
SAE, “3016—taxonomy and definitions for terms related to on-road motor vehicle automated driving systems.” https://www.sae.org/standards/ content/j3016 202104/, 2021. Accessed: 15-07-2023.
K. Bimbraw, “Autonomous cars: Past, present and future a review of the developments in the last century, the present scenario and the expected future of autonomous vehicle technology,” in 2015 12th international conference on informatics in control, automation and robotics (ICINCO), vol. 1, pp. 191–198, IEEE, 2015.
Rodionova, I. Alvarez, M. S. Elli, F. Oboril, J. Quast, and R. Mangharam, “How safe is safe enough? automatic safety constraints boundary estimation for decision-making in automated vehicles,” in 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1457–1464, IEEE, 2020.
D. Sportillo, A. Paljic, and L. Ojeda, “On-road evaluation of autonomous driving training,” in 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 182–190, IEEE, 2019.
X. Zhao, V. Robu, D. Flynn, K. Salako, and L. Strigini, “Assessing the safety and reliability of autonomous vehicles from road testing,” in 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), pp. 13–23, IEEE, 2019.
S. Baltodano, S. Sibi, N. Martelaro, N. Gowda, and W. Ju, “The rrads platform: a real road autonomous driving simulator,” in Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 281–288, 2015.
Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in Conference on robot learning, pp. 1–16, PMLR, 2017.
G. Rong, B. H. Shin, H. Tabatabaee, Q. Lu, S. Lemke, M. Mozeiko,ˇ E. Boise, G. Uhm, M. Gerow, S. Mehta, et al., “Lgsvl simulator: A high fidelity simulator for autonomous driving,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–6, IEEE, 2020.
N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?,” Transportation Research Part A: Policy and Practice, vol. 94, pp. 182– 193, 2016.
T. D. Son, A. Bhave, and H. Van der Auweraer, “Simulation-based testing framework for autonomous driving development,” in 2019 IEEE International Conference on Mechatronics (ICM), vol. 1, pp. 576–583, IEEE, 2019.
D. J. Fremont, E. Kim, Y. V. Pant, S. A. Seshia, A. Acharya, X. Bruso, P. Wells, S. Lemke, Q. Lu, and S. Mehta, “Formal scenario-based testing of autonomous vehicles: From simulation to the real world,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–8, IEEE, 2020.
Z. Zhong, Y. Tang, Y. Zhou, V. d. O. Neves, Y. Liu, and B. Ray, “A survey on scenario-based testing for automated driving systems in highfidelity simulation,” arXiv preprint arXiv:2112.00964, 2021.
H. Elrofai, D. Worm, and O. Op den Camp, “Scenario identification for validation of automated driving functions,” in Advanced Microsystems for Automotive Applications 2016: Smart Systems for the Automobile of the Future, pp. 153–163, Springer, 2016.
S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and M. Maurer, “Defining and substantiating the terms scene, situation, and scenario for automated driving,” in 2015 IEEE 18th international conference on intelligent transportation systems, pp. 982–988, IEEE, 2015.
E. de Gelder, O. O. den Camp, and N. de Boer, “Scenario categories for the assessment of automated vehicles,” CETRAN, Singapore, Version, vol. 1, 2020.
S. Tang, Z. Zhang, Y. Zhang, J. Zhou, Y. Guo, S. Liu, S. Guo, Y.-F. Li, L. Ma, Y. Xue, et al., “A survey on automated driving system testing: Landscapes and trends,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 5, pp. 1–62, 2023.
G. Lou, Y. Deng, X. Zheng, M. Zhang, and T. Zhang, “Testing of autonomous driving systems: where are we and where should we go?,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 31–43, 2022.
W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on safety-critical driving scenario generation—a methodological perspective,” IEEE Transactions on Intelligent Transportation Systems, 2023.
K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic mapping studies in software engineering,” in 12th international conference on evaluation and assessment in software engineering (EASE), BCS Learning & Development, 2008.
F. Khan, H. Anwar, and D. Pfahl, “A process for scenario prioritization and selection in simulation-based safety testing of automated driving systems,” in International Conference on Product-Focused Software Process Improvement, (Austria), pp. 89–99, Springer, 2023.
C. Birchler, N. Ganz, S. Khatiri, A. Gambi, and S. Panichella, “Costeffective simulation-based test selection in self-driving cars software,” Science of Computer Programming, vol. 226, p. 102926, 2023.
C. Birchler, S. Khatiri, P. Derakhshanfar, S. Panichella, and A. Panichella, “Single and multi-objective test cases prioritization for self-driving cars in virtual environments,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 2, pp. 1–30, 2023.
Y. Deng, X. Zheng, M. Zhang, G. Lou, and T. Zhang, “Scenario-based test reduction and prioritization for multi-module autonomous driving systems,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 82–93, 2022.
C. Lu, H. Zhang, T. Yue, and S. Ali, “Search-based selection and prioritization of test scenarios for autonomous driving systems,” in International Symposium on Search Based Software Engineering, pp. 41–55, Springer, 2021.
J. Bach, J. Langner, S. Otten, E. Sax, and M. Holzapfel, “Test scenario¨ selection for system-level verification and validation of geolocationdependent automotive control systems,” in 2017 International Conference on Engineering, Technology and Innovation (ICE/ITMC), pp. 203– 210, IEEE, 2017.
Iso, “Pas 21448-road vehicles-safety of the intended functionality,” Int. Organization for Standardization, 2019.
Mercedes-Benz, “Safety first for automated driving (safad).” https://group.mercedes-benz.com/innovation/case/autonomous/ safety-first-for-automated-driving-2.html, 2019. Accessed: 19-072023.
Waymo, “Waymo safety report: On the road to fully self-driving.” https: //g.co/kgs/N6nC77u, 2018. Accessed: 18-07-2023.
E. Castellano, A. Cetinkaya, and P. Arcaini, “Analysis of road representations in search-based testing of autonomous driving systems,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), pp. 167–178, IEEE, 2021.
Gambi, M. Muller, and G. Fraser, “Asfault: Testing self-driving car¨ software using search-based procedural content generation,” in 2019 IEEE/ACM 41st International Conference on Software Engineering:Companion Proceedings (ICSE-Companion), pp. 27–30, IEEE, 2019.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Fauzia Khan, Hina Anwar, Dietmar Pfahl
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
License Terms:
Except where otherwise noted, content on this website is lincesed under a Creative Commons Attribution Non-Commercial License (CC BY NC)
Use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes, is permitted.
Copyright to any article published by WiPiEC retained by the author(s). Authors grant WiPiEC Journal a license to publish the article and identify itself as the original publisher. Authors also grant any third party the right to use the article freely as long as it is not used for commercial purposes and its original authors, citation details, and publisher are identified, in accordance with CC BY NC license. Fore more information on license terms, click here.