Identifying Underrepresented Driving Scenarios in Training Datasets with SafeML Robustness Monitoring for Autonomous Driving Vehicles

This study investigates the evaluation of the representativeness of datasets in autonomous driving systems using statistical distance metrics. Eight different datasets, created in the Carla simulation environment, cover various driving conditions, including normal and impaired lighting scenarios, al...

Full description

Saved in:
Bibliographic Details
Published in:2024 8th International Conference on System Reliability and Safety (ICSRS) pp. 245 - 253
Main Authors: Matthias, Bergler, Ramin, Tavakoli-Kolagari, Kristina, Lundqvist
Format: Conference Proceeding
Language:English
Published: IEEE 20.11.2024
Subjects:
Autonomous Driving
Autonomous vehicles
Lighting
Monitoring
Power measurement
Power system reliability
Real-time systems
Ro-bustness
Robustness
SafeML
Safety
Simulation
Time measurement
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract This study investigates the evaluation of the representativeness of datasets in autonomous driving systems using statistical distance metrics. Eight different datasets, created in the Carla simulation environment, cover various driving conditions, including normal and impaired lighting scenarios, alternative routes, and challenging conditions such as power outages. The datasets were designed to represent similar routes under different conditions. Various distance metrics- Wasserstein, Kuiper, Anderson-Darling, Chernoff, DTS, and CVM-were applied to measure pairwise dataset distances. We anticipated that the dataset for a given route under ideal conditions would exhibit a large distance measure (of any of the listed distance measures) compared to the same route under impaired conditions (e.g., a power failure at the streetlights). However, we were particularly interested in whether a measurable jump at a (potential threshold) value could be recognized even with a smaller drop in dataset condition quality. The results of the study show that a normalization of these distance measures enables precise divergence comparisons and the determination of meaningful threshold values. This in turn means that normalized deviation measures can effectively identify deviations in real time, hence contributing to the development and monitoring of more reliable autonomous driving models.
AbstractList This study investigates the evaluation of the representativeness of datasets in autonomous driving systems using statistical distance metrics. Eight different datasets, created in the Carla simulation environment, cover various driving conditions, including normal and impaired lighting scenarios, alternative routes, and challenging conditions such as power outages. The datasets were designed to represent similar routes under different conditions. Various distance metrics- Wasserstein, Kuiper, Anderson-Darling, Chernoff, DTS, and CVM-were applied to measure pairwise dataset distances. We anticipated that the dataset for a given route under ideal conditions would exhibit a large distance measure (of any of the listed distance measures) compared to the same route under impaired conditions (e.g., a power failure at the streetlights). However, we were particularly interested in whether a measurable jump at a (potential threshold) value could be recognized even with a smaller drop in dataset condition quality. The results of the study show that a normalization of these distance measures enables precise divergence comparisons and the determination of meaningful threshold values. This in turn means that normalized deviation measures can effectively identify deviations in real time, hence contributing to the development and monitoring of more reliable autonomous driving models.
Author Kristina, Lundqvist
Ramin, Tavakoli-Kolagari
Matthias, Bergler
Author_xml – sequence: 1
  givenname: Bergler
  orcidid: 0000-0002-2941-7948
  surname: Matthias
  fullname: Matthias, Bergler
  email: matthias.bergler@th-nuernberg.de
  organization: Nuremberg Institute of Technology,Department for Computer Science,Nuremberg,Germany
– sequence: 2
  givenname: Tavakoli-Kolagari
  orcidid: 0000-0002-7470-3767
  surname: Ramin
  fullname: Ramin, Tavakoli-Kolagari
  email: ramin.tavakolikolagari@th-nuernberg.de
  organization: Nuremberg Institute of Technology,Department for Computer Science,Nuremberg,Germany
– sequence: 3
  givenname: Lundqvist
  orcidid: 0000-0003-0904-3712
  surname: Kristina
  fullname: Kristina, Lundqvist
  email: kristina.lundqvist@mdu.se
  organization: Mälardalens University,Dependable Software Engineering,Västerås,Sweden
BookMark eNo9kM1OAjEUhWuiC0XewEVfAGzntjPTJQF_SCAmDLgld-itNIGWtIOGB_C9lfizOsl3Tr7FuWGXIQZijEsxlFKY--m4WTQlCFUOC1Go4TcrKg3mgvVNZWrQArTSor5mn1NLofPu5MMbXwVLKdEhUf6GZPkk-fdz0WwoYPIxcx_4MqEPZzrBDjN1mX_4bssbdDSf8UVsj7kLlDOfx-C7mM5TFxMfHbsY4j4e87_3lbZ-s6N8y64c7jL1f7PHVo8Py_HzYPbyNB2PZgMvQehBKVErC22JDtFAKWuJhdMgKxTCalKmrRBqEFhoCahKo9pWOwGVdRtjK-ixux-vJ6L1Ifk9ptP67x34AiBqYnY
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ICSRS63046.2024.10927539
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9798350354508
EndPage 253
ExternalDocumentID 10927539
Genre orig-research
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i1305-61a54d3b6afaa936181a2f5317a00d5e49b7a3830a2513a4694bb5f037dfc9d73
IEDL.DBID RIE
IngestDate Thu Apr 10 08:20:01 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i1305-61a54d3b6afaa936181a2f5317a00d5e49b7a3830a2513a4694bb5f037dfc9d73
ORCID 0000-0002-7470-3767
0000-0002-2941-7948
0000-0003-0904-3712
PageCount 9
ParticipantIDs ieee_primary_10927539
PublicationCentury 2000
PublicationDate 2024-Nov.-20
PublicationDateYYYYMMDD 2024-11-20
PublicationDate_xml – month: 11
  year: 2024
  text: 2024-Nov.-20
  day: 20
PublicationDecade 2020
PublicationTitle 2024 8th International Conference on System Reliability and Safety (ICSRS)
PublicationTitleAbbrev ICSRS
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.8905463
Snippet This study investigates the evaluation of the representativeness of datasets in autonomous driving systems using statistical distance metrics. Eight different...
SourceID ieee
SourceType Publisher
StartPage 245
SubjectTerms Autonomous Driving
Autonomous vehicles
Lighting
Monitoring
Power measurement
Power system reliability
Real-time systems
Ro-bustness
Robustness
SafeML
Safety
Simulation
Time measurement
Training
Title Identifying Underrepresented Driving Scenarios in Training Datasets with SafeML Robustness Monitoring for Autonomous Driving Vehicles
URI https://ieeexplore.ieee.org/document/10927539
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELVoxcAEiCK-5YE1xbKTuh5RSwUSVFVTULfKjs8iS4KahH_A_-bstEUMDGyRFcWSrcu7d77nR8gtgqJMIOORy0BHMTAXKSRwkVRMcKcUt7ENZhNyOh0ul2q2EasHLQwAhOYz6PvHcJZvy6zxpTKMcMUxvVYd0pFStmKtbXcOU3dPo3SeDvxRH_I-Hve3r_8yTgm4MTn854xHpPejwKOzHbYckz0oTshXq6oNyiQaDIvClZRePgSWjte5Lw7QNIMCCXBZ0bygi40DBB3rGvGqrqgvvNJUO3h5pvPSNFXtf3a0jW1f5KOYxtL7pvZqh7Kpdt99g_fQQtcjr5OHxegx2tgoRDkCVILkUCexFWagndZKDBDTNXcYe1IzZhOIlZEaiSrTmOsIjXw5NiZxTEjrMmWlOCXdoizgjFDODBND7QRiXywxeRFcYn5inDTSGZ2ck55fw9VHe1PGart8F3-MX5IDv1Ne28fZFenW6wauyX72WefV-ibs7zfh5Kq5
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PT8IwFG4UTfSkRoy_7cHrsLYdpUcDEohACEPDjbRbG3fZDNv8D_y_fe0A48GDt6XJ1qTN2_e-1_f1Q-geQFGEJqaBjY0KuCE2kEDgAiEJo1ZKmvDEm02IyaSzWMjpWqzutTDGGN98Zlru0Z_lJ3lcuVIZRLikkF7LXbQXck4fa7nWpj-HyIdhN5pFbXfYB8yP8tbmhV_WKR45-kf_nPMYNX80eHi6RZcTtGOyU_RV62q9Ngl7yyJ_KaUTEJkE91apKw_gKDYZUOC8wGmG52sPCNxTJSBWWWBXesWRsmY8wrNcV0Xpfne4jm5X5sOQyOKnqnR6h7wqtt99M---ia6JXvvP8-4gWBspBClAVAj0UIU8YbqtrFKStQHVFbUQfUIRkoSGSy0UUFWiINthChgz1zq0hInExjIR7Aw1sjwz5whTognrKMsA_biA9IVRARmKtkILq1V4gZpuDZcf9V0Zy83yXf4xfocOBvPxaDkaTl6u0KHbNaf0o-QaNcpVZW7QfvxZpsXq1u_1N-yprgA
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2024+8th+International+Conference+on+System+Reliability+and+Safety+%28ICSRS%29&rft.atitle=Identifying+Underrepresented+Driving+Scenarios+in+Training+Datasets+with+SafeML+Robustness+Monitoring+for+Autonomous+Driving+Vehicles&rft.au=Matthias%2C+Bergler&rft.au=Ramin%2C+Tavakoli-Kolagari&rft.au=Kristina%2C+Lundqvist&rft.date=2024-11-20&rft.pub=IEEE&rft.spage=245&rft.epage=253&rft_id=info:doi/10.1109%2FICSRS63046.2024.10927539&rft.externalDocID=10927539