Emperror: A Flexible Generative Perception Error Model for Probing Self-Driving Planners

To handle the complexities of real-world traffic, learning planners for self-driving from data is a promising direction. While recent approaches have shown great progress, they typically assume a setting in which the ground-truth world state is available as input. However, when deployed, planning ne...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	IEEE robotics and automation letters Jg. 10; H. 6; S. 5807 - 5814
Hauptverfasser:	Hanselmann, Niklas, Doll, Simon, Cordts, Marius, Lensch, Hendrik P.A., Geiger, Andreas
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	Piscataway IEEE 01.06.2025 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:	autonomous agents Collision rates Complexity theory Decoding Deep learning methods Detectors Failure modes Heavily-tailed distribution Noise Noise measurement object detection Perception Planning segmentation and categorization Three-dimensional displays Trajectory Transformers
ISSN:	2377-3766, 2377-3766
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	To handle the complexities of real-world traffic, learning planners for self-driving from data is a promising direction. While recent approaches have shown great progress, they typically assume a setting in which the ground-truth world state is available as input. However, when deployed, planning needs to be robust to the long-tail of errors incurred by a noisy perception system, which is often neglected in evaluation. To address this, previous work has proposed drawing adversarial samples from a perception error model (PEM) mimicking the noise characteristics of a target object detector. However, these methods use simple PEMs that fail to accurately capture all failure modes of detection. In this letter, we present Emperror , a novel transformer-based generative PEM, apply it to stress-test an imitation learning (IL)-based planner and show that it imitates modern detectors more faithfully than previous work. Furthermore, it is able to produce realistic noisy inputs that increase the planner's collision rate by up to 85%, demonstrating its utility as a valuable tool for a more complete evaluation of self-driving planners.
AbstractList	To handle the complexities of real-world traffic, learning planners for self-driving from data is a promising direction. While recent approaches have shown great progress, they typically assume a setting in which the ground-truth world state is available as input. However, when deployed, planning needs to be robust to the long-tail of errors incurred by a noisy perception system, which is often neglected in evaluation. To address this, previous work has proposed drawing adversarial samples from a perception error model (PEM) mimicking the noise characteristics of a target object detector. However, these methods use simple PEMs that fail to accurately capture all failure modes of detection. In this letter, we present Emperror , a novel transformer-based generative PEM, apply it to stress-test an imitation learning (IL)-based planner and show that it imitates modern detectors more faithfully than previous work. Furthermore, it is able to produce realistic noisy inputs that increase the planner's collision rate by up to 85%, demonstrating its utility as a valuable tool for a more complete evaluation of self-driving planners.
Author	Hanselmann, Niklas Doll, Simon Lensch, Hendrik P.A. Cordts, Marius Geiger, Andreas
Author_xml	– sequence: 1 givenname: Niklas orcidid: 0000-0001-7387-4583 surname: Hanselmann fullname: Hanselmann, Niklas email: Niklas.Hanselmann@mercedes-benz.com organization: Mercedes-Benz AG R&D, Sindelfingen, Germany – sequence: 2 givenname: Simon orcidid: 0000-0001-7985-0830 surname: Doll fullname: Doll, Simon organization: Mercedes-Benz AG R&D, Sindelfingen, Germany – sequence: 3 givenname: Marius orcidid: 0000-0001-8729-9233 surname: Cordts fullname: Cordts, Marius organization: Mercedes-Benz AG R&D, Sindelfingen, Germany – sequence: 4 givenname: Hendrik P.A. orcidid: 0000-0003-3616-8668 surname: Lensch fullname: Lensch, Hendrik P.A. organization: University of Tübingen, Tübingen, Germany – sequence: 5 givenname: Andreas orcidid: 0000-0002-8151-3726 surname: Geiger fullname: Geiger, Andreas organization: University of Tübingen, Tübingen, Germany
BookMark	eNp9kE1Lw0AQhhepYK29e_Cw4Dl1P7K7WW-ltlWoWPwAb2E3mZUtaRI3adF_b0I9FA-e5h14nxl4ztGgrEpA6JKSCaVE36yepxNGmJhwIZlK9AkaMq5UxJWUg6N8hsZNsyGEUMEU12KI3ufbGkKowi2e4kUBX94WgJdQQjCt3wNeQ8igbn1V4nnfw49VDgV2XVqHyvryA79A4aK74Pf9si5M2cHNBTp1pmhg_DtH6G0xf53dR6un5cNsuooyLmUbOeaESYAp4_JcJso6ZqmLE5uJOIu5sirh0jqXJ1xRp23CWcxySIQQSmSx5SN0fbhbh-pzB02bbqpdKLuXKadaKa01j7sWObSyUDVNAJfWwW9N-E4pSXuFaacw7RWmvwo7RP5BMt-aXkQbjC_-A68OoAeAoz9aUUYk_wHad3_b
CODEN	IRALC6
CitedBy_id	crossref_primary_10_3390_e27090947
Cites_doi	10.1109/cvpr42600.2020.01164 10.1109/CVPR52729.2023.00928 10.1007/978-3-030-58574-7_19 10.1109/CVPR52688.2022.01679 10.1109/CVPR.2016.90 10.1109/CVPR.2018.00798 10.1109/ICRA48891.2023.10161501 10.1109/ITSC.2018.8570015 10.1007/978-3-030-58523-5_40 10.1561/0600000079 10.1007/978-3-031-20077-9_1 10.1109/IROS45743.2020.9340696 10.1109/CVPR52729.2023.01712 10.1016/j.artint.2021.103500 10.32657/10356/154942 10.1109/CVPR52733.2024.01395 10.1007/978-3-031-19839-7_20 10.1109/CVPR46437.2021.01026 10.1109/CVPR46437.2021.00978 10.1109/CVPR46437.2021.01161 10.1109/ICCV51070.2023.00335 10.1007/978-3-030-58452-8_13 10.1109/ICCV51070.2023.00766 10.24963/ijcai.2020/483
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/LRA.2025.3562789
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	2377-3766
EndPage	5814
ExternalDocumentID	10_1109_LRA_2025_3562789 10971206
Genre	orig-research
GrantInformation_xml	– fundername: German Federal Ministry for Economic Affairs and Climate Action – fundername: ERC Starting Grant through LEGO-3D grantid: 850533
GroupedDBID	0R~ 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c366t-f2f5a8e27afdd687bf2b1f48bc54c437b7836bffd8371f9b83242de855575c4b3
IEDL.DBID	RIE
ISICitedReferencesCount	1
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001479448500016&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2377-3766
IngestDate	Mon Jun 30 07:44:25 EDT 2025 Sat Nov 29 07:57:47 EST 2025 Tue Nov 18 21:39:27 EST 2025 Wed Aug 27 02:03:23 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	6
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c366t-f2f5a8e27afdd687bf2b1f48bc54c437b7836bffd8371f9b83242de855575c4b3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0003-3616-8668 0000-0001-7387-4583 0000-0001-8729-9233 0000-0001-7985-0830 0000-0002-8151-3726
PQID	3197799934
PQPubID	4437225
PageCount	8
ParticipantIDs	crossref_primary_10_1109_LRA_2025_3562789 crossref_citationtrail_10_1109_LRA_2025_3562789 ieee_primary_10971206 proquest_journals_3197799934
PublicationCentury	2000
PublicationDate	2025-06-01
PublicationDateYYYYMMDD	2025-06-01
PublicationDate_xml	– month: 06 year: 2025 text: 2025-06-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE robotics and automation letters
PublicationTitleAbbrev	LRA
PublicationYear	2025
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref35 ref12 ref37 Filos (ref11) 2020 ref14 ref36 Caesar (ref9) 2021 ref31 ref30 ref32 Deasy (ref33) 2020 ref1 Hu (ref6) 2021 ref17 ref16 ref38 ref18 Ju (ref19) 2023 Montali (ref10) 2023; 36 Sadeghi (ref13) 2023 ref24 Wang (ref2) 2021 ref23 ref26 ref25 ref20 ref42 ref22 ref21 Sadeghi (ref15) 2021 Sohn (ref28) 2015 Renz (ref7) 2022 ref27 Higgins (ref34) 2017 Kingma (ref41) 2015 Ba (ref39) 2016 ref3 Dauner (ref8) 2023 Kingma (ref29) 2014 ref5 Clevert (ref40) 2015 Li (ref4) 2023
References_xml	– ident: ref32 doi: 10.1109/cvpr42600.2020.01164 – ident: ref23 doi: 10.1109/CVPR52729.2023.00928 – start-page: 459 volume-title: Proc. Conf. Robot Learn. year: 2022 ident: ref7 article-title: Plant: Explainable planning transformers via object-level representations – start-page: 10647 volume-title: Proc. Neural Inf. Process. Syst. year: 2020 ident: ref33 article-title: Constraining variational inference with geometric Jensen-Shannon divergence – ident: ref18 doi: 10.1007/978-3-030-58574-7_19 – year: 2023 ident: ref19 article-title: Perception imitation: Towards synthesis-free simulator for autonomous vehicles – ident: ref25 doi: 10.1109/CVPR52688.2022.01679 – year: 2015 ident: ref40 article-title: Fast and accurate deep network learning by exponential linear units (ELUS) – ident: ref38 doi: 10.1109/CVPR.2016.90 – ident: ref20 doi: 10.1109/CVPR.2018.00798 – year: 2016 ident: ref39 article-title: Layer normalization – ident: ref12 doi: 10.1109/ICRA48891.2023.10161501 – volume-title: Proc. Neural Inf. Process. Syst. Workshops year: 2021 ident: ref15 article-title: A step towards efficient evaluation of complex perception tasks in simulation – volume-title: Proc. Int. Conf. Learn. Representations year: 2015 ident: ref41 article-title: Adam: A method for stochastic optimization – ident: ref16 doi: 10.1109/ITSC.2018.8570015 – ident: ref5 doi: 10.1007/978-3-030-58523-5_40 – ident: ref1 doi: 10.1561/0600000079 – ident: ref30 doi: 10.1007/978-3-031-20077-9_1 – ident: ref22 doi: 10.1109/IROS45743.2020.9340696 – volume-title: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops year: 2021 ident: ref9 article-title: NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles – ident: ref35 doi: 10.1109/CVPR52729.2023.01712 – volume: 36 start-page: 59151 volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2023 ident: ref10 article-title: The Waymo Open Sim Agents Challenge – ident: ref42 doi: 10.1016/j.artint.2021.103500 – ident: ref17 doi: 10.32657/10356/154942 – ident: ref37 doi: 10.1109/CVPR52733.2024.01395 – ident: ref26 doi: 10.1007/978-3-031-19839-7_20 – ident: ref31 doi: 10.1109/CVPR46437.2021.01026 – year: 2023 ident: ref13 article-title: Attacking motion planners using adversarial perception errors – ident: ref24 doi: 10.1109/CVPR46437.2021.00978 – volume-title: Proc. Int. Conf. Learn. Representations year: 2014 ident: ref29 article-title: Auto-encoding variational bayes – start-page: 15273 volume-title: Proc. IEEE/CVF Int. Conf. Comput. Vis. year: 2021 ident: ref6 article-title: FIERY: Future instance segmentation in bird’s-eye view from surround monocular cameras – start-page: 3145 volume-title: Proc. Int. Conf. Mach. Learn. year: 2020 ident: ref11 article-title: Can autonomous vehicles identify, recover from, and adapt to distribution shifts? – start-page: 180 volume-title: Proc. Conf. Robot Learn. year: 2021 ident: ref2 article-title: Detr3d: 3 d object detection from multi-view images via 3D-to-2D queries – year: 2023 ident: ref4 article-title: Graph-based topology reasoning for driving scenes – ident: ref21 doi: 10.1109/CVPR46437.2021.01161 – ident: ref3 doi: 10.1109/ICCV51070.2023.00335 – start-page: 1268 volume-title: Proc. Conf. Robot Learn. year: 2023 ident: ref8 article-title: Parting with misconceptions about learning-based vehicle motion planning – start-page: 3483 volume-title: Proc. Neural Inf. Process. Syst. year: 2015 ident: ref28 article-title: Learning structured output representation using deep conditional generative models – ident: ref27 doi: 10.1007/978-3-030-58452-8_13 – ident: ref36 doi: 10.1109/ICCV51070.2023.00766 – ident: ref14 doi: 10.24963/ijcai.2020/483 – volume-title: Proc. Int. Conf. Learn. Representations year: 2017 ident: ref34 article-title: beta-VAE: Learning basic visual concepts with a constrained variational framework
SSID	ssj0001527395
Score	2.3069048
Snippet	To handle the complexities of real-world traffic, learning planners for self-driving from data is a promising direction. While recent approaches have shown...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	5807
SubjectTerms	autonomous agents Collision rates Complexity theory Decoding Deep learning methods Detectors Failure modes Heavily-tailed distribution Noise Noise measurement object detection Perception Planning segmentation and categorization Three-dimensional displays Trajectory Transformers
Title	Emperror: A Flexible Generative Perception Error Model for Probing Self-Driving Planners
URI	https://ieeexplore.ieee.org/document/10971206 https://www.proquest.com/docview/3197799934
Volume	10
WOSCitedRecordID	wos001479448500016&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE/IET Electronic Library (IEL) (UW System Shared) customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2377-3766 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001527395 issn: 2377-3766 databaseCode: M~E dateStart: 20160101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELagYoCBZxGFUnlgYUhb5-UcWwWtGEpV8VK3KHZsVClKUdoy8tvxOWmphEBiy3COorvY953t-z5CrkIQoP2EORAJlDDTyklAcEcrZbI5k4xZMp3XIR-NoskExlWzuu2FUUrZy2eqjY_2LD-dySVulXXwtJS5SLC9zXlYNmt9b6gglRgEq6PILnSGjz1TALpB2zNJnqOQ-0bqsVoqPxZgm1UGB__8nkOyX8FH2ivjfUS2VH5M9jZIBU_IpG-QcFHMihvaowPkuxSZoiW9NK5tdLy-y0L7aEdRDy2jBr3SMbIy5W_0SWXauSumuNtAra6RQYl18jLoP9_eO5V-giO9MFw42tVBEimXJzpNw4gL7Qqm_UjIwJe-xwV2cAitU1OkMg0iQnCVqigIDIaTvvBOSS2f5eqMUOM9kBB0pclePkSQQJgCmPnLkN6QqQbprFwby4pcHDUustgWGV2ITTBiDEZcBaNBrtcj3ktijT9s6-j8DbvS7w3SXIUvrqbePDZrCucG9nr--S_DLsguvr288NUktUWxVJdkR34spvOiRbYfPvst-299AbBBy2M
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PT8IwFG4MmqgHf2JEUXvw4mFA96utN6IQjJMQRcNtWbvWkCxgBvj329cNJDGaeNvhNVveW_u-1_Z9H0LXIRdc-wlxOBMgYaaVk3BBHa2UyeZEEmLJdN4i2u-z0YgPymZ12wujlLKXz1QDHu1ZfjqVC9gqa8JpKXGBYHsTpLPKdq3vLRUgE-PB8jCyxZvRc9uUgG7Q8EyapyDlvpZ8rJrKjyXY5pXu_j-_6ADtlQASt4uIH6INNTlCu2u0gsdo1DFYOM-n-S1u4y4wXopM4YJgGlY3PFjdZsEdsMOgiJZhg1_xAHiZJu_4RWXauc_HsN-ArbKRwYlV9NrtDO96Tqmg4EgvDOeOdnWQMOXSRKdpyKjQriDaZ0IGvvQ9KqCHQ2idmjKVaC4YwKtUsSAwKE76wjtBlcl0ok4RNt7jkgctafKXzxlPeJhybmYwAYJDomqouXRtLEt6cVC5yGJbZrR4bIIRQzDiMhg1dLMa8VFQa_xhWwXnr9kVfq-h-jJ8cTn5ZrFZVSg1wNfzz34ZdoW2e8OnKI4e-o_naAfeVFz_qqPKPF-oC7QlP-fjWX5p_7AvombNew
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Emperror%3A+A+Flexible+Generative+Perception+Error+Model+for+Probing+Self-Driving+Planners&rft.jtitle=IEEE+robotics+and+automation+letters&rft.au=Hanselmann%2C+Niklas&rft.au=Doll%2C+Simon&rft.au=Cordts%2C+Marius&rft.au=Lensch%2C+Hendrik+P.A.&rft.date=2025-06-01&rft.pub=IEEE&rft.eissn=2377-3766&rft.volume=10&rft.issue=6&rft.spage=5807&rft.epage=5814&rft_id=info:doi/10.1109%2FLRA.2025.3562789&rft.externalDocID=10971206
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2377-3766&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2377-3766&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2377-3766&client=summon