Policy Iteration Based Approximate Dynamic Programming Toward Autonomous Driving in Constrained Dynamic Environment

In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the driving scenarios are complex. Particularly, most of the existing methods cannot be generalized to dynamically changing scenarios with varying s...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	IEEE transactions on intelligent transportation systems Jg. 24; H. 5; S. 1 - 11
Hauptverfasser:	Lin, Ziyu, Ma, Jun, Duan, Jingliang, Li, Shengbo Eben, Ma, Haitong, Cheng, Bo, Lee, Tong Heng
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	New York IEEE 01.05.2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:	Algorithms approximate dynamic programming Approximation algorithms Autonomous driving Autonomous vehicles constrained optimization Dynamic programming Heuristic algorithms Iterative methods Motion planning Multilayers Neural networks Path planning Path tracking Planning reinforcement learning Roads Task analysis Tracking control Vehicle dynamics
ISSN:	1524-9050, 1558-0016
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the driving scenarios are complex. Particularly, most of the existing methods cannot be generalized to dynamically changing scenarios with varying surrounding vehicles. To address this problem, this development here investigates the framework of integrated decision and control. As part of the modules, static path planning determines the reference candidates ahead, and then the optimal path-tracking controller realizes the specific autonomous driving task. An innovative and effective constrained finite-horizon approximate dynamic programming (ADP) algorithm is herein presented to generate the desired control policy for effective path tracking. With the generalized policy neural network that maps from the state to the control input, the proposed algorithm preserves the high effectiveness for the motion planning problem towards changing driving environments with varying surrounding vehicles. Moreover, the algorithm attains the noteworthy advantage of alleviating the typically heavy computational loads with the mode of offline training and online execution. As a result of the utilization of multi-layer neural networks in conjunction with the actor-critic framework, the constrained ADP method is capable of handling complex and multidimensional scenarios. Finally, various simulations have been carried out to show that the constrained ADP algorithm is effective.
AbstractList	In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the driving scenarios are complex. Particularly, most of the existing methods cannot be generalized to dynamically changing scenarios with varying surrounding vehicles. To address this problem, this development here investigates the framework of integrated decision and control. As part of the modules, static path planning determines the reference candidates ahead, and then the optimal path-tracking controller realizes the specific autonomous driving task. An innovative and effective constrained finite-horizon approximate dynamic programming (ADP) algorithm is herein presented to generate the desired control policy for effective path tracking. With the generalized policy neural network that maps from the state to the control input, the proposed algorithm preserves the high effectiveness for the motion planning problem towards changing driving environments with varying surrounding vehicles. Moreover, the algorithm attains the noteworthy advantage of alleviating the typically heavy computational loads with the mode of offline training and online execution. As a result of the utilization of multi-layer neural networks in conjunction with the actor-critic framework, the constrained ADP method is capable of handling complex and multidimensional scenarios. Finally, various simulations have been carried out to show that the constrained ADP algorithm is effective.
Author	Duan, Jingliang Li, Shengbo Eben Ma, Haitong Cheng, Bo Ma, Jun Lee, Tong Heng Lin, Ziyu
Author_xml	– sequence: 1 givenname: Ziyu orcidid: 0000-0003-0532-0030 surname: Lin fullname: Lin, Ziyu organization: School of Vehicle and Mobility, Tsinghua University, Beijing, China – sequence: 2 givenname: Jun orcidid: 0000-0002-9405-8232 surname: Ma fullname: Ma, Jun organization: Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China – sequence: 3 givenname: Jingliang orcidid: 0000-0002-3697-1576 surname: Duan fullname: Duan, Jingliang organization: School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, China – sequence: 4 givenname: Shengbo Eben orcidid: 0000-0003-4923-3633 surname: Li fullname: Li, Shengbo Eben organization: School of Vehicle and Mobility, Tsinghua University, Beijing, China – sequence: 5 givenname: Haitong orcidid: 0000-0002-9943-0638 surname: Ma fullname: Ma, Haitong organization: School of Vehicle and Mobility, Tsinghua University, Beijing, China – sequence: 6 givenname: Bo orcidid: 0000-0002-2785-516X surname: Cheng fullname: Cheng, Bo organization: School of Vehicle and Mobility, Tsinghua University, Beijing, China – sequence: 7 givenname: Tong Heng surname: Lee fullname: Lee, Tong Heng organization: Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
BookMark	eNp9kEFPwyAYhomZidv0B5h4IPHcCbSF9ji3qUuWuMR6JkjZwrLCBDrdv5e6mRgPniDwPh-8zwD0jDUKgGuMRhij8q6aVy8jgkg6SknKclqcgT7O8yJBCNNetydZUqIcXYCB95t4muUY94Ff2q2WBzgPyomgrYH3wqsajnc7Zz91I4KC04MRjZZw6ezaiabRZg0r-yFcjLXBGtvY1sOp0_vuRhs4scYHJ7SJg37gmdlrZ02jTLgE5yux9erqtA7B68Osmjwli-fH-WS8SCQps5AUVJaICSZLgWtEZSyS1nUpFCGE1YVIiVSypCwv2eqtyKmqFcYppTXGhBVplg7B7XFurPLeKh_4xrbOxCc5KTBmhGUUxRQ7pqSz3ju14lKHbxVdhS3HiHeGeWeYd4b5yXAk8R9y56Ixd_iXuTkyWin1K48IpfEzX8pcinE
CODEN	ITISFG
CitedBy_id	crossref_primary_10_1016_j_robot_2024_104676 crossref_primary_10_1109_JAS_2024_124950 crossref_primary_10_1109_TSMC_2024_3417230 crossref_primary_10_3390_drones9080569 crossref_primary_10_1002_rnc_7659 crossref_primary_10_1109_TVT_2024_3399826 crossref_primary_10_1016_j_robot_2023_104616 crossref_primary_10_1109_TASE_2023_3347264 crossref_primary_10_1007_s11071_025_11393_9 crossref_primary_10_3390_electronics14071483 crossref_primary_10_1109_TIM_2025_3545523 crossref_primary_10_1007_s10489_023_05233_9 crossref_primary_10_3390_s24175746 crossref_primary_10_1109_LCSYS_2025_3547629 crossref_primary_10_1109_TSMC_2024_3412172 crossref_primary_10_1002_oca_3142 crossref_primary_10_3390_electronics13050936 crossref_primary_10_1016_j_enconman_2023_117753 crossref_primary_10_1007_s11071_024_09843_x crossref_primary_10_1016_j_neucom_2024_127835 crossref_primary_10_1007_s10489_024_05593_w crossref_primary_10_1109_LRA_2024_3381088 crossref_primary_10_3389_fnbot_2025_1549414 crossref_primary_10_1016_j_trc_2024_104654 crossref_primary_10_1016_j_eswa_2025_128994 crossref_primary_10_1109_TITS_2023_3336670 crossref_primary_10_1109_TSMC_2024_3392756
Cites_doi	10.1007/978-3-319-50815-3 10.1007/s12532-018-0139-4 10.1109/YAC.2016.7804933 10.1109/ICUS50048.2020.9274944 10.1109/TNNLS.2022.3165846 10.1109/TIE.2022.3153800 10.1109/TITS.2019.2916354 10.1109/TNNLS.2019.2900510 10.1109/IVWorkshops54471.2021.9669260 10.1504/IJVAS.2005.008237 10.1109/TITS.2021.3094215 10.1109/TITS.2022.3194571 10.1016/j.neucom.2021.04.134 10.1109/TITS.2020.3036984 10.1049/iet-its.2019.0249 10.1109/TSMC.1987.289329 10.1109/TITS.2020.3046646 10.1049/iet-its.2019.0317 10.1007/978-981-19-7784-8 10.1109/ChiCC.2014.6896497 10.1109/TCST.2010.2049203 10.1109/TNNLS.2022.3225090 10.1109/TITS.2015.2453404 10.1109/TCYB.2020.3032711 10.1109/TCST.2017.2753169 10.1109/TCYB.2022.3163816 10.1109/TNNLS.2021.3082568 10.1002/9780470182963 10.1038/497181a 10.1109/ICCCR49711.2021.9349412
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD FR3 JQ2 KR7 L7M L~C L~D
DOI	10.1109/TITS.2023.3237568
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database Engineering Research Database ProQuest Computer Science Collection Civil Engineering Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Civil Engineering Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Engineering Research Database Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Civil Engineering Abstracts
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	1558-0016
EndPage	11
ExternalDocumentID	10_1109_TITS_2023_3237568 10026660
Genre	orig-research
GrantInformation_xml	– fundername: NSF China grantid: U20A20334; 52072213; 52202487 – fundername: Tsinghua University-Toyota Joint Research Center for AI Technology of Automated Vehicle – fundername: National Key Research and Development Program of China; National Key R&D Program of China grantid: 2022YFB2502901 funderid: 10.13039/501100012166
GroupedDBID	-~X 0R~ 29I 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK ACNCT AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS HZ~ IFIPE IPLJI JAVBF LAI M43 O9- OCL P2P PQQKQ RIA RIE RNS AAYXX AETIX AGSQL AIBXA CITATION EJD H~9 ZY4 7SC 7SP 8FD FR3 JQ2 KR7 L7M L~C L~D
ID	FETCH-LOGICAL-c294t-86c907a7c9a1d06c0163dd9ae2227d8a32cec967597fb856ede11366d11278343
IEDL.DBID	RIE
ISICitedReferencesCount	35
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001054283300024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1524-9050
IngestDate	Sun Nov 30 04:33:09 EST 2025 Tue Nov 18 21:58:05 EST 2025 Sat Nov 29 06:35:02 EST 2025 Wed Aug 27 02:18:20 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	5
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c294t-86c907a7c9a1d06c0163dd9ae2227d8a32cec967597fb856ede11366d11278343
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-3697-1576 0000-0002-2785-516X 0000-0002-9943-0638 0000-0003-4923-3633 0000-0003-0532-0030 0000-0002-9405-8232
PQID	2811727460
PQPubID	75735
PageCount	11
ParticipantIDs	crossref_citationtrail_10_1109_TITS_2023_3237568 ieee_primary_10026660 proquest_journals_2811727460 crossref_primary_10_1109_TITS_2023_3237568
PublicationCentury	2000
PublicationDate	2023-05-01
PublicationDateYYYYMMDD	2023-05-01
PublicationDate_xml	– month: 05 year: 2023 text: 2023-05-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on intelligent transportation systems
PublicationTitleAbbrev	TITS
PublicationYear	2023
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref12 ref15 ref14 ref31 ref30 ref11 ref33 ref10 dixit (ref7) 2019; 21 ref2 ref1 guan (ref32) 2021 ref17 ref19 ref18 howard (ref23) 1960 powell (ref22) 2007 werbos (ref21) 1992 ref24 ref26 ref25 ref20 duan (ref16) 2022; 33 ref28 ref27 ref29 ref8 ref9 ref4 ref3 ref6 ref5
References_xml	– ident: ref25 doi: 10.1007/978-3-319-50815-3 – ident: ref33 doi: 10.1007/s12532-018-0139-4 – ident: ref27 doi: 10.1109/YAC.2016.7804933 – ident: ref24 doi: 10.1109/ICUS50048.2020.9274944 – ident: ref13 doi: 10.1109/TNNLS.2022.3165846 – ident: ref9 doi: 10.1109/TIE.2022.3153800 – volume: 21 start-page: 2310 year: 2019 ident: ref7 article-title: Trajectory planning for autonomous high-speed overtaking in structured environments using robust MPC publication-title: IEEE Trans Intell Transp Syst doi: 10.1109/TITS.2019.2916354 – ident: ref15 doi: 10.1109/TNNLS.2019.2900510 – ident: ref31 doi: 10.1109/IVWorkshops54471.2021.9669260 – ident: ref4 doi: 10.1504/IJVAS.2005.008237 – ident: ref12 doi: 10.1109/TITS.2021.3094215 – ident: ref11 doi: 10.1109/TITS.2022.3194571 – start-page: 493 year: 1992 ident: ref21 article-title: Approximate dynamic programming for real time control and neural modelling publication-title: Handbook of Intelligent Control Neural Fuzzy and Adaptive Approaches – ident: ref2 doi: 10.1016/j.neucom.2021.04.134 – ident: ref8 doi: 10.1109/TITS.2020.3036984 – ident: ref17 doi: 10.1049/iet-its.2019.0249 – ident: ref28 doi: 10.1109/TSMC.1987.289329 – ident: ref20 doi: 10.1109/TITS.2020.3046646 – ident: ref14 doi: 10.1049/iet-its.2019.0317 – year: 2021 ident: ref32 article-title: Mixed policy gradient publication-title: arXiv 2102 11513 – ident: ref19 doi: 10.1007/978-981-19-7784-8 – ident: ref26 doi: 10.1109/ChiCC.2014.6896497 – ident: ref3 doi: 10.1109/TCST.2010.2049203 – ident: ref29 doi: 10.1109/TNNLS.2022.3225090 – ident: ref5 doi: 10.1109/TITS.2015.2453404 – ident: ref10 doi: 10.1109/TCYB.2020.3032711 – ident: ref6 doi: 10.1109/TCST.2017.2753169 – ident: ref30 doi: 10.1109/TCYB.2022.3163816 – year: 1960 ident: ref23 publication-title: Dynamic Programming and Markov Processes – volume: 33 start-page: 6584 year: 2022 ident: ref16 article-title: Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors publication-title: IEEE Trans Neural Netw Learn Syst doi: 10.1109/TNNLS.2021.3082568 – year: 2007 ident: ref22 publication-title: Approximate Dynamic Programming Solving the Curses of Dimensionality doi: 10.1002/9780470182963 – ident: ref1 doi: 10.1038/497181a – ident: ref18 doi: 10.1109/ICCCR49711.2021.9349412
SSID	ssj0014511
Score	2.52796
Snippet	In the area of autonomous driving, it typically brings great difficulty in solving the motion planning problem since the vehicle model is nonlinear and the...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	1
SubjectTerms	Algorithms approximate dynamic programming Approximation algorithms Autonomous driving Autonomous vehicles constrained optimization Dynamic programming Heuristic algorithms Iterative methods Motion planning Multilayers Neural networks Path planning Path tracking Planning reinforcement learning Roads Task analysis Tracking control Vehicle dynamics
Title	Policy Iteration Based Approximate Dynamic Programming Toward Autonomous Driving in Constrained Dynamic Environment
URI	https://ieeexplore.ieee.org/document/10026660 https://www.proquest.com/docview/2811727460
Volume	24
WOSCitedRecordID	wos001054283300024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-0016 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014511 issn: 1524-9050 databaseCode: RIE dateStart: 20000101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA46POjBnxOnU3LwJHRr0585TrfhQMbACruVNkmhoJ2srfjn-17azoEoeCkt5IXQr03eS977PkJu7UAlAnfuzTRhhqNSmAc5Sw0eC4DbEkkgdaHwkz-fB8slXzTF6roWRimlk8_UAG_1Wb5ciQq3yoZIFwruNkTou77v1cVamyMDJNrS5KjMMbjptkeYlsmH4Sx8HqBO-MBmtu8irerWIqRVVX5MxXp9mR79c2TH5LBxJOmoRv6E7Kj8lBxs0QuekaIm_aUzzZwMANB7WLMkHSGP-GcGvqqi41qRni7qPK03MKShTqWlo6rEiodVVdDxOsONB5rlFBU-ta4EdNQaT76r5brkZToJHx6NRmTBEIw7pRF4AuLj2Bc8tqTpCXABbSl5rLBIVgaxzYQC2HwIPNIkcD0lFcrAeBIcNRTpsM9JJ1_l6oLQJIXgTsCFJdJJfDv23NRnwoZGlgMPPWK2bz0SDQM5Dvg10pGIySMEKkKgogaoHrnbmLzX9Bt_Ne4iMlsNa1B6pN9iGzV_aBExrLCFkNwzL38xuyL72Hud3dgnnXJdqWuyJz7KrFjf6I_vC3kc198
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fS8MwEA4yBfXBnxOnU_Pgk9CtTX_mcbqNDecYWGFvoU1SGOgmWyf--d6l3RyIgi-lhVwb-rXJXXL3fYTcupFOJa7c21nKLE9nMA5yllk8kQC3I9NImULhQTgcRuMxH5XF6qYWRmttks90A0_NXr6aySUulTWRLhTcbYjQt1E6yy_KtdabBki1ZehRmWdx219tYjo2b8b9-LmBSuENl7mhj8SqG9OQ0VX5MRibGaZ7-M--HZGD0pWkrQL7Y7Klpydkf4Ng8JQsCtpf2jfcyQABvYdZS9EWMol_TsBb1bRdaNLTUZGp9QaGNDbJtLS1zLHmYbZc0PZ8gksPdDKlqPFplCXgRivjzne9XJW8dDvxQ88qZRYsybiXW1EgIUJOQskTR9mBBCfQVYonGstkVZS4TGoALoTQI0sjP9BKoxBMoMBVQ5kO94xUprOpPic0zSC8k3BgqfLS0E0CPwuZdKGR48FFjdirty5kyUGOHX4VJhaxuUCgBAIlSqBq5G5t8l4QcPzVuIrIbDQsQKmR-gpbUf6jC8GwxhaC8sC--MXshuz24qeBGPSHj5dkD59U5DrWSSWfL_UV2ZEf-WQxvzYf4hflR9sq
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Policy+Iteration+Based+Approximate+Dynamic+Programming+Toward+Autonomous+Driving+in+Constrained+Dynamic+Environment&rft.jtitle=IEEE+transactions+on+intelligent+transportation+systems&rft.au=Lin%2C+Ziyu&rft.au=Ma%2C+Jun&rft.au=Duan%2C+Jingliang&rft.au=Shengbo+Eben+Li&rft.date=2023-05-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=1524-9050&rft.eissn=1558-0016&rft.volume=24&rft.issue=5&rft.spage=5003&rft_id=info:doi/10.1109%2FTITS.2023.3237568&rft.externalDBID=NO_FULL_TEXT
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1524-9050&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1524-9050&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1524-9050&client=summon