An Efficient Impulsive Adaptive Dynamic Programming Algorithm for Stochastic Systems

In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive "events," instead of two regular time indexes. Based on this general matrix, the polic...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on cybernetics Vol. 53; no. 9; pp. 5545 - 5559
Main Authors: Liang, Mingming, Wang, Yonghua, Liu, Derong
Format: Journal Article
Language:English
Published: United States IEEE 01.09.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Adaptive algorithms
Adaptive dynamic programming (ADP)
Aerospace electronics
Algorithms
Approximation algorithms
Control theory
Convergence
Dynamic programming
Heuristic algorithms
impulsive stochastic systems
Iterative methods
Markov processes
optimal control
Performance indices
policy iteration
Probability distribution
Stability analysis
Stochastic systems
transition matrix
ISSN:2168-2267, 2168-2275, 2168-2275
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive "events," instead of two regular time indexes. Based on this general matrix, the policy iteration-based impulsive adaptive dynamic programming (IADP) algorithm along with its variant, which is a more efficient IADP (EIADP) algorithm, are developed in order to solve the optimal impulsive control problems of discrete stochastic systems. Through analyzing the monotonicity, stability, and convergency properties of the obtained iterative value functions and control laws, it is proved that the IADP and EIADP algorithms both converge to the optimal impulsive performance index function. By dividing the whole impulsive policy into smaller pieces, the proposed EIADP algorithm updates the iterative policies in a "piece-by-piece" manner according to the actual hardware constraints. This feature of the EIADP method enables these ADP-based algorithms to be fully optimized to run on all "sizes" of computing devices including the ones with low memory spaces. A simulation experiment is conducted to validate the effectiveness of the present methods.
AbstractList In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive ``events,'' instead of two regular time indexes. Based on this general matrix, the policy iteration-based impulsive adaptive dynamic programming (IADP) algorithm along with its variant, which is a more efficient IADP (EIADP) algorithm, are developed in order to solve the optimal impulsive control problems of discrete stochastic systems. Through analyzing the monotonicity, stability, and convergency properties of the obtained iterative value functions and control laws, it is proved that the IADP and EIADP algorithms both converge to the optimal impulsive performance index function. By dividing the whole impulsive policy into smaller pieces, the proposed EIADP algorithm updates the iterative policies in a ``piece-by-piece'' manner according to the actual hardware constraints. This feature of the EIADP method enables these ADP-based algorithms to be fully optimized to run on all ``sizes'' of computing devices including the ones with low memory spaces. A simulation experiment is conducted to validate the effectiveness of the present methods.
In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive "events," instead of two regular time indexes. Based on this general matrix, the policy iteration-based impulsive adaptive dynamic programming (IADP) algorithm along with its variant, which is a more efficient IADP (EIADP) algorithm, are developed in order to solve the optimal impulsive control problems of discrete stochastic systems. Through analyzing the monotonicity, stability, and convergency properties of the obtained iterative value functions and control laws, it is proved that the IADP and EIADP algorithms both converge to the optimal impulsive performance index function. By dividing the whole impulsive policy into smaller pieces, the proposed EIADP algorithm updates the iterative policies in a "piece-by-piece" manner according to the actual hardware constraints. This feature of the EIADP method enables these ADP-based algorithms to be fully optimized to run on all "sizes" of computing devices including the ones with low memory spaces. A simulation experiment is conducted to validate the effectiveness of the present methods.In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns for all system states between two impulsive "events," instead of two regular time indexes. Based on this general matrix, the policy iteration-based impulsive adaptive dynamic programming (IADP) algorithm along with its variant, which is a more efficient IADP (EIADP) algorithm, are developed in order to solve the optimal impulsive control problems of discrete stochastic systems. Through analyzing the monotonicity, stability, and convergency properties of the obtained iterative value functions and control laws, it is proved that the IADP and EIADP algorithms both converge to the optimal impulsive performance index function. By dividing the whole impulsive policy into smaller pieces, the proposed EIADP algorithm updates the iterative policies in a "piece-by-piece" manner according to the actual hardware constraints. This feature of the EIADP method enables these ADP-based algorithms to be fully optimized to run on all "sizes" of computing devices including the ones with low memory spaces. A simulation experiment is conducted to validate the effectiveness of the present methods.
Author Liang, Mingming
Wang, Yonghua
Liu, Derong
Author_xml – sequence: 1
  givenname: Mingming
  orcidid: 0000-0002-9299-4465
  surname: Liang
  fullname: Liang, Mingming
  email: liangmingming@gdut.edu.cn
  organization: School of Automation, Guangdong University of Technology, Guangzhou, China
– sequence: 2
  givenname: Yonghua
  orcidid: 0000-0002-0051-7224
  surname: Wang
  fullname: Wang, Yonghua
  email: wangyonghua@gdut.edu.cn
  organization: School of Automation, Guangdong University of Technology, Guangzhou, China
– sequence: 3
  givenname: Derong
  orcidid: 0000-0003-3715-4778
  surname: Liu
  fullname: Liu, Derong
  email: derong@uic.edu
  organization: School of Automation, Guangdong University of Technology, Guangzhou, China
BackLink https://www.ncbi.nlm.nih.gov/pubmed/35380980$$D View this record in MEDLINE/PubMed
BookMark eNp9kU9vGyEQxVGVqPnTfICqUrVSL7nYBXbZhaPjpm2kSK0U55ATYtnBIVrABbaSv32x7PqQQ7jMCP0ew7x3gU588IDQR4LnhGDxdbV8uplTTOm8Joxzwd-hc0paPqO0YyfHvu3O0FVKL7gcXq4Ef4_OalZzLDg-R6uFr26NsdqCz9Wd20xjsn-hWgxqk3fNt61XzurqdwzrqJyzfl0txnWINj-7yoRYPeSgn1XKBXrYpgwufUCnRo0Jrg71Ej1-v10tf87uf_24Wy7uZ7oWNM8IqXsOGgO0PcaNadqGsoEyikXPFDGMqAGMEMBp3-ke10oNBjfdMPBBNb2oL9H1_t1NDH8mSFk6mzSMo_IQpiRp23QtY11LCvrlFfoSpujL7yTlrMbFM0YL9flATb2DQW6idSpu5X-_CkD2gI4hpQjmiBAsd7HIXSxyF4s8xFI03SuNtlllG3yOyo5vKj_tlRYAjpNExzAuK_0DQoCZUQ
CODEN ITCEB8
CitedBy_id crossref_primary_10_1109_TCYB_2025_3530951
crossref_primary_10_1007_s11071_024_09840_0
crossref_primary_10_1109_TSMC_2024_3421658
crossref_primary_10_1109_TCYB_2025_3531224
crossref_primary_10_1109_TCYB_2024_3366974
crossref_primary_10_1109_TSMC_2023_3318650
crossref_primary_10_1109_TFUZZ_2024_3402348
crossref_primary_10_1109_TCYB_2024_3491582
Cites_doi 10.1109/TIE.2020.3001840
10.1016/j.neunet.2009.08.009
10.1109/TCYB.2019.2896340
10.1080/17442508.2016.1197925
10.1109/TNNLS.2020.3041469
10.1007/978-3-319-50815-3
10.1239/aap/1427814583
10.1109/JAS.2021.1003814
10.1109/TSMC.2020.3042876
10.1137/18M1229365
10.1080/00207170110081705
10.1109/TCYB.2020.2970969
10.1109/TCYB.2019.2906694
10.1109/TIE.2020.2978699
10.1109/TSMCB.2012.2216523
10.1109/JAS.2017.7510739
10.1002/asjc.769
10.2514/1.G002848
10.2514/6.2013-4787
10.1109/TNNLS.2020.3021037
10.1142/0906
10.1016/j.ast.2020.106233
10.1109/TII.2020.2972383
10.1109/TNNLS.2013.2281663
10.1109/JAS.2016.7510262
10.1109/TNNLS.2013.2280013
10.1109/TCYB.2016.2586082
10.1016/j.ifacol.2018.11.428
10.1109/TCYB.2019.2957406
10.1109/LRA.2020.2978451
10.1109/TCYB.2015.2417170
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID 97E
ESBDL
RIA
RIE
AAYXX
CITATION
NPM
7SC
7SP
7TB
8FD
F28
FR3
H8D
JQ2
L7M
L~C
L~D
7X8
DOI 10.1109/TCYB.2022.3158898
DatabaseName IEEE Xplore (IEEE)
IEEE Xplore Open Access Journals
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
PubMed
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Mechanical & Transportation Engineering Abstracts
Technology Research Database
ANTE: Abstracts in New Technology & Engineering
Engineering Research Database
Aerospace Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
MEDLINE - Academic
DatabaseTitle CrossRef
PubMed
Aerospace Database
Technology Research Database
Computer and Information Systems Abstracts – Academic
Mechanical & Transportation Engineering Abstracts
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Engineering Research Database
Advanced Technologies Database with Aerospace
ANTE: Abstracts in New Technology & Engineering
Computer and Information Systems Abstracts Professional
MEDLINE - Academic
DatabaseTitleList PubMed
MEDLINE - Academic

Aerospace Database
Database_xml – sequence: 1
  dbid: NPM
  name: PubMed
  url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed
  sourceTypes: Index Database
– sequence: 2
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
– sequence: 3
  dbid: 7X8
  name: MEDLINE - Academic
  url: https://search.proquest.com/medline
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Sciences (General)
EISSN 2168-2275
EndPage 5559
ExternalDocumentID 35380980
10_1109_TCYB_2022_3158898
9750061
Genre orig-research
Journal Article
GrantInformation_xml – fundername: National Natural Science Foundation of China
  grantid: 62073085
  funderid: 10.13039/501100001809
– fundername: Basic and Applied Basic Research Foundation of Guangdong Province; Guangdong Basic and Applied Basic Research Foundation
  grantid: 2021A1515110870
  funderid: 10.13039/501100021171
GroupedDBID 0R~
4.4
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACIWK
AENEX
AGQYO
AGSQL
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
EJD
ESBDL
HZ~
IFIPE
IPLJI
JAVBF
M43
O9-
OCL
PQQKQ
RIA
RIE
RNS
AAYXX
CITATION
NPM
7SC
7SP
7TB
8FD
F28
FR3
H8D
JQ2
L7M
L~C
L~D
7X8
ID FETCH-LOGICAL-c392t-113b8ec0ee6b004f46425d25209b5a1f51adef99e82b7cb03aadf047dd8da4b93
IEDL.DBID RIE
ISICitedReferencesCount 12
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000779675100001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 2168-2267
2168-2275
IngestDate Thu Oct 02 10:38:57 EDT 2025
Sat Oct 04 23:56:54 EDT 2025
Mon Jul 21 05:45:54 EDT 2025
Sat Nov 29 02:02:36 EST 2025
Tue Nov 18 22:38:11 EST 2025
Wed Aug 27 02:18:15 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 9
Language English
License https://creativecommons.org/licenses/by/4.0/legalcode
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c392t-113b8ec0ee6b004f46425d25209b5a1f51adef99e82b7cb03aadf047dd8da4b93
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ORCID 0000-0002-0051-7224
0000-0002-9299-4465
0000-0003-3715-4778
OpenAccessLink https://ieeexplore.ieee.org/document/9750061
PMID 35380980
PQID 2853027552
PQPubID 85422
PageCount 15
ParticipantIDs proquest_journals_2853027552
proquest_miscellaneous_2647655761
ieee_primary_9750061
crossref_primary_10_1109_TCYB_2022_3158898
pubmed_primary_35380980
crossref_citationtrail_10_1109_TCYB_2022_3158898
PublicationCentury 2000
PublicationDate 2023-09-01
PublicationDateYYYYMMDD 2023-09-01
PublicationDate_xml – month: 09
  year: 2023
  text: 2023-09-01
  day: 01
PublicationDecade 2020
PublicationPlace United States
PublicationPlace_xml – name: United States
– name: Piscataway
PublicationTitle IEEE transactions on cybernetics
PublicationTitleAbbrev TCYB
PublicationTitleAlternate IEEE Trans Cybern
PublicationYear 2023
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref12
ref15
ref14
ref31
ref30
ref11
ref10
ref32
ref2
ref1
ref17
ref16
ref19
ref18
lawden (ref20) 1963
ref24
ref23
ref26
ref25
ref22
ref21
ref28
ref27
ref29
liu (ref5) 2013; 43
ref8
ref7
ref9
ref4
ref3
ref6
References_xml – ident: ref18
  doi: 10.1109/TIE.2020.3001840
– ident: ref31
  doi: 10.1016/j.neunet.2009.08.009
– ident: ref24
  doi: 10.1109/TCYB.2019.2896340
– ident: ref27
  doi: 10.1080/17442508.2016.1197925
– ident: ref15
  doi: 10.1109/TNNLS.2020.3041469
– ident: ref1
  doi: 10.1007/978-3-319-50815-3
– ident: ref26
  doi: 10.1239/aap/1427814583
– ident: ref14
  doi: 10.1109/JAS.2021.1003814
– ident: ref8
  doi: 10.1109/TSMC.2020.3042876
– ident: ref28
  doi: 10.1137/18M1229365
– year: 1963
  ident: ref20
  publication-title: Optimal Trajectories for Space Navigation
– ident: ref23
  doi: 10.1080/00207170110081705
– ident: ref10
  doi: 10.1109/TCYB.2020.2970969
– ident: ref29
  doi: 10.1109/TCYB.2019.2906694
– ident: ref16
  doi: 10.1109/TIE.2020.2978699
– volume: 43
  start-page: 779
  year: 2013
  ident: ref5
  article-title: Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems
  publication-title: IEEE Trans Cybern
  doi: 10.1109/TSMCB.2012.2216523
– ident: ref6
  doi: 10.1109/JAS.2017.7510739
– ident: ref32
  doi: 10.1002/asjc.769
– ident: ref19
  doi: 10.2514/1.G002848
– ident: ref21
  doi: 10.2514/6.2013-4787
– ident: ref30
  doi: 10.1109/TNNLS.2020.3021037
– ident: ref22
  doi: 10.1142/0906
– ident: ref12
  doi: 10.1016/j.ast.2020.106233
– ident: ref17
  doi: 10.1109/TII.2020.2972383
– ident: ref3
  doi: 10.1109/TNNLS.2013.2281663
– ident: ref7
  doi: 10.1109/JAS.2016.7510262
– ident: ref4
  doi: 10.1109/TNNLS.2013.2280013
– ident: ref11
  doi: 10.1109/TCYB.2016.2586082
– ident: ref25
  doi: 10.1016/j.ifacol.2018.11.428
– ident: ref9
  doi: 10.1109/TCYB.2019.2957406
– ident: ref13
  doi: 10.1109/LRA.2020.2978451
– ident: ref2
  doi: 10.1109/TCYB.2015.2417170
SSID ssj0000816898
Score 2.4024487
Snippet In this study, a novel general impulsive transition matrix is defined, which can reveal the transition dynamics and probability distribution evolution patterns...
SourceID proquest
pubmed
crossref
ieee
SourceType Aggregation Database
Index Database
Enrichment Source
Publisher
StartPage 5545
SubjectTerms Adaptive algorithms
Adaptive dynamic programming (ADP)
Aerospace electronics
Algorithms
Approximation algorithms
Control theory
Convergence
Dynamic programming
Heuristic algorithms
impulsive stochastic systems
Iterative methods
Markov processes
optimal control
Performance indices
policy iteration
Probability distribution
Stability analysis
Stochastic systems
transition matrix
Title An Efficient Impulsive Adaptive Dynamic Programming Algorithm for Stochastic Systems
URI https://ieeexplore.ieee.org/document/9750061
https://www.ncbi.nlm.nih.gov/pubmed/35380980
https://www.proquest.com/docview/2853027552
https://www.proquest.com/docview/2647655761
Volume 53
WOSCitedRecordID wos000779675100001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 2168-2275
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000816898
  issn: 2168-2267
  databaseCode: RIE
  dateStart: 20130101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Rb9MwED6NCaG9AGMwCmMyEg8DkdWx4zh-LGPTJqFp0goqT5FjO2xSm1Rtyu_n7LgRD4DEm6Wcnch3l7vznb8DeKeYyQxnOtE8z5JMFHWCRoEnaOxqaz0CergL8-2LvL4uZjN1swMfh7swzrlQfOZO_TDk8m1rNv6obKzQvFEf6zyQMu_vag3nKaGBRGh9y3CQoFchYxIzpWo8Pfv-CYNBxjBGFQXS7cEjjrpOlceD_M0ihRYrf_c2g9W5ePJ_3_sUHkfvkkx6cdiHHdc8g_2ov2tyEkGm3x_AdNKQ8wAfgUuQq8VyM_eF7GRi9dL_AcnnvlU9uekLuBZo4shk_qNd3Xd3C4KuLrntWnOnPc4zibjnz-Hrxfn07DKJHRYSg35Rl6QprwpnqHO5V986w2hEWOZLYyqh01qk2rpaKVewSpqKcq1tTTNpbWF1Vin-AnabtnEvgaSiUppxzayvVcy5Ni7ljipuM5VLyUdAt7tcmgg_7rtgzMsQhlBVeh6Vnkdl5NEIPgxTlj32xr-IDzwDBsK49yM42rKyjNq5LlkhQrpWsBG8HR6jXvlkiW5cu0GaPJO5wGgMlzjsRWBYeys5r_78ztew55vS95VoR7DbrTbuDTw0P7v79eoYhXdWHAfh_QUq5-a3
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3db9MwED9NA8FegDE-CgOMxAMgwhx_JPFjGZs2UapJFDSeIsd22KQ2qdqUv5-z40Y8ABJvlnJ2It9d7s53_h3AK8WMMJzpRPNMJEIWdYJGgSdo7GprPQJ6uAvzbZJPp8XlpbrYgXfDXRjnXCg-c-_9MOTybWs2_qjsSKF5oz7WuSGFYLS_rTWcqIQWEqH5LcNBgn5FHtOYKVVHs-PvHzAcZAyjVFkg3R7c4qjtVHlEyN9sUmiy8nd_M9id07v_98X34E70L8m4F4h92HHNfdiPGrwmryPM9JsDmI0bchIAJHAJcr5Ybua-lJ2MrV76fyD52DerJxd9CdcCjRwZz3-0q-vuakHQ2SVfutZcaY_0TCLy-QP4enoyOz5LYo-FxKBn1CVpyqvCGepc5hW4FhiPSMt8cUwldVrLVFtXK-UKVuWmolxrW1ORW1tYLSrFH8Ju0zbuMZBUVkozrpn11YoZ18al3FHFrVBZnvMR0O0ulyYCkPs-GPMyBCJUlZ5HpedRGXk0grfDlGWPvvEv4gPPgIEw7v0IDresLKN-rktWyJCwlWwEL4fHqFk-XaIb126QJhN5JjEewyUe9SIwrL2VnCd_fucLuH02-zwpJ-fTT09hz7eo7-vSDmG3W23cM7hpfnbX69XzIMK_AOF16RY
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Efficient+Impulsive+Adaptive+Dynamic+Programming+Algorithm+for+Stochastic+Systems&rft.jtitle=IEEE+transactions+on+cybernetics&rft.au=Liang%2C+Mingming&rft.au=Wang%2C+Yonghua&rft.au=Liu%2C+Derong&rft.date=2023-09-01&rft.issn=2168-2267&rft.eissn=2168-2275&rft.volume=53&rft.issue=9&rft.spage=5545&rft.epage=5559&rft_id=info:doi/10.1109%2FTCYB.2022.3158898&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TCYB_2022_3158898
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2168-2267&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2168-2267&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2168-2267&client=summon