Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch
With the increasing expansion of the power grid, economic dispatch problems have received considerable attention. A multi-agent coordinated deep reinforcement learning algorithm is proposed to deal with distributed nonconvex economic dispatch problems. In the algorithm, agents run independent reinfo...
Uloženo v:
| Vydáno v: | IEEE transactions on power systems Ročník 38; číslo 1; s. 204 - 217 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
New York
IEEE
01.01.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Témata: | |
| ISSN: | 0885-8950, 1558-0679 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | With the increasing expansion of the power grid, economic dispatch problems have received considerable attention. A multi-agent coordinated deep reinforcement learning algorithm is proposed to deal with distributed nonconvex economic dispatch problems. In the algorithm, agents run independent reinforcement learning algorithms and update their local Q-functions with a newly defined joint reward. The double network structure is adopted to approximate the Q-function so that the offline trained model can be used online to provide recommended power outputs for time-varying demands in real-time. By introducing the reward network, the competition mechanism between the reward network and the target network is established to determine a progressively stable target value, which achieves coordination among agents and pledges the losses of the Q-networks to converge well. Theoretical analysis is given and case studies are conducted to prove the advantages compared with existing approaches. |
|---|---|
| AbstractList | With the increasing expansion of the power grid, economic dispatch problems have received considerable attention. A multi-agent coordinated deep reinforcement learning algorithm is proposed to deal with distributed nonconvex economic dispatch problems. In the algorithm, agents run independent reinforcement learning algorithms and update their local Q-functions with a newly defined joint reward. The double network structure is adopted to approximate the Q-function so that the offline trained model can be used online to provide recommended power outputs for time-varying demands in real-time. By introducing the reward network, the competition mechanism between the reward network and the target network is established to determine a progressively stable target value, which achieves coordination among agents and pledges the losses of the Q-networks to converge well. Theoretical analysis is given and case studies are conducted to prove the advantages compared with existing approaches. |
| Author | Lin, Zhiyun Yan, Gangfeng Shi, Xiasheng Ding, Lifu |
| Author_xml | – sequence: 1 givenname: Lifu orcidid: 0000-0003-1860-9992 surname: Ding fullname: Ding, Lifu email: dinglifu@zju.edu.cn organization: College of Electrical Engineering, Zhejiang University, Hangzhou, Zhejiang, China – sequence: 2 givenname: Zhiyun orcidid: 0000-0002-5523-4467 surname: Lin fullname: Lin, Zhiyun email: linzy@sustech.edu.cn organization: Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen, China – sequence: 3 givenname: Xiasheng orcidid: 0000-0001-9079-5705 surname: Shi fullname: Shi, Xiasheng email: shixiasheng@zju.edu.cn organization: School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, Jiangsu, China – sequence: 4 givenname: Gangfeng orcidid: 0000-0002-3598-4949 surname: Yan fullname: Yan, Gangfeng email: ygf@zju.edu.cn organization: College of Electrical Engineering, Zhejiang University, Hangzhou, Zhejiang, China |
| BookMark | eNp9kU1LxDAQhoMouH78Ab0UPGdN0k3THNf1E9YPdNVjSeN0jbRJTVNR8MebuuLBg6eBeeeZgWe20Lp1FhDao2RMKZGHi5vH27sxI4yNU8plzvgaGlHOc0wyIdfRiOQ5x7nkZBNtdd0LISSLwQh9LpRfQsAPqu4Bz1zTQjDBOIuPVAdPyWVfB4OnS7AhOQZok1swtnJeQzO05qC8NXaZTOul8yY8N0kMk2PTBW_KPsQNV85qZ9_gPTmJ1TVGD3Grgn7eQRuVqjvY_anb6P70ZDE7x_Prs4vZdI41kzxgJcocSlURxVMhOaU5TGgGVArFKpCEVVwSrQmnnKpJmTKQGQitU0WqchIb2-hgtbf17rWHLhQvrvc2niyY4IKKTGaTOJWvprR3XeehKrQJanARvDJ1QUkxuC6-XReD6-LHdUTZH7T1plH-439ofwUZAPgFpEjjZ0T6Bfi4jyQ |
| CODEN | ITPSEG |
| CitedBy_id | crossref_primary_10_1109_TSG_2025_3549723 crossref_primary_10_1016_j_ijepes_2024_109867 crossref_primary_10_1007_s13369_025_10181_2 crossref_primary_10_1016_j_dche_2025_100257 crossref_primary_10_1109_ACCESS_2024_3514362 crossref_primary_10_1109_TPWRS_2024_3395248 crossref_primary_10_1016_j_ijepes_2024_109980 crossref_primary_10_1109_TPWRS_2024_3419894 crossref_primary_10_1016_j_apenergy_2024_123611 crossref_primary_10_1007_s11432_024_4223_2 crossref_primary_10_1016_j_energy_2025_137895 crossref_primary_10_1080_00207721_2024_2448770 crossref_primary_10_1109_TII_2025_3567398 crossref_primary_10_3390_en17081878 crossref_primary_10_1016_j_epsr_2024_110711 crossref_primary_10_1109_TAES_2025_3541162 crossref_primary_10_1049_gtd2_13001 crossref_primary_10_1109_ACCESS_2023_3337118 crossref_primary_10_1109_TII_2024_3465601 crossref_primary_10_3390_app15158605 crossref_primary_10_1016_j_enbuild_2024_114410 crossref_primary_10_1109_TPWRS_2022_3221439 crossref_primary_10_3390_en18071837 crossref_primary_10_1080_21642583_2025_2467077 crossref_primary_10_1109_TPWRS_2023_3288039 crossref_primary_10_1109_TAC_2024_3384341 crossref_primary_10_1109_TIA_2025_3529675 |
| Cites_doi | 10.1109/TCYB.2019.2921475 10.1049/ip-c.1993.0074 10.1109/TNNLS.2018.2801880 10.1109/TNN.1998.712192 10.1109/TII.2013.2287807 10.1109/SFCS.2003.1238221 10.1109/TSP.2017.2669896 10.1016/j.epsr.2007.11.007 10.1007/978-981-13-8285-7_4 10.1109/TII.2019.2933443 10.1109/TSG.2020.3011739 10.1109/59.260861 10.1109/TPWRS.2020.2999890 10.1109/TII.2019.2908450 10.21236/ada333248 10.1109/TSG.2014.2363096 10.1109/TSG.2018.2833108 10.1109/MPE.2009.934876 10.1016/j.arcontrol.2008.03.004 10.1109/TSG.2021.3052998 10.1109/TSG.2021.3060027 10.1016/j.eneco.2016.11.014 10.1109/TSG.2017.2756041 10.1109/TPWRS.2006.889132 10.1109/SURV.2011.101911.00087 10.1109/TPWRS.2005.857924 10.1109/TPWRS.2003.814889 10.1109/TPWRS.2018.2889989 10.1109/TPWRS.2021.3070161 10.3390/en13010123 10.1007/978-3-319-92040-5_19 10.1016/j.ins.2010.02.015 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
| DBID | 97E RIA RIE AAYXX CITATION 7SP 7TB 8FD FR3 KR7 L7M |
| DOI | 10.1109/TPWRS.2022.3159825 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Electronics & Communications Abstracts Mechanical & Transportation Engineering Abstracts Technology Research Database Engineering Research Database Civil Engineering Abstracts Advanced Technologies Database with Aerospace |
| DatabaseTitle | CrossRef Civil Engineering Abstracts Engineering Research Database Technology Research Database Mechanical & Transportation Engineering Abstracts Advanced Technologies Database with Aerospace Electronics & Communications Abstracts |
| DatabaseTitleList | Civil Engineering Abstracts |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1558-0679 |
| EndPage | 217 |
| ExternalDocumentID | 10_1109_TPWRS_2022_3159825 9736677 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Natural Science Foundation of China grantid: 62173118 funderid: 10.13039/501100001809 |
| GroupedDBID | -~X .DC 0R~ 29I 3EH 4.4 5GY 5VS 6IK 85S 97E AAJGR AARMG AASAJ AAWTH ABAZT ABFSI ABQJQ ABVLG ACGFO ACGFS ACIWK ACKIV AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 E.L EBS EJD HZ~ H~9 ICLAB IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ O9- OCL P2P RIA RIE RNS TAE TN5 VH1 VJK AAYXX CITATION 7SP 7TB 8FD FR3 KR7 L7M |
| ID | FETCH-LOGICAL-c295t-a7b8ebaf0a53795118e416e197a2fe902f590cc05151a4b32e96e7cc3a0fb44b3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 34 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000922154400017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0885-8950 |
| IngestDate | Fri Jul 25 19:03:01 EDT 2025 Sat Nov 29 02:52:27 EST 2025 Tue Nov 18 22:32:09 EST 2025 Wed Aug 27 02:14:28 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 1 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c295t-a7b8ebaf0a53795118e416e197a2fe902f590cc05151a4b32e96e7cc3a0fb44b3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0001-9079-5705 0000-0003-1860-9992 0000-0002-3598-4949 0000-0002-5523-4467 |
| PQID | 2757176964 |
| PQPubID | 85441 |
| PageCount | 14 |
| ParticipantIDs | crossref_primary_10_1109_TPWRS_2022_3159825 crossref_citationtrail_10_1109_TPWRS_2022_3159825 proquest_journals_2757176964 ieee_primary_9736677 |
| PublicationCentury | 2000 |
| PublicationDate | 2023-Jan. 2023-1-00 20230101 |
| PublicationDateYYYYMMDD | 2023-01-01 |
| PublicationDate_xml | – month: 01 year: 2023 text: 2023-Jan. |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on power systems |
| PublicationTitleAbbrev | TPWRS |
| PublicationYear | 2023 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref35 ref12 ref34 ref14 ref36 ref31 Melo (ref33) 2001 ref30 ref11 ref10 ref32 ref2 ref1 ref39 ref38 ref19 ref18 Romoff (ref29) 2018 Sunehag (ref15) 2018; 3 Lauer (ref23) 2000 Lowe (ref17) 2017; 30 ref24 ref26 ref25 ref20 ref22 ref21 ref28 ref27 ref8 ref7 Park (ref37) 1998; 6 Rashid (ref16) 2018 ref9 ref4 ref3 ref6 ref5 |
| References_xml | – start-page: 674 volume-title: Proc. 2nd Conf. Robot Learn. year: 2018 ident: ref29 article-title: Reward estimation for variance reduction in deep reinforcement learning – ident: ref25 doi: 10.1109/TCYB.2019.2921475 – ident: ref3 doi: 10.1049/ip-c.1993.0074 – ident: ref24 doi: 10.1109/TNNLS.2018.2801880 – ident: ref34 doi: 10.1109/TNN.1998.712192 – ident: ref11 doi: 10.1109/TII.2013.2287807 – ident: ref30 doi: 10.1109/SFCS.2003.1238221 – volume: 3 start-page: 2085 volume-title: Proc. 17th Int. Conf. Autonomous Agents Multiagent Syst. year: 2018 ident: ref15 article-title: Value-decomposition networks for cooperative multi-agent learning based on team reward – ident: ref7 doi: 10.1109/TSP.2017.2669896 – ident: ref38 doi: 10.1016/j.epsr.2007.11.007 – volume: 30 volume-title: Adv. Neural Inf. Process. Syst. year: 2017 ident: ref17 article-title: Multi-agent actor-critic for mixed cooperative-competitive environments – ident: ref31 doi: 10.1007/978-981-13-8285-7_4 – ident: ref26 doi: 10.1109/TII.2019.2933443 – ident: ref19 doi: 10.1109/TSG.2020.3011739 – ident: ref4 doi: 10.1109/59.260861 – ident: ref18 doi: 10.1109/TPWRS.2020.2999890 – ident: ref10 doi: 10.1109/TII.2019.2908450 – ident: ref14 doi: 10.21236/ada333248 – ident: ref32 doi: 10.1109/TSG.2014.2363096 – ident: ref9 doi: 10.1109/TSG.2018.2833108 – ident: ref1 doi: 10.1109/MPE.2009.934876 – ident: ref6 doi: 10.1016/j.arcontrol.2008.03.004 – ident: ref20 doi: 10.1109/TSG.2021.3052998 – ident: ref21 doi: 10.1109/TSG.2021.3060027 – ident: ref28 doi: 10.1016/j.eneco.2016.11.014 – ident: ref8 doi: 10.1109/TSG.2017.2756041 – volume: 6 start-page: 103 issue: 2 year: 1998 ident: ref37 article-title: New approach to economic load dispatch based on improved evolutionary programming publication-title: Int. J. Eng. Intell. Syst. Elect. Eng. Commun. – ident: ref5 doi: 10.1109/TPWRS.2006.889132 – ident: ref2 doi: 10.1109/SURV.2011.101911.00087 – ident: ref35 doi: 10.1109/TPWRS.2005.857924 – ident: ref36 doi: 10.1109/TPWRS.2003.814889 – ident: ref12 doi: 10.1109/TPWRS.2018.2889989 – start-page: 535 volume-title: Proc. 17th Int. Conf. Mach. Learn. year: 2000 ident: ref23 article-title: An algorithm for distributed reinforcement learning in cooperative multi-agent systems – ident: ref27 doi: 10.1109/TPWRS.2021.3070161 – start-page: 4295 volume-title: Proc. Int. Conf. Mach. Learn. year: 2018 ident: ref16 article-title: QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning – start-page: 1 year: 2001 ident: ref33 article-title: Convergence of Q-learning: A simple proof – ident: ref22 doi: 10.3390/en13010123 – ident: ref13 doi: 10.1007/978-3-319-92040-5_19 – ident: ref39 doi: 10.1016/j.ins.2010.02.015 |
| SSID | ssj0006679 |
| Score | 2.5688422 |
| Snippet | With the increasing expansion of the power grid, economic dispatch problems have received considerable attention. A multi-agent coordinated deep reinforcement... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 204 |
| SubjectTerms | Algorithms Approximation algorithms Competition Deep learning Deep neural network distributed economic dispatch Estimation Heuristic algorithms Inference algorithms Machine learning multi-agent deep reinforcement learning Multiagent systems nonconvex optimization Observability Power dispatch Reinforcement learning Training |
| Title | Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch |
| URI | https://ieeexplore.ieee.org/document/9736677 https://www.proquest.com/docview/2757176964 |
| Volume | 38 |
| WOSCitedRecordID | wos000922154400017&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-0679 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0006679 issn: 0885-8950 databaseCode: RIE dateStart: 19860101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT9wwEB0B6qE9lAKtunxUPvTWujhObMfHBYo4rRBdWm6RPZkA0rK7WnYRB348tpNdFbWq1FsU21GUl4xnJjPvAXx2mGWkCuTos5itKjNeovI8d5KsLZw2ukliE2YwKK-u7PkafF31whBRKj6jb_Ew_cuvJ7iIqbJDa3KtjVmHdWN026u1srphwLYeo-KlVWLZICPs4fD818WPEApKGSLUSFinXmxCSVXlD1Oc9pfTzf-7s3fwtvMjWb8FfgvWaLwNb35jF9yBp2Gq8uY_3WhB_Dg5yKlAix-FratmqfeW92NvFTshmrILSjSqmDKGrGNevWb90fVkdju_uWNhkJ1Ept0okhWuMJiMU9X6I1v2N8fhabDuN-_h8vT78PiMd2ILHKVVc-6ML8m7RjiVGxvjDgq-GmXWONmQFbJRViBGSZjMFT4PUGoyiLkTjS_CiQ-wMZ6M6SMwiQYl5irL67IgzH1wY7SovfNNLaRXPciWT7_Cjok8CmKMqhSRCFslxKqIWNUh1oMvqzXTlofjn7N3IkarmR08Pdhfglx1n-p9JY0KIa22utj9-6o9eB015tu8yz5szGcLOoBX-DC_vZ99Sm_hMxMp3Fs |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fb9MwED6NDQn2sAEbomwDP_AGZo4Tx_Fj90tDjGoaBfYW2ZfLNqm0VddOPOyPn-2k1dDQJN6i2I6ifMn57nL3fQAfLCYJqQw5uiRkq4qEF6gcT60kYzKb67yOYhO61yvOz83pEnxa9MIQUSw-o8_hMP7Lr0Y4C6myXaPTPNf6CayoLJOi6dZa2F0_ZBqfUfHCKDFvkRFmt3_66-y7Dwal9DFqoKxTf21DUVflgTGOO8zR-v_d2wtYaz1J1m2gfwlLNHwFq_f4BTfgth_rvPlPO5gR348ucizR4nt-86pY7L7l3dBdxQ6IxuyMIpEqxpwha7lXL1h3cDGaXE0vfzM_yA4C126QyfJX6I2GsW79D5t3OIfhsbfvl5vw4-iwv3_MW7kFjtKoKbfaFeRsLaxKtQmRB3lvjRKjrazJCFkrIxCDKExiM5d6MHPSiKkVtcv8idewPBwN6Q0wiRolpipJqyIjTJ13ZHJROevqSkinOpDMn36JLRd5kMQYlDEmEaaMiJUBsbJFrAMfF2vGDRPHo7M3AkaLmS08Hdieg1y2H-t1KbXyQW1u8uztv1e9h2fH_W8n5cmX3tcteB4U55sszDYsTycz2oGneDO9up68i2_kHa1W36I |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Target-Value-Competition-Based+Multi-Agent+Deep+Reinforcement+Learning+Algorithm+for+Distributed+Nonconvex+Economic+Dispatch&rft.jtitle=IEEE+transactions+on+power+systems&rft.au=Ding%2C+Lifu&rft.au=Lin%2C+Zhiyun&rft.au=Shi%2C+Xiasheng&rft.au=Gangfeng+Yan&rft.date=2023-01-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=0885-8950&rft.eissn=1558-0679&rft.volume=38&rft.issue=1&rft.spage=204&rft_id=info:doi/10.1109%2FTPWRS.2022.3159825&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0885-8950&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0885-8950&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0885-8950&client=summon |