An Adaptive Q-Learning Algorithm Developed for Agent-Based Computational Modeling of Electricity Market
Balancing between exploration and exploitation with adaptation of the Q -learning (QL) parameters to the condition of dynamic uncertain environment has always been a significant subject of interest in the context of reinforcement learning. The peculiarities of the electricity market have provided su...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on systems, man and cybernetics. Part C, Applications and reviews Jg. 40; H. 5; S. 547 - 556 |
|---|---|
| Hauptverfasser: | , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
New-York, NY
IEEE
01.09.2010
Institute of Electrical and Electronics Engineers |
| Schlagworte: | |
| ISSN: | 1094-6977, 1558-2442 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Balancing between exploration and exploitation with adaptation of the Q -learning (QL) parameters to the condition of dynamic uncertain environment has always been a significant subject of interest in the context of reinforcement learning. The peculiarities of the electricity market have provided such complex dynamic economic environment, and consequently have increased the requirement for advancement of the learning methods. In this economic system, the agent's market power plays a vital role in bidding decision-making problem. In order to improve the QL method, as main idea, adaptation of its parameters to the market power is proposed for making a good balance between exploration and exploitation. To implement this adaptation process, due to the fuzzy nature of human's decision-making process, a fuzzy system is designed to map each agent's market power into the QL parameters. Therefore, a fuzzy QL method is developed to model the power supplier's strategic bidding behavior in a computational electricity market. In the simulation framework, the QL algorithm selects the power supplier's bidding strategy according to the past experiences and the values of the parameters, which show the human's risk characteristic. The application of the proposed methodology for the power supplier in a multiarea power system shows the performance improvement in comparison to the QL with fixed parameters. |
|---|---|
| AbstractList | Balancing between exploration and exploitation with adaptation of the Q -learning (QL) parameters to the condition of dynamic uncertain environment has always been a significant subject of interest in the context of reinforcement learning. The peculiarities of the electricity market have provided such complex dynamic economic environment, and consequently have increased the requirement for advancement of the learning methods. In this economic system, the agent's market power plays a vital role in bidding decision-making problem. In order to improve the QL method, as main idea, adaptation of its parameters to the market power is proposed for making a good balance between exploration and exploitation. To implement this adaptation process, due to the fuzzy nature of human's decision-making process, a fuzzy system is designed to map each agent's market power into the QL parameters. Therefore, a fuzzy QL method is developed to model the power supplier's strategic bidding behavior in a computational electricity market. In the simulation framework, the QL algorithm selects the power supplier's bidding strategy according to the past experiences and the values of the parameters, which show the human's risk characteristic. The application of the proposed methodology for the power supplier in a multiarea power system shows the performance improvement in comparison to the QL with fixed parameters. |
| Author | Mashhadi, Habib Rajabi Rahimiyan, Morteza |
| Author_xml | – sequence: 1 givenname: Morteza surname: Rahimiyan fullname: Rahimiyan, Morteza email: morteza_rahimiyan@yahoo.com organization: Dept. of Electr. Eng., Ferdowsi Univ. of Mashhad, Mashhad, Iran – sequence: 2 givenname: Habib Rajabi surname: Mashhadi fullname: Mashhadi, Habib Rajabi email: h_mashhadi@um.ac.ir organization: Dept. of Electr. Eng., Ferdowsi Univ. of Mashhad, Mashhad, Iran |
| BackLink | http://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=23173729$$DView record in Pascal Francis |
| BookMark | eNp9kMFu1DAQQC1UJNrCD8AlFyQuKbbjxPYxpKVF2lVVUc7RxJ4sBm8cbG-l_j3Z7qqHHnoaj_SeR3pn5GQKExLykdELxqj-ev9z3XUXnC47p0IwKd6QU1bXquRC8JPlTbUoGy3lO3KW0h9KmRC6OiWbdipaC3N2D1jclSuEOLlpU7R-E6LLv7fFJT6gDzPaYgyxaDc45fIbpGXvwnbeZcguTOCLdbDo92oYiyuPJkdnXH4s1hD_Yn5P3o7gE344znPy6_vVfXdTrm6vf3TtqjS8obkUwiqkio8ClbAATIIdmNLKGitVDQbNwLHmw8CAN4Mcm8ZWqKUSaqRgTXVOvhz-nWP4t8OU-61LBr2HCcMu9YyrqtENrfSCfj6ikAz4McJkXOrn6LYQH3teMVlJvuf4gTMxpBRxfEYY7ff1-6f6_b5-f6y_SOqFtMR4SpUjOP-6-umgOkR8vlWLmmvJqv9mr5WC |
| CODEN | ITCRFH |
| CitedBy_id | crossref_primary_10_1016_j_epsr_2024_110404 crossref_primary_10_1007_s00202_025_03015_9 crossref_primary_10_1016_j_ijepes_2023_108954 crossref_primary_10_1007_s10846_015_0222_2 crossref_primary_10_1109_ACCESS_2022_3217497 crossref_primary_10_1002_acs_1220 crossref_primary_10_1016_j_arcontrol_2020_03_001 crossref_primary_10_1109_JIOT_2019_2899673 crossref_primary_10_1109_JSYST_2014_2329314 crossref_primary_10_1016_j_neucom_2024_128068 crossref_primary_10_1080_14697688_2024_2420609 crossref_primary_10_1109_TMECH_2019_2899365 crossref_primary_10_1177_1748006X19869750 crossref_primary_10_1016_j_apenergy_2025_126590 crossref_primary_10_1109_TSG_2019_2936142 crossref_primary_10_3390_en81212419 crossref_primary_10_1109_TASE_2023_3327264 crossref_primary_10_1049_iet_stg_2019_0129 crossref_primary_10_1016_j_cie_2021_107217 crossref_primary_10_1016_j_energy_2016_07_083 crossref_primary_10_1049_iet_rpg_2019_0786 crossref_primary_10_1007_s40314_022_01868_5 crossref_primary_10_1109_TAC_2016_2545106 crossref_primary_10_3390_pr8030368 crossref_primary_10_1016_j_engappai_2013_06_016 crossref_primary_10_1016_j_epsr_2014_06_001 crossref_primary_10_1109_TPWRS_2017_2688344 crossref_primary_10_1007_s00170_018_2690_6 crossref_primary_10_1109_TPWRS_2011_2144626 crossref_primary_10_1109_COMST_2019_2916177 crossref_primary_10_1016_j_apenergy_2014_02_004 crossref_primary_10_1016_j_renene_2020_08_089 crossref_primary_10_1109_TPWRS_2022_3173654 crossref_primary_10_1109_TSG_2015_2393059 crossref_primary_10_1109_TCYB_2016_2542923 crossref_primary_10_1016_j_epsr_2014_02_013 crossref_primary_10_1109_TNSM_2021_3049381 crossref_primary_10_1016_j_eneco_2025_108688 crossref_primary_10_1109_TPWRS_2018_2823641 crossref_primary_10_1016_j_rser_2023_113379 crossref_primary_10_1109_TSG_2011_2168244 crossref_primary_10_1016_j_apenergy_2017_03_121 crossref_primary_10_1007_s10994_013_5340_0 crossref_primary_10_1109_TSMC_2014_2373336 crossref_primary_10_1007_s00521_017_3106_5 crossref_primary_10_3390_en12152891 crossref_primary_10_1016_j_ifacol_2019_06_027 crossref_primary_10_1016_j_ifacol_2017_08_1217 crossref_primary_10_1109_TSG_2022_3214202 crossref_primary_10_1109_TSG_2012_2215349 crossref_primary_10_1515_itit_2019_0016 |
| Cites_doi | 10.1109/9780470545584 10.1109/5326.897075 10.1016/j.eneco.2008.01.003 10.1109/MIS.2003.1249170 10.1017/CBO9780511753985 10.1049/cp:20000400 10.1109/TSMCC.2007.913919 10.1109/TSMCC.2005.860575 10.1109/4235.956713 10.1109/ICCIAS.2006.294112 10.1016/j.epsr.2007.01.009 10.1109/59.982211 10.1109/TSMCA.2008.2001059 10.1016/j.ijepes.2006.03.002 10.1109/4235.956714 10.1541/ieejeiss.123.1134 10.1109/TSMCC.2004.843188 10.1109/TSMCC.2008.2001691 10.1109/TPWRS.2006.888977 10.1093/0199280290.001.0001 10.1613/jair.301 10.1109/TSMCC.2005.860578 10.1109/TSMCC.2007.913909 10.1109/TPWRS.2002.807041 10.1109/TSMCA.2005.854231 |
| ContentType | Journal Article |
| Copyright | 2015 INIST-CNRS |
| Copyright_xml | – notice: 2015 INIST-CNRS |
| DBID | 97E RIA RIE AAYXX CITATION IQODW 7SC 7SP 7TB 8FD F28 FR3 JQ2 L7M L~C L~D |
| DOI | 10.1109/TSMCC.2010.2044174 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Pascal-Francis Computer and Information Systems Abstracts Electronics & Communications Abstracts Mechanical & Transportation Engineering Abstracts Technology Research Database ANTE: Abstracts in New Technology & Engineering Engineering Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Mechanical & Transportation Engineering Abstracts Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Engineering Research Database Advanced Technologies Database with Aerospace ANTE: Abstracts in New Technology & Engineering Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Sciences (General) Applied Sciences Economics |
| EISSN | 1558-2442 |
| EndPage | 556 |
| ExternalDocumentID | 23173729 10_1109_TSMCC_2010_2044174 5452971 |
| Genre | orig-research |
| GroupedDBID | -~X 0R~ 29I 4.4 5VS 6IK 97E AAJGR AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFS AETIX AGQYO AGSQL AHBIQ AI. AIBXA ALLEH ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD F5P HZ~ H~9 IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL PZZ RIA RIE RNS VH1 AAYXX CITATION IQODW RIG 7SC 7SP 7TB 8FD F28 FR3 JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c260t-44d8e082f4e84daa17adb1898dcd785acecb2e52bb1a26b7f66d3e97848f0adc3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 68 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000283128300005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1094-6977 |
| IngestDate | Thu Sep 04 22:27:41 EDT 2025 Mon Jul 21 09:15:42 EDT 2025 Sat Nov 29 06:00:07 EST 2025 Tue Nov 18 22:31:04 EST 2025 Tue Aug 26 17:10:57 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | false |
| Issue | 5 |
| Keywords | Parameter estimation Adaptive algorithm Methodology Expert system Economic sciences Modeling Dynamic conditions Economic market Uncertain system Agent-based computational modeling Learning algorithm Human Q-learning (QL) Bidding electricity market Computer simulation Decision support system Decision making risk strategy Reinforcement learning Intelligent agent Fuzzy logic Multiagent system Electrical network Fuzzy decision Artificial intelligence Power |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html CC BY 4.0 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c260t-44d8e082f4e84daa17adb1898dcd785acecb2e52bb1a26b7f66d3e97848f0adc3 |
| Notes | ObjectType-Article-2 SourceType-Scholarly Journals-1 ObjectType-Feature-1 content type line 23 |
| PQID | 1283696039 |
| PQPubID | 23500 |
| PageCount | 10 |
| ParticipantIDs | crossref_primary_10_1109_TSMCC_2010_2044174 crossref_citationtrail_10_1109_TSMCC_2010_2044174 ieee_primary_5452971 pascalfrancis_primary_23173729 proquest_miscellaneous_1283696039 |
| PublicationCentury | 2000 |
| PublicationDate | 2010-09-01 |
| PublicationDateYYYYMMDD | 2010-09-01 |
| PublicationDate_xml | – month: 09 year: 2010 text: 2010-09-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | New-York, NY |
| PublicationPlace_xml | – name: New-York, NY |
| PublicationTitle | IEEE transactions on systems, man and cybernetics. Part C, Applications and reviews |
| PublicationTitleAbbrev | TSMCC |
| PublicationYear | 2010 |
| Publisher | IEEE Institute of Electrical and Electronics Engineers |
| Publisher_xml | – name: IEEE – name: Institute of Electrical and Electronics Engineers |
| References | ref13 ref12 ref15 ref14 ref30 ref11 ref32 ref10 watkins (ref24) 1989 tesfatsion (ref5) 2006 ref2 ref16 ref19 lau (ref1) 2008; 38 newbery (ref27) 2004 sutton (ref18) 1998 ref23 kaelbling (ref17) 1996; 4 ref25 ref20 david (ref28) 2000; 1 ref22 xin (ref33) 1996 ref21 lahlou (ref7) 2007 ref29 gomes-exposito (ref26) 2009 ref8 ref9 (ref31) 1992 ref4 ref3 ref6 |
| References_xml | – ident: ref30 doi: 10.1109/9780470545584 – ident: ref20 doi: 10.1109/5326.897075 – year: 2007 ident: ref7 article-title: multi-agent modelling of electricity markets: transaction processes and generation capacity expansion under competition – ident: ref22 doi: 10.1016/j.eneco.2008.01.003 – ident: ref6 doi: 10.1109/MIS.2003.1249170 – ident: ref29 doi: 10.1017/CBO9780511753985 – year: 2006 ident: ref5 publication-title: Handbook of Computational Economics Volume 2 Agent-Based Computational Economics – volume: 1 start-page: 242 year: 2000 ident: ref28 article-title: market power in generation markets publication-title: Proceedings of APSCOM 2000 - International Conference on Advances in Power System Control Operation and Management doi: 10.1049/cp:20000400 – ident: ref19 doi: 10.1109/TSMCC.2007.913919 – ident: ref4 doi: 10.1109/TSMCC.2005.860575 – ident: ref8 doi: 10.1109/4235.956713 – ident: ref14 doi: 10.1109/ICCIAS.2006.294112 – ident: ref13 doi: 10.1016/j.epsr.2007.01.009 – ident: ref32 doi: 10.1109/59.982211 – year: 1996 ident: ref33 publication-title: A Course in Fuzzy Systems and Control – year: 2004 ident: ref27 article-title: a review of the monitoring of market power – year: 1989 ident: ref24 article-title: learning from delayed rewards – volume: 38 start-page: 1210 year: 2008 ident: ref1 article-title: predicting interactions between agents in agent-based modeling and simulation of sociotechnical systems publication-title: IEEE Trans Syst Man Cybern A Syst Hum doi: 10.1109/TSMCA.2008.2001059 – year: 1992 ident: ref31 article-title: 1992 horizontal merger guidelines – ident: ref12 doi: 10.1016/j.ijepes.2006.03.002 – ident: ref9 doi: 10.1109/4235.956714 – ident: ref23 doi: 10.1541/ieejeiss.123.1134 – year: 1998 ident: ref18 publication-title: Reinforcement Learning An Introduction – ident: ref21 doi: 10.1109/TSMCC.2004.843188 – year: 2009 ident: ref26 publication-title: Electric Energy Systems Analysis and Operation – ident: ref11 doi: 10.1109/TSMCC.2008.2001691 – ident: ref16 doi: 10.1109/TPWRS.2006.888977 – ident: ref25 doi: 10.1093/0199280290.001.0001 – volume: 4 start-page: 237 year: 1996 ident: ref17 article-title: reinforcement learning: a survey publication-title: J Artif Intell Res doi: 10.1613/jair.301 – ident: ref2 doi: 10.1109/TSMCC.2005.860578 – ident: ref10 doi: 10.1109/TSMCC.2007.913909 – ident: ref15 doi: 10.1109/TPWRS.2002.807041 – ident: ref3 doi: 10.1109/TSMCA.2005.854231 |
| SSID | ssj0014493 |
| Score | 1.671253 |
| Snippet | Balancing between exploration and exploitation with adaptation of the Q -learning (QL) parameters to the condition of dynamic uncertain environment has always... |
| SourceID | proquest pascalfrancis crossref ieee |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 547 |
| SubjectTerms | Adaptation Agent-based computational modeling Algorithms Applied sciences Artificial intelligence Computer science; control theory; systems Decision making Dynamics Economics Electric power generation Electrical engineering. Electrical power engineering Electrical power engineering Electricity electricity market Electricity supply industry Environmental economics Exact sciences and technology Fuzzy systems Learning and adaptive systems Learning systems Markets Mathematical models Power generation economics Power networks and lines Power supplies Power system economics Power system modeling Power system simulation Q -learning (QL) risk strategy Simulation Software |
| Title | An Adaptive Q-Learning Algorithm Developed for Agent-Based Computational Modeling of Electricity Market |
| URI | https://ieeexplore.ieee.org/document/5452971 https://www.proquest.com/docview/1283696039 |
| Volume | 40 |
| WOSCitedRecordID | wos000283128300005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-2442 dateEnd: 20121231 omitProxy: false ssIdentifier: ssj0014493 issn: 1094-6977 databaseCode: RIE dateStart: 19980101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT9wwEB5R1AM9QHmJ5SUjcSgqhmzijeNjugL1UFCrAuIWOfZkWQkStA9-Px7HG4FaVeotUpwoyjf2fPY8PoDj2FFg5XgCR1VVXDhCwLUwMY-TNEuERqN9--K7H_L6Oru_Vz-X4LSrhUFEn3yGZ3TpY_m2MXM6KjsnPWxFBeMfpJRtrVYXMRBCtcn0SvDUkZpFgUykzm9-Xw2HbRZXHJHklnjnhLyqCuVE6qn7LVWrZ_HH0uz9zeXa_33pZ1gNvJLlrSGswxLWG_DpTbfBDVgP83jKvoRm0yebMMprllv9TKse-8VDu9URyx9HzWQ8e3hiIa0ILXP8luVUisW_Od9nWasIEU4TGamqUW07ayp24cV1xsZRfHbl66q34Pby4mb4nQfxBW7cFmfGhbAZOn5QCcyE1bovtS37mcqssTIbaIOmjHEQl2Vfx2kpqzS1Cbo9qciqSFuTbMNy3dS4A0yYlOKXRhqLYmBsiTaWiDKhGaqTQQ_6CzQKEzqTk0DGY-F3KJEqPIIFIVgEBHvwtXvmue3L8c_Rm4RRNzLA04PDd6B39x3pJfke1YOjhRUUbtpRLEXX2MynhXPrpIQYJWr37-_eg5U20YDS0fZheTaZ4wF8NC-z8XRy6G33FegQ7CM |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bi9QwFD4sq6A-qLurOF7WCD4oGrdN00se67DLijOD4ij7VtLkdBxY22Uu_n5z0kxxUQTfCk1L6XeS8yXn8gG8FI4CK8cTOKqm4dIRAq6lEVwkWZFIjUb79sXfJvlsVlxcqE978HaohUFEn3yG7-jSx_JtZ7Z0VHZCetiKCsZvpFKKuK_WGmIGUqo-nV5JnjlasyuRidTJ_Mt0PO7zuEREolvymhvyuiqUFanX7sc0vaLFH4uz9zhn9_7vW-_D3cAsWdmbwgHsYXsId37rN3gIB2Emr9mr0G769REsypaVVl_Rusc-89BwdcHKy0W3Wm6-_2AhsQgtcwyXlVSMxd8772dZrwkRzhMZ6apRdTvrGnbq5XWWxpF8NvWV1Q_g69npfHzOg_wCN26Ts-FS2gIdQ2gkFtJqHefa1nGhCmtsXqTaoKkFpqKuYy2yOm-yzCbodqWyaCJtTfIQ9tuuxUfApMkogmlyY1GmxtZoRY6YJzRHdZKOIN6hUZnQm5wkMi4rv0eJVOURrAjBKiA4gjfDM1d9Z45_jj4ijIaRAZ4RHF8DfbjvaC8J-KgRvNhZQeUmHkVTdIvddl05x05aiFGiHv_93c_h1vl8OqkmH2Yfn8DtPu2AktOewv5mtcVncNP83CzXq2Nvx78AUfnvag |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Adaptive+Q-Learning+Algorithm+Developed+for+Agent-Based+Computational+Modeling+of+Electricity+Market&rft.jtitle=IEEE+transactions+on+systems%2C+man+and+cybernetics.+Part+C%2C+Applications+and+reviews&rft.au=Rahimiyan%2C+Morteza&rft.au=Mashhadi%2C+Habib+Rajabi&rft.date=2010-09-01&rft.pub=IEEE&rft.issn=1094-6977&rft.volume=40&rft.issue=5&rft.spage=547&rft.epage=556&rft_id=info:doi/10.1109%2FTSMCC.2010.2044174&rft.externalDocID=5452971 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1094-6977&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1094-6977&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1094-6977&client=summon |