Optimal Control of Microgrids with Multi-stage Mixed-integer Nonlinear Programming Guided $Q$-learning Algorithm

This paper proposes an energy management system (EMS) for the real-time operation of a pilot stochastic and dynamic microgrid on a university campus in Malta consisting of a diesel generator, photovoltaic panels, and batteries. The objective is to minimize the total daily operation costs, which incl...

Full description

Saved in:
Bibliographic Details
Published in:Journal of modern power systems and clean energy Vol. 8; no. 6; pp. 1151 - 1159
Main Authors: Yeliz Yoldas, Selcuk Goren, Ahmet Onen
Format: Journal Article
Language:English
Published: IEEE 01.11.2020
Subjects:
Cost minimization
energy management system
microgrid
real-time optimization
reinforcement learning
ISSN:2196-5420
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract This paper proposes an energy management system (EMS) for the real-time operation of a pilot stochastic and dynamic microgrid on a university campus in Malta consisting of a diesel generator, photovoltaic panels, and batteries. The objective is to minimize the total daily operation costs, which include the degradation cost of batteries, the cost of energy bought from the main grid, the fuel cost of the diesel generator, and the emission cost. The optimization problem is modeled as a finite Markov decision process (MDP) by combining network and technical constraints, and Q-learning algorithm is adopted to solve the sequential decision subproblems. The proposed algorithm decomposes a multi-stage mixed-integer nonlinear programming (MINLP) problem into a series of single-stage problems so that each subproblem can be solved by using Bellman's equation. To prove the effectiveness of the proposed algorithm, three case studies are taken into consideration: 1 minimizing the daily energy cost; 2 minimizing the emission cost; 3 minimizing the daily energy cost and emission cost simultaneously. Moreover, each case is operated under different battery operation conditions to investigate the battery lifetime. Finally, performance comparisons are carried out with a conventional Q-learning algorithm.
AbstractList This paper proposes an energy management system (EMS) for the real-time operation of a pilot stochastic and dynamic microgrid on a university campus in Malta consisting of a diesel generator, photovoltaic panels, and batteries. The objective is to minimize the total daily operation costs, which include the degradation cost of batteries, the cost of energy bought from the main grid, the fuel cost of the diesel generator, and the emission cost. The optimization problem is modeled as a finite Markov decision process (MDP) by combining network and technical constraints, and Q-learning algorithm is adopted to solve the sequential decision subproblems. The proposed algorithm decomposes a multi-stage mixed-integer nonlinear programming (MINLP) problem into a series of single-stage problems so that each subproblem can be solved by using Bellman's equation. To prove the effectiveness of the proposed algorithm, three case studies are taken into consideration: 1 minimizing the daily energy cost; 2 minimizing the emission cost; 3 minimizing the daily energy cost and emission cost simultaneously. Moreover, each case is operated under different battery operation conditions to investigate the battery lifetime. Finally, performance comparisons are carried out with a conventional Q-learning algorithm.
Author Yeliz Yoldas
Ahmet Onen
Selcuk Goren
Author_xml – sequence: 1
  fullname: Yeliz Yoldas
  organization: Abdullah Gul University,Department of Electrical and Electronics Engineering,Kayseri,Turkey,38080
– sequence: 2
  fullname: Selcuk Goren
  organization: Abdullah Gul University,Department of Industrial Engineering,Kayseri,Turkey,38080
– sequence: 3
  fullname: Ahmet Onen
  organization: Abdullah Gul University,Department of Electrical and Electronics Engineering,Kayseri,Turkey,38080
BookMark eNotj01PAjEQhhujiYj8AG89cF3s9mOXHglBJAHBRM-b2ba7lnS3pFui_nuLeprJ82aeyXuHrnvfG4QecjJjYs7Y4-6wXM0ooWRGCBGkuEIjmssiE5ySWzQZhmPiuaSiKNgInfanaDtweOn7GLzDvsE7q4Jvg9UD_rTxA-_OLtpsiNCalH0Zndk-mtYE_OJ7Z3sDAR8uF9B1tm_x-my10Xj6Os1cyvoLW7jWhyTr7tFNA24wk_85Ru9Pq7flc7bdrzfLxTZTTPCY5Zozk1oUBgQRTcNrJXhadEkoV4xCASUQQ2pSayh0rriQAkDSEuqcgmRjtPnzag_H6hRSyfBdebDVL_ChrSBEq5ypaNIwKTifF8DnVEmgpmEqfVdASibZDx_cakY
CitedBy_id crossref_primary_10_1016_j_heliyon_2024_e31280
crossref_primary_10_1109_TPWRS_2022_3187069
crossref_primary_10_1007_s00202_025_03230_4
crossref_primary_10_3390_en15062251
crossref_primary_10_1109_ACCESS_2024_3440885
crossref_primary_10_1016_j_tej_2022_107129
crossref_primary_10_1016_j_seta_2023_103377
crossref_primary_10_1109_ACCESS_2025_3525843
crossref_primary_10_1016_j_isatra_2022_12_008
crossref_primary_10_1109_TASC_2024_3468074
crossref_primary_10_3390_su15118952
crossref_primary_10_3390_en16010090
crossref_primary_10_3390_en14185688
crossref_primary_10_3390_en15228739
crossref_primary_10_24018_ejeng_2025_10_3_3269
crossref_primary_10_1007_s13198_021_01479_z
crossref_primary_10_1002_tee_23980
crossref_primary_10_2514_1_G008165
ContentType Journal Article
DBID DOA
DOI 10.35833/MPCE.2020.000506
DatabaseName DOAJ Directory of Open Access Journals (ODIN)
DatabaseTitleList
Database_xml – sequence: 1
  dbid: DOA
  name: DOAJ Directory of Open Access Journals
  url: https://www.doaj.org/
  sourceTypes: Open Website
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2196-5420
EndPage 1159
ExternalDocumentID oai_doaj_org_article_2e0b3954486a482c9a2ef3c202ca0739
GroupedDBID -SC
-S~
4.4
5VR
5VS
8FE
8FG
9D9
9DC
AAFWJ
AAKKN
ABAZT
ABEEZ
ABJCF
ABVLG
ACGFS
AFGXO
AFPKN
AFUIB
AHBYD
AHYZX
ALMA_UNASSIGNED_HOLDINGS
AMKLP
ASPBG
AVWKF
BENPR
C24
C6C
CAJEC
ESBDL
GROUPED_DOAJ
HCIFZ
JAVBF
KQ8
L6V
M~E
OK1
PIMPY
PROAC
Q--
SOJ
U1G
U5M
ID FETCH-LOGICAL-c354t-1d43e2026ea505ff4bc5405fd7024c32a6a7a0e0b0bda6d1c4595aa927ab12a93
IEDL.DBID DOA
ISICitedReferencesCount 22
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000608833600012&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Fri Oct 03 12:39:40 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 6
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c354t-1d43e2026ea505ff4bc5405fd7024c32a6a7a0e0b0bda6d1c4595aa927ab12a93
OpenAccessLink https://doaj.org/article/2e0b3954486a482c9a2ef3c202ca0739
PageCount 9
ParticipantIDs doaj_primary_oai_doaj_org_article_2e0b3954486a482c9a2ef3c202ca0739
PublicationCentury 2000
PublicationDate 2020-11-01
PublicationDateYYYYMMDD 2020-11-01
PublicationDate_xml – month: 11
  year: 2020
  text: 2020-11-01
  day: 01
PublicationDecade 2020
PublicationTitle Journal of modern power systems and clean energy
PublicationYear 2020
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0001925663
Score 2.302986
Snippet This paper proposes an energy management system (EMS) for the real-time operation of a pilot stochastic and dynamic microgrid on a university campus in Malta...
SourceID doaj
SourceType Open Website
StartPage 1151
SubjectTerms Cost minimization
energy management system
microgrid
real-time optimization
reinforcement learning
Title Optimal Control of Microgrids with Multi-stage Mixed-integer Nonlinear Programming Guided $Q$-learning Algorithm
URI https://doaj.org/article/2e0b3954486a482c9a2ef3c202ca0739
Volume 8
WOSCitedRecordID wos000608833600012&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVAON
  databaseName: DOAJ Directory of Open Access Journals
  databaseCode: DOA
  dateStart: 20130101
  customDbUrl:
  isFulltext: true
  eissn: 2196-5420
  dateEnd: 20241231
  titleUrlDefault: https://www.doaj.org/
  omitProxy: false
  ssIdentifier: ssj0001925663
  providerName: Directory of Open Access Journals
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  databaseCode: M~E
  dateStart: 20130101
  customDbUrl:
  isFulltext: true
  eissn: 2196-5420
  dateEnd: 99991231
  titleUrlDefault: https://road.issn.org
  omitProxy: false
  ssIdentifier: ssj0001925663
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV07T8MwELZQxQAD4ine8tDVqmMnTj2WqoWlpUggdYsc-xIV0YfSFjHx2znHGcrEwpAMiXVR7pL4u_i77whpSyGd0gaY5cq3MDMJ6zoAJopcgLORxa1uNpGOx93pVE92Wn15TliQBw6O6wjgudQJZhHKxF1htRFQSIs5uzV-lcl_fXmqd5Kp94BbEKfIsIwpfWVRZzTpDzAfFJ7KxRP-W6S_nk2Gx-SogYG0Fy5_QvZgcUoOd8QBz8jqGd_mOQ7qBzY5XRZ05OlzZTVza-p_oNK6fJYhwCsBz32BY7X8A1R0HCQwTEUngYE1R6P0cTtz4Gj7pc2adhEl7X2UywqNzc_J23Dw2n9iTX8EZmUSb1jkYgl4RwoM4piiiHPr8VfhUpx4rRRGmdRw9B_PnVEusnGiE2O0SE0eCaPlBWktlgu4JJTniOucSKzkEDuVa1_TaxxogYOtVFfkwTsrWwUJjMyLUtcHMFRZE6rsr1Bd_4eRG3LggxjKAW9Ja1Nt4Y7s28_NbF3d108B7kffgx-ZULk5
linkProvider Directory of Open Access Journals
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimal+Control+of+Microgrids+with+Multi-stage+Mixed-integer+Nonlinear+Programming+Guided+%24Q%24-learning+Algorithm&rft.jtitle=Journal+of+modern+power+systems+and+clean+energy&rft.au=Yeliz+Yoldas&rft.au=Selcuk+Goren&rft.au=Ahmet+Onen&rft.date=2020-11-01&rft.pub=IEEE&rft.eissn=2196-5420&rft.volume=8&rft.issue=6&rft.spage=1151&rft.epage=1159&rft_id=info:doi/10.35833%2FMPCE.2020.000506&rft.externalDBID=DOA&rft.externalDocID=oai_doaj_org_article_2e0b3954486a482c9a2ef3c202ca0739