How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system
Both model predictive control (MPC) and deep reinforcement learning control (DRL) have been presented as a way to approximate the true optimality of a dynamic programming problem, and these two have shown significant operational cost saving potentials for building energy systems. However, there is s...
Saved in:
| Published in: | Energy (Oxford) Vol. 273; no. C; p. 127073 |
|---|---|
| Main Authors: | , , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
United Kingdom
Elsevier Ltd
01.06.2023
Elsevier |
| Subjects: | |
| ISSN: | 0360-5442 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Both model predictive control (MPC) and deep reinforcement learning control (DRL) have been presented as a way to approximate the true optimality of a dynamic programming problem, and these two have shown significant operational cost saving potentials for building energy systems. However, there is still a lack of in-depth quantitative studies on their approximation levels to the true optimality, especially in the building energy domain. To fill in the gap, this paper provides a numerical framework that enables the evaluation of the optimality levels of different controllers for building energy systems. This framework is then used to comprehensively compare the optimal control performance of both MPC and DRL controllers with given computation budgets for a single zone fan coil unit system. Note the optimality is estimated based on a user-specific selection of trade-off weights among energy costs, thermal comfort and control slew rates. Compared with the best optimality we can find through expensive optimization simulations, the best DRL agent can maximally approximate the optimality by 96.54%, which outperforms the best MPC whose optimality level is 90.11%. However, due to the stochasticity, the DRL agent is only expected to approximate the optimality by 90.42%, which is almost equivalent to the best MPC. Except for Proximal Policy Optimization (PPO), all DRL agents can have a better approximation to the optimality than the best MPC, and are expected to have better approximation than the MPC with a prediction horizon of 32 steps (15 min per step). In terms of reducing energy cost and thermal discomfort, MPC can outperform the rule-based control (RBC) by 18.47%–25.44%. DRL can be expected to outperform RBC by 18.95%–25.65% ,and the best DRL control policy can outperform RBC by 20.29%–29.72%. Although the comparison of the optimality level is performed in a perfect setting, e.g., MPC assumes perfect models, and DRL assumes a perfect offline training process and online deployment process, this can shed insight on their capabilities of approximating to the original dynamic programming problem.
•Learning- and model-based controllers are evaluated in a containerized environment.•Both controllers can approximate the true optimality by up to 90%–96%.•Most DRL controllers have equivalent performance as the best MPC controller.•DRL controllers can outperform the best MPC controller by up to 6%. |
|---|---|
| AbstractList | Both model predictive control (MPC) and deep reinforcement learning control (DRL) have been presented as a way to approximate the true optimality of a dynamic programming problem, and these two have shown significant operational cost saving potentials for building energy systems. However, there is still a lack of in-depth quantitative studies on their approximation levels to the true optimality, especially in the building energy domain. To fill in the gap, this paper provides a numerical framework that enables the evaluation of the optimality levels of different controllers for building energy systems. This framework is then used to comprehensively compare the optimal control performance of both MPC and DRL controllers with given computation budgets for a single zone fan coil unit system. Note the optimality is estimated based on a user-specific selection of trade-off weights among energy costs, thermal comfort and control slew rates. Compared with the best optimality we can find through expensive optimization simulations, the best DRL agent can maximally approximate the optimality by 96.54%, which outperforms the best MPC whose optimality level is 90.11%. However, due to the stochasticity, the DRL agent is only expected to approximate the optimality by 90.42%, which is almost equivalent to the best MPC. Except for Proximal Policy Optimization (PPO), all DRL agents can have a better approximation to the optimality than the best MPC, and are expected to have better approximation than the MPC with a prediction horizon of 32 steps (15 min per step). In terms of reducing energy cost and thermal discomfort, MPC can outperform the rule-based control (RBC) by 18.47%–25.44%. DRL can be expected to outperform RBC by 18.95%–25.65% ,and the best DRL control policy can outperform RBC by 20.29%–29.72%. Although the comparison of the optimality level is performed in a perfect setting, e.g., MPC assumes perfect models, and DRL assumes a perfect offline training process and online deployment process, this can shed insight on their capabilities of approximating to the original dynamic programming problem. Both model predictive control (MPC) and deep reinforcement learning control (DRL) have been presented as a way to approximate the true optimality of a dynamic programming problem, and these two have shown significant operational cost saving potentials for building energy systems. However, there is still a lack of in-depth quantitative studies on their approximation levels to the true optimality, especially in the building energy domain. To fill in the gap, this paper provides a numerical framework that enables the evaluation of the optimality levels of different controllers for building energy systems. This framework is then used to comprehensively compare the optimal control performance of both MPC and DRL controllers with given computation budgets for a single zone fan coil unit system. Note the optimality is estimated based on a user-specific selection of trade-off weights among energy costs, thermal comfort and control slew rates. Compared with the best optimality we can find through expensive optimization simulations, the best DRL agent can maximally approximate the optimality by 96.54%, which outperforms the best MPC whose optimality level is 90.11%. However, due to the stochasticity, the DRL agent is only expected to approximate the optimality by 90.42%, which is almost equivalent to the best MPC. Except for Proximal Policy Optimization (PPO), all DRL agents can have a better approximation to the optimality than the best MPC, and are expected to have better approximation than the MPC with a prediction horizon of 32 steps (15 min per step). In terms of reducing energy cost and thermal discomfort, MPC can outperform the rule-based control (RBC) by 18.47%–25.44%. DRL can be expected to outperform RBC by 18.95%–25.65% ,and the best DRL control policy can outperform RBC by 20.29%–29.72%. Although the comparison of the optimality level is performed in a perfect setting, e.g., MPC assumes perfect models, and DRL assumes a perfect offline training process and online deployment process, this can shed insight on their capabilities of approximating to the original dynamic programming problem. •Learning- and model-based controllers are evaluated in a containerized environment.•Both controllers can approximate the true optimality by up to 90%–96%.•Most DRL controllers have equivalent performance as the best MPC controller.•DRL controllers can outperform the best MPC controller by up to 6%. |
| ArticleNumber | 127073 |
| Author | Fu, Yangyang O’Neill, Zheng Xu, Shichao Zhu, Qi Adetola, Veronica |
| Author_xml | – sequence: 1 givenname: Yangyang surname: Fu fullname: Fu, Yangyang email: yangyang.fu@tamu.edu organization: J. Mike Walker’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX, USA – sequence: 2 givenname: Shichao orcidid: 0000-0002-9026-6245 surname: Xu fullname: Xu, Shichao email: shichaoxu2023@u.northwestern.edu organization: Department of Computer Science, Northwestern University, IL, USA – sequence: 3 givenname: Qi orcidid: 0000-0002-7700-4099 surname: Zhu fullname: Zhu, Qi email: qzhu@northwestern.edu organization: Department of Computer Engineering, Northwestern University, IL, USA – sequence: 4 givenname: Zheng orcidid: 0000-0002-8839-7174 surname: O’Neill fullname: O’Neill, Zheng email: zoneill@tamu.edu organization: J. Mike Walker’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX, USA – sequence: 5 givenname: Veronica surname: Adetola fullname: Adetola, Veronica email: veronica.adetola@pnnl.gov organization: Pacific Northwest National Laboratory, Richland, WA, USA |
| BackLink | https://www.osti.gov/biblio/1963981$$D View this record in Osti.gov |
| BookMark | eNqFkT9vFDEQxV0Eifz7BimsVDS7jO272zUFCEVAIkWiCbU1a89efPLZwd47dHwAPje-bKoUUFnWvPc0v3ln7CSmSIxdCWgFiNX7TUuR8vrQSpCqFbKDTp2wU1AraJaLhXzLzkrZAMCy1_qU_blNv_g6JccxEw-EOfq4bgYs5LhNccop8H1bWr5NjsKrwZgyDwkdL49-nKrxE7-LeyqTX-PkUyw8RY681Ekg_rtuyoedD67--bwmL4cy0faCvRkxFLp8ec_Zj69fHm5um_vv3-5uPt83VslONT0CaqURO-0QnBJ2KZ3Sg0YNqhNWAonBYW9dtxitllKvOgDRD25QGnqhztn1nJvqjqZYP5F9rDSR7GSEXin9LHo3i55y-rmrNGbri6UQMFLaFaPEUkmo4q5KP8xSm1MpmUZTI5_Rp4w-GAHm2IrZmBnXHFsxcyvVvHhlfsp-i_nwP9vH2Ub1UHtP-chB0ZLz-Yjhkv93wF_ppa4i |
| CitedBy_id | crossref_primary_10_1016_j_gloei_2023_11_002 crossref_primary_10_3390_en16145326 crossref_primary_10_1109_TASE_2025_3551649 crossref_primary_10_1016_j_jobe_2024_111310 crossref_primary_10_1007_s12273_025_1275_1 crossref_primary_10_1016_j_buildenv_2025_113359 crossref_primary_10_1186_s40807_025_00152_4 crossref_primary_10_3390_ma16155321 crossref_primary_10_1016_j_apenergy_2024_125046 crossref_primary_10_1016_j_energy_2025_138517 crossref_primary_10_1016_j_aej_2025_09_017 crossref_primary_10_1016_j_enbuild_2023_113295 crossref_primary_10_1016_j_apenergy_2024_123753 crossref_primary_10_1016_j_enbuild_2023_113770 crossref_primary_10_1016_j_egyr_2025_07_003 crossref_primary_10_1016_j_enbuild_2025_115299 crossref_primary_10_1016_j_apenergy_2023_121785 crossref_primary_10_1016_j_jobe_2025_111909 crossref_primary_10_1016_j_scs_2025_106623 crossref_primary_10_1080_23744731_2024_2444818 crossref_primary_10_1016_j_engappai_2025_110831 |
| Cites_doi | 10.1016/j.enbuild.2019.01.018 10.1145/3563357.3564064 10.1145/3486611.3492412 10.1109/TCST.2015.2415411 10.1016/j.buildenv.2015.04.033 10.1016/j.autcon.2022.104128 10.1126/science.aar6404 10.1016/j.apenergy.2021.118346 10.1145/3061639.3062224 10.1016/j.arcontrol.2020.09.001 10.1109/TIA.2019.2932963 10.1109/TSG.2016.2597002 10.1177/0278364917710318 10.1016/j.apenergy.2021.117639 10.1016/j.apenergy.2020.115621 10.1016/j.enbuild.2020.110490 10.1007/s12532-018-0139-4 10.1016/j.apenergy.2013.05.081 10.1080/19401493.2021.1986574 10.1609/aaai.v30i1.10295 10.1109/ACC.2010.5530680 10.1109/MS.2015.62 10.1016/j.ifacol.2017.08.747 10.3384/ecp21181325 10.1016/j.foar.2012.11.001 10.1109/TSG.2016.2628897 10.3166/ejc.11.310-334 10.1016/j.enbuild.2019.07.029 10.1016/S0378-7788(00)00114-6 10.1109/9.704994 |
| ContentType | Journal Article |
| Copyright | 2023 Elsevier Ltd |
| Copyright_xml | – notice: 2023 Elsevier Ltd |
| DBID | AAYXX CITATION 7S9 L.6 OTOTI |
| DOI | 10.1016/j.energy.2023.127073 |
| DatabaseName | CrossRef AGRICOLA AGRICOLA - Academic OSTI.GOV |
| DatabaseTitle | CrossRef AGRICOLA AGRICOLA - Academic |
| DatabaseTitleList | AGRICOLA |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Economics Environmental Sciences |
| ExternalDocumentID | 1963981 10_1016_j_energy_2023_127073 S036054422300467X |
| GroupedDBID | --K --M .DC .~1 0R~ 1B1 1RT 1~. 1~5 4.4 457 4G. 5GY 5VS 7-5 71M 8P~ 9JN AABNK AACTN AAEDT AAEDW AAHBH AAHCO AAIKC AAIKJ AAKOC AALRI AAMNW AAOAW AAQFI AARJD AAXKI AAXUO ABJNI ABMAC ACDAQ ACGFS ACIWK ACRLP ADBBV ADEZE AEBSH AEKER AENEX AFJKZ AFKWA AFRAH AFTJW AGHFR AGUBO AGYEJ AHIDL AIEXJ AIKHN AITUG AJOXV AKRWK ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AXJTR BELTK BKOJK BLXMC CS3 DU5 EBS EFJIC EO8 EO9 EP2 EP3 FDB FIRID FNPLU FYGXN G-Q GBLVA IHE J1W JARJE KOM LY6 M41 MO0 N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. Q38 RNS ROL RPZ SDF SDG SES SPC SPCBC SSR SSZ T5K TN5 XPP ZMT ~02 ~G- 29G 6TJ 9DU AAQXK AATTM AAYWO AAYXX ABDPE ABFNM ABWVN ABXDB ACLOT ACRPL ACVFH ADCNI ADMUD ADNMO ADXHL AEIPS AEUPX AFPUW AGQPQ AHHHB AIGII AIIUN AKBMS AKYEP ANKPU APXCP ASPBG AVWKF AZFZN CITATION EFKBS EFLBG EJD FEDTE FGOYB G-2 HVGLF HZ~ R2- SAC SEW WUQ ~HD 7S9 L.6 AAIAV ABYKQ OTOTI RIG |
| ID | FETCH-LOGICAL-c3273-8a0a939aa79da0d31c52d39b9a90371c20e1bda8cd74fc9229670018bdb390813 |
| ISSN | 0360-5442 |
| IngestDate | Mon Apr 01 04:54:15 EDT 2024 Sat Sep 27 22:02:50 EDT 2025 Tue Nov 18 22:43:12 EST 2025 Sat Nov 29 07:19:03 EST 2025 Tue Dec 03 03:44:28 EST 2024 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | C |
| Keywords | Model predictive control Deep reinforcement learning Building energy and control system Approximate dynamic programming problem |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c3273-8a0a939aa79da0d31c52d39b9a90371c20e1bda8cd74fc9229670018bdb390813 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 USDOE EE0009150 |
| ORCID | 0000-0002-8839-7174 0000-0002-9026-6245 0000-0002-7700-4099 0000000290266245 0000000277004099 0000000288397174 |
| OpenAccessLink | https://www.osti.gov/biblio/1963981 |
| PQID | 3153209637 |
| PQPubID | 24069 |
| ParticipantIDs | osti_scitechconnect_1963981 proquest_miscellaneous_3153209637 crossref_citationtrail_10_1016_j_energy_2023_127073 crossref_primary_10_1016_j_energy_2023_127073 elsevier_sciencedirect_doi_10_1016_j_energy_2023_127073 |
| PublicationCentury | 2000 |
| PublicationDate | 20230601 |
| PublicationDateYYYYMMDD | 2023-06-01 |
| PublicationDate_xml | – month: 06 year: 2023 text: 20230601 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | United Kingdom |
| PublicationPlace_xml | – name: United Kingdom |
| PublicationTitle | Energy (Oxford) |
| PublicationYear | 2023 |
| Publisher | Elsevier Ltd Elsevier |
| Publisher_xml | – name: Elsevier Ltd – name: Elsevier |
| References | Zhang, Chong, Pan, Zhang, Lam (b22) 2019; 199 Bertsekas (b29) 2005; 11 Rao (b47) 2009; 135 Brandi, Fiorentini, Capozzoli (b27) 2022; 135 Fabietti, Gorecki, Qureshi, Bitlislioğlu, Lymperopoulos, Jones (b12) 2016; 9 Fu, Han, Baker, Zuo (b35) 2020; 277 Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra, Riedmiller (b18) 2013 Blum, Arroyo, Huang, Drgoňa, Jorissen, Walnum, Chen, Benne, Vrabie, Wetter (b28) 2021; 14 Van Cutsem, Kayal, Blum, Pritoni (b15) 2019 Schreiber, Eschweiler, Baranski, Müller (b21) 2020; 229 Silver, Hubert, Schrittwieser, Antonoglou, Lai, Guez, Lanctot, Sifre, Kumaran, Graepel (b17) 2018; 362 Arroyo, Manna, Spiessens, Helsen (b26) 2022; 309 Levine, Pastor, Krizhevsky, Ibarz, Quillen (b19) 2018; 37 Westphalen, Koszalinski (b3) 1999; 20 Bruno, Giannoccaro, La Scala (b11) 2019; 55 Scokaert, Rawlings (b30) 1998; 43 Bertsekas (b23) 2019 Wetter M, Benne K, Ravache B. Software Architecture and Implementation of Modelica Buildings Library Coupling for Spawn of EnergyPlus. In: Modelica Conferences. 2021, p. 325–34. Drgoňa, Arroyo, Figueroa, Blum, Arendt, Kim, Ollé, Oravec, Wetter, Vrabie (b7) 2020 Sturzenegger, Gyalistras, Morari, Smith (b14) 2015; 24 Blochwitz, Otter, Arnold, Bausch, Clauß, Elmqvist, Junghanns, Mauss, Monteiro, Neidhold (b33) 2011 Wilcox, Marion (b46) 2008 Fu, Zuo, Wetter, VanGilder, Han, Plamondon (b4) 2019; 186 Fu, Xu, Zhu, O’Neill (b31) 2021 Xu, Fu, Wang, Yang, O’Neill, Wang, Zhu (b39) 2022 (b2) 2022 Görges (b24) 2017; 50 Anderson (b32) 2015; 32 Moritz, Nishihara, Wang, Tumanov, Liaw, Liang, Elibol, Yang, Paul, Jordan, Stoica (b43) 2018 Liaw, Liang, Nishihara, Moritz, Gonzalez, Stoica (b44) 2018 Weng, Chen, Yan, You, Duburcq, Zhang, Su, Zhu (b42) 2021 Nagy, Kazmi, Cheaib, Driesen (b25) 2018 Zhang, Blum, Grahovac, Hu, Granderson, Wetter (b6) 2020 Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1. 2016. Vrettos, Kara, MacDonald, Andersson, Callaway (b13) 2016; 9 Judkoff, Neymark (b45) 1995 Candanedo, Dehkordi, Stylianou (b10) 2013; 111 Cígler J, Gyalistras D, Široky J, Tiet V, Ferkl L. Beyond theory: the challenge of implementing model predictive control in buildings. In: Proceedings of 11th Rehva World Congress, Clima, vol. 250. 2013. Fu, O’Neill, Yang, Adetola, Wen, Ren, Wagner, Zhu, Wu (b36) 2021; 303 Andersson, Gillis, Horn, Rawlings, Diehl (b41) 2019; 11 Liang, Quinte, Jia, Sun (b8) 2015; 92 Lu X, Fu Y, Xu S, Zhu Q, O’Neill Z, Yang Z. Comparison Study Of High- performance Rule-based HVAC Control With Deep Reinforcement Learning-based Control In A Multi-zone VAV System. In: International High Performance Buildings Conference. 2022, URL. Oldewurtel F, Gyalistras D, Gwerder M, Jones C, Parisio A, Stauch V, Lehmann B, Morari M. Increasing energy efficiency in building climate control using weather forecasts and model predictive control. In: Clima-RHEVA World Congress. (CONF). 2010. Brockman, Cheung, Pettersson, Schneider, Schulman, Tang, Zaremba (b34) 2016 EIA (b1) 2022 Wei T, Wang Y, Zhu Q. Deep reinforcement learning for building HVAC control. In: Proceedings of the 54th Annual DAC 2017. 2017, p. 1–6. Aghemo, Virgone, Fracastoro, Pellegrino, Blaso, Savoyat, Johannes (b5) 2013; 2 Crawley, Lawrie, Winkelmann, Buhl, Huang, Pedersen, Strand, Liesen, Fisher, Witte (b37) 2001; 33 Zhang (10.1016/j.energy.2023.127073_b6) 2020 Blum (10.1016/j.energy.2023.127073_b28) 2021; 14 Drgoňa (10.1016/j.energy.2023.127073_b7) 2020 10.1016/j.energy.2023.127073_b16 10.1016/j.energy.2023.127073_b38 Candanedo (10.1016/j.energy.2023.127073_b10) 2013; 111 Bruno (10.1016/j.energy.2023.127073_b11) 2019; 55 Van Cutsem (10.1016/j.energy.2023.127073_b15) 2019 Schreiber (10.1016/j.energy.2023.127073_b21) 2020; 229 Sturzenegger (10.1016/j.energy.2023.127073_b14) 2015; 24 Liaw (10.1016/j.energy.2023.127073_b44) 2018 Mnih (10.1016/j.energy.2023.127073_b18) 2013 Brandi (10.1016/j.energy.2023.127073_b27) 2022; 135 (10.1016/j.energy.2023.127073_b2) 2022 Fu (10.1016/j.energy.2023.127073_b36) 2021; 303 Weng (10.1016/j.energy.2023.127073_b42) 2021 Görges (10.1016/j.energy.2023.127073_b24) 2017; 50 Anderson (10.1016/j.energy.2023.127073_b32) 2015; 32 Moritz (10.1016/j.energy.2023.127073_b43) 2018 Scokaert (10.1016/j.energy.2023.127073_b30) 1998; 43 10.1016/j.energy.2023.127073_b40 Judkoff (10.1016/j.energy.2023.127073_b45) 1995 Rao (10.1016/j.energy.2023.127073_b47) 2009; 135 Arroyo (10.1016/j.energy.2023.127073_b26) 2022; 309 Fu (10.1016/j.energy.2023.127073_b4) 2019; 186 Wilcox (10.1016/j.energy.2023.127073_b46) 2008 Liang (10.1016/j.energy.2023.127073_b8) 2015; 92 Levine (10.1016/j.energy.2023.127073_b19) 2018; 37 10.1016/j.energy.2023.127073_b48 10.1016/j.energy.2023.127073_b9 Vrettos (10.1016/j.energy.2023.127073_b13) 2016; 9 Fu (10.1016/j.energy.2023.127073_b31) 2021 Brockman (10.1016/j.energy.2023.127073_b34) 2016 10.1016/j.energy.2023.127073_b20 EIA (10.1016/j.energy.2023.127073_b1) 2022 Westphalen (10.1016/j.energy.2023.127073_b3) 1999; 20 Nagy (10.1016/j.energy.2023.127073_b25) 2018 Aghemo (10.1016/j.energy.2023.127073_b5) 2013; 2 Fabietti (10.1016/j.energy.2023.127073_b12) 2016; 9 Crawley (10.1016/j.energy.2023.127073_b37) 2001; 33 Andersson (10.1016/j.energy.2023.127073_b41) 2019; 11 Bertsekas (10.1016/j.energy.2023.127073_b29) 2005; 11 Bertsekas (10.1016/j.energy.2023.127073_b23) 2019 Fu (10.1016/j.energy.2023.127073_b35) 2020; 277 Silver (10.1016/j.energy.2023.127073_b17) 2018; 362 Xu (10.1016/j.energy.2023.127073_b39) 2022 Zhang (10.1016/j.energy.2023.127073_b22) 2019; 199 Blochwitz (10.1016/j.energy.2023.127073_b33) 2011 |
| References_xml | – volume: 32 start-page: 102 year: 2015 end-page: c3 ident: b32 article-title: Docker [software engineering] publication-title: Ieee Softw. – volume: 362 start-page: 1140 year: 2018 end-page: 1144 ident: b17 article-title: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play publication-title: Science – volume: 55 start-page: 7052 year: 2019 end-page: 7061 ident: b11 article-title: A demand response implementation in tertiary buildings through model predictive control publication-title: IEEE Trans Ind Appl – volume: 92 start-page: 256 year: 2015 end-page: 268 ident: b8 article-title: MPC control for improving energy efficiency of a building air handler for multi-zone VAVs publication-title: Build Environ – volume: 9 start-page: 1657 year: 2016 end-page: 1666 ident: b12 article-title: Experimental implementation of frequency regulation services using commercial buildings publication-title: IEEE Trans Smart Grid – volume: 14 start-page: 586 year: 2021 end-page: 610 ident: b28 article-title: Building optimization testing framework (BOPTEST) for simulation-based benchmarking of control strategies in buildings publication-title: J. Build. Perform. Simul. – volume: 229 year: 2020 ident: b21 article-title: Application of two promising reinforcement learning algorithms for load shifting in a cooling supply system publication-title: Energy Build – volume: 24 start-page: 1 year: 2015 end-page: 12 ident: b14 article-title: Model predictive climate control of a swiss office building: Implementation, results, and cost–benefit analysis publication-title: IEEE Trans Control Syst Technol – volume: 2 start-page: 147 year: 2013 end-page: 161 ident: b5 article-title: Management and monitoring of public buildings through ICT based systems: Control rules for energy saving with lighting and HVAC services publication-title: Front. Archit. Res. – start-page: 276 year: 2021 end-page: 280 ident: b31 article-title: Containerized framework for building control performance comparisons: Model predictive control vs deep reinforcement learning control publication-title: BuildSys 2021 - Proceedings of the 2021 ACM International Conference on Systems for Energy-Efficient Built Environments – volume: 186 start-page: 108 year: 2019 end-page: 125 ident: b4 article-title: Equation-based object-oriented modeling and simulation for data center cooling: A case study publication-title: Energy Build – year: 2018 ident: b44 article-title: Tune: A research platform for distributed model selection and training – reference: Cígler J, Gyalistras D, Široky J, Tiet V, Ferkl L. Beyond theory: the challenge of implementing model predictive control in buildings. In: Proceedings of 11th Rehva World Congress, Clima, vol. 250. 2013. – year: 2013 ident: b18 article-title: Playing atari with deep reinforcement learning – reference: Wei T, Wang Y, Zhu Q. Deep reinforcement learning for building HVAC control. In: Proceedings of the 54th Annual DAC 2017. 2017, p. 1–6. – volume: 20 start-page: 33700 year: 1999 end-page: 33745 ident: b3 article-title: Energy consumption characteristics of commercial building HVAC systems. Volume II: Thermal distribution, auxiliary equipment, and ventilation publication-title: Arthur D. Little Inc (ADLI) – volume: 277 year: 2020 ident: b35 article-title: Assessments of data centers for provision of frequency regulation publication-title: Appl Energy – year: 2018 ident: b25 article-title: Deep reinforcement learning for optimal control of space heating – volume: 11 start-page: 1 year: 2019 end-page: 36 ident: b41 article-title: CasADi: a software framework for nonlinear optimization and optimal control publication-title: Math Program Comput – volume: 309 year: 2022 ident: b26 article-title: Reinforced model predictive control (RL-MPC) for building energy management publication-title: Appl Energy – volume: 11 start-page: 310 year: 2005 end-page: 334 ident: b29 article-title: Dynamic programming and suboptimal control: A survey from ADP to MPC publication-title: Eur J Control – volume: 37 start-page: 421 year: 2018 end-page: 436 ident: b19 article-title: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection publication-title: Int J Robot Res – year: 1995 ident: b45 article-title: International energy agency building energy simulation test (BESTEST) and diagnostic method – year: 2021 ident: b42 article-title: Tianshou: a highly modularized deep reinforcement learning library – volume: 111 start-page: 1032 year: 2013 end-page: 1045 ident: b10 article-title: Model-based predictive control of an ice storage device in a building cooling system publication-title: Appl Energy – start-page: 561 year: 2018 end-page: 577 ident: b43 article-title: Ray: A distributed framework for emerging AI applications publication-title: 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) – year: 2020 ident: b6 article-title: Development and Verification of Control Sequences for Single-Zone Variable Air Volume System Based on ASHRAEGuideline 36 – year: 2020 ident: b7 article-title: All you need to know about model predictive control for buildings publication-title: Annu Rev Control – volume: 199 start-page: 472 year: 2019 end-page: 490 ident: b22 article-title: Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning publication-title: Energy Build – year: 2022 ident: b1 article-title: Monthly energy review december 2022 – year: 2008 ident: b46 article-title: Users manual for TMY3 data sets – start-page: 105 year: 2011 end-page: 114 ident: b33 article-title: The functional mockup interface for tool independent exchange of simulation models publication-title: Proceedings of the 8th International Modelica Conference – volume: 33 start-page: 319 year: 2001 end-page: 331 ident: b37 article-title: EnergyPlus: creating a new-generation building energy simulation program publication-title: Energy Build – volume: 303 year: 2021 ident: b36 article-title: Modeling and evaluation of cyber-attacks on grid-interactive efficient buildings publication-title: Appl Energy – volume: 135 year: 2022 ident: b27 article-title: Comparison of online and offline deep reinforcement learning with model predictive control for thermal energy management publication-title: Autom Constr – year: 2016 ident: b34 article-title: Openai gym – reference: Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1. 2016. – year: 2022 ident: b2 article-title: How much energy is consumed in U.S. buildings? – year: 2019 ident: b23 article-title: Reinforcement Learning and Optimal Control – volume: 43 start-page: 1163 year: 1998 end-page: 1169 ident: b30 article-title: Constrained linear quadratic regulation publication-title: IEEE Trans Automat Control – volume: 135 start-page: 497 year: 2009 end-page: 528 ident: b47 article-title: A survey of numerical methods for optimal control publication-title: Adv. Astronaut. Sci. – start-page: 89 year: 2022 end-page: 98 ident: b39 article-title: Accelerate online reinforcement learning for building HVAC control with heterogeneous expert guidances publication-title: Proceedings of the 9th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation – start-page: 1 year: 2019 end-page: 6 ident: b15 article-title: Comparison of MPC formulations for building control under commercial time-of-use tariffs publication-title: 2019 IEEE Milan PowerTech – reference: Wetter M, Benne K, Ravache B. Software Architecture and Implementation of Modelica Buildings Library Coupling for Spawn of EnergyPlus. In: Modelica Conferences. 2021, p. 325–34. – reference: Lu X, Fu Y, Xu S, Zhu Q, O’Neill Z, Yang Z. Comparison Study Of High- performance Rule-based HVAC Control With Deep Reinforcement Learning-based Control In A Multi-zone VAV System. In: International High Performance Buildings Conference. 2022, URL. – volume: 9 start-page: 3213 year: 2016 end-page: 3223 ident: b13 article-title: Experimental demonstration of frequency regulation by commercial buildings—Part I: Modeling and hierarchical control design publication-title: IEEE Trans Smart Grid – reference: Oldewurtel F, Gyalistras D, Gwerder M, Jones C, Parisio A, Stauch V, Lehmann B, Morari M. Increasing energy efficiency in building climate control using weather forecasts and model predictive control. In: Clima-RHEVA World Congress. (CONF). 2010. – volume: 50 start-page: 4920 year: 2017 end-page: 4928 ident: b24 article-title: Relations between model predictive control and reinforcement learning publication-title: IFAC-PapersOnLine – volume: 186 start-page: 108 year: 2019 ident: 10.1016/j.energy.2023.127073_b4 article-title: Equation-based object-oriented modeling and simulation for data center cooling: A case study publication-title: Energy Build doi: 10.1016/j.enbuild.2019.01.018 – volume: 20 start-page: 33700 issue: October year: 1999 ident: 10.1016/j.energy.2023.127073_b3 article-title: Energy consumption characteristics of commercial building HVAC systems. Volume II: Thermal distribution, auxiliary equipment, and ventilation publication-title: Arthur D. Little Inc (ADLI) – start-page: 89 year: 2022 ident: 10.1016/j.energy.2023.127073_b39 article-title: Accelerate online reinforcement learning for building HVAC control with heterogeneous expert guidances doi: 10.1145/3563357.3564064 – start-page: 276 year: 2021 ident: 10.1016/j.energy.2023.127073_b31 article-title: Containerized framework for building control performance comparisons: Model predictive control vs deep reinforcement learning control doi: 10.1145/3486611.3492412 – volume: 24 start-page: 1 issue: 1 year: 2015 ident: 10.1016/j.energy.2023.127073_b14 article-title: Model predictive climate control of a swiss office building: Implementation, results, and cost–benefit analysis publication-title: IEEE Trans Control Syst Technol doi: 10.1109/TCST.2015.2415411 – volume: 92 start-page: 256 year: 2015 ident: 10.1016/j.energy.2023.127073_b8 article-title: MPC control for improving energy efficiency of a building air handler for multi-zone VAVs publication-title: Build Environ doi: 10.1016/j.buildenv.2015.04.033 – start-page: 105 year: 2011 ident: 10.1016/j.energy.2023.127073_b33 article-title: The functional mockup interface for tool independent exchange of simulation models – volume: 135 issn: 0926-5805 year: 2022 ident: 10.1016/j.energy.2023.127073_b27 article-title: Comparison of online and offline deep reinforcement learning with model predictive control for thermal energy management publication-title: Autom Constr doi: 10.1016/j.autcon.2022.104128 – year: 2018 ident: 10.1016/j.energy.2023.127073_b25 – year: 2018 ident: 10.1016/j.energy.2023.127073_b44 – year: 2021 ident: 10.1016/j.energy.2023.127073_b42 – volume: 362 start-page: 1140 issue: 6419 year: 2018 ident: 10.1016/j.energy.2023.127073_b17 article-title: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play publication-title: Science doi: 10.1126/science.aar6404 – volume: 309 year: 2022 ident: 10.1016/j.energy.2023.127073_b26 article-title: Reinforced model predictive control (RL-MPC) for building energy management publication-title: Appl Energy doi: 10.1016/j.apenergy.2021.118346 – start-page: 561 year: 2018 ident: 10.1016/j.energy.2023.127073_b43 article-title: Ray: A distributed framework for emerging AI applications – ident: 10.1016/j.energy.2023.127073_b20 doi: 10.1145/3061639.3062224 – year: 2020 ident: 10.1016/j.energy.2023.127073_b6 – year: 2020 ident: 10.1016/j.energy.2023.127073_b7 article-title: All you need to know about model predictive control for buildings publication-title: Annu Rev Control doi: 10.1016/j.arcontrol.2020.09.001 – volume: 55 start-page: 7052 issue: 6 year: 2019 ident: 10.1016/j.energy.2023.127073_b11 article-title: A demand response implementation in tertiary buildings through model predictive control publication-title: IEEE Trans Ind Appl doi: 10.1109/TIA.2019.2932963 – volume: 9 start-page: 1657 issue: 3 year: 2016 ident: 10.1016/j.energy.2023.127073_b12 article-title: Experimental implementation of frequency regulation services using commercial buildings publication-title: IEEE Trans Smart Grid doi: 10.1109/TSG.2016.2597002 – volume: 37 start-page: 421 issue: 4–5 year: 2018 ident: 10.1016/j.energy.2023.127073_b19 article-title: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection publication-title: Int J Robot Res doi: 10.1177/0278364917710318 – year: 2019 ident: 10.1016/j.energy.2023.127073_b23 – volume: 303 issn: 03062619 year: 2021 ident: 10.1016/j.energy.2023.127073_b36 article-title: Modeling and evaluation of cyber-attacks on grid-interactive efficient buildings publication-title: Appl Energy doi: 10.1016/j.apenergy.2021.117639 – volume: 277 issn: 03062619 year: 2020 ident: 10.1016/j.energy.2023.127073_b35 article-title: Assessments of data centers for provision of frequency regulation publication-title: Appl Energy doi: 10.1016/j.apenergy.2020.115621 – volume: 229 year: 2020 ident: 10.1016/j.energy.2023.127073_b21 article-title: Application of two promising reinforcement learning algorithms for load shifting in a cooling supply system publication-title: Energy Build doi: 10.1016/j.enbuild.2020.110490 – volume: 11 start-page: 1 issue: 1 year: 2019 ident: 10.1016/j.energy.2023.127073_b41 article-title: CasADi: a software framework for nonlinear optimization and optimal control publication-title: Math Program Comput doi: 10.1007/s12532-018-0139-4 – volume: 111 start-page: 1032 year: 2013 ident: 10.1016/j.energy.2023.127073_b10 article-title: Model-based predictive control of an ice storage device in a building cooling system publication-title: Appl Energy doi: 10.1016/j.apenergy.2013.05.081 – volume: 14 start-page: 586 issue: 5 year: 2021 ident: 10.1016/j.energy.2023.127073_b28 article-title: Building optimization testing framework (BOPTEST) for simulation-based benchmarking of control strategies in buildings publication-title: J. Build. Perform. Simul. doi: 10.1080/19401493.2021.1986574 – volume: 135 start-page: 497 issue: 1 year: 2009 ident: 10.1016/j.energy.2023.127073_b47 article-title: A survey of numerical methods for optimal control publication-title: Adv. Astronaut. Sci. – ident: 10.1016/j.energy.2023.127073_b48 doi: 10.1609/aaai.v30i1.10295 – start-page: 1 year: 2019 ident: 10.1016/j.energy.2023.127073_b15 article-title: Comparison of MPC formulations for building control under commercial time-of-use tariffs – year: 2022 ident: 10.1016/j.energy.2023.127073_b1 – year: 2008 ident: 10.1016/j.energy.2023.127073_b46 – ident: 10.1016/j.energy.2023.127073_b9 doi: 10.1109/ACC.2010.5530680 – year: 2013 ident: 10.1016/j.energy.2023.127073_b18 – volume: 32 start-page: 102 issue: 3 year: 2015 ident: 10.1016/j.energy.2023.127073_b32 article-title: Docker [software engineering] publication-title: Ieee Softw. doi: 10.1109/MS.2015.62 – ident: 10.1016/j.energy.2023.127073_b40 – volume: 50 start-page: 4920 issue: 1 year: 2017 ident: 10.1016/j.energy.2023.127073_b24 article-title: Relations between model predictive control and reinforcement learning publication-title: IFAC-PapersOnLine doi: 10.1016/j.ifacol.2017.08.747 – ident: 10.1016/j.energy.2023.127073_b38 doi: 10.3384/ecp21181325 – volume: 2 start-page: 147 issue: 2 year: 2013 ident: 10.1016/j.energy.2023.127073_b5 article-title: Management and monitoring of public buildings through ICT based systems: Control rules for energy saving with lighting and HVAC services publication-title: Front. Archit. Res. doi: 10.1016/j.foar.2012.11.001 – volume: 9 start-page: 3213 issue: 4 year: 2016 ident: 10.1016/j.energy.2023.127073_b13 article-title: Experimental demonstration of frequency regulation by commercial buildings—Part I: Modeling and hierarchical control design publication-title: IEEE Trans Smart Grid doi: 10.1109/TSG.2016.2628897 – volume: 11 start-page: 310 issue: 4–5 year: 2005 ident: 10.1016/j.energy.2023.127073_b29 article-title: Dynamic programming and suboptimal control: A survey from ADP to MPC publication-title: Eur J Control doi: 10.3166/ejc.11.310-334 – year: 1995 ident: 10.1016/j.energy.2023.127073_b45 – year: 2016 ident: 10.1016/j.energy.2023.127073_b34 – year: 2022 ident: 10.1016/j.energy.2023.127073_b2 – volume: 199 start-page: 472 year: 2019 ident: 10.1016/j.energy.2023.127073_b22 article-title: Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning publication-title: Energy Build doi: 10.1016/j.enbuild.2019.07.029 – volume: 33 start-page: 319 issue: 4 year: 2001 ident: 10.1016/j.energy.2023.127073_b37 article-title: EnergyPlus: creating a new-generation building energy simulation program publication-title: Energy Build doi: 10.1016/S0378-7788(00)00114-6 – ident: 10.1016/j.energy.2023.127073_b16 – volume: 43 start-page: 1163 issue: 8 year: 1998 ident: 10.1016/j.energy.2023.127073_b30 article-title: Constrained linear quadratic regulation publication-title: IEEE Trans Automat Control doi: 10.1109/9.704994 |
| SSID | ssj0005899 |
| Score | 2.5292323 |
| Snippet | Both model predictive control (MPC) and deep reinforcement learning control (DRL) have been presented as a way to approximate the true optimality of a dynamic... |
| SourceID | osti proquest crossref elsevier |
| SourceType | Open Access Repository Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 127073 |
| SubjectTerms | Approximate dynamic programming problem Building energy and control system Deep reinforcement learning energy energy costs issues and policy Model predictive control operating costs |
| Title | How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system |
| URI | https://dx.doi.org/10.1016/j.energy.2023.127073 https://www.proquest.com/docview/3153209637 https://www.osti.gov/biblio/1963981 |
| Volume | 273 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVESC databaseName: Elsevier SD Freedom Collection Journals 2021 issn: 0360-5442 databaseCode: AIEXJ dateStart: 19950101 customDbUrl: isFulltext: true dateEnd: 99991231 titleUrlDefault: https://www.sciencedirect.com omitProxy: false ssIdentifier: ssj0005899 providerName: Elsevier |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Rb9MwELZKhwQvCAbTygAZibcoUROnTfyEJtQKeJhADKniJXJsp-1UJdOyVoUfwB_hj3IXO0mzCsYeeIkqx3ac3hff5fLdHSFvfKmyOOPS5WM9dsOUcVeE8LiDZRHIIB2PUqmqYhPR2Vk8m_FPvd6vOhZms4ryPN5u-eV_FTW0gbAxdPYO4m4mhQb4DUKHI4gdjv8keCwSN8dcxcjpskUh5i5qK9UQ0zde6ZkiODdOIOlwVQjllItlZoKhp7u5OCrWXO4IB10MK-38KMBITW1pbUebQEKTHbrj8zcnMLXp1rDpG__DdF1pAZHPvwurRaF1VrV-WSCpv2id21Xr52XL1LFUDY6-7MqZ_W2h7SzWlRGwlnLVhHABSsKwsz0HEdvZYPFDuWnY2_uNG-LCM_fq4QW8tns31fYNFdgQE2vO20ViZklwlsTMco8cBNGIw9Z5cPphMvvYMoniqkxps_o6QrOiEe6v5k8WUL8AUe6ZBJWdc_6YPLIvKPTUAOsJ6en8kDyo49fLQ3I0aWMjoaNVDuVT8hOQRxF5FJBHu8ijFmAUkUd3kNecAFhQRB6tkfeWdnFHi5wKanBHEXe0xh01d08N7p6Rr9PJ-bv3ri3z4UoGwnVjMRSccSEirsRQMV-OAsV4ygXHfJIyGGo_VSKWKgozyYOAY2iZH6cKdhawaNkR6edw1WNCs5ipWHAteKhC7SswtYJQZ0MFO1Em9XhAWP3XJ9LmwMdSLKvkb4IfELcZdWlywNzSP6qlmlg71tinCUD1lpEnCAIchSmcJXLdYBhqSR77A_K6xkYCSgC_7IlcF-syYT7Wd4Fe0fM7rvWEPGyfxBekf3211i_Jfbm5XpZXryzSfwNwCdf5 |
| linkProvider | Elsevier |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+good+are+learning-based+control+v.s.+model-based+control+for+load+shifting%3F+Investigations+on+a+single+zone+building+energy+system&rft.jtitle=Energy+%28Oxford%29&rft.au=Fu%2C+Yangyang&rft.au=Xu%2C+Shichao&rft.au=Zhu%2C+Qi&rft.au=O%E2%80%99Neill%2C+Zheng&rft.date=2023-06-01&rft.issn=0360-5442&rft.volume=273&rft.spage=127073&rft_id=info:doi/10.1016%2Fj.energy.2023.127073&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_energy_2023_127073 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0360-5442&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0360-5442&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0360-5442&client=summon |