Adaptive Environment Generation for Continual Learning: Integrating Constraint Logic Programming With Deep Reinforcement Learning
In this article, we introduce a novel framework that combines constraint logic programming (CLP) with deep reinforcement learning (DRL) to create adaptive environments for continual learning. We focus on two challenging domains: Sudoku puzzles and scheduling problems, where environment complexity ev...
Saved in:
| Published in: | IEEE transactions on cognitive and developmental systems Vol. 17; no. 3; pp. 540 - 553 |
|---|---|
| Main Authors: | , |
| Format: | Journal Article |
| Language: | English |
| Published: |
Piscataway
IEEE
01.06.2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 2379-8920, 2379-8939 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | In this article, we introduce a novel framework that combines constraint logic programming (CLP) with deep reinforcement learning (DRL) to create adaptive environments for continual learning. We focus on two challenging domains: Sudoku puzzles and scheduling problems, where environment complexity evolves based on the agent's performance. By integrating CLP, we dynamically adjust problem difficulty in response to the agent's learning trajectory, ensuring a progressively challenging environment that fosters enhanced problem-solving skills. Empirical results across 500 000 episodes show substantial improvements in solve rates, increasing from 6% to 86% for sudoku puzzles and 7% to 79% for scheduling problems, alongside significant reductions in the average steps required to solve each problem. The proposed adaptive environment generation demonstrates the potential of CLP in advancing RL agents' continual learning capabilities by dynamically regulating complexity, thus improving their adaptability and learning efficiency. This framework contributes to the broader fields of reinforcement learning and procedural content generation by introducing an innovative approach to continual adaptation in complex environments. |
|---|---|
| AbstractList | In this article, we introduce a novel framework that combines constraint logic programming (CLP) with deep reinforcement learning (DRL) to create adaptive environments for continual learning. We focus on two challenging domains: Sudoku puzzles and scheduling problems, where environment complexity evolves based on the agent's performance. By integrating CLP, we dynamically adjust problem difficulty in response to the agent's learning trajectory, ensuring a progressively challenging environment that fosters enhanced problem-solving skills. Empirical results across 500 000 episodes show substantial improvements in solve rates, increasing from 6% to 86% for sudoku puzzles and 7% to 79% for scheduling problems, alongside significant reductions in the average steps required to solve each problem. The proposed adaptive environment generation demonstrates the potential of CLP in advancing RL agents' continual learning capabilities by dynamically regulating complexity, thus improving their adaptability and learning efficiency. This framework contributes to the broader fields of reinforcement learning and procedural content generation by introducing an innovative approach to continual adaptation in complex environments. |
| Author | Idrissi, Abdellah Boutyour, Youness |
| Author_xml | – sequence: 1 givenname: Youness orcidid: 0009-0006-5684-2751 surname: Boutyour fullname: Boutyour, Youness email: youness.boutyour@um5r.ac.ma organization: Department of Computer Science, Faculty of Sciences, Mohammed V University, Rabat, Morocco – sequence: 2 givenname: Abdellah orcidid: 0000-0001-5696-3007 surname: Idrissi fullname: Idrissi, Abdellah organization: Department of Computer Science, Faculty of Sciences, Mohammed V University, Rabat, Morocco |
| BookMark | eNpNUEtLw0AQXkTBWvsDBA8LnlP3lU3WW0lrLRQUrXgM22RSt7S7cbMtePSfm9gqMoeZ4XvBd4FOrbOA0BUlQ0qJul1k45chI0wMuUhjkbIT1GM8UVGquDr9uxk5R4OmWRNCqORJKpIe-hqVug5mD3hi98Y7uwUb8BQseB2Ms7hyHmfOBmN3eoPnoL01dnWHZzbAquPYVYc3wWvTKuduZQr85F2Lbbcd-GbCOx4D1PgZjG3tCvjJ-LW6RGeV3jQwOO4-er2fLLKHaP44nWWjeVQwIUPEY8FIIpfLEpQElSw5tJ_iaVLEEhItJKuYoCUVFSkqKXihlZCiZKXSy4oo3kc3B9_au48dNCFfu523bWTOGY1VzGk7fUQPrMK7pvFQ5bU3W-0_c0ryruy8Kzvvys6PZbea64PGAMA_fsJlLGL-DR5cfsM |
| CODEN | ITCDA4 |
| Cites_doi | 10.1007/978-3-031-33309-5_5 10.1109/TG.2022.3175795 10.1016/j.asoc.2024.111305 10.1109/ICRA48506.2021.9561593 10.1109/TCIAIG.2013.2290371 10.4204/EPTCS.325.38 10.1016/j.patcog.2023.110238 10.1007/978-3-031-37189-9_39 10.1109/TG.2021.3076368 10.1016/j.compag.2024.109162 10.1109/ACCESS.2023.3313725 10.1109/tnnls.2023.3348422 10.1007/s10489-020-01758-5 10.1109/TCDS.2023.3287987 10.1145/3649921.3659844 10.1093/logcom/exad032 10.1109/TCDS.2022.3218940 10.1016/j.ifacol.2023.10.1688 10.1109/CoG52621.2021.9619131 10.1007/s11042-023-14945-6 10.1007/s11432-021-3347-8 10.1007/978-3-642-20589-7_14 10.1109/TPAMI.2022.3185549 10.1109/TG.2020.3046133 10.5220/0007257904110418 10.1016/j.neucom.2022.10.058 10.1260/147807708784640081 10.1111/tops.12143 10.1109/ICTAI59109.2023.00038 10.1109/TCDS.2022.3231055 10.1007/s10994-023-06311-2 10.4271/2017-01-0237 10.1145/3459991 10.1073/pnas.1611835114 10.1613/jair.1.13673 10.1109/TCIAIG.2011.2158545 10.1115/1.4046293 10.1613/jair.1.14174 10.1016/j.ins.2022.11.051 10.1016/j.inffus.2019.12.004 10.1007/s10489-020-01786-1 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TCDS.2024.3485482 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Anatomy & Physiology |
| EISSN | 2379-8939 |
| EndPage | 553 |
| ExternalDocumentID | 10_1109_TCDS_2024_3485482 10736545 |
| Genre | orig-research |
| GroupedDBID | 0R~ 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABJNI ABQJQ ABVLG ACGFS AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF M43 O9- OCL RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c246t-3542076bbde96e97b3e76b9387c56e7a462f241d14f0cf643ca9464d2d9abf093 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001502494600009&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2379-8920 |
| IngestDate | Mon Jun 30 07:38:13 EDT 2025 Sat Nov 29 07:53:45 EST 2025 Wed Jun 11 06:03:45 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c246t-3542076bbde96e97b3e76b9387c56e7a462f241d14f0cf643ca9464d2d9abf093 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0009-0006-5684-2751 0000-0001-5696-3007 |
| PQID | 3215953131 |
| PQPubID | 85513 |
| PageCount | 14 |
| ParticipantIDs | proquest_journals_3215953131 crossref_primary_10_1109_TCDS_2024_3485482 ieee_primary_10736545 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-06-01 |
| PublicationDateYYYYMMDD | 2025-06-01 |
| PublicationDate_xml | – month: 06 year: 2025 text: 2025-06-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | Piscataway |
| PublicationPlace_xml | – name: Piscataway |
| PublicationTitle | IEEE transactions on cognitive and developmental systems |
| PublicationTitleAbbrev | TCDS |
| PublicationYear | 2025 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref15 ref14 ref11 ref10 ref17 ref16 ref19 ref18 ref46 ref45 ref47 ref42 ref41 ref43 Clavera (ref5) 1803 ref8 ref7 ref9 ref4 ref3 ref6 ref35 ref34 ref36 ref31 ref30 ref33 ref32 Chesani (ref44); 2214 Sutton (ref1) 2018 ref2 ref39 Justesen (ref50) 2018 Berthier (ref40) 2013 Rolnick (ref48) 2019; 32 ref24 ref23 ref26 ref25 ref20 ref22 ref21 ref28 ref27 Glorian (ref37) 2021; 210 ref29 Schwarz (ref49) 2018; 80 Smith (ref38) 2023 |
| References_xml | – ident: ref14 doi: 10.1007/978-3-031-33309-5_5 – ident: ref22 doi: 10.1109/TG.2022.3175795 – ident: ref8 doi: 10.1016/j.asoc.2024.111305 – volume: 210 start-page: 1 volume-title: Proc. 27th Int. Conf. Princ. Pract. Constraint Program. (CP) year: 2021 ident: ref37 article-title: The dungeon variations problem using constraint programming – ident: ref11 doi: 10.1109/ICRA48506.2021.9561593 – ident: ref21 doi: 10.1109/TCIAIG.2013.2290371 – ident: ref41 doi: 10.4204/EPTCS.325.38 – ident: ref16 doi: 10.1016/j.patcog.2023.110238 – ident: ref35 doi: 10.1007/978-3-031-37189-9_39 – ident: ref24 doi: 10.1109/TG.2021.3076368 – ident: ref45 doi: 10.1016/j.compag.2024.109162 – volume: 32 start-page: 350 year: 2019 ident: ref48 article-title: Experience replay for continual learning publication-title: Adv. Neur. Inf. Process. Syst. – ident: ref25 doi: 10.1109/ACCESS.2023.3313725 – ident: ref33 doi: 10.1109/tnnls.2023.3348422 – ident: ref9 doi: 10.1007/s10489-020-01758-5 – ident: ref2 doi: 10.1109/TCDS.2023.3287987 – volume: 80 start-page: 4528 volume-title: Int. Conf. Mach. Learn. year: 2018 ident: ref49 article-title: Progress and compress: A scalable framework for continual learning – ident: ref47 doi: 10.1145/3649921.3659844 – ident: ref13 doi: 10.1093/logcom/exad032 – ident: ref26 doi: 10.1109/TCDS.2022.3218940 – ident: ref43 doi: 10.1016/j.ifacol.2023.10.1688 – ident: ref27 doi: 10.1109/CoG52621.2021.9619131 – ident: ref6 doi: 10.1007/s11042-023-14945-6 – ident: ref7 doi: 10.1007/s11432-021-3347-8 – ident: ref46 doi: 10.1007/978-3-642-20589-7_14 – ident: ref28 doi: 10.1109/TPAMI.2022.3185549 – ident: ref36 doi: 10.1109/TG.2020.3046133 – ident: ref39 doi: 10.5220/0007257904110418 – ident: ref31 doi: 10.1016/j.neucom.2022.10.058 – ident: ref34 doi: 10.1260/147807708784640081 – ident: ref4 doi: 10.1111/tops.12143 – ident: ref30 doi: 10.1109/ICTAI59109.2023.00038 – ident: ref17 doi: 10.1109/TCDS.2022.3231055 – ident: ref29 doi: 10.1007/s10994-023-06311-2 – year: 2018 ident: ref50 article-title: Illuminating generalization in deep reinforcement learning through procedural level generation – ident: ref42 doi: 10.4271/2017-01-0237 – ident: ref3 doi: 10.1145/3459991 – ident: ref19 doi: 10.1073/pnas.1611835114 – ident: ref20 doi: 10.1613/jair.1.13673 – ident: ref23 doi: 10.1109/TCIAIG.2011.2158545 – ident: ref12 doi: 10.1115/1.4046293 – volume-title: An Introduction year: 2018 ident: ref1 article-title: Reinforcement learning: An introduction – issue: 11347 year: 1803 ident: ref5 article-title: Learning to adapt: Meta-learning for model-based control – ident: ref10 doi: 10.1613/jair.1.14174 – ident: ref32 doi: 10.1016/j.ins.2022.11.051 – ident: ref15 doi: 10.1016/j.inffus.2019.12.004 – year: 2023 ident: ref38 article-title: Procedural Constraint-Based Generation for Game Development – volume: 2214 start-page: 2018 volume-title: Proc. 33rd Italian Conf. Comput. Log. ident: ref44 article-title: A decision support system for food recycling based on constraint logic programming and ontological reasoning – ident: ref18 doi: 10.1007/s10489-020-01786-1 – year: 2013 ident: ref40 article-title: Pattern-based constraint satisfaction and logic puzzles |
| SSID | ssj0001637847 |
| Score | 2.3447504 |
| Snippet | In this article, we introduce a novel framework that combines constraint logic programming (CLP) with deep reinforcement learning (DRL) to create adaptive... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Index Database Publisher |
| StartPage | 540 |
| SubjectTerms | Adaptive environments Complexity Complexity theory constraint logic programming Constraints continual learning Continuing education Deep learning Dynamic scheduling Indexes Logic Logic programming Problem solving procedural content generation Procedural generation reinforcement learning Schedules Scheduling Training Trajectory |
| Title | Adaptive Environment Generation for Continual Learning: Integrating Constraint Logic Programming With Deep Reinforcement Learning |
| URI | https://ieeexplore.ieee.org/document/10736545 https://www.proquest.com/docview/3215953131 |
| Volume | 17 |
| WOSCitedRecordID | wos001502494600009&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2379-8939 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001637847 issn: 2379-8920 databaseCode: RIE dateStart: 20160101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3LS8MwGA9uePDiaxOnU3IQD0K1TdKk8Tb2QEFEfKC30iRfdYfV4TrBo_-5Sdr5QDxILw1pS8ivyffI9_0-hA4AYhoqkgUhUXHARBwHirlIAKOtOu9kLPeJwhfi8jJ5eJBXdbK6z4UBAB98Bsfu1p_lm2c9d64yu8IF5VbkN1BDCFEla305VDgViS8oRqiQQSLJ4hQzCuXJbX9wY61Bwo4pS6yWTn7IIV9Y5ddu7EXMaO2fg1tHq7UuiXsV-BtoCYpN1OoV1o6evOFD7KM7vdu8hd57Jpu6rQ0Pv1LbcEU67bDBVnnFjqpq7DhKcU27-niKz2s-Cdtw_TNfU6LErkazxldVdNfEdd6Pyyc8AJjia_B0rNp7Hj8_1UZ3o-Ft_yyoyy8EmjBeBjRmJBRcKQOSgxSKgm1JmggdcxAZ4yS38t9ELA91bjUbnUnGmSFGZioPJd1CzeK5gG2EGU8SYJSZiOYszlTCTK7tFTlyHW6gg44WYKTTimUj9dZJKFOHXOqQS2vkOqjtZv_bg9XEd1B3gV9aL8RZSq1KI-0-Q6OdP17bRSvE1fT1npUuapYvc9hDy_q1HM9e9v0_9gH9CdCC |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1JS8QwFH64gV7cxXHNQTwI1TZJk8bb4ILiOIiO6K20yavOYcbBqYJH_7lJ2nFBPEgvDWlDydfkLXnvewA7iDELc5oFIc3jgMs4DnLuIgGMtuq8k7HCJwq3ZLud3N-rqzpZ3efCIKIPPsN9d-vP8s2TfnGuMrvCJRNW5I_DZMztiFW61pdLRTCZ-JJilEkVJIqOzjGjUB10jo5vrD1I-T7jidXT6Q9J5Eur_NqPvZA5nfvn583DbK1NkmYF_wKMYX8Rlpp9a0n33sgu8fGd3nG-BO9Nkw3c5kZOvpLbSEU77dAhVn0ljqyq61hKSU28-nBIzmtGCdtw_UNfVaIkrkqzJldVfFfPdd51y0dyjDgg1-gJWbX3PX4OtQy3pyedo7OgLsAQaMpFGbCY01CKPDeoBCqZM7QtxRKpY4Ey44IWVgMwES9CXVjdRmeKC26oUVlehIqtwET_qY-rQLhIEuSMm4gVPM7yhJtC2yty9DrCYAP2RmCkg4pnI_X2SahSh1zqkEtr5Bqw7Gb_24PVxDdgY4RfWi_FYcqsUqPsTsOitT9e24bps85lK22dty_WYYa6Cr_ez7IBE-XzC27ClH4tu8PnLf-_fQD7StPJ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adaptive+Environment+Generation+for+Continual+Learning%3A+Integrating+Constraint+Logic+Programming+With+Deep+Reinforcement+Learning&rft.jtitle=IEEE+transactions+on+cognitive+and+developmental+systems&rft.au=Boutyour%2C+Youness&rft.au=Idrissi%2C+Abdellah&rft.date=2025-06-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=2379-8920&rft.eissn=2379-8939&rft.volume=17&rft.issue=3&rft.spage=540&rft.epage=553&rft_id=info:doi/10.1109%2FTCDS.2024.3485482&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2379-8920&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2379-8920&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2379-8920&client=summon |