A dynamic self-improving ramp metering algorithm based on multi-agent deep reinforcement learning
We present a novel ramp metering algorithm that incorporates multi-agent deep reinforcement learning (DRL) techniques, which utilizes monitoring data from loop detectors. Our proposed approach employed a multi-agent DRL framework to generate optimized ramp metering schedules for each ramp meter in r...
Saved in:
| Published in: | Transportation letters Vol. ahead-of-print; no. ahead-of-print; pp. 1 - 9 |
|---|---|
| Main Authors: | , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
Taylor & Francis
08.08.2024
|
| Subjects: | |
| ISSN: | 1942-7867, 1942-7875 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | We present a novel ramp metering algorithm that incorporates multi-agent deep reinforcement learning (DRL) techniques, which utilizes monitoring data from loop detectors. Our proposed approach employed a multi-agent DRL framework to generate optimized ramp metering schedules for each ramp meter in real-time, enhancing the operational efficiency of urban freeways with less investment. To simplify the implementation and training of the algorithm, we developed a simulation platform based on SUMO microscopic traffic simulator. We conducted a series of simulation experiments, including local and coordinated ramp metering scenarios with various traffic demands profiles. The simulation results indicate that the proposed DRL-based algorithm outperforms the state-of-the-practice ramp metering methods, considering a comprehensive evaluation index encompassing mainstream speed at the bottleneck and queue length on ramp. Additionally, the method exhibits robustness, scalability, and the potential for further improvement through online learning during implementation. |
|---|---|
| AbstractList | We present a novel ramp metering algorithm that incorporates multi-agent deep reinforcement learning (DRL) techniques, which utilizes monitoring data from loop detectors. Our proposed approach employed a multi-agent DRL framework to generate optimized ramp metering schedules for each ramp meter in real-time, enhancing the operational efficiency of urban freeways with less investment. To simplify the implementation and training of the algorithm, we developed a simulation platform based on SUMO microscopic traffic simulator. We conducted a series of simulation experiments, including local and coordinated ramp metering scenarios with various traffic demands profiles. The simulation results indicate that the proposed DRL-based algorithm outperforms the state-of-the-practice ramp metering methods, considering a comprehensive evaluation index encompassing mainstream speed at the bottleneck and queue length on ramp. Additionally, the method exhibits robustness, scalability, and the potential for further improvement through online learning during implementation. |
| Author | Du, Yuchuan Deng, Fuwen Shen, Yu Jin, Jiandong |
| Author_xml | – sequence: 1 givenname: Fuwen orcidid: 0000-0002-1150-682X surname: Deng fullname: Deng, Fuwen email: dengfw@sdtbu.edu.cn organization: Shandong Technology and Business University – sequence: 2 givenname: Jiandong surname: Jin fullname: Jin, Jiandong organization: Peking University – sequence: 3 givenname: Yu surname: Shen fullname: Shen, Yu organization: Tongji University – sequence: 4 givenname: Yuchuan surname: Du fullname: Du, Yuchuan organization: Tongji University |
| BookMark | eNqFkM1KAzEUhYNUsK0-gpAXmJpMZiZT3FiKfyC40fWQSW5qJD8liUrf3hlbXbjQ1b33cL8D58zQxAcPCJ1TsqCkJRd0WZW8bfiiJCVblCWjDWuP0HTUC97yevKzN_wEzVJ6JaRpWkKnSKyw2nnhjMQJrC6M28bwbvwGR-G22EGGOF7CbkI0-cXhXiRQOHjs3mw2hdiAz1gBbHEE43WIEtwoWRDRD-gpOtbCJjg7zDl6vrl-Wt8VD4-39-vVQyEZJbmAvly2qpGgtOKE6JZCw1gLXJRKVLVkZUWBDKeCJR-eK6blkKmndS1V3fRsji73vjKGlCLoTpossgk-R2FsR0k3ttV9t9WNbXWHtga6_kVvo3Ei7v7lrvbcV3QnPkK0qstiZ0PUUXhpUsf-tvgE4LSFqw |
| CitedBy_id | crossref_primary_10_1177_00375497251331487 crossref_primary_10_3390_su162210055 crossref_primary_10_1016_j_trc_2025_105077 crossref_primary_10_1109_TITS_2024_3521460 crossref_primary_10_3390_electronics13234794 crossref_primary_10_1155_atr_2838943 |
| Cites_doi | 10.1016/0191-2615(80)90015-6 10.1109/AHPCAI57455.2022.10087798 10.3141/1634-02 10.1007/978-1-4471-3008-6 10.1016/j.trc.2015.08.016 10.1109/ITSC.2019.8916903 10.1080/19427867.2022.2146302 10.1145/3072959.3073602 10.1016/j.trc.2004.08.001 10.1109/JAS.2016.7508798 10.1038/nature14236 10.1016/j.engappai.2014.01.007 10.1016/j.trc.2019.09.023 10.1016/0191-2607(90)90048-B 10.1287/trsc.6.2.114 10.1109/ICCA.2014.6871097 10.1080/19427867.2018.1477491 10.1109/TITS.2007.908724 10.1002/rnc.5237 10.1016/S0191-2615(03)00012-2 10.1080/19427867.2019.1700005 10.3141/1856-08 10.1049/cp:19940452 10.1109/TITS.2022.3141730 10.1038/nature16961 10.1016/j.trc.2022.103584 10.1109/ITSC.2018.8569938 10.1016/j.trpro.2015.09.070 10.1109/TITS.2018.2790167 10.1016/0041-1647(74)90026-4 10.1155/2021/6669028 10.3141/1603-12 |
| ContentType | Journal Article |
| Copyright | 2023 Informa UK Limited, trading as Taylor & Francis Group 2023 |
| Copyright_xml | – notice: 2023 Informa UK Limited, trading as Taylor & Francis Group 2023 |
| DBID | AAYXX CITATION |
| DOI | 10.1080/19427867.2023.2231638 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Economics |
| EISSN | 1942-7875 |
| EndPage | 9 |
| ExternalDocumentID | 10_1080_19427867_2023_2231638 2231638 |
| Genre | Research Article |
| GroupedDBID | 002 0BK 0R~ 30N 4.4 AAGDL AAHIA AAJMT AALDU AAMIU AAPUL AAQRR ABCCY ABFIM ABLIJ ABPAQ ABPEM ABXUL ABXYU ACGFS ACTIO ADCVX ADGTB ADMLS AEISY AFRVT AGDLA AIJEM AIYEW AKBVH AKOOK ALMA_UNASSIGNED_HOLDINGS ALQZU AQRUH AQTUD AWYRJ BLEHA CCCUG DGEBU DKSSO E01 EBS H13 HCLVR HZ~ IPNFZ KYCEM LJTGL M4Z P75 P7B RIG RNANH ROSJB RTWRZ TASJS TBQAZ TDBHL TEN TFL TFT TFW TTHFI TUROJ ZGOLN AAYXX CITATION |
| ID | FETCH-LOGICAL-c310t-eb298d6cedfd700f81e6338e7a2da45c3241e0e7ade97eb243fc942b155cd56b3 |
| IEDL.DBID | TFW |
| ISICitedReferencesCount | 8 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001018772700001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1942-7867 |
| IngestDate | Sat Nov 29 02:09:03 EST 2025 Tue Nov 18 22:04:11 EST 2025 Mon Oct 20 23:45:42 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | ahead-of-print |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c310t-eb298d6cedfd700f81e6338e7a2da45c3241e0e7ade97eb243fc942b155cd56b3 |
| ORCID | 0000-0002-1150-682X |
| PageCount | 9 |
| ParticipantIDs | informaworld_taylorfrancis_310_1080_19427867_2023_2231638 crossref_citationtrail_10_1080_19427867_2023_2231638 crossref_primary_10_1080_19427867_2023_2231638 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-08-08 |
| PublicationDateYYYYMMDD | 2024-08-08 |
| PublicationDate_xml | – month: 08 year: 2024 text: 2024-08-08 day: 08 |
| PublicationDecade | 2020 |
| PublicationTitle | Transportation letters |
| PublicationYear | 2024 |
| Publisher | Taylor & Francis |
| Publisher_xml | – name: Taylor & Francis |
| References | e_1_3_3_30_1 Mnih V. (e_1_3_3_21_1) 2016 Wattleworth J. A. (e_1_3_3_43_1) 1967; 157 Wu C. (e_1_3_3_44_1) 2018 Schulman J. (e_1_3_3_28_1) 2017 e_1_3_3_18_1 Shalev-Shwartz S. (e_1_3_3_29_1) 2016 e_1_3_3_39_1 e_1_3_3_19_1 e_1_3_3_14_1 Markos Papageorgiou J.-M. B. (e_1_3_3_20_1) 1990; 1320 e_1_3_3_37_1 e_1_3_3_13_1 e_1_3_3_38_1 e_1_3_3_16_1 e_1_3_3_35_1 e_1_3_3_36_1 e_1_3_3_10_1 e_1_3_3_33_1 e_1_3_3_34_1 e_1_3_3_12_1 e_1_3_3_31_1 e_1_3_3_32_1 Wang J. J. (e_1_3_3_41_1) 1973; 469 Lipp L. (e_1_3_3_17_1) 1991; 1320 Wang Z. (e_1_3_3_40_1) 2016 Brockman G. (e_1_3_3_2_1) 2016 Heess N. (e_1_3_3_9_1) 2017 e_1_3_3_7_1 e_1_3_3_6_1 e_1_3_3_8_1 Jacobson L. (e_1_3_3_11_1) 1989; 1732 e_1_3_3_25_1 e_1_3_3_24_1 e_1_3_3_27_1 e_1_3_3_46_1 e_1_3_3_26_1 e_1_3_3_3_1 e_1_3_3_45_1 e_1_3_3_5_1 e_1_3_3_23_1 e_1_3_3_42_1 e_1_3_3_4_1 Kingma D. (e_1_3_3_15_1) 2014; 1412 e_1_3_3_22_1 |
| References_xml | – ident: e_1_3_3_24_1 doi: 10.1016/0191-2615(80)90015-6 – ident: e_1_3_3_45_1 doi: 10.1109/AHPCAI57455.2022.10087798 – volume: 1412 start-page: 6980 year: 2014 ident: e_1_3_3_15_1 article-title: Adam: A Method for Stochastic Optimization publication-title: ArXiv Preprint – ident: e_1_3_3_36_1 doi: 10.3141/1634-02 – ident: e_1_3_3_3_1 doi: 10.1007/978-1-4471-3008-6 – ident: e_1_3_3_13_1 doi: 10.1016/j.trc.2015.08.016 – volume: 157 start-page: 1 year: 1967 ident: e_1_3_3_43_1 article-title: Peak Period Analysis and Control of a Freeway System with Discussion publication-title: Highway Research Record – ident: e_1_3_3_5_1 doi: 10.1109/ITSC.2019.8916903 – ident: e_1_3_3_23_1 doi: 10.1080/19427867.2022.2146302 – ident: e_1_3_3_37_1 doi: 10.1145/3072959.3073602 – ident: e_1_3_3_10_1 doi: 10.1016/j.trc.2004.08.001 – volume: 469 start-page: 16 year: 1973 ident: e_1_3_3_41_1 article-title: Computer Model for Optimal Freeway On-Ramp Control publication-title: Highway Research Record – ident: e_1_3_3_16_1 doi: 10.1109/JAS.2016.7508798 – volume: 1320 start-page: 3 year: 1991 ident: e_1_3_3_17_1 article-title: Benefits of Central Computer Control for Denver Ramp-Metering System publication-title: Transportation Research Record – ident: e_1_3_3_22_1 doi: 10.1038/nature14236 – year: 2016 ident: e_1_3_3_29_1 article-title: Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving publication-title: ArXiv Preprint – volume-title: IEEE Conference on Intelligent Transportation Systems (ITSC) year: 2018 ident: e_1_3_3_44_1 – volume: 1732 start-page: 20 year: 1989 ident: e_1_3_3_11_1 article-title: Real-Time Metering Algorithm for Centralized Control publication-title: Transportation Research Record – ident: e_1_3_3_14_1 doi: 10.1016/j.engappai.2014.01.007 – ident: e_1_3_3_7_1 doi: 10.1016/j.trc.2019.09.023 – ident: e_1_3_3_25_1 doi: 10.1016/0191-2607(90)90048-B – ident: e_1_3_3_39_1 doi: 10.1287/trsc.6.2.114 – year: 2016 ident: e_1_3_3_2_1 article-title: Openai Gym publication-title: ArXiv Preprint – ident: e_1_3_3_6_1 doi: 10.1109/ICCA.2014.6871097 – ident: e_1_3_3_46_1 doi: 10.1080/19427867.2018.1477491 – ident: e_1_3_3_27_1 doi: 10.1109/TITS.2007.908724 – ident: e_1_3_3_35_1 doi: 10.1002/rnc.5237 – ident: e_1_3_3_32_1 doi: 10.1016/S0191-2615(03)00012-2 – ident: e_1_3_3_34_1 doi: 10.1080/19427867.2019.1700005 – ident: e_1_3_3_31_1 doi: 10.3141/1856-08 – ident: e_1_3_3_33_1 doi: 10.1049/cp:19940452 – ident: e_1_3_3_42_1 doi: 10.1109/TITS.2022.3141730 – ident: e_1_3_3_30_1 doi: 10.1038/nature16961 – ident: e_1_3_3_8_1 doi: 10.1016/j.trc.2022.103584 – volume: 1320 start-page: 58 year: 1990 ident: e_1_3_3_20_1 article-title: ALINEA a Local Feedback Control Law for on Ramp Metering publication-title: Transportation Research Reccord – volume-title: International Conference on Machine Learning year: 2016 ident: e_1_3_3_40_1 – ident: e_1_3_3_19_1 doi: 10.1109/ITSC.2018.8569938 – year: 2017 ident: e_1_3_3_28_1 article-title: Proximal Policy Optimization Algorithms publication-title: ArXiv Preprint – ident: e_1_3_3_12_1 doi: 10.1016/j.trpro.2015.09.070 – ident: e_1_3_3_38_1 doi: 10.1109/TITS.2018.2790167 – ident: e_1_3_3_4_1 doi: 10.1016/0041-1647(74)90026-4 – volume-title: International Conference on Machine Learning year: 2016 ident: e_1_3_3_21_1 – year: 2017 ident: e_1_3_3_9_1 article-title: Emergence of Locomotion Behaviours in Rich Environments publication-title: ArXiv Preprint – ident: e_1_3_3_18_1 doi: 10.1155/2021/6669028 – ident: e_1_3_3_26_1 doi: 10.3141/1603-12 |
| SSID | ssj0066801 |
| Score | 2.3508277 |
| Snippet | We present a novel ramp metering algorithm that incorporates multi-agent deep reinforcement learning (DRL) techniques, which utilizes monitoring data from loop... |
| SourceID | crossref informaworld |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 1 |
| SubjectTerms | adaptive control deep reinforcement learning ramp metering |
| Title | A dynamic self-improving ramp metering algorithm based on multi-agent deep reinforcement learning |
| URI | https://www.tandfonline.com/doi/abs/10.1080/19427867.2023.2231638 |
| Volume | ahead-of-print |
| WOSCitedRecordID | wos001018772700001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAWR databaseName: Taylor & Francis Journals Complete customDbUrl: eissn: 1942-7875 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0066801 issn: 1942-7867 databaseCode: TFW dateStart: 20090101 isFulltext: true titleUrlDefault: https://www.tandfonline.com providerName: Taylor & Francis |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV07T8MwELZQhQQLb0R5yQOrS5yXnbFCVEwVQxHdIsePUqlNqyTw-zk7DmoHYIDxpFwUX-y776y7-xC6i1hhGFUJ4RE1BPCtIRmERUKV0JCOKZlR48gm2HjMp9Ps2VcT1r6s0ubQph0U4Xy1PdyiqLuKuHvIu0PGUzaw1N8DiG8WU4AXhtBvj-Zk9Nr54jTljgDZahCr0vXwfPeWrei0Nbt0I-qMDv_he4_QgYeceNjukWO0o8sTtNd1JNenSAyxaonpca0Xhsy7iwZcieUaL23FjJXEYraq5s3bEtvYp_CqxK4ekQjbn4WV1mtcabcc6W4dseekmJ2hl9Hj5OGJeOoFIgHvNQTy7YyrVGplFAsCw6lOIZnVTIRKxIkEGEZ1AKLSGYOH48hIWGUB6ESqJC2ic9QrV6W-QJgDphNCBRKSy1hqXvDUcBMymhVRYXTYR3Fn8lz6ueSWHmORUz--tLNfbu2Xe_v10eBLbd0O5vhNIdv8n3njbkRMS1-SRz_qXv5B9wrtgxi7okF-jXpN9a5v0K78aOZ1deu26ydZy-aw |
| linkProvider | Taylor & Francis |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV07T8MwELagIJWFN6I8PbC61HnZGRGiKqJ0KqJb5PhRKrVplQZ-P2cnQe0ADDBazlnxxfF9d7q7D6Ebn6WGURUS7lNDAN8aEoNZJFQJDe6YkjE1jmyCDQZ8NIpXa2FsWqX1oU3ZKMLd1fbntsHoOiXuFhxvj_GItS33dxsMnAUVm2grBFtr--cPu6_1bRxF3FEgWxFiZeoqnu-WWbNPa91LV-xOd-8_3ngf7VaoE9-Vx-QAbejsEDXrouTlERJ3WJXc9Hipp4ZM6lgDzsVsgWc2acaOxHQ8zyfF2wxb86fwPMMuJZEIW6KFldYLnGu3H-kCj7iipRgfo5fuw_C-Ryr2BSIB8hUEXO6Yq0hqZRTrdAynOgJ_VjPhKRGEEpAY1R0YKh0zeDjwjYRdpgBQpAqj1D9BjWye6VOEOcA6IVRHgn8ZSM1THhluPEbj1E-N9looqHWeyKo1uWXImCa06mBa6y-x-ksq_bVQ-0tsUfbm-E0gXv2gSeGCIqZkMEn8H2XP_iB7jZq94XM_6T8Ons7RDkwFLoeQX6BGkb_rS7QtP4rJMr9yZ_cTAyTq2g |
| linkToPdf | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LT8MwDI5gIODCGzGeOXDN6GtNepyACQSadhhityrNAyZtXdUWfj9O2qLtABzgGLWuGteNP1u2P4SufJpo6souYb6rCeBbTSJwi8SVXEE4JkXkaks2QQcDNh5Hw7qasKjLKk0MratBEfasNj93JnVTEXcNcbdHWUg7hvq7A_7NYIpVtAbQOTRGPuq_NIdxGDLLgGxEiJFpmni-e8ySe1oaXrrgdvo7__DCu2i7xpy4VxnJHlpR6T7abFqSiwPEe1hWzPS4UFNNJk2mAed8luGZKZkxKz59neeT8m2GjfOTeJ5iW5BIuGnQwlKpDOfKbkfYtCOuSSleD9Fz_250c09q7gUiAPCVBALuiMlQKKkldRzNXBVCNKso9yQPugJwmKscWEoVUbg58LWAXSYAT4Tshol_hFrpPFXHCDMAdZxLR0B0GQjFEhZqpj3qRomfaOW1UdCoPBb1YHLDjzGN3Xp-aaO_2OgvrvXXRp0vsayazPGbQLT4PePSpkR0xV8S-z_KnvxB9hJtDG_78dPD4PEUbcGVwBYQsjPUKvN3dY7WxUc5KfILa7mfSoTpjA |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+dynamic+self-improving+ramp+metering+algorithm+based+on+multi-agent+deep+reinforcement+learning&rft.jtitle=Transportation+letters&rft.au=Deng%2C+Fuwen&rft.au=Jin%2C+Jiandong&rft.au=Shen%2C+Yu&rft.au=Du%2C+Yuchuan&rft.date=2024-08-08&rft.pub=Taylor+%26+Francis&rft.issn=1942-7867&rft.eissn=1942-7875&rft.volume=ahead-of-print&rft.issue=ahead-of-print&rft.spage=1&rft.epage=9&rft_id=info:doi/10.1080%2F19427867.2023.2231638&rft.externalDocID=2231638 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1942-7867&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1942-7867&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1942-7867&client=summon |