Robust Coordinated Reinforcement Learning for MAC Design in Sensor Networks
In this paper, we propose a medium access control (MAC) design method for wireless sensor networks based on decentralized coordinated reinforcement learning. Our solution maps the MAC resource allocation problem first to a factor graph, and then, based on the dependencies between sensors, transforms...
Saved in:
| Published in: | IEEE journal on selected areas in communications Vol. 37; no. 10; pp. 2211 - 2224 |
|---|---|
| Main Authors: | , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
IEEE
01.10.2019
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 0733-8716, 1558-0008 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | In this paper, we propose a medium access control (MAC) design method for wireless sensor networks based on decentralized coordinated reinforcement learning. Our solution maps the MAC resource allocation problem first to a factor graph, and then, based on the dependencies between sensors, transforms it into a coordination graph, on which the max-sum algorithm is employed to find the optimal transmission actions for sensors. We have theoretically analyzed the system and determined the convergence guarantees for decentralized coordinated learning in sensor networks. As part of this analysis, we derive a novel sufficient condition for the convergence of max-sum on graphs with cycles and employ it to render the learning process robust. In addition, we reduce the complexity of applying max-sum to our optimization problem by expressing coordination as a multiple knapsack problem (MKP). The complexity of the proposed solution can be, thus, bounded by the capacities of the MKP. Our simulations reveal the benefits coming from adaptivity and sensors' coordination, both inherent in the proposed learning-based MAC. |
|---|---|
| AbstractList | In this paper, we propose a medium access control (MAC) design method for wireless sensor networks based on decentralized coordinated reinforcement learning. Our solution maps the MAC resource allocation problem first to a factor graph, and then, based on the dependencies between sensors, transforms it into a coordination graph, on which the max-sum algorithm is employed to find the optimal transmission actions for sensors. We have theoretically analyzed the system and determined the convergence guarantees for decentralized coordinated learning in sensor networks. As part of this analysis, we derive a novel sufficient condition for the convergence of max-sum on graphs with cycles and employ it to render the learning process robust. In addition, we reduce the complexity of applying max-sum to our optimization problem by expressing coordination as a multiple knapsack problem (MKP). The complexity of the proposed solution can be, thus, bounded by the capacities of the MKP. Our simulations reveal the benefits coming from adaptivity and sensors' coordination, both inherent in the proposed learning-based MAC. |
| Author | Thomos, Nikolaos Nisioti, Eleni |
| Author_xml | – sequence: 1 givenname: Eleni orcidid: 0000-0001-7170-7108 surname: Nisioti fullname: Nisioti, Eleni email: e.nisioti@essex.ac.uk organization: School of Computer Science and Electronic Engineering, University of Essex, Colchester, U.K – sequence: 2 givenname: Nikolaos orcidid: 0000-0001-7266-2642 surname: Thomos fullname: Thomos, Nikolaos email: nthomos@essex.ac.uk organization: School of Computer Science and Electronic Engineering, University of Essex, Colchester, U.K |
| BookMark | eNp9kE1LAzEQhoNUsK3-APGy4HlrJtnuJseyflsVWj0v2c1sSW2TmqSI_94tLR48eBp4mWfe4RmQnnUWCTkHOgKg8upxPilHjIIcMcm5EMUR6cN4LFJKqeiRPi04T0UB-QkZhLCkFLJMsD55mrl6G2JSOue1sSqiTmZobOt8g2u0MZmi8tbYRdJFyfOkTK4xmIVNjE3maEMXvmD8cv4jnJLjVq0Cnh3mkLzf3ryV9-n09e6hnEzTpvstpgXWQrGWMUStaygkw7GiNRcZz3QDBUjMudItF4A8p20j6jxrQTaa6YyxnA_J5f7uxrvPLYZYLd3W266yYkwyKYDDbgv2W413IXhsq403a-W_K6DVzlm1c1btnFUHZx1T_GEaE1U0zkavzOpf8mJPGkT8bRKF5ACS_wBBwnsI |
| CODEN | ISACEM |
| CitedBy_id | crossref_primary_10_1109_ACCESS_2022_3211653 crossref_primary_10_1109_JIOT_2020_3025365 crossref_primary_10_1109_JIOT_2021_3132006 crossref_primary_10_3390_e25010101 crossref_primary_10_1109_TMM_2021_3052339 crossref_primary_10_1007_s00521_022_07515_8 crossref_primary_10_1109_TNSE_2022_3201121 crossref_primary_10_1016_j_comnet_2024_110631 crossref_primary_10_1038_s41598_023_48956_y crossref_primary_10_3390_s21237925 crossref_primary_10_4018_IJSIR_287549 crossref_primary_10_1016_j_comnet_2020_107646 |
| Cites_doi | 10.1109/PIMRC.2018.8580848 10.1109/TCOMM.2010.120710.100054 10.1109/TWC.2007.348337 10.1109/IPSN.2005.1440895 10.1162/089976600300015880 10.1007/11780519_1 10.3233/AIC-2010-0476 10.1109/TIT.2007.909166 10.1145/1015330.1015410 10.1016/S0004-3702(98)00023-X 10.1007/BF00992698 10.1561/2200000001 10.1002/9780470316870 10.1109/TCOM.1983.1095828 10.1109/18.910585 10.1287/mnsc.5.1.97 10.1007/BF01580430 10.1145/1478462.1478502 10.1016/j.artint.2010.11.001 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019 |
| DBID | 97E RIA RIE AAYXX CITATION 7SP 8FD L7M |
| DOI | 10.1109/JSAC.2019.2933887 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Electronics & Communications Abstracts Technology Research Database Advanced Technologies Database with Aerospace |
| DatabaseTitle | CrossRef Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1558-0008 |
| EndPage | 2224 |
| ExternalDocumentID | 10_1109_JSAC_2019_2933887 8793119 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: European Union Horizon 2020 Research and Innovation Programme under the Marie Sklodowska-Curie Research and Innovation Staff Exchange Grant through the project RECENT grantid: 823903 |
| GroupedDBID | -~X .DC 0R~ 29I 3EH 4.4 41~ 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK ACNCT ADRHT AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD HZ~ H~9 IBMZZ ICLAB IES IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P RIA RIE RNS TN5 VH1 AAYXX CITATION 7SP 8FD L7M RIG |
| ID | FETCH-LOGICAL-c293t-7eb8a2f22eeddb1792e5a0b38434dc1719e63adf381e360fc8b64f19cd2d42263 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 14 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000487055400004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0733-8716 |
| IngestDate | Mon Jun 30 10:17:59 EDT 2025 Sat Nov 29 03:23:01 EST 2025 Tue Nov 18 21:25:27 EST 2025 Wed Aug 27 02:42:10 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 10 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c293t-7eb8a2f22eeddb1792e5a0b38434dc1719e63adf381e360fc8b64f19cd2d42263 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0001-7170-7108 0000-0001-7266-2642 |
| PQID | 2292981316 |
| PQPubID | 85481 |
| PageCount | 14 |
| ParticipantIDs | proquest_journals_2292981316 ieee_primary_8793119 crossref_primary_10_1109_JSAC_2019_2933887 crossref_citationtrail_10_1109_JSAC_2019_2933887 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-10-01 |
| PublicationDateYYYYMMDD | 2019-10-01 |
| PublicationDate_xml | – month: 10 year: 2019 text: 2019-10-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE journal on selected areas in communications |
| PublicationTitleAbbrev | J-SAC |
| PublicationYear | 2019 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref14 ref30 murphy (ref20) 1999 ref10 ref2 claus (ref6) 1998 ref19 ref18 guestrin (ref8) 2002 jordan (ref29) 2007; 1 bernardo (ref28) 1994 ref24 ref26 guestrin (ref5) 2002 ref25 ref22 ref21 martello (ref11) 1990 naparstek (ref16) 2017 zhang (ref7) 2011 ref27 kok (ref23) 2006; 7 toni (ref15) 2018 ref9 ref4 ref3 sun (ref1) 2018 challita (ref17) 2017 |
| References_xml | – year: 1990 ident: ref11 publication-title: Knapsack Problems Algorithms and Computer Implementations – ident: ref4 doi: 10.1109/PIMRC.2018.8580848 – start-page: 467 year: 1999 ident: ref20 article-title: Loopy belief propagation for approximate inference: An empirical study publication-title: Proc 15th Conf Uncertainty Artif Intell – ident: ref3 doi: 10.1109/TCOMM.2010.120710.100054 – start-page: 1523 year: 2002 ident: ref5 article-title: Multiagent planning with factored MDPs publication-title: Advances in Neural Information Processing Systems 14 – ident: ref14 doi: 10.1109/TWC.2007.348337 – ident: ref10 doi: 10.1109/IPSN.2005.1440895 – year: 2017 ident: ref16 article-title: Deep multi-user reinforcement learning for distributed dynamic spectrum access publication-title: arXiv 1704 02613 – year: 2018 ident: ref1 article-title: Application of machine learning in wireless networks: Key techniques and open issues publication-title: arXiv 1809 08707 – ident: ref22 doi: 10.1162/089976600300015880 – year: 2017 ident: ref17 article-title: Proactive resource management in LTE-U systems: A deep learning perspective publication-title: arXiv 1702 07031 – start-page: 1 year: 2011 ident: ref7 article-title: Coordinated multi-agent reinforcement learning in networked distributed POMDPs publication-title: Proc 25th Conf Artif Intell (AAAI) – ident: ref19 doi: 10.1007/11780519_1 – ident: ref9 doi: 10.3233/AIC-2010-0476 – start-page: 1 year: 1998 ident: ref6 article-title: The dynamics of reinforcement learning in cooperative multiagent systems publication-title: Proc AAAI – ident: ref12 doi: 10.1109/TIT.2007.909166 – start-page: 227 year: 2002 ident: ref8 article-title: Coordinated reinforcement learning publication-title: Proc ICML – ident: ref18 doi: 10.1145/1015330.1015410 – ident: ref25 doi: 10.1016/S0004-3702(98)00023-X – volume: 7 start-page: 1789 year: 2006 ident: ref23 article-title: Collaborative multiagent reinforcement learning by payoff propagation publication-title: J Mach Learn Res – ident: ref26 doi: 10.1007/BF00992698 – year: 2018 ident: ref15 article-title: IRSA transmission optimization via online learning publication-title: arXiv 1801 09060 – volume: 1 start-page: 1 year: 2007 ident: ref29 article-title: Graphical models, exponential families, and variational inference publication-title: Found Trends Mach Learn doi: 10.1561/2200000001 – year: 1994 ident: ref28 publication-title: Bayesian Theory doi: 10.1002/9780470316870 – ident: ref13 doi: 10.1109/TCOM.1983.1095828 – ident: ref24 doi: 10.1109/18.910585 – ident: ref27 doi: 10.1287/mnsc.5.1.97 – ident: ref30 doi: 10.1007/BF01580430 – ident: ref2 doi: 10.1145/1478462.1478502 – ident: ref21 doi: 10.1016/j.artint.2010.11.001 |
| SSID | ssj0014482 |
| Score | 2.3915107 |
| Snippet | In this paper, we propose a medium access control (MAC) design method for wireless sensor networks based on decentralized coordinated reinforcement learning.... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 2211 |
| SubjectTerms | Access control Algorithms Complexity Complexity theory Computer simulation Convergence Coordination coordination graphs irregular repetition slotted ALOHA Knapsack problem Machine learning max-sum algorithm Media Access Protocol Medium access control Optimization POMDP Q-learning Reinforcement learning Remote sensors Resource allocation Resource management Sensors Throughput Wireless sensor networks |
| Title | Robust Coordinated Reinforcement Learning for MAC Design in Sensor Networks |
| URI | https://ieeexplore.ieee.org/document/8793119 https://www.proquest.com/docview/2292981316 |
| Volume | 37 |
| WOSCitedRecordID | wos000487055400004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-0008 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014482 issn: 0733-8716 databaseCode: RIE dateStart: 19830101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwEA9z-KAPfk1xOiUPPomdTdKl7eOYDlEcsinsrTTpVQbSytr593tJuzJQBN9KSPpxSXr3y939jpArP8alq5nnDESIACWMpaNi8Bw3SKWIuTZclLbYhD-ZBPN5-NIiN00uDADY4DPom0vry09yvTJHZbcBLiZmOD63fF9WuVqNxwBhhvUY-EI4BgTUHkx81u3jbDgyQVxhH3WbsNFzGzrIFlX58Se26mW8_78XOyB7tRlJh9W8H5IWZEdkd4NcsEOeprlaFSUd5YgvFxnalAmdgiVK1fZMkNbcqu8Um-jzcETvbDgHXWR0hugWGydVkHhxTN7G96-jB6cuneBo_MbS8UEFMU85RxWYKNx0HAaxq0TgCS_RzGch4FQkKeprENJNdaCkl7JQJzwxubXihLSzPINTQqXkwFXigjZYkrlqoEDyIE1TQ9cFrEvctTAjXfOKm_IWH5HFF24YGflHRv5RLf8uuW6GfFakGn917hiBNx1rWXdJbz1jUb3tiohztPYCJpg8-33UOdkx966i8XqkXS5XcEG29Ve5KJaXdkV9A10Mx1A |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED_GFNQHv6Y4nZoHn8RuTdL143FMx3RziFPwrTTpVQbSidv8-72kdQwUwbcSEtJekt79cne_A7gIEtq6mntOW0YEUKLEd1SCnuOGmS8ToQ0XpS02EYxG4ctL9FCBq2UuDCLa4DNsmkfry0-nemGuylohbSZuOD7X2p4n3CJba-kzIKBhfQaBlI6BAaUPk2Zr3Y07XRPGFTVJu0kbP7eihWxZlR__Yqtgejv_e7Vd2C4NSdYpVn4PKpjvw9YKvWANBo9TtZjNWXdKCHOSk1WZske0VKna3gqykl31lVETu-902bUN6GCTnI0J31LjqAgTnx3Ac-_mqdt3yuIJjqZvnDsBqjARmRCkBFNFx05gO3GVDD3ppZoHPEJajDQjjY3SdzMdKt_LeKRTkZrsWnkI1Xya4xEw3xcoVOqiNmiSu6qt0BdhlmWGsAt5HdxvYca6ZBY3BS7eYosw3Cg28o-N_ONS_nW4XA55L2g1_upcMwJfdixlXYfG94rF5cGbxUKQvRdyyf3j30edw0b_6X4YD29HgxPYNPMUsXkNqM4_FngK6_pzPpl9nNnd9QXmPsqX |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Robust+Coordinated+Reinforcement+Learning+for+MAC+Design+in+Sensor+Networks&rft.jtitle=IEEE+journal+on+selected+areas+in+communications&rft.au=Nisioti%2C+Eleni&rft.au=Thomos%2C+Nikolaos&rft.date=2019-10-01&rft.issn=0733-8716&rft.eissn=1558-0008&rft.volume=37&rft.issue=10&rft.spage=2211&rft.epage=2224&rft_id=info:doi/10.1109%2FJSAC.2019.2933887&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_JSAC_2019_2933887 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0733-8716&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0733-8716&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0733-8716&client=summon |