A Heuristically Assisted Deep Reinforcement Learning Approach for Network Slice Placement

Network Slice placement with the problem of allocation of resources from a virtualized substrate network is an optimization problem which can be formulated as a multi-objective Integer Linear Programming (ILP) problem. However, to cope with the complexity of such a continuous task and seeking for op...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE eTransactions on network and service management Vol. 19; no. 4; pp. 4794 - 4806
Main Authors:	Alves Esteves, Jose Jurandir, Boubendir, Amina, Guillemin, Fabrice, Sens, Pierre
Format:	Journal Article
Language:	English
Published:	New York IEEE 01.12.2022 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Algorithms automation Bandwidth Computer Science Convergence Deep learning deep reinforcement learning Feature extraction Integer programming large scale Linear programming Machine learning Network slicing Networking and Internet Architecture Optimization Placement Random access memory Reinforcement learning Substrates Training Automation Deep Reinforcement Learning Placement Large Scale Optimization Network Slicing
ISSN:	1932-4537, 1932-4537
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Network Slice placement with the problem of allocation of resources from a virtualized substrate network is an optimization problem which can be formulated as a multi-objective Integer Linear Programming (ILP) problem. However, to cope with the complexity of such a continuous task and seeking for optimality and automation, the use of Machine Learning (ML) techniques appear as a promising approach. We introduce a hybrid placement solution based on Deep Reinforcement Learning (DRL) and a dedicated optimization heuristic based on the "Power of Two Choices" principle. The DRL algorithm uses the so-called Asynchronous Advantage Actor Critic (A3C) algorithm for fast learning, and Graph Convolutional Networks (GCN) to automate feature extraction from the physical substrate network. The proposed Heuristically-Assisted DRL (HA-DRL) allows for the acceleration of the learning process and substantial gain in resource usage when compared against other state-of-the-art approaches, as evidenced by evaluation results.
AbstractList	Network Slice placement with the problem of allocation of resources from a virtualized substrate network is an optimization problem which can be formulated as a multi-objective Integer Linear Programming (ILP) problem. However, to cope with the complexity of such a continuous task and seeking for optimality and automation, the use of Machine Learning (ML) techniques appear as a promising approach. We introduce a hybrid placement solution based on Deep Reinforcement Learning (DRL) and a dedicated optimization heuristic based on the "Power of Two Choices" principle. The DRL algorithm uses the so-called Asynchronous Advantage Actor Critic (A3C) algorithm for fast learning, and Graph Convolutional Networks (GCN) to automate feature extraction from the physical substrate network. The proposed Heuristically-Assisted DRL (HA-DRL) allows for the acceleration of the learning process and substantial gain in resource usage when compared against other state-of-the-art approaches, as evidenced by evaluation results. Network Slice placement with the problem of allocation of resources from a virtualized substrate network is an optimization problem which can be formulated as a multiobjective Integer Linear Programming (ILP) problem. However, to cope with the complexity of such a continuous task and seeking for optimality and automation, the use of Machine Learning (ML) techniques appear as a promising approach. We introduce a hybrid placement solution based on Deep Reinforcement Learning (DRL) and a dedicated optimization heuristic based on the "Power of Two Choices" principle. The DRL algorithm uses the so-called Asynchronous Advantage Actor Critic (A3C) algorithm for fast learning, and Graph Convolutional Networks (GCN) to automate feature extraction from the physical substrate network. The proposed Heuristically-Assisted DRL (HA-DRL) allows for the acceleration of the learning process and substantial gain in resource usage when compared against other state-of-theart approaches, as evidenced by evaluation results.
Author	Guillemin, Fabrice Sens, Pierre Boubendir, Amina Alves Esteves, Jose Jurandir
Author_xml	– sequence: 1 givenname: Jose Jurandir orcidid: 0000-0002-1365-7277 surname: Alves Esteves fullname: Alves Esteves, Jose Jurandir email: josejurandir.alvesesteves@orange.com organization: Orange Labs, Chatilon, France – sequence: 2 givenname: Amina orcidid: 0000-0002-7472-148X surname: Boubendir fullname: Boubendir, Amina email: amina.boubendir@orange.com organization: Orange Labs, Chatilon, France – sequence: 3 givenname: Fabrice orcidid: 0000-0001-5960-2274 surname: Guillemin fullname: Guillemin, Fabrice email: fabrice.guillemin@orange.com organization: Orange Labs, Chatilon, France – sequence: 4 givenname: Pierre orcidid: 0000-0002-5156-7715 surname: Sens fullname: Sens, Pierre email: pierre.sens@lip6.fr organization: LIP6-Inria, Sorbonne University, Paris, France
BackLink	https://hal.science/hal-03954579$$DView record in HAL
BookMark	eNp9kEtLxDAUhYMo-PwB4ibgysWMuUnTpsvia4TxgTMuXIU03mi0tmPSUfz3tlRFXLi6r-9cDmeTrNZNjYTsAhsDsPxwfjm7GHPGYSxAcGBihWxALvgokSJb_dWvk80YnxiTCnK-Qe4KOsFl8LH11lTVBy1i7Aa8p8eIC3qDvnZNsPiCdUunaELt6wdaLBahMfaRdjd6ie17E57prPIW6XVlBnqbrDlTRdz5qlvk9vRkfjQZTa_Ozo-K6ch2PmGESiIY58pSqVLIUjKHDm1pEpfeW2tsaqE0hjuZlRmqxIHlkKRcJFmiMEWxRQ6Gv4-m0ovgX0z40I3xelJMdb9jIpeJzPI36Nj9ge3svy4xtvqpWYa6s6d5loHkTKmeygbKhibGgE5b35rWN3UbjK80MN1nrvvMdZ-5_sq8U8If5beh_zR7g8Yj4g-fp4IrnohPh9WPJA
CODEN	ITNSC4
CitedBy_id	crossref_primary_10_1109_TNSM_2023_3287651 crossref_primary_10_1109_COMST_2024_3390613 crossref_primary_10_1109_TPDS_2023_3310013 crossref_primary_10_1109_TNSM_2024_3454758 crossref_primary_10_1109_JIOT_2023_3264253 crossref_primary_10_1109_TNSM_2024_3437217 crossref_primary_10_1109_TNSM_2023_3258692 crossref_primary_10_1109_MCOM_006_2200534 crossref_primary_10_1109_TCCN_2024_3349452 crossref_primary_10_1002_dac_5463 crossref_primary_10_1109_COMST_2023_3316615 crossref_primary_10_1109_JIOT_2024_3356750 crossref_primary_10_3390_s25103139 crossref_primary_10_1109_COMST_2024_3352910 crossref_primary_10_1007_s10922_023_09773_w crossref_primary_10_1002_ett_5016 crossref_primary_10_1109_TNSM_2022_3175910
Cites_doi	10.23919/CNSM50824.2020.9269099 10.1109/ICCV.2017.43 10.1109/SURV.2013.013013.00155 10.1016/j.neucom.2018.01.025 10.23919/CNSM.2017.8255993 10.1109/COMST.2015.2477041 10.1016/j.ejor.2016.01.003 10.1109/NFV-SDN.2017.8169880 10.1109/CCNC49032.2021.9369463 10.1007/s10846-017-0731-2 10.1109/TCYB.2013.2253094 10.1109/INFCOM.2009.5061987 10.1109/TNSM.2019.2947905 10.1109/71.963420 10.1109/JSAC.2020.2986662 10.1109/CLOUD.2016.0057 10.1109/INFCOMW.2019.8845184 10.1109/COMST.2018.2884835 10.1016/j.ins.2019.05.012 10.1145/3326285.3329056 10.1109/ACCESS.2015.2507158 10.1007/s10732-007-9031-5 10.23919/JCC.2019.12.001 10.1109/CCNC.2018.8319276 10.1109/INFCOMW.2019.8845171 10.1016/j.endm.2016.03.028 10.1109/TNSM.2016.2598420
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 Distributed under a Creative Commons Attribution 4.0 International License
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 – notice: Distributed under a Creative Commons Attribution 4.0 International License
DBID	97E RIA RIE AAYXX CITATION 1XC VOOES
DOI	10.1109/TNSM.2021.3132103
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Hyper Article en Ligne (HAL) Hyper Article en Ligne (HAL) (Open Access)
DatabaseTitle	CrossRef
DatabaseTitleList
Database_xml	– sequence: 1 dbid: RIE name: IEL url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1932-4537
EndPage	4806
ExternalDocumentID	oai:HAL:hal-03954579v1 10_1109_TNSM_2021_3132103 9632824
Genre	orig-research
GrantInformation_xml	– fundername: European Commission funderid: 10.13039/501100000780
GroupedDBID	0R~ 4.4 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABJNI ABQJQ ABVLG ACGFO ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AIBXA AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 EBS EJD HZ~ IES IFIPE IPLJI JAVBF LAI M43 O9- OCL P2P RIA RIE AAYXX CITATION 1XC VOOES
ID	FETCH-LOGICAL-c3211-e85e1affbb88b35b50fefecba4f6dccac6c1baa2f57b7e84f1c2146234748e6e3
IEDL.DBID	RIE
ISSN	1932-4537
IngestDate	Tue Oct 14 20:44:27 EDT 2025 Sun Nov 30 04:16:37 EST 2025 Sat Nov 29 03:13:15 EST 2025 Tue Nov 18 22:30:52 EST 2025 Wed Aug 27 02:18:08 EDT 2025
IsDoiOpenAccess	true
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	4
Keywords	Automation Deep Reinforcement Learning Placement Large Scale Optimization Network Slicing
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 Distributed under a Creative Commons Attribution 4.0 International License: http://creativecommons.org/licenses/by/4.0
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c3211-e85e1affbb88b35b50fefecba4f6dccac6c1baa2f57b7e84f1c2146234748e6e3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-1365-7277 0000-0001-5960-2274 0000-0002-7472-148X 0000-0002-5156-7715
OpenAccessLink	https://hal.science/hal-03954579
PQID	2771520881
PQPubID	85504
PageCount	13
ParticipantIDs	hal_primary_oai_HAL_hal_03954579v1 crossref_citationtrail_10_1109_TNSM_2021_3132103 proquest_journals_2771520881 crossref_primary_10_1109_TNSM_2021_3132103 ieee_primary_9632824
PublicationCentury	2000
PublicationDate	2022-12-01
PublicationDateYYYYMMDD	2022-12-01
PublicationDate_xml	– month: 12 year: 2022 text: 2022-12-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE eTransactions on network and service management
PublicationTitleAbbrev	T-NSM
PublicationYear	2022
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref35 ref12 ref15 ref14 ref31 ref30 ref11 ref33 ref10 ref32 (ref2) 2020 ref1 ref17 ref16 ref19 ref18 Fischer (ref4) 2011 Mnih (ref24) ref25 ref20 ref21 Bianchi (ref26); 242 Defferrard (ref22) ref28 ref27 ref29 ref8 Sutton (ref34) 2015 ref7 ref9 (ref3) 2017 Kipf (ref23) ref6 ref5
References_xml	– ident: ref10 doi: 10.23919/CNSM50824.2020.9269099 – ident: ref27 doi: 10.1109/ICCV.2017.43 – ident: ref8 doi: 10.1109/SURV.2013.013013.00155 – ident: ref13 doi: 10.1016/j.neucom.2018.01.025 – ident: ref16 doi: 10.23919/CNSM.2017.8255993 – ident: ref1 doi: 10.1109/COMST.2015.2477041 – ident: ref32 doi: 10.1016/j.ejor.2016.01.003 – ident: ref35 doi: 10.1109/NFV-SDN.2017.8169880 – ident: ref17 doi: 10.1109/CCNC49032.2021.9369463 – ident: ref29 doi: 10.1007/s10846-017-0731-2 – volume-title: Reinforcement learning: An introduction year: 2015 ident: ref34 – ident: ref28 doi: 10.1109/TCYB.2013.2253094 – ident: ref15 doi: 10.1109/INFCOM.2009.5061987 – ident: ref20 doi: 10.1109/TNSM.2019.2947905 – ident: ref33 doi: 10.1109/71.963420 – year: 2020 ident: ref2 article-title: Management and orchestration; 5G Network Resource Model (NRM); Stage 2 and stage 3 (Release 17) – ident: ref11 doi: 10.1109/JSAC.2020.2986662 – ident: ref14 doi: 10.1109/CLOUD.2016.0057 – volume: 242 start-page: 169 volume-title: Proc. 20th Eur. Conf. Artif. Intell. ident: ref26 article-title: Heuristically accelerated reinforcement learning: Theoretical and experimental results – start-page: 1 year: 2011 ident: ref4 article-title: ALEVIN–A framework to develop, compare, and analyze virtual network embedding algorithms publication-title: Open Access J. Electron. Commun. EASST – start-page: 1928 volume-title: Proc. Int. Conf. Mach. Learn. ident: ref24 article-title: Asynchronous methods for deep reinforcement learning – ident: ref21 doi: 10.1109/INFCOMW.2019.8845184 – start-page: 1 volume-title: Proc. 5th Int. Conf. Learn. Represent. (ICLR) ident: ref23 article-title: Semi-supervised classification with graph convolutional networks – year: 2017 ident: ref3 article-title: Network functions Virtualisation (NFV); evolution and ecosystem; report on network slicing support – ident: ref7 doi: 10.1109/COMST.2018.2884835 – ident: ref18 doi: 10.1016/j.ins.2019.05.012 – ident: ref19 doi: 10.1145/3326285.3329056 – ident: ref30 doi: 10.1109/ACCESS.2015.2507158 – ident: ref25 doi: 10.1007/s10732-007-9031-5 – ident: ref5 doi: 10.23919/JCC.2019.12.001 – start-page: 3844 volume-title: Proc. 30th Int. Conf. Neural Process. Syst. (NIPS) ident: ref22 article-title: Convolutional neural networks on graphs with fast localized spectral filtering – ident: ref31 doi: 10.1109/CCNC.2018.8319276 – ident: ref12 doi: 10.1109/INFCOMW.2019.8845171 – ident: ref9 doi: 10.1016/j.endm.2016.03.028 – ident: ref6 doi: 10.1109/TNSM.2016.2598420
SSID	ssj0058192
Score	2.404942
Snippet	Network Slice placement with the problem of allocation of resources from a virtualized substrate network is an optimization problem which can be formulated as...
SourceID	hal proquest crossref ieee
SourceType	Open Access Repository Aggregation Database Enrichment Source Index Database Publisher
StartPage	4794
SubjectTerms	Algorithms automation Bandwidth Computer Science Convergence Deep learning deep reinforcement learning Feature extraction Integer programming large scale Linear programming Machine learning Network slicing Networking and Internet Architecture Optimization Placement Random access memory Reinforcement learning Substrates Training
Title	A Heuristically Assisted Deep Reinforcement Learning Approach for Network Slice Placement
URI	https://ieeexplore.ieee.org/document/9632824 https://www.proquest.com/docview/2771520881 https://hal.science/hal-03954579
Volume	19
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEL customDbUrl: eissn: 1932-4537 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0058192 issn: 1932-4537 databaseCode: RIE dateStart: 20040101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NaxsxEB1ik0N7yEfTEidOEKGn0k1Wu1pLOpp84ENiTJ1CelokeZQEjBPi2JB_H41WNoWWQm_L7gh29bSjGWn0HsBXXRmpNLdZXhiRiZAyZzqvVFaVuStsSYoiPopNyOFQ3d3p0QZ8X5-FQcRYfIandBn38idPbkFLZWdhsIQMQbSgJWWvOau18roVEXulXUue67Pb4fgmZH8FPyVuQr7SxErzTuuBqh6jnMofPjhOLFfb__dKO7CVAkjWbxDfhQ2cfYKPv9EK7sGvPhvgoqFgNtPpGwsYEJoTdoH4zH5gZEt1cWGQJYLVe9ZP7OIsPGPDpjqcjafBkbARrbWT9Wf4eXV5ez7IkoRC5sJX8wxVhdx4b61SoedtlXv06KwRvjcJ4Lme49aYwlfSSlTCc0dK30UppFDYw_ILtGdPM9wHJqWZeC-NsNaLwjqtsdQh3uRWhU5A1YF81cG1S_ziJHMxrWOekeuaMKkJkzph0oFv6ybPDbnGv4xPAmprO6LFHvSva7qXlzoEglIveQf2CKO1VYKnA90VyHX6R-d1IWUIXoKX5Qd_b3UIHwo67BCLV7rQfn1Z4BFsuuXr4_zlOA6_d1YO2Lk
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1RSxwxEB7UFtQH22rFa20bSp_E1c1u9pI8Hlq50nOReoI-hSQ3aQvHKeoJ_ffN5HJHQRH6tuxOYDdfdjKTTL4P4IturFSau6KsrChETJkLXTaqaOrSV64mRZGQxCZk26rLS322BPuLszCImIrP8IAu017-6NpPaansMA6WmCGIZXhByln5tNbc7zZE7ZX3LXmpD4ft-WnM_yp-QOyEfK6KlWee5V9U95gEVR554TS1nLz6v5d6DRs5hGS9GeZvYAknm7D-D7HgFlz1WB-nMxJmOx7_YREFwnPEjhFv2A9MfKk-LQ2yTLH6k_UyvziLz1g7qw9n5-PoStgZrbaT9Vu4OPk6POoXWUSh8PGreYGqQW5DcE6p2PeuKQMG9M6K0B1F-HzXc2dtFRrpJCoRuCet76oWUijsYr0NK5PrCe4Ak9KOQpBWOBdE5bzWWOsYcXKnYieg6kA572DjM8M4CV2MTco0Sm0IE0OYmIxJB_YWTW5m9BrPGX-OqC3siBi73xsYulfWOoaCUj_wDmwRRgurDE8Hducgm_yX3plKyhi-RD_L3z3d6hOs9oenAzP41n5_D2sVHX1IpSy7sHJ_O8UP8NI_3P--u_2YhuJfxMfcAg
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Heuristically+Assisted+Deep+Reinforcement+Learning+Approach+for+Network+Slice+Placement&rft.jtitle=IEEE+eTransactions+on+network+and+service+management&rft.au=Alves+Esteves%2C+Jose+Jurandir&rft.au=Boubendir%2C+Amina&rft.au=Guillemin%2C+Fabrice&rft.au=Sens%2C+Pierre&rft.date=2022-12-01&rft.issn=1932-4537&rft.eissn=1932-4537&rft.volume=19&rft.issue=4&rft.spage=4794&rft.epage=4806&rft_id=info:doi/10.1109%2FTNSM.2021.3132103&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TNSM_2021_3132103
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1932-4537&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1932-4537&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1932-4537&client=summon