AoI Minimization in Status Update Control With Energy Harvesting Sensors

Information freshness is crucial for time-critical IoT applications, e.g., monitoring and control. We consider an IoT status update system with users, energy harvesting sensors, and a cache-enabled edge node. The users receive time-sensitive information about physical quantities, each measured by a...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on communications Ročník 69; číslo 12; s. 8335 - 8351
Hlavní autori: Hatami, Mohammad, Leinonen, Markus, Codreanu, Marian
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York IEEE 01.12.2021
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
ISSN:0090-6778, 1558-0857, 1558-0857
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Abstract Information freshness is crucial for time-critical IoT applications, e.g., monitoring and control. We consider an IoT status update system with users, energy harvesting sensors, and a cache-enabled edge node. The users receive time-sensitive information about physical quantities, each measured by a sensor. Users demand for the information from the edge node whose cache stores the most recently received measurements from each sensor. To serve a request, the edge node either commands the sensor to send an update or retrieves the aged measurement from the cache. We aim at finding the best actions of the edge node to minimize the average AoI of the served measurements at the users, termed on-demand AoI. We model this problem as a Markov decision process and develop reinforcement learning (RL) algorithms: model-based value iteration and model-free Q-learning. We also propose a Q-learning method for the realistic case where the edge node is informed about the sensors' battery levels only via the status updates. The case under transmission limitations is also addressed. Furthermore, properties of an optimal policy are characterized. Simulation results show that an optimal policy is a threshold-based policy and that the proposed RL methods significantly reduce the average cost compared to several baselines.
AbstractList Information freshness is crucial for time-critical IoT applications, e.g., monitoring and control. We consider an IoT status update system with users, energy harvesting sensors, and a cache-enabled edge node. The users receive time-sensitive information about physical quantities, each measured by a sensor. Users demand for the information from the edge node whose cache stores the most recently received measurements from each sensor. To serve a request, the edge node either commands the sensor to send an update or retrieves the aged measurement from the cache. We aim at finding the best actions of the edge node to minimize the average AoI of the served measurements at the users, termed on-demand AoI. We model this problem as a Markov decision process and develop reinforcement learning (RL) algorithms: model-based value iteration and model-free Q-learning. We also propose a Q-learning method for the realistic case where the edge node is informed about the sensors battery levels only via the status updates. The case under transmission limitations is also addressed. Furthermore, properties of an optimal policy are characterized. Simulation results show that an optimal policy is a threshold-based policy and that the proposed RL methods significantly reduce the average cost compared to several baselines.
Author Hatami, Mohammad
Leinonen, Markus
Codreanu, Marian
Author_xml – sequence: 1
  givenname: Mohammad
  orcidid: 0000-0002-0730-7668
  surname: Hatami
  fullname: Hatami, Mohammad
  email: mohammad.hatami@oulu.fi
  organization: Centre for Wireless Communications—Radio Technologies, University of Oulu, Oulu, Finland
– sequence: 2
  givenname: Markus
  orcidid: 0000-0002-5639-3144
  surname: Leinonen
  fullname: Leinonen, Markus
  email: markus.leinonen@oulu.fi
  organization: Centre for Wireless Communications—Radio Technologies, University of Oulu, Oulu, Finland
– sequence: 3
  givenname: Marian
  orcidid: 0000-0003-0210-4375
  surname: Codreanu
  fullname: Codreanu, Marian
  email: marian.codreanu@liu.se
  organization: Department of Science and Technology, Linköping University, Linköping, Sweden
BackLink https://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-182230$$DView record from Swedish Publication Index (Linköpings universitet)
BookMark eNp9kE9PwjAchhuDiYB-Ab008Tzsv23dkUwUEogHUI9NNzosGS22nQY_veNPOHjw9Lu8z_t78_RAx1ijALjFaIAxyh4W-ctsNiCI4AHFmCUcX4AujmMeIR6nHdBFKENRkqb8CvS8XyOEGKK0C8ZDO4EzbfRG_8igrYHawHmQofHwdbuUQcHcmuBsDd91-IAjo9xqB8fSfSkftFnBuTLeOn8NLitZe3Vzun2weBot8nE0fXme5MNpVNI0DRHhZSnTSmZVUdIl5RkrsCKMFJIrKTHOlllJCUsJlxgVFeNKJbyiEsW0wGVM-yA61vpvtW0KsXV6I91OWKnFo34bCutWotaNwJwQitr8_TG_dfazaSeLtW2caRcKkrTmOOGMtSlyTJXOeu9Ude7FSOwFi4NgsRcsToJbiP-BSh0ODoOTuv4fvTuiWil1_pXFLEkzQn8BpG-LBg
CODEN IECMBT
CitedBy_id crossref_primary_10_1109_TCOMM_2022_3208873
crossref_primary_10_1109_TGCN_2024_3484132
crossref_primary_10_1109_TCOMM_2024_3524026
crossref_primary_10_1109_TGCN_2024_3479460
crossref_primary_10_3390_e26121018
crossref_primary_10_1016_j_suscom_2025_101114
crossref_primary_10_1109_TCCN_2022_3214240
crossref_primary_10_1109_TCOMM_2024_3367712
crossref_primary_10_1109_TCCN_2024_3408462
crossref_primary_10_1109_LWC_2022_3193186
crossref_primary_10_1016_j_dcan_2024_08_015
crossref_primary_10_1109_TITS_2024_3472033
crossref_primary_10_1109_TVT_2023_3336547
crossref_primary_10_1109_ACCESS_2023_3299340
crossref_primary_10_1109_TVT_2023_3336824
crossref_primary_10_1109_JSAC_2024_3431518
crossref_primary_10_1109_JIOT_2023_3339514
crossref_primary_10_3390_s23249687
crossref_primary_10_1109_COMST_2023_3329472
crossref_primary_10_1109_TMC_2025_3527559
crossref_primary_10_1109_TCOMM_2024_3524036
crossref_primary_10_1109_TCOMM_2023_3282256
crossref_primary_10_1109_TWC_2024_3520324
crossref_primary_10_1186_s42162_024_00425_0
crossref_primary_10_1109_TMC_2024_3356229
crossref_primary_10_1109_TNET_2023_3270388
crossref_primary_10_1109_TWC_2023_3259721
crossref_primary_10_1109_ACCESS_2024_3525263
crossref_primary_10_1109_JIOT_2023_3324879
crossref_primary_10_1016_j_engappai_2025_112210
crossref_primary_10_1109_JIOT_2024_3403849
crossref_primary_10_1109_TWC_2023_3278460
crossref_primary_10_1109_JIOT_2023_3283549
crossref_primary_10_1109_TII_2024_3475422
crossref_primary_10_1109_TCOMM_2022_3141786
crossref_primary_10_1109_JIOT_2023_3300921
crossref_primary_10_1002_ett_4586
crossref_primary_10_1109_TCOMM_2024_3471969
Cites_doi 10.1109/TIT.2018.2871079
10.1109/SURV.2011.060710.00094
10.1109/TII.2014.2300753
10.1109/LWC.2018.2890605
10.1109/TMC.2019.2936199
10.1109/PIMRC48278.2020.9217302
10.1109/JPROC.2014.2357031
10.1109/TWC.2019.2920351
10.1109/TNET.2020.3041654
10.1002/9780470980033
10.1109/INFOCOMWKSHPS50562.2020.9162973
10.1109/INFCOM.2012.6195689
10.1109/ISIT.2017.8006506
10.1109/TCOMM.2020.2991992
10.1109/ICC.2016.7511440
10.1109/GCWkshps45667.2019.9024463
10.1287/moor.24.2.293
10.1561/1300000060
10.1109/ISIT.2019.8849636
10.1109/TCOMM.2019.2931538
10.2200/S00954ED2V01Y201909CNT023
10.1109/TIT.2016.2533395
10.1109/LSP.2017.2686377
10.1109/JIOT.2018.2882583
10.1109/INFCOMW.2019.8845182
10.1109/PIMRC.2019.8904358
10.1109/INFOCOM.2019.8737508
10.1109/ISIT.2015.7283009
10.1109/TIT.2019.2938969
10.1109/ITA.2015.7308962
10.1109/TCOMM.2013.052013.120565
10.1109/IEEECONF44664.2019.9048659
10.1109/TCCN.2019.2916097
10.1109/ACCESS.2019.2899001
10.1109/COMST.2020.2988367
10.1109/CISS48834.2020.1570629110
10.1109/GLOBECOM38437.2019.9013454
10.1109/TGCN.2017.2778501
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2021
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2021
DBID 97E
ESBDL
RIA
RIE
AAYXX
CITATION
7SP
8FD
L7M
ABXSW
ADTPV
AOWAS
D8T
DG8
ZZAVC
DOI 10.1109/TCOMM.2021.3114681
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005–Present
IEEE Xplore Open Access Journals
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Electronics & Communications Abstracts
Technology Research Database
Advanced Technologies Database with Aerospace
SWEPUB Linköpings universitet full text
SwePub
SwePub Articles
SWEPUB Freely available online
SWEPUB Linköpings universitet
SwePub Articles full text
DatabaseTitle CrossRef
Technology Research Database
Advanced Technologies Database with Aerospace
Electronics & Communications Abstracts
DatabaseTitleList
Technology Research Database

Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL) (UW System Shared)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 1558-0857
EndPage 8351
ExternalDocumentID oai_DiVA_org_liu_182230
10_1109_TCOMM_2021_3114681
9546792
Genre orig-research
GrantInformation_xml – fundername: HPY Research Foundation and Riitta ja Jorma J. Takanen Foundation
– fundername: Infotech Oulu, the Academy of Finland
  grantid: 323698
  funderid: 10.13039/501100018948
– fundername: Academy of Finland
  grantid: 319485
  funderid: 10.13039/501100002341
– fundername: Academy of Finland 6Genesis Flagship
  grantid: 318927
  funderid: 10.13039/501100002341
– fundername: European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement
  grantid: 793402 (COMPRESS NETS)
GroupedDBID -~X
.DC
0R~
29I
3EH
4.4
5GY
5VS
6IK
85S
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABFSI
ABQJQ
ABVLG
ACGFO
ACGFS
ACIWK
ACKIV
ACNCT
AENEX
AETIX
AGQYO
AGSQL
AHBIQ
AI.
AIBXA
AKJIK
AKQYR
ALLEH
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
E.L
EBS
EJD
ESBDL
HZ~
H~9
IAAWW
IBMZZ
ICLAB
IES
IFIPE
IFJZH
IPLJI
JAVBF
LAI
M43
MS~
O9-
OCL
P2P
RIA
RIE
RNS
TAE
TN5
VH1
ZCA
ZCG
AAYXX
CITATION
7SP
8FD
L7M
ABXSW
ADTPV
AOWAS
D8T
DG8
ZZAVC
ID FETCH-LOGICAL-c377t-28cca7fa9fbc3d3894b1e242ba8eaa119d9c324728a10bf48ee68f3a053b1c53
IEDL.DBID RIE
ISICitedReferencesCount 60
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000731147500035&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 0090-6778
1558-0857
IngestDate Tue Nov 04 16:56:11 EST 2025
Mon Jun 30 10:13:27 EDT 2025
Tue Nov 18 21:32:52 EST 2025
Sat Nov 29 04:08:22 EST 2025
Wed Aug 27 03:02:53 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 12
Keywords Internet of Things (IoT)
Q-learning
dynamic programming
Minimization
value iteration algorithm (VIA)
Sensor systems
Batteries
Energy harvesting
reinforcement learning (RL)
Wireless sensor networks
age of information (AoI)
Sensors
Monitoring
Language English
License https://creativecommons.org/licenses/by/4.0/legalcode
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c377t-28cca7fa9fbc3d3894b1e242ba8eaa119d9c324728a10bf48ee68f3a053b1c53
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0002-0730-7668
0000-0002-5639-3144
0000-0003-0210-4375
OpenAccessLink https://ieeexplore.ieee.org/document/9546792
PQID 2610982844
PQPubID 85472
PageCount 17
ParticipantIDs crossref_primary_10_1109_TCOMM_2021_3114681
proquest_journals_2610982844
crossref_citationtrail_10_1109_TCOMM_2021_3114681
swepub_primary_oai_DiVA_org_liu_182230
ieee_primary_9546792
PublicationCentury 2000
PublicationDate 2021-12-01
PublicationDateYYYYMMDD 2021-12-01
PublicationDate_xml – month: 12
  year: 2021
  text: 2021-12-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on communications
PublicationTitleAbbrev TCOMM
PublicationYear 2021
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref35
ref34
ref12
bastopcu (ref18) 2020
ref37
ref15
ref36
ref14
ref31
ref30
ref33
ref11
ref32
ref10
ref2
ref39
ref17
ref16
ref19
mankar (ref13) 2020
da xu (ref1) 2014; 10
ref24
ref23
ref26
ref25
ref20
ref41
ref22
ref21
ref28
ref27
ref29
ref8
ref7
sutton (ref38) 2018
ref9
ref4
ref3
ref6
ref5
ref40
References_xml – ident: ref8
  doi: 10.1109/TIT.2018.2871079
– ident: ref5
  doi: 10.1109/SURV.2011.060710.00094
– volume: 10
  start-page: 2233
  year: 2014
  ident: ref1
  article-title: Internet of Things in industries: A survey
  publication-title: IEEE Trans Ind Informat
  doi: 10.1109/TII.2014.2300753
– ident: ref16
  doi: 10.1109/LWC.2018.2890605
– year: 2020
  ident: ref13
  article-title: Throughput and age of information in a cellular-based IoT network
  publication-title: arXiv 2005 09547
– ident: ref39
  doi: 10.1109/TMC.2019.2936199
– ident: ref36
  doi: 10.1109/PIMRC48278.2020.9217302
– ident: ref6
  doi: 10.1109/JPROC.2014.2357031
– ident: ref22
  doi: 10.1109/TWC.2019.2920351
– ident: ref35
  doi: 10.1109/TNET.2020.3041654
– ident: ref41
  doi: 10.1002/9780470980033
– ident: ref12
  doi: 10.1109/INFOCOMWKSHPS50562.2020.9162973
– ident: ref7
  doi: 10.1109/INFCOM.2012.6195689
– ident: ref17
  doi: 10.1109/ISIT.2017.8006506
– ident: ref27
  doi: 10.1109/TCOMM.2020.2991992
– ident: ref14
  doi: 10.1109/ICC.2016.7511440
– ident: ref31
  doi: 10.1109/GCWkshps45667.2019.9024463
– ident: ref40
  doi: 10.1287/moor.24.2.293
– year: 2018
  ident: ref38
  publication-title: Reinforcement Learning An Introduction
– ident: ref11
  doi: 10.1561/1300000060
– ident: ref21
  doi: 10.1109/ISIT.2019.8849636
– ident: ref26
  doi: 10.1109/TCOMM.2019.2931538
– ident: ref10
  doi: 10.2200/S00954ED2V01Y201909CNT023
– ident: ref9
  doi: 10.1109/TIT.2016.2533395
– ident: ref2
  doi: 10.1109/LSP.2017.2686377
– ident: ref24
  doi: 10.1109/JIOT.2018.2882583
– ident: ref32
  doi: 10.1109/INFCOMW.2019.8845182
– ident: ref3
  doi: 10.1109/PIMRC.2019.8904358
– ident: ref34
  doi: 10.1109/INFOCOM.2019.8737508
– ident: ref19
  doi: 10.1109/ISIT.2015.7283009
– ident: ref23
  doi: 10.1109/TIT.2019.2938969
– ident: ref28
  doi: 10.1109/ITA.2015.7308962
– ident: ref37
  doi: 10.1109/TCOMM.2013.052013.120565
– ident: ref29
  doi: 10.1109/IEEECONF44664.2019.9048659
– year: 2020
  ident: ref18
  article-title: Information freshness in cache updating systems
  publication-title: arXiv 2004 09475
– ident: ref30
  doi: 10.1109/TCCN.2019.2916097
– ident: ref4
  doi: 10.1109/ACCESS.2019.2899001
– ident: ref33
  doi: 10.1109/COMST.2020.2988367
– ident: ref15
  doi: 10.1109/CISS48834.2020.1570629110
– ident: ref25
  doi: 10.1109/GLOBECOM38437.2019.9013454
– ident: ref20
  doi: 10.1109/TGCN.2017.2778501
SSID ssj0004033
Score 2.6367784
Snippet Information freshness is crucial for time-critical IoT applications, e.g., monitoring and control. We consider an IoT status update system with users, energy...
SourceID swepub
proquest
crossref
ieee
SourceType Open Access Repository
Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 8335
SubjectTerms age of information (AoI)
Algorithms
Batteries
dynamic programming
Energy harvesting
Internet of Things (IoT)
Iterative methods
Machine learning
Markov processes
Minimization
Monitoring
Nodes
Optimization
Q-learning
reinforcement learning (RL)
Sensor systems
Sensors
value iteration algorithm (VIA)
Wireless sensor networks
Title AoI Minimization in Status Update Control With Energy Harvesting Sensors
URI https://ieeexplore.ieee.org/document/9546792
https://www.proquest.com/docview/2610982844
https://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-182230
Volume 69
WOSCitedRecordID wos000731147500035&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE/IET Electronic Library (IEL) (UW System Shared)
  customDbUrl:
  eissn: 1558-0857
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0004033
  issn: 1558-0857
  databaseCode: RIE
  dateStart: 19720101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTuQwEC0xiANzYJ0RzSYfRlyYQJzN9rHVgECiGSR6gJvlpTITCdKoF74f20k3ICEkbjk4keVnp-ol9V4B_EIXlbAsbUQp5lGGaRIJjUVUqNhQo4o0CT4Ft5fs6orf34vrBfg918IgYig-wyN_Gf7l26GZ-k9lxyJ3x1q4F-43xopGq_WqgYzT1nHSl7MzPhPIxOJ40PvT7zsqmFDHUL3UiL4LQqGryvsE861paAg0Z6tfm-IarLQJJek2O2AdFrDegO9vbAY34bw7vCD9qq4eW9ElqWris8zpmPx98pSf9JqKdXJXTf6T06AHJL5tkPfgqP-RG0d2h6PxDxicnQ5651HbQSEyKWOTKOEOIFYqUWqTWpebZJqiC8pacVSKUmGFcRkVS7iisS4zjljwMlXuZGpq8vQnLNbDGreAxEILtJpbK4rMYs5VwmxWGKE543mRdoDOVlSa1l3cN7l4kIFlxEIGFKRHQbYodOBwfs9T463x6ehNv9zzke1Kd2B3Bpxsj99YJt5E3nHJLOvAQQPm_D7vp31S3Xalw00-VFPpGJbjYdsfP34Hlv0kmgKWXVicjKa4B0vmeVKNR_thC74Al-TZIQ
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NT9wwEB0hWqn0UEoBdQulPiAubSDOp31cbUGLursgdQvcLH9MIBLNov3o76_teBeQqkrccnAiy8_OzEvmvQE4RBuVsKpMRCnmUYZpEnGFRVTIWFMtizTxPgVXg3I0Yjc3_HINvq20MIjoi8_w2F36f_lmohfuU9kJz-2x5vaF-8p1zgpqrUcVZJwGz0lX0F6ypUQm5ifj3sVwaMlgQi1HdWIj-iwM-b4qz1PMp7ahPtScbb5sku_hXUgpSbfdA1uwhs0HePvEaHAb-t3JORnWTf07yC5J3RCXZy5m5NeDI_2k19ask-t6fkdOvSKQuMZBzoWjuSU_Ld2dTGc7MD47Hff6UeihEOm0LOdRwixEZSV5pXRqbHaSKYo2LCvJUEpKueHa5lRlwiSNVZUxxIJVqbRnU1Gdp7uw3kwa_Agk5oqjUcwYXmQGcyaT0mSF5oqVLC_SDtDligod_MVdm4t74XlGzIVHQTgUREChA19X9zy07hr_Hb3tlns1Mqx0B_aXwIlwAGcicTbylk1mWQeOWjBX9zlH7e_1VVdY3MR9vRCWY1km9unfj_8Cb_rj4UAMzkc_9mDDTagtZ9mH9fl0gZ_htf4zr2fTA78d_wIPzNxq
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AoI+Minimization+in+Status+Update+Control+With+Energy+Harvesting+Sensors&rft.jtitle=IEEE+transactions+on+communications&rft.au=Hatami%2C+Mohammad&rft.au=Leinonen%2C+Markus&rft.au=Codreanu%2C+Marian&rft.date=2021-12-01&rft.pub=IEEE&rft.issn=0090-6778&rft.volume=69&rft.issue=12&rft.spage=8335&rft.epage=8351&rft_id=info:doi/10.1109%2FTCOMM.2021.3114681&rft.externalDocID=9546792
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0090-6778&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0090-6778&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0090-6778&client=summon