EnosLib: A Library for Experiment-Driven Research in Distributed Computing

Despite the importance of experiment-driven research in the distributed computing community, there has been little progress in helping researchers conduct their experiments. In most cases, they have to achieve tedious and time-consuming development and instrumentation activities to deal with the spe...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on parallel and distributed systems Vol. 33; no. 6; pp. 1464 - 1477
Main Authors: Cherrueau, Ronan-Alexandre, Delavergne, Marie, van Kempen, Alexandre, Lebre, Adrien, Pertin, Dimitri, Balderrama, Javier Rojas, Simonet, Anthony, Simonin, Matthieu
Format: Journal Article
Language:English
Published: New York IEEE 01.06.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Institute of Electrical and Electronics Engineers
Subjects:
Benchmark testing
Codes
Computer networks
Computer Science
distributed computing experimentation library
Distributed processing
Distributed, Parallel, and Cluster Computing
Experiment-driven research
Experimentation
Experiments
Libraries
Networking and Internet Architecture
Other
performance evaluation
Protocols
Software
Source code
Task analysis
Performance evaluation
Experiment-driven research
Distributed computing experimentation library
ISSN:1045-9219, 1558-2183
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Despite the importance of experiment-driven research in the distributed computing community, there has been little progress in helping researchers conduct their experiments. In most cases, they have to achieve tedious and time-consuming development and instrumentation activities to deal with the specifics of testbeds and the system under study. In order to relieve researchers of the burden of those efforts, we have developed EnosLib : a Python library that takes into account best experimentation practices and leverages modern toolkits on automatic deployment and configuration systems. EnosLib helps researchers not only in the process of developing their experimental artifacts, but also in running them over different infrastructures. To demonstrate the relevance of our library, we discuss three experimental engines built on top of EnosLib , and used to conduct empirical studies on complex software stacks between 2016 and 2019 (database systems, communication buses and OpenStack). By introducing EnosLib , our goal is to gather academic and industrial actors of our community around a library that aggregates everyday experiment-driven research operations. A library that has been already adopted by open-source projects and members of the scientific community thanks to its ease of use and extension.
AbstractList Despite the importance of experiment-driven research in the distributed computing community, there has been little progress in helping researchers conduct their experiments. In most cases, they have to achieve tedious and time-consuming development and instrumentation activities to deal with the specifics of testbeds and the system under study. In order to relieve researchers of the burden of those efforts, we have developed ENOSLIB: a Python library that takes into account best experimentation practices and leverages modern toolkits on automatic deployment and configuration systems. ENOSLIB helps researchers not only in the process of developing their experimental artifacts, but also in running them over different infrastructures. To demonstrate the relevance of our library, we discuss three experimental engines built on top of ENOSLIB, and used to conduct empirical studies on complex software stacks between 2016 and 2019 (database systems, communication buses and OpenStack). By introducing ENOSLIB, our goal is to gather academic and industrial actors of our community around a library that aggregates everyday experiment-driven research operations. A library that has been already adopted by open-source projects and members of the scientific community thanks to its ease of use and extension.
Author Simonin, Matthieu
Balderrama, Javier Rojas
Simonet, Anthony
van Kempen, Alexandre
Cherrueau, Ronan-Alexandre
Lebre, Adrien
Delavergne, Marie
Pertin, Dimitri
Author_xml – sequence: 1
  givenname: Ronan-Alexandre
  surname: Cherrueau
  fullname: Cherrueau, Ronan-Alexandre
  email: Ronan-Alexandre.Cherrueau@inria.fr
  organization: Inria, LS2N, IMT Atlantique, Nantes, France
– sequence: 2
  givenname: Marie
  orcidid: 0000-0001-8020-959X
  surname: Delavergne
  fullname: Delavergne, Marie
  email: Marie.Delavergne.@inria.fr
  organization: Inria, LS2N, IMT Atlantique, Nantes, France
– sequence: 3
  givenname: Alexandre
  surname: van Kempen
  fullname: van Kempen, Alexandre
  email: Alexandre.Kempen@inria.fr
  organization: Inria, LS2N, IMT Atlantique, Nantes, France
– sequence: 4
  givenname: Adrien
  orcidid: 0000-0002-0305-4130
  surname: Lebre
  fullname: Lebre, Adrien
  email: Adrien.Lebre@inria.fr
  organization: Inria, LS2N, IMT Atlantique, Nantes, France
– sequence: 5
  givenname: Dimitri
  surname: Pertin
  fullname: Pertin, Dimitri
  email: Dimitri.Pertin@inria.fr
  organization: Inria, LS2N, IMT Atlantique, Nantes, France
– sequence: 6
  givenname: Javier Rojas
  orcidid: 0000-0002-3341-2525
  surname: Balderrama
  fullname: Balderrama, Javier Rojas
  email: Javier.Balderrama@inria.fr
  organization: Inria, LS2N, IMT Atlantique, Nantes, France
– sequence: 7
  givenname: Anthony
  orcidid: 0000-0002-4072-8886
  surname: Simonet
  fullname: Simonet, Anthony
  email: asb@iex.ec
  organization: iExec Blockchain Tech, Lyon, Auvergne-Rhône-Alpes, France
– sequence: 8
  givenname: Matthieu
  surname: Simonin
  fullname: Simonin, Matthieu
  email: matthieu.simonin@inria.fr
  organization: Inria Rennes - Bretagne Atlantique, Rennes, France
BackLink https://inria.hal.science/hal-03324177$$DView record in HAL
BookMark eNo9kE9Lw0AQxRdRsK1-APES8OQhdf9kk11vpa1WCShaz8smmdgtbRJ3k6Lf3g0pncsbht8bHm-Mzqu6AoRuCJ4SguXD-n3xOaWYkikjfrg8QyPCuQgpEezc7zjioaREXqKxc1uMScRxNEKvy6p2qckeg1ngxWr7F5S1DZa_DVizh6oNF9YcoAo-wIG2-SYwVbAwrrUm61oognm9b7rWVN9X6KLUOwfXR52gr6fler4K07fnl_ksDXNGaRvSnENeFCDKTGaYi4QCjSRQnBdERiyRQpO4lEQkBWihWSFxmWc5BSGY4DpmE3Q__N3onWp8SJ9Z1dqo1SxV_Q0zRiOSJAfi2buBbWz904Fr1bbubOXjKcqFwEkUS-wpMlC5rZ2zUJ7eEqz6elVfr-rrVcd6ved28BgAOPGSsyj2Qf8B7vx2lA
CODEN ITDSEO
Cites_doi 10.1201/9781351036863-5
10.1145/2723872.2723882
10.1109/GCE.2010.5676126
10.1109/WF-IoT.2015.7389098
10.1145/3297858.3304013
10.1145/62138.62141
10.5220/0007721103630370
10.1145/2479871.2479892
10.1109/CLUSTER49012.2020.00028
10.1007/978-3-540-39671-0_4
10.1109/ICSR.1998.685738
10.1109/CCGrid49817.2020.00-81
10.1145/1897852.1897868
10.1145/2509578.2514738
10.1109/CloudCom.2013.119
10.1145/2908080.2908083
10.1109/CCGrid.2014.33
10.1371/journal.pcbi.1003285
10.1109/INFCOMW.2018.8407024
10.1109/IPDPSW.2017.157
10.1109/FiCloud.2017.51
10.1145/3318216.3363299
10.1145/361598.361623
10.1109/CloudCom2018.2018.00023
10.1145/2882903.2899401
10.1145/2723872.2723883
10.1109/TPDS.2019.2907950
10.1109/PDP.2013.32
10.1145/2815400.2815401
10.1109/CCGRID.2017.87
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
Attribution
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
– notice: Attribution
DBID 97E
RIA
RIE
AAYXX
CITATION
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
1XC
VOOES
DOI 10.1109/TPDS.2021.3111159
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Technology Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
Hyper Article en Ligne (HAL)
Hyper Article en Ligne (HAL) (Open Access)
DatabaseTitle CrossRef
Technology Research Database
Computer and Information Systems Abstracts – Academic
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts Professional
DatabaseTitleList

Technology Research Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
Computer Science
EISSN 1558-2183
EndPage 1477
ExternalDocumentID oai:HAL:hal-03324177v1
10_1109_TPDS_2021_3111159
9534688
Genre orig-research
GrantInformation_xml – fundername: Institut national de recherche en informatique et en automatique
  funderid: 10.13039/100012950
– fundername: Inria and Orange Labs
– fundername: CNRS
– fundername: RENATER
GroupedDBID --Z
-~X
.DC
0R~
29I
4.4
5GY
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFO
ACIWK
AENEX
AGQYO
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EBS
EJD
HZ~
IEDLZ
IFIPE
IPLJI
JAVBF
LAI
M43
MS~
O9-
OCL
P2P
PQQKQ
RIA
RIE
RNS
TN5
TWZ
UHB
AAYXX
CITATION
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
1XC
VOOES
ID FETCH-LOGICAL-c322t-2c5ecdde8fb9b05872e249e20cd1943798a16f9187dea8a3d90fcbc2e88385a63
IEDL.DBID RIE
ISICitedReferencesCount 14
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000712560900003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 1045-9219
IngestDate Tue Oct 14 19:57:06 EDT 2025
Mon Jun 30 06:56:06 EDT 2025
Sat Nov 29 06:06:48 EST 2025
Wed Aug 27 02:28:36 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 6
Keywords Performance evaluation
Experiment-driven research
Distributed computing experimentation library
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
Attribution: http://creativecommons.org/licenses/by
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c322t-2c5ecdde8fb9b05872e249e20cd1943798a16f9187dea8a3d90fcbc2e88385a63
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0002-0305-4130
0000-0002-3341-2525
0000-0002-4072-8886
0000-0001-8020-959X
OpenAccessLink https://inria.hal.science/hal-03324177
PQID 2588074690
PQPubID 85437
PageCount 14
ParticipantIDs hal_primary_oai_HAL_hal_03324177v1
crossref_primary_10_1109_TPDS_2021_3111159
proquest_journals_2588074690
ieee_primary_9534688
PublicationCentury 2000
PublicationDate 2022-06-01
PublicationDateYYYYMMDD 2022-06-01
PublicationDate_xml – month: 06
  year: 2022
  text: 2022-06-01
  day: 01
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on parallel and distributed systems
PublicationTitleAbbrev TPDS
PublicationYear 2022
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Institute of Electrical and Electronics Engineers
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
– name: Institute of Electrical and Electronics Engineers
References ref35
ref13
ref34
ref12
rojas balderrama (ref32) 2020
ref37
hemminger (ref18) 2005
ref36
ref14
hochstein (ref19) 2017
ref30
stoller (ref42) 2008
ref33
ref11
ref10
delavergne (ref15) 2018
cherrueau (ref8) 2017
desprez (ref16) 2012
ref2
ref39
ref17
ref38
papadopoulos (ref29) 2020
ricci (ref31) 2014; 39
ref24
ref23
ref26
ref20
ref22
ref44
ref21
ref43
(ref1) 2018
ref28
manaouil (ref25) 2021
simonin (ref41) 2016
ref7
ref9
ref3
maricq (ref27) 2018
ref6
ref5
ref40
balouek (ref4) 2012
References_xml – year: 2018
  ident: ref1
– ident: ref23
  doi: 10.1201/9781351036863-5
– start-page: 409
  year: 2018
  ident: ref27
  article-title: Taming performance variability
  publication-title: Proc 13th USENIX Symp Operating Syst Des Implementation
– ident: ref5
  doi: 10.1145/2723872.2723882
– ident: ref44
  doi: 10.1109/GCE.2010.5676126
– start-page: 93
  year: 2020
  ident: ref29
  article-title: Methodological principles for reproducible performance evaluation in cloud computing
  publication-title: Software Engineering Fachtagung des GI-Fachbereichs Softwaretechnik
– ident: ref2
  doi: 10.1109/WF-IoT.2015.7389098
– ident: ref17
  doi: 10.1145/3297858.3304013
– ident: ref24
  doi: 10.1145/62138.62141
– ident: ref39
  doi: 10.5220/0007721103630370
– year: 2020
  ident: ref32
  article-title: Identify potential bottlenecks in the neutron API, plan a performance testing and more, OpenStack Superuser blog
– ident: ref14
  doi: 10.1145/2479871.2479892
– ident: ref34
  doi: 10.1109/CLUSTER49012.2020.00028
– ident: ref12
  doi: 10.1007/978-3-540-39671-0_4
– ident: ref20
  doi: 10.1109/ICSR.1998.685738
– ident: ref37
  doi: 10.1109/CCGrid49817.2020.00-81
– ident: ref7
  doi: 10.1145/1897852.1897868
– year: 2012
  ident: ref16
  article-title: Supporting experimental computer science
– start-page: 113
  year: 2008
  ident: ref42
  article-title: Large-scale virtualization in the emulab network testbed
  publication-title: Proc USENIX Annu Tech Conf
– ident: ref3
  doi: 10.1145/2509578.2514738
– ident: ref21
  doi: 10.1109/CloudCom.2013.119
– ident: ref40
  doi: 10.1145/2908080.2908083
– ident: ref6
  doi: 10.1109/CCGrid.2014.33
– ident: ref36
  doi: 10.1371/journal.pcbi.1003285
– ident: ref10
  doi: 10.1109/INFCOMW.2018.8407024
– ident: ref22
  doi: 10.1109/IPDPSW.2017.157
– ident: ref13
  doi: 10.1109/FiCloud.2017.51
– ident: ref28
  doi: 10.1145/3318216.3363299
– ident: ref30
  doi: 10.1145/361598.361623
– start-page: 3
  year: 2012
  ident: ref4
  article-title: Adding virtualization capabilities to the Grid'5000 testbed
  publication-title: Proc Int Conf Cloud Comput Services Sci
– year: 2017
  ident: ref19
  publication-title: Ansible Up and Running Automating Configuration Management and Deployment the Easy Way
– ident: ref33
  doi: 10.1109/CloudCom2018.2018.00023
– ident: ref11
  doi: 10.1145/2882903.2899401
– ident: ref35
  doi: 10.1145/2723872.2723883
– year: 2017
  ident: ref8
  article-title: Toward fog, edge, and NFV deployments: Evaluating OpenStack WANwide
– year: 2005
  ident: ref18
  article-title: Network emulation with NetEm
  publication-title: Proc 6th Australia's National Linux Conf
– ident: ref26
  doi: 10.1109/TPDS.2019.2907950
– year: 2016
  ident: ref41
  article-title: Chasing 1000 nodes scale
– start-page: 193
  year: 2021
  ident: ref25
  article-title: Kubernetes wanwide: A deployment scenario to expose and use edge computing resources?
  publication-title: Proc 29th Euromicro Int Conf Parallel Distrib Netw -Based Process
– ident: ref38
  doi: 10.1109/PDP.2013.32
– volume: 39
  start-page: 36
  year: 2014
  ident: ref31
  article-title: Introducing cloudlab: Scientific infrastructure for advancing cloud architectures and applications
  publication-title: Login The Magazine of USENIX and SAGE
– ident: ref43
  doi: 10.1145/2815400.2815401
– year: 2018
  ident: ref15
  article-title: Keystone in the context of fog/edge massively distributed clouds
  publication-title: OpenStack Summit Vancounver Canada
– ident: ref9
  doi: 10.1109/CCGRID.2017.87
SSID ssj0014504
Score 2.4478374
Snippet Despite the importance of experiment-driven research in the distributed computing community, there has been little progress in helping researchers conduct...
SourceID hal
proquest
crossref
ieee
SourceType Open Access Repository
Aggregation Database
Index Database
Publisher
StartPage 1464
SubjectTerms Benchmark testing
Codes
Computer networks
Computer Science
distributed computing experimentation library
Distributed processing
Distributed, Parallel, and Cluster Computing
Experiment-driven research
Experimentation
Experiments
Libraries
Networking and Internet Architecture
Other
performance evaluation
Protocols
Software
Source code
Task analysis
Title EnosLib: A Library for Experiment-Driven Research in Distributed Computing
URI https://ieeexplore.ieee.org/document/9534688
https://www.proquest.com/docview/2588074690
https://inria.hal.science/hal-03324177
Volume 33
WOSCitedRecordID wos000712560900003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 1558-2183
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0014504
  issn: 1045-9219
  databaseCode: RIE
  dateStart: 19900101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB7a4kEPVlvF-GIRT2JsnvvwVmxLkVIKVugtbHY32Esqff1-dzcPCnoxp5BssmG_zOzMzs43AI8qwDzzsXK5yqgbcRy5NDNFBKSIQskiQbBNFJ6Q6ZQuFmzWgOc6F0YpZTefqRdzamP5ciV2Zqmsx-IwwpQ2oUkILnK16ohBFNtSgdq7iF2mxbCMYPoe681ngw_tCQa-dlD1YWhJD-ag5pfZAWlLq_zSx3aSGbX_93lncFoak6hfoH8ODZV3oF0VakCl3Hbg5IB1sAvvw3y1mSzTV9RHZdoC0qYrGtZk_-5gbZQgqrbloWWOBoZh1xTHUhIVPeiXXcDnaDh_G7tlSQVXaMnduoGIldAajWYpS72YkkBp_0sFnpA-M9SElPs4Yz4lUnHKNVpeJlIRKEpDGnMcXkIrX-XqCpAUTArhpZkMSRSHggsZsgwzDb2-xbkDT9UgJ98Fc0ZiPQ6PJQaRxCCSlIg48KBhqNsZzutxf5KYa16obT6fkL3vQNcMet2qHG8HbivUklIAN0kQU0Pzo33_67-fuoHjwGQy2AWVW2ht1zt1B0div11u1vf23_oBbbPMzQ
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD7oFNQH7-K8BvFJrOu9iW_DTabWITjBt5AmKe6lk236-83JsjLQF_tU2rQp-XqSc5Kc7wO41GEqyiDVntAl9WKRxh4tUURAyThSLJZZahOF86zfp-_v7GUJrutcGK213Xymb_DUruWrkfzCqbIWS6I4pXQZVlA5y2Vr1WsGcWLFAk18kXjMGKJbwwx81hq8dF5NLBgGJkQ1BxKTLoxCyx-4B9KKq_zqke0wc7_1vw_chk3nTpL2DP8dWNLVLmzNpRqIs9xd2FjgHdyDx241muTD4pa0iUtcIMZ5Jd2a7t_rjLEbJPONeWRYkQ5y7KI8llZkVoN52T683XcHdz3PiSp40tju1AtloqXp02hZsMJPaBZqE4Hp0JcqYEhOSEWQliygmdKCCoOXX8pChprSiCYijQ6gUY0qfQhESaak9ItSRVmcRFJIFbEyZQZ8c0uIJlzNG5l_zrgzuI05fMYREY6IcIdIEy4MDHU5ZL3utXOO1_zIeH1Bln0HTdjDRq9LufZuwskcNe5McMLDhCLRj4n-j_5-6hzWeoPnnOcP_adjWA8xr8FOr5xAYzr-0qewKr-nw8n4zP5nP0Nj0BY
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EnosLib%3A+A+Library+for+Experiment-Driven+Research+in+Distributed+Computing&rft.jtitle=IEEE+transactions+on+parallel+and+distributed+systems&rft.au=Cherrueau%2C+Ronan-Alexandre&rft.au=Delavergne%2C+Marie&rft.au=van+Kempen%2C+Alexandre&rft.au=Lebre%2C+Adrien&rft.date=2022-06-01&rft.pub=Institute+of+Electrical+and+Electronics+Engineers&rft.issn=1045-9219&rft.volume=33&rft.issue=6&rft.spage=1464&rft.epage=1477&rft_id=info:doi/10.1109%2FTPDS.2021.3111159&rft.externalDBID=HAS_PDF_LINK&rft.externalDocID=oai%3AHAL%3Ahal-03324177v1
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1045-9219&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1045-9219&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1045-9219&client=summon