Dynamic Provisioning and Execution of HPC Workflows Using Python

High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:2016 6th Workshop on Python for High-Performance and Scientific Computing (PyHPC) s. 1 - 8
Hlavní autori: Harris, Chris, O'Leary, Patrick, Grauer, Michael, Chaudhary, Aashish, Kotfila, Chris, O'Bara, Robert
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: IEEE 01.11.2016
Predmet:
Cloud computing
Complexity theory
Dynamic scheduling
Electronic mail
Open source software
Structural beams
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Abstract High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In this paper, we describe the development of the Python-based cumulus, which addresses many of these barriers. cumulus is a platform for the dynamic provisioning and execution of HPC workflows. cumulus provides the infrastructure needed to build applications that leverage traditional or Cloud-based HPC resources in their workflows. Finally, we demonstrate the use of cumulus in both web and desktop simulation applications, as well as in an Apache Spark-based analysis application.
AbstractList High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In this paper, we describe the development of the Python-based cumulus, which addresses many of these barriers. cumulus is a platform for the dynamic provisioning and execution of HPC workflows. cumulus provides the infrastructure needed to build applications that leverage traditional or Cloud-based HPC resources in their workflows. Finally, we demonstrate the use of cumulus in both web and desktop simulation applications, as well as in an Apache Spark-based analysis application.
Author Grauer, Michael
Chaudhary, Aashish
Harris, Chris
O'Bara, Robert
Kotfila, Chris
O'Leary, Patrick
Author_xml – sequence: 1
  givenname: Chris
  surname: Harris
  fullname: Harris, Chris
  email: chris.harris@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 2
  givenname: Patrick
  surname: O'Leary
  fullname: O'Leary, Patrick
  email: patrick.oleary@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 3
  givenname: Michael
  surname: Grauer
  fullname: Grauer, Michael
  email: michael.grauer@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 4
  givenname: Aashish
  surname: Chaudhary
  fullname: Chaudhary, Aashish
  email: aashish.chaudhary@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 5
  givenname: Chris
  surname: Kotfila
  fullname: Kotfila, Chris
  email: chris.kotfila@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 6
  givenname: Robert
  surname: O'Bara
  fullname: O'Bara, Robert
  email: bob.obara@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
BookMark eNotjLtOwzAUQI0EA5SuLCz-gYbr6_q1gUKhSJXIQMVYOc41WLQ2Ssojf08RTEc6Ojpn7DiXTIxdCKiEAHfVjMumrhCErgDUEZs6Y4UCBwoR8JRd347Z71LgTV8-05BKTvmF-9zxxTeFj_1B8BL5YcKfS_8Wt-Vr4OvhN2rG_WvJ5-wk-u1A039O2Ppu8VQvZ6vH-4f6ZjVLQhk1kzFS5y0qMlrHeRfa4CB63drYeiO9QO1B6rkyRIgthNiBVqRReAcOg5ywy79vIqLNe592vh83xkptpZU_fvdGYQ
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/PyHPC.2016.005
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Xplore
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9781509052202
1509052208
EndPage 8
ExternalDocumentID 7836838
Genre orig-research
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i1575-3ffeda825e766f4dcbc90fa6b8fba73a126a036457ee22b0cfd065e621a9092c3
IEDL.DBID RIE
IngestDate Thu Jun 29 18:37:37 EDT 2023
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i1575-3ffeda825e766f4dcbc90fa6b8fba73a126a036457ee22b0cfd065e621a9092c3
OpenAccessLink https://www.osti.gov/biblio/1595260
PageCount 8
ParticipantIDs ieee_primary_7836838
PublicationCentury 2000
PublicationDate 2016-Nov.
PublicationDateYYYYMMDD 2016-11-01
PublicationDate_xml – month: 11
  year: 2016
  text: 2016-Nov.
PublicationDecade 2010
PublicationTitle 2016 6th Workshop on Python for High-Performance and Scientific Computing (PyHPC)
PublicationTitleAbbrev PYHPC
PublicationYear 2016
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.6263936
Snippet High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms Cloud computing
Complexity theory
Dynamic scheduling
Electronic mail
Open source software
Structural beams
Title Dynamic Provisioning and Execution of HPC Workflows Using Python
URI https://ieeexplore.ieee.org/document/7836838
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB7a4sGTSiu-ycGjsZvsdje5CbWlByl7UOmt5DGBgnSlD7X_3iS7VAQv3kLmkMwkMM9vBuBWKOEJTnrfZMBpZk1CtWApzVBylJkULsKjX5-K6VTMZrJswd0eC4OIsfgM78My5vJtZbYhVNYPiAORija0iyKvsVpNH0aWyH65m5TDUKwV0guDX9NSorIYH_3vmGPo_aDuSLnXJyfQwmUXHh7rkfGBUuPAPY1495-MvtDEX0MqR_ydSIh7u7fqc01iHQApd6EvQA9exqPn4YQ2Uw_ognnbiabOoVXeccMiz52XnjYycSrXwmlVpIrxXMXkYYHIuU6Ms96MwJwzJRPJTXoKnWW1xDMg2iijGFppLIYIWoDVOuEy5rjSWWrPoRu4n7_XjS3mDeMXf29fwmEQbg3Eu4LOZrXFazgwH5vFenUTX-MbJ9ePLQ
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5qFfSk0opvc_Do2t1supvchNpSsZY9VOmtZJMJFGS39KH235tkS0Xw4i1kDslMAvP8ZgBuueSWYIT1Tdo0YFqFQc6jOGAoKAomuPHw6LdBOhzy8VhkNbjbYmEQ0Ref4b1b-ly-LtXKhcpaDnHAY74Du23GaFihtTadGKNQtLJ1P-u4ci2XYGj_mpfi1UXv8H8HHUHzB3dHsq1GOYYaFg14eKyGxjtKhQS3NCILTbpfqPy_IaUh9k7ERb7Ne_m5IL4SgGRr1xmgCa-97qjTDzZzD4JpZK2nIDYGtbSuG6ZJYqz8ciVCI5Ocm1ymsYxoIn36MEWkNA-V0daQwIRGUoSCqvgE6kVZ4CmQXEklI9RCaXQxNAesNdywyFCZs1ifQcNxP5lVrS0mG8bP_96-gf3-6GUwGTwNny_gwAm6guVdQn05X-EV7KmP5XQxv_Yv8w3JJpJ0
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2016+6th+Workshop+on+Python+for+High-Performance+and+Scientific+Computing+%28PyHPC%29&rft.atitle=Dynamic+Provisioning+and+Execution+of+HPC+Workflows+Using+Python&rft.au=Harris%2C+Chris&rft.au=O%27Leary%2C+Patrick&rft.au=Grauer%2C+Michael&rft.au=Chaudhary%2C+Aashish&rft.date=2016-11-01&rft.pub=IEEE&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FPyHPC.2016.005&rft.externalDocID=7836838