Dynamic Provisioning and Execution of HPC Workflows Using Python

High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In...

Full description

Saved in:
Bibliographic Details
Published in:2016 6th Workshop on Python for High-Performance and Scientific Computing (PyHPC) pp. 1 - 8
Main Authors: Harris, Chris, O'Leary, Patrick, Grauer, Michael, Chaudhary, Aashish, Kotfila, Chris, O'Bara, Robert
Format: Conference Proceeding
Language:English
Published: IEEE 01.11.2016
Subjects:
Cloud computing
Complexity theory
Dynamic scheduling
Electronic mail
Open source software
Structural beams
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In this paper, we describe the development of the Python-based cumulus, which addresses many of these barriers. cumulus is a platform for the dynamic provisioning and execution of HPC workflows. cumulus provides the infrastructure needed to build applications that leverage traditional or Cloud-based HPC resources in their workflows. Finally, we demonstrate the use of cumulus in both web and desktop simulation applications, as well as in an Apache Spark-based analysis application.
AbstractList High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production of better products, more quickly, and at reduced cost. However, HPC workflows are difficult to implement and use for a variety of reasons. In this paper, we describe the development of the Python-based cumulus, which addresses many of these barriers. cumulus is a platform for the dynamic provisioning and execution of HPC workflows. cumulus provides the infrastructure needed to build applications that leverage traditional or Cloud-based HPC resources in their workflows. Finally, we demonstrate the use of cumulus in both web and desktop simulation applications, as well as in an Apache Spark-based analysis application.
Author Grauer, Michael
Chaudhary, Aashish
Harris, Chris
O'Bara, Robert
Kotfila, Chris
O'Leary, Patrick
Author_xml – sequence: 1
  givenname: Chris
  surname: Harris
  fullname: Harris, Chris
  email: chris.harris@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 2
  givenname: Patrick
  surname: O'Leary
  fullname: O'Leary, Patrick
  email: patrick.oleary@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 3
  givenname: Michael
  surname: Grauer
  fullname: Grauer, Michael
  email: michael.grauer@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 4
  givenname: Aashish
  surname: Chaudhary
  fullname: Chaudhary, Aashish
  email: aashish.chaudhary@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 5
  givenname: Chris
  surname: Kotfila
  fullname: Kotfila, Chris
  email: chris.kotfila@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
– sequence: 6
  givenname: Robert
  surname: O'Bara
  fullname: O'Bara, Robert
  email: bob.obara@kitware.com
  organization: Kitware Inc., Clifton Park, NY, USA
BookMark eNotjLtOwzAUQI0EA5SuLCz-gYbr6_q1gUKhSJXIQMVYOc41WLQ2Ssojf08RTEc6Ojpn7DiXTIxdCKiEAHfVjMumrhCErgDUEZs6Y4UCBwoR8JRd347Z71LgTV8-05BKTvmF-9zxxTeFj_1B8BL5YcKfS_8Wt-Vr4OvhN2rG_WvJ5-wk-u1A039O2Ppu8VQvZ6vH-4f6ZjVLQhk1kzFS5y0qMlrHeRfa4CB63drYeiO9QO1B6rkyRIgthNiBVqRReAcOg5ywy79vIqLNe592vh83xkptpZU_fvdGYQ
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/PyHPC.2016.005
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL) (UW System Shared)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9781509052202
1509052208
EndPage 8
ExternalDocumentID 7836838
Genre orig-research
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i1575-3ffeda825e766f4dcbc90fa6b8fba73a126a036457ee22b0cfd065e621a9092c3
IEDL.DBID RIE
IngestDate Thu Jun 29 18:37:37 EDT 2023
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i1575-3ffeda825e766f4dcbc90fa6b8fba73a126a036457ee22b0cfd065e621a9092c3
OpenAccessLink https://www.osti.gov/biblio/1595260
PageCount 8
ParticipantIDs ieee_primary_7836838
PublicationCentury 2000
PublicationDate 2016-Nov.
PublicationDateYYYYMMDD 2016-11-01
PublicationDate_xml – month: 11
  year: 2016
  text: 2016-Nov.
PublicationDecade 2010
PublicationTitle 2016 6th Workshop on Python for High-Performance and Scientific Computing (PyHPC)
PublicationTitleAbbrev PYHPC
PublicationYear 2016
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.6263076
Snippet High-performance computing (HPC) workflows over the last several decades have proven to assist in the understanding of scientific phenomena and the production...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms Cloud computing
Complexity theory
Dynamic scheduling
Electronic mail
Open source software
Structural beams
Title Dynamic Provisioning and Execution of HPC Workflows Using Python
URI https://ieeexplore.ieee.org/document/7836838
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB7a4sGTSiu-ycGjsbvZNY-bUFt6kLIHhd5KNplAQXalD7X_3iS7VAQv3kIG8pgE5vnNANx6oSCYQ0Mzo5Hm0iJVEgXVzjjNS2ZZGbuWPIvZTM7nqujA3R4Lg4gx-QzvwzDG8m1ttsFVNgyIA5nJLnSF4A1Wq63DmCZqWOymxSgka4XwwsOvbilRWEyO_rfNMQx-UHek2MuTE-hg1YfHp6ZlfKA0OHBPI978J-MvNPHXkNoRfyYS_N7urf5ck5gHQIpdqAswgNfJ-GU0pW3XA7pMve5EM-fQam-4oeDc5daURiWBb9KVWmQ6ZVzH4KFAZKxMjLNejUDOUq0SxUx2Cr2qrvAMCBfMeg0JrXEsl2Uiw1JGGSdDGTdrz6Efbr94bwpbLNqLX_w9fQmHgbkNEO8KepvVFq_hwHxsluvVTXyNb9rxj7M
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5qFfSk0opvc_Bo7G52u8nehNqyYi17qNBbySYTKMiu9KH235tkS0Xw4i1kIMlMAjOZmW8G4NYqBc4MKhopiTQWGmkqkFNplJFJwTQrfNeSIR-NxGSS5g2422JhENEnn-G9G_pYvq7UyrnKOg5xICKxA7vdOGZBjdbaVGIMg7STr7O859K1XICh-6tfilcXg8P_bXQE7R_cHcm3GuUYGli24OGxbhrvKDUS3NKILDXpf6Hy74ZUhtgzEef5Nm_V54L4TACSr11lgDa8DvrjXkY3fQ_oLLTWE42MQS3t1w15kphYq0KlgZOcMIXkkQxZIn34kCMyVgTKaGtIYMJCmQYpU9EJNMuqxFMgCWfa2kiolWGxKALhllKpMsIVctP6DFqO--l7XdpiumH8_O_pG9jPxi_D6fBp9HwBB07QNSzvEprL-QqvYE99LGeL-bW_mW-Eq5L6
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2016+6th+Workshop+on+Python+for+High-Performance+and+Scientific+Computing+%28PyHPC%29&rft.atitle=Dynamic+Provisioning+and+Execution+of+HPC+Workflows+Using+Python&rft.au=Harris%2C+Chris&rft.au=O%27Leary%2C+Patrick&rft.au=Grauer%2C+Michael&rft.au=Chaudhary%2C+Aashish&rft.date=2016-11-01&rft.pub=IEEE&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FPyHPC.2016.005&rft.externalDocID=7836838