LIDC: A Location Independent Multi-Cluster Computing Framework for Data Intensive Science

Scientific communities are increasingly using geographically distributed computing platforms. The current methods of compute placement predominantly use logically centralized controllers such as Kubernetes (K8s) to match tasks to available resources. However, this centralized approach is unsuitable...

Full description

Saved in:
Bibliographic Details
Published in:SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis pp. 760 - 764
Main Authors: Timilsina, Sankalpa, Shannigrahi, Susmit
Format: Conference Proceeding
Language:English
Published: IEEE 17.11.2024
Subjects:
Collaboration
Complexity theory
Conferences
data intensive science
Decentralized control
Distributed computing
High performance computing
Manuals
orchestration systems
Semantics
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Scientific communities are increasingly using geographically distributed computing platforms. The current methods of compute placement predominantly use logically centralized controllers such as Kubernetes (K8s) to match tasks to available resources. However, this centralized approach is unsuitable in multi-organizational collaborations. Furthermore, workflows often need to use manual configurations tailored for a single platform and cannot adapt to dynamic changes across infrastructure.Our work introduces a decentralized control plane for placing computations on geographically dispersed compute clusters using semantic names. We assign semantic names to computations to match requests with named Kubernetes (K8s) service endpoints. We show that this approach provides multiple benefits. First, it allows placement of computational jobs to be independent of location, enabling any cluster with sufficient resources to execute the computation. Second, it facilitates dynamic compute placement without requiring prior knowledge of cluster locations or predefined configurations.
AbstractList Scientific communities are increasingly using geographically distributed computing platforms. The current methods of compute placement predominantly use logically centralized controllers such as Kubernetes (K8s) to match tasks to available resources. However, this centralized approach is unsuitable in multi-organizational collaborations. Furthermore, workflows often need to use manual configurations tailored for a single platform and cannot adapt to dynamic changes across infrastructure.Our work introduces a decentralized control plane for placing computations on geographically dispersed compute clusters using semantic names. We assign semantic names to computations to match requests with named Kubernetes (K8s) service endpoints. We show that this approach provides multiple benefits. First, it allows placement of computational jobs to be independent of location, enabling any cluster with sufficient resources to execute the computation. Second, it facilitates dynamic compute placement without requiring prior knowledge of cluster locations or predefined configurations.
Author Shannigrahi, Susmit
Timilsina, Sankalpa
Author_xml – sequence: 1
  givenname: Sankalpa
  surname: Timilsina
  fullname: Timilsina, Sankalpa
  email: stimilsin43@tntech.edu
  organization: Tennessee Tech,Computer Science Department,Cookeville,TN
– sequence: 2
  givenname: Susmit
  surname: Shannigrahi
  fullname: Shannigrahi, Susmit
  email: sshannigrahi@tntech.edu
  organization: Tennessee Tech,Computer Science Department,Cookeville,TN
BookMark eNotj81KxDAURiMoqOM8gS7yAq03v03dDR1HByouRhFXQ9LcSrCTDm2q-PYWdPOd1TnwXZLT2Eck5JpBzhiUt7vqTQsuIefAZQ7AwJyQZVmURigQSikpzslyHIMDDcpIMOqCvNfbdXVHV7TuG5tCH-k2ejziPDHRp6lLIau6aUw40Ko_HKcU4gfdDPaA3_3wSdt-oGub7KwljGP4QrprAsYGr8hZa7sRl_9ckNfN_Uv1mNXPD9tqVWeWK50y1IYL6z2U3hhfugIKcL7R6JEJ1Wp0rdaFbFrvJXMg0MmWC6MRC-a1dWJBbv66ARH3xyEc7PCzn89zUNqIX5toVBY
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/SCW63240.2024.00108
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9798350355543
EndPage 764
ExternalDocumentID 10820568
Genre orig-research
GrantInformation_xml – fundername: National Science Foundation
  funderid: 10.13039/100000001
GroupedDBID 6IE
6IL
ACM
ALMA_UNASSIGNED_HOLDINGS
CBEJK
RIE
RIL
ID FETCH-LOGICAL-a256t-e6823add09d88d9b7070bdc6ede135f6ebf6674cfdd41b03eb4f2386ee71d6ab3
IEDL.DBID RIE
ISICitedReferencesCount 0
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001451792300085&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 01:59:34 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a256t-e6823add09d88d9b7070bdc6ede135f6ebf6674cfdd41b03eb4f2386ee71d6ab3
PageCount 5
ParticipantIDs ieee_primary_10820568
PublicationCentury 2000
PublicationDate 2024-Nov.-17
PublicationDateYYYYMMDD 2024-11-17
PublicationDate_xml – month: 11
  year: 2024
  text: 2024-Nov.-17
  day: 17
PublicationDecade 2020
PublicationTitle SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
PublicationTitleAbbrev SC-W
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssib060584085
Score 1.8972452
Snippet Scientific communities are increasingly using geographically distributed computing platforms. The current methods of compute placement predominantly use...
SourceID ieee
SourceType Publisher
StartPage 760
SubjectTerms Collaboration
Complexity theory
Conferences
data intensive science
Decentralized control
Distributed computing
High performance computing
Manuals
orchestration systems
Semantics
Title LIDC: A Location Independent Multi-Cluster Computing Framework for Data Intensive Science
URI https://ieeexplore.ieee.org/document/10820568
WOSCitedRecordID wos001451792300085&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEB1s8eBJxYrf5OA1uh_ZJOtNthYLpRRUrKeSbCYgSCvt1t_vZHdbvXjwtgQ2C5PMvnzMew_g2gSNFFVaHjmZcWE88lx6yitHE0g64TNszCbUeKyn03zSktVrLgwi1sVneBMe67t8tyjX4aiMMpzwKpO6Ax2lZEPW2kyecL0X1LpaZaE4ym-fitcgRh7RLjAJGtlx8JD85aFSQ8hg_58fP4DeDxmPTbYwcwg7OD-Ct9GwX9yxezZaNGdubLj1s61Yzarlxcc6qCCwxriB3mWDTSUWo6Uq65vKsG0JO2uzvAcvg4fn4pG3Lgnc0HKl4ih1ktJfKsqd1i63ipLYulKiwzjNvETrpVSi9M6J2EYpWuEJpyWiip00Nj2G7nwxxxNgidEG0WqrMhQ6px4J36jJJRqNEfIUeiEus89GCGO2CcnZH-3nsBdCH6h7sbqAbrVc4yXsll_V-2p5VQ_fN-gYnRQ
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFH7oFPSk4sTf5uA12h9pknqTzrFhHQMnztNImhcYyCaz8-83abvpxYO3EEgLyXv90uR93wdwrbxGiig0DQxPKFMWacqtyyvjAogbZhOszSbEYCDH43TYkNUrLgwiVsVneOOb1V2-mRdLf1TmMtzhVcLlJmx566yGrrUKH3_B5_W6Gm2hMEhvn7NXL0ceuP_AyKtkh95F8peLSgUi3b1_vn4f2j90PDJcA80BbODsEN7yfie7I_ckn9enbqS_drQtScWrpdn70usgkNq6wY0l3VUtFnGbVdJRpSLrInbS5HkbXroPo6xHG58EqtyGpaTIZRS771SQGilNqoVLY20KjgbDOLEcteVcsMIaw0IdxKiZdUjNEUVouNLxEbRm8xkeA4mUVIhaapEgk6l7okM412UiiUoxfgJtPy-Tj1oKY7KaktM_-q9gpzd6yid5f_B4Brt-GTyRLxTn0CoXS7yA7eKrnH4uLqul_AZGD6Bd
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=SC24-W%3A+Workshops+of+the+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis&rft.atitle=LIDC%3A+A+Location+Independent+Multi-Cluster+Computing+Framework+for+Data+Intensive+Science&rft.au=Timilsina%2C+Sankalpa&rft.au=Shannigrahi%2C+Susmit&rft.date=2024-11-17&rft.pub=IEEE&rft.spage=760&rft.epage=764&rft_id=info:doi/10.1109%2FSCW63240.2024.00108&rft.externalDocID=10820568