Accelerating Communications in Federated Applications with Transparent Object Proxies

Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute-such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step compu...

Full description

Saved in:
Bibliographic Details
Published in:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) pp. 01 - 16
Main Authors: Pauloski, J. Gregory, Hayot-Sasson, Valerie, Ward, Logan, Hudson, Nathaniel, Sabino, Charlie, Baughman, Matt, Chard, Kyle, Foster, Ian
Format: Conference Proceeding
Language:English
Published: ACM 11.11.2023
Subjects:
Computational modeling
Data Communication and Storage
Distributed Computing
Distributed databases
Federated Computing
High performance computing
Open-source Software
Prefetching
Protocols
Python
Semantics
Serverless computing
ISSN:2167-4337
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute-such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step computations spanning federated collections of cloud, high-performance computing (HPC), and edge systems, but passing data among computational steps via cloud storage can incur high costs. Here, we overcome this obstacle with a new programming paradigm that decouples control flow from data flow by extending the pass-by-reference model to distributed applications. We describe ProxyStore, a system that implements this paradigm by providing object proxies that act as wide-area object references with just-in-time resolution. This proxy model enables data producers to communicate data unilaterally, transparently, and efficiently to both local and remote consumers. We demonstrate the benefits of this model with synthetic bench-marks and real-world scientific applications, running across various computing platforms.
AbstractList Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute-such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step computations spanning federated collections of cloud, high-performance computing (HPC), and edge systems, but passing data among computational steps via cloud storage can incur high costs. Here, we overcome this obstacle with a new programming paradigm that decouples control flow from data flow by extending the pass-by-reference model to distributed applications. We describe ProxyStore, a system that implements this paradigm by providing object proxies that act as wide-area object references with just-in-time resolution. This proxy model enables data producers to communicate data unilaterally, transparently, and efficiently to both local and remote consumers. We demonstrate the benefits of this model with synthetic bench-marks and real-world scientific applications, running across various computing platforms.
Author Hayot-Sasson, Valerie
Sabino, Charlie
Baughman, Matt
Hudson, Nathaniel
Chard, Kyle
Ward, Logan
Foster, Ian
Pauloski, J. Gregory
Author_xml – sequence: 1
  givenname: J. Gregory
  surname: Pauloski
  fullname: Pauloski, J. Gregory
  organization: University of Chicago
– sequence: 2
  givenname: Valerie
  surname: Hayot-Sasson
  fullname: Hayot-Sasson, Valerie
  organization: University of Chicago
– sequence: 3
  givenname: Logan
  surname: Ward
  fullname: Ward, Logan
  organization: Argonne National Laboratory
– sequence: 4
  givenname: Nathaniel
  surname: Hudson
  fullname: Hudson, Nathaniel
  organization: University of Chicago
– sequence: 5
  givenname: Charlie
  surname: Sabino
  fullname: Sabino, Charlie
  organization: University of Chicago
– sequence: 6
  givenname: Matt
  surname: Baughman
  fullname: Baughman, Matt
  organization: University of Chicago
– sequence: 7
  givenname: Kyle
  surname: Chard
  fullname: Chard, Kyle
  organization: University of Chicago Argonne National Laboratory
– sequence: 8
  givenname: Ian
  surname: Foster
  fullname: Foster, Ian
  organization: University of Chicago Argonne National Laboratory
BookMark eNo9jMtKw0AUQEdRsNas3biYH0i9N_PMshSrQqEu2nWZx41OaSchiah_b0VxdTgcONfsIreZGLtFmCFKdS-URWPlTGgwIM0ZK2pTWwknQ6irczapUJtSCmGuWDEMewAQFUi0MGHbeQh0oN6NKb_yRXs8vucUTtbmgafMlxR_IkU-77rDf_lI4xvf9C4Pnespj3zt9xRG_tK3n4mGG3bZuMNAxR-nbLt82CyeytX68XkxX5VOSD2WlbHWe9VEA16BiQZlAAHKhUY3iKEhBboiayMgRQ2APmplvDax9uBJTNnd7zcR0a7r09H1XzsEaRUoEN_qf1O-
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1145/3581784.3607047
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798400701092
EISSN 2167-4337
EndPage 16
ExternalDocumentID 10485050
Genre orig-research
GrantInformation_xml – fundername: Department of Energy (DOE)
  grantid: DE-AC02-06CH11357
  funderid: 10.13039/100000015
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
OCL
RIE
RIL
ID FETCH-LOGICAL-a346t-2788bb5fd70b507d714c0305acf6f11cfe5062e88d01ed6001bd657b67d9b0be3
IEDL.DBID RIE
ISICitedReferencesCount 10
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001461755900022&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:09:33 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a346t-2788bb5fd70b507d714c0305acf6f11cfe5062e88d01ed6001bd657b67d9b0be3
PageCount 16
ParticipantIDs ieee_primary_10485050
PublicationCentury 2000
PublicationDate 2023-Nov.-11
PublicationDateYYYYMMDD 2023-11-11
PublicationDate_xml – month: 11
  year: 2023
  text: 2023-Nov.-11
  day: 11
PublicationDecade 2020
PublicationTitle International Conference for High Performance Computing, Networking, Storage and Analysis (Online)
PublicationTitleAbbrev SC
PublicationYear 2023
Publisher ACM
Publisher_xml – name: ACM
SSID ssj0003204180
ssib053141430
Score 1.9396582
Snippet Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute-such as when fast networks make it cost-effective...
SourceID ieee
SourceType Publisher
StartPage 01
SubjectTerms Computational modeling
Data Communication and Storage
Distributed Computing
Distributed databases
Federated Computing
High performance computing
Open-source Software
Prefetching
Protocols
Python
Semantics
Serverless computing
Title Accelerating Communications in Federated Applications with Transparent Object Proxies
URI https://ieeexplore.ieee.org/document/10485050
WOSCitedRecordID wos001461755900022&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV25TsNAEF1BREEVjiBubUHrsLvey2WEiKhCCiKli_ZEaRyUOIjPz44PLomCznJja3fGM2_W7z2E7qIrSMGYzYgOMgGUELIiyiIT3DCf0Ji2eWM2oSYTPZ8X05asXnNhQgj1z2dhCJf1Wb5fuS2MylKGc50qdkLo-0rJhqzVBU-KJU47KXH4DOeMcKpJK-dDubgHqS-l-TCXKc5_-anU5WTc_-eLHKHBFzEPTz9LzjHaC-UJ6nfODLhN1FM0GzmX6gnsbvmKf5BANnhZ4jFISKQu0-PRtwNsDENZ3MidA0esws8WxjTwyI-EqAdoNn58eXjKWgOFzORcVhlL-NZaEb0iNvV9XlHuIMGNizJS6mIQRLKgtSc0eGh9rJdCWal8YYkN-RnqlasynCPMpDCSC8uNYVxHbkAEKOQqOi8INeICDWCZFm-NRsaiW6HLP-5foUMwbgdWH6XXqFett-EGHbj3arlZ39Y7uwNDVKOo
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTsMwELVQQYJTWYrY8YFriu3YjnOsEFURpfTQSr1V3oJ6SVGbIj4fTxY2iQO3KJdE9kxm3jjvPYRuMpuSlDETEeVlACjeR2km00hwzVxAY8rEldlEMhqp2Swd12T1kgvjvS9_PvNduCzP8t3SbmBUFjKcq1CxA0LfFpwzUtG1mvAJ0cRpIyYOH-KYEU4VqQV9KBe3IPaVKN6NZYj0X44qZUHpt__5Kvuo80XNw-PPonOAtnx-iNqNNwOuU_UITXvWhooC-5u_4B80kDVe5LgPIhKhz3S49-0IG8NYFleC58ASK_CzgUENPPI9YOoOmvbvJ3eDqLZQiHTMZRGxgHCNEZlLiAmdn0sot5Di2mYyo9RmXhDJvFKOUO-g-TFOisTIxKWGGB8fo1a-zP0JwkwKLbkwXGvGVcY1yAD5OMmsE4RqcYo6sEzz10olY96s0Nkf96_R7mDyNJwPH0aP52gPbNyB40fpBWoVq42_RDv2rVisV1flLn8A9d6m7w
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis+%28Online%29&rft.atitle=Accelerating+Communications+in+Federated+Applications+with+Transparent+Object+Proxies&rft.au=Pauloski%2C+J.+Gregory&rft.au=Hayot-Sasson%2C+Valerie&rft.au=Ward%2C+Logan&rft.au=Hudson%2C+Nathaniel&rft.date=2023-11-11&rft.pub=ACM&rft.eissn=2167-4337&rft.spage=01&rft.epage=16&rft_id=info:doi/10.1145%2F3581784.3607047&rft.externalDocID=10485050