Accelerating Communications in Federated Applications with Transparent Object Proxies

Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute-such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step compu...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International Conference for High Performance Computing, Networking, Storage and Analysis (Online) s. 01 - 16
Hlavní autoři: Pauloski, J. Gregory, Hayot-Sasson, Valerie, Ward, Logan, Hudson, Nathaniel, Sabino, Charlie, Baughman, Matt, Chard, Kyle, Foster, Ian
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: ACM 11.11.2023
Témata:
Computational modeling
Data Communication and Storage
Distributed Computing
Distributed databases
Federated Computing
High performance computing
Open-source Software
Prefetching
Protocols
Python
Semantics
Serverless computing
ISSN:2167-4337
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Advances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute-such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step computations spanning federated collections of cloud, high-performance computing (HPC), and edge systems, but passing data among computational steps via cloud storage can incur high costs. Here, we overcome this obstacle with a new programming paradigm that decouples control flow from data flow by extending the pass-by-reference model to distributed applications. We describe ProxyStore, a system that implements this paradigm by providing object proxies that act as wide-area object references with just-in-time resolution. This proxy model enables data producers to communicate data unilaterally, transparently, and efficiently to both local and remote consumers. We demonstrate the benefits of this model with synthetic bench-marks and real-world scientific applications, running across various computing platforms.
ISSN:2167-4337
DOI:10.1145/3581784.3607047