A Serverless Engine for High Energy Physics Distributed Analysis

The Large Hadron Collider (LHC) at CERN has generated in the last decade an unprecedented volume of data for the High-Energy Physics (HEP) field. Scientific collaborations interested in analysing such data very often require computing power beyond a single machine. This issue has been tackled tradit...

Full description

Saved in:
Bibliographic Details
Published in:2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid) pp. 575 - 584
Main Authors: Kusnierz, Jacek, Padulano, Vincenzo E., Malawski, Maciej, Burkiewicz, Kamil, Saavedra, Enric Tejedor, Alonso-Jorda, Pedro, Pitt, Michael, Avati, Valentina
Format: Conference Proceeding
Language:English
Published: IEEE 01.05.2022
Subjects:
AWS
C++ languages
CERN
Codes
Computer architecture
Distributed Computing
HEP
Lambda
Large Hadron Collider
MapReduce
ROOT
Runtime
Serverless
Serverless computing
Web services
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The Large Hadron Collider (LHC) at CERN has generated in the last decade an unprecedented volume of data for the High-Energy Physics (HEP) field. Scientific collaborations interested in analysing such data very often require computing power beyond a single machine. This issue has been tackled traditionally by running analyses in distributed environments using stateful, managed batch computing systems. While this approach has been effective so far, current estimates for future computing needs of the field present large scaling challenges. Such a managed approach may not be the only viable way to tackle them and an interesting alternative could be provided by serverless architectures, to enable an even larger scaling potential. This work describes a novel approach to running real HEP scientific applications through a distributed serverless computing engine. The engine is built upon ROOT, a well-established HEP data analysis software, and distributes its computations to a large pool of concurrent executions on Amazon Web Services Lambda Serverless Platform. Thanks to the developed tool, physicists are able to access datasets stored at CERN (also those that are under restricted access policies) and process it on remote infrastructures outside of their typical environment. The analysis of the serverless functions is monitored at runtime to gather performance metrics, both for data- and computation-intensive workloads.
DOI:10.1109/CCGrid54584.2022.00067