BenchCloud: A Platform for Scalable Performance Benchmarking

Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately measure resource consumption, including time and memory, which are essential for understanding the tools' capabilities. BenchExec, a widely...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE/ACM International Conference on Automated Software Engineering : [proceedings] s. 2386 - 2389
Hlavní autoři: Beyer, Dirk, Chien, Po-Chun, Jankola, Marek
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: ACM 27.10.2024
Témata:
Benchmark testing
Benchmarking
Cloud computing
Containers
Distributed computing
Heterogeneous networks
Job-distribution system
Memory management
Organizations
Performance evaluation
Reliability
Remote execution
Resource management
Software
Software engineering
Time measurement
Tool competition
ISSN:2643-1572
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately measure resource consumption, including time and memory, which are essential for understanding the tools' capabilities. BenchExec, a widely used benchmarking framework, reliably measures resource usage for tools executed locally on a single node. This paper describes BenchCloud, a solution for elastic and scalable job distribution across hundreds of nodes, enabling large-scale experiments on distributed and heterogeneous computing environments. BenchCloud seamlessly integrates with BenchExec, allowing BenchExec to delegate the actual execution to BenchCloud. The system has been employed in several prominent international competitions in automated reasoning, including SMT-COMP, SV-COMP, and Test-Comp, underscoring its importance in rigorous tool evaluation across various research domains. It helps to ensure both internal and external validity of the experimental results. This paper presents an overview of BenchCloud's architecture and highlights its primary use cases in facilitating scalable benchmarking.Demonstration video: https://youtu.be/aBfQytqPm0URunning system: https://benchcloud.sosy-lab.org/CCS Concepts* General and reference → Cross-computing tools and techniques; * Computing methodologies → Distributed computing methodologies; * Computer-systems organization → Cloud computing; Client-server architectures; * Software and its engineering → Software verification and validation.
ISSN:2643-1572
DOI:10.1145/3691620.3695358