BenchCloud: A Platform for Scalable Performance Benchmarking
Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately measure resource consumption, including time and memory, which are essential for understanding the tools' capabilities. BenchExec, a widely...
Gespeichert in:
| Veröffentlicht in: | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] S. 2386 - 2389 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
ACM
27.10.2024
|
| Schlagworte: | |
| ISSN: | 2643-1572 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately measure resource consumption, including time and memory, which are essential for understanding the tools' capabilities. BenchExec, a widely used benchmarking framework, reliably measures resource usage for tools executed locally on a single node. This paper describes BenchCloud, a solution for elastic and scalable job distribution across hundreds of nodes, enabling large-scale experiments on distributed and heterogeneous computing environments. BenchCloud seamlessly integrates with BenchExec, allowing BenchExec to delegate the actual execution to BenchCloud. The system has been employed in several prominent international competitions in automated reasoning, including SMT-COMP, SV-COMP, and Test-Comp, underscoring its importance in rigorous tool evaluation across various research domains. It helps to ensure both internal and external validity of the experimental results. This paper presents an overview of BenchCloud's architecture and highlights its primary use cases in facilitating scalable benchmarking.Demonstration video: https://youtu.be/aBfQytqPm0URunning system: https://benchcloud.sosy-lab.org/CCS Concepts* General and reference → Cross-computing tools and techniques; * Computing methodologies → Distributed computing methodologies; * Computer-systems organization → Cloud computing; Client-server architectures; * Software and its engineering → Software verification and validation. |
|---|---|
| AbstractList | Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately measure resource consumption, including time and memory, which are essential for understanding the tools' capabilities. BenchExec, a widely used benchmarking framework, reliably measures resource usage for tools executed locally on a single node. This paper describes BenchCloud, a solution for elastic and scalable job distribution across hundreds of nodes, enabling large-scale experiments on distributed and heterogeneous computing environments. BenchCloud seamlessly integrates with BenchExec, allowing BenchExec to delegate the actual execution to BenchCloud. The system has been employed in several prominent international competitions in automated reasoning, including SMT-COMP, SV-COMP, and Test-Comp, underscoring its importance in rigorous tool evaluation across various research domains. It helps to ensure both internal and external validity of the experimental results. This paper presents an overview of BenchCloud's architecture and highlights its primary use cases in facilitating scalable benchmarking.Demonstration video: https://youtu.be/aBfQytqPm0URunning system: https://benchcloud.sosy-lab.org/CCS Concepts* General and reference → Cross-computing tools and techniques; * Computing methodologies → Distributed computing methodologies; * Computer-systems organization → Cloud computing; Client-server architectures; * Software and its engineering → Software verification and validation. |
| Author | Chien, Po-Chun Beyer, Dirk Jankola, Marek |
| Author_xml | – sequence: 1 givenname: Dirk surname: Beyer fullname: Beyer, Dirk organization: LMU Munich,Munich,Germany – sequence: 2 givenname: Po-Chun surname: Chien fullname: Chien, Po-Chun organization: LMU Munich,Munich,Germany – sequence: 3 givenname: Marek surname: Jankola fullname: Jankola, Marek organization: LMU Munich,Munich,Germany |
| BookMark | eNotT0tLw0AYXEXBtubsxcP-gbT7ffsWLzX4goIFey_bzRcNphtJ6sF_3_VxmRmGYYaZsrPUJ2LsCsQcQOmFNB4MinlmLbU7YYW33ikhLKBy9pRN0ChZgrZ4wYpxbHciS20AzITd3lGK71XXf9U3fMnXXTg0_bDnGfhrDF3YdcTXNPyYIUXiv_l9GD7a9HbJzpvQjVT884xtHu431VO5enl8rparMqDzh1IqYU1DpKAR3iB45VHUPqLXzqjauhCFI2N0iA0Ej7WqNWpAoawlB3LGrv9qWyLafg5tnv_eQi7V-Yk8ArDHR_w |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1145/3691620.3695358 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE Xplore IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798400712487 |
| EISSN | 2643-1572 |
| EndPage | 2389 |
| ExternalDocumentID | 10765057 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IF 6IH 6IK 6IL 6IM 6IN 6J9 AAJGR AAWTH ABLEC ACREN ADYOE ADZIZ AFYQB ALMA_UNASSIGNED_HOLDINGS AMTXH BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI M43 OCL RIE RIL |
| ID | FETCH-LOGICAL-a289t-34076fee41f0962194920d9c295864d78ac08e665acf1a92d4d525120477e813 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 1 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001353105400219&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Jan 15 06:20:43 EST 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a289t-34076fee41f0962194920d9c295864d78ac08e665acf1a92d4d525120477e813 |
| OpenAccessLink | https://doi.org/10.1145/3691620.3695358 |
| PageCount | 4 |
| ParticipantIDs | ieee_primary_10765057 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-Oct.-27 |
| PublicationDateYYYYMMDD | 2024-10-27 |
| PublicationDate_xml | – month: 10 year: 2024 text: 2024-Oct.-27 day: 27 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] |
| PublicationTitleAbbrev | ASE |
| PublicationYear | 2024 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssib057256116 ssj0051577 |
| Score | 2.2928855 |
| Snippet | Performance evaluation is a crucial method for assessing automated-reasoning tools. Evaluating automated tools requires rigorous benchmarking to accurately... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 2386 |
| SubjectTerms | Benchmark testing Benchmarking Cloud computing Containers Distributed computing Heterogeneous networks Job-distribution system Memory management Organizations Performance evaluation Reliability Remote execution Resource management Software Software engineering Time measurement Tool competition |
| Title | BenchCloud: A Platform for Scalable Performance Benchmarking |
| URI | https://ieeexplore.ieee.org/document/10765057 |
| WOSCitedRecordID | wos001353105400219&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwED7RioGpPIp4ywNrIH7biAUqKgZUZejQrXJiRyCVFpWU38_ZTSkLA4tjWR4iP-6-z_Z3B3BNcWYd8o5My9pnIr6tMdLHuLelLGktTOBJKPyiRyMzmdiiFasnLUwIIT0-Czexmu7y_aJaxaMy3OFaRUDdgY7Wai3W2iweqdF504h11mYY_bTWbSwfKuQtVwiEGHJUZSWPCd5_JVNJvmTY--df7EN_q8ojxY-_OYCdMD-E3iYtA2l36RHcP-L3dTBbrPwdeSDFzDURmRIssJObRbEUKbaCAZL6v7t0at6H8fBpPHjO2iQJmUOu1GQcGZmqQxC0RjaC9kdYlntbMSuNEl4bV-UmKCVdVVNnmRdeRkyTC62DofwYuvPFPJwAQUvDhWSlQX4tvDVWsiACl1Xuck_r8hT6cTCmH-swGNPNOJz90X4OewwRQDT0TF9At1muwiXsVl_N2-fyKk3eN1b-lxM |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwED1BQYKpfBTxjQfWQOzYsY1YoKIqolQZOnSrnNgRSP1AJeX3c3ZTysLAkkSRh8iO796z_e4BXFMcWYO8I5KitBH3Z2uUsL7ubS5yWnLlkiAU7sl-Xw2HOqvF6kEL45wLh8_cjX8Me_l2Viz8UhnOcJl6QL0JW946SyzlWqvfR0hM39SjnWUgxkwtZV3Nh3Jxm6QIhRiy1FSLxFu8_7JTCdmk0_znd-xBa63LI9lPxtmHDTc9gObKmIHU8_QQ7h_x_tYezxb2jjyQbGwqj00JXrCRGXu5FMnWkgES2k9MWDdvwaDzNGh3o9omITLIlqooQU6Wls5xWiIfwQjENYutLpgWKuVWKlPEyqWpMEVJjWaWW-FRTcyldIomR9CYzqbuGAjGmoQLlitk2NxqpQVz3CWiiE1saZmfQMt3xuhjWQhjtOqH0z_eX8FOd_DaG_We-y9nsMsQD_iwz-Q5NKr5wl3AdvFVvX_OL8NAfgNuoJpe |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=BenchCloud%3A+A+Platform+for+Scalable+Performance+Benchmarking&rft.au=Beyer%2C+Dirk&rft.au=Chien%2C+Po-Chun&rft.au=Jankola%2C+Marek&rft.date=2024-10-27&rft.pub=ACM&rft.eissn=2643-1572&rft.spage=2386&rft.epage=2389&rft_id=info:doi/10.1145%2F3691620.3695358&rft.externalDocID=10765057 |