AI-driven Java Performance Testing: Balancing Result Quality with Testing Time
Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the so...
Uloženo v:
| Vydáno v: | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] s. 443 - 454 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
ACM
27.10.2024
|
| Témata: | |
| ISSN: | 2643-1572 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the software undergoes a warm-up phase of execution, due to just-in-time compilation. During this phase, performance measurements are subject to severe fluctuations, which may adversely affect quality of performance test results. Both practitioners and researchers have proposed approaches to mitigate this issue. Practitioners typically rely on a fixed number of iterated executions that are used to warm-up the software before starting to collect performance measurements (state-of-practice). Researchers have developed techniques that can dynamically stop warm-up iterations at runtime (state-of-the-art). However, these approaches often provide suboptimal estimates of the warm-up phase, resulting in either insufficient or excessive warm-up iterations, which may degrade result quality or increase testing time. There is still a lack of consensus on how to properly address this problem. Here, we propose and study an AI-based framework to dynamically halt warm-up iterations at runtime. Specifically, our framework leverages recent advances in AI for Time Series Classification (TSC) to predict the end of the warm-up phase during test execution. We conduct experiments by training three different TSC models on half a million of measurement segments obtained from JMH microbenchmark executions. We find that our framework significantly improves the accuracy of the warm-up estimates provided by state-of-practice and state-of-the-art methods. This higher estimation accuracy results in a net improvement in either result quality or testing time for up to +35.3% of the microbenchmarks. Our study highlights that integrating AI to dynamically estimate the end of the warm-up phase can enhance the cost-effectiveness of Java performance testing.CCS CONCEPTS*Software and its engineering → Software performance; Software testing and debugging. |
|---|---|
| AbstractList | Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the software undergoes a warm-up phase of execution, due to just-in-time compilation. During this phase, performance measurements are subject to severe fluctuations, which may adversely affect quality of performance test results. Both practitioners and researchers have proposed approaches to mitigate this issue. Practitioners typically rely on a fixed number of iterated executions that are used to warm-up the software before starting to collect performance measurements (state-of-practice). Researchers have developed techniques that can dynamically stop warm-up iterations at runtime (state-of-the-art). However, these approaches often provide suboptimal estimates of the warm-up phase, resulting in either insufficient or excessive warm-up iterations, which may degrade result quality or increase testing time. There is still a lack of consensus on how to properly address this problem. Here, we propose and study an AI-based framework to dynamically halt warm-up iterations at runtime. Specifically, our framework leverages recent advances in AI for Time Series Classification (TSC) to predict the end of the warm-up phase during test execution. We conduct experiments by training three different TSC models on half a million of measurement segments obtained from JMH microbenchmark executions. We find that our framework significantly improves the accuracy of the warm-up estimates provided by state-of-practice and state-of-the-art methods. This higher estimation accuracy results in a net improvement in either result quality or testing time for up to +35.3% of the microbenchmarks. Our study highlights that integrating AI to dynamically estimate the end of the warm-up phase can enhance the cost-effectiveness of Java performance testing.CCS CONCEPTS*Software and its engineering → Software performance; Software testing and debugging. |
| Author | Di Menna, Federico Cortellessa, Vittorio Traini, Luca |
| Author_xml | – sequence: 1 givenname: Luca surname: Traini fullname: Traini, Luca email: luca.traini@univaq.it organization: University of L'Aquila,Italy – sequence: 2 givenname: Federico surname: Di Menna fullname: Di Menna, Federico email: federico.dimenna@graduate.univaq.it organization: University of L'Aquila,Italy – sequence: 3 givenname: Vittorio surname: Cortellessa fullname: Cortellessa, Vittorio email: vittorio.cortellessa@univaq.it organization: University of L'Aquila,Italy |
| BookMark | eNo1kMtOwzAURA0CiVKyZsPCP5Di68QvdqUqUFTxUvbVTXwNlpIUJWlR_54gYHWORqNZzDk7abctMXYJYgaQq-tMO9BSzEYqAeaIJc44mwthQObWHLOJ1HmWgjLyjCV9H0sxqtIAesKe5qvUd3FPLX_EPfIX6sK2a7CtiBfUD7F9v-G3WI_BqPyN-l098Ncd1nE48K84fPzXeBEbumCnAeuekj9OWXG3LBYP6fr5frWYr1OU1g1p0HlpsVRZqAS6oIMJldIVSoGlMx6Uosx7BCs9Oh9yBGcFSVlVYLyx2ZRd_c5GItp8drHB7rABYfTPA9k3k3BQvQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1145/3691620.3695017 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798400712487 |
| EISSN | 2643-1572 |
| EndPage | 454 |
| ExternalDocumentID | 10765017 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IF 6IH 6IK 6IL 6IM 6IN 6J9 AAJGR AAWTH ABLEC ACREN ADYOE ADZIZ AFYQB ALMA_UNASSIGNED_HOLDINGS AMTXH BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI M43 OCL RIE RIL |
| ID | FETCH-LOGICAL-a289t-f64b8ab53fc0a9f6f7fc56ca20ab97d155e3dda182da9df4a1980e22cc17d783 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 4 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001353105400036&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Jan 15 06:20:39 EST 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a289t-f64b8ab53fc0a9f6f7fc56ca20ab97d155e3dda182da9df4a1980e22cc17d783 |
| OpenAccessLink | https://doi.org/10.1145/3691620.3695017 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_10765017 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-Oct.-27 |
| PublicationDateYYYYMMDD | 2024-10-27 |
| PublicationDate_xml | – month: 10 year: 2024 text: 2024-Oct.-27 day: 27 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] |
| PublicationTitleAbbrev | ASE |
| PublicationYear | 2024 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssib057256116 ssj0051577 |
| Score | 2.3121006 |
| Snippet | Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 443 |
| SubjectTerms | Accuracy Artificial intelligence Java JMH Microbenchmarking Runtime Software testing Steady-state Testing Time series analysis Time Series Classification Training Tuning |
| Title | AI-driven Java Performance Testing: Balancing Result Quality with Testing Time |
| URI | https://ieeexplore.ieee.org/document/10765017 |
| WOSCitedRecordID | wos001353105400036&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELWgYmAqH0V8ywOrIYkTX8wGiAoYqgp16FY5PhshoRS1aSX-Ped8UDEwsERWFEWRk7v3zvF7x9gVOgiu3VLQAUWaFyhMlBbCIyVDlVsttambTcBolE-netyK1WstjHOu3nzmrsOw_pePc7sKS2UU4UCEIoZttg0AjVir-3gyIPCOA9dp0jDhNEDr5ROn2Y1URIQSqlGVDrf41UylxpJh_59PsccGG1UeH__gzT7bcuUB63dtGXgbpYdsdPcscBGyGH8xa8PHG20AnwRTjfLtlt-HLY2WhvzVLVcfFW_MNL54WJjtLuNBIDJgk-Hj5OFJtG0ThKHqqRJepUVuikx6GxntlQdvM2VNEplCAxKBcBLRUGGBRqNPTazzyCWJtTEg5PKI9cp56Y4Zx4ygTDuNkoiW0ZHJHVWP6IECWWXSnLBBmJ7ZZ2OMMetm5vSP82dsNyFOEFJ_AuesVy1W7oLt2HX1vlxc1q_zG-6vn9o |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF60Cnqqj4pv9-B1Nckm2aw3FUurNRTJobcy2YcIkkqbFvz3zuahePDgJSwhhLDJzPfNZr9vCLnURjjXbs7woFmY5JqBF-bMakyGcaIkl1A1mxBpmkwmctyI1SstjDGm2nxmrtyw-pevZ2rplsowwgUSCl-sk40oDAO_lmu1n08kEL59x3bqRIxILUTj5uOH0TWPkQoFWKXG0t3kVzuVCk363X8-xw7p_ejy6PgbcXbJmin2SLdtzECbON0n6e2Q6bnLY_QRVkDHP-oAmjlbjeL1ht65TY0Kh_TFLJbvJa3tND6pW5ptL6NOItIjWf8hux-wpnECA6yfSmbjME8gj7hVHkgbW2FVFCsIPMil0EghDNcasLTQILUNwZeJZ4JAKV9okfAD0ilmhTkkVEcIZtJIzZFqgfQgMVg_aiswlOOIwxHpuemZftTWGNN2Zo7_OH9BtgbZ82g6GqZPJ2Q7QIbggCAQp6RTzpfmjGyqVfm2mJ9Xr_YLb7mjIQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=AI-driven+Java+Performance+Testing%3A+Balancing+Result+Quality+with+Testing+Time&rft.au=Traini%2C+Luca&rft.au=Di+Menna%2C+Federico&rft.au=Cortellessa%2C+Vittorio&rft.date=2024-10-27&rft.pub=ACM&rft.eissn=2643-1572&rft.spage=443&rft.epage=454&rft_id=info:doi/10.1145%2F3691620.3695017&rft.externalDocID=10765017 |