An Experience Report of Generating Load Tests Using Log-Recovered Workloads at Varying Granularities of User Behaviour
Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at different levels of granularity. A coarse-grained load test may miss detailed user behaviour, leading to a non-representative load test; while...
Uloženo v:
| Vydáno v: | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] s. 669 - 681 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.11.2019
|
| Témata: | |
| ISSN: | 2643-1572 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at different levels of granularity. A coarse-grained load test may miss detailed user behaviour, leading to a non-representative load test; while an extremely fine-grained load test would simply replay user actions step by step, leading to load tests that are costly to develop, execute and maintain. Workload recovery is at core of these load tests. Prior research often captures the workload as the frequency of user actions. However, there exists much valuable information in the context and sequences of user actions. Such richer information would ensure that the load tests that leverage such workloads are more field-representative. In this experience paper, we study the use of different granularities of user behaviour, i.e., basic user actions, basic user actions with contextual information and user action sequences with contextual information, when recovering workloads for use in the load testing of large-scale systems. We propose three approaches that are based on the three granularities of user behaviour and evaluate our approaches on four subject systems, namely Apache James, OpenMRS, Google Borg, and an ultra-large-scale industrial system (SA) from Alibaba. Our results show that our approach that is based on user action sequences with contextual information outperforms the other two approaches and can generate more representative load tests with similar throughput and CPU usage to the original field workload (i.e., mostly statistically insignificant or with small/trivial effect sizes). Such representative load tests are generated only based on a small number of clusters of users, leading to a low cost of conducting/maintaining such tests. Finally, we demonstrate that our approaches can detect injected users in the original field workloads with high precision and recall. Our paper demonstrates the importance of user action sequences with contextual information in the workload recovery of large-scale systems. |
|---|---|
| AbstractList | Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at different levels of granularity. A coarse-grained load test may miss detailed user behaviour, leading to a non-representative load test; while an extremely fine-grained load test would simply replay user actions step by step, leading to load tests that are costly to develop, execute and maintain. Workload recovery is at core of these load tests. Prior research often captures the workload as the frequency of user actions. However, there exists much valuable information in the context and sequences of user actions. Such richer information would ensure that the load tests that leverage such workloads are more field-representative. In this experience paper, we study the use of different granularities of user behaviour, i.e., basic user actions, basic user actions with contextual information and user action sequences with contextual information, when recovering workloads for use in the load testing of large-scale systems. We propose three approaches that are based on the three granularities of user behaviour and evaluate our approaches on four subject systems, namely Apache James, OpenMRS, Google Borg, and an ultra-large-scale industrial system (SA) from Alibaba. Our results show that our approach that is based on user action sequences with contextual information outperforms the other two approaches and can generate more representative load tests with similar throughput and CPU usage to the original field workload (i.e., mostly statistically insignificant or with small/trivial effect sizes). Such representative load tests are generated only based on a small number of clusters of users, leading to a low cost of conducting/maintaining such tests. Finally, we demonstrate that our approaches can detect injected users in the original field workloads with high precision and recall. Our paper demonstrates the importance of user action sequences with contextual information in the workload recovery of large-scale systems. |
| Author | Hassan, Ahmed E. Shang, Weiyi Wang, Yong Chen, Jinfu Lin, Jiangbin |
| Author_xml | – sequence: 1 givenname: Jinfu surname: Chen fullname: Chen, Jinfu organization: Concordia University – sequence: 2 givenname: Weiyi surname: Shang fullname: Shang, Weiyi organization: Concordia University – sequence: 3 givenname: Ahmed E. surname: Hassan fullname: Hassan, Ahmed E. organization: Queen's University – sequence: 4 givenname: Yong surname: Wang fullname: Wang, Yong organization: Alibaba Group – sequence: 5 givenname: Jiangbin surname: Lin fullname: Lin, Jiangbin organization: Alibaba Group |
| BookMark | eNotjk1Lw0AURUdRsK1u3biZP5A4H51ksqwlRqEg1EaX5SV5qaN1JsykQf-9KXV1uZfD4U7JhXUWCbnlLOacZfeL1zwWjGcxYyzRZ2TKU6G5UEzzczIRyVxGXKXiikxD-GRMjSWdkGFhaf7ToTdoa6Rr7JzvqWtpgRY99Mbu6MpBQzcY-kDLcBp20RprN6DHhr47_7UfkUChp2_gf49I4cEe9uBNbzAcfWVATx_wAwbjDv6aXLawD3jznzNSPuab5VO0eimel4tVBCKVfQRQMykaJrNEIrQJ1q3SMq0qmTUVphI0n2sl5tC0kGjZJlWmhEAheKNA1Shn5O7kNYi47bz5Hv9t9UgpJeUfZKBd0g |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/ASE.2019.00068 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library Online IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 1728125081 9781728125084 |
| EISSN | 2643-1572 |
| EndPage | 681 |
| ExternalDocumentID | 8952553 |
| Genre | orig-research |
| GroupedDBID | 29I 6IE 6IF 6IH 6IK 6IL 6IM 6IN 6J9 AAJGR AAWTH ABLEC ACREN ADYOE ADZIZ AFYQB ALMA_UNASSIGNED_HOLDINGS AMTXH APO BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI M43 OCL RIE RIL |
| ID | FETCH-LOGICAL-a273t-aac032d03963eaf6ecf5837bb39dbe73a8148524adfa683f6b9522e221d5a5ce3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 19 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000533303400062&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:39:58 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a273t-aac032d03963eaf6ecf5837bb39dbe73a8148524adfa683f6b9522e221d5a5ce3 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_8952553 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-Nov. |
| PublicationDateYYYYMMDD | 2019-11-01 |
| PublicationDate_xml | – month: 11 year: 2019 text: 2019-Nov. |
| PublicationDecade | 2010 |
| PublicationTitle | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] |
| PublicationTitleAbbrev | ASE |
| PublicationYear | 2019 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0051577 ssib040743839 |
| Score | 2.24653 |
| Snippet | Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 669 |
| SubjectTerms | Google Large-scale systems Load tests Software engineering Software log analysis Software performance Software systems Task analysis Testing Workload recovery |
| Title | An Experience Report of Generating Load Tests Using Log-Recovered Workloads at Varying Granularities of User Behaviour |
| URI | https://ieeexplore.ieee.org/document/8952553 |
| WOSCitedRecordID | wos000533303400062&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT8JAEN4A8eAJFYzv7MGjlT52u-2RGNADISQC4Ua23VljYlpTCr_fmbaABy_emk3T7PubmX7zDWOPvrBCAcW_JPqqwihwtJXaCUH4RnkgTBXMWU7UdBqtVvGsxZ4OuTAAUJHP4Jkeq3_5Jk-3FCobRLFECzhos7ZSqs7V2u8dQVAYkcJlfQsjTCvViDR6bjwYvo-Ix0XilC7Jqv4qpVIhybj7vz6csf4xJY_PDmBzzlqQXbDuviYDb45oj-2GGT_KF_Pavua55bW-NJGc-STXhs8RDja8Ygxgw4dDfuiOCndyip9_4Ssbrku-1AUlQvFXxDRirFYCrPS9Be5d3qgrbos-W4xH85c3p6mt4Gg0WEpH69QNfOMGeABB2xBSK9FXTZIgNgmoQEfoJ0lfaGN1GAU2THDcPvi-Z6SWKQSXrJPlGVwxbmOhUhMBeiqpwEsLTQ7rQmS1jNF-094169E0rr9r-Yx1M4M3fzffslNapzrd7451ymIL9-wk3ZWfm-KhWvMfq5CuIA |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT8JAEN4gmugJFYxv9-DRSrvd7eNIDIixEhKBcCNLd9aYGDBQ-P3OtAU8ePHWbJpm39_M9JtvGLsX0soQKP6l0FeVJgRHW6WdAKQwoQfS5MGcURL2etF4HPcr7GGbCwMAOfkMHukx_5dv5umKQmXNKFZoAft7bF9JKbwiW2uzeySBYUQal8U9jEAdhqVMo-fGzdZ7m5hcJE_pkrDqr2IqOZZ0av_rxTFr7JLyeH8LNyesArNTVttUZeDlIa2zdWvGdwLGvLCw-dzyQmGaaM48mWvDBwgIS55zBrDhwyFPdE2lOzlF0L_wlSXXGR_pBaVC8WdENeKs5hKs9L0h7l5e6iuuFg027LQHT12nrK7gaDRZMkfr1PWFcX08gqBtAKlV6K1Op35sphD6OkJPSQmpjdVB5NtgiuMWIIRnlFYp-GesOpvP4JxxG8swNRGgr5JKvLbQ6LAuRFarGC047V2wOk3j5LsQ0JiUM3j5d_MdO-wO3pJJ8tJ7vWJHtGZF8t81q2aLFdywg3SdfS4Xt_n6_wDlEbFn |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=An+Experience+Report+of+Generating+Load+Tests+Using+Log-Recovered+Workloads+at+Varying+Granularities+of+User+Behaviour&rft.au=Chen%2C+Jinfu&rft.au=Shang%2C+Weiyi&rft.au=Hassan%2C+Ahmed+E.&rft.au=Wang%2C+Yong&rft.date=2019-11-01&rft.pub=IEEE&rft.eissn=2643-1572&rft.spage=669&rft.epage=681&rft_id=info:doi/10.1109%2FASE.2019.00068&rft.externalDocID=8952553 |