Evaluating System-Level Test Generation for Industrial Software: A Comparison between Manual, Combinatorial and Model-Based Testing
Adequate testing of safety-critical systems is vital to ensure correct functional and non-functional operations. Previous research has shown that testing such systems requires a lot of effort, thus automated testing techniques have found a certain degree of success. However, automated testing has no...
Uloženo v:
| Vydáno v: | Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test s. 148 - 159 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
ACM
01.05.2022
|
| Témata: | |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Adequate testing of safety-critical systems is vital to ensure correct functional and non-functional operations. Previous research has shown that testing such systems requires a lot of effort, thus automated testing techniques have found a certain degree of success. However, automated testing has not replaced the need for manual testing, rather a common industrial practice exhibits a balance between automated and manual testing. In this respect, comparing manual testing with automated testing techniques continues to be an interesting topic to investigate. The need for this investigation is most apparent at system-level testing of industrial systems, where there is a lack of results on how different testing techniques perform concerning both structural and system-level metrics such as Modified Condition/Decision Coverage (MC/DC) and requirement coverage. In addition to the coverage, the cost of these techniques will also determine their efficiency and thus practical viability. In this paper, we have developed cost models for efficiency measurement and performed an experimental evaluation of manual testing, model-based testing (MBT), and combinatorial testing (CT) in terms of MC/DC and requirement coverage. The evaluation is done in an industrial context of a safety-critical system that controls several functions on-board the passenger trains. We have reported the dominant conditions of MC/DC affected by each technique while generating MC/DC adequate test suites. Moreover, we investigated differences and overlaps of test cases generated by each of the three techniques. The results showed that all test suites achieved 100% requirement coverage except the test suite generated by the pairwise testing strategy. However, MBT-generated test suites were more MC/DC adequate and provided a higher number of both similar and unique test cases. Moreover, unique test cases generated by MBT had an observable effect on MC/DC, which will complement manual testing to increase MC/DC coverage. The least dominant MC/DC condition fulfilled by the generated test cases by all three techniques is the 'independent effect of a condition on the outcomes of a decision'. Lastly, the evaluation also showed CT as the most efficient testing technique amongst the three in terms of time required for test generation and execution, but with an added cost parameter of manual identification of expected outcomes. CCS CONCEPTS *Computer systems organization →Embedded software; * Software and its engineering →Software verification and validation. |
|---|---|
| AbstractList | Adequate testing of safety-critical systems is vital to ensure correct functional and non-functional operations. Previous research has shown that testing such systems requires a lot of effort, thus automated testing techniques have found a certain degree of success. However, automated testing has not replaced the need for manual testing, rather a common industrial practice exhibits a balance between automated and manual testing. In this respect, comparing manual testing with automated testing techniques continues to be an interesting topic to investigate. The need for this investigation is most apparent at system-level testing of industrial systems, where there is a lack of results on how different testing techniques perform concerning both structural and system-level metrics such as Modified Condition/Decision Coverage (MC/DC) and requirement coverage. In addition to the coverage, the cost of these techniques will also determine their efficiency and thus practical viability. In this paper, we have developed cost models for efficiency measurement and performed an experimental evaluation of manual testing, model-based testing (MBT), and combinatorial testing (CT) in terms of MC/DC and requirement coverage. The evaluation is done in an industrial context of a safety-critical system that controls several functions on-board the passenger trains. We have reported the dominant conditions of MC/DC affected by each technique while generating MC/DC adequate test suites. Moreover, we investigated differences and overlaps of test cases generated by each of the three techniques. The results showed that all test suites achieved 100% requirement coverage except the test suite generated by the pairwise testing strategy. However, MBT-generated test suites were more MC/DC adequate and provided a higher number of both similar and unique test cases. Moreover, unique test cases generated by MBT had an observable effect on MC/DC, which will complement manual testing to increase MC/DC coverage. The least dominant MC/DC condition fulfilled by the generated test cases by all three techniques is the 'independent effect of a condition on the outcomes of a decision'. Lastly, the evaluation also showed CT as the most efficient testing technique amongst the three in terms of time required for test generation and execution, but with an added cost parameter of manual identification of expected outcomes. CCS CONCEPTS *Computer systems organization →Embedded software; * Software and its engineering →Software verification and validation. |
| Author | Zafar, Muhammad Nouman Enoiu, Eduard Afzal, Wasif |
| Author_xml | – sequence: 1 givenname: Muhammad Nouman surname: Zafar fullname: Zafar, Muhammad Nouman email: muhammad.nouman.zafar@mdh.se organization: Mälardalen University,Sweden – sequence: 2 givenname: Wasif surname: Afzal fullname: Afzal, Wasif email: wasif.afzal@mdh.se organization: Mälardalen University,Sweden – sequence: 3 givenname: Eduard surname: Enoiu fullname: Enoiu, Eduard email: eduard.enoiu@mdh.se organization: Mälardalen University,Sweden |
| BookMark | eNotjctOAjEUhmuiiYqsXbjpAzjYy1xad0gQSSAuwDU5nZ6aMUNLpgXC2hd3RFff4vsvt-TSB4-E3HM24jwvnmQh8lzxUc9KyOKCDHWlesGkFqpk12QY4xdjTKiKc1HckO_pAdo9pMZ_0tUpJtxmCzxgS9cYE52hx66XwVMXOjr3dh9T10BLV8GlI3T4TMd0ErY76JrYpwymI6KnS_B7aB9_lWk8pHAugbd0GSy22QtEtOeP_viOXDloIw7_OSAfr9P15C1bvM_mk_EiA6HylMnSKKZqzWWtS0SnreWscKZ0gIXoocA4IawwNVruOBh0NSCXjBluoZYD8vC32yDiZtc1W-hOG13pMmda_gCzxWOo |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK ESBDL RIE RIL |
| DOI | 10.1145/3524481.3527235 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings Open Access资源_IEL Journals IEEE Xplore IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9781450392860 1450392865 |
| EndPage | 159 |
| ExternalDocumentID | 9796409 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Vinnova funderid: 10.13039/501100001858 |
| GroupedDBID | 6IE 6IL ACM ALMA_UNASSIGNED_HOLDINGS APO CBEJK ESBDL GUFHI LHSKQ RIE RIL |
| ID | FETCH-LOGICAL-a284t-36b808c913c96eef9dd105fb6fae52b6f8abf22d2bced1f1abefcae1300b1dac3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 8 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000850254300016&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:51:38 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a284t-36b808c913c96eef9dd105fb6fae52b6f8abf22d2bced1f1abefcae1300b1dac3 |
| OpenAccessLink | https://ieeexplore.ieee.org/document/9796409 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_9796409 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-May |
| PublicationDateYYYYMMDD | 2022-05-01 |
| PublicationDate_xml | – month: 05 year: 2022 text: 2022-May |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test |
| PublicationTitleAbbrev | AST |
| PublicationYear | 2022 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssj0002871125 |
| Score | 1.8697959 |
| Snippet | Adequate testing of safety-critical systems is vital to ensure correct functional and non-functional operations. Previous research has shown that testing such... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 148 |
| SubjectTerms | Automation Combinatorial testing Control systems Costs Manuals Measurement Organizations Safety Critical Systems System-level Tests Test Coverage Test Generation |
| Title | Evaluating System-Level Test Generation for Industrial Software: A Comparison between Manual, Combinatorial and Model-Based Testing |
| URI | https://ieeexplore.ieee.org/document/9796409 |
| WOSCitedRecordID | wos000850254300016&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePCk0opvcvDYtPvK7sablhYPtRSs0luZPFYEWaW2-gP8485kl60HL56yD5aBCZuZSeb7PsauksDmGWDmFkgbi0SaXIDUGickNYABJgNPYPo0yabTfLFQsxbrNVgY55xvPnN9uvRn-fbNbGirbKAIN0lovZ0sSyusVrOfQpk_BuuavSdM5ABTC6w9wj6OWURybr_kU3z0GO__z-4B625heHzWBJhD1nJlh32Pan7u8plXdONiQo0_fI7rO69YpMnZHLNRvhXm4A-43n7Byl3zGz5sxAd53afF74G4SXv0CmtlqsTpIygtJ7W0V3GL0c56G2i4yx7Ho_nwTtRKCgIw_KxFnOo8yI0KY6NS5wplLeZVhU4LcDLCIQddRJGNtHE2LELQrjDg6KhLhxZMfMTa5VvpjhkPjNSFlanGzCaRCn9gazUovM9S0MqcsA45cPlekWUsa9-d_v34jO1FhCfwHYTnrL1ebdwF2zWf65eP1aWf4R_NAquW |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF6KCnpSacW3e_DYtHltkvWmpaViWgpW6a3MPiKCpFJb_QH-cWeSkHrw4mnzIAzMkv1mduf7hrHr0DVJDBi5ucIETih04oBQCick0oAAE0MhYPqcxuNxMpvJSYO1ay6MtbYoPrMduizO8s1Cr2mrrCuJN0lsvW0RYt5TsrXqHRWK_RGuK_0eLxRdDC4w-_A6OMY-NXT71UClwI_B_v8sH7DWhojHJzXEHLKGzZvsu18pdOcvvBQcd1Iq_eFTXOF5qSNN7uYYj_JNaw7-iCvuFyztDb_lvbr9IK8qtfgISJ20Ta8wW6ZcnD6C3HDql_bm3CHemcIGGm6xp0F_2hs6VS8FBxCAVk4QqcRNtPQCLSNrM2kMRlaZijKwwschAZX5vvGVtsbLPFA202DpsEt5BnRwxLbyRW6PGXe1UJkRkcLYJhQSf2FjFEi8jyNQUp-wJjlw_l7KZcwr353-_fiK7Q6no3Se3o8fztieT-yCop7wnG2tlmt7wXb05-r1Y3lZzPYP4Suu3Q |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+of+the+3rd+ACM%2FIEEE+International+Conference+on+Automation+of+Software+Test&rft.atitle=Evaluating+System-Level+Test+Generation+for+Industrial+Software%3A+A+Comparison+between+Manual%2C+Combinatorial+and+Model-Based+Testing&rft.au=Zafar%2C+Muhammad+Nouman&rft.au=Afzal%2C+Wasif&rft.au=Enoiu%2C+Eduard&rft.date=2022-05-01&rft.pub=ACM&rft.spage=148&rft.epage=159&rft_id=info:doi/10.1145%2F3524481.3527235&rft.externalDocID=9796409 |