TerzoN: Human-in-the-Loop Software Testing with a Composite Oracle
Software testing is difficult, tedious, and may consume 28%–50% of software engineering labor. Automatic test generators aim to ease this burden but have important trade-offs. Fuzzers use an implicit oracle that can detect obviously invalid results, but the oracle problem has no general solution, an...
Saved in:
| Published in: | Proceedings of the ACM on software engineering Vol. 2; no. FSE; pp. 1983 - 2005 |
|---|---|
| Main Authors: | , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York, NY, USA
ACM
19.06.2025
|
| Subjects: | |
| ISSN: | 2994-970X, 2994-970X |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Software testing is difficult, tedious, and may consume 28%–50% of software engineering labor. Automatic test generators aim to ease this burden but have important trade-offs. Fuzzers use an implicit oracle that can detect obviously invalid results, but the oracle problem has no general solution, and an implicit oracle cannot automatically evaluate correctness. Test suite generators like EvoSuite use the program under test as the oracle and therefore cannot evaluate correctness. Property-based testing tools evaluate correctness, but users have difficulty coming up with properties to test and understanding whether their properties are correct. Consequently, practitioners create many test suites manually and often use an example-based oracle to tediously specify correct input and output examples. To help bridge the gaps among various oracle and tool types, we present the Composite Oracle, which organizes various oracle types into a hierarchy and renders a single test result per example execution. To understand the Composite Oracle’s practical properties, we built TerzoN, a test suite generator that includes a particular instantiation of the Composite Oracle. TerzoN displays all the test results in an integrated view composed from the results of three types of oracles and finds some types of test assertion inconsistencies that might otherwise lead to misleading test results. We evaluated TerzoN in a randomized controlled trial with 14 professional software engineers with a popular industry tool, fast-check, as the control. Participants using TerzoN elicited 72% more bugs (p < 0.01), accurately described more than twice the number of bugs (p < 0.01) and tested 16% more quickly (p < 0.05) relative to fast-check. |
|---|---|
| AbstractList | Software testing is difficult, tedious, and may consume 28%–50% of software engineering labor. Automatic test generators aim to ease this burden but have important trade-offs. Fuzzers use an implicit oracle that can detect obviously invalid results, but the oracle problem has no general solution, and an implicit oracle cannot automatically evaluate correctness. Test suite generators like EvoSuite use the program under test as the oracle and therefore cannot evaluate correctness. Property-based testing tools evaluate correctness, but users have difficulty coming up with properties to test and understanding whether their properties are correct. Consequently, practitioners create many test suites manually and often use an example-based oracle to tediously specify correct input and output examples. To help bridge the gaps among various oracle and tool types, we present the Composite Oracle, which organizes various oracle types into a hierarchy and renders a single test result per example execution. To understand the Composite Oracle’s practical properties, we built TerzoN, a test suite generator that includes a particular instantiation of the Composite Oracle. TerzoN displays all the test results in an integrated view composed from the results of three types of oracles and finds some types of test assertion inconsistencies that might otherwise lead to misleading test results. We evaluated TerzoN in a randomized controlled trial with 14 professional software engineers with a popular industry tool, fast-check, as the control. Participants using TerzoN elicited 72% more bugs (p < 0.01), accurately described more than twice the number of bugs (p < 0.01) and tested 16% more quickly (p < 0.05) relative to fast-check. |
| ArticleNumber | FSE089 |
| Author | Wei, Amy Davis, Matthew C. Myers, Brad A. Sunshine, Joshua |
| Author_xml | – sequence: 1 givenname: Matthew C. orcidid: 0000-0003-2366-8436 surname: Davis fullname: Davis, Matthew C. email: mcd2@cs.cmu.edu organization: Carnegie Mellon University, Pittsburgh, USA – sequence: 2 givenname: Amy orcidid: 0009-0005-6046-1815 surname: Wei fullname: Wei, Amy email: weia@umich.edu organization: University of Michigan, Ann Arbor, USA – sequence: 3 givenname: Brad A. orcidid: 0000-0002-4769-0219 surname: Myers fullname: Myers, Brad A. email: bam@cs.cmu.edu organization: Carnegie Mellon University, Pittsburgh, USA – sequence: 4 givenname: Joshua orcidid: 0000-0002-9672-5297 surname: Sunshine fullname: Sunshine, Joshua email: sunshine@cs.cmu.edu organization: Carnegie Mellon University, Pittsburgh, USA |
| BookMark | eNpt0E1LAzEQBuAgFay1ePeUm6dosh_JxpsWtUKxB_fgbZlmJzbS3ZQkUvTXW2kVEU8zzPswh_eYDHrfIyGngl8IUZSXucp0XuoDMsy0LphW_Hnwaz8i4xhfOefbixCKD8lNjeHDP17R6VsHPXM9S0tkM-_X9MnbtIGAtMaYXP9CNy4tKdCJ79Y-uoR0HsCs8IQcWlhFHO_niNR3t_Vkymbz-4fJ9YxBpTWrECUsMgRtrLQWFrkpC1MujIZWobYyV6BEpW3WyjbnogBoucwqYYWUXIl8RNjurQk-xoC2MS5Bcr5PAdyqEbz5qqDZV7D153_8OrgOwvs_8mwnwXQ_6Dv8BAmeZG8 |
| CitedBy_id | crossref_primary_10_1145_3765754 |
| Cites_doi | 10.1145/3183440.3195001 10.1109/ISSRE.1997.630851 10.1145/3597503.3639581 10.1007/s10664-021-10072-8 10.1109/ASE56229.2023.00193 10.1109/TSE.2012.14 10.1145/3587157 10.1145/3580446 10.21105/joss.01891 10.1109/QSIC.2014.33 10.1109/TSE.2017.2776152 10.1016/j.jss.2013.02.061 10.1145/2884781.2884847 10.1109/CHASE52884.2021.00026 10.1109/TSE.2013.59 10.1145/174800.174808 10.1109/MC.2016.200 10.1145/3597503.3608128 10.1145/2771783.2771801 10.1007/s10664-013-9279-3 10.1007/s10664-017-9570-9 10.1145/3611643.3616327 10.1109/ICSE.1998.671118 10.1109/ICSE.2007.37 10.1016/j.jss.2018.03.052 10.1145/351240.351266 10.2307/2288652 10.1176/appi.ajp.2012.12070999 10.1109/ICICoS56336.2022.9930600 10.1109/TSE.2019.2946563 10.1080/00031305.1998.10480559 10.1109/TSE.2014.2372785 10.48550/arXiv.2307.04346 10.1145/191666.191729 10.1145/581339.581359 10.1145/1131421.1131423 10.1145/3477132.3483540 10.1145/2699688 10.1093/acprof:oso/9780195385540.001.0001 |
| ContentType | Journal Article |
| Copyright | Copyright is held by the owner/author(s). Publication rights licensed to ACM. |
| Copyright_xml | – notice: Copyright is held by the owner/author(s). Publication rights licensed to ACM. |
| DBID | AAYXX CITATION |
| DOI | 10.1145/3729359 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | CrossRef |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 2994-970X |
| EndPage | 2005 |
| ExternalDocumentID | 10_1145_3729359 3729359 |
| GrantInformation_xml | – fundername: National Science Foundation grantid: 1910264, 2150217, 233977 funderid: https:\/\/doi.org\/10.13039\/100000001 |
| GroupedDBID | AAKMM ACM AEJOY AKRVB ALMA_UNASSIGNED_HOLDINGS LHSKQ M~E AAYXX CITATION |
| ID | FETCH-LOGICAL-a899-8ee6ab2ea9cf6ffab3c54c5bc9ad7e9f637a7189f2d6d3014aad06281f1660713 |
| ISSN | 2994-970X |
| IngestDate | Sat Nov 29 07:51:01 EST 2025 Tue Nov 18 21:24:15 EST 2025 Mon Jul 14 20:49:04 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | FSE |
| Keywords | Empirical software engineering software testing usable testing user study experiments automatic test generation composite oracle human subjects |
| Language | English |
| License | Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-a899-8ee6ab2ea9cf6ffab3c54c5bc9ad7e9f637a7189f2d6d3014aad06281f1660713 |
| ORCID | 0000-0002-9672-5297 0000-0003-2366-8436 0009-0005-6046-1815 0000-0002-4769-0219 |
| OpenAccessLink | https://dl.acm.org/doi/10.1145/3729359 |
| PageCount | 23 |
| ParticipantIDs | crossref_citationtrail_10_1145_3729359 crossref_primary_10_1145_3729359 acm_primary_3729359 |
| PublicationCentury | 2000 |
| PublicationDate | 20250619 2025-06-19 |
| PublicationDateYYYYMMDD | 2025-06-19 |
| PublicationDate_xml | – month: 06 year: 2025 text: 20250619 day: 19 |
| PublicationDecade | 2020 |
| PublicationPlace | New York, NY, USA |
| PublicationPlace_xml | – name: New York, NY, USA |
| PublicationTitle | Proceedings of the ACM on software engineering |
| PublicationTitleAbbrev | ACM PACMSE |
| PublicationYear | 2025 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| References | Nicolas Dubien. 2024. fast-check official documentation | fast-check. https://fast-check.dev Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, and Andy Zaidman. 2017. Developer testing in the ide: Patterns, beliefs, and behavior. IEEE Transactions on Software Engineering, 45, 3 (2017), 261–284. Marc Fisher, Mingming Cao, Gregg Rothermel, Curtis R Cook, and Margaret M Burnett. 2002. Automated test case generation for spreadsheets. In Proceedings of the 24th International Conference on Software Engineering (ICSE). 141–151. Matthew C. Davis, Sangheon Choi, Sam Estep, Brad A. Myers, and Joshua Sunshine. 2023. NaNofuzz: A Usable Tool for Automatic Test Generation. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA. 1114–1126. isbn:9798400703270 https://doi.org/10.1145/3611643.3616327 10.1145/3611643.3616327 Sebastian P Ng, Tafline Murnane, Karl Reed, D Grant, and Tsong Yueh Chen. 2004. A preliminary survey on software testing practices in Australia. In 2004 Australian Software Engineering Conference. Proceedings.. 116–125. Sebastiano Panichella, Annibale Panichella, Moritz Beller, Andy Zaidman, and Harald C Gall. 2016. The impact of test case summaries on bug fixing performance: An empirical investigation. In Proceedings of the 38th international conference on software engineering. 547–558. G. Rothermel, L. Li, and M. Burnett. 1997. Testing strategies for form-based visual programs. In Proceedings The Eighth International Symposium on Software Reliability Engineering. 96–107. https://doi.org/10.1109/ISSRE.1997.630851 10.1109/ISSRE.1997.630851 Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the fifth ACM SIGPLAN international conference on Functional programming. 268–279. Eduard Enoiu and Robert Feldt. 2021. Towards Human-Like Automated Test Generation: Perspectives from Cognition and Problem Solving. In 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). 123–124. https://doi.org/10.1109/CHASE52884.2021.00026 10.1109/CHASE52884.2021.00026 Harrison Goldstein, Joseph W Cutler, Adam Stein, Benjamin C Pierce, and Andrew Head. 2022. Some Problems with Properties. In Proc. Workshop on the Human Aspects of Types and Reasoning Assistants (HATRA). Pankaj Jalote. 2008. A concise introduction to software engineering. Springer Science & Business Media. Jeshua S Kracht, Jacob Z Petrovic, and Kristen R Walcott-Justice. 2014. Empirically evaluating the quality of automatically generated and manually written test suites. In 2014 14th International Conference on Quality Software. 256–265. Matthew C. Davis, Amy Wei, Sangheon Choi, and Sam Estep. 2024. NaNofuzz - Visual Studio Marketplace. https://marketplace.visualstudio.com/items?itemName=penrose.nanofuzz [Online; accessed 2024-09-01] Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2023. CAT-LM Training Language Models on Aligned Code And Tests. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). 409–420. https://doi.org/10.1109/ASE56229.2023.00193 10.1109/ASE56229.2023.00193 Brad A Myers. 1994. Challenges of HCI Design and Implementation. Interactions, 1, 1 (1994), jan, 73–83. issn:1072-5520 https://doi.org/10.1145/174800.174808 10.1145/174800.174808 Mathieu Nassif, Alexa Hernandez, Ashvitha Sridharan, and Martin P Robillard. 2021. Generating unit tests for documentation. IEEE Transactions on Software Engineering. David R. MacIver, Zac Hatfield-Dodds, and many other contributors. 2019. Hypothesis: A new approach to property-based testing. https://doi.org/10.21105/joss.01891 10.21105/joss.01891 Sebastian Baltes and Paul Ralph. 2022. Sampling in software engineering research: a critical review and guidelines. Empirical Software Engineering, 27, 4 (2022), April, 94. issn:1573-7616 https://doi.org/10.1007/s10664-021-10072-8 10.1007/s10664-021-10072-8 Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 152–158. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology, 3, 2 (2006), 77–101. GitHub. 2023. GitHub Codespaces. https://github.com/features/codespaces [Online; accessed 2023-07-21] Valentin JM Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering, 47, 11 (2019), 2312–2331. Matthew C. Davis, Emad Aghayi, Thomas D. Latoza, Xiaoyin Wang, Brad A Myers, and Joshua Sunshine. 2023. What’s (Not) Working in Programmer User Studies? ACM Trans. Softw. Eng. Methodol., 32, 5 (2023), Article 120, jul, 32 pages. issn:1049-331X https://doi.org/10.1145/3587157 10.1145/3587157 Nicolas Dubien. 2024. fast-check. https://www.npmjs.com/package/fast-check Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, and Peng Cheng. 2021. UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers.. In USENIX Security Symposium. 2777–2794. Google DevOps Research and Assessment. 2024. 2024 State of DevOps Report. Google. John Wrenn, Tim Nelson, and Shriram Krishnamurthi. 2021. Using Relational Problems to Teach Property-Based Testing. The art science and engineering of programming, 5, 2 (2021). Stephen Cass. 2024. Top Programming Languages 2024 - IEEE Spectrum. https://spectrum.ieee.org/top-programming-languages-2024 Nicolas Dubien. 2024. dubzzz/fast-check. https://github.com/dubzzz/fast-check original-date: 2017-10-30T23:41:11Z Novi Setiani, Ridi Ferdiana, and Rudy Hartanto. 2022. Understandable Automatic Generated Unit Tests using Semantic and Format Improvement. In 2022 6th International Conference on Informatics and Computational Sciences (ICICoS). 122–127. https://doi.org/10.1109/ICICoS56336.2022.9930600 10.1109/ICICoS56336.2022.9930600 Jerry L. Hintze and Ray D. Nelson. 1998. Violin Plots: A Box Plot-Density Trace Synergism. The American Statistician, 52, 2 (1998), May, 181–184. issn:0003-1305 https://doi.org/10.1080/00031305.1998.10480559 10.1080/00031305.1998.10480559 Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In 29th International Conference on Software Engineering (ICSE’07). IEEE, Minneapolis, MN, USA. 75–84. isbn:978-0-7695-2828-1 https://doi.org/10.1109/ICSE.2007.37 ISSN: 0270-5257 10.1109/ICSE.2007.37 Karen J Rothermel, Curtis R Cook, Margaret M Burnett, Justin Schonfeld, Thomas RG Green, and Gregg Rothermel. 2000. WYSIWYT testing in the spreadsheet paradigm: An empirical evaluation. In Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium. 230–239. Amy J Ko, Thomas D LaToza, and Margaret M Burnett. 2015. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering, 20, 1 (2015), 110–141. Caitlin Sadowski and Thomas Zimmermann. 2019. Rethinking productivity in software engineering. Springer Nature. Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785 10.1109/TSE.2014.2372785 James Bornholt, Rajeev Joshi, Vytautas Astrauskas, Brendan Cully, Bernhard Kragl, Seth Markle, Kyle Sauri, Drew Schleit, Grant Slatton, Serdar Tasiran, Jacob Van Geffen, and Andrew Warfield. 2021. Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP ’21). Association for Computing Machinery, New York, NY, USA. 836–850. isbn:9781450387095 https://doi.org/10.1145/3477132.3483540 10.1145/3477132.3483540 Michał Zalewski. 2014. american fuzzy lop. https://lcamtuf.coredump.cx/afl/ [Online; accessed 2023-06-27] Andrea Arcuri. 2018. An experience report on applying software testing academic results in industry: we need usable automated test generation. Empirical Software Engineering, 23, 4 (2018), 1959–1981. Tristan Teufel and contributors. 2022. Jest Runner. https://github.com/firsttris/vscode-jest-runner [Online; accessed 2022-11-10] Gerard O’Regan. 2019. Fundamentals of Software Testing. In Concise Guide to Software Testing. Springer, 59–78. Harrison Goldstein, Joseph W. Cutler, Daniel Dickstein, Benjamin C. Pierce, and Andrew Head. 2024. Property-Based Testing in Practice. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machinery, New York, NY, USA. 1–13. isbn:9798400702174 https://doi.org/10.1145/3597503.3639581 10.1145/3597503.3639581 José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software development: A controlled experiment and think-aloud observations. In Proceedings of the 2015 international symposium on software testing and analysis. 338–349. Microsoft. 2023. Overview of Microsoft IntelliTester. https://learn.microsoft.com/en-us/visualstudio/test/intellitest-manual/ [Online; accessed 2023-01-27] Alex Groce, Todd Kulesza, Chaoqiang Zhang, Shalini Shamasunder, Margaret Burnett, Weng-Keen Wong, Simone Stumpf, Shubhomoy Das, Amber Shinsel, Forrest Bice, and Kevin McIntosh. 2014. You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems. IEEE Transactions on Software Engineering, 40, 3 (2014), March, 307–323. issn:1939-3520 https://doi.org/10.1109/TSE.2013.59 10.1109/TSE.2013.59 Priyadarshi Tripathy and Kshirasagar Naik. 2011. Software t Brooke John (e_1_2_1_9_1) 1996 Tripathy Priyadarshi (e_1_2_1_64_1) e_1_2_1_60_1 Sadowski Caitlin (e_1_2_1_59_1) e_1_2_1_20_1 Roy Devjeet (e_1_2_1_58_1) 2020 Li Yuwei (e_1_2_1_36_1) 2021 e_1_2_1_24_1 e_1_2_1_22_1 e_1_2_1_43_1 e_1_2_1_28_1 e_1_2_1_49_1 e_1_2_1_26_1 Source Facebook Open (e_1_2_1_62_1) 2024 e_1_2_1_31_1 e_1_2_1_54_1 Zalewski Michał (e_1_2_1_67_1) 2023 e_1_2_1_56_1 e_1_2_1_6_1 e_1_2_1_12_1 e_1_2_1_35_1 e_1_2_1_50_1 e_1_2_1_4_1 e_1_2_1_10_1 O’Regan Gerard (e_1_2_1_47_1) e_1_2_1_52_1 e_1_2_1_2_1 e_1_2_1_39_1 e_1_2_1_14_1 e_1_2_1_37_1 e_1_2_1_18_1 Nassif Mathieu (e_1_2_1_44_1) 2021 Ng Sebastian P (e_1_2_1_45_1) 2004 Grano Giovanni (e_1_2_1_29_1) 2018 (e_1_2_1_41_1) 2023 Rothermel Karen J (e_1_2_1_57_1) 2000 Wrenn John (e_1_2_1_66_1) 2021 Somers James (e_1_2_1_61_1) 2023 Teufel Tristan (e_1_2_1_63_1) 2022 e_1_2_1_42_1 e_1_2_1_65_1 e_1_2_1_40_1 e_1_2_1_23_1 e_1_2_1_46_1 e_1_2_1_21_1 e_1_2_1_48_1 Goldstein Harrison (e_1_2_1_27_1) 2022 Jalote Pankaj (e_1_2_1_33_1) e_1_2_1_7_1 Codespaces GitHub (e_1_2_1_25_1) 2023 e_1_2_1_30_1 e_1_2_1_55_1 e_1_2_1_5_1 Braun Virginia (e_1_2_1_8_1) 2006 e_1_2_1_3_1 e_1_2_1_13_1 e_1_2_1_34_1 e_1_2_1_51_1 e_1_2_1_1_1 e_1_2_1_11_1 e_1_2_1_32_1 e_1_2_1_53_1 e_1_2_1_17_1 e_1_2_1_38_1 e_1_2_1_15_1 Davis Matthew C. (e_1_2_1_16_1) 2024 e_1_2_1_19_1 |
| References_xml | – reference: Sebastian Baltes and Paul Ralph. 2022. Sampling in software engineering research: a critical review and guidelines. Empirical Software Engineering, 27, 4 (2022), April, 94. issn:1573-7616 https://doi.org/10.1007/s10664-021-10072-8 10.1007/s10664-021-10072-8 – reference: Brad A Myers, Amy J Ko, Thomas D LaToza, and YoungSeok Yoon. 2016. Programmers are users too: Human-centered methods for improving programming tools. Computer, 49, 7 (2016), 44–52. – reference: James Somers. 2023. What if writing tests was a joyful experience? https://blog.janestreet.com/the-joy-of-expect-tests/ [Online; accessed 2023-01-22] – reference: Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785 10.1109/TSE.2014.2372785 – reference: Microsoft. 2023. Overview of Microsoft IntelliTester. https://learn.microsoft.com/en-us/visualstudio/test/intellitest-manual/ [Online; accessed 2023-01-27] – reference: G. Rothermel, L. Li, and M. Burnett. 1997. Testing strategies for form-based visual programs. In Proceedings The Eighth International Symposium on Software Reliability Engineering. 96–107. https://doi.org/10.1109/ISSRE.1997.630851 10.1109/ISSRE.1997.630851 – reference: Robert Rosenthal and Ralph L Rosnow. 2008. Essentials of behavioral research: Methods and data analysis. – reference: G. Rothermel, L. Li, C. DuPuis, and M. Burnett. 1998. What you see is what you test: a methodology for testing form-based visual programs. In Proceedings of the 20th International Conference on Software Engineering. 198–207. https://doi.org/10.1109/ICSE.1998.671118 ISSN: 0270-5257 10.1109/ICSE.1998.671118 – reference: Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2024. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machinery, New York, NY, USA. 1–13. isbn:9798400702174 https://doi.org/10.1145/3597503.3608128 10.1145/3597503.3608128 – reference: Marc Fisher, Gregg Rothermel, Darren Brown, Mingming Cao, Curtis Cook, and Margaret Burnett. 2006. Integrating automated test generation into the WYSIWYT spreadsheet testing methodology. ACM Transactions on Software Engineering and Methodology (TOSEM), 15, 2 (2006), 150–194. – reference: John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry, 189, 194 (1996), 4–7. – reference: Caitlin Sadowski and Thomas Zimmermann. 2019. Rethinking productivity in software engineering. Springer Nature. – reference: Priyadarshi Tripathy and Kshirasagar Naik. 2011. Software testing and quality assurance: theory and practice. John Wiley & Sons. – reference: Michał Zalewski. 2014. american fuzzy lop. https://lcamtuf.coredump.cx/afl/ [Online; accessed 2023-06-27] – reference: Matthew C Davis and Amy Wei. 2025. Reproduction Package for Article “TerzoN: Human-in-the-Loop Software Testing with a Composite Oracle”. https://doi.org/10.1145/3580446 10.1145/3580446 – reference: James Bornholt, Rajeev Joshi, Vytautas Astrauskas, Brendan Cully, Bernhard Kragl, Seth Markle, Kyle Sauri, Drew Schleit, Grant Slatton, Serdar Tasiran, Jacob Van Geffen, and Andrew Warfield. 2021. Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP ’21). Association for Computing Machinery, New York, NY, USA. 836–850. isbn:9781450387095 https://doi.org/10.1145/3477132.3483540 10.1145/3477132.3483540 – reference: ISO. 2018. Ergonomics of human-system interaction—Part 11: Usability: Definitions and concepts ISO 9241–11: 2018 (en). – reference: Jeshua S Kracht, Jacob Z Petrovic, and Kristen R Walcott-Justice. 2014. Empirically evaluating the quality of automatically generated and manually written test suites. In 2014 14th International Conference on Quality Software. 256–265. – reference: Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, and Peng Cheng. 2021. UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers.. In USENIX Security Symposium. 2777–2794. – reference: Gerard O’Regan. 2019. Fundamentals of Software Testing. In Concise Guide to Software Testing. Springer, 59–78. – reference: Anastasia Danilova. 2022. How to Conduct Security Studies with Software Developers. Ph. D. Dissertation. Universitäts-und Landesbibliothek Bonn. – reference: Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2023. CAT-LM Training Language Models on Aligned Code And Tests. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). 409–420. https://doi.org/10.1109/ASE56229.2023.00193 10.1109/ASE56229.2023.00193 – reference: Andrea Arcuri. 2018. An experience report on applying software testing academic results in industry: we need usable automated test generation. Empirical Software Engineering, 23, 4 (2018), 1959–1981. – reference: Facebook Open Source. 2023. Jest - Delightful Javascript Testing. https://jestjs.io/ [Online; accessed 2024-07-08] – reference: Novi Setiani, Ridi Ferdiana, and Rudy Hartanto. 2022. Understandable Automatic Generated Unit Tests using Semantic and Format Improvement. In 2022 6th International Conference on Informatics and Computational Sciences (ICICoS). 122–127. https://doi.org/10.1109/ICICoS56336.2022.9930600 10.1109/ICICoS56336.2022.9930600 – reference: Matthew C. Davis, Amy Wei, Sangheon Choi, and Sam Estep. 2024. NaNofuzz - Visual Studio Marketplace. https://marketplace.visualstudio.com/items?itemName=penrose.nanofuzz [Online; accessed 2024-09-01] – reference: Google DevOps Research and Assessment. 2024. 2024 State of DevOps Report. Google. – reference: Mathieu Nassif, Alexa Hernandez, Ashvitha Sridharan, and Martin P Robillard. 2021. Generating unit tests for documentation. IEEE Transactions on Software Engineering. – reference: Sebastiano Panichella, Annibale Panichella, Moritz Beller, Andy Zaidman, and Harald C Gall. 2016. The impact of test case summaries on bug fixing performance: An empirical investigation. In Proceedings of the 38th international conference on software engineering. 547–558. – reference: Darrel A. Regier, William E. Narrow, Diana E. Clarke, Helena C. Kraemer, S. Janet Kuramoto, Emily A. Kuhl, and David J. Kupfer. 2013. DSM-5 Field Trials in the United States and Canada, Part II: Test-Retest Reliability of Selected Categorical Diagnoses. American Journal of Psychiatry, 170, 1 (2013), Jan., 59–70. issn:0002-953X https://doi.org/10.1176/appi.ajp.2012.12070999 Publisher: American Psychiatric Publishing 10.1176/appi.ajp.2012.12070999 – reference: Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology, 3, 2 (2006), 77–101. – reference: Matthew C. Davis, Emad Aghayi, Thomas D. Latoza, Xiaoyin Wang, Brad A Myers, and Joshua Sunshine. 2023. What’s (Not) Working in Programmer User Studies? ACM Trans. Softw. Eng. Methodol., 32, 5 (2023), Article 120, jul, 32 pages. issn:1049-331X https://doi.org/10.1145/3587157 10.1145/3587157 – reference: Jerry L. Hintze and Ray D. Nelson. 1998. Violin Plots: A Box Plot-Density Trace Synergism. The American Statistician, 52, 2 (1998), May, 181–184. issn:0003-1305 https://doi.org/10.1080/00031305.1998.10480559 10.1080/00031305.1998.10480559 – reference: Devjeet Roy, Ziyi Zhang, Maggie Ma, Venera Arnaoudova, Annibale Panichella, Sebastiano Panichella, Danielle Gonzalez, and Mehdi Mirakhorli. 2020. DeepTC-Enhancer: Improving the readability of automatically generated tests. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 287–298. – reference: Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the fifth ACM SIGPLAN international conference on Functional programming. 268–279. – reference: Amy J Ko, Thomas D LaToza, and Margaret M Burnett. 2015. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering, 20, 1 (2015), 110–141. – reference: Marc Fisher, Mingming Cao, Gregg Rothermel, Curtis R Cook, and Margaret M Burnett. 2002. Automated test case generation for spreadsheets. In Proceedings of the 24th International Conference on Software Engineering (ICSE). 141–151. – reference: Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, and Andy Zaidman. 2017. Developer testing in the ide: Patterns, beliefs, and behavior. IEEE Transactions on Software Engineering, 45, 3 (2017), 261–284. – reference: Bilal Amir and Paul Ralph. 2018. There is no random sampling in software engineering research. In Proceedings of the 40th International Conference on Software Engineering: companion proceeedings. 344–345. – reference: Stephen Cass. 2024. Top Programming Languages 2024 - IEEE Spectrum. https://spectrum.ieee.org/top-programming-languages-2024 – reference: David R. MacIver, Zac Hatfield-Dodds, and many other contributors. 2019. Hypothesis: A new approach to property-based testing. https://doi.org/10.21105/joss.01891 10.21105/joss.01891 – reference: Valentin JM Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering, 47, 11 (2019), 2312–2331. – reference: Cyrus R. Mehta and Nitin R. Patel. 1983. A Network Algorithm for Performing Fisher’s Exact Test in r × c Contingency Tables. J. Amer. Statist. Assoc., 78, 382 (1983), 427–434. issn:0162-1459 https://doi.org/10.2307/2288652 Publisher: [American Statistical Association, Taylor & Francis, Ltd.] 10.2307/2288652 – reference: Brad A Myers. 1994. Challenges of HCI Design and Implementation. Interactions, 1, 1 (1994), jan, 73–83. issn:1072-5520 https://doi.org/10.1145/174800.174808 10.1145/174800.174808 – reference: Sebastian P Ng, Tafline Murnane, Karl Reed, D Grant, and Tsong Yueh Chen. 2004. A preliminary survey on software testing practices in Australia. In 2004 Australian Software Engineering Conference. Proceedings.. 116–125. – reference: Matthew C. Davis, Sangheon Choi, Sam Estep, Brad A. Myers, and Joshua Sunshine. 2023. NaNofuzz: A Usable Tool for Automatic Test Generation. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA. 1114–1126. isbn:9798400703270 https://doi.org/10.1145/3611643.3616327 10.1145/3611643.3616327 – reference: Nicolas Dubien. 2024. dubzzz/fast-check. https://github.com/dubzzz/fast-check original-date: 2017-10-30T23:41:11Z – reference: Karen J Rothermel, Curtis R Cook, Margaret M Burnett, Justin Schonfeld, Thomas RG Green, and Gregg Rothermel. 2000. WYSIWYT testing in the spreadsheet paradigm: An empirical evaluation. In Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium. 230–239. – reference: Giovanni Grano, Simone Scalabrino, Harald C Gall, and Rocco Oliveto. 2018. An empirical investigation on the readability of manual and generated test cases. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC). 348–3483. – reference: Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2024. Can Large Language Models Write Good Property-Based Tests? https://doi.org/10.48550/arXiv.2307.04346 arXiv:2307.04346 [cs] 10.48550/arXiv.2307.04346 – reference: GitHub. 2023. GitHub Codespaces. https://github.com/features/codespaces [Online; accessed 2023-07-21] – reference: Harrison Goldstein, Joseph W Cutler, Adam Stein, Benjamin C Pierce, and Andrew Head. 2022. Some Problems with Properties. In Proc. Workshop on the Human Aspects of Types and Reasoning Assistants (HATRA). – reference: José Miguel Rojas, Gordon Fraser, and Andrea Arcuri. 2015. Automated unit test generation during software development: A controlled experiment and think-aloud observations. In Proceedings of the 2015 international symposium on software testing and analysis. 338–349. – reference: Nicolas Dubien. 2024. fast-check official documentation | fast-check. https://fast-check.dev/ – reference: Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In 29th International Conference on Software Engineering (ICSE’07). IEEE, Minneapolis, MN, USA. 75–84. isbn:978-0-7695-2828-1 https://doi.org/10.1109/ICSE.2007.37 ISSN: 0270-5257 10.1109/ICSE.2007.37 – reference: Eduard Enoiu and Robert Feldt. 2021. Towards Human-Like Automated Test Generation: Perspectives from Cognition and Problem Solving. In 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). 123–124. https://doi.org/10.1109/CHASE52884.2021.00026 10.1109/CHASE52884.2021.00026 – reference: Alex Groce, Todd Kulesza, Chaoqiang Zhang, Shalini Shamasunder, Margaret Burnett, Weng-Keen Wong, Simone Stumpf, Shubhomoy Das, Amber Shinsel, Forrest Bice, and Kevin McIntosh. 2014. You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems. IEEE Transactions on Software Engineering, 40, 3 (2014), March, 307–323. issn:1939-3520 https://doi.org/10.1109/TSE.2013.59 10.1109/TSE.2013.59 – reference: Saswat Anand, Edmund K. Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, Antonia Bertolino, J. Jenny Li, and Hong Zhu. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 86, 8 (2013), 1978–2001. issn:0164-1212 https://doi.org/10.1016/j.jss.2013.02.061 10.1016/j.jss.2013.02.061 – reference: Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. IEEE Transactions on Software Engineering, 39, 2 (2013), Feb., 276–291. issn:1939-3520 https://doi.org/10.1109/TSE.2012.14 10.1109/TSE.2012.14 – reference: Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 152–158. – reference: Marllos Paiva Prado and Auri Marcelo Rizzo Vincenzi. 2018. Towards cognitive support for unit testing: A qualitative study with practitioners. Journal of Systems and Software, 141 (2018), 66–84. issn:0164-1212 https://doi.org/10.1016/j.jss.2018.03.052 10.1016/j.jss.2018.03.052 – reference: Gordon Fraser, Matt Staats, Phil McMinn, Andrea Arcuri, and Frank Padberg. 2015. Does automated unit test generation really help software testers? a controlled empirical study. ACM Transactions on Software Engineering and Methodology (TOSEM), 24, 4 (2015), 1–49. – reference: Pankaj Jalote. 2008. A concise introduction to software engineering. Springer Science & Business Media. – reference: John Wrenn, Tim Nelson, and Shriram Krishnamurthi. 2021. Using Relational Problems to Teach Property-Based Testing. The art science and engineering of programming, 5, 2 (2021). – reference: Harrison Goldstein, Joseph W. Cutler, Daniel Dickstein, Benjamin C. Pierce, and Andrew Head. 2024. Property-Based Testing in Practice. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machinery, New York, NY, USA. 1–13. isbn:9798400702174 https://doi.org/10.1145/3597503.3639581 10.1145/3597503.3639581 – reference: Tristan Teufel and contributors. 2022. Jest Runner. https://github.com/firsttris/vscode-jest-runner [Online; accessed 2022-11-10] – reference: Nicolas Dubien. 2024. fast-check. https://www.npmjs.com/package/fast-check – ident: e_1_2_1_1_1 doi: 10.1145/3183440.3195001 – ident: e_1_2_1_55_1 doi: 10.1109/ISSRE.1997.630851 – ident: e_1_2_1_26_1 doi: 10.1145/3597503.3639581 – ident: e_1_2_1_4_1 doi: 10.1007/s10664-021-10072-8 – ident: e_1_2_1_51_1 doi: 10.1109/ASE56229.2023.00193 – volume-title: A concise introduction to software engineering ident: e_1_2_1_33_1 – ident: e_1_2_1_23_1 doi: 10.1109/TSE.2012.14 – ident: e_1_2_1_13_1 doi: 10.1145/3587157 – ident: e_1_2_1_19_1 – ident: e_1_2_1_15_1 doi: 10.1145/3580446 – volume-title: https://github.com/firsttris/vscode-jest-runner [Online year: 2022 ident: e_1_2_1_63_1 – ident: e_1_2_1_38_1 doi: 10.21105/joss.01891 – ident: e_1_2_1_18_1 – ident: e_1_2_1_35_1 doi: 10.1109/QSIC.2014.33 – ident: e_1_2_1_6_1 doi: 10.1109/TSE.2017.2776152 – ident: e_1_2_1_12_1 – ident: e_1_2_1_2_1 doi: 10.1016/j.jss.2013.02.061 – ident: e_1_2_1_49_1 doi: 10.1145/2884781.2884847 – ident: e_1_2_1_20_1 doi: 10.1109/CHASE52884.2021.00026 – ident: e_1_2_1_30_1 doi: 10.1109/TSE.2013.59 – ident: e_1_2_1_42_1 doi: 10.1145/174800.174808 – volume-title: Using Relational Problems to Teach Property-Based Testing. The art science and engineering of programming, 5, 2 year: 2021 ident: e_1_2_1_66_1 – volume-title: Software testing and quality assurance: theory and practice ident: e_1_2_1_64_1 – ident: e_1_2_1_43_1 doi: 10.1109/MC.2016.200 – ident: e_1_2_1_37_1 doi: 10.1145/3597503.3608128 – volume-title: Proceedings.. 116–125 year: 2004 ident: e_1_2_1_45_1 – volume-title: american fuzzy lop. https://lcamtuf.coredump.cx/afl/ [Online year: 2023 ident: e_1_2_1_67_1 – ident: e_1_2_1_53_1 doi: 10.1145/2771783.2771801 – ident: e_1_2_1_34_1 doi: 10.1007/s10664-013-9279-3 – ident: e_1_2_1_3_1 doi: 10.1007/s10664-017-9570-9 – ident: e_1_2_1_14_1 doi: 10.1145/3611643.3616327 – volume-title: Overview of Microsoft IntelliTester. https://learn.microsoft.com/en-us/visualstudio/test/intellitest-manual/ [Online year: 2023 ident: e_1_2_1_41_1 – volume-title: 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC). 348–3483 year: 2018 ident: e_1_2_1_29_1 – ident: e_1_2_1_56_1 doi: 10.1109/ICSE.1998.671118 – ident: e_1_2_1_48_1 doi: 10.1109/ICSE.2007.37 – ident: e_1_2_1_50_1 doi: 10.1016/j.jss.2018.03.052 – ident: e_1_2_1_10_1 – ident: e_1_2_1_11_1 doi: 10.1145/351240.351266 – volume-title: Jest - Delightful Javascript Testing. https://jestjs.io/ [Online year: 2024 ident: e_1_2_1_62_1 – ident: e_1_2_1_32_1 – ident: e_1_2_1_40_1 doi: 10.2307/2288652 – volume-title: NaNofuzz - Visual Studio Marketplace. https://marketplace.visualstudio.com/items?itemName=penrose.nanofuzz [Online year: 2024 ident: e_1_2_1_16_1 – ident: e_1_2_1_52_1 doi: 10.1176/appi.ajp.2012.12070999 – ident: e_1_2_1_60_1 doi: 10.1109/ICICoS56336.2022.9930600 – volume-title: Using thematic analysis in psychology. Qualitative research in psychology, 3, 2 year: 2006 ident: e_1_2_1_8_1 – ident: e_1_2_1_39_1 doi: 10.1109/TSE.2019.2946563 – ident: e_1_2_1_31_1 doi: 10.1080/00031305.1998.10480559 – volume-title: Rethinking productivity in software engineering ident: e_1_2_1_59_1 – ident: e_1_2_1_5_1 doi: 10.1109/TSE.2014.2372785 – volume-title: https://github.com/features/codespaces [Online year: 2023 ident: e_1_2_1_25_1 – ident: e_1_2_1_65_1 doi: 10.48550/arXiv.2307.04346 – ident: e_1_2_1_46_1 doi: 10.1145/191666.191729 – volume-title: Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium. 230–239 year: 2000 ident: e_1_2_1_57_1 – year: 2021 ident: e_1_2_1_44_1 article-title: Generating unit tests for documentation publication-title: IEEE Transactions on Software Engineering. – volume-title: Proc. Workshop on the Human Aspects of Types and Reasoning Assistants (HATRA). year: 2022 ident: e_1_2_1_27_1 – ident: e_1_2_1_21_1 doi: 10.1145/581339.581359 – volume-title: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 287–298 year: 2020 ident: e_1_2_1_58_1 – ident: e_1_2_1_17_1 – ident: e_1_2_1_22_1 doi: 10.1145/1131421.1131423 – volume-title: USENIX Security Symposium. 2777–2794 year: 2021 ident: e_1_2_1_36_1 – ident: e_1_2_1_7_1 doi: 10.1145/3477132.3483540 – ident: e_1_2_1_24_1 doi: 10.1145/2699688 – volume-title: Concise Guide to Software Testing ident: e_1_2_1_47_1 – volume-title: SUS-A quick and dirty usability scale. Usability evaluation in industry, 189, 194 year: 1996 ident: e_1_2_1_9_1 – volume-title: What if writing tests was a joyful experience? https://blog.janestreet.com/the-joy-of-expect-tests/ [Online year: 2023 ident: e_1_2_1_61_1 – ident: e_1_2_1_54_1 doi: 10.1093/acprof:oso/9780195385540.001.0001 – ident: e_1_2_1_28_1 |
| SSID | ssj0002991170 |
| Score | 2.2946146 |
| Snippet | Software testing is difficult, tedious, and may consume 28%–50% of software engineering labor. Automatic test generators aim to ease this burden but have... |
| SourceID | crossref acm |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 1983 |
| SubjectTerms | Human-centered computing Software and its engineering Software testing and debugging User studies |
| SubjectTermsDisplay | Human-centered computing -- User studies Software and its engineering -- Software testing and debugging |
| Title | TerzoN: Human-in-the-Loop Software Testing with a Composite Oracle |
| URI | https://dl.acm.org/doi/10.1145/3729359 |
| Volume | 2 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2994-970X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0002991170 issn: 2994-970X databaseCode: M~E dateStart: 20240101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Bb9MwFLbK4MCFwQBRNpAPiEtkWOIkrncr1SYOrExqD71VTvIsJm1p1a5j2mG_gZ_Me4mdhDIJOHCJqsi2Ir-v7z0_v_c9xt7pzCqplBUhgBExmlRhwkSJVKYaIDbS2rhqNqHG48Fsps96vR--Fub6QpXl4OZGL_-rqPEdCptKZ_9B3M2i-AJ_o9DxiWLH598JHla3izGd9KsAvTgvBTp54stisQwmqHS_U67XlMg1fBjWVFqBsrcg-Loyfj3nsp41Jm7tEwqGo1O6ZFj71aAlNWwj3469wDUUD0Yf2kugKoNg2NZS-M4TAd3sB8Nm5GRDyWSlC_Ovv21MN0gRJZRM5VRhpcsiYiDW6nBWm5173jllHHUwdzI57qjWUNcdb5yZpmjY_SYgJrYMuo2Ujmv8F5LtLePXpCTWBdrJ3E18wB5GKtGkJ0_v2qgdfjO16qGOhf7r60psmvvRzSVHJ7_sODodj2X6lD1xRw0-rCHyjPWg3GO7vo0Hd1r9OftUI-aI_4YX7vHCHV444YUb3uCF13h5waYnx9PRZ-FaawiDB2wxAEhNFoHRuU2tNZnMkzhPslybQoG2qVQGnRZtoyIt6NBtTEHFtqENUyIklC_ZTrko4RXj-M_OoDi0lFIQJ0qb2KLthSgMbaRAyj7bw82YL2vuFL-9ffbeb848d2T01BPlYr4lhz7jzUC_xtaQ138ess8et6g8YDtXqw28YY_y66vz9eptJeKfwwV0uw |
| linkProvider | ISSN International Centre |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=TerzoN%3A+Human-in-the-Loop+Software+Testing+with+a+Composite+Oracle&rft.jtitle=Proceedings+of+the+ACM+on+software+engineering&rft.au=Davis%2C+Matthew+C.&rft.au=Wei%2C+Amy&rft.au=Myers%2C+Brad+A.&rft.au=Sunshine%2C+Joshua&rft.date=2025-06-19&rft.issn=2994-970X&rft.eissn=2994-970X&rft.volume=2&rft.issue=FSE&rft.spage=1983&rft.epage=2005&rft_id=info:doi/10.1145%2F3729359&rft.externalDBID=n%2Fa&rft.externalDocID=10_1145_3729359 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2994-970X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2994-970X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2994-970X&client=summon |