View in EDS

Preempting flaky tests via non-idempotent-outcome tests.

Saved in:
Bibliographic Details
Title: Preempting flaky tests via non-idempotent-outcome tests.
Authors: Wei, Anjiang, Yi, Pu, Li, Zhengxi, Xie, Tao, Marinov, Darko, Lam, Wing
Source: ICSE: International Conference on Software Engineering; 2022, p1730-1742, 13p
Subject Terms: REGRESSION analysis, COMPUTER software, JAVA programming language, PYTHON programming language, RESEARCH
Abstract: Regression testing can greatly help in software development, but it can be seriously undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on the same code commit. Flaky tests are an emerging topic in both research and industry. Prior work has identified multiple categories of flaky tests, developed techniques for detecting these flaky tests, and analyzed some detected flaky tests. To proactively detect, i.e., preempt, flaky tests, we propose to detect non-idempotent-outcome (NIO) tests, a novel category related to flaky tests. In particular, we run each test twice in the same test execution environment, e.g., run each Java test twice in the same Java Virtual Machine. A test is NIO if it passes in the first run but fails in the second. Each NIO test has side effects and "self-pollutes" the state shared among test runs. We perform experiments on both Java and Python open-source projects, detecting 223 NIO Java tests and 138 NIO Python tests. We have inspected all 361 detected tests and opened pull requests that fix 268 tests, with 192 already accepted, only 6 rejected, and the remaining 70 pending. [ABSTRACT FROM AUTHOR]
Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Complementary Index
FullText Text:
  Availability: 0
Header DbId: edb
DbLabel: Complementary Index
An: 185195821
RelevancyScore: 941
AccessLevel: 6
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 940.765075683594
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Preempting flaky tests via non-idempotent-outcome tests.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wei%2C+Anjiang%22">Wei, Anjiang</searchLink><br /><searchLink fieldCode="AR" term="%22Yi%2C+Pu%22">Yi, Pu</searchLink><br /><searchLink fieldCode="AR" term="%22Li%2C+Zhengxi%22">Li, Zhengxi</searchLink><br /><searchLink fieldCode="AR" term="%22Xie%2C+Tao%22">Xie, Tao</searchLink><br /><searchLink fieldCode="AR" term="%22Marinov%2C+Darko%22">Marinov, Darko</searchLink><br /><searchLink fieldCode="AR" term="%22Lam%2C+Wing%22">Lam, Wing</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: ICSE: International Conference on Software Engineering; 2022, p1730-1742, 13p
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22REGRESSION+analysis%22">REGRESSION analysis</searchLink><br /><searchLink fieldCode="DE" term="%22COMPUTER+software%22">COMPUTER software</searchLink><br /><searchLink fieldCode="DE" term="%22JAVA+programming+language%22">JAVA programming language</searchLink><br /><searchLink fieldCode="DE" term="%22PYTHON+programming+language%22">PYTHON programming language</searchLink><br /><searchLink fieldCode="DE" term="%22RESEARCH%22">RESEARCH</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Regression testing can greatly help in software development, but it can be seriously undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on the same code commit. Flaky tests are an emerging topic in both research and industry. Prior work has identified multiple categories of flaky tests, developed techniques for detecting these flaky tests, and analyzed some detected flaky tests. To proactively detect, i.e., preempt, flaky tests, we propose to detect non-idempotent-outcome (NIO) tests, a novel category related to flaky tests. In particular, we run each test twice in the same test execution environment, e.g., run each Java test twice in the same Java Virtual Machine. A test is NIO if it passes in the first run but fails in the second. Each NIO test has side effects and "self-pollutes" the state shared among test runs. We perform experiments on both Java and Python open-source projects, detecting 223 NIO Java tests and 138 NIO Python tests. We have inspected all 361 detected tests and opened pull requests that fix 268 tests, with 192 already accepted, only 6 rejected, and the remaining 70 pending. [ABSTRACT FROM AUTHOR]
– Name: Abstract
  Label:
  Group: Ab
  Data: <i>Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://erproxy.cvtisr.sk/sfx/access?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edb&AN=185195821
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1145/3510003.3510170
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 1730
    Subjects:
      – SubjectFull: REGRESSION analysis
        Type: general
      – SubjectFull: COMPUTER software
        Type: general
      – SubjectFull: JAVA programming language
        Type: general
      – SubjectFull: PYTHON programming language
        Type: general
      – SubjectFull: RESEARCH
        Type: general
    Titles:
      – TitleFull: Preempting flaky tests via non-idempotent-outcome tests.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wei, Anjiang
      – PersonEntity:
          Name:
            NameFull: Yi, Pu
      – PersonEntity:
          Name:
            NameFull: Li, Zhengxi
      – PersonEntity:
          Name:
            NameFull: Xie, Tao
      – PersonEntity:
          Name:
            NameFull: Marinov, Darko
      – PersonEntity:
          Name:
            NameFull: Lam, Wing
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Text: 2022
              Type: published
              Y: 2022
          Titles:
            – TitleFull: ICSE: International Conference on Software Engineering
              Type: main
ResultId 1