How to judge testing progress

It is usual to base the assessment of software testing progress on a coverage measure such as code coverage or specification coverage, or on the percentage of the input domain exercised. In this paper it is argued that these characteristics do not provide good indications of the degree to which the...

Full description

Saved in:
Bibliographic Details
Published in:Information and software technology Vol. 46; no. 5; pp. 323 - 328
Main Author: Weyuker, Elaine J.
Format: Journal Article
Language:English
Published: Amsterdam Elsevier B.V 15.04.2004
Elsevier Science Ltd
Subjects:
Computer programming
Distributed processing
Operational distribution
Software
Software testing
Studies
Test data adequacy
Test progress
Testing
Software testing
Test data adequacy
Test progress
Operational distribution
ISSN:0950-5849, 1873-6025
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract It is usual to base the assessment of software testing progress on a coverage measure such as code coverage or specification coverage, or on the percentage of the input domain exercised. In this paper it is argued that these characteristics do not provide good indications of the degree to which the software has been tested. Instead we propose that the assessment of testing progress be based on the total percentage of the probability mass that corresponds to the test cases selected and run. To do this, it is necessary to collect data that profiles how the software will be used once it is operational in the field. By so doing, we are able to accurately determine how much testing has been done, and whether it has met the standards of completeness for the product under consideration.
AbstractList It is usual to base the assessment of software testing progress on a coverage measure such as code coverage or specification coverage, or on the percentage of the input domain exercised. In this paper it is argued that these characteristics do not provide good indications of the degree to which the software has been tested. Instead we propose that the assessment of testing progress be based on the total percentage of the probability mass that corresponds to the test cases selected and run. To do this, it is necessary to collect data that profiles how the software will be used once it is operational in the field. By so doing, we are able to accurately determine how much testing has been done, and whether it has met the standards of completeness for the product under consideration. [PUBLICATION ABSTRACT]
It is usual to base the assessment of software testing progress on a coverage measure such as code coverage or specification coverage, or on the percentage of the input domain exercised. In this paper it is argued that these characteristics do not provide good indications of the degree to which the software has been tested. Instead we propose that the assessment of testing progress be based on the total percentage of the probability mass that corresponds to the test cases selected and run. To do this, it is necessary to collect data that profiles how the software will be used once it is operational in the field. By so doing, we are able to accurately determine how much testing has been done, and whether it has met the standards of completeness for the product under consideration.
Author Weyuker, Elaine J.
Author_xml – sequence: 1
  givenname: Elaine J.
  surname: Weyuker
  fullname: Weyuker, Elaine J.
  email: weyuker@research.att.com
  organization: AT & T Labs—Research, Room No C213, 180 Park Avenue, Florharn Park, NJ 07932, USA
BookMark eNqFkE1LAzEQhoNUsK3-A4XF-66TzSa78SBI8QsKXvQc9mNSstSkJqnivzdlPXnQ01zmed-ZZ0Fm1lkk5JxCQYGKq7EwVgenixKAFSALgOaIzGlTs1xAyWdkDpJDzptKnpBFCCMArYHBnFw8us8sumzcDxvMIoZo7CbbebfxGMIpOdbtNuDZz1yS1_u7l9Vjvn5-eFrdrvOeMRrzWsiKIwOhq04gR14KzoZGY1N2HVJeAYBkgpWDQFrrsm57rdt2YI2osKs7tiSXU24qft-nI9To9t6mSkWlqErKOKSl62mp9y4Ej1r1JrbROBt9a7aKgjrYUKOabKiDDQVSJRsJrn7BO2_eWv_1H3YzYZi-_zDoVegN2h4H47GPanDm74BvGQt77Q
CitedBy_id crossref_primary_10_1108_13287261211255310
crossref_primary_10_1016_j_tcs_2015_02_002
Cites_doi 10.1145/227607.227608
10.1109/52.199724
10.1109/32.888628
10.1145/151299.151326
10.1145/584369.584373
10.1109/52.536461
10.1109/32.464549
10.1002/(SICI)1097-024X(199606)26:6<613::AID-SPE23>3.0.CO;2-5
ContentType Journal Article
Copyright 2003 Elsevier B.V.
Copyright Elsevier Science Ltd. Apr 15, 2004
Copyright_xml – notice: 2003 Elsevier B.V.
– notice: Copyright Elsevier Science Ltd. Apr 15, 2004
DBID AAYXX
CITATION
7SC
8FD
JQ2
L7M
L~C
L~D
DOI 10.1016/j.infsof.2003.09.008
DatabaseName CrossRef
Computer and Information Systems Abstracts
Technology Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DatabaseTitle CrossRef
Computer and Information Systems Abstracts
Technology Research Database
Computer and Information Systems Abstracts – Academic
Advanced Technologies Database with Aerospace
ProQuest Computer Science Collection
Computer and Information Systems Abstracts Professional
DatabaseTitleList Computer and Information Systems Abstracts
DeliveryMethod fulltext_linktorsrc
Discipline Business
EISSN 1873-6025
EndPage 328
ExternalDocumentID 666736561
10_1016_j_infsof_2003_09_008
S0950584903002015
Genre Feature
GroupedDBID --K
--M
-~X
.DC
.~1
0R~
1B1
1~.
1~5
29I
4.4
457
4G.
5GY
5VS
7-5
71M
77K
8P~
9JN
AABNK
AACTN
AAEDT
AAEDW
AAIAV
AAIKJ
AAKOC
AALRI
AAOAW
AAQFI
AAQXK
AAXUO
AAYFN
AAYOK
ABBOA
ABFNM
ABFRF
ABJNI
ABMAC
ABTAH
ABXDB
ABYKQ
ACDAQ
ACGFO
ACGFS
ACGOD
ACNNM
ACRLP
ACZNC
ADBBV
ADEZE
ADJOM
ADMUD
AEBSH
AEFWE
AEKER
AENEX
AFKWA
AFTJW
AGHFR
AGUBO
AGYEJ
AHHHB
AHZHX
AIALX
AIEXJ
AIKHN
AITUG
AJBFU
AJOXV
ALMA_UNASSIGNED_HOLDINGS
AMFUW
AMRAJ
AOUOD
ASPBG
AVWKF
AXJTR
AZFZN
BKOJK
BKOMP
BLXMC
CS3
DU5
EBS
EFJIC
EFLBG
EJD
EO8
EO9
EP2
EP3
FDB
FEDTE
FGOYB
FIRID
FNPLU
FYGXN
G-Q
G8K
GBLVA
GBOLZ
HLZ
HVGLF
HZ~
IHE
J1W
KOM
LG9
M41
MO0
MS~
N9A
O-L
O9-
OAUVE
OZT
P-8
P-9
P2P
PC.
PQQKQ
Q38
R2-
RIG
ROL
RPZ
SBC
SDF
SDG
SDP
SES
SEW
SPC
SPCBC
SSV
SSZ
T5K
TWZ
UHS
UNMZH
WH7
WUQ
XFK
ZY4
~G-
77I
9DU
AATTM
AAXKI
AAYWO
AAYXX
ABDPE
ABWVN
ACLOT
ACRPL
ACVFH
ADCNI
ADNMO
AEIPS
AEUPX
AFJKZ
AFPUW
AGQPQ
AIGII
AIIUN
AKBMS
AKRWK
AKYEP
ANKPU
APXCP
CITATION
EFKBS
~HD
7SC
8FD
AFXIZ
AGCQF
AGRNS
JQ2
L7M
L~C
L~D
SSH
ID FETCH-LOGICAL-c331t-76945e306f4b6e5e52653d8fe82bbe15400093632d6e17f27acffaad3864eb7b3
ISICitedReferencesCount 2
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000189229500008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 0950-5849
IngestDate Fri Jul 25 07:05:47 EDT 2025
Sat Nov 29 03:40:31 EST 2025
Tue Nov 18 22:19:50 EST 2025
Fri Feb 23 02:23:55 EST 2024
IsPeerReviewed true
IsScholarly true
Issue 5
Keywords Software testing
Test data adequacy
Test progress
Operational distribution
Language English
License https://www.elsevier.com/tdm/userlicense/1.0
LinkModel OpenURL
MergedId FETCHMERGED-LOGICAL-c331t-76945e306f4b6e5e52653d8fe82bbe15400093632d6e17f27acffaad3864eb7b3
Notes SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
PQID 196421350
PQPubID 41979
PageCount 6
ParticipantIDs proquest_journals_196421350
crossref_citationtrail_10_1016_j_infsof_2003_09_008
crossref_primary_10_1016_j_infsof_2003_09_008
elsevier_sciencedirect_doi_10_1016_j_infsof_2003_09_008
PublicationCentury 2000
PublicationDate 2004-04-15
PublicationDateYYYYMMDD 2004-04-15
PublicationDate_xml – month: 04
  year: 2004
  text: 2004-04-15
  day: 15
PublicationDecade 2000
PublicationPlace Amsterdam
PublicationPlace_xml – name: Amsterdam
PublicationTitle Information and software technology
PublicationYear 2004
Publisher Elsevier B.V
Elsevier Science Ltd
Publisher_xml – name: Elsevier B.V
– name: Elsevier Science Ltd
References Weyuker, Vokolos (BIB9) 2000; 26
Weyuker (BIB6) 1996; 5
Avritzer, Weyuker (BIB2) 1995; September
Avritzer, Kondek, Liu, Weyuker (BIB4) 2002
Whittaker, Poore (BIB10) 1993; 2
Avritzer, Ros, Weyuker (BIB1) 1996; September
Musa (BIB5) 1993; 10
Weyuker, Vokolos (BIB8) 1998
Weyuker, Avritzer (BIB7) 2002; June
Avritzer, Weyuker (BIB3) 1996; 26
Avritzer (10.1016/j.infsof.2003.09.008_BIB4) 2002
Weyuker (10.1016/j.infsof.2003.09.008_BIB9) 2000; 26
Avritzer (10.1016/j.infsof.2003.09.008_BIB1) 1996; September
Avritzer (10.1016/j.infsof.2003.09.008_BIB2) 1995; September
Avritzer (10.1016/j.infsof.2003.09.008_BIB3) 1996; 26
Whittaker (10.1016/j.infsof.2003.09.008_BIB10) 1993; 2
Musa (10.1016/j.infsof.2003.09.008_BIB5) 1993; 10
Weyuker (10.1016/j.infsof.2003.09.008_BIB6) 1996; 5
Weyuker (10.1016/j.infsof.2003.09.008_BIB7) 2002; June
Weyuker (10.1016/j.infsof.2003.09.008_BIB8) 1998
References_xml – start-page: 80
  year: 1998
  end-page: 87
  ident: BIB8
  article-title: Performance testing of software systems
  publication-title: Proceedings of ACM Workshop on Software and Performance (WOSP-98)
– volume: September
  start-page: 705
  year: 1995
  end-page: 716
  ident: BIB2
  article-title: The automatic generation of load test suites and the assessment of the resulting software
  publication-title: IEEE Transactions of Software Engineering
– volume: June
  start-page: 152
  year: 2002
  end-page: 158
  ident: BIB7
  article-title: A metric to predict software scalability
  publication-title: Proceedings of IEEE/Eighth International Symposium on Software Metrics (Metrics 2002) Ottawa, Canada
– volume: September
  start-page: 76
  year: 1996
  end-page: 82
  ident: BIB1
  article-title: Reliability testing of rule-based systems
  publication-title: IEEE Software
– year: 2002
  ident: BIB4
  article-title: Software performance testing based on workload characterization
  publication-title: Proceedings of ACM/Third International Workshop on Software and Performance (WOSP 2002) Rome, Italy, July
– volume: 10
  start-page: 14
  year: 1993
  end-page: 32
  ident: BIB5
  article-title: Operational profiles in software reliability engineering
  publication-title: IEEE Software
– volume: 26
  start-page: 1147
  year: 2000
  end-page: 1156
  ident: BIB9
  article-title: Experience with performance testing of software systems-issues, an approach, and case study
  publication-title: IEEE Transactions on Software Engineering
– volume: 5
  start-page: 87
  year: 1996
  end-page: 98
  ident: BIB6
  article-title: Using failure cost information for testing and reliability assessment
  publication-title: ACM Transactions of Software Engineering and Methodology
– volume: 2
  start-page: 93
  year: 1993
  end-page: 106
  ident: BIB10
  article-title: Markov analysis of software specifications
  publication-title: ACM Transactions on Software Engineering and Methodology
– volume: 26
  year: 1996
  ident: BIB3
  article-title: Deriving workloads for performance testing
  publication-title: Software Practice and Experience
– volume: 5
  start-page: 87
  issue: 2
  year: 1996
  ident: 10.1016/j.infsof.2003.09.008_BIB6
  article-title: Using failure cost information for testing and reliability assessment
  publication-title: ACM Transactions of Software Engineering and Methodology
  doi: 10.1145/227607.227608
– volume: 10
  start-page: 14
  issue: 2
  year: 1993
  ident: 10.1016/j.infsof.2003.09.008_BIB5
  article-title: Operational profiles in software reliability engineering
  publication-title: IEEE Software
  doi: 10.1109/52.199724
– volume: 26
  start-page: 1147
  issue: 12
  year: 2000
  ident: 10.1016/j.infsof.2003.09.008_BIB9
  article-title: Experience with performance testing of software systems-issues, an approach, and case study
  publication-title: IEEE Transactions on Software Engineering
  doi: 10.1109/32.888628
– volume: 2
  start-page: 93
  issue: 1
  year: 1993
  ident: 10.1016/j.infsof.2003.09.008_BIB10
  article-title: Markov analysis of software specifications
  publication-title: ACM Transactions on Software Engineering and Methodology
  doi: 10.1145/151299.151326
– year: 2002
  ident: 10.1016/j.infsof.2003.09.008_BIB4
  article-title: Software performance testing based on workload characterization
  publication-title: Proceedings of ACM/Third International Workshop on Software and Performance (WOSP 2002) Rome, Italy, July
  doi: 10.1145/584369.584373
– volume: September
  start-page: 76
  year: 1996
  ident: 10.1016/j.infsof.2003.09.008_BIB1
  article-title: Reliability testing of rule-based systems
  publication-title: IEEE Software
  doi: 10.1109/52.536461
– volume: September
  start-page: 705
  year: 1995
  ident: 10.1016/j.infsof.2003.09.008_BIB2
  article-title: The automatic generation of load test suites and the assessment of the resulting software
  publication-title: IEEE Transactions of Software Engineering
  doi: 10.1109/32.464549
– volume: 26
  issue: 6
  year: 1996
  ident: 10.1016/j.infsof.2003.09.008_BIB3
  article-title: Deriving workloads for performance testing
  publication-title: Software Practice and Experience
  doi: 10.1002/(SICI)1097-024X(199606)26:6<613::AID-SPE23>3.0.CO;2-5
– start-page: 80
  year: 1998
  ident: 10.1016/j.infsof.2003.09.008_BIB8
  article-title: Performance testing of software systems
  publication-title: Proceedings of ACM Workshop on Software and Performance (WOSP-98)
– volume: June
  start-page: 152
  year: 2002
  ident: 10.1016/j.infsof.2003.09.008_BIB7
  article-title: A metric to predict software scalability
  publication-title: Proceedings of IEEE/Eighth International Symposium on Software Metrics (Metrics 2002) Ottawa, Canada
SSID ssj0017030
Score 1.6980842
Snippet It is usual to base the assessment of software testing progress on a coverage measure such as code coverage or specification coverage, or on the percentage of...
SourceID proquest
crossref
elsevier
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 323
SubjectTerms Computer programming
Distributed processing
Operational distribution
Software
Software testing
Studies
Test data adequacy
Test progress
Testing
Title How to judge testing progress
URI https://dx.doi.org/10.1016/j.infsof.2003.09.008
https://www.proquest.com/docview/196421350
Volume 46
WOSCitedRecordID wos000189229500008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVESC
  databaseName: Elsevier SD Freedom Collection Journals 2021
  customDbUrl:
  eissn: 1873-6025
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0017030
  issn: 0950-5849
  databaseCode: AIEXJ
  dateStart: 19950101
  isFulltext: true
  titleUrlDefault: https://www.sciencedirect.com
  providerName: Elsevier
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Lb9NAEF5BixAXxFO0pcgHxKVayfHau_axqlKVKgocUpTbyvswalQ5aeNA--87sw-HNoLCgYsV2XGU3c-e_XZmvhlCPtZGA2s3nFZNKmheMw120CqqFFavhA1D6cIF30ZiPC6n0-praMa5dO0ERNuW19fV4r9CDecAbJTO_gPc_Y_CCfgMoMMRYIfjXwHvmsTND2YrFGJ1WETD6c3n3_tki1nMXu-Viwfefd50PzERrNt0t9ubVUi_GF7USExP77gLXJaJF0x6H1bUsUTTsU4e8k7BlAIl8UbMepNYCkZ56uXJ0WYGt-H5r1FpZwCZVw9vGGbvI5jhbgJG4-qwuvqyableiGLwffxFHp-NRnIynE7uXnXrLsdOpUBDB58WlxTbh2GYPfRSeUy2M1FUYN62Dz8Pp6d9QAkNmy-76AcYVZQu1W_zT_2Opdxbrx0Jmbwgz8PuITn0qL8kj2z7ijyN4oXXZB_AT7p54sBPAvhJBP8NOTseTo5OaGiAQTVjg44KXuWFhU1dkytuC4utDJgpG1tmSlkgv84hxVlmuB2IJhO1bpq6NqzkuVVCsbdkq5239h1JjDY5OiBQEpTXwAt1xhSHJcYUqtGm2iEsDljqUB0em5RcyJgGOJN-mrBxKZNpJWGadgjt71r46igPfF_EuZSB4XnmJuE5eeDOvTj1MrxsS4nF5LIBK9LdP17dI8_Wb8J7stVdrew-eaJ_dOfLqw_hSbkFnJd3FQ
linkProvider Elsevier
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+to+judge+testing+progress&rft.jtitle=Information+and+software+technology&rft.au=Weyuker%2C+Elaine+J&rft.date=2004-04-15&rft.pub=Elsevier+Science+Ltd&rft.issn=0950-5849&rft.eissn=1873-6025&rft.volume=46&rft.issue=5&rft.spage=323&rft_id=info:doi/10.1016%2Fj.infsof.2003.09.008&rft.externalDBID=NO_FULL_TEXT&rft.externalDocID=666736561
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0950-5849&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0950-5849&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0950-5849&client=summon