On negative results when using sentiment analysis tools for software engineering research

Recent years have seen an increasing attention to social aspects of software engineering, including studies of emotions and sentiments experienced and expressed by the software developers. Most of these studies reuse existing sentiment analysis tools such as SentiStrength and NLTK . However, these t...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	Empirical software engineering : an international journal Jg. 22; H. 5; S. 2543 - 2584
Hauptverfasser:	Jongeling, Robbert, Sarkar, Proshanta, Datta, Subhajit, Serebrenik, Alexander
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	New York Springer US 01.10.2017 Springer Nature B.V
Schlagworte:	Compilers Computer Science Contrarian investing Data mining Engineering research Interpreters Product reviews Programming Languages Sentiment analysis Social factors Software Software engineering Software Engineering/Programming and Operating Systems Studies Negative results Replication study Sentiment analysis tools
ISSN:	1382-3256, 1573-7616
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	Recent years have seen an increasing attention to social aspects of software engineering, including studies of emotions and sentiments experienced and expressed by the software developers. Most of these studies reuse existing sentiment analysis tools such as SentiStrength and NLTK . However, these tools have been trained on product reviews and movie reviews and, therefore, their results might not be applicable in the software engineering domain. In this paper we study whether the sentiment analysis tools agree with the sentiment recognized by human evaluators (as reported in an earlier study) as well as with each other. Furthermore, we evaluate the impact of the choice of a sentiment analysis tool on software engineering studies by conducting a simple study of differences in issue resolution times for positive, negative and neutral texts. We repeat the study for seven datasets (issue trackers and Stack Overflow questions) and different sentiment analysis tools and observe that the disagreement between the tools can lead to diverging conclusions. Finally, we perform two replications of previously published studies and observe that the results of those studies cannot be confirmed when a different sentiment analysis tool is used.
AbstractList	Recent years have seen an increasing attention to social aspects of software engineering, including studies of emotions and sentiments experienced and expressed by the software developers. Most of these studies reuse existing sentiment analysis tools such as SentiStrength and NLTK. However, these tools have been trained on product reviews and movie reviews and, therefore, their results might not be applicable in the software engineering domain. In this paper we study whether the sentiment analysis tools agree with the sentiment recognized by human evaluators (as reported in an earlier study) as well as with each other. Furthermore, we evaluate the impact of the choice of a sentiment analysis tool on software engineering studies by conducting a simple study of differences in issue resolution times for positive, negative and neutral texts. We repeat the study for seven datasets (issue trackers and Stack Overflow questions) and different sentiment analysis tools and observe that the disagreement between the tools can lead to diverging conclusions. Finally, we perform two replications of previously published studies and observe that the results of those studies cannot be confirmed when a different sentiment analysis tool is used. Recent years have seen an increasing attention to social aspects of software engineering, including studies of emotions and sentiments experienced and expressed by the software developers. Most of these studies reuse existing sentiment analysis tools such as SentiStrength and NLTK . However, these tools have been trained on product reviews and movie reviews and, therefore, their results might not be applicable in the software engineering domain. In this paper we study whether the sentiment analysis tools agree with the sentiment recognized by human evaluators (as reported in an earlier study) as well as with each other. Furthermore, we evaluate the impact of the choice of a sentiment analysis tool on software engineering studies by conducting a simple study of differences in issue resolution times for positive, negative and neutral texts. We repeat the study for seven datasets (issue trackers and Stack Overflow questions) and different sentiment analysis tools and observe that the disagreement between the tools can lead to diverging conclusions. Finally, we perform two replications of previously published studies and observe that the results of those studies cannot be confirmed when a different sentiment analysis tool is used.
Author	Jongeling, Robbert Datta, Subhajit Serebrenik, Alexander Sarkar, Proshanta
Author_xml	– sequence: 1 givenname: Robbert surname: Jongeling fullname: Jongeling, Robbert organization: Eindhoven University of Technology – sequence: 2 givenname: Proshanta surname: Sarkar fullname: Sarkar, Proshanta organization: IBM India Private Limited – sequence: 3 givenname: Subhajit surname: Datta fullname: Datta, Subhajit organization: Singapore University of Technology and Design – sequence: 4 givenname: Alexander orcidid: 0000-0002-1418-0095 surname: Serebrenik fullname: Serebrenik, Alexander email: a.serebrenik@tue.nl organization: Eindhoven University of Technology
BookMark	eNp9kE1LAzEQhoNUsFZ_gLeA59Ukm012j1L8gkIvvXgK2d1Ju2Wb1Exq6793SwVB0MvMHN7nZXguycgHD4TccHbHGdP3yJlSMmNcZZWs8uxwRsa80HmmFVej4c5LkeWiUBfkEnHNGKu0LMbkbe6ph6VN3QfQCLjrE9L9CjzdYeeXFMGnbjMMar3tP7FDmkLokboQKQaX9jYCBb_sPEA8EkMJ2Nisrsi5sz3C9feekMXT42L6ks3mz6_Th1nWSKZTVua1lqLV1oqWCdcUSnLlSiFKKXlpQbVtVdRN3RayqZyrWl3VqnWVEKwuQeYTcnuq3cbwvgNMZh12cfgVjWAqV4xLlQ8pfko1MSBGcGYbu42Nn4YzcxRoTgLNINAcBZrDwOhfTNOlQVTwKdqu_5cUJxK3RyUQf376G_oCJJqJaw
CitedBy_id	crossref_primary_10_1109_ACCESS_2024_3488904 crossref_primary_10_1371_journal_pone_0246464 crossref_primary_10_1007_s10515_021_00283_0 crossref_primary_10_1145_3241743 crossref_primary_10_1007_s10664_018_9659_9 crossref_primary_10_3233_JIFS_211217 crossref_primary_10_1109_ACCESS_2021_3108624 crossref_primary_10_1007_s10664_019_09716_7 crossref_primary_10_1007_s10664_022_10242_2 crossref_primary_10_1109_MS_2020_2968557 crossref_primary_10_1109_ACCESS_2024_3439450 crossref_primary_10_1145_3583562 crossref_primary_10_1145_3479497 crossref_primary_10_1145_3424308 crossref_primary_10_1016_j_infsof_2022_107018 crossref_primary_10_3390_su132313356 crossref_primary_10_7759_s44389_024_02851_3 crossref_primary_10_1016_j_jss_2020_110740 crossref_primary_10_1145_3757739 crossref_primary_10_1109_TG_2021_3072545 crossref_primary_10_1016_j_ins_2018_12_038 crossref_primary_10_1016_j_jss_2025_112339 crossref_primary_10_1109_MS_2018_2874568 crossref_primary_10_1016_j_jss_2019_05_026 crossref_primary_10_1145_3491211 crossref_primary_10_1016_j_jss_2023_111804 crossref_primary_10_1016_j_infsof_2017_10_009 crossref_primary_10_1016_j_ribaf_2025_102875 crossref_primary_10_1007_s10664_025_10617_1 crossref_primary_10_1145_3610092 crossref_primary_10_1007_s10664_021_09977_1 crossref_primary_10_3390_su14052677 crossref_primary_10_1145_3359224 crossref_primary_10_1007_s10515_020_00274_7 crossref_primary_10_32604_cmc_2022_020431 crossref_primary_10_1007_s10664_024_10468_2 crossref_primary_10_3390_app12083709 crossref_primary_10_1007_s10515_021_00295_w crossref_primary_10_7717_peerj_cs_1498 crossref_primary_10_1016_j_jss_2020_110704 crossref_primary_10_1109_TSE_2018_2883603 crossref_primary_10_1016_j_infsof_2019_08_002 crossref_primary_10_1016_j_jimonfin_2024_103129 crossref_primary_10_3390_info15010008 crossref_primary_10_1080_19312458_2019_1671966 crossref_primary_10_1007_s10664_022_10183_w crossref_primary_10_1007_s10664_021_09960_w crossref_primary_10_1016_j_procs_2024_01_120 crossref_primary_10_1145_3697009 crossref_primary_10_3390_info14010011 crossref_primary_10_1109_MS_2019_2919573 crossref_primary_10_1109_MS_2019_2922949 crossref_primary_10_1007_s10270_020_00809_2 crossref_primary_10_1016_j_procs_2019_12_042 crossref_primary_10_3390_ijerph18115993 crossref_primary_10_1016_j_jss_2019_02_020 crossref_primary_10_1109_TSE_2019_2900245 crossref_primary_10_1093_joc_jqz023 crossref_primary_10_1016_j_infsof_2019_05_012 crossref_primary_10_1049_iet_sen_2019_0054 crossref_primary_10_1109_TSE_2021_3098242 crossref_primary_10_1145_3491039 crossref_primary_10_1016_j_jss_2018_11_016 crossref_primary_10_1145_3490388 crossref_primary_10_1007_s10664_019_09743_4 crossref_primary_10_1109_ACCESS_2019_2909746 crossref_primary_10_1007_s10462_023_10555_8 crossref_primary_10_1007_s42979_023_02595_2
Cites_doi	10.1109/SERA.2016.7516145 10.1109/MS.2012.169 10.2307/3001968 10.1214/aoms/1177697819 10.1080/01621459.1971.10482356 10.1109/ICSME.2014.51 10.1109/ICSTW.2011.12 10.1145/1964897.1964899 10.1145/2124436.2124453 10.1109/ICSE.2013.6606588 10.1109/QUATIC.2012.36 10.1145/2630088.2652480 10.1016/j.infsof.2015.05.003 10.1007/978-3-642-33266-1_58 10.1109/CHASE.2015.32 10.1016/j.infsof.2016.01.004 10.1145/1656274.1656278 10.1109/ICSM.2015.7332474 10.1145/1985374.1985381 10.2466/pms.1992.74.3.835 10.1145/1868328.1868336 10.1109/MSR.2012.6224307 10.7717/peerj-cs.73 10.1109/MSR.2015.35 10.1109/ICSM.2015.7332508 10.1109/JCSSE.2013.6567355 10.1109/MSR.2015.8 10.1109/MSR.2012.6224268 10.1037/h0026256 10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U 10.1145/1978942.1979409 10.18653/v1/D13-1170 10.1109/ICSE.2015.170 10.1080/01621459.1961.10482090 10.1002/0471445428 10.1214/12-EJS691 10.1109/SocialCom.2013.35 10.1109/MSR.2009.5069486 10.1145/2597073.2597086 10.1145/2145204.2145401 10.1109/CGC.2013.71 10.1109/WAINA.2009.190 10.1007/10720076_29 10.1145/2597073.2597117 10.1002/asi.21416 10.5329/RESI.2014.1302006 10.1007/11552253_12 10.1145/42005.42007 10.1145/2465478.2465482 10.1145/2384616.2384664 10.1145/69605.357970 10.1109/HICSS.2012.421 10.3115/1118693.1118704 10.1145/1460563.1460654 10.1002/asi.21662 10.1145/2597073.2597118 10.3115/1220575.1220619 10.1007/s10664-013-9244-1 10.1007/978-3-319-18612-2_11 10.1145/2491411.2494578 10.1109/MSR.2012.6224267 10.1109/MSR.2013.6624052 10.1007/978-3-642-04277-5_18 10.1017/CBO9780511527685 10.1145/2601248.2601289 10.1145/2124295.2124371 10.1609/icwsm.v4i1.14009 10.1145/2804381.2804387 10.1145/2810146.2810147 10.1007/s10664-008-9060-1 10.1109/SCC.2014.80 10.1007/BF01908075 10.1007/s10664-007-9037-5
ContentType	Journal Article
Copyright	The Author(s) 2017 Copyright Springer Science & Business Media 2017
Copyright_xml	– notice: The Author(s) 2017 – notice: Copyright Springer Science & Business Media 2017
DBID	C6C AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D
DOI	10.1007/s10664-016-9493-x
DatabaseName	Springer Nature OA Free Journals CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional
DatabaseTitleList	Computer and Information Systems Abstracts
DeliveryMethod	fulltext_linktorsrc
Discipline	Computer Science
EISSN	1573-7616
EndPage	2584
ExternalDocumentID	10_1007_s10664_016_9493_x
GrantInformation_xml	– fundername: Eindhoven University of Technology
GroupedDBID	-4Z -59 -5G -BR -EM -Y2 -~C .86 .DC .VR 06D 0R~ 0VY 199 1N0 1SB 2.D 203 28- 29G 2J2 2JN 2JY 2KG 2LR 2P1 2VQ 2~H 30V 4.4 406 408 409 40D 40E 5GY 5QI 5VS 67Z 6NX 78A 8FE 8FG 8TC 8UJ 95- 95. 95~ 96X AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AAOBN AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYOK AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDZT ABECU ABFTD ABFTV ABHLI ABHQN ABJCF ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABULA ABWNU ABXPI ACAOD ACBXY ACDTI ACGFS ACHSB ACHXU ACIWK ACKNC ACMDZ ACMLO ACOKC ACOMO ACPIV ACSNA ACZOJ ADHHG ADHIR ADIMF ADINQ ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEBTG AEFIE AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AENEX AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFEXP AFGCZ AFKRA AFLOW AFQWF AFWTZ AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARAPS ARMRJ ASPBG AVWKF AXYYD AYJHY AZFZN B-. BA0 BBWZM BDATZ BENPR BGLVJ BGNMA BSONS C6C CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 EBLON EBS EIOEI EJD ESBYG FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNWQR GQ6 GQ7 GQ8 GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ KDC KOV KOW L6V LAK LLZTM M4Y M7S MA- N2Q NB0 NDZJH NPVJJ NQJWS NU0 O9- O93 O9G O9I O9J OAM P19 P62 P9O PF0 PT4 PT5 PTHSS Q2X QOK QOS R4E R89 R9I RHV RNI RNS ROL RPX RSV RZC RZE RZK S0W S16 S1Z S26 S27 S28 S3B SAP SCJ SCLPG SCO SDH SDM SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 T16 TSG TSK TSV TUC U2A UG4 UOJIU UTJUX UZXMN VC2 VFIZW W23 W48 WK8 YLTOR Z45 Z7R Z7S Z7V Z7X Z7Z Z81 Z83 Z86 Z88 Z8M Z8N Z8P Z8R Z8T Z8U Z8W Z92 ZMTXR ~EX AAPKM AAYXX ABBRH ABDBE ABFSG ABRTQ ACSTC ADHKG AEZWR AFDZB AFFHD AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION PHGZM PHGZT PQGLB 7SC 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c407t-83b742d7aa2d02fc56416f82284418ae6dd95bcbd54c9ff9d79b6df9220b8e43
IEDL.DBID	RSV
ISICitedReferencesCount	112
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000407921200008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1382-3256
IngestDate	Tue Dec 02 18:44:57 EST 2025 Sat Nov 29 07:48:37 EST 2025 Tue Nov 18 21:58:10 EST 2025 Fri Feb 21 02:35:47 EST 2025
IsDoiOpenAccess	true
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	5
Keywords	Negative results Replication study Sentiment analysis tools
Language	English
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c407t-83b742d7aa2d02fc56416f82284418ae6dd95bcbd54c9ff9d79b6df9220b8e43
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-1418-0095
OpenAccessLink	https://link.springer.com/10.1007/s10664-016-9493-x
PQID	2063601463
PQPubID	326341
PageCount	42
ParticipantIDs	proquest_journals_2063601463 crossref_primary_10_1007_s10664_016_9493_x crossref_citationtrail_10_1007_s10664_016_9493_x springer_journals_10_1007_s10664_016_9493_x
PublicationCentury	2000
PublicationDate	2017-10-01
PublicationDateYYYYMMDD	2017-10-01
PublicationDate_xml	– month: 10 year: 2017 text: 2017-10-01 day: 01
PublicationDecade	2010
PublicationPlace	New York
PublicationPlace_xml	– name: New York – name: Dordrecht
PublicationSubtitle	An International Journal
PublicationTitle	Empirical software engineering : an international journal
PublicationTitleAbbrev	Empir Software Eng
PublicationYear	2017
Publisher	Springer US Springer Nature B.V
Publisher_xml	– name: Springer US – name: Springer Nature B.V
References	Greiler M, Herzig K, Czerwonka J (2015) Code ownership and software quality: A replication study. In: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR ’15, pp. 2–12. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org.library.sutd.edu.sg:2048/citation.cfm?id=2820518.2820522 DestefanisGOrtuMCounsellSSwiftSMarchesiMTonelliRPeer J Comput Sci20162e73135 Islam MR, Zibran MF (2016) Towards understanding and exploiting developers’ emotional variations in software engineering. In: 2016 IEEE 14th International Conference on Software Engineering Research, Management and Applications (SERA), pp 185–192. doi:10.1109/SERA.2016.7516145 Datta S, Sindhgatta R, Sengupta B (2012) Talk versus work: characteristics of developer collaboration on the Jazz platform. In: Proceedings of the ACM international conference on Object oriented programming systems languages and applications, OOPSLA ’12, pp 655–668. ACM, New York, NY, USA. doi:10.1145/2384616.2384664 Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? classifying user reviews for software maintenance and evolution. In: ICSME. IEEE, pp 281–290 Sun X, Li B, Leung H, Li B, Li Y (2015) MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks. Inf Softw Technol 66:1–12. doi:10.1016/j.infsof.2015.05.003. http://www.sciencedirect.com/science/article/pii/S0950584915001007 Wang S, Lo D, Vasilescu B, Serebrenik A (2014) EnTagRec: An enhanced tag recommendation system for software information sites. In: ICSME. IEEE, pp 291–300 KonietschkeFHothornLABrunnerERank-based multiple test procedures and simultaneous confidence intervalsElectronic Journal of Statistics20126738759298842710.1214/12-EJS6911334.62083 Mishne G, Glance NS (2006) Predicting movie sales from blogger sentiment. In: AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, pp 155–158 Novielli N, Calefato F, Lanubile F (2015) The challenges of sentiment detection in the social programmer ecosystem. In: Proceedings of the 7th International Workshop on Social Software Engineering, SSE 2015, pp 33–40. ACM, New York, NY, USA. doi:10.1145/2804381.2804387 ThelwallMBuckleyKPaltoglouGSentiment strength detection for the social webJ Am Soc Inf Sci Technol201263116317310.1002/asi.21662 Goul M, Marjanovic O, Baxley S, Vizecky K (2012) Managing the Enterprise Business Intelligence App Store: Sentiment Analysis Supported Requirements Engineering. In: 2012 45th Hawaii International Conference on System Science (HICSS). doi:10.1109/HICSS.2012.421, pp 4168–4177 Sheskin DJ (2007) Handbook of parametric and nonparametric statistical procedures, 4 edn. Chapman & Hall PangBLeeLOpinion mining and sentiment analysisFound Trends Inf Retr200721-21135 Ortu M, Destefanis G, Kassab M, Counsell S, Marchesi M, Tonelli R (2015) Would you mind fixing this issue? - an empirical analysis of politeness and attractiveness in software developed using agile boards. In: Lassenius C, Dingsøyr T, Paasivaara M (eds) Agile Processes, in Software Engineering, and Extreme Programming - 16th International Conference, XP 2015, Helsinki, Finland, May 25–29, 2015, Proceedings, Lecture Notes in Business Information Processing, vol 212. Springer, pp 129–140. doi:10.1007/978-3-319-18612-2_11 Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, Grenning J, Highsmith J, Hunt A, Jeffries R, Kern J, Marick B, Martin RC, Mellor S, Schwaber K, Sutherland J, Thomas D (2001) Manifesto for agile software development http://agilemanifesto.org/principles.html Last accessed: October 14, 2015 Barkmann H, Lincke R, Löwe W (2009) Quantitative evaluation of software quality metrics in open-source projects. In: IEEE International Workshop on Quantitative Evaluation of large-scale Systems and Technologies, pp 1067–1072 Ortu M, Adams B, Destefanis G, Tourani P, Marchesi M, Tonelli R (2015) Are bullies more productive? empirical study of affectiveness vs. issue fixing time. In: MSR HubertLArabiePComparing partitionsJ Classif19852119321810.1007/BF019080750587.62128 Pletea D, Vasilescu B, Serebrenik A (2014) Security and emotion: Sentiment analysis of security discussions on GitHub. In: MSR. ACM, New York, NY, USA, pp 348–351. doi:10.1145/2597073.2597117 Schröter A, Aranda J, Damian D, Kwan I (2012) To talk or not to talk: factors that influence communication around changesets. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, CSCW ’12, pp 1317–1326. ACM, New York, NY, USA. doi:10.1145/2145204.2145401 Capiluppi A, Serebrenik A, Singer L (2013) Assessing technical candidates on the social web. Software. IEEE 30(1):45–51. doi:10.1109/MS.2012.169 ShullFJCarverJCVegasSJuristoNThe role of replications in empirical software engineeringEmpir Softw Eng200813221121810.1007/s10664-008-9060-1 de Magalhães CVC, da Silva FQB, Santos RES (2014) Investigations about replication of empirical studies in software engineering: Preliminary findings from a mapping study. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, pp 37:1–37:10. ACM, New York, NY, USA. doi:10.1145/2601248.2601289 VieraAJGarrettJMUnderstanding interobserver agreement: the kappa statisticFam Med2005375360363 Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp 422–431. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2486788.2486844 PritchardPSome negative results concerning prime number generatorsCommun ACM1984271535778432310.1145/69605.3579700589.10001 ZimmermanDWZumboBDParametric alternatives to the Student t test under violation of normality and homogeneity of variancePercept Mot Skills1992743183584410.2466/pms.1992.74.3.835 Sfetsos P, Adamidis P, Angelis L, Stamelos I, Deligiannis I (2012) Investigating the impact of personality and temperament traits on pair programming: a controlled experiment replication. In: 2012 Eighth International Conference on the Quality of Information and Communications Technology (QUATIC), pp 57–65. doi:10.1109/QUATIC.2012.36 Howard MJ, Gupta S, Pollock LL, Vijay-Shanker K (2013) Automatically mining software-based, semantically-similar words from comment-code mappings. In: Zimmermann T, Penta MD, Kim S (eds) MSR, pp 377–386. IEEE Computer Society VasilescuBSerebrenikAGoeminneMMensTOn the variation and specialisation of workload – a case study of the Gnome ecosystem communityEmpir Softw Eng2013194955100810.1007/s10664-013-9244-1 ThelwallMBuckleyKPaltoglouGCaiDKappasASentiment in short strength detection informal textJ Am Soc Inf Sci Technol201061122544255810.1002/asi.21416 Agarwal A, Xie B, Vovsha I, Rambow O, Passonneau R (2011) Sentiment Analysis of Twitter Data. In: Proceedings of the Workshop on Languages in Social Media, LSM ’11, pp 30–38. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=2021109.2021114 Giraud-CarrierCDunhamMHOn the importance of sharing negative resultsSigkdd Explor. Newsl.20111223410.1145/1964897.1964899 Lindsey MR (2011) What went wrong?: Negative results from VoIP service providers. In: Proceedings of the 5th International Conference on Principles, Systems and Applications of IP Telecommunications, IPTcomm ’11, pp 13:1–13:3. ACM, New York, NY, USA. doi:10.1145/2124436.2124453 TonellaPTorchianoMDu BoisBSystäTEmpirical studies in reverse engineering: State of the art and future trendsEmpir Softw Eng200712555157110.1007/s10664-007-9037-5 Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in GitHub: An empirical study. In: MSR, pp 352–355, ACM, New York, NY, USA RousinopoulosAIRoblesGGonzález-BarahonaJMSentiment analysis of Free/Open Source developers: preliminary findings from a case studyRevista Eletrônica de Sistemas de Informação20141326:16:2110.5329/RESI.2014.1302006 Honkela T, Izzatdust Z, Lagus K (2012) Text mining for wellbeing: Selecting stories using semantic and pragmatic features. In: Artificial Neural Networks and Machine Learning, Part II, LNCS, vol 7553. Springer, pp 467–474 Danescu-Niculescu-Mizil C, Sudhof M, Jurafsky D, Leskovec J, Potts C (2013) A computational approach to politeness with application to social factors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pp 250–259. The Association for Computer Linguistics. http://aclweb.org/anthology/P/P13/P13-1025.pdf BrunnerEMunzelUThe Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample ApproximationBiometrical Journal20004211725174456110.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U0969.62033 Garcia D, Zanetti MS, Schweitzer F (2013) The role of emotions in contributors activity: A case study on the Gentoo community. In: International Conference on Cloud and Green Computing, pp 410–417 Vasilescu B, Filkov V, Serebrenik A (2013) StackOverflow and github: associations between software development and crowdsourced knowledge. In: 2013 International Conference on Social Computing (SocialCom), pp 188–195. doi:10.1109/SocialCom.2013.35 Abbasi A, Hassan A, Dhar M (2014) Benchmarking Twitter sentiment analysis tools. In: International Conference on Language Resources and Evaluation. ELRA, Reykjavik, Iceland, pp 823–829 CohenJWeighted kappa: Nominal scale agreement with provision for scaled disagreement or partial creditPsychol Bull196870421322010.1037/h0026256 Shihab E, Kamei Y, Bhattacharya P (2012) Mining challenge 2012: the Android platform. In: MSR, pp 112–115 Batista GEAPA, Carvalho ACPLF, Monard MC (2000) Applying one-sided selection to unbalanced datasets. In: Mexican International Conference on Artificial Intelligence: Advances in Artificial Intelligence. Springer-Verlag, London, UK, UK, pp 315–325 Gamon M, Aue A, Corston-O 9493_CR39 9493_CR36 9493_CR37 9493_CR34 9493_CR32 9493_CR76 9493_CR33 9493_CR30 9493_CR74 9493_CR31 9493_CR75 9493_CR72 F Konietschke (9493_CR41) 2012; 6 9493_CR70 9493_CR71 JL Fleiss (9493_CR23) 2003 FJ Shull (9493_CR73) 2008; 13 KR Gabriel (9493_CR26) 1969; 40 9493_CR49 9493_CR47 9493_CR48 9493_CR45 9493_CR46 9493_CR43 9493_CR87 9493_CR44 9493_CR88 9493_CR42 OJ Dunn (9493_CR21) 1961; 56 9493_CR83 9493_CR40 9493_CR84 9493_CR81 9493_CR82 9493_CR80 F Wilcoxon (9493_CR89) 1945; 1 AI Rousinopoulos (9493_CR67) 2014; 13 P Pritchard (9493_CR64) 1984; 27 P Tonella (9493_CR79) 2007; 12 B Vasilescu (9493_CR85) 2013; 19 M Hall (9493_CR35) 2009; 11 9493_CR16 M Thelwall (9493_CR77) 2012; 63 9493_CR17 9493_CR14 9493_CR58 9493_CR15 9493_CR59 9493_CR12 9493_CR56 9493_CR57 9493_CR9 9493_CR54 9493_CR11 9493_CR55 9493_CR52 J Cohen (9493_CR13) 1968; 70 9493_CR53 9493_CR50 9493_CR51 9493_CR92 9493_CR90 9493_CR91 B Pang (9493_CR60) 2007; 2 M Thelwall (9493_CR78) 2010; 61 DW Zimmerman (9493_CR93) 1992; 74 AJ Viera (9493_CR86) 2005; 37 E Brunner (9493_CR10) 2000; 42 9493_CR27 9493_CR28 9493_CR25 9493_CR69 9493_CR24 9493_CR68 9493_CR22 9493_CR66 9493_CR63 9493_CR20 9493_CR61 9493_CR62 G Destefanis (9493_CR19) 2016; 2 L Hubert (9493_CR38) 1985; 2 9493_CR7 9493_CR8 WM Rand (9493_CR65) 1971; 66 9493_CR5 9493_CR6 9493_CR3 C Giraud-Carrier (9493_CR29) 2011; 12 9493_CR4 9493_CR1 9493_CR18 9493_CR2
References_xml	– reference: DunnOJMultiple comparisons among meansJ Am Stat Assoc196156293526412495210.1080/01621459.1961.104820900103.37001 – reference: Dajsuren Y, van den Brand MGJ, Serebrenik A, Roubtsov S (2013) Simulink models are also software: Modularity assessment. In: Proceedings of the 9th International ACM Sigsoft Conference on Quality of Software Architectures, QoSA ’13, pp 99–106. ACM, New York, NY, USA. doi:10.1145/2465478.2465482 – reference: Howard MJ, Gupta S, Pollock LL, Vijay-Shanker K (2013) Automatically mining software-based, semantically-similar words from comment-code mappings. In: Zimmermann T, Penta MD, Kim S (eds) MSR, pp 377–386. IEEE Computer Society – reference: ShullFJCarverJCVegasSJuristoNThe role of replications in empirical software engineeringEmpir Softw Eng200813221121810.1007/s10664-008-9060-1 – reference: Abbasi A, Hassan A, Dhar M (2014) Benchmarking Twitter sentiment analysis tools. In: International Conference on Language Resources and Evaluation. ELRA, Reykjavik, Iceland, pp 823–829 – reference: Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? classifying user reviews for software maintenance and evolution. In: ICSME. IEEE, pp 281–290 – reference: RandWMObjective criteria for the evaluation of clustering methodsJ Am Stat Assoc19716633684685010.1080/01621459.1971.10482356 – reference: Shihab E, Kamei Y, Bhattacharya P (2012) Mining challenge 2012: the Android platform. In: MSR, pp 112–115 – reference: Yu HF, Ho CH, Juan YC, Lin CJ (2013) Libshorttext: A library for short-text classification and analysis. Tech. rep., Technical Report. http://www.csie.ntu.edu.tw/~cjlin/papers/libshorttext.pdf – reference: Yu Y, Wang H, Yin G, Wang T (2016) Reviewer recommendation for pull-requests in github: What can we learn from code review and bug assignment?. Inf Softw Technol 74:204–218. doi:10.1016/j.infsof.2016.01.004. http://www.sciencedirect.com/science/article/pii/S0950584916000069 – reference: VieraAJGarrettJMUnderstanding interobserver agreement: the kappa statisticFam Med2005375360363 – reference: Tukey JW (1951) Quick and dirty methods in statistics, part II, Simple analysis for standard designs. In: American Society for Quality Control, pp 189–197 – reference: CohenJWeighted kappa: Nominal scale agreement with provision for scaled disagreement or partial creditPsychol Bull196870421322010.1037/h0026256 – reference: DestefanisGOrtuMCounsellSSwiftSMarchesiMTonelliRPeer J Comput Sci20162e73135 – reference: Murgia A, Tourani P, Adams B, Ortu M (2014) Do developers feel emotions? an exploratory analysis of emotions in software artifacts. In: MSR, pp 262-271, ACM, New York, NY, USA – reference: Barkmann H, Lincke R, Löwe W (2009) Quantitative evaluation of software quality metrics in open-source projects. In: IEEE International Workshop on Quantitative Evaluation of large-scale Systems and Technologies, pp 1067–1072 – reference: ThelwallMBuckleyKPaltoglouGCaiDKappasASentiment in short strength detection informal textJ Am Soc Inf Sci Technol201061122544255810.1002/asi.21416 – reference: Davidov D, Tsur O, Rappoport A (2010) Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. In: Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pp. 107–116. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=1870568.1870582 – reference: Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Empirical Methods in Natural Language Processing, pp 1631–1642. Ass. for Comp. Linguistics – reference: Bakeman R, Gottman JM (1997) Observing interaction: an introduction to sequential analysis. Cambridge University Press. https://books.google.nl/books?id=CMj2SmcijhEC – reference: Sun X, Li B, Leung H, Li B, Li Y (2015) MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks. Inf Softw Technol 66:1–12. doi:10.1016/j.infsof.2015.05.003. http://www.sciencedirect.com/science/article/pii/S0950584915001007 – reference: Mende T (2010) Replication of defect prediction studies: Problems, pitfalls and recommendations. In: Proceedings of the 6th International Conference on Predictive Models in Software Engineering, PROMISE ’10, pp 5:1–5:10. ACM, New York, NY, USA. doi:10.1145/1868328.1868336 – reference: Li TH, Liu R, Sukaviriya N, Li Y, Yang J, Sandin M, Lee J (2014) Incident ticket analytics for it application management services. In: 2014 IEEE International Conference on Services Computing (SCC), pp 568–574. doi:10.1109/SCC.2014.80 – reference: Sheskin DJ (2007) Handbook of parametric and nonparametric statistical procedures, 4 edn. Chapman & Hall – reference: van Rijsbergen CJ (1979) Information Retrieval, 2nd edn. Butterworth-Heinemann, Newton, MA, USA – reference: Tourani P, Jiang Y, Adams B (2014) Monitoring sentiment in open source mailing lists: exploratory study on the apache ecosystem. In: Proceedings of 24th Annual International Conference on Computer Science and Software Engineering, CASCON ’14, pp 34–44. IBM Corp., Riverton, NJ, USA. http://dl.acm.org/citation.cfm?id=2735522.2735528 – reference: Vasilescu B, Filkov V, Serebrenik A (2013) StackOverflow and github: associations between software development and crowdsourced knowledge. In: 2013 International Conference on Social Computing (SocialCom), pp 188–195. doi:10.1109/SocialCom.2013.35 – reference: RousinopoulosAIRoblesGGonzález-BarahonaJMSentiment analysis of Free/Open Source developers: preliminary findings from a case studyRevista Eletrônica de Sistemas de Informação20141326:16:2110.5329/RESI.2014.1302006 – reference: Wilson T, Wiebe J, Hoffmann P (2005) Recognizing contextual polarity in phrase-level sentiment analysis. In: Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Stroudsburg, PA, USA, pp 347–354 – reference: BrunnerEMunzelUThe Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample ApproximationBiometrical Journal20004211725174456110.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U0969.62033 – reference: HubertLArabiePComparing partitionsJ Classif19852119321810.1007/BF019080750587.62128 – reference: WilcoxonFIndividual comparisons by ranking methodsBiom Bull194516808310.2307/3001968 – reference: Ortu M, Adams B, Destefanis G, Tourani P, Marchesi M, Tonelli R (2015) Are bullies more productive? empirical study of affectiveness vs. issue fixing time. In: MSR – reference: Datta S, Sindhgatta R, Sengupta B (2012) Talk versus work: characteristics of developer collaboration on the Jazz platform. In: Proceedings of the ACM international conference on Object oriented programming systems languages and applications, OOPSLA ’12, pp 655–668. ACM, New York, NY, USA. doi:10.1145/2384616.2384664 – reference: FleissJLLevinBPaikMCStatistical methods for rates and proportions20033rd ednNJWiley Series in Probability and Statistics. Wiley, Hoboken10.1002/04714454281034.62113 – reference: Pletea D, Vasilescu B, Serebrenik A (2014) Security and emotion: Sentiment analysis of security discussions on GitHub. In: MSR. ACM, New York, NY, USA, pp 348–351. doi:10.1145/2597073.2597117 – reference: Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, Grenning J, Highsmith J, Hunt A, Jeffries R, Kern J, Marick B, Martin RC, Mellor S, Schwaber K, Sutherland J, Thomas D (2001) Manifesto for agile software development http://agilemanifesto.org/principles.html Last accessed: October 14, 2015 – reference: Fuhr N, Muller P (1987) Probabilistic search term weighting - some negative results. In: Proceedings of the 10th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’87, pp 13–18. ACM, New York, NY, USA. doi:10.1145/42005.42007 – reference: Gousios G (2013) The GHTorrent dataset and tool suite. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR’13, pp 233–236. http://dl.acm.org/citation.cfm?id=2487085.2487132 – reference: Kucuktunc O, Cambazoglu BB, Weber I, Ferhatosmanoglu H (2012) A Large-scale Sentiment Analysis for Yahoo! Answers. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM ’12, pp 633–642. ACM, New York, NY, USA. doi:10.1145/2124295.2124371 – reference: ThelwallMBuckleyKPaltoglouGSentiment strength detection for the social webJ Am Soc Inf Sci Technol201263116317310.1002/asi.21662 – reference: Gamon M, Aue A, Corston-Oliver S, Ringger E (2005) Pulse: Mining customer opinions from free text. In: Proceedings of the 6th International Conference on Advances in Intelligent Data Analysis, IDA’05. Springer-Verlag, Berlin, Heidelberg, pp 121–132. doi:10.1007/11552253_12 – reference: HallMFrankEHolmesGPfahringerBReutemannPWittenIHThe Weka data mining software: An upyearSIGKDD Explor Newsl2009111101810.1145/1656274.1656278 – reference: Dewan P (2015) Towards Emotion-Based Collaborative Software Engineering. In: 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp 109–112. doi:10.1109/CHASE.2015.32 – reference: Lanza M, Di Penta M, Xie T (2012) (eds.): 9th IEEE Working Conference of Mining Software Repositories, MSR 2012, June 2-3, 2012, Zurich, Switzerland. IEEE Computer Society. http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=6220358 – reference: GabrielKRSimultaneous test procedures—some theory of multiple comparisonsAnn Math Stat196940122425024093110.1214/aoms/11776978190198.23602 – reference: Täht D (2014) The value of repeatable experiments and negative results: - a journey through the history and future of aqm and fair queuing algorithms. In: Proceedings of the 2014 ACM SIGCOMM Workshop on Capacity Sharing Workshop, CSWS ’14, pp 1–2. ACM, New York, NY, USA. doi:10.1145/2630088.2652480 – reference: Tumasjan A, Sprenger TO, Sandner PG, Welpe IM (2010) Predicting elections with twitter: What 140 characters reveal about political sentiment. In: International AAAI Conference on Weblogs and Social Media, pp 178–185 – reference: Cataldo M, Herbsleb JD (2008) Communication networks in geographically distributed software development. In: Proceedings of the 2008 ACM conference on Computer supported cooperative work, CSCW ’08, pp 579–588. ACM, New York, NY, USA. doi:10.1145/1460563.1460654 – reference: Honkela T, Izzatdust Z, Lagus K (2012) Text mining for wellbeing: Selecting stories using semantic and pragmatic features. In: Artificial Neural Networks and Machine Learning, Part II, LNCS, vol 7553. Springer, pp 467–474 – reference: Vasilescu B, Serebrenik A, van den Brand MGJ (2011) By no means: a study on aggregating software metrics. In: Concas G, Tempero ED, Zhang H, Penta MD (eds) Proceedings of the 2nd International Workshop on Emerging Trends in Software Metrics, WETSoM 2011, Waikiki, Honolulu, HI, USA, May 24, 2011. ACM, pp 23–26. doi:10.1145/1985374.1985381 – reference: de Magalhães CVC, da Silva FQB, Santos RES (2014) Investigations about replication of empirical studies in software engineering: Preliminary findings from a mapping study. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, pp 37:1–37:10. ACM, New York, NY, USA. doi:10.1145/2601248.2601289 – reference: Capiluppi A, Serebrenik A, Singer L (2013) Assessing technical candidates on the social web. Software. IEEE 30(1):45–51. doi:10.1109/MS.2012.169 – reference: Asaduzzaman M, Bullock MC, Roy CK, Schneider KA Bug introducing changes: A case study with android. In: Lanza et al. [43], pp 116–119. doi:10.1109/MSR.2012.6224267 – reference: Ortu M, Destefanis G, Kassab M, Counsell S, Marchesi M, Tonelli R (2015) Would you mind fixing this issue? - an empirical analysis of politeness and attractiveness in software developed using agile boards. In: Lassenius C, Dingsøyr T, Paasivaara M (eds) Agile Processes, in Software Engineering, and Extreme Programming - 16th International Conference, XP 2015, Helsinki, Finland, May 25–29, 2015, Proceedings, Lecture Notes in Business Information Processing, vol 212. Springer, pp 129–140. doi:10.1007/978-3-319-18612-2_11 – reference: Schröter A, Aranda J, Damian D, Kwan I (2012) To talk or not to talk: factors that influence communication around changesets. In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, CSCW ’12, pp 1317–1326. ACM, New York, NY, USA. doi:10.1145/2145204.2145401 – reference: Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Valletta, Malta. http://www.lrec-conf.org/proceedings/lrec2010/pdf/769_Paper.pdf – reference: Bird S, Loper E, Klein E (2009) Natural language processing with Python. O’Reilly Media Inc – reference: Goul M, Marjanovic O, Baxley S, Vizecky K (2012) Managing the Enterprise Business Intelligence App Store: Sentiment Analysis Supported Requirements Engineering. In: 2012 45th Hawaii International Conference on System Science (HICSS). doi:10.1109/HICSS.2012.421, pp 4168–4177 – reference: PangBLeeLOpinion mining and sentiment analysisFound Trends Inf Retr200721-21135 – reference: Guzman E, Bruegge B (2013) Towards emotional awareness in software development teams. In: Joint Meeting on Foundations of Software Engineering, pp 671–674, ACM, New York, NY, USA – reference: Garcia D, Zanetti MS, Schweitzer F (2013) The role of emotions in contributors activity: A case study on the Gentoo community. In: International Conference on Cloud and Green Computing, pp 410–417 – reference: Mishne G, Glance NS (2006) Predicting movie sales from blogger sentiment. In: AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, pp 155–158 – reference: Mohammad SM, Kiritchenko S, Zhu X (2013) NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets. arXiv:1308.6242[cs] – reference: PritchardPSome negative results concerning prime number generatorsCommun ACM1984271535778432310.1145/69605.3579700589.10001 – reference: Santos JM, Embrechts M (2009) On the use of the adjusted rand index as a metric for evaluating supervised classification. In: International Conference on Artificial Neural Networks, LNCS, vol 5769. Springer, pp 175–184 – reference: Pang B, Lee L, Vaithyanathan S (2002) Thumbs Up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pp 79–86. Association for Computational Linguistics, Stroudsburg, PA, USA. doi:10.3115/1118693.1118704 – reference: Agarwal A, Xie B, Vovsha I, Rambow O, Passonneau R (2011) Sentiment Analysis of Twitter Data. In: Proceedings of the Workshop on Languages in Social Media, LSM ’11, pp 30–38. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=2021109.2021114 – reference: Jongeling R, Datta S, Serebrenik A (2015) Choosing your weapons: On sentiment analysis tools for software engineering research. In: ICSME, pp 531–535. IEEE. doi:10.1109/ICSM.2015.7332508 – reference: Linstead E, Baldi P (2009) Mining the coherence of GNOME bug reports with statistical topic models. In: Godfrey MW, Whitehead J (eds) Proceedings of the 6th International Working Conference on Mining Software Repositories, MSR 2009 (Co-located with ICSE), Vancouver, BC, Canada, May 16–17, 2009, Proceedings, pp 99–102. IEEE Computer Society. doi:10.1109/MSR.2009.5069486 – reference: Martie L, Palepu VK, Sajnani H, Lopes CV Trendy bugs: Topic trends in the android bug reports. In: Lanza et al. [43], pp 120–123. doi:10.1109/MSR.2012.6224268 – reference: Novielli N, Calefato F, Lanubile F (2015) The challenges of sentiment detection in the social programmer ecosystem. In: Proceedings of the 7th International Workshop on Social Software Engineering, SSE 2015, pp 33–40. ACM, New York, NY, USA. doi:10.1145/2804381.2804387 – reference: Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp 422–431. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2486788.2486844 – reference: Islam MR, Zibran MF (2016) Towards understanding and exploiting developers’ emotional variations in software engineering. In: 2016 IEEE 14th International Conference on Software Engineering Research, Management and Applications (SERA), pp 185–192. doi:10.1109/SERA.2016.7516145 – reference: Giraud-CarrierCDunhamMHOn the importance of sharing negative resultsSigkdd Explor. Newsl.20111223410.1145/1964897.1964899 – reference: TonellaPTorchianoMDu BoisBSystäTEmpirical studies in reverse engineering: State of the art and future trendsEmpir Softw Eng200712555157110.1007/s10664-007-9037-5 – reference: Wang S, Lo D, Vasilescu B, Serebrenik A (2014) EnTagRec: An enhanced tag recommendation system for software information sites. In: ICSME. IEEE, pp 291–300 – reference: Batista GEAPA, Carvalho ACPLF, Monard MC (2000) Applying one-sided selection to unbalanced datasets. In: Mexican International Conference on Artificial Intelligence: Advances in Artificial Intelligence. Springer-Verlag, London, UK, UK, pp 315–325 – reference: Pak A, Paroubek P (2010) Twitter Based System: Using Twitter for Disambiguating Sentiment Ambiguous Adjectives. In: Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, pp. 436–439. Association for Computational Linguistics, Stroudsburg, PA, USA. http://dl.acm.org/citation.cfm?id=1859664.1859761 – reference: KonietschkeFHothornLABrunnerERank-based multiple test procedures and simultaneous confidence intervalsElectronic Journal of Statistics20126738759298842710.1214/12-EJS6911334.62083 – reference: Ortu M, Destefanis G, Adams B, Murgia A, Marchesi M, Tonelli R (2015) The JIRA repository dataset: Understanding social aspects of software development. In: Proceedings of the 11th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE ’15, pp 1:1–1:4. ACM, New York, NY, USA. doi:10.1145/2810146.2810147 – reference: Fontana FA, Mariani E, Morniroli A, Sormani R, Tonello A (2011) An experience report on using code smells detection tools. In: ICST Workshops, pp 450–457. IEEE – reference: Danescu-Niculescu-Mizil C, Sudhof M, Jurafsky D, Leskovec J, Potts C (2013) A computational approach to politeness with application to social factors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pp 250–259. The Association for Computer Linguistics. http://aclweb.org/anthology/P/P13/P13-1025.pdf – reference: Costa JM, Cataldo M, de Souza CR (2011) The scale and evolution of coordination needs in large-scale distributed projects: implications for the future generation of collaborative tools. In: Proceedings of the 2011 annual conference on Human factors in computing systems, CHI ’11, pp 3151–3160. ACM, New York, NY, USA. doi:10.1145/1978942.1979409 – reference: VasilescuBSerebrenikAGoeminneMMensTOn the variation and specialisation of workload – a case study of the Gnome ecosystem communityEmpir Softw Eng2013194955100810.1007/s10664-013-9244-1 – reference: Lindsey MR (2011) What went wrong?: Negative results from VoIP service providers. In: Proceedings of the 5th International Conference on Principles, Systems and Applications of IP Telecommunications, IPTcomm ’11, pp 13:1–13:3. ACM, New York, NY, USA. doi:10.1145/2124436.2124453 – reference: Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in GitHub: An empirical study. In: MSR, pp 352–355, ACM, New York, NY, USA – reference: Leopairote W, Surarerks A, Prompoon N (2013) Evaluating software quality in use using user reviews mining. In: 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE), pp 257–262. doi:10.1109/JCSSE.2013.6567355 – reference: Greiler M, Herzig K, Czerwonka J (2015) Code ownership and software quality: A replication study. In: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR ’15, pp. 2–12. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org.library.sutd.edu.sg:2048/citation.cfm?id=2820518.2820522 – reference: Sfetsos P, Adamidis P, Angelis L, Stamelos I, Deligiannis I (2012) Investigating the impact of personality and temperament traits on pair programming: a controlled experiment replication. In: 2012 Eighth International Conference on the Quality of Information and Communications Technology (QUATIC), pp 57–65. doi:10.1109/QUATIC.2012.36 – reference: Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’94, pp. 3–12. Springer-Verlag New York, Inc., New York, NY, USA. http://dl.acm.org.dianus.libr.tue.nl/citation.cfm?id=188490.188495 – reference: ZimmermanDWZumboBDParametric alternatives to the Student t test under violation of normality and homogeneity of variancePercept Mot Skills1992743183584410.2466/pms.1992.74.3.835 – reference: Vivian R, Tarmazdi H, Falkner K, Falkner N, Szabo C (2015) The development of a dashboard tool for visualising online teamwork discussions. In: Proceedings of the 37th International Conference on Software Engineering - Volume 2, ICSE ’15, pp 380–388. IEEE Press, Piscataway, NJ, USA. http://dl.acm.org/citation.cfm?id=2819009.2819070 – ident: 9493_CR39 doi: 10.1109/SERA.2016.7516145 – ident: 9493_CR11 doi: 10.1109/MS.2012.169 – ident: 9493_CR66 – volume: 1 start-page: 80 issue: 6 year: 1945 ident: 9493_CR89 publication-title: Biom Bull doi: 10.2307/3001968 – volume: 40 start-page: 224 issue: 1 year: 1969 ident: 9493_CR26 publication-title: Ann Math Stat doi: 10.1214/aoms/1177697819 – ident: 9493_CR43 – volume: 66 start-page: 846 issue: 336 year: 1971 ident: 9493_CR65 publication-title: J Am Stat Assoc doi: 10.1080/01621459.1971.10482356 – ident: 9493_CR88 doi: 10.1109/ICSME.2014.51 – ident: 9493_CR24 doi: 10.1109/ICSTW.2011.12 – volume: 12 start-page: 3 issue: 2 year: 2011 ident: 9493_CR29 publication-title: Sigkdd Explor. Newsl. doi: 10.1145/1964897.1964899 – ident: 9493_CR47 doi: 10.1145/2124436.2124453 – ident: 9493_CR22 doi: 10.1109/ICSE.2013.6606588 – ident: 9493_CR70 doi: 10.1109/QUATIC.2012.36 – ident: 9493_CR76 doi: 10.1145/2630088.2652480 – ident: 9493_CR75 doi: 10.1016/j.infsof.2015.05.003 – ident: 9493_CR36 doi: 10.1007/978-3-642-33266-1_58 – ident: 9493_CR18 – ident: 9493_CR52 – volume: 37 start-page: 360 issue: 5 year: 2005 ident: 9493_CR86 publication-title: Fam Med – ident: 9493_CR20 doi: 10.1109/CHASE.2015.32 – ident: 9493_CR92 doi: 10.1016/j.infsof.2016.01.004 – volume: 11 start-page: 10 issue: 1 year: 2009 ident: 9493_CR35 publication-title: SIGKDD Explor Newsl doi: 10.1145/1656274.1656278 – ident: 9493_CR62 doi: 10.1109/ICSM.2015.7332474 – ident: 9493_CR84 doi: 10.1145/1985374.1985381 – volume: 74 start-page: 835 issue: 31 year: 1992 ident: 9493_CR93 publication-title: Percept Mot Skills doi: 10.2466/pms.1992.74.3.835 – ident: 9493_CR51 doi: 10.1145/1868328.1868336 – ident: 9493_CR72 doi: 10.1109/MSR.2012.6224307 – volume: 2 start-page: 1 issue: e73 year: 2016 ident: 9493_CR19 publication-title: Peer J Comput Sci doi: 10.7717/peerj-cs.73 – ident: 9493_CR56 doi: 10.1109/MSR.2015.35 – ident: 9493_CR2 – ident: 9493_CR40 doi: 10.1109/ICSM.2015.7332508 – ident: 9493_CR44 doi: 10.1109/JCSSE.2013.6567355 – ident: 9493_CR32 doi: 10.1109/MSR.2015.8 – ident: 9493_CR50 doi: 10.1109/MSR.2012.6224268 – volume: 70 start-page: 213 issue: 4 year: 1968 ident: 9493_CR13 publication-title: Psychol Bull doi: 10.1037/h0026256 – volume: 42 start-page: 17 issue: 1 year: 2000 ident: 9493_CR10 publication-title: Biometrical Journal doi: 10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U – ident: 9493_CR14 doi: 10.1145/1978942.1979409 – ident: 9493_CR74 doi: 10.18653/v1/D13-1170 – ident: 9493_CR87 doi: 10.1109/ICSE.2015.170 – volume: 56 start-page: 52 issue: 293 year: 1961 ident: 9493_CR21 publication-title: J Am Stat Assoc doi: 10.1080/01621459.1961.10482090 – volume-title: Statistical methods for rates and proportions year: 2003 ident: 9493_CR23 doi: 10.1002/0471445428 – volume: 6 start-page: 738 year: 2012 ident: 9493_CR41 publication-title: Electronic Journal of Statistics doi: 10.1214/12-EJS691 – ident: 9493_CR83 doi: 10.1109/SocialCom.2013.35 – ident: 9493_CR48 doi: 10.1109/MSR.2009.5069486 – ident: 9493_CR54 doi: 10.1145/2597073.2597086 – ident: 9493_CR69 doi: 10.1145/2145204.2145401 – ident: 9493_CR59 – ident: 9493_CR28 doi: 10.1109/CGC.2013.71 – ident: 9493_CR6 doi: 10.1109/WAINA.2009.190 – ident: 9493_CR7 doi: 10.1007/10720076_29 – ident: 9493_CR63 doi: 10.1145/2597073.2597117 – volume: 61 start-page: 2544 issue: 12 year: 2010 ident: 9493_CR78 publication-title: J Am Soc Inf Sci Technol doi: 10.1002/asi.21416 – volume: 13 start-page: 6:1 issue: 2 year: 2014 ident: 9493_CR67 publication-title: Revista Eletrônica de Sistemas de Informação doi: 10.5329/RESI.2014.1302006 – ident: 9493_CR27 doi: 10.1007/11552253_12 – ident: 9493_CR80 – ident: 9493_CR45 – ident: 9493_CR1 – ident: 9493_CR9 – ident: 9493_CR25 doi: 10.1145/42005.42007 – ident: 9493_CR15 doi: 10.1145/2465478.2465482 – ident: 9493_CR17 doi: 10.1145/2384616.2384664 – volume: 27 start-page: 53 issue: 1 year: 1984 ident: 9493_CR64 publication-title: Commun ACM doi: 10.1145/69605.357970 – ident: 9493_CR31 – ident: 9493_CR16 – ident: 9493_CR30 doi: 10.1109/HICSS.2012.421 – ident: 9493_CR61 doi: 10.3115/1118693.1118704 – ident: 9493_CR12 doi: 10.1145/1460563.1460654 – ident: 9493_CR71 – volume: 63 start-page: 163 issue: 1 year: 2012 ident: 9493_CR77 publication-title: J Am Soc Inf Sci Technol doi: 10.1002/asi.21662 – ident: 9493_CR33 doi: 10.1145/2597073.2597118 – ident: 9493_CR90 doi: 10.3115/1220575.1220619 – volume: 2 start-page: 1 issue: 1-2 year: 2007 ident: 9493_CR60 publication-title: Found Trends Inf Retr – volume: 19 start-page: 955 issue: 4 year: 2013 ident: 9493_CR85 publication-title: Empir Softw Eng doi: 10.1007/s10664-013-9244-1 – ident: 9493_CR4 – ident: 9493_CR58 doi: 10.1007/978-3-319-18612-2_11 – ident: 9493_CR34 doi: 10.1145/2491411.2494578 – ident: 9493_CR3 doi: 10.1109/MSR.2012.6224267 – ident: 9493_CR37 doi: 10.1109/MSR.2013.6624052 – ident: 9493_CR81 – ident: 9493_CR68 doi: 10.1007/978-3-642-04277-5_18 – ident: 9493_CR8 – ident: 9493_CR5 doi: 10.1017/CBO9780511527685 – ident: 9493_CR49 doi: 10.1145/2601248.2601289 – ident: 9493_CR42 doi: 10.1145/2124295.2124371 – ident: 9493_CR82 doi: 10.1609/icwsm.v4i1.14009 – ident: 9493_CR55 doi: 10.1145/2804381.2804387 – ident: 9493_CR57 doi: 10.1145/2810146.2810147 – volume: 13 start-page: 211 issue: 2 year: 2008 ident: 9493_CR73 publication-title: Empir Softw Eng doi: 10.1007/s10664-008-9060-1 – ident: 9493_CR91 – ident: 9493_CR46 doi: 10.1109/SCC.2014.80 – ident: 9493_CR53 – volume: 2 start-page: 193 issue: 1 year: 1985 ident: 9493_CR38 publication-title: J Classif doi: 10.1007/BF01908075 – volume: 12 start-page: 551 issue: 5 year: 2007 ident: 9493_CR79 publication-title: Empir Softw Eng doi: 10.1007/s10664-007-9037-5
SSID	ssj0009745
Score	2.5640619
Snippet	Recent years have seen an increasing attention to social aspects of software engineering, including studies of emotions and sentiments experienced and...
SourceID	proquest crossref springer
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	2543
SubjectTerms	Compilers Computer Science Contrarian investing Data mining Engineering research Interpreters Product reviews Programming Languages Sentiment analysis Social factors Software Software engineering Software Engineering/Programming and Operating Systems Studies
Title	On negative results when using sentiment analysis tools for software engineering research
URI	https://link.springer.com/article/10.1007/s10664-016-9493-x https://www.proquest.com/docview/2063601463
Volume	22
WOSCitedRecordID	wos000407921200008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVAVX databaseName: SpringerLINK Contemporary 1997-Present customDbUrl: eissn: 1573-7616 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0009745 issn: 1382-3256 databaseCode: RSV dateStart: 19970101 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LSwMxEA6iHrxYn1itkoMnJbDuI4-jiMWDqGgp9bQkm6QVyla6W-vPd5Luuioq6HmzQ5gkM98wM98gdMwjS1XMKYQlSkGAoixxtFskEBbckw1p4scB9a_ZzQ0fDMRd1cdd1NXudUrSW-oPzW6UuooJSkQsIgLAcQXEcTev4f6h3zDtMj-Z2HHrkQgcep3K_E7EZ2fUIMwvSVHva7qtf-1yA61X0BKfL-7CJloy-RZq1WMbcPWKt9HjbY5zM_SM3xii7dm4LPB8ZHLsiuCH2LUjec5_LCvGElxOJuMCA77FBZjtuZwabBoiQ1wxBo12UK972bu4ItWEBZJBIFcSHikIjTWTMtRBaLOEAj6zgBk4oCQuDdVaJCpTOokzYa3QTCiqrQjDQHETR7toOZ_kZg9hm0gaaC4tU2cxM1wpxcB02oDRWMtYt1FQazrNKvZxNwRjnDa8yU5zqas4c5pLX9vo5P2X5wX1xm-LO_XxpdUrLNIwcGxo4AuiNjqtj6v5_KOw_T-tPkBrofP1vsKvg5bL6cwcotXspXwqpkf-cr4BAb3hXQ
linkProvider	Springer Nature
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnZ3NS8MwGMaDqKAX5ydOp-bgSQnUNs3HUcQxcU7RMeapJE2yCaOTtXP--SZda1VU0HPTUNIkz_OSN78XgGMWGCIxIzYskdIGKNIgh91CHjdWnoxPwrwcUK9NOx3W7_O74h53Wma7l0eS-U794bIbIS5jgiCOeYCscVzCVrAcMP_-oVeRdmlemdix9VBgBb08yvyui89iVDnML4eiudY0a__6ynWwVlhLeD6fCxtgQSeboFaWbYDFKt4Cj7cJTPQgJ35DG21PR1kKZ0OdQJcEP4DuOlLO_IeiIJbAbDwepdD6W5jabXsmJhrqCmQIC2LQcBt0m5fdixYqKiyg2AZyGWKBtKGxokL4yvNNHBLrz4z1DMy6JCY0UYqHMpYqxDE3hivKJVGG-74nmcbBDlhMxoneBdCEgniKCUPlGaaaSSmp3TqNRwlWAqs68MqRjuKCPu6KYIyiipvsRi5yGWdu5KLXOjh5f-V5jt74rXGj_H1RsQrTyPccDc1qQVAHp-Xvqh7_2Nnen1ofgZVW96Ydta861_tg1Xe6n2f7NcBiNpnqA7Acv2RP6eQwn6hvmubkQQ
linkToPdf	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LSwMxEA6iIl6sT6xWzcGTErruI4-jqEWx1IIi9bQkm6QVyrZ0t9afb7LNuioqiOfNhjB5zDfMzPcBcEwDjUVIsQlLhDABitDI0m4hj2njnrSPo0IO6LFNOh3a67Gu0znNymr3MiU572mwLE1p3hxL3fzQ-IaxrZ7AiIUsQAZELoW2jt6G6_ePFesuKVSKLc8eCoxzL9Oa303x2TFVaPNLgrTwO63av1e8DtYc5ITn8zOyARZUuglqpZwDdLd7CzzdpTBV_YIJHJoofDrMMzgbqBTa4vg-tG1KhRYA5I7JBOaj0TCDBvfCzDznMz5RUFUEh9AxCQ22wUPr6uHiGjnlBZSYAC9HNBAmZJaEc196vk4ibHCbNliCGvREucJSskgkQkZhwrRmkjCBpWa-7wmqwmAHLKajVO0CqCOOPUm5JuIsJIoKIYh5UrVHcCh5KOvAK60eJ46V3IpjDOOKT9laLraVaNZy8WsdnLz_Mp5Tcvw2uFFuZexuZxb7nmVJMz4iqIPTcuuqzz9Otven0UdgpXvZits3ndt9sOpbOFAUATbAYj6ZqgOwnLzkz9nksDizbwXH7SU
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+negative+results+when+using+sentiment+analysis+tools+for+software+engineering+research&rft.jtitle=Empirical+software+engineering+%3A+an+international+journal&rft.au=Jongeling%2C+Robbert&rft.au=Sarkar%2C+Proshanta&rft.au=Datta%2C+Subhajit&rft.au=Serebrenik%2C+Alexander&rft.date=2017-10-01&rft.pub=Springer+US&rft.issn=1382-3256&rft.eissn=1573-7616&rft.volume=22&rft.issue=5&rft.spage=2543&rft.epage=2584&rft_id=info:doi/10.1007%2Fs10664-016-9493-x&rft.externalDocID=10_1007_s10664_016_9493_x
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1382-3256&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1382-3256&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1382-3256&client=summon