Sentiment analysis for software engineering how far can we go?

Sentiment analysis has been applied to various software engineering (SE) tasks, such as evaluating app reviews or analyzing developers' emotions in commit messages. Studies indicate that sentiment analysis tools provide unreliable results when used out-of-the-box, since they are not designed to...

Full description

Saved in:
Bibliographic Details
Published in:2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) pp. 94 - 104
Main Authors: Lin, Bin, Zampetti, Fiorella, Bavota, Gabriele, Di Penta, Massimiliano, Lanza, Michele, Oliveto, Rocco
Format: Conference Proceeding
Language:English
Published: New York, NY, USA ACM 27.05.2018
Series:ACM Conferences
Subjects:
Information systems > Information retrieval > Retrieval tasks and goals > Sentiment analysis
Motion pictures
NLP
Sentiment analysis
Software
Software engineering
Task analysis
Training
sentiment analysis
software engineering
NLP
ISBN:9781450356381, 1450356389
ISSN:1558-1225
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Sentiment analysis has been applied to various software engineering (SE) tasks, such as evaluating app reviews or analyzing developers' emotions in commit messages. Studies indicate that sentiment analysis tools provide unreliable results when used out-of-the-box, since they are not designed to process SE datasets. The silver bullet for a successful application of sentiment analysis tools to SE datasets might be their customization to the specific usage context. We describe our experience in building a software library recommender exploiting developers' opinions mined from Stack Overflow. To reach our goal, we retrained---on a set of 40k manually labeled sentences/words extracted from Stack Overflow---a state-of-the-art sentiment analysis tool exploiting deep learning. Despite such an effort- and time-consuming training process, the results were negative. We changed our focus and performed a thorough investigation of the accuracy of commonly used tools to identify the sentiment of SE related texts. Meanwhile, we also studied the impact of different datasets on tool performance. Our results should warn the research community about the strong limitations of current sentiment analysis tools.
AbstractList Sentiment analysis has been applied to various software engineering (SE) tasks, such as evaluating app reviews or analyzing developers' emotions in commit messages. Studies indicate that sentiment analysis tools provide unreliable results when used out-of-the-box, since they are not designed to process SE datasets. The silver bullet for a successful application of sentiment analysis tools to SE datasets might be their customization to the specific usage context. We describe our experience in building a software library recommender exploiting crowdsourced opinions mined from Stack Overflow (e.g., what is the sentiment of developers about the usability of a library). To reach our goal, we retrained-on a set of 40k manually labeled sentences/words extracted from Stack Overflow-a state-of-the-art sentiment analysis tool exploiting deep learning. Despite such an effort- and time-consuming training process, the results were negative. We changed our focus and performed a thorough investigation of the accuracy of these tools on a variety of SE datasets. Our results should warn the research community about the strong limitations of current sentiment analysis tools.
Sentiment analysis has been applied to various software engineering (SE) tasks, such as evaluating app reviews or analyzing developers' emotions in commit messages. Studies indicate that sentiment analysis tools provide unreliable results when used out-of-the-box, since they are not designed to process SE datasets. The silver bullet for a successful application of sentiment analysis tools to SE datasets might be their customization to the specific usage context. We describe our experience in building a software library recommender exploiting developers' opinions mined from Stack Overflow. To reach our goal, we retrained---on a set of 40k manually labeled sentences/words extracted from Stack Overflow---a state-of-the-art sentiment analysis tool exploiting deep learning. Despite such an effort- and time-consuming training process, the results were negative. We changed our focus and performed a thorough investigation of the accuracy of commonly used tools to identify the sentiment of SE related texts. Meanwhile, we also studied the impact of different datasets on tool performance. Our results should warn the research community about the strong limitations of current sentiment analysis tools.
Author Lin, Bin
Oliveto, Rocco
Di Penta, Massimiliano
Lanza, Michele
Zampetti, Fiorella
Bavota, Gabriele
Author_xml – sequence: 1
  givenname: Bin
  surname: Lin
  fullname: Lin, Bin
  organization: Università della Svizzera italiana (USI), Switzerland
– sequence: 2
  givenname: Fiorella
  surname: Zampetti
  fullname: Zampetti, Fiorella
  organization: University of Sannio, Italy
– sequence: 3
  givenname: Gabriele
  surname: Bavota
  fullname: Bavota, Gabriele
  organization: Università della Svizzera italiana (USI), Switzerland
– sequence: 4
  givenname: Massimiliano
  surname: Di Penta
  fullname: Di Penta, Massimiliano
  organization: University of Sannio, Italy
– sequence: 5
  givenname: Michele
  surname: Lanza
  fullname: Lanza, Michele
  organization: Università della Svizzera italiana (USI), Switzerland
– sequence: 6
  givenname: Rocco
  surname: Oliveto
  fullname: Oliveto, Rocco
  organization: University of Molise, Italy
BookMark eNqNkDtPwzAUhc1LopTODCwZkVDC9dseUcVLqsQAzJbbXFeG1kF2JNR_T6CZmFjOGb6jM3xn5Dh1CQm5oNBQKuQNpwaolM1vW3lAZlabAQCXiht6SCYDNTVlTB79YadkVso7ADBlhKFqQq5fMPVxO0Tlk9_sSixV6HJVutB_-YwVpnVMiDmm9Tk5CX5TcDb2lLzd373OH-vF88PT_HZRe2ZlX7e21S0oywKFAIZqyWAljF15LUDYpQE0BiXTKlgpW6a518pzQIrAW8H5lFzufyMius8ctz7vnBGSg9IDvdpTv9q6Zdd9FEfB_Zhxoxk3mhmmzT-nbpkjBv4N9IVeLg
CODEN IEEPAD
ContentType Conference Proceeding
Copyright 2018 ACM
Copyright_xml – notice: 2018 ACM
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1145/3180155.3180195
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList

Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9781450356381
1450356389
EISSN 1558-1225
EndPage 104
ExternalDocumentID 8453067
Genre orig-research
GroupedDBID 6IE
6IF
6IG
6IL
6IM
6IN
AAJGR
ABLEC
ABQGA
ACM
ADPZR
ALMA_UNASSIGNED_HOLDINGS
APO
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
GUFHI
IEGSK
IJVOP
LHSKQ
OCL
RIB
RIC
RIE
RIL
RIO
-~X
.4S
.DC
123
23M
29O
5VS
6IH
6IK
8US
AAWTH
ADZIZ
AFFNX
ARCSS
AVWKF
CHZPO
EDO
FEDTE
I-F
I07
IPLJI
M43
RNS
XOL
ID FETCH-LOGICAL-a295t-d9d7d0692f10f0817520c489ca74049b80e88e5276f955d273a76a30e1e03d433
IEDL.DBID RIE
ISBN 9781450356381
1450356389
ISICitedReferencesCount 129
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000454843300013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:08:00 EDT 2025
Fri Sep 13 11:04:35 EDT 2024
IsPeerReviewed false
IsScholarly true
Keywords sentiment analysis
software engineering
NLP
Language English
License Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org.
LinkModel DirectLink
MeetingName ICSE '18: 40th International Conference on Software Engineering
MergedId FETCHMERGED-LOGICAL-a295t-d9d7d0692f10f0817520c489ca74049b80e88e5276f955d273a76a30e1e03d433
PageCount 11
ParticipantIDs acm_books_10_1145_3180155_3180195
ieee_primary_8453067
acm_books_10_1145_3180155_3180195_brief
PublicationCentury 2000
PublicationDate 20180527
2018-May
PublicationDateYYYYMMDD 2018-05-27
2018-05-01
PublicationDate_xml – month: 05
  year: 2018
  text: 20180527
  day: 27
PublicationDecade 2010
PublicationPlace New York, NY, USA
PublicationPlace_xml – name: New York, NY, USA
PublicationSeriesTitle ACM Conferences
PublicationTitle 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)
PublicationTitleAbbrev ICSE
PublicationYear 2018
Publisher ACM
Publisher_xml – name: ACM
SSID ssj0002684816
ssj0006499
Score 2.4995284
Snippet Sentiment analysis has been applied to various software engineering (SE) tasks, such as evaluating app reviews or analyzing developers' emotions in commit...
SourceID ieee
acm
SourceType Publisher
StartPage 94
SubjectTerms Information systems -- Information retrieval -- Retrieval tasks and goals -- Sentiment analysis
Motion pictures
NLP
Sentiment analysis
Software
Software engineering
Task analysis
Training
Subtitle how far can we go?
Title Sentiment analysis for software engineering
URI https://ieeexplore.ieee.org/document/8453067
WOSCitedRecordID wos000454843300013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ1LS8NAEMeXtnjwVLUV64sVBC-mzWM3u-vFQ7H2VAoq9hY2uxPowVTS1n59Z5O0KgjiKQ9CCJPNzm82M_Mn5FoYi9zAuWe1n2GAAoGXasDDKGOhhQihwJRiE2IykbOZmjbI7a4WBgDK5DPou93yX75dmLVbKhtIxh3hNklTiLiq1dqtp7iuJdKhTj0Lx4jydSufgPEBDl1HB_1y68Qkmtq8_VBUKR3KqP2_Rzkg3a_KPDrd-ZxD0oD8iLS30gy0_lI7yOAuC8jdg27bjlDEU_qEs-5GF0C_9SG8o-PFho50QYc6p69AHxf3XfIyengejr1aK8HToeIrzyorrB-rMAv8DN284KFvmFRGC4ZBQCp9kBJ4KOJMcW4RWrSIdYTvBvzIsig6Jq18kcMJoaFiGsMIoa3IGNKGTHWQxtbBSGykkj1yhbZLXBCwTKq6Zp7U9k1q-_bIzZ_XJGkxh6xHOs66yXvVXCOpDXv6--kzso_EIquMw3PSWhVruCB75mM1XxaX5Yj4BJv6rdg
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlZ1LS8NAEMeHtgp6qtqK9bmC4MW0eexmN148iLViLQUr9hY22Q30YCppa7--s0laFQTxlAchhMlm5zebmfkDXPBYITcwZilpJxigaMeKpMZDL6Gu0h5CQZyLTfDBQIzHwbACV-taGK11nnym22Y3_5evpvHCLJV1BGWGcKuwwSh17aJaa72iYvqWCAM75TzsI8yXzXwcyjo4eA0ftPOtkZOoyvjth6ZK7lK69f89zA40v2rzyHDtdXahotM9qK_EGUj5rTaQwk0ekLkHWTUeIQio5Bnn3aXMNPnWifCa9KZL0pUZuZUpedXkfnrThJfu3ei2Z5VqCZZ0Aza3VKC4sv3ATRw7QUfPmWvHVASx5BTDgEjYWgjNXO4nAWMKsUVyX3r4drTtKep5-1BLp6k-AOIGVGIgwaXiCUXeEJF0Il8ZHPFjEYgWnKPtQhMGzMKispmFpX3D0r4tuPzzmjDKJjppQcNYN3wv2muEpWEPfz99Blu90VM_7D8MHo9gG_lFFPmHx1CbZwt9Apvxx3wyy07z0fEJ3eGxHw
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2018+IEEE%2FACM+40th+International+Conference+on+Software+Engineering+%28ICSE%29&rft.atitle=Sentiment+Analysis+for+Software+Engineering%3A+How+Far+Can+We+Go%3F&rft.au=Lin%2C+Bin&rft.au=Zampetti%2C+Fiorella&rft.au=Bavota%2C+Gabriele&rft.au=Di+Penta%2C+Massimiliano&rft.date=2018-05-01&rft.pub=ACM&rft.eissn=1558-1225&rft.spage=94&rft.epage=104&rft_id=info:doi/10.1145%2F3180155.3180195&rft.externalDocID=8453067
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9781450356381/lc.gif&client=summon&freeimage=true
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9781450356381/mc.gif&client=summon&freeimage=true
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9781450356381/sc.gif&client=summon&freeimage=true