How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering

Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. I...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings / International Conference on Software Engineering pp. 2270 - 2282
Main Authors: Choudhuri, Rudrajit, Liu, Dylan, Steinmacher, Igor, Gerosa, Marco, Sarma, Anita
Format: Conference Proceeding
Language:English
Published: ACM 14.04.2024
Subjects:
Chatbots
ChatGPT
Empirical Study
Generative AI
Navigation
Productivity
Software Engineering
Task analysis
Uncertainty
ISSN:1558-1225
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
AbstractList Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
Author Choudhuri, Rudrajit
Gerosa, Marco
Steinmacher, Igor
Sarma, Anita
Liu, Dylan
Author_xml – sequence: 1
  givenname: Rudrajit
  surname: Choudhuri
  fullname: Choudhuri, Rudrajit
  email: choudhru@oregonstate.edu
  organization: Oregon State University,Corvallis,OR,USA
– sequence: 2
  givenname: Dylan
  surname: Liu
  fullname: Liu, Dylan
  email: liudy@oregonstate.edu
  organization: Oregon State University,Corvallis,OR,USA
– sequence: 3
  givenname: Igor
  surname: Steinmacher
  fullname: Steinmacher, Igor
  email: igor.steinmacher@nau.edu
  organization: Northern Arizona University,Flagstaff,AZ,USA
– sequence: 4
  givenname: Marco
  surname: Gerosa
  fullname: Gerosa, Marco
  email: marco.gerosa@nau.edu
  organization: Northern Arizona University,Flagstaff,AZ,USA
– sequence: 5
  givenname: Anita
  surname: Sarma
  fullname: Sarma, Anita
  email: anita.sarma@oregonstate.edu
  organization: Oregon State University,Corvallis,OR,USA
BookMark eNotj81KAzEYRaMoWOus3bjIC0zNl59JspKh9A8GXFgRVyXTfGkDNlMy1eLbO6Kry71wDtxbcpW6hITcA5sASPUolNWKiYmohOUMLkhhtTWSMc04aHlJRqCUKYFzdUOKvo8tU1IoXUkxIu_L7kznLtM6I33DJ7reI13n-Hk47nvqkv8t7qOnXaALTJjdKX4hrVc0JtqgyymmHX3pwunsBsMs7WJCzMN4R67DAGLxn2PyOp-tp8uyeV6spnVTOi44lL5VSmqsALnyMtgtbNGw1ntvpLBVqJgMxnuHADzYSoNx2OqtEcpg8EGKMXn480ZE3BxzPLj8vYHho2WgxA_XJ1N1
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
ESBDL
RIE
RIO
DOI 10.1145/3597503.3639201
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Xplore Open Access Journals
IEEE Xplore
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798400702174
EISSN 1558-1225
EndPage 2282
ExternalDocumentID 10549015
Genre orig-research
GrantInformation_xml – fundername: National Science Foundation
  grantid: 2235601,2236198,2247929,2303042,2303043
  funderid: 10.13039/100000001
GroupedDBID -~X
.4S
.DC
29O
5VS
6IE
6IF
6IH
6IK
6IL
6IM
6IN
8US
AAJGR
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
ARCSS
AVWKF
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
EDO
ESBDL
FEDTE
I-F
IEGSK
IJVOP
IPLJI
M43
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-a2321-db5547e61e25d4f9c1ce80bddd84396f604f8ddae112f96718aeb7c8358efdf43
IEDL.DBID RIE
IngestDate Wed Aug 27 01:53:12 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a2321-db5547e61e25d4f9c1ce80bddd84396f604f8ddae112f96718aeb7c8358efdf43
OpenAccessLink https://ieeexplore.ieee.org/document/10549015
PageCount 13
ParticipantIDs ieee_primary_10549015
PublicationCentury 2000
PublicationDate 2024-April-14
PublicationDateYYYYMMDD 2024-04-14
PublicationDate_xml – month: 04
  year: 2024
  text: 2024-April-14
  day: 14
PublicationDecade 2020
PublicationTitle Proceedings / International Conference on Software Engineering
PublicationTitleAbbrev ICSE
PublicationYear 2024
Publisher ACM
Publisher_xml – name: ACM
SSID ssib054357643
ssib055306466
ssj0006499
Score 2.4257932
Snippet Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work....
SourceID ieee
SourceType Publisher
StartPage 2270
SubjectTerms Chatbots
ChatGPT
Empirical Study
Generative AI
Navigation
Productivity
Software Engineering
Task analysis
Uncertainty
Title How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
URI https://ieeexplore.ieee.org/document/10549015
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELWgYmDiq4hveWBNcWLXcSZUIaoioaoSRZSpin1n1CVBaUv_Puc0hS4MbHGGyPKd773Eee8Yu6WwEtHFOLIIaaQkiChLUESYKeO9jr0xtVD4OR0OzWSSjRqxeq2FQcT65zPshMv6LB9KtwyfymiH09uMCJLy3TTVa7HWJnm6hPvplrdUaIejVeAqTVnWxO0bb59Yde8kMemukB1JEJ2EljBbzVVqbOkf_HNWh6z9q9Ljox_8OWI7WByzg02bBt7s2hP2PihXvJ9XvFchf8N7TqnBx9UsBHLO8wLCgLKQl56vXahDCeS9Jz4reOO_-sFfqF6vcnrCloNhm732H8cPg6jpqBDlxJziCCyxhxR1jEkXlM9c7NAICwCGiIn2WihvAHIkFuYzTbiVo00dsTSDHrySp6xVlAWeMe6k9Vagct6iUjaz0uUIICCRUroMz1k7LNX0c22aMd2s0sUf9y_ZfkJ8IRzUxOqKtRbVEq_ZnvtazObVTR3qbySxp-s
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELUQIMFUPor4xgOrixM7iTOhClG1olSVKKJMVWyfUZcEpS39-5zTFLowsMUZIst3vvcS570j5BbDikQXAqbBJkwKy1kaAmeQSuVcHDilKqFwPxkM1HicDmuxeqWFAYDq5zNo-cvqLN8WZuE_leEOx7cZ7iXlO5GUIV_JtdbpEyHyJxvuUr4hTiw9W6kLc4zsvnb3CWR0J5BLR1y0BIJ06JvCbLRXqdCl0_jnvA5I81enR4c_CHRItiA_Io11owZa79tj8t4tlrSTlbRdAn2De4rJQUfl1IdyRrPc-gHmIS0cXflQ-yJI2z06zWntwPpBX7BiLzN8woaHYZO8dh5HD11W91RgGXKngFmN_CGBOIAwstKlJjCguLbWKqQmsYu5dMraDJCHuTRG5MpAJwZ5mgJnnRQnZDsvcjgl1AjtNAdpnAYpdaqFycBabkMhhEnhjDT9Uk0-V7YZk_Uqnf9x_4bsdUfP_Um_N3i6IPshsgd_bBPIS7I9LxdwRXbN13w6K6-rsH8DELmrMg
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Conference+on+Software+Engineering&rft.atitle=How+Far+Are+We%3F+The+Triumphs+and+Trials+of+Generative+AI+in+Learning+Software+Engineering&rft.au=Choudhuri%2C+Rudrajit&rft.au=Liu%2C+Dylan&rft.au=Steinmacher%2C+Igor&rft.au=Gerosa%2C+Marco&rft.date=2024-04-14&rft.pub=ACM&rft.eissn=1558-1225&rft.spage=2270&rft.epage=2282&rft_id=info:doi/10.1145%2F3597503.3639201&rft.externalDocID=10549015