Validating AI-Generated Code with Live Programming

AI-powered programming assistants are increasingly gaining popularity, with GitHub Copilot alone used by over a million developers worldwide. These tools are far from perfect, however, producing code suggestions that may be incorrect in subtle ways. As a result, developers face a new challenge: vali...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org
Main Authors: Ferdowsi, Kasra, Huang, Ruanqianqian, James, Michael B, Polikarpova, Nadia, Lerner, Sorin
Format: Paper
Language:English
Published: Ithaca Cornell University Library, arXiv.org 23.02.2024
Subjects:
Interaction models
Programming
ISSN:2331-8422
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract AI-powered programming assistants are increasingly gaining popularity, with GitHub Copilot alone used by over a million developers worldwide. These tools are far from perfect, however, producing code suggestions that may be incorrect in subtle ways. As a result, developers face a new challenge: validating AI's suggestions. This paper explores whether Live Programming (LP), a continuous display of a program's runtime values, can help address this challenge. To answer this question, we built a Python editor that combines an AI-powered programming assistant with an existing LP environment. Using this environment in a between-subjects study (N=17), we found that by lowering the cost of validation by execution, LP can mitigate over- and under-reliance on AI-generated programs and reduce the cognitive load of validation for certain types of tasks.
AbstractList AI-powered programming assistants are increasingly gaining popularity, with GitHub Copilot alone used by over a million developers worldwide. These tools are far from perfect, however, producing code suggestions that may be incorrect in subtle ways. As a result, developers face a new challenge: validating AI's suggestions. This paper explores whether Live Programming (LP), a continuous display of a program's runtime values, can help address this challenge. To answer this question, we built a Python editor that combines an AI-powered programming assistant with an existing LP environment. Using this environment in a between-subjects study (N=17), we found that by lowering the cost of validation by execution, LP can mitigate over- and under-reliance on AI-generated programs and reduce the cognitive load of validation for certain types of tasks.
Author Polikarpova, Nadia
James, Michael B
Ferdowsi, Kasra
Huang, Ruanqianqian
Lerner, Sorin
Author_xml – sequence: 1
  givenname: Kasra
  surname: Ferdowsi
  fullname: Ferdowsi, Kasra
– sequence: 2
  givenname: Ruanqianqian
  surname: Huang
  fullname: Huang, Ruanqianqian
– sequence: 3
  givenname: Michael
  surname: James
  middlename: B
  fullname: James, Michael B
– sequence: 4
  givenname: Nadia
  surname: Polikarpova
  fullname: Polikarpova, Nadia
– sequence: 5
  givenname: Sorin
  surname: Lerner
  fullname: Lerner, Sorin
BookMark eNotjUFLw0AQRhdRsLb9Ad4CnhMnM9ns7rEEbQsBPZRey6Q7qSltopu0-vMN6Ok7vMf7HtRt27Wi1GMKSWa1hmcOP801QYI8Aaez9EZNkCiNbYZ4r-Z9fwQAzA1qTROFWz41noemPUSLdbyUVgIP4qOi8xJ9N8NHVDZXid5Ddwh8Po_eTN3VfOpl_r9TtXl92RSruHxbrotFGbNGiDOqIQMjla-dYwRBo0eAxIDGWL831si-YkcGXeW1RamcFfI-ZcsoNFVPf9nP0H1dpB92x-4S2vFxhxYN5UAO6Bf0H0as
ContentType Paper
Copyright 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID 8FE
8FG
ABJCF
ABUWG
AFKRA
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
HCIFZ
L6V
M7S
PHGZM
PHGZT
PIMPY
PKEHL
PQEST
PQGLB
PQQKQ
PQUKI
PRINS
PTHSS
DOI 10.48550/arxiv.2306.09541
DatabaseName ProQuest SciTech Collection
ProQuest Technology Collection
Materials Science & Engineering Collection
ProQuest Central (Alumni)
ProQuest Central UK/Ireland
ProQuest Central Essentials - QC
ProQuest Central
Technology Collection
ProQuest One Community College
ProQuest Central Korea
SciTech Premium Collection
ProQuest Engineering Collection
Engineering Database
ProQuest Central Premium
ProQuest One Academic (New)
Publicly Available Content Database
ProQuest One Academic Middle East (New)
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Applied & Life Sciences
ProQuest One Academic (retired)
ProQuest One Academic UKI Edition
ProQuest Central China
Engineering Collection
DatabaseTitle Publicly Available Content Database
Engineering Database
Technology Collection
ProQuest One Academic Middle East (New)
ProQuest Central Essentials
ProQuest One Academic Eastern Edition
ProQuest Central (Alumni Edition)
SciTech Premium Collection
ProQuest One Community College
ProQuest Technology Collection
ProQuest SciTech Collection
ProQuest Central China
ProQuest Central
ProQuest One Applied & Life Sciences
ProQuest Engineering Collection
ProQuest One Academic UKI Edition
ProQuest Central Korea
Materials Science & Engineering Collection
ProQuest Central (New)
ProQuest One Academic
ProQuest One Academic (New)
Engineering Collection
DatabaseTitleList Publicly Available Content Database
Database_xml – sequence: 1
  dbid: PIMPY
  name: ProQuest Publicly Available Content Database
  url: http://search.proquest.com/publiccontent
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Physics
EISSN 2331-8422
Genre Working Paper/Pre-Print
GroupedDBID 8FE
8FG
ABJCF
ABUWG
AFKRA
ALMA_UNASSIGNED_HOLDINGS
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
FRJ
HCIFZ
L6V
M7S
M~E
PHGZM
PHGZT
PIMPY
PKEHL
PQEST
PQGLB
PQQKQ
PQUKI
PRINS
PTHSS
ID FETCH-LOGICAL-a520-43f0407ebdf99a20e275a5223a02778dc787ecba93729bd582eb98e3dd1a8a2e3
IEDL.DBID M7S
IngestDate Mon Jun 30 09:11:17 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a520-43f0407ebdf99a20e275a5223a02778dc787ecba93729bd582eb98e3dd1a8a2e3
Notes SourceType-Working Papers-1
ObjectType-Working Paper/Pre-Print-1
content type line 50
OpenAccessLink https://www.proquest.com/docview/2827360390?pq-origsite=%requestingapplication%
PQID 2827360390
PQPubID 2050157
ParticipantIDs proquest_journals_2827360390
PublicationCentury 2000
PublicationDate 20240223
PublicationDateYYYYMMDD 2024-02-23
PublicationDate_xml – month: 02
  year: 2024
  text: 20240223
  day: 23
PublicationDecade 2020
PublicationPlace Ithaca
PublicationPlace_xml – name: Ithaca
PublicationTitle arXiv.org
PublicationYear 2024
Publisher Cornell University Library, arXiv.org
Publisher_xml – name: Cornell University Library, arXiv.org
SSID ssj0002672553
Score 1.861206
SecondaryResourceType preprint
Snippet AI-powered programming assistants are increasingly gaining popularity, with GitHub Copilot alone used by over a million developers worldwide. These tools are...
SourceID proquest
SourceType Aggregation Database
SubjectTerms Interaction models
Programming
Title Validating AI-Generated Code with Live Programming
URI https://www.proquest.com/docview/2827360390
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV07T8MwED5BCxITb_EoVQbW0MR285gQVK2oBFUEFSpTZccX1IG0JKXi53N2UhiQWBgjR6ckti_ffff5DuBS-SnTASE3mmHtCsWlK32JrqcwE1qzVARVs4lwNIomkzipCbeyllWufaJ11HqeGo68Q6FByAOPQvTrxbtrukaZ7GrdQmMTmqZKgm-le0_fHAsLQkLMvEpm2tJdHVl8zlZG_RxcEbgQ_i8XbP8rg93_PtEeNBO5wGIfNjA_gG2r50zLQ2DPhK_NyYX81bkZulVxaQKXTm-u0THkq3NPfs5JKnnWG913BONBf9y7c-v2CK7sUswneEYbMESlsziWzEMWdmmAcWnSspFOaStiqmRsEnNKdyOGKo6Qa-3LSDLkx9DI5zmegCNYJg0_nAkbjkRKoOacbDKyj553Cq31F5jWS7yc_rz-2d_D57BDloQ9B85b0FgWH3gBW-lqOSuLNjRv-6PksW1njq6S4UPy8gW6hKJZ
linkProvider ProQuest
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1LT8JAEJ4gaPTkOz5Qe9Bjpd1d-jgYY1ACAQmJxHAju92t4WDBFlF_lP_R2S3Vg4k3Dp63mTQz0-l8M9_sAJwLNyLSw8wNLSxtJii3ucuV7QgVMylJxLx82YTf6wXDYdgvwWcxC6NplUVMNIFaTiJdI68hNPCp5yBEv56-2HprlO6uFis0crfoqI83hGzZVfsW7XtBSPNu0GjZi60CNq8jVGI0Rr_1lZBxGHLiKOLX8YBQrruZgYzQg1UkeKj7WULWA6JEGCgqpcsDThRFsStQwSyChIYp-PBd0iGejwk6zXun5qawGk_fx3NNtvYuMZdh7q-Ib35jzc1_poAtqPT5VKXbUFLJDqwZtmqU7QJ5RPSg5zKSJ-umbedXZ2PqbDUmUlm6tGx1MYpb_Zx89ozP7cFgGW-5D-VkkqgDsBiJua5-x8yArUAwJSlFmQTlK8c5hGqh8NHiA85GP9o--vv4DNZbg_vuqNvudY5hA6UyM_FOq1Cepa_qBFaj-WycpafGWSwYLdk2XwKX-uE
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Validating+AI-Generated+Code+with+Live+Programming&rft.jtitle=arXiv.org&rft.au=Ferdowsi%2C+Kasra&rft.au=Huang%2C+Ruanqianqian&rft.au=James%2C+Michael+B&rft.au=Polikarpova%2C+Nadia&rft.date=2024-02-23&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.2306.09541