Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Several advances in deep learning have been successfully applied to the software development process. Of recent interest is the use of neural language models to build tools, such as Copilot, that assist in writing code. In this paper we perform a comparative empirical analysis of Copilot-generated c...
Saved in:
| Published in: | Empirical software engineering : an international journal Vol. 28; no. 6; p. 129 |
|---|---|
| Main Authors: | , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
Springer US
01.11.2023
Springer Nature B.V |
| Subjects: | |
| ISSN: | 1382-3256, 1573-7616 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Several advances in deep learning have been successfully applied to the software development process. Of recent interest is the use of neural language models to build tools, such as Copilot, that assist in writing code. In this paper we perform a comparative empirical analysis of Copilot-generated code from a security perspective. The aim of this study is to determine if Copilot is as bad as human developers. We investigate whether Copilot is just as likely to introduce the same software vulnerabilities as human developers. Using a dataset of C/C++ vulnerabilities, we prompt Copilot to generate suggestions in scenarios that led to the introduction of vulnerabilities by human developers. The suggestions are inspected and categorized in a 2-stage process based on whether the original vulnerability or fix is reintroduced. We find that Copilot replicates the original vulnerable code about 33% of the time while replicating the fixed code at a 25% rate. However this behaviour is not consistent: Copilot is more likely to introduce some types of vulnerabilities than others and is also more likely to generate vulnerable code in response to prompts that correspond to older vulnerabilities. Overall, given that in a significant number of cases it did not replicate the vulnerabilities previously introduced by human developers, we conclude that Copilot, despite performing differently across various vulnerability types, is not as bad as human developers at introducing vulnerabilities in code. |
|---|---|
| AbstractList | Several advances in deep learning have been successfully applied to the software development process. Of recent interest is the use of neural language models to build tools, such as Copilot, that assist in writing code. In this paper we perform a comparative empirical analysis of Copilot-generated code from a security perspective. The aim of this study is to determine if Copilot is as bad as human developers. We investigate whether Copilot is just as likely to introduce the same software vulnerabilities as human developers. Using a dataset of C/C++ vulnerabilities, we prompt Copilot to generate suggestions in scenarios that led to the introduction of vulnerabilities by human developers. The suggestions are inspected and categorized in a 2-stage process based on whether the original vulnerability or fix is reintroduced. We find that Copilot replicates the original vulnerable code about 33% of the time while replicating the fixed code at a 25% rate. However this behaviour is not consistent: Copilot is more likely to introduce some types of vulnerabilities than others and is also more likely to generate vulnerable code in response to prompts that correspond to older vulnerabilities. Overall, given that in a significant number of cases it did not replicate the vulnerabilities previously introduced by human developers, we conclude that Copilot, despite performing differently across various vulnerability types, is not as bad as human developers at introducing vulnerabilities in code. |
| ArticleNumber | 129 |
| Author | Nagappan, Meiyappan Asokan, N. Asare, Owura |
| Author_xml | – sequence: 1 givenname: Owura orcidid: 0000-0002-6755-605X surname: Asare fullname: Asare, Owura email: oasare@uwaterloo.ca organization: Cheriton School of Computer Science, University of Waterloo – sequence: 2 givenname: Meiyappan surname: Nagappan fullname: Nagappan, Meiyappan organization: Cheriton School of Computer Science, University of Waterloo – sequence: 3 givenname: N. surname: Asokan fullname: Asokan, N. organization: Cheriton School of Computer Science, University of Waterloo |
| BookMark | eNp9kM1KAzEUhYNUsFZfwFXAdfQmmcnMrESKtoWiG12HTJKpKdNJTTKCO1_D1_NJnFpBcNHVuXDPd3_OKRp1vrMIXVC4ogDFdaQgREaAcUKBl0DoERrTvOCkEFSMhpqXjHCWixN0GuMaAKoiy8foYRHxzKV5X399fEY89VvX-oRVxLUyO3npN6qLWCXsuhS86bXrVvitbzsbVO1al5yNQw9rb-zNGTpuVBvt-a9O0PP93dN0TpaPs8X0dkk0p1Ui2lbMcFbzXDHKAbJaV0YpKK02kGU55Ywb3ahSGcUYM1TnBnJoNBNlo23DJ-hyP3cb_GtvY5Jr34duWClZKQqRCc7p4Cr3Lh18jME2UrukkvPDJ8q1koLcpSf36ckhPfmTntyh7B-6DW6jwvthiO-hOJi7lQ1_Vx2gvgFyQYS8 |
| CitedBy_id | crossref_primary_10_3390_info16050401 crossref_primary_10_1002_spe_3432 crossref_primary_10_3390_a17070287 crossref_primary_10_1007_s10664_025_10614_4 crossref_primary_10_1007_s10664_023_10380_1 crossref_primary_10_1145_3709354 crossref_primary_10_1145_3709355 crossref_primary_10_1145_3709353 crossref_primary_10_1145_3708533 crossref_primary_10_32604_cmc_2025_062552 crossref_primary_10_1016_j_csi_2024_103938 crossref_primary_10_1145_3715108 crossref_primary_10_1016_j_jss_2024_112204 crossref_primary_10_1145_3708519 crossref_primary_10_1007_s10664_025_10658_6 crossref_primary_10_1016_j_infsof_2024_107610 crossref_primary_10_1016_j_infsof_2025_107737 crossref_primary_10_1145_3709360 crossref_primary_10_1016_j_jss_2024_112330 crossref_primary_10_3390_math12050629 crossref_primary_10_1002_spe_70005 crossref_primary_10_1016_j_jss_2024_112135 crossref_primary_10_1109_ACCESS_2024_3403858 crossref_primary_10_1109_TDSC_2025_3535218 crossref_primary_10_1145_3660807 crossref_primary_10_1145_3714464 crossref_primary_10_1007_s10515_024_00451_y crossref_primary_10_1109_TAI_2025_3527398 crossref_primary_10_1016_j_jss_2025_112519 |
| Cites_doi | 10.1109/TNNLS.2020.3019893 10.1162/tacl_a_00105 10.1145/3586030 10.1162/neco.1997.9.8.1735 10.1162/neco.10.1145/3383458 10.1145/3510454.3516866 10.3115/v1/D14-1082 10.18653/v1/W16-0106 10.1145/3106237.3106290 10.1109/ICSE.2012.6227135 10.1007/s10664-023-10380-1 10.1145/3524842.3528440 10.18653/v1/2020.findings-emnlp.139 10.1126/science.abq1158 10.1145/3520312.3534864 10.1145/3379597.3387501 10.1109/SP46214.2022.9833571 10.1145/3520312.3534862 10.1145/3524459.3527351 10.1145/2594291.2594321 10.1109/SP46215.2023.10179324 10.18653/v1/P17-1041 10.1145/3512290.3528700 10.1145/3368089.3417058 10.1145/3524842.3528470 10.1109/ICSE43902.2021.00107 10.1145/3491101.3519665 10.1145/2983990.2984041 10.1109/TSE.2021.3087402 |
| ContentType | Journal Article |
| Copyright | The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. |
| Copyright_xml | – notice: The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. |
| DBID | AAYXX CITATION 7SC 8FD 8FE 8FG ABJCF AFKRA ARAPS BENPR BGLVJ CCPQU DWQXO HCIFZ JQ2 L6V L7M L~C L~D M7S P5Z P62 PHGZM PHGZT PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS S0W |
| DOI | 10.1007/s10664-023-10380-1 |
| DatabaseName | CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection ProQuest Materials Science & Engineering ProQuest Central UK/Ireland Advanced Technologies & Computer Science Collection ProQuest Central Technology Collection ProQuest One ProQuest Central SciTech Premium Collection ProQuest Computer Science Collection ProQuest Engineering Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional Engineering Database ProQuest advanced technologies & aerospace journals ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Premium ProQuest One Academic ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China Engineering collection DELNET Engineering & Technology Collection |
| DatabaseTitle | CrossRef Technology Collection Technology Research Database Computer and Information Systems Abstracts – Academic ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Computer Science Collection Computer and Information Systems Abstracts SciTech Premium Collection ProQuest One Community College ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences ProQuest Engineering Collection ProQuest Central Korea ProQuest Central (New) Advanced Technologies Database with Aerospace Engineering Collection Advanced Technologies & Aerospace Collection Engineering Database ProQuest One Academic Eastern Edition ProQuest Technology Collection ProQuest SciTech Collection Computer and Information Systems Abstracts Professional Advanced Technologies & Aerospace Database ProQuest One Academic UKI Edition ProQuest DELNET Engineering and Technology Collection Materials Science & Engineering Collection ProQuest One Academic ProQuest One Academic (New) |
| DatabaseTitleList | Technology Collection |
| Database_xml | – sequence: 1 dbid: BENPR name: ProQuest Central url: https://www.proquest.com/central sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1573-7616 |
| ExternalDocumentID | 10_1007_s10664_023_10380_1 |
| GrantInformation_xml | – fundername: WHJIL |
| GroupedDBID | -4Z -59 -5G -BR -EM -Y2 -~C .86 .DC .VR 06D 0R~ 0VY 199 1N0 1SB 2.D 203 28- 29G 2J2 2JN 2JY 2KG 2LR 2P1 2VQ 2~H 30V 4.4 406 408 409 40D 40E 5GY 5QI 5VS 67Z 6NX 78A 8FE 8FG 8TC 8UJ 95- 95. 95~ 96X AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AAOBN AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYOK AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDZT ABECU ABFTD ABFTV ABHLI ABHQN ABJCF ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABULA ABWNU ABXPI ACAOD ACBXY ACDTI ACGFS ACHSB ACHXU ACIWK ACKNC ACMDZ ACMLO ACOKC ACOMO ACPIV ACSNA ACZOJ ADHHG ADHIR ADIMF ADINQ ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEBTG AEFIE AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AENEX AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFEXP AFGCZ AFKRA AFLOW AFQWF AFWTZ AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARAPS ARMRJ ASPBG AVWKF AXYYD AYJHY AZFZN B-. BA0 BBWZM BDATZ BENPR BGLVJ BGNMA BSONS CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 EBLON EBS EIOEI EJD ESBYG FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNWQR GQ6 GQ7 GQ8 GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ KDC KOV KOW L6V LAK LLZTM M4Y M7S MA- N2Q NB0 NDZJH NPVJJ NQJWS NU0 O9- O93 O9G O9I O9J OAM P19 P62 P9O PF0 PT4 PT5 PTHSS Q2X QOK QOS R4E R89 R9I RHV RNI RNS ROL RPX RSV RZC RZE RZK S0W S16 S1Z S26 S27 S28 S3B SAP SCJ SCLPG SCO SDH SDM SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 T16 TSG TSK TSV TUC U2A UG4 UOJIU UTJUX UZXMN VC2 VFIZW W23 W48 WK8 YLTOR Z45 Z7R Z7S Z7V Z7X Z7Z Z81 Z83 Z86 Z88 Z8M Z8N Z8P Z8R Z8T Z8U Z8W Z92 ZMTXR ~EX AAPKM AAYXX ABBRH ABDBE ABFSG ABRTQ ACSTC ADHKG AEZWR AFDZB AFFHD AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION PHGZM PHGZT PQGLB 7SC 8FD DWQXO JQ2 L7M L~C L~D PKEHL PQEST PQQKQ PQUKI PRINS |
| ID | FETCH-LOGICAL-c319t-ce92d32b35a213004bc9daa08ecd04451323dcfa8ada222d1c5d050fc268fcef3 |
| IEDL.DBID | M7S |
| ISICitedReferencesCount | 51 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001076229600001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1382-3256 |
| IngestDate | Tue Dec 02 15:58:16 EST 2025 Tue Nov 18 22:30:01 EST 2025 Sat Nov 29 05:37:47 EST 2025 Fri Feb 21 02:40:56 EST 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 6 |
| Keywords | code security copilot software engineering language models |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c319t-ce92d32b35a213004bc9daa08ecd04451323dcfa8ada222d1c5d050fc268fcef3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0002-6755-605X |
| PQID | 2867646331 |
| PQPubID | 326341 |
| ParticipantIDs | proquest_journals_2867646331 crossref_citationtrail_10_1007_s10664_023_10380_1 crossref_primary_10_1007_s10664_023_10380_1 springer_journals_10_1007_s10664_023_10380_1 |
| PublicationCentury | 2000 |
| PublicationDate | 2023-11-01 |
| PublicationDateYYYYMMDD | 2023-11-01 |
| PublicationDate_xml | – month: 11 year: 2023 text: 2023-11-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York – name: Dordrecht |
| PublicationSubtitle | An International Journal |
| PublicationTitle | Empirical software engineering : an international journal |
| PublicationTitleAbbrev | Empir Software Eng |
| PublicationYear | 2023 |
| Publisher | Springer US Springer Nature B.V |
| Publisher_xml | – name: Springer US – name: Springer Nature B.V |
| References | Brown, T.B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss,G. Krueger, T. Henighan, R. Child, A. Ramesh, D.M. Ziegler, J. Wu, C. Winter,C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark,C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. 2020,July.Language Models are Few-Shot Learners. arXiv:2005.14165 [cs] Asare, O., M. Nagappan, and N. Asokan. 2022. Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code? _eprint: 2204.04741 Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł Kaiser, and I. Polosukhin 2017.Attention is All You Need.In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA,pp.6000–6010. Curran Associates Inc.event-place: Long Beach, California, USA Tabnine. 2022.Code Faster with AI Completions Hindle, A., E.T. Barr, Z. Su, M. Gabel, and P. Devanbu 2012.On the Naturalness of Software.In Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, pp.837–847. IEEE Press. event-place: Zurich, Switzerland Hardmeier, C. 2016, December.A Neural Model for Part-of-Speech Tagging in Historical Texts.In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, pp.922–931. The COLING 2016 Organizing Committee Hellendoorn, V.J. and P. Devanbu 2017, August.Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, Paderborn Germany, pp.763–773. ACM Xu, F.F., U. Alon, G. Neubig, and V.J. Hellendoorn 2022, June. A systematic evaluation of large language models of code.In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego CA USA, pp.1–10. ACM Dohmke, T. 2022, June.GitHub Copilot is generally available to all developers Bielik, P., V. Raychev, and M. Vechev 2016.PHOG: probabilistic model for code. In International Conference on Machine Learning, pp. 2933–2942. PMLR Ziegler, A., E. Kalliamvakou, X.A. Li, A. Rice, D. Rifkin, S. Simister,G. Sittampalam, and E. Aftandilian 2022, June.Productivity assessment of neural code completion.In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego CA USA, pp.21–29. ACM Prenner, J., H. Babii, and R. Robbes 2022, May. Can OpenAI’s Codex Fix Bugs?: An evaluation on QuixBugs.2022 IEEE/ACM International Workshop on Automated Program Repair (APR), Los Alamitos, CA, USA, pp.69–75. IEEE Computer Society Nijkamp, E., B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2022.CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.arXiv preprint Yan, W. and Y. Li. 2022, April.WhyGen: Explaining ML-powered Code Generation by Referring to Training Examples. arXiv:2204.07940 Svyatkovskiy, A., S.K. Deng, S. Fu, and N. Sundaresan 2020, November.IntelliCode compose: code generation using transformer.In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event USA, pp.1433–1443. ACM HochreiterSSchmidhuberJNovember. Long Short-Term MemoryNeural Computation1997981735178010.1162/neco.1997.9.8.1735 Zhang, J., J. Cambronero, S. Gulwani, V. Le, R. Piskac, G. Soares, and G. Verbruggen. 2022.Repairing Bugs in Python Assignments Using Large Language Models Feng, Z., D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu,D. Jiang, and M. Zhou. 2020, September.CodeBERT: A Pre-Trained Model for Programming and Natural Languages.arXiv:2002.08155 Galassi, A., M. Lippi, and P. Torroni. 2021, October. Natural Language Processing.IEEE Transactions on Neural Networks and Learning Systems 32(10): 4291–4308. https://doi.org/10.1109/TNNLS.2020.3019893 Vaithilingam, P., T. Zhang, and E.L. Glassman 2022, April. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models.In CHI Conference on Human Factors in Computing Systems Extended Abstracts, New Orleans LA USA, pp.1–7. ACM Le, T.H.M., H. Chen, and M.A. Babar. 2020, June.Deep Learning for Source Code Modeling and Generation:Models, Applications, and Challenges. ACM Comput. Surv. 53(3)https://doi.org/10.1162/neco.10.1145/3383458 Chen, D. and C. Manning 2014, October. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, pp. 740–750. Association for Computational Linguistics Chen, M., J. Tworek, H. Jun, Q. Yuan, H.P.d.O. Pinto, J. Kaplan, H. Edwards,Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov,H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov,A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F.P. Such,D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W.H. Guss,A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain,W. Saunders, C. Hesse, A.N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa,A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder,B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. 2021,July.Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs] Dakhel, A.M., V. Majdinasab, A. Nikanjam, F. Khomh, M.C. Desmarais, Z. Ming,and Jiang. 2022, June. GitHub Copilot AI pair programmer: Asset or Liability? arXiv:2206.15331 Lu, S., D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement,D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano,M. Gong, M. Zhou, N. Duan, N. Sundaresan, S.K. Deng, S. Fu, and S. Liu. 2021,March.CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. arXiv:2102.04664 ChakrabortySKrishnaRDingYRayBLearning Based Vulnerability Detection: Are We There Yet?IEEE Transactions on Software Engineering20224893280329610.1109/TSE.2021.3087402 GitHub Inc. 2019.CodeQL Devlin, J., M.W. Chang, K. Lee, and K. Toutanova. 2019, May. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 Jiang, N., T. Lutellier, and L. Tan 2021, May. CURE: Code-Aware Neural Machine Translation for Automatic Program Repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp.1161–1173.ISSN: 1558-1225 Barke, S., M.B. James, and N. Polikarpova. 2022, August. Grounded Copilot: How Programmers Interact with Code-Generating Models. arXiv:2206.15000 Nguyen, N. and S. Nadi 2022.Empirical Evaluation of GitHub Copilot’s Code Suggestions.In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), pp.1–5 Synopsys 2022.Source Security and Risk Analysis Report. Technical report, Synopsys Inc Ciniselli, M., L. Pascarella, and G. Bavota. 2022, April.To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? arXiv:2204.06894 Pearce, H., B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri 2022, May.Asleep at the Keyboard? Assessing the Security of GitHub Copilot Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp.754–768.ISSN: 2375-1207 Desai, A. and A. Deo. 2022. Introducing Amazon CodeWhisperer, the ML-powered coding companion Bengio, Y., R. Ducharme, and P. Vincent 2000.A Neural Probabilistic Language Model. In Advances in Neural Information Processing Systems,Volume 13. MIT Press Yin, P. and G. Neubig. 2017, April. A Syntactic Neural Model for General-Purpose Code Generation.arXiv:1704.01696 GitHub Inc. 2021.GitHub Copilot Your AI pair programmer Sobania, D., M. Briesch, and F. Rothlauf 2022, July.Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming.In Proceedings of the Genetic and Evolutionary Computation Conference, Boston Massachusetts, pp.1019–1027. ACM Fan, J., Y. Li, S. Wang, and T.N. Nguyen 2020, June.A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, Seoul Republic of Korea, pp. 508–512.ACM Pearce, H., B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt 2023, May.Examining Zero-Shot Vulnerability Repair with Large Language Models.In 2023 2023 IEEE Symposium on Security and Privacy(SP) (SP), Los Alamitos, CA, USA, pp.1–18. IEEE Computer Society Raychev, V., M. Vechev, and E. Yahav 2014, June.Code completion with statistical language models.In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, Edinburgh United Kingdom, pp.419–428. ACM ZhouJCaoYWangXLiPXuWDeep Recurrent Models with Fast-Forward Connections for Neural Machine TranslationTransactions of the Association for Computational Linguistics2016437138310.1162/tacl_a_00105 Li, Y., D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R Leblond, T. Eccles,J. Keeling, F. Gimeno, A.D. Lago, T. Hubert, P. Choy, C.d.M. d’Autume,I. Babuschkin, X. Chen, P.S. Huang, J. Welbl, S. Gowal, A. Cherepanov,J. Molloy, D.J. Mankowitz, E.S. Robson, P. Kohli, N. de Freitas,K. Kavukcuoglu, and O. Vinyals. 2022.Competition-Level Code Generation with AlphaCode Yin, J., X. Jiang, Z. Lu, L. Shang, H. Li, and X. Li 2016. Neural Generative Question Answering.In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, pp.2972–2978. AAAI Press.event-place: New York, New York, USA 10380_CR1 10380_CR41 10380_CR40 10380_CR19 S Chakraborty (10380_CR6) 2022; 48 10380_CR18 10380_CR17 10380_CR39 10380_CR9 10380_CR16 10380_CR38 10380_CR8 10380_CR15 10380_CR37 10380_CR7 10380_CR14 10380_CR36 10380_CR13 10380_CR35 10380_CR5 10380_CR12 10380_CR34 10380_CR4 10380_CR11 10380_CR33 10380_CR3 10380_CR10 10380_CR32 J Zhou (10380_CR44) 2016; 4 10380_CR2 10380_CR31 10380_CR30 S Hochreiter (10380_CR22) 1997; 9 10380_CR29 10380_CR28 10380_CR27 10380_CR26 10380_CR25 10380_CR24 10380_CR23 10380_CR45 10380_CR21 10380_CR43 10380_CR20 10380_CR42 |
| References_xml | – reference: Bielik, P., V. Raychev, and M. Vechev 2016.PHOG: probabilistic model for code. In International Conference on Machine Learning, pp. 2933–2942. PMLR – reference: Ziegler, A., E. Kalliamvakou, X.A. Li, A. Rice, D. Rifkin, S. Simister,G. Sittampalam, and E. Aftandilian 2022, June.Productivity assessment of neural code completion.In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego CA USA, pp.21–29. ACM – reference: ChakrabortySKrishnaRDingYRayBLearning Based Vulnerability Detection: Are We There Yet?IEEE Transactions on Software Engineering20224893280329610.1109/TSE.2021.3087402 – reference: Li, Y., D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R Leblond, T. Eccles,J. Keeling, F. Gimeno, A.D. Lago, T. Hubert, P. Choy, C.d.M. d’Autume,I. Babuschkin, X. Chen, P.S. Huang, J. Welbl, S. Gowal, A. Cherepanov,J. Molloy, D.J. Mankowitz, E.S. Robson, P. Kohli, N. de Freitas,K. Kavukcuoglu, and O. Vinyals. 2022.Competition-Level Code Generation with AlphaCode – reference: Nijkamp, E., B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2022.CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.arXiv preprint – reference: Pearce, H., B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt 2023, May.Examining Zero-Shot Vulnerability Repair with Large Language Models.In 2023 2023 IEEE Symposium on Security and Privacy(SP) (SP), Los Alamitos, CA, USA, pp.1–18. IEEE Computer Society – reference: Vaithilingam, P., T. Zhang, and E.L. Glassman 2022, April. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models.In CHI Conference on Human Factors in Computing Systems Extended Abstracts, New Orleans LA USA, pp.1–7. ACM – reference: Ciniselli, M., L. Pascarella, and G. Bavota. 2022, April.To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? arXiv:2204.06894 – reference: Desai, A. and A. Deo. 2022. Introducing Amazon CodeWhisperer, the ML-powered coding companion – reference: Yin, P. and G. Neubig. 2017, April. A Syntactic Neural Model for General-Purpose Code Generation.arXiv:1704.01696 – reference: Feng, Z., D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu,D. Jiang, and M. Zhou. 2020, September.CodeBERT: A Pre-Trained Model for Programming and Natural Languages.arXiv:2002.08155 – reference: Galassi, A., M. Lippi, and P. Torroni. 2021, October. Natural Language Processing.IEEE Transactions on Neural Networks and Learning Systems 32(10): 4291–4308. https://doi.org/10.1109/TNNLS.2020.3019893 – reference: Brown, T.B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss,G. Krueger, T. Henighan, R. Child, A. Ramesh, D.M. Ziegler, J. Wu, C. Winter,C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark,C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. 2020,July.Language Models are Few-Shot Learners. arXiv:2005.14165 [cs] – reference: Svyatkovskiy, A., S.K. Deng, S. Fu, and N. Sundaresan 2020, November.IntelliCode compose: code generation using transformer.In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event USA, pp.1433–1443. ACM – reference: GitHub Inc. 2019.CodeQL – reference: Devlin, J., M.W. Chang, K. Lee, and K. Toutanova. 2019, May. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 – reference: Synopsys 2022.Source Security and Risk Analysis Report. Technical report, Synopsys Inc – reference: Barke, S., M.B. James, and N. Polikarpova. 2022, August. Grounded Copilot: How Programmers Interact with Code-Generating Models. arXiv:2206.15000 – reference: Sobania, D., M. Briesch, and F. Rothlauf 2022, July.Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming.In Proceedings of the Genetic and Evolutionary Computation Conference, Boston Massachusetts, pp.1019–1027. ACM – reference: Yin, J., X. Jiang, Z. Lu, L. Shang, H. Li, and X. Li 2016. Neural Generative Question Answering.In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, pp.2972–2978. AAAI Press.event-place: New York, New York, USA – reference: Prenner, J., H. Babii, and R. Robbes 2022, May. Can OpenAI’s Codex Fix Bugs?: An evaluation on QuixBugs.2022 IEEE/ACM International Workshop on Automated Program Repair (APR), Los Alamitos, CA, USA, pp.69–75. IEEE Computer Society – reference: ZhouJCaoYWangXLiPXuWDeep Recurrent Models with Fast-Forward Connections for Neural Machine TranslationTransactions of the Association for Computational Linguistics2016437138310.1162/tacl_a_00105 – reference: Hellendoorn, V.J. and P. Devanbu 2017, August.Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, Paderborn Germany, pp.763–773. ACM – reference: Nguyen, N. and S. Nadi 2022.Empirical Evaluation of GitHub Copilot’s Code Suggestions.In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), pp.1–5 – reference: Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł Kaiser, and I. Polosukhin 2017.Attention is All You Need.In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA,pp.6000–6010. Curran Associates Inc.event-place: Long Beach, California, USA – reference: Lu, S., D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement,D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano,M. Gong, M. Zhou, N. Duan, N. Sundaresan, S.K. Deng, S. Fu, and S. Liu. 2021,March.CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. arXiv:2102.04664 – reference: Dakhel, A.M., V. Majdinasab, A. Nikanjam, F. Khomh, M.C. Desmarais, Z. Ming,and Jiang. 2022, June. GitHub Copilot AI pair programmer: Asset or Liability? arXiv:2206.15331 – reference: Chen, M., J. Tworek, H. Jun, Q. Yuan, H.P.d.O. Pinto, J. Kaplan, H. Edwards,Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov,H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov,A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F.P. Such,D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W.H. Guss,A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain,W. Saunders, C. Hesse, A.N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa,A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder,B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. 2021,July.Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs] – reference: Dohmke, T. 2022, June.GitHub Copilot is generally available to all developers – reference: GitHub Inc. 2021.GitHub Copilot Your AI pair programmer – reference: Hindle, A., E.T. Barr, Z. Su, M. Gabel, and P. Devanbu 2012.On the Naturalness of Software.In Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, pp.837–847. IEEE Press. event-place: Zurich, Switzerland – reference: Raychev, V., M. Vechev, and E. Yahav 2014, June.Code completion with statistical language models.In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, Edinburgh United Kingdom, pp.419–428. ACM – reference: Yan, W. and Y. Li. 2022, April.WhyGen: Explaining ML-powered Code Generation by Referring to Training Examples. arXiv:2204.07940 – reference: Fan, J., Y. Li, S. Wang, and T.N. Nguyen 2020, June.A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, Seoul Republic of Korea, pp. 508–512.ACM – reference: Tabnine. 2022.Code Faster with AI Completions – reference: Asare, O., M. Nagappan, and N. Asokan. 2022. Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code? _eprint: 2204.04741 – reference: Bengio, Y., R. Ducharme, and P. Vincent 2000.A Neural Probabilistic Language Model. In Advances in Neural Information Processing Systems,Volume 13. MIT Press – reference: Pearce, H., B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri 2022, May.Asleep at the Keyboard? Assessing the Security of GitHub Copilot Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp.754–768.ISSN: 2375-1207 – reference: Xu, F.F., U. Alon, G. Neubig, and V.J. Hellendoorn 2022, June. A systematic evaluation of large language models of code.In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego CA USA, pp.1–10. ACM – reference: Chen, D. and C. Manning 2014, October. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, pp. 740–750. Association for Computational Linguistics – reference: Jiang, N., T. Lutellier, and L. Tan 2021, May. CURE: Code-Aware Neural Machine Translation for Automatic Program Repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp.1161–1173.ISSN: 1558-1225 – reference: Hardmeier, C. 2016, December.A Neural Model for Part-of-Speech Tagging in Historical Texts.In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, pp.922–931. The COLING 2016 Organizing Committee – reference: Le, T.H.M., H. Chen, and M.A. Babar. 2020, June.Deep Learning for Source Code Modeling and Generation:Models, Applications, and Challenges. ACM Comput. Surv. 53(3)https://doi.org/10.1162/neco.10.1145/3383458 – reference: Zhang, J., J. Cambronero, S. Gulwani, V. Le, R. Piskac, G. Soares, and G. Verbruggen. 2022.Repairing Bugs in Python Assignments Using Large Language Models – reference: HochreiterSSchmidhuberJNovember. Long Short-Term MemoryNeural Computation1997981735178010.1162/neco.1997.9.8.1735 – ident: 10380_CR8 – ident: 10380_CR16 doi: 10.1109/TNNLS.2020.3019893 – ident: 10380_CR18 – ident: 10380_CR43 – volume: 4 start-page: 371 year: 2016 ident: 10380_CR44 publication-title: Transactions of the Association for Computational Linguistics doi: 10.1162/tacl_a_00105 – ident: 10380_CR2 doi: 10.1145/3586030 – volume: 9 start-page: 1735 issue: 8 year: 1997 ident: 10380_CR22 publication-title: Neural Computation doi: 10.1162/neco.1997.9.8.1735 – ident: 10380_CR24 doi: 10.1162/neco.10.1145/3383458 – ident: 10380_CR26 – ident: 10380_CR40 doi: 10.1145/3510454.3516866 – ident: 10380_CR28 – ident: 10380_CR7 doi: 10.3115/v1/D14-1082 – ident: 10380_CR41 doi: 10.18653/v1/W16-0106 – ident: 10380_CR20 doi: 10.1145/3106237.3106290 – ident: 10380_CR21 doi: 10.1109/ICSE.2012.6227135 – ident: 10380_CR1 doi: 10.1007/s10664-023-10380-1 – ident: 10380_CR36 – ident: 10380_CR11 – ident: 10380_CR13 – ident: 10380_CR5 – ident: 10380_CR9 doi: 10.1145/3524842.3528440 – ident: 10380_CR15 doi: 10.18653/v1/2020.findings-emnlp.139 – ident: 10380_CR25 doi: 10.1126/science.abq1158 – ident: 10380_CR45 doi: 10.1145/3520312.3534864 – ident: 10380_CR14 doi: 10.1145/3379597.3387501 – ident: 10380_CR38 – ident: 10380_CR3 – ident: 10380_CR29 doi: 10.1109/SP46214.2022.9833571 – ident: 10380_CR19 – ident: 10380_CR17 – ident: 10380_CR39 doi: 10.1145/3520312.3534862 – ident: 10380_CR31 doi: 10.1145/3524459.3527351 – ident: 10380_CR32 doi: 10.1145/2594291.2594321 – ident: 10380_CR30 doi: 10.1109/SP46215.2023.10179324 – ident: 10380_CR10 – ident: 10380_CR12 – ident: 10380_CR42 doi: 10.18653/v1/P17-1041 – ident: 10380_CR35 – ident: 10380_CR33 doi: 10.1145/3512290.3528700 – ident: 10380_CR34 doi: 10.1145/3368089.3417058 – ident: 10380_CR27 doi: 10.1145/3524842.3528470 – ident: 10380_CR23 doi: 10.1109/ICSE43902.2021.00107 – ident: 10380_CR37 doi: 10.1145/3491101.3519665 – ident: 10380_CR4 doi: 10.1145/2983990.2984041 – volume: 48 start-page: 3280 issue: 9 year: 2022 ident: 10380_CR6 publication-title: IEEE Transactions on Software Engineering doi: 10.1109/TSE.2021.3087402 |
| SSID | ssj0009745 |
| Score | 2.6170175 |
| Snippet | Several advances in deep learning have been successfully applied to the software development process. Of recent interest is the use of neural language models... |
| SourceID | proquest crossref springer |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 129 |
| SubjectTerms | Compilers Computer Science Datasets Deep learning Empirical analysis Interpreters Language Mining Software Repositories (MSR) Natural language processing Neural networks Programming Languages Software development Software engineering Software Engineering/Programming and Operating Systems |
| SummonAdditionalLinks | – databaseName: SpringerLINK Contemporary 1997-Present dbid: RSV link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1PS8MwFA-iHrw4_-J0Sg7eNNAmbZaeRIZzggzxz9itpC8tDMY21m5nv4Zfz0_iS9Y6FRW0lx6ShvBeXn8veXm_R8hpGADigkENhEqzAIxkCSIZCwQ3mUjQ63fl3nq3zW5X9fvRXZkUlle33auQpPtTf0h2kzJgiDHMknp7DPc8awh3yprj_UNvSbXbdKWJLbkeE4joZarM92N8hqOlj_klLOrQpl373zy3yGbpXdLLxXLYJivpaIfUqsoNtDTkXdK9yen1oOjMktfnl5y2xpPBcFxQndNEG_typftyqgs6sFfZzQxwCnQ-G1qSanefFnfY2EZtSvzFHnlqXz22OqysrMAATa5gkEbcCJ6IUHMbzwoSiIzWnkrBeJayTHBhINNKG40OhPEhNF7oZcClyiDNxD5ZHY1H6QGhSvoGkhQfZbcykVIRoE-jfclBG87rxK8EHENJO26rXwzjJWGyFViMAoudwGK_Ts7ev5ksSDd-7d2o9BaXBpjHXMmmDKQQ2Hxe6WnZ_PNoh3_rfkQ2uFO1PZRpkNViOkuPyTrMi0E-PXEL8w2pudtu priority: 102 providerName: Springer Nature |
| Title | Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code? |
| URI | https://link.springer.com/article/10.1007/s10664-023-10380-1 https://www.proquest.com/docview/2867646331 |
| Volume | 28 |
| WOSCitedRecordID | wos001076229600001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAVX databaseName: SpringerLINK Contemporary 1997-Present customDbUrl: eissn: 1573-7616 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0009745 issn: 1382-3256 databaseCode: RSV dateStart: 19970101 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV3LSsQwFL34WrjxLY4vsnCnwTZpM-lKVHyBDIMvxE1Jb1oYGGbGace1v-Hv-SUmMbUo6MYu0kXaEHKS3Jvk5hyAvThCYxe0QSCWikaoBc2MJaMRZ7rgmfH6ndzbw3W705GPj0nXb7iVPqyynhPdRK2HaPfID5kUbREJzsOj0TO1qlH2dNVLaEzDrGVJYC5077Yh3W07kWJLs0e5se3-0oy_OidERI3FopYiPKDhd8PUeJs_Dkid3Tlf_G-Nl2DBe5zk-LOLLMNUPliBxVrNgfjBvQqdq5Jc9KrLSfb--laS0-Go1x9WRJUkU9q-nJxfSVRFeja8XU_QVJi8TPqWuNrF2JpVt8kj9pr80Rrcn5_dnV5Sr7ZA0QzDimKeMM1ZxmPF7BlXlGGilQpkjjqwNGaccY2Fkkor41ToEGMdxEGBTMgC84Kvw8xgOMg3gEgRasxy80i7vEmkTND4OSoUDJVmrAVh3dQpeipyq4jRTxsSZQtPauBJHTxp2IL9r39Gn0Qcf369XWOS-kFZpg0gLTioUW2yfy9t8-_StmCeuY5kN2a2YaYaT_IdmMOXqleOd2H25KzTvdl1XdOk3fjJpDe3Dx9vvelV |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V3NbtQwEB6VggQXyl_F0hZ8gBNYJHbidQ6oqgplV11WSBTUW3DGjhRptbs02SJuvEZfgofiSfB4EyKQ6K0HcsnBjqXYn2fG9vj7AJ6mCXq_YP0IpNrwBK3ihfdkPJHClrLwUX-Qe_s0GU6n-vQ0e78BP7q7MJRW2dnEYKjtAmmP_KXQaqgSJWW8v_zCSTWKTlc7CY01LI7dt69-yVa_Gr_24_tMiKM3J4cj3qoKcPRwazi6TFgpCpkaQWc5SYGZNSbSDm1EdF1SSIul0cYa7zxtjKmN0qhEoXSJrpS-3WtwPSHrH1IFP_Qkv8Mgiky0flz6WKK9pNNe1VMq4d5DcqIkj3j8pyPso9u_DmSDnzva-t966A7cbiNqdrCeAndhw83vwVanVsFa43UfpuOava2a0ar4-f2iZoeLZTVbNMzUrDCWXkGusGamYRWl79sV-g5i56sZEXOHHOLK1b6MEQ3A_gP4eCV_tQ2b88XcPQSmVWyxcP7RtHzLtM7Qx3EmVgKNFWIAcTe0ObZU66T4Mct7kmiCQ-7hkAc45PEAnv_-ZrkmGrm09m6Hgbw1OnXeA2AALzoU9cX_bu3R5a09gZujk3eTfDKeHu_ALRFATJtQu7DZnK3cHtzA86aqzx6H6cDg81Wj6xdxZkSI |
| linkToPdf | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V3LbtQwFL0qBSE2lKcYaMELWIHVxE48zqKqUMvQUavRLABVbIJjO1Kk0czQZIrY8Rv8Cp_TL-FeT9IIJLrrgmyySGIp9vF92NfnALxME4t-weEIpNrwxDrFC_RkPJHClbLAqD_IvX06GU4m-vQ0m27Ar-4sDJVVdjYxGGq3sLRGviu0GqpESRnvlm1ZxPRwtL_8yklBinZaOzmNNUSO_fdvmL7Ve-NDHOtXQozefTg44q3CALcIvYZbnwknRSFTI2hfJyls5oyJtLcuIuouKaSzpdHGGXSkLrapi9KotELp0vpSYrs34OYQc0xK_Kbp557wdxgEkonij0uMK9oDO-2xPaUSjt6SEz15xOM_nWIf6f61ORt83mjrf-6te3C3jbTZ2_XUuA8bfv4AtjoVC9YatYcwGdfsfdUcrYqLHz9rdrBYVrNFw0zNCuPoFmQMa2YaVlFZv1tZ7Cx2vpoRYXeoLa58jc8Y0QPsP4KP1_JXj2Fzvpj7J8C0ip0tPF6a0rpM68xifGdiJaxxQgwg7oY5ty0FOymBzPKePJqgkSM08gCNPB7A68tvlmsCkivf3u7wkLfGqM57MAzgTYeo_vG_W3t6dWsv4DaCKj8ZT46fwR0R8ExrU9uw2Zyt_A7csudNVZ89DzODwZfrBtdvkB5Naw |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Is+GitHub%E2%80%99s+Copilot+as+bad+as+humans+at+introducing+vulnerabilities+in+code%3F&rft.jtitle=Empirical+software+engineering+%3A+an+international+journal&rft.au=Asare%2C+Owura&rft.au=Nagappan%2C+Meiyappan&rft.au=Asokan%2C+N&rft.date=2023-11-01&rft.pub=Springer+Nature+B.V&rft.issn=1382-3256&rft.eissn=1573-7616&rft.volume=28&rft.issue=6&rft.spage=129&rft_id=info:doi/10.1007%2Fs10664-023-10380-1&rft.externalDBID=HAS_PDF_LINK |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1382-3256&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1382-3256&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1382-3256&client=summon |