Zobrazit v EDS

Literate Programming with LLMs? - A Study on Rosetta Code and CodeNet

Uloženo v:
Podrobná bibliografie
Název: Literate Programming with LLMs? - A Study on Rosetta Code and CodeNet
Autoři: Sun, Simin, 1993, Staron, Miroslaw, 1977
Zdroj: IEEE Transactions on Software Engineering. In Press
Témata: Code-related Tasks, Literate Programming, Computation Experiment, Large Language Model(LLM)
Popis: Literate programming, a concept introduced by Knuth in 1984, emphasized the importance of combining human-readable documentation with machine-readable code as writing literate programs is a prerequisite for software quality. Our objective with this paper is to evaluate whether generative AI models, Large Language Models (LLM) like GPT-4, LLaMA or Falcon, are capable of literate programming because of their extensive use in software engineering. To truly achieve literate programming, LLMs must generate natural language descriptions and corresponding code with aligned semantics based on user prompts. In addition, their internal representation of programs should allow us to recognize both programming languages and their descriptions. To evaluate their capabilities, we conducted a study using the Rosetta Code and CodeNet repositories. We perform four computational experiments using the Rosetta Code repository, encompassing 1,228 tasks across 926 programming languages, and validate our findings on the larger CodeNet dataset, which includes 55 tasks and 52 languages. Our findings show that LLMs in the trillion-parameter class are capable of literate programming, while models in the million- and billion-parameter classes are better at recognizing programming languages than tasks. Based on these results, we conclude that modern LLMs inhibit a deeper ability to encode programming languages and the semantics of programming tasks, bringing us closer to realizing the full potential of literate programming.
Popis souboru: electronic
Přístupová URL adresa: https://research.chalmers.se/publication/549267
https://research.chalmers.se/publication/549267/file/549267_Fulltext.pdf
Databáze: SwePub
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://research.chalmers.se/publication/549267#
    Name: EDS - SwePub (s4221598)
    Category: fullText
    Text: View record in SwePub
  – Url: https://resolver.ebscohost.com/openurl?sid=EBSCO:edsswe&genre=article&issn=00985589&ISBN=&volume=In%20Press&issue=&date=20250101&spage=&pages=&title=IEEE Transactions on Software Engineering&atitle=Literate%20Programming%20with%20LLMs%3F%20-%20A%20Study%20on%20Rosetta%20Code%20and%20CodeNet&aulast=Sun%2C%20Simin&id=DOI:10.1109/TSE.2025.3629828
    Name: Full Text Finder
    Category: fullText
    Text: Full Text Finder
    Icon: https://imageserver.ebscohost.com/branding/images/FTF.gif
    MouseOverText: Full Text Finder
  – Url: https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=EBSCO&SrcAuth=EBSCO&DestApp=WOS&ServiceName=TransferToWoS&DestLinkType=GeneralSearchSummary&Func=Links&author=Sun%20S
    Name: ISI
    Category: fullText
    Text: Nájsť tento článok vo Web of Science
    Icon: https://imagesrvr.epnet.com/ls/20docs.gif
    MouseOverText: Nájsť tento článok vo Web of Science
Header DbId: edsswe
DbLabel: SwePub
An: edsswe.oai.research.chalmers.se.2a5ab5bf.badf.4264.a89e.940cc8c7cedf
RelevancyScore: 1065
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 1064.736328125
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Literate Programming with LLMs? - A Study on Rosetta Code and CodeNet
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sun%2C+Simin%22">Sun, Simin</searchLink>, 1993<br /><searchLink fieldCode="AR" term="%22Staron%2C+Miroslaw%22">Staron, Miroslaw</searchLink>, 1977
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <i>IEEE Transactions on Software Engineering</i>. In Press
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Code-related+Tasks%22">Code-related Tasks</searchLink><br /><searchLink fieldCode="DE" term="%22Literate+Programming%22">Literate Programming</searchLink><br /><searchLink fieldCode="DE" term="%22Computation+Experiment%22">Computation Experiment</searchLink><br /><searchLink fieldCode="DE" term="%22Large+Language+Model%28LLM%29%22">Large Language Model(LLM)</searchLink>
– Name: Abstract
  Label: Description
  Group: Ab
  Data: Literate programming, a concept introduced by Knuth in 1984, emphasized the importance of combining human-readable documentation with machine-readable code as writing literate programs is a prerequisite for software quality. Our objective with this paper is to evaluate whether generative AI models, Large Language Models (LLM) like GPT-4, LLaMA or Falcon, are capable of literate programming because of their extensive use in software engineering. To truly achieve literate programming, LLMs must generate natural language descriptions and corresponding code with aligned semantics based on user prompts. In addition, their internal representation of programs should allow us to recognize both programming languages and their descriptions. To evaluate their capabilities, we conducted a study using the Rosetta Code and CodeNet repositories. We perform four computational experiments using the Rosetta Code repository, encompassing 1,228 tasks across 926 programming languages, and validate our findings on the larger CodeNet dataset, which includes 55 tasks and 52 languages. Our findings show that LLMs in the trillion-parameter class are capable of literate programming, while models in the million- and billion-parameter classes are better at recognizing programming languages than tasks. Based on these results, we conclude that modern LLMs inhibit a deeper ability to encode programming languages and the semantics of programming tasks, bringing us closer to realizing the full potential of literate programming.
– Name: Format
  Label: File Description
  Group: SrcInfo
  Data: electronic
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://research.chalmers.se/publication/549267" linkWindow="_blank">https://research.chalmers.se/publication/549267</link><br /><link linkTarget="URL" linkTerm="https://research.chalmers.se/publication/549267/file/549267_Fulltext.pdf" linkWindow="_blank">https://research.chalmers.se/publication/549267/file/549267_Fulltext.pdf</link>
PLink https://erproxy.cvtisr.sk/sfx/access?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsswe&AN=edsswe.oai.research.chalmers.se.2a5ab5bf.badf.4264.a89e.940cc8c7cedf
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1109/TSE.2025.3629828
    Languages:
      – Text: English
    Subjects:
      – SubjectFull: Code-related Tasks
        Type: general
      – SubjectFull: Literate Programming
        Type: general
      – SubjectFull: Computation Experiment
        Type: general
      – SubjectFull: Large Language Model(LLM)
        Type: general
    Titles:
      – TitleFull: Literate Programming with LLMs? - A Study on Rosetta Code and CodeNet
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sun, Simin
      – PersonEntity:
          Name:
            NameFull: Staron, Miroslaw
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 00985589
            – Type: issn-print
              Value: 19393520
            – Type: issn-locals
              Value: SWEPUB_FREE
            – Type: issn-locals
              Value: CTH_SWEPUB
          Numbering:
            – Type: volume
              Value: In Press
          Titles:
            – TitleFull: IEEE Transactions on Software Engineering
              Type: main
ResultId 1