View in EDS

More polished, not necessarily more learned: LLMs and perceived text quality in higher education

Saved in:
Bibliographic Details
Title: More polished, not necessarily more learned: LLMs and perceived text quality in higher education
Authors: Tärning, Betty, Tjøstheim, Trond A., Wallin, Annika
Contributors: Lund University, Profile areas and other strong research environments, Lund University Profile areas, LU Profile Area: Natural and Artificial Cognition, Lunds universitet, Profilområden och andra starka forskningsmiljöer, Lunds universitets profilområden, LU profilområde: Naturlig och artificiell kognition, Originator, Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, Cognitive modeling, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, Kognitiv modellering, Originator, Lund University, Profile areas and other strong research environments, Strategic research areas (SRA), eSSENCE: The e-Science Collaboration, Lunds universitet, Profilområden och andra starka forskningsmiljöer, Strategiska forskningsområden (SFO), eSSENCE: The e-Science Collaboration, Originator, Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, The Educational Technology Group, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, The Educational Technology Group, Originator, Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, Cognitive Science, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, Kognitionsvetenskap, Originator
Source: Frontiers in Artificial Intelligence.
Subject Terms: Natural Sciences, Computer and Information Sciences, Artificial Intelligence, Naturvetenskap, Data- och informationsvetenskap (Datateknik), Artificiell intelligens, Social Sciences, Educational Sciences, Samhällsvetenskap, Utbildningsvetenskap
Description: The use of Large Language Models (LLMs) such as ChatGPT is a prominent topic in higher education, prompting debate over their educational impact. Studies on the effect of LLMs on learning in higher education often rely on self-reported data, leaving an opening for complimentary methodologies. This study contributes by analysing actual course grades as well as ratings by fellow students to investigate how LLMs can affect academic outcomes. We investigated whether using LLMs affected students’ learning by allowing them to choose one of three options for a written assignment: (1) composing the text without LLM assistance; (2) writing a first draft and using an LLM for revisions; or (3) generating a first draft with an LLM and then revising it themselves. Students’ learning was measured by their scores on a mid-course exam and final course grades. Additionally, we assessed how the students rate the quality of fellow students’ texts for each of the three conditions. Finally we examined how accurately fellow students could identify which LLM option (1–3) was used for a given text. Our results indicate only a weak effect of LLM use. However, writing a first draft and using an LLM for revisions compared favourably to the ‘no LLM’ baseline in terms of final grades. Ratings for fellow students’ texts was higher for texts created using option 3, specifically regarding how well-written they were judged to be. Regarding text classification, students most accurately predicted the ‘no LLM’ baseline, but were unable to identify texts that were generated by an LLM and then edited by a student at a rate better than chance.
File Description: electronic
Access URL: https://lucris.lub.lu.se/ws/files/234491786/Ta_rning_2025_-_More_polished_not_necessarily_more_learned.pdf
Database: SwePub
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://lucris.lub.lu.se/ws/files/234491786/Ta_rning_2025_-_More_polished_not_necessarily_more_learned.pdf#
    Name: EDS - SwePub (s4221598)
    Category: fullText
    Text: View record in SwePub
  – Url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&db=pmc&term=2624-8212[TA]+AND+[PG]+AND+2025[PDAT]
    Name: FREE - PubMed Central (ISSN based link)
    Category: fullText
    Text: Full Text
    Icon: https://imageserver.ebscohost.com/NetImages/iconPdf.gif
    MouseOverText: Check this PubMed for the article full text.
  – Url: https://resolver.ebscohost.com/openurl?sid=EBSCO:edsswe&genre=article&issn=26248212&ISBN=&volume=&issue=&date=20251201&spage=&pages=&title=Frontiers in Artificial Intelligence&atitle=More%20polished%2C%20not%20necessarily%20more%20learned%3A%20LLMs%20and%20perceived%20text%20quality%20in%20higher%20education&aulast=T%C3%A4rning%2C%20Betty&id=DOI:10.3389/frai.2025.1653992
    Name: Full Text Finder
    Category: fullText
    Text: Full Text Finder
    Icon: https://imageserver.ebscohost.com/branding/images/FTF.gif
    MouseOverText: Full Text Finder
  – Url: https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=EBSCO&SrcAuth=EBSCO&DestApp=WOS&ServiceName=TransferToWoS&DestLinkType=GeneralSearchSummary&Func=Links&author=T%C3%A4rning%20B
    Name: ISI
    Category: fullText
    Text: Nájsť tento článok vo Web of Science
    Icon: https://imagesrvr.epnet.com/ls/20docs.gif
    MouseOverText: Nájsť tento článok vo Web of Science
Header DbId: edsswe
DbLabel: SwePub
An: edsswe.oai.portal.research.lu.se.publications.4a1cd6ff.393f.4aa5.806d.7220a170da98
RelevancyScore: 1124
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 1123.91345214844
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: More polished, not necessarily more learned: LLMs and perceived text quality in higher education
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Tärning%2C+Betty%22">Tärning, Betty</searchLink><br /><searchLink fieldCode="AR" term="%22Tjøstheim%2C+Trond+A%2E%22">Tjøstheim, Trond A.</searchLink><br /><searchLink fieldCode="AR" term="%22Wallin%2C+Annika%22">Wallin, Annika</searchLink>
– Name: Author
  Label: Contributors
  Group: Au
  Data: Lund University, Profile areas and other strong research environments, Lund University Profile areas, LU Profile Area: Natural and Artificial Cognition, Lunds universitet, Profilområden och andra starka forskningsmiljöer, Lunds universitets profilområden, LU profilområde: Naturlig och artificiell kognition, Originator<br />Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, Cognitive modeling, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, Kognitiv modellering, Originator<br />Lund University, Profile areas and other strong research environments, Strategic research areas (SRA), eSSENCE: The e-Science Collaboration, Lunds universitet, Profilområden och andra starka forskningsmiljöer, Strategiska forskningsområden (SFO), eSSENCE: The e-Science Collaboration, Originator<br />Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, The Educational Technology Group, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, The Educational Technology Group, Originator<br />Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, Cognitive Science, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, Kognitionsvetenskap, Originator
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <i>Frontiers in Artificial Intelligence</i>.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Natural+Sciences%22">Natural Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+and+Information+Sciences%22">Computer and Information Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Naturvetenskap%22">Naturvetenskap</searchLink><br /><searchLink fieldCode="DE" term="%22Data-+och+informationsvetenskap+%28Datateknik%29%22">Data- och informationsvetenskap (Datateknik)</searchLink><br /><searchLink fieldCode="DE" term="%22Artificiell+intelligens%22">Artificiell intelligens</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Sciences%22">Social Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Sciences%22">Educational Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Samhällsvetenskap%22">Samhällsvetenskap</searchLink><br /><searchLink fieldCode="DE" term="%22Utbildningsvetenskap%22">Utbildningsvetenskap</searchLink>
– Name: Abstract
  Label: Description
  Group: Ab
  Data: The use of Large Language Models (LLMs) such as ChatGPT is a prominent topic in higher education, prompting debate over their educational impact. Studies on the effect of LLMs on learning in higher education often rely on self-reported data, leaving an opening for complimentary methodologies. This study contributes by analysing actual course grades as well as ratings by fellow students to investigate how LLMs can affect academic outcomes. We investigated whether using LLMs affected students’ learning by allowing them to choose one of three options for a written assignment: (1) composing the text without LLM assistance; (2) writing a first draft and using an LLM for revisions; or (3) generating a first draft with an LLM and then revising it themselves. Students’ learning was measured by their scores on a mid-course exam and final course grades. Additionally, we assessed how the students rate the quality of fellow students’ texts for each of the three conditions. Finally we examined how accurately fellow students could identify which LLM option (1–3) was used for a given text. Our results indicate only a weak effect of LLM use. However, writing a first draft and using an LLM for revisions compared favourably to the ‘no LLM’ baseline in terms of final grades. Ratings for fellow students’ texts was higher for texts created using option 3, specifically regarding how well-written they were judged to be. Regarding text classification, students most accurately predicted the ‘no LLM’ baseline, but were unable to identify texts that were generated by an LLM and then edited by a student at a rate better than chance.
– Name: Format
  Label: File Description
  Group: SrcInfo
  Data: electronic
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://lucris.lub.lu.se/ws/files/234491786/Ta_rning_2025_-_More_polished_not_necessarily_more_learned.pdf" linkWindow="_blank">https://lucris.lub.lu.se/ws/files/234491786/Ta_rning_2025_-_More_polished_not_necessarily_more_learned.pdf</link>
PLink https://erproxy.cvtisr.sk/sfx/access?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsswe&AN=edsswe.oai.portal.research.lu.se.publications.4a1cd6ff.393f.4aa5.806d.7220a170da98
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.3389/frai.2025.1653992
    Languages:
      – Text: English
    Subjects:
      – SubjectFull: Natural Sciences
        Type: general
      – SubjectFull: Computer and Information Sciences
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Naturvetenskap
        Type: general
      – SubjectFull: Data- och informationsvetenskap (Datateknik)
        Type: general
      – SubjectFull: Artificiell intelligens
        Type: general
      – SubjectFull: Social Sciences
        Type: general
      – SubjectFull: Educational Sciences
        Type: general
      – SubjectFull: Samhällsvetenskap
        Type: general
      – SubjectFull: Utbildningsvetenskap
        Type: general
    Titles:
      – TitleFull: More polished, not necessarily more learned: LLMs and perceived text quality in higher education
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Tärning, Betty
      – PersonEntity:
          Name:
            NameFull: Tjøstheim, Trond A.
      – PersonEntity:
          Name:
            NameFull: Wallin, Annika
      – PersonEntity:
          Name:
            NameFull: Lund University, Profile areas and other strong research environments, Lund University Profile areas, LU Profile Area: Natural and Artificial Cognition, Lunds universitet, Profilområden och andra starka forskningsmiljöer, Lunds universitets profilområden, LU profilområde: Naturlig och artificiell kognition, Originator
      – PersonEntity:
          Name:
            NameFull: Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, Cognitive modeling, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, Kognitiv modellering, Originator
      – PersonEntity:
          Name:
            NameFull: Lund University, Profile areas and other strong research environments, Strategic research areas (SRA), eSSENCE: The e-Science Collaboration, Lunds universitet, Profilområden och andra starka forskningsmiljöer, Strategiska forskningsområden (SFO), eSSENCE: The e-Science Collaboration, Originator
      – PersonEntity:
          Name:
            NameFull: Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, The Educational Technology Group, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, The Educational Technology Group, Originator
      – PersonEntity:
          Name:
            NameFull: Lund University, Joint Faculties of Humanities and Theology, Departments, Department of Philosophy, Cognitive Science, Lunds universitet, Humanistiska och teologiska fakulteterna, Institutioner, Filosofiska institutionen, Kognitionsvetenskap, Originator
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 26248212
            – Type: issn-locals
              Value: SWEPUB_FREE
            – Type: issn-locals
              Value: LU_SWEPUB
          Titles:
            – TitleFull: Frontiers in Artificial Intelligence
              Type: main
ResultId 1