Assessing chatbots ability to produce leaflets on cataract surgery: Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi
To evaluate leaflets on cataract surgery produced by 7 common free chatbots. UK-based ophthalmologists carrying out online research. Data were collected from the responses of 7 freely available online chatbots. Analysis of answers given by 7 chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Goo...
Uloženo v:
| Vydáno v: | Journal of cataract and refractive surgery Ročník 51; číslo 5; s. 371 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
United States
01.05.2025
|
| Témata: | |
| ISSN: | 1873-4502, 1873-4502 |
| On-line přístup: | Zjistit podrobnosti o přístupu |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | To evaluate leaflets on cataract surgery produced by 7 common free chatbots.
UK-based ophthalmologists carrying out online research.
Data were collected from the responses of 7 freely available online chatbots.
Analysis of answers given by 7 chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi) was prompted to "make a patient information leaflet on cataract surgery." Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade level readability score, and material reliability.
The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5. ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic, and Perplexity included reliable references; the other chatbots provided no references.
This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology. |
|---|---|
| AbstractList | To evaluate leaflets on cataract surgery produced by 7 common free chatbots.
UK-based ophthalmologists carrying out online research.
Data were collected from the responses of 7 freely available online chatbots.
Analysis of answers given by 7 chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi) was prompted to "make a patient information leaflet on cataract surgery." Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade level readability score, and material reliability.
The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5. ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic, and Perplexity included reliable references; the other chatbots provided no references.
This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology. This study aimed to evaluate leaflets on cataract surgery produced by seven common free chatbots.PURPOSEThis study aimed to evaluate leaflets on cataract surgery produced by seven common free chatbots.Usage of conversational artificial intelligence services (chatbots) is becoming more prevalent in all aspects of life, including healthcare. Cataract surgery is the most commonly performed operation in the world, with numbers set to increase. Possible applications for chatbots include information giving and education, allowing clinicians to allocate their time more efficiently.SETTINGUsage of conversational artificial intelligence services (chatbots) is becoming more prevalent in all aspects of life, including healthcare. Cataract surgery is the most commonly performed operation in the world, with numbers set to increase. Possible applications for chatbots include information giving and education, allowing clinicians to allocate their time more efficiently.Analysis of answers given by seven chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity and Pi) were prompted to "make a patient information leaflet on cataract surgery".DESIGNAnalysis of answers given by seven chatbots (Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity and Pi) were prompted to "make a patient information leaflet on cataract surgery".Answers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade Level readability score and material reliability.METHODSAnswers were evaluated using the DISCERN instrument, Patient Education Materials Assessment Tool (PEMAT), presence of misinformation, the Flesch-Kincaid Grade Level readability score and material reliability.The highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5.ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic and Perplexity included reliable references; the other chatbots provided no references.RESULTSThe highest overall scored response was from ChatSonic, followed by Bing AI and then Perplexity. The lowest scoring was ChatGPT 3.5.ChatSonic achieved the highest DISCERN and PEMAT scores, and had the highest Flesch-Kincaid Grade level. The lowest DISCERN and PEMAT scores were for Pi. Only ChatGPT 3.5 included some misinformation in its response. Bing AI, ChatSonic and Perplexity included reliable references; the other chatbots provided no references.This study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology.CONCLUSIONSThis study demonstrates a range of answers given by chatbots creating a cataract surgery leaflet, suggesting variation in their development and reliability. ChatGPT 3.5 scored the most poorly. However, ChatSonic indicated promise in how technology may be used to assist information giving in ophthalmology. |
| Author | Thompson, Polly Thornton, Richard Ramsden, Conor M |
| Author_xml | – sequence: 1 givenname: Polly orcidid: 0009-0003-0542-1090 surname: Thompson fullname: Thompson, Polly organization: From the The West of England Eye Unit, Royal Devon University Hospital NHS Foundation Trust, Exeter, United Kingdom (Thompson, Ramsden); Royal Eye Infirmary, Derriford Hospital, University Hospitals Plymouth NHS Foundation Trust, Plymouth, United Kingdom (Thornton); The University of Exeter, Exeter, United Kingdom (Ramsden) – sequence: 2 givenname: Richard surname: Thornton fullname: Thornton, Richard – sequence: 3 givenname: Conor M orcidid: 0000-0002-4320-2994 surname: Ramsden fullname: Ramsden, Conor M |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/39885649$$D View this record in MEDLINE/PubMed |
| BookMark | eNpNUMtOwzAQtFARfcAvgI8ckmDHsRNzaysolSpRid4jx96UVGlc7ESif8EnE6AI9rKz2tkZ7YzRoLENIHRDSUSJTO920U47H5F_RUUcn6ERzVIWJpzEg394iMbe73pSEjN-gYZMZhkXiRyhj6n34H3VbLF-VW1hW49VUdVVe8StxQdnTacB16DKGvqdbbBWrXJKt9h3bgvueI9nX-fTZfAtsVhvMIv435DYAM97_GKbSgd4Ye22BjxTzgR4De5Qw3vvFmDVGLyuLtF5qWoPV6c-QZvHh838KVw9L5bz6SrUTPA47FOgikFaMklIGSteatDElKXK0pRKpY1ICyHAkKRISUELqYXWRgueCiN5PEG3P7L9h28d-DbfV15DXasGbOdzRgWVlAspeur1idoVezD5wVV75Y75b4jxJ5fydzc |
| CitedBy_id | crossref_primary_10_1016_j_acra_2025_08_056 crossref_primary_10_1177_11206721251367562 crossref_primary_10_1097_j_jcrs_0000000000001683 |
| ContentType | Journal Article |
| Copyright | Copyright © 2025 Published by Wolters Kluwer on behalf of ASCRS and ESCRS. |
| Copyright_xml | – notice: Copyright © 2025 Published by Wolters Kluwer on behalf of ASCRS and ESCRS. |
| DBID | CGR CUY CVF ECM EIF NPM 7X8 |
| DOI | 10.1097/j.jcrs.0000000000001622 |
| DatabaseName | Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed MEDLINE - Academic |
| DatabaseTitle | MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) MEDLINE - Academic |
| DatabaseTitleList | MEDLINE MEDLINE - Academic |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | no_fulltext_linktorsrc |
| Discipline | Medicine |
| EISSN | 1873-4502 |
| ExternalDocumentID | 39885649 |
| Genre | Journal Article |
| GroupedDBID | --- --K .1- .FO 0R~ 1B1 1P~ 1~5 4.4 457 4G. 5GY 7-5 AAAAV AAEDT AAGIX AAHPQ AAIQE AALRI AAQKA AASCR ABASU ABDIG ABIID ABJNI ABLJU ABMAC ABPXF ABVCZ ABZZY ACGFO ACGFS ACILI ACLDA ACXJB ACZKN ADBBV ADGGA ADHPY AENEX AEVXI AFBFQ AFCTW AFDTB AFRHN AFTJW AHOMT AHQNM AINUH AJCLO AJIOK AJNWD AJUYK AJZMW AKCTQ AKRWK AKULP ALKUP ALMA_UNASSIGNED_HOLDINGS ALMTX AMJPA AMKUR AMNEI AOQMC BELOY C45 C5W CGR CS3 CUY CVF DIWNM DU5 EBS ECM EEVPB EIF ERAAH F5P FCALG FDB GBLVA GNXGY GQDEL HLJTE IHE IKREB IKYAY J1W KOM L7B MO0 NPM O9- OF- OPF OQ~ OVD P2P RLZ RPZ SEL SES SJN SSZ TEORI TSPGW UV1 XH2 Z5R 7X8 ACBKD ADKSD ADSXY |
| ID | FETCH-LOGICAL-c3652-1091a3e7f3900f2a5fcec0dffa87719acd67b66ed04b70b1b9c6ccdc6576d952 |
| IEDL.DBID | 7X8 |
| ISICitedReferencesCount | 3 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001569141600013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1873-4502 |
| IngestDate | Sun Nov 09 14:06:09 EST 2025 Thu Jun 26 01:51:50 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 5 |
| Language | English |
| License | Copyright © 2025 Published by Wolters Kluwer on behalf of ASCRS and ESCRS. |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c3652-1091a3e7f3900f2a5fcec0dffa87719acd67b66ed04b70b1b9c6ccdc6576d952 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| ORCID | 0009-0003-0542-1090 0000-0002-4320-2994 |
| PMID | 39885649 |
| PQID | 3161915696 |
| PQPubID | 23479 |
| ParticipantIDs | proquest_miscellaneous_3161915696 pubmed_primary_39885649 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-May |
| PublicationDateYYYYMMDD | 2025-05-01 |
| PublicationDate_xml | – month: 05 year: 2025 text: 2025-May |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Journal of cataract and refractive surgery |
| PublicationTitleAlternate | J Cataract Refract Surg |
| PublicationYear | 2025 |
| SSID | ssj0004235 |
| Score | 2.4743927 |
| Snippet | To evaluate leaflets on cataract surgery produced by 7 common free chatbots.
UK-based ophthalmologists carrying out online research.
Data were collected from... This study aimed to evaluate leaflets on cataract surgery produced by seven common free chatbots.PURPOSEThis study aimed to evaluate leaflets on cataract... |
| SourceID | proquest pubmed |
| SourceType | Aggregation Database Index Database |
| StartPage | 371 |
| SubjectTerms | Cataract Extraction Comprehension Generative Artificial Intelligence Humans Internet Patient Education as Topic - methods |
| Title | Assessing chatbots ability to produce leaflets on cataract surgery: Bing AI, chatGPT 3.5, chatGPT 4o, ChatSonic, Google Bard, Perplexity, and Pi |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/39885649 https://www.proquest.com/docview/3161915696 |
| Volume | 51 |
| WOSCitedRecordID | wos001569141600013&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1bS-QwFA66I-LL6q7X1ZUI-zjRtmmTZl9EZdUFHQoOMm9DruowNON0dmF_hX95T9oO44sg2IfQC2lKcppzknPO9yH0g2sw2UCxEeVSRVJDFRE8yQkz0gkew906kfb-hvd6-WAginbDrWrDKudzYj1RG6_DHvkJBdNEwGJDsNPJMwmsUcG72lJoLKMOBVMmSDUfvEILT2qCzTjnlECbyTy-S_CT0fFIT6sGvbA9YhYYdN-yM2t9c7n-0S_dQJ9bSxOfNaLxBS3Z8itavW196ZvopfH3gurC-lHOlJ9VuAHt_odnHk9qKFiLxzbw-8IzX-Kw1xOyqnDVJFP_xOeh-tnvbv2Kq6KP6XG2uEh9F1_A-V3A3-3iK-8fxhafg1B2cWGnkwDGCa3J0uDiaQv1L3_1L65JS89ANGVZQgKkqKSWOyqiyCUyc9rqyDgnc85jIbVhXDFmTZQqHqlYCc20NprBEseILNlGn0pf2l2EcymzRFHupBRpZmC8MhMwTqkz1EK5h47mPT0E6Q8uDVla_6caLvp6D-00wzWcNDAdQyryPGOp-PaO2vtoLQnEvnUk4wHqOPj37Xe0ov_OnqrpYS1WUPaK2__3VtYR |
| linkProvider | ProQuest |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Assessing+chatbots+ability+to+produce+leaflets+on+cataract+surgery%3A+Bing+AI%2C+chatGPT+3.5%2C+chatGPT+4o%2C+ChatSonic%2C+Google+Bard%2C+Perplexity+and+Pi&rft.jtitle=Journal+of+cataract+and+refractive+surgery&rft.au=Thompson%2C+Dr+Polly&rft.au=Thornton%2C+Dr+Richard&rft.au=Ramsden%2C+Mr+Conor&rft.date=2025-05-01&rft.issn=1873-4502&rft.eissn=1873-4502&rft_id=info:doi/10.1097%2Fj.jcrs.0000000000001622&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1873-4502&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1873-4502&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1873-4502&client=summon |