Finding “similar” universities using ChatGPT for institutional benchmarking: A large‐scale comparison of European universities

The study objective was to evaluate the efficacy of ChatGPT in identifying “similar” institutions for benchmarking the research performance of a university. Benchmarking is deemed a promising approach to compare “similar with similar” as a better alternative to rankings (comparing “different” univer...

Full description

Saved in:
Bibliographic Details
Published in:Journal of the Association for Information Science and Technology Vol. 76; no. 9; pp. 1174 - 1187
Main Authors: Lepori, Benedetto, Bornmann, Lutz, Gay, Mario
Format: Journal Article
Language:English
Published: Hoboken, USA John Wiley & Sons, Inc 01.09.2025
ISSN:2330-1635, 2330-1643
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The study objective was to evaluate the efficacy of ChatGPT in identifying “similar” institutions for benchmarking the research performance of a university. Benchmarking is deemed a promising approach to compare “similar with similar” as a better alternative to rankings (comparing “different” universities). Current approaches either focus on a limited number of “quantitative” dimensions or are too complex for most users. We conducted large‐scale testing by tasking ChatGPT with identifying the most similar European universities in terms of research performance, utilizing the European Tertiary Education Register data. We tested whether the peers suggested by ChatGPT were similar to the focal university on size, research intensity, and subject composition. Additionally, we evaluated whether providing more specific instructions improved the results. The findings offer a nuanced perspective on the potential and risks of using ChatGPT to identify peer institutions for benchmarking. On one hand, solely using ChatGPT would replicate the visibility biases associated with university rankings, thereby undermining the rationale for benchmarking. On the other hand, relying on semantic associations might capture dimensions of university similarity that are relevant and difficult to capture through quantitative methods. We finally reflected on the broader implications for scholars in higher education and science studies research.
ISSN:2330-1635
2330-1643
DOI:10.1002/asi.25010