Coding with the machines: machine-assisted coding of rare event data

Abstract While machine coding of data has dramatically advanced in recent years, the literature raises significant concerns about validation of LLM classification showing, for example, that reliability varies greatly by prompt and temperature tuning, across subject areas and tasks—especially in “zer...

Celý popis

Uložené v:

Podrobná bibliografia
Vydané v:	PNAS nexus Ročník 3; číslo 5; s. pgae165
Hlavní autori:	Overos, Henry David, Hlatky, Roman, Pathak, Ojashwi, Goers, Harriet, Gouws-Dewar, Jordan, Smith, Katy, Chew, Keith Padraic, Birnir, Jóhanna K, Liu, Amy H
Médium:	Journal Article
Jazyk:	English
Vydavateľské údaje:	US Oxford University Press 01.05.2024
Predmet:	Accuracy Algorithms Coders Coding Datasets Dictionaries Electronic data processing Engineering Language Large language models Machine learning Methods Minority & ethnic groups Performance assessment Performance evaluation Politics Prompt engineering Social and Political Sciences United States GPT BERT machine coding machine learning political event data
ISSN:	2752-6542, 2752-6542
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!

Popis
Shrnutí:	Abstract While machine coding of data has dramatically advanced in recent years, the literature raises significant concerns about validation of LLM classification showing, for example, that reliability varies greatly by prompt and temperature tuning, across subject areas and tasks—especially in “zero-shot” applications. This paper contributes to the discussion of validation in several different ways. To test the relative performance of supervised and semi-supervised algorithms when coding political data, we compare three models’ performances to each other over multiple iterations for each model and to trained expert coding of data. We also examine changes in performance resulting from prompt engineering and pre-processing of source data. To ameliorate concerns regarding LLM’s pre-training on test data, we assess performance by updating an existing dataset beyond what is publicly available. Overall, we find that only GPT-4 approaches trained expert coders when coding contexts familiar to human coders and codes more consistently across contexts. We conclude by discussing some benefits and drawbacks of machine coding moving forward.
Bibliografia:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 Competing Interest: The authors declare no competing interest.
ISSN:	2752-6542 2752-6542
DOI:	10.1093/pnasnexus/pgae165