Coding with the machines: machine-assisted coding of rare event data

Abstract While machine coding of data has dramatically advanced in recent years, the literature raises significant concerns about validation of LLM classification showing, for example, that reliability varies greatly by prompt and temperature tuning, across subject areas and tasks—especially in “zer...

Full description

Saved in:
Bibliographic Details
Published in:PNAS nexus Vol. 3; no. 5; p. pgae165
Main Authors: Overos, Henry David, Hlatky, Roman, Pathak, Ojashwi, Goers, Harriet, Gouws-Dewar, Jordan, Smith, Katy, Chew, Keith Padraic, Birnir, Jóhanna K, Liu, Amy H
Format: Journal Article
Language:English
Published: US Oxford University Press 01.05.2024
Subjects:
Accuracy
Algorithms
Coders
Coding
Datasets
Dictionaries
Electronic data processing
Engineering
Language
Large language models
Machine learning
Methods
Minority & ethnic groups
Performance assessment
Performance evaluation
Politics
Prompt engineering
Social and Political Sciences
United States
GPT
BERT
machine coding
machine learning
political event data
ISSN:2752-6542, 2752-6542
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Abstract While machine coding of data has dramatically advanced in recent years, the literature raises significant concerns about validation of LLM classification showing, for example, that reliability varies greatly by prompt and temperature tuning, across subject areas and tasks—especially in “zero-shot” applications. This paper contributes to the discussion of validation in several different ways. To test the relative performance of supervised and semi-supervised algorithms when coding political data, we compare three models’ performances to each other over multiple iterations for each model and to trained expert coding of data. We also examine changes in performance resulting from prompt engineering and pre-processing of source data. To ameliorate concerns regarding LLM’s pre-training on test data, we assess performance by updating an existing dataset beyond what is publicly available. Overall, we find that only GPT-4 approaches trained expert coders when coding contexts familiar to human coders and codes more consistently across contexts. We conclude by discussing some benefits and drawbacks of machine coding moving forward.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
Competing Interest: The authors declare no competing interest.
ISSN:2752-6542
2752-6542
DOI:10.1093/pnasnexus/pgae165