Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches

Supervised machine learning methods are increasingly employed in political science. Such models require costly manual labeling of documents. In this paper, we introduce active learning, a framework in which data to be labeled by human coders are not chosen at random but rather targeted in such a way...

Full description

Saved in:
Bibliographic Details
Published in:Political analysis Vol. 28; no. 4; pp. 532 - 551
Main Authors: Miller, Blake, Linder, Fridolin, Mebane, Walter R.
Format: Journal Article
Language:English
Published: New York, USA Cambridge University Press 01.10.2020
Subjects:
Active learning
Algorithms
Automatic text analysis
Automation
Classification
Computer science
Data
Datasets
Efficiency
Labeling
Machine learning
Methods
Political science
Random sampling
Reliability
Researchers
Simulation
Social networks
Terrorism
active learning
statistical analysis of texts
intercoder reliability
supervised learning
ISSN:1047-1987, 1476-4989
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Supervised machine learning methods are increasingly employed in political science. Such models require costly manual labeling of documents. In this paper, we introduce active learning, a framework in which data to be labeled by human coders are not chosen at random but rather targeted in such a way that the required amount of data to train a machine learning model can be minimized. We study the benefits of active learning using text data examples. We perform simulation studies that illustrate conditions where active learning can reduce the cost of labeling text data. We perform these simulations on three corpora that vary in size, document length, and domain. We find that in cases where the document class of interest is not balanced, researchers can label a fraction of the documents one would need using random sampling (or “passive” learning) to achieve equally performing classifiers. We further investigate how varying levels of intercoder reliability affect the active learning procedures and find that even with low reliability, active learning performs more efficiently than does random sampling.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1047-1987
1476-4989
DOI:10.1017/pan.2020.4