Topic models do not model topics: epistemological remarks and steps towards best practices

The social sciences and digital humanities have recently adopted the machine learning technique of topic modeling to address research questions in their fields. This is problematic in a number of ways, some of which have not received much attention in the debate yet. This paper adds epistemological...

Full description

Saved in:
Bibliographic Details
Published in:Journal of data mining and digital humanities Vol. 2021
Main Author: Shadrova, Anna
Format: Journal Article
Language:English
Published: INRIA 27.10.2021
Nicolas Turenne
Subjects:
[info.info-dl]computer science [cs]/digital libraries [cs.dl]
[shs.langue]humanities and social sciences/linguistics
[shs]humanities and social sciences
Computer Science
digital humanities
Digital Libraries
Humanities and Social Sciences
information extraction for scientific inquiry
Linguistics
topic modeling
Topic modeling
digital humanities
information extraction for scientific inquiry
ISSN:2416-5999, 2416-5999
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The social sciences and digital humanities have recently adopted the machine learning technique of topic modeling to address research questions in their fields. This is problematic in a number of ways, some of which have not received much attention in the debate yet. This paper adds epistemological concerns centering around the interface between topic modeling and linguistic concepts and the argumentative embedding of evidence obtained through topic modeling. It concludes that topic modeling in its present state of methodological integration does not meet the requirements of an independent research method. It operates from relevantly unrealistic assumptions, is non-deterministic, cannot effectively be validated against a reasonable number of competing models, does not lock into a well-defined linguistic interface, and does not scholarly model topics in the sense of themes or content. These features are intrinsic and make the interpretation of its results prone to apophenia (the human tendency to perceive random sets of elements as meaningful patterns) and confirmation bias (the human tendency to perceptually prefer patterns that are in alignment with pre-existing biases). While partial validation of the statistical model is possible, a conceptual validation would require an extended triangulation with other methods and human ratings, and clarification of whether statistical distinctivity of lexical co-occurrence correlates with conceputal topics in any reliable way.
ISSN:2416-5999
2416-5999
DOI:10.46298/jdmdh.7595