What is wrong with topic modeling? And how to fix it using search-based software engineering

Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeling technique is Latent Dirichlet allocation. When running on different datasets, LDA suffers from “order effects”, i.e., different topics are generated if the order of training data is shuffled. Su...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Information and software technology Ročník 98; s. 74 - 88
Hlavní autoři: Agrawal, Amritanshu, Fu, Wei, Menzies, Tim
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier B.V 01.06.2018
Témata:
Differential evolution
LDA
Stability
Topic modeling
Tuning
Topic modeling
LDA
Differential evolution
Stability
Tuning
ISSN:0950-5849, 1873-6025
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeling technique is Latent Dirichlet allocation. When running on different datasets, LDA suffers from “order effects”, i.e., different topics are generated if the order of training data is shuffled. Such order effects introduce a systematic error for any study. This error can relate to misleading results; specifically, inaccurate topic descriptions and a reduction in the efficacy of text mining classification results. To provide a method in which distributions generated by LDA are more stable and can be used for further analysis. We use LDADE, a search-based software engineering tool which uses Differential Evolution (DE) to tune the LDA’s parameters. LDADE is evaluated on data from a programmer information exchange site (Stackoverflow), title and abstract text of thousands of Software Engineering (SE) papers, and software defect reports from NASA. Results were collected across different implementations of LDA (Python+Scikit-Learn, Scala+Spark) across Linux platform and for different kinds of LDAs (VEM, Gibbs sampling). Results were scored via topic stability and text mining classification accuracy. In all treatments: (i) standard LDA exhibits very large topic instability; (ii) LDADE’s tunings dramatically reduce cluster instability; (iii) LDADE also leads to improved performances for supervised as well as unsupervised learning. Due to topic instability, using standard LDA with its “off-the-shelf” settings should now be depreciated. Also, in future, we should require SE papers that use LDA to test and (if needed) mitigate LDA topic instability. Finally, LDADE is a candidate technology for effectively and efficiently reducing that instability.
ISSN:0950-5849
1873-6025
DOI:10.1016/j.infsof.2018.02.005