A Bayesian index of association: comparison with other measures and performance - Oleinik, Anton (Quality & quantity, 2024)
The article discusses a Bayesian measure of association, B-index, and compares it with the other existing measures of agreement, association, and similarity, both chance-corrected and non-corrected: Scott’s π, Krippendorff’s α, Cohen’s κ, Bennett, Alpert & Goldstein’s S, Cosine similarity, and the Jaccard similarity coefficient. PageRank adapted to particularities of annotation is also added to this list. Two versions of B-index are considered: with the informative and non-informative priors. An algorithm for calculating B-index written in pseudocode is provided. Particular attention is devoted to the uses of those measures in content analysis, communication studies, computational linguistics, psychology, computer science and network science. Real-world data gathered using an online platform for content analysis allowed comparing the behavior of all eight measures included in the scope of analysis. Three short texts (164 data points/sentences in total) were coded by 66 annotators. The behaviors of B-index with the non-informative prior and Bennett, Alpert & Goldstein’s S have some common patterns.