Computing inter-rater reliability and its variance in the presence of high agreement

Pi (π) and kappa (κ) statistics are widely used in the areas of psychiatry and psychological testing to compute the extent of agreement between raters on nominally scaled data. It is a fact that these coefficients occasionally yield unexpected results in situations known as the paradoxes of kappa. T...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:British journal of mathematical & statistical psychology Ročník 61; číslo 1; s. 29 - 48
Hlavný autor: Gwet, Kilem Li
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Oxford, UK Blackwell Publishing Ltd 01.05.2008
British Psychological Society
Predmet:
Analysis of Variance
Humans
Monte Carlo Method
Monte Carlo simulation
Observer Variation
Probability
Psychiatry
Psychological tests
Psychological Tests - statistics & numerical data
Psychology
Psychometrics - statistics & numerical data
Reproducibility of Results
Statistical analysis
ISSN:0007-1102, 2044-8317
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Pi (π) and kappa (κ) statistics are widely used in the areas of psychiatry and psychological testing to compute the extent of agreement between raters on nominally scaled data. It is a fact that these coefficients occasionally yield unexpected results in situations known as the paradoxes of kappa. This paper explores the origin of these limitations, and introduces an alternative and more stable agreement coefficient referred to as the AC1 coefficient. Also proposed are new variance estimators for the multiple‐rater generalized π and AC1 statistics, whose validity does not depend upon the hypothesis of independence between raters. This is an improvement over existing alternative variances, which depend on the independence assumption. A Monte‐Carlo simulation study demonstrates the validity of these variance estimators for confidence interval construction, and confirms the value of AC1 as an improved alternative to existing inter‐rater reliability statistics.
Bibliografia:ArticleID:BMSP253
istex:29A04BD84BF5C2B0E9624DF373427AB30F020AA5
ark:/67375/WNG-15CDWBWT-J
SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-2
content type line 23
ObjectType-Article-1
ObjectType-Feature-2
ISSN:0007-1102
2044-8317
DOI:10.1348/000711006X126600