When coders are reliable: The application of three measures to assess inter-rater reliability/agreement with doctor–patient communication data coded with the VR-CoDES

To investigate whether different measures of inter-rater reliability will compute similar estimates with nominal data commonly encountered in communication studies. To make recommendations how reliability should be computed and described for communication coding instruments. The raw data from an int...

Full description

Saved in:
Bibliographic Details
Published in:Patient education and counseling Vol. 82; no. 3; pp. 341 - 345
Main Authors: Fletcher, Ian, Mazzi, Mariangela, Nuebling, Matthias
Format: Journal Article
Language:English
Published: Oxford Elsevier Ireland Ltd 01.03.2011
Elsevier
Subjects:
Biological and medical sciences
Clinical Coding - methods
Clinical Coding - standards
Clinical Competence - standards
Coding
Communication
Doctor-Patient communication
Guidance
Humans
Inter-rater study
Internal Medicine
Intraclass correlation coefficient
Kappa
Medical sciences
Miscellaneous
Observer Variation
Physician-Patient Relations
Public health. Hygiene
Public health. Hygiene-occupational medicine
Reliability
Reproducibility of Results
Research Design
Sensitivity
Sensitivity analysis
Sensitivity and specificity
Videotape Recording
VR-CoDES
Sensitivity and specificity
Intraclass correlation coefficient
Inter-rater study
VR-CoDES
Kappa
Statistical data
Evaluation
Coefficient
Physician patient relation
Computerized processing
Reliability
Statistics
Data gathering
Communication
ISSN:0738-3991, 1873-5134, 1873-5134
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To investigate whether different measures of inter-rater reliability will compute similar estimates with nominal data commonly encountered in communication studies. To make recommendations how reliability should be computed and described for communication coding instruments. The raw data from an inter-rater study with three coders were analysed with; Cohen's κ, sensitivity and specificity measures, Fleiss's multirater κ j , and an intraclass correlation coefficient (ICC). Minor differences were found between Cohen's κ and an ICC model across paired data (largest margin = 0.01). There were negligible differences between the multirater estimates e.g. κ j (0.52) and ICC (0.53). Sensitivity analyses were in general agreement with the multirater estimates. It is more practical to analyse nominal data with >2 raters with an appropriate model ICC for inter-rater studies, and little difference exists between Cohen's κ or an ICC. Alternatives to Cohen's κ are readily available, but researchers need to be aware of the different ICC definitions. An ICC model should be fully described in reports. Investigators are encouraged to supply confidence limits with inter-rater data, and to revisit guidance regarding the relative strengths of agreement of reliability coefficients.
Bibliography:ObjectType-Article-2
SourceType-Scholarly Journals-1
ObjectType-Undefined-1
ObjectType-Feature-3
content type line 23
ObjectType-Article-1
ObjectType-Feature-2
ISSN:0738-3991
1873-5134
1873-5134
DOI:10.1016/j.pec.2011.01.004