On Designing Paired Comparison Experiments for Subjective Multimedia Quality Assessment

This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on multimedia Vol. 16; no. 2; pp. 564 - 571
Main Author: Lee, Jong-Seok
Format: Journal Article
Language:English
Published: New York, NY IEEE 01.02.2014
Institute of Electrical and Electronics Engineers
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Applied sciences
Bradley-Terry model
Computer science; control theory; systems
Computer simulation
Convergence
Correlation coefficients
Encoding
Errors
Exact sciences and technology
Mathematical models
mean opinion score (MOS)
Multimedia
Multimedia communication
paired comparison
Quality assessment
Quality of experience (QoE)
Roots
Simulation
Software
Streaming media
subjective quality assessment
Video coding
Correlation coefficient
Performance evaluation
Multimedia
Paired comparison
Computer simulation
Bradley-Terry model
Case based reasoning
Modeling
Subjective evaluation
Mean square error
Human error
subjective quality assessment
mean opinion score (MOS)
Quality of experience (QoE)
Service quality
ISSN:1520-9210, 1941-0077
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined.
Bibliography:ObjectType-Article-2
SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
content type line 23
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2013.2292590