On Designing Paired Comparison Experiments for Subjective Multimedia Quality Assessment
This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment...
Saved in:
| Published in: | IEEE transactions on multimedia Vol. 16; no. 2; pp. 564 - 571 |
|---|---|
| Main Author: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York, NY
IEEE
01.02.2014
Institute of Electrical and Electronics Engineers The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 1520-9210, 1941-0077 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined. |
|---|---|
| AbstractList | This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined. |
| Author | Jong-Seok Lee |
| Author_xml | – sequence: 1 givenname: Jong-Seok surname: Lee fullname: Lee, Jong-Seok |
| BackLink | http://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=28178415$$DView record in Pascal Francis |
| BookMark | eNp9kU1r3DAQhkVJoUnae6EXQSn04u2MbEv2MWzTD8iSlqb0aMayFLR4pa0kl-bfV8suOeTQ0wzM887Xe8HOfPCGsdcIK0ToP9xtNisBWK-E6EXbwzN2jn2DFYBSZyVvBVS9QHjBLlLaAmDTgjpnv249_2iSu_fO3_Nv5KKZ-Drs9hRdCp5f_92b6HbG58RtiPzHMm6Nzu6P4ZtlzqUyOeLfF5pdfuBXKZmUDvRL9tzSnMyrU7xkPz9d362_VDe3n7-ur24q3QjIlVW2thNODfVKE4ydHMUEiKqrOw0olcFRgGiVoVZoOcI44jSSIEu91VbUl-z9se8-ht-LSXnYuaTNPJM3YUnD4e6-lQhQ0LdP0G1Yoi_bFQqaBkQjsVDvThQlTbON5LVLw748geLDILqyW4Nt4eDI6RhSisY-IgjDwZGhODIcHBlOjhSJfCLRLlN2wedIbv6f8M1R6Iwxj3OkVG2nZP0PndWaQg |
| CODEN | ITMUF8 |
| CitedBy_id | crossref_primary_10_1145_3176648 crossref_primary_10_1109_TMM_2015_2484963 crossref_primary_10_1109_TMM_2016_2574623 crossref_primary_10_3390_s17020321 crossref_primary_10_1109_TIP_2014_2387416 crossref_primary_10_1016_j_compind_2025_104358 crossref_primary_10_1109_TMM_2016_2557728 crossref_primary_10_1016_j_neucom_2022_07_065 crossref_primary_10_1109_TIP_2025_3539477 crossref_primary_10_1109_TMM_2023_3327613 crossref_primary_10_1109_TSMC_2016_2531654 crossref_primary_10_1109_TCSVT_2022_3208100 crossref_primary_10_1109_TMM_2020_3032026 crossref_primary_10_1109_TBC_2020_3047218 crossref_primary_10_1109_TCSVT_2022_3164918 crossref_primary_10_1109_MWC_2015_7054721 crossref_primary_10_1109_TMM_2016_2559942 crossref_primary_10_1109_TMM_2023_3330096 crossref_primary_10_1016_j_image_2021_116150 crossref_primary_10_1186_s13638_020_01670_z crossref_primary_10_1109_TMM_2024_3391907 crossref_primary_10_1109_TBC_2018_2828607 crossref_primary_10_1109_TMM_2016_2581582 crossref_primary_10_1109_TMM_2017_2711860 crossref_primary_10_1109_TMM_2021_3103394 crossref_primary_10_1109_TBC_2018_2832465 |
| Cites_doi | 10.1109/QOMEX.2009.5246961 10.1117/1.2181547 10.1109/TBC.2008.2000733 10.1145/1882261.1866186 10.2307/2334029 10.1109/TIP.2010.2042111 10.1109/GLOCOMW.2012.6477780 10.1109/ICIP.2012.6466938 10.2307/2332613 10.1109/MNET.2010.5430141 10.1109/TMM.2011.2157333 10.1145/1631272.1631339 10.1214/12-STS396 10.1016/j.displa.2004.09.003 10.1016/j.ress.2010.02.013 10.1007/s11042-012-1011-6 10.1109/TMM.2012.2190924 10.1214/aos/1018031267 10.2737/RMRS-GTR-216 10.1109/TCSVT.2010.2041829 10.1109/JSTSP.2012.2215306 |
| ContentType | Journal Article |
| Copyright | 2015 INIST-CNRS Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Feb 2014 |
| Copyright_xml | – notice: 2015 INIST-CNRS – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Feb 2014 |
| DBID | 97E RIA RIE AAYXX CITATION IQODW 7SC 7SP 8FD JQ2 L7M L~C L~D F28 FR3 |
| DOI | 10.1109/TMM.2013.2292590 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Pascal-Francis Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional ANTE: Abstracts in New Technology & Engineering Engineering Research Database |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional Engineering Research Database ANTE: Abstracts in New Technology & Engineering |
| DatabaseTitleList | Technology Research Database Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science Applied Sciences |
| EISSN | 1941-0077 |
| EndPage | 571 |
| ExternalDocumentID | 3237418401 28178415 10_1109_TMM_2013_2292590 6675876 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Basic Science Research Program grantid: 2013R1A1A1007822 – fundername: Ministry of Science, ICT and Future Planning (MSIP), Korea |
| GroupedDBID | -~X 0R~ 29I 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD HZ~ H~9 IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P PQQKQ RIA RIE RNS TN5 VH1 ZY4 AAYXX CITATION ABTAH IQODW 7SC 7SP 8FD JQ2 L7M L~C L~D F28 FR3 |
| ID | FETCH-LOGICAL-c420t-f7f3fd1d4a97ca0b86b2d0117838c0167e1b20257ea52c6b0bb1dba2afa9fcf23 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 32 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000330245800023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1520-9210 |
| IngestDate | Sun Sep 28 11:20:36 EDT 2025 Sun Nov 09 06:38:14 EST 2025 Wed Apr 02 07:25:10 EDT 2025 Tue Nov 18 22:11:51 EST 2025 Sat Nov 29 02:03:46 EST 2025 Tue Aug 26 16:49:27 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 2 |
| Keywords | Correlation coefficient Performance evaluation Multimedia Paired comparison Computer simulation Bradley-Terry model Case based reasoning Modeling Subjective evaluation Mean square error Human error subjective quality assessment mean opinion score (MOS) Quality of experience (QoE) Service quality |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html CC BY 4.0 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c420t-f7f3fd1d4a97ca0b86b2d0117838c0167e1b20257ea52c6b0bb1dba2afa9fcf23 |
| Notes | ObjectType-Article-2 SourceType-Scholarly Journals-1 ObjectType-Feature-1 content type line 14 content type line 23 |
| PQID | 1504402461 |
| PQPubID | 75737 |
| PageCount | 8 |
| ParticipantIDs | proquest_journals_1504402461 crossref_primary_10_1109_TMM_2013_2292590 ieee_primary_6675876 proquest_miscellaneous_1520956100 crossref_citationtrail_10_1109_TMM_2013_2292590 pascalfrancis_primary_28178415 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-02-01 |
| PublicationDateYYYYMMDD | 2014-02-01 |
| PublicationDate_xml | – month: 02 year: 2014 text: 2014-02-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | New York, NY |
| PublicationPlace_xml | – name: New York, NY – name: Piscataway |
| PublicationTitle | IEEE transactions on multimedia |
| PublicationTitleAbbrev | TMM |
| PublicationYear | 2014 |
| Publisher | IEEE Institute of Electrical and Electronics Engineers The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: Institute of Electrical and Electronics Engineers – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 simons (ref12) 1999; 27 ref15 ref14 ref11 ref10 ref1 ref16 ref19 (ref2) 2002 brown (ref9) 2009 rubinstein (ref6) 2010; 29 ref23 ref25 ref20 ref22 ref21 david (ref17) 1969 marden (ref18) 1995 ref8 ref7 ref4 ref3 ref5 ponomarenko (ref24) 2009; 10 |
| References_xml | – year: 2002 ident: ref2 publication-title: Methodology for the Subjective Assessment of the Quality of Television Pictures – ident: ref7 doi: 10.1109/QOMEX.2009.5246961 – ident: ref11 doi: 10.1117/1.2181547 – volume: 10 start-page: 30 year: 2009 ident: ref24 article-title: TID2008- a database for evaluation of full-reference visual quality assessment metrics publication-title: Adv Modern Radioelectron – year: 1995 ident: ref18 publication-title: Analyzing and Modeling Rank Data – ident: ref1 doi: 10.1109/TBC.2008.2000733 – volume: 29 start-page: 160:1 year: 2010 ident: ref6 article-title: A comparative study of image retargeting publication-title: ACM Trans Graph doi: 10.1145/1882261.1866186 – ident: ref16 doi: 10.2307/2334029 – ident: ref20 doi: 10.1109/TIP.2010.2042111 – ident: ref23 doi: 10.1109/GLOCOMW.2012.6477780 – year: 1969 ident: ref17 publication-title: The Method of Paired Comparisons – ident: ref14 doi: 10.1109/ICIP.2012.6466938 – ident: ref21 doi: 10.2307/2332613 – ident: ref3 doi: 10.1109/MNET.2010.5430141 – ident: ref4 doi: 10.1109/TMM.2011.2157333 – ident: ref22 doi: 10.1145/1631272.1631339 – ident: ref25 doi: 10.1214/12-STS396 – ident: ref10 doi: 10.1016/j.displa.2004.09.003 – ident: ref15 doi: 10.1016/j.ress.2010.02.013 – ident: ref5 doi: 10.1007/s11042-012-1011-6 – ident: ref13 doi: 10.1109/TMM.2012.2190924 – volume: 27 start-page: 1041 year: 1999 ident: ref12 article-title: Asymptotics when the number of parameters tends to infinity in the Bradley-Terry model for paired comparisons publication-title: Ann Statist doi: 10.1214/aos/1018031267 – year: 2009 ident: ref9 publication-title: An Enquiry into the Method of Paired Comparison Reliability Scaling and Thurstone's Law of Comparative Judgment doi: 10.2737/RMRS-GTR-216 – ident: ref19 doi: 10.1109/TCSVT.2010.2041829 – ident: ref8 doi: 10.1109/JSTSP.2012.2215306 |
| SSID | ssj0014507 |
| Score | 2.2308104 |
| Snippet | This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the... |
| SourceID | proquest pascalfrancis crossref ieee |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 564 |
| SubjectTerms | Applied sciences Bradley-Terry model Computer science; control theory; systems Computer simulation Convergence Correlation coefficients Encoding Errors Exact sciences and technology Mathematical models mean opinion score (MOS) Multimedia Multimedia communication paired comparison Quality assessment Quality of experience (QoE) Roots Simulation Software Streaming media subjective quality assessment Video coding |
| Title | On Designing Paired Comparison Experiments for Subjective Multimedia Quality Assessment |
| URI | https://ieeexplore.ieee.org/document/6675876 https://www.proquest.com/docview/1504402461 https://www.proquest.com/docview/1520956100 |
| Volume | 16 |
| WOSCitedRecordID | wos000330245800023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Xplore customDbUrl: eissn: 1941-0077 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014507 issn: 1520-9210 databaseCode: RIE dateStart: 19990101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS-UwEB9U9qAH3fUDu6sSwYtgNU3TjxzFD_bix0HRW0maBBTpE997gv-9M2lfWHERvLUkpSGTmUwyM78fwJ5WZSGdtWlu8Gwi6f7G-Fyn6NrTDpQLX9lANlFdXtb39-p6Dg5iLYxzLiSfuUN6DLF8O2qndFVGzH0Fau88zFdV2ddqxYiBLEJpNG5HPFV4jpmFJLk6urm4oByu_FAIhd4-_7AFBU4VyojUY5wU37NZfDLMYbc5X_neOH_C8uBVsuN-GfyCOdetwsqMsYENCrwKS__AD67B3VXHTkMGB76ya43Wz7KTSEzIziL6_5ihb8vQyDz29pGFut1QdcJ6FI43dhwxPtfh9vzs5uRvOhAtpK0UfJL6yufeZlZqVbWam7o0whJYXJ3XLdUpuAwlicrtdCHa0nBjMmu00F4r33qRb8BCN-rcJrDaSCPQZlEAUFaqrJUtVaW1USYz3ucJHM3mvmkHFHIiw3hqwmmEqwal1ZC0mkFaCezHL557BI4v-q6RNGK_QRAJ7HwQb2wXdUZx1yKBrZm8m0GHxw26yhIXryyzBHZjM2ofhVR050ZT6iMIyTHj_Pf_f_0HFnGAss_z3oKFycvUbcOP9nXyMH7ZCUv4HWII7ss |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fSxwxEB6sLVQfaqsVV61NoS-Frmaz2R95FKtY6l19uFLflmSTQEX2xLsT_O-dye6FSkuhb7sky4Z8mckkM_MNwEetykI6a9Pc4NlE0v2N8blO0bSnHSgXvrKh2EQ1HtdXV-pyBT7HXBjnXAg-c4f0GHz5dtou6KqMKvcVKL3P4HkhpeB9tlb0GcgiJEfjhsRThSeZpVOSq6PJaERRXPmhEArtff5kEwpVVSgmUs9wWnxfz-IP1Rz2m7ON_xvpa3g12JXsuF8Ib2DFdZuwsazZwAYR3oT13wgIt-Dn9459CTEc-MouNeo_y05iaUJ2Gvn_ZwytW4Zq5rrXkCxk7oa8E9bzcDyw48jy-RZ-nJ1OTs7TodRC2uJEzlNf-dzbzEqtqlZzU5dGWKKLq_O6pUwFlyGWKN5OF6ItDTcms0YL7bXyrRf5Nqx2087tAKuNNAK1FrkAZaXKWtlSVVobZTLjfZ7A0XLum3bgIadyGDdNOI9w1SBaDaHVDGgl8Cl-cdtzcPyj7xahEfsNQCRw8ATe2C7qjDyvRQL7S7ybQYpnDRrLEpevLLMEPsRmlD9yqujOTRfURxCXY8b57t9__R5enk9GF83F1_G3PVjDwco-6nsfVud3C_cOXrT381-zu4OwnB8B7tPyEg |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+Designing+Paired+Comparison+Experiments+for+Subjective+Multimedia+Quality+Assessment&rft.jtitle=IEEE+transactions+on+multimedia&rft.au=Jong-Seok+Lee&rft.date=2014-02-01&rft.pub=IEEE&rft.issn=1520-9210&rft.volume=16&rft.issue=2&rft.spage=564&rft.epage=571&rft_id=info:doi/10.1109%2FTMM.2013.2292590&rft.externalDocID=6675876 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1520-9210&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1520-9210&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1520-9210&client=summon |