Enhancing Simulations With Intra-subject Variability for Improved Psychophysical Assessments
Psychometric properties of perceptual assessments, like reliability, depend on the stochastic properties of psychophysical sampling procedures resulting in method variability, as well as inter- and intra-subject variability. Method variability is commonly minimized by optimizing sampling procedures...
Saved in:
| Published in: | bioRxiv |
|---|---|
| Main Authors: | , , |
| Format: | Paper |
| Language: | English |
| Published: |
Cold Spring Harbor
Cold Spring Harbor Laboratory Press
21.03.2018
Cold Spring Harbor Laboratory |
| Edition: | 1.1 |
| Subjects: | |
| ISSN: | 2692-8205, 2692-8205 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Psychometric properties of perceptual assessments, like reliability, depend on the stochastic properties of psychophysical sampling procedures resulting in method variability, as well as inter- and intra-subject variability. Method variability is commonly minimized by optimizing sampling procedures through computer simulations. Inter-subject variability is inherent in the population of interest and cannot be acted upon. In contrast, intra-subject variability introduced by confounds cannot be simply quantified from experimental data, as this data also includes method variability. Therefore, this aspect is generally neglected when developing assessments. Yet, comparing method variability and intra-subject variability could give insights on whether effort should be invested in optimizing the sampling procedure, or in addressing potential confounds instead. We propose a new approach to estimate and model intra-subject variability of psychometric functions by combining computer simulations and behavioral data, and to account for it when simulating experiments. The approach was illustrated in a real-world scenario of proprioceptive difference threshold assessments. The behavioral study revealed a test-retest reliability of 0.212. Computer simulations lacking intra-subject variability predicted a reliability of 0.768, whereas the new approach including an intra-subject variability model lead to a realistic estimate of reliability (0.207). Such a model also allows computing the theoretically maximally attainable reliability (0.552) assuming an ideal sampling procedure. Comparing the reliability estimates when exclusively accounting for method variability versus intra-subject variability reveals that intra-subject variability should be reduced by addressing confounds and that only optimizing the sampling procedure may be insufficient to achieve a high reliability. The new approach also allows accelerating the development of assessments by simulating the converging behavior of the reliability confidence interval with a large number of subjects and retests without requiring additional experiments. Having such a tool of predictive value is especially important for target populations where time is scarce, such as for assessments in clinical settings. |
|---|---|
| AbstractList | Psychometric properties of perceptual assessments, like reliability, depend on the stochastic properties of psychophysical sampling procedures resulting in method variability, as well as inter- and intra-subject variability. Method variability is commonly minimized by optimizing sampling procedures through computer simulations. Inter-subject variability is inherent in the population of interest and cannot be acted upon. In contrast, intra-subject variability introduced by confounds cannot be simply quantified from experimental data, as this data also includes method variability. Therefore, this aspect is generally neglected when developing assessments. Yet, comparing method variability and intra-subject variability could give insights on whether effort should be invested in optimizing the sampling procedure, or in addressing potential confounds instead. We propose a new approach to estimate and model intra-subject variability of psychometric functions by combining computer simulations and behavioral data, and to account for it when simulating experiments. The approach was illustrated in a real-world scenario of proprioceptive difference threshold assessments. The behavioral study revealed a test-retest reliability of 0.212. Computer simulations lacking intra-subject variability predicted a reliability of 0.768, whereas the new approach including an intra-subject variability model lead to a realistic estimate of reliability (0.207). Such a model also allows computing the theoretically maximally attainable reliability (0.552) assuming an ideal sampling procedure. Comparing the reliability estimates when exclusively accounting for method variability versus intra-subject variability reveals that intra-subject variability should be reduced by addressing confounds and that only optimizing the sampling procedure may be insufficient to achieve a high reliability. The new approach also allows accelerating the development of assessments by simulating the converging behavior of the reliability confidence interval with a large number of subjects and retests without requiring additional experiments. Having such a tool of predictive value is especially important for target populations where time is scarce, such as for assessments in clinical settings. |
| Author | Gassert, Roger Lambercy, Olivier Rinderknecht, Mike Domenik |
| Author_xml | – sequence: 1 givenname: Mike surname: Rinderknecht middlename: Domenik fullname: Rinderknecht, Mike Domenik – sequence: 2 givenname: Olivier surname: Lambercy fullname: Lambercy, Olivier – sequence: 3 givenname: Roger surname: Gassert fullname: Gassert, Roger |
| BookMark | eNpNkF1LwzAARYNMcM75D4SAz9V8tEn6OMacg4GCQ1-EkqSpzWiTmaTD_nsH88Gne-HA5XCvwcR5ZwC4xegBY4QfiSio4BdgSlhJMkFQMfnXr8A8xj1CiJQMU55PwefKtdJp677gm-2HTibrXYQfNrVw41KQWRzU3ugE32WwUtnOphE2PsBNfwj-aGr4Gkfd-kM7RqtlBxcxmhh741K8AZeN7KKZ_-UM7J5Wu-Vztn1Zb5aLbaYE4RnHmucckZxIVhfCaEyZqQnCda5rWtCyoazkiipCuWmwbpiiZW0wZzJXhcR0Bu7Os8r68GOP1SHYXoaxOp9x4vdnfhL-HkxM1d4PwZ2MKoKYKAVmSNBf_UhgGQ |
| ContentType | Paper |
| Copyright | 2018. This article is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ ( the License ). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. 2018, Posted by Cold Spring Harbor Laboratory |
| Copyright_xml | – notice: 2018. This article is published under http://creativecommons.org/licenses/by-nc-nd/4.0/ ( the License ). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. – notice: 2018, Posted by Cold Spring Harbor Laboratory |
| DBID | 8FE 8FH ABUWG AFKRA AZQEC BBNVY BENPR BHPHI CCPQU DWQXO GNUQQ HCIFZ LK8 M7P PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS FX. |
| DOI | 10.1101/285387 |
| DatabaseName | ProQuest SciTech Collection ProQuest Natural Science Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland ProQuest Central Essentials - QC Biological Science Collection ProQuest Central Natural Science Collection ProQuest One ProQuest Central ProQuest Central Student SciTech Premium Collection Biological Sciences Biological Science Database ProQuest Central Premium ProQuest One Academic Publicly Available Content Database (ProQuest) ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China bioRxiv |
| DatabaseTitle | Publicly Available Content Database ProQuest Central Student ProQuest One Academic Middle East (New) ProQuest Biological Science Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Natural Science Collection Biological Science Database ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences ProQuest One Academic UKI Edition Natural Science Collection ProQuest Central Korea Biological Science Collection ProQuest Central (New) ProQuest One Academic ProQuest One Academic (New) |
| DatabaseTitleList | Publicly Available Content Database |
| Database_xml | – sequence: 1 dbid: PIMPY name: Publicly Available Content Database url: http://search.proquest.com/publiccontent sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Biology |
| EISSN | 2692-8205 |
| Edition | 1.1 |
| ExternalDocumentID | 285387v1 |
| Genre | Working Paper/Pre-Print |
| GroupedDBID | 8FE 8FH ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BBNVY BENPR BHPHI CCPQU DWQXO GNUQQ HCIFZ LK8 M7P NQS PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PROAC RHI FX. |
| ID | FETCH-LOGICAL-b827-71c7470242a6d58ec136ed201d4cd3539f3697b3b237ef1cf6b39de176a4b5a13 |
| IEDL.DBID | M7P |
| ISSN | 2692-8205 |
| IngestDate | Tue Jan 07 18:55:39 EST 2025 Fri Jul 25 09:20:27 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Keywords | method variability behavioral data proprioception assessment parameter estimation by sequential testing (PEST) psychophysics reliability |
| Language | English |
| License | This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at http://creativecommons.org/licenses/by-nc-nd/4.0 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-b827-71c7470242a6d58ec136ed201d4cd3539f3697b3b237ef1cf6b39de176a4b5a13 |
| Notes | SourceType-Working Papers-1 ObjectType-Working Paper/Pre-Print-1 content type line 50 |
| ORCID | 0000-0002-0760-7054 0000-0002-6373-8518 0000-0002-9825-8776 |
| OpenAccessLink | https://www.proquest.com/docview/2068981608?pq-origsite=%requestingapplication% |
| PQID | 2068981608 |
| PQPubID | 2050091 |
| PageCount | 10 |
| ParticipantIDs | biorxiv_primary_285387 proquest_journals_2068981608 |
| PublicationCentury | 2000 |
| PublicationDate | 20180321 |
| PublicationDateYYYYMMDD | 2018-03-21 |
| PublicationDate_xml | – month: 03 year: 2018 text: 20180321 day: 21 |
| PublicationDecade | 2010 |
| PublicationPlace | Cold Spring Harbor |
| PublicationPlace_xml | – name: Cold Spring Harbor |
| PublicationTitle | bioRxiv |
| PublicationYear | 2018 |
| Publisher | Cold Spring Harbor Laboratory Press Cold Spring Harbor Laboratory |
| Publisher_xml | – name: Cold Spring Harbor Laboratory Press – name: Cold Spring Harbor Laboratory |
| References | Prins, Kingdom (285387v1.18) 2009 de Vet, Terwee, Knol, Bouter (285387v1.3) 2006; 59 Gescheider (285387v1.6) 1997 Green (285387v1.7) 1993; 93 Findlay (285387v1.4) 1978; 23 Kontsevich, Tyler (285387v1.11) 1999; 39 Taylor, Douglas Creelman (285387v1.27) 1967; 41 Strasburger (285387v1.24) 2001; 63 Rinderknecht, Ranzani, Popp, Lambercy, Gassert (285387v1.21) 2018 Lexell, Downham (285387v1.13) 2005; 84 King-Smith, Grigsby, Vingrys, Benes, Supowit (285387v1.10) 1994; 34 Watson, Fitzhugh (285387v1.28) 1990; 47 Rinderknecht, Lambercy, Raible, Liepert, Gassert (285387v1.19) 2017; 12 Simpson (285387v1.23) 1989; 45 Pentland (285387v1.17) 1980; 28 Leek, Hanna, Marshall (285387v1.12) 1991; 90 Madigan, Williams (285387v1.15) 1987; 42 Rinderknecht, Popp, Lambercy, Gassert (285387v1.20) 2014 Shrout, Fleiss (285387v1.22) 1979; 86 Wichmann, Hill (285387v1.29) 2001; 63 Taylor (285387v1.26) 1971; 49 Macmillan, Douglas Creelman (285387v1.14) 2005 Kaernbach (285387v1.9) 1991; 49 Fründ, Haenel, Wichmann (285387v1.5) 2011; 11 Chapuis, De Grave, Lambercy, Gassert (285387v1.1) 2010 Cohen, Maunsell (285387v1.2) 2011; 31 Hall (285387v1.8) 1981; 69 Streiner, Norman (285387v1.25) 2008 Oldfield (285387v1.16) 1971; 9 |
| References_xml | – volume: 86 start-page: 420 year: 1979 end-page: 428 ident: 285387v1.22 article-title: Intraclass correlations: uses in assessing rater reliability publication-title: Psychol Bull – volume: 42 start-page: 240 year: 1987 end-page: 249 ident: 285387v1.15 article-title: Maximum-likelihood psychometric procedures in two-alternative forced-choice: evaluation and recommendations publication-title: Perception & Psychophysics – volume: 90 start-page: 1385 year: 1991 end-page: 1397 ident: 285387v1.12 article-title: An interleaved tracking procedure to monitor unstable psychometric functions publication-title: The Journal of the Acoustical Society of America – volume: 45 start-page: 572 year: 1989 end-page: 576 ident: 285387v1.23 article-title: The step method: A new adaptive psychophysical procedure publication-title: Perception & Psychophysics – volume: 84 start-page: 719 year: 2005 end-page: 723 ident: 285387v1.13 article-title: How to assess the reliability of measurements in rehabilitation publication-title: Am J Phys Med Rehabil – year: 1997 ident: 285387v1.6 publication-title: Psychophysics: The Fundamentals – volume: 59 start-page: 1033 year: 2006 end-page: 1039 ident: 285387v1.3 article-title: When to use agreement versus reliability measures publication-title: J Clin Epidemiol – volume: 93 start-page: 2096 year: 1993 end-page: 2105 ident: 285387v1.7 article-title: A maximum-likelihood method for estimating thresholds in a yes-no task publication-title: The Journal of the Acoustical Society of America – volume: 49 start-page: 505 issue: Suppl 2 year: 1971 ident: 285387v1.26 article-title: On the efficiency of psychophysical measurement publication-title: The Journal of the Acoustical Society of America – year: 2008 ident: 285387v1.25 publication-title: Health measurement scales: a practical guide to their development and use – volume: 9 start-page: 97 year: 1971 end-page: 113 ident: 285387v1.16 article-title: The assessment and analysis of handedness: the Edinburgh inventory publication-title: Neuropsychologia – volume: 49 start-page: 227 year: 1991 end-page: 229 ident: 285387v1.9 article-title: Simple adaptive testing with the weighted up-down method publication-title: Perception & Psychophysics – volume: 39 start-page: 2729 year: 1999 end-page: 2737 ident: 285387v1.11 article-title: Bayesian adaptive estimation of psychometric slope and threshold publication-title: Vision Research – volume: 47 start-page: 87 year: 1990 end-page: 91 ident: 285387v1.28 article-title: The method of constant stimuli is inefficient publication-title: Perception & Psychophysics – year: 2005 ident: 285387v1.14 publication-title: Detection Theory: A User’s Guide – volume: 28 start-page: 377 year: 1980 end-page: 379 ident: 285387v1.17 article-title: Maximum likelihood estimation: The best PEST publication-title: Attention, Perception, & Psychophysics – start-page: 3 year: 2014 end-page: 10 ident: 285387v1.20 publication-title: Lecture Notes in Computer Science – volume: 69 start-page: 1763 year: 1981 ident: 285387v1.8 article-title: Hybrid adaptive procedure for estimation of psychometric functions publication-title: The Journal of the Acoustical Society of America – volume: 63 start-page: 1293 year: 2001 end-page: 1313 ident: 285387v1.29 article-title: The psychometric function: I. Fitting, sampling, and goodness of fit publication-title: Perception & Psychophysics – year: 2018 ident: 285387v1.21 article-title: Method for Improving Psychophysical Threshold Estimates by Detecting Sustained Inattention publication-title: bioRxiv – year: 2009 ident: 285387v1.18 publication-title: Palamedes: Matlab routines for analyzing psychophysical data – volume: 31 start-page: 15802 year: 2011 end-page: 15806 ident: 285387v1.2 article-title: When attention wanders: how uncontrolled fluctuations in attention affect performance publication-title: J Neurosci – volume: 34 start-page: 885 year: 1994 end-page: 912 ident: 285387v1.10 article-title: Efficient and unbiased modifications of the QUEST threshold method: theory, simulations, experimental evaluation and practical implementation publication-title: Vision Research – volume: 41 start-page: 782 year: 1967 ident: 285387v1.27 article-title: PEST: Efficient estimates on probability functions publication-title: The Journal of the Acoustical Society of America – volume: 63 start-page: 1348 year: 2001 end-page: 1355 ident: 285387v1.24 article-title: Converting between measures of slope of the psychometric function publication-title: Perception & Psychophysics – volume: 23 start-page: 181 year: 1978 end-page: 185 ident: 285387v1.4 article-title: Estimates on probability functions: A more virulent PEST publication-title: Attention, Perception, & Psychophysics – volume: 12 start-page: 635 year: 2017 end-page: 643 ident: 285387v1.19 article-title: Age-based model for metacar-pophalangeal joint proprioception in elderly publication-title: Clin Interv Aging – volume: 11 year: 2011 ident: 285387v1.5 article-title: Inference for psychometric functions in the presence of nonstationary behavior publication-title: J Vis – start-page: 417 year: 2010 end-page: 424 ident: 285387v1.1 publication-title: In Haptics Symposium, 2010 IEEE |
| SSID | ssj0002961374 |
| Score | 1.5303633 |
| SecondaryResourceType | preprint |
| Snippet | Psychometric properties of perceptual assessments, like reliability, depend on the stochastic properties of psychophysical sampling procedures resulting in... |
| SourceID | biorxiv proquest |
| SourceType | Open Access Repository Aggregation Database |
| SubjectTerms | Mathematical models Neuroscience Proprioception Psychophysics Quantitative psychology Sampling Simulation |
| Title | Enhancing Simulations With Intra-subject Variability for Improved Psychophysical Assessments |
| URI | https://www.proquest.com/docview/2068981608 https://www.biorxiv.org/content/10.1101/285387 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1NS8MwGA66OfDk53A6JQevxTZpk-YkKhsOdBQdc4JQkjR1PdjNdRvu35u02TwIXjyWXMKbvF9P3j4PAJccBcJU4g5n2t18l6cOSxlxFKah7ogEx6Uky_CB9vvhaMQiC7gVdqxyHRPLQJ1MpMHIdZNOQhZ6xA2vp5-OUY0yr6tWQmMb1A1LAi5H96INxoKYTlYlETMiTDs-cgMrL6Qv4hXSmcpM0jVENpl9Zctf0bhMMd29_25uH9QjPlWzA7Cl8kPQqEQmV0fgrZOPDalG_g6fsw-r1lXAl2w-hj0D7TrFQhg0Bg5131zRdq-grmVhBTioBFZRcmqPFN5s2DyLYzDodgZ3947VVHBEiKhDPan7B5OXOUmCUEkPE5XoIiDxZYIDzFJMGBVYIExV6smUCMwS5VHCfRFwDzdBLZ_k6gRAN0Cc6uzOfV1SlargikrJsBfIRAYub4GmNWw8rYgz4sriLdBeGzG2DlPEPxY8_Xv5DOzq7Za_BSKvDWrz2UKdgx25nGfF7ALUbzv96OmivAf6K-o9Rq_fl2a8Ng |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1NT9wwEB3RBQQnPlrEx7b1AY5RE3tjx4cKIQpixXa1EivggBTZjgM5kF02y8f-qP5Hxk6WHir1xoFzpCjJG897noznAewrGmunxAMlcbl1QpUHMpc8sEwkuCPSinlLlsue6PeT62s5WIA_87Mwrq1ynhN9os5GxtXIcZPOE5lEPEwOxw-Bc41yf1fnFhp1WJzb2TNu2aqf3V-I7wGlpyfD47OgcRUIdEJFICKDCtoxk-JZnFgTMW4zpMGsYzIWM5kzLoVmmjJh88jkXDOZ2Uhw1dGxihje9hMsooqgie8UHLyVdKhEbvRznymXmGdoGDduRhj3PygSo2vcW9bFaPJSPP2T_D2jna59sG-xDosDNbaTDViw5SYs1xaas89wc1LeuZEh5S25KO4bL7KKXBXTO9J1heugetSu1kQuFa413wo8I6jUSV1OsRmpOWDcBCw5eptVWn2B4Xu8zxa0ylFpt4GEMVUCtYvqoGD0nudWGCNZFJvMxKHaga0Gx3RcjwVJa4B3oD3HLG3SQZX-BWz3_5e_w8rZ8Hcv7XX753uwio_uD0DSqA2t6eTRfoUl8zQtqsk3H3oE0neG9xUuPBOh |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enhancing+Simulations+With+Intra-subject+Variability+for+Improved+Psychophysical+Assessments&rft.jtitle=bioRxiv&rft.au=Rinderknecht%2C+Mike+Domenik&rft.au=Lambercy%2C+Olivier&rft.au=Gassert%2C+Roger&rft.date=2018-03-21&rft.pub=Cold+Spring+Harbor+Laboratory+Press&rft.issn=2692-8205&rft.eissn=2692-8205&rft_id=info:doi/10.1101%2F285387 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2692-8205&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2692-8205&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2692-8205&client=summon |