Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concern...

Full description

Saved in:

Bibliographic Details
Published in:	Behavior research methods Vol. 55; no. 8; pp. 3953 - 3964
Main Authors:	Hauser, David J., Moss, Aaron J., Rosenzweig, Cheskie, Jaffe, Shalom N., Robinson, Jonathan, Litman, Leib
Format:	Journal Article
Language:	English
Published:	New York Springer US 01.12.2023
Subjects:	Attention Behavioral Science and Psychology Cognitive Psychology Crowdsourcing - methods Data Accuracy Humans Psychology Self Report Surveys and Questionnaires Response bias Data quality Participant recruitment Test validity
ISSN:	1554-3528, 1554-3528
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. In a pre-registered study, participants ( N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk’s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.
AbstractList	Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch's vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk. Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. In a pre-registered study, participants ( N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk’s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk. Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch's vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch's vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.
Author	Robinson, Jonathan Rosenzweig, Cheskie Litman, Leib Hauser, David J. Jaffe, Shalom N. Moss, Aaron J.
Author_xml	– sequence: 1 givenname: David J. orcidid: 0000-0002-5480-5213 surname: Hauser fullname: Hauser, David J. email: david.hauser@queensu.ca organization: Department of Psychology, Queen’s University – sequence: 2 givenname: Aaron J. orcidid: 0000-0003-4396-4128 surname: Moss fullname: Moss, Aaron J. organization: CloudResearch – sequence: 3 givenname: Cheskie orcidid: 0000-0003-2403-6940 surname: Rosenzweig fullname: Rosenzweig, Cheskie organization: CloudResearch, Department of Clinical Psychology, Columbia University – sequence: 4 givenname: Shalom N. orcidid: 0000-0002-7523-5213 surname: Jaffe fullname: Jaffe, Shalom N. organization: CloudResearch, Department of Psychology, Lander College – sequence: 5 givenname: Jonathan orcidid: 0000-0001-5353-7969 surname: Robinson fullname: Robinson, Jonathan organization: CloudResearch, Department of Computer Science, Lander College – sequence: 6 givenname: Leib orcidid: 0000-0001-6598-2701 surname: Litman fullname: Litman, Leib organization: CloudResearch, Department of Psychology, Lander College
BackLink	https://www.ncbi.nlm.nih.gov/pubmed/36326997$$D View this record in MEDLINE/PubMed
BookMark	eNp9kMtKxDAUQIMovn_AhWTppppHM02WMviCEUF0HW7aVKuZZkwaGXf-hr_nlxidUcSFqwTuOXmcLbTa-94itEfJIa-EPIqUl0wWhLGCUKVUMV9Bm1SIsuCCydVf-w20FeMDIVwyWq6jDT7ibKRUtYn0yTO4BEPX3-Gx86m5ttFCqO_fX98iPp7Ngn-2DT4LPs0wRAw4epeGzve49QHnsXF2mv0aNzAAfkrguuEF5_nlTQqPO2itBRft7nLdRrenJzfj82JydXYxPp4UdSnkUFgKsjTAQRouKiMVKxsiiSGjRohGSJv_K6EiyihDKtPWRtVi1DJJGWfKlnwbHSzOzS96SjYOetrF2joHvfUpalZxWtEq0xndX6LJTG2jZ6GbQnjR31EywBZAHXyMwbY_CCX6s7xelNe5vP4qr-dZkn-kuhvgM9QQoHP_q3yhxnxPf2eDfvAp9DnXf9YHvSKZ2g
CitedBy_id	crossref_primary_10_1016_j_isci_2025_112833 crossref_primary_10_1177_20539517231188723 crossref_primary_10_1016_j_jenvp_2025_102608 crossref_primary_10_1016_j_ijhcs_2023_103103 crossref_primary_10_1016_j_watres_2024_121254 crossref_primary_10_1080_29984688_2025_2545754 crossref_primary_10_1007_s10862_024_10171_y crossref_primary_10_3758_s13421_025_01760_7 crossref_primary_10_1080_10410236_2023_2282833 crossref_primary_10_1177_02654075251317425 crossref_primary_10_1002_sd_70133 crossref_primary_10_1080_14681994_2025_2506085 crossref_primary_10_1371_journal_pone_0305364 crossref_primary_10_1177_00222437241268491 crossref_primary_10_1177_10731911241256439 crossref_primary_10_1080_10410236_2025_2484024 crossref_primary_10_1002_jeab_70038 crossref_primary_10_1016_j_paid_2025_113290 crossref_primary_10_1080_07421222_2024_2415774 crossref_primary_10_1016_j_jretconser_2025_104262 crossref_primary_10_1016_j_chb_2024_108342 crossref_primary_10_1016_j_jad_2024_12_084 crossref_primary_10_1093_pnasnexus_pgaf275 crossref_primary_10_2196_38552 crossref_primary_10_1007_s41233_025_00072_7 crossref_primary_10_1002_jcop_70005 crossref_primary_10_1108_JSM_12_2024_0634 crossref_primary_10_1007_s40615_025_02602_7 crossref_primary_10_1161_CIRCOUTCOMES_124_011069 crossref_primary_10_1177_21674795251323193 crossref_primary_10_1080_10696679_2024_2385374 crossref_primary_10_1007_s00146_024_02121_9 crossref_primary_10_1080_00952990_2025_2504152 crossref_primary_10_1016_j_appet_2024_107304 crossref_primary_10_1080_13546783_2024_2443149 crossref_primary_10_1007_s11109_024_09934_w crossref_primary_10_1016_j_electstud_2024_102760 crossref_primary_10_1080_09658211_2025_2467134 crossref_primary_10_2139_ssrn_4506020 crossref_primary_10_1016_j_brat_2024_104568 crossref_primary_10_1123_ijsc_2024_0205 crossref_primary_10_1017_jdm_2025_10 crossref_primary_10_1111_joms_13277 crossref_primary_10_1002_bdm_2353 crossref_primary_10_1017_S0034412525000198 crossref_primary_10_2196_70926 crossref_primary_10_1007_s11031_025_10133_1 crossref_primary_10_1038_s41597_025_05687_1 crossref_primary_10_1007_s11142_025_09902_7 crossref_primary_10_2196_63032 crossref_primary_10_1111_jasp_13081 crossref_primary_10_1111_1468_5973_12620 crossref_primary_10_3390_met15091012 crossref_primary_10_1007_s10508_025_03105_6 crossref_primary_10_1007_s12144_024_05895_9 crossref_primary_10_3758_s13428_023_02142_0 crossref_primary_10_1080_08959285_2024_2327656 crossref_primary_10_1521_soco_2024_42_3_233 crossref_primary_10_1891_RE_24_16 crossref_primary_10_1016_j_jocrd_2023_100806 crossref_primary_10_1111_1468_5973_12454 crossref_primary_10_1016_j_ecresq_2024_10_001 crossref_primary_10_1177_01461672241283862 crossref_primary_10_1177_08997640251343049 crossref_primary_10_1027_1864_9335_a000544 crossref_primary_10_1027_1866_5888_a000339 crossref_primary_10_1111_jomf_13072 crossref_primary_10_1038_s41598_025_90813_7 crossref_primary_10_1016_j_jretconser_2023_103652 crossref_primary_10_1073_pnas_2508479122 crossref_primary_10_2308_TAR_2022_0307 crossref_primary_10_1007_s44202_024_00229_z crossref_primary_10_1080_08946566_2025_2469048 crossref_primary_10_1080_17522439_2025_2505513 crossref_primary_10_1016_j_eatbeh_2025_101978 crossref_primary_10_1371_journal_pone_0320570 crossref_primary_10_1007_s12144_024_07050_w crossref_primary_10_1080_10615806_2024_2391841 crossref_primary_10_2308_TAR_2023_0027 crossref_primary_10_1016_j_jesp_2023_104489 crossref_primary_10_1038_s41598_025_93100_7 crossref_primary_10_5465_amj_2025_4004 crossref_primary_10_1177_00031224241232599 crossref_primary_10_1111_cogs_13477 crossref_primary_10_1177_07334648251317758 crossref_primary_10_1177_0272989X241275645 crossref_primary_10_1016_j_pubrev_2024_102517 crossref_primary_10_1016_j_bodyim_2024_101721 crossref_primary_10_1177_09567976251333666 crossref_primary_10_1007_s11292_025_09660_1 crossref_primary_10_1017_S0954579423000822 crossref_primary_10_3390_bs15070844 crossref_primary_10_1016_j_amepre_2024_09_009 crossref_primary_10_1007_s10591_024_09728_6 crossref_primary_10_3390_bs14080647 crossref_primary_10_1080_14681994_2024_2332622 crossref_primary_10_1016_j_jacceco_2024_101707 crossref_primary_10_1038_s41598_025_86056_1 crossref_primary_10_1080_13218719_2024_2421167 crossref_primary_10_1080_07317115_2024_2399588 crossref_primary_10_1098_rsos_250361 crossref_primary_10_1177_17456916241242734 crossref_primary_10_1080_10510974_2024_2353393 crossref_primary_10_1016_j_paid_2023_112111 crossref_primary_10_1002_ab_70005 crossref_primary_10_1016_j_chiabu_2024_106789 crossref_primary_10_1016_j_cognition_2024_106050
Cites_doi	10.1371/journal.pone.0157732 10.31234/osf.io/wr8ds 10.1093/poq/nfw030 10.3758/s13428-013-0434-y 10.1002/acp.2350050305 10.1016/j.jesp.2015.07.006 10.20982/tqmp.16.5.p472 10.1177/1745691617706516 10.1016/j.jesp.2016.09.006 10.1016/j.jesp.2009.03.009 10.1177/1948550619875149 10.1037/pspa0000056 10.3758/BF03193146 10.3758/s13428-014-0483-x 10.1016/j.jesp.2017.01.006 10.1111/ajps.12081 10.1016/0010-0285(73)90033-9 10.31234/osf.io/jbc9d 10.33774/apsa-2020-96t72 10.1093/ijpor/edw027 10.2308/bria-18-044 10.1016/j.copsyc.2016.07.001 10.1017/psrm.2020.6 10.1177/1065912919882101 10.1017/psrm.2021.57 10.1111/j.1468-0017.2006.00297.x 10.3758/s13428-016-0727-z 10.1177/2515245919838479 10.1002/bdm.1753 10.3758/s13428-015-0578-z 10.1016/j.jesp.2013.10.005 10.1037/xge0000920 10.1371/journal.pone.0226394 10.1177/1745691610393980 10.1017/S1930297500002205 10.1177/01461672952111004
ContentType	Journal Article
Copyright	The Author(s) 2022 2022. The Author(s).
Copyright_xml	– notice: The Author(s) 2022 – notice: 2022. The Author(s).
DBID	C6C AAYXX CITATION CGR CUY CVF ECM EIF NPM 7X8
DOI	10.3758/s13428-022-01999-x
DatabaseName	Springer Nature OA Free Journals CrossRef Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed MEDLINE - Academic
DatabaseTitle	CrossRef MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) MEDLINE - Academic
DatabaseTitleList	MEDLINE MEDLINE - Academic CrossRef
Database_xml	– sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database
DeliveryMethod	fulltext_linktorsrc
Discipline	Psychology
EISSN	1554-3528
EndPage	3964
ExternalDocumentID	36326997 10_3758_s13428_022_01999_x
Genre	Journal Article
GroupedDBID	--- -55 -5G -BR -DZ -EM -ET -~C -~X 0-V 06D 0R~ 0VY 199 1N0 203 23N 2J2 2JN 2JY 2KG 2KM 2LR 2VQ 30V 3V. 4.4 406 408 40E 53G 5GY 7X7 875 88E 8AO 8FI 8FJ 8G5 8TC 8UJ 95. 96X AAAVM AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AAKPC AANZL AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYQN AAYTO AAYZH AAZMS ABAKF ABDZT ABECU ABFTV ABHLI ABIVO ABJNI ABJOX ABJUD ABKCH ABMQK ABNWP ABPLI ABPPZ ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABUWG ABXPI ACAOD ACBXY ACDTI ACGFS ACHQT ACHSB ACHXU ACIWK ACKIV ACKNC ACMDZ ACMLO ACNCT ACOKC ACPIV ACPRK ACZOJ ADBBV ADHHG ADHIR ADINQ ADKNI ADKPE ADURQ ADYFF ADZKW AEBTG AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AEOHA AEPYU AESKC AETCA AEVLU AEXYK AFBBN AFFNX AFKRA AFQWF AFRAH AFWTZ AFZKB AGAYW AGDGC AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHMBA AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALIPV ALMA_UNASSIGNED_HOLDINGS ALSLI AMKLP AMXSW AMYLF AMYQR AOCGG ARALO ARMRJ ASPBG AVWKF AXYYD AYQZM AZFZN AZQEC B-. BAWUL BENPR BGNMA BPHCQ BVXVI C1A C6C CAG CCPQU COF CSCUP DDRTE DIK DNIVK DPUIP DWQXO E3Z EBD EBLON EBS EIOEI EJD EMOBN ESBYG F5P FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRRFC FSGXE FYUFA GGCAI GGRSB GJIRD GNUQQ GNWQR GQ3 GQ6 GQ7 GUQSH H13 HF~ HMCUK HMJXF HRMNR HVGLF HZ~ H~9 IAO IHR IKXTQ INH IPY IRVIT ITC ITM IWAJR J-C JBSCW JZLTJ KOV LLZTM M1P M2M M2O M2R M4Y MVM N2Q N9A NB0 NPVJJ NQJWS NU0 O9- O93 O9G O9J OHT OK1 P2P P9L PADUT PF- PQQKQ PROAC PSQYO PSYQQ PT4 R9I RIG ROL RPV RSV S16 S1Z S27 S3B SBS SBU SCLPG SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW SSXJD STPWE SV3 SZN T13 TN5 TR2 TSG TUC TUS U2A U9L UG4 UKHRP UOJIU UPT UTJUX UZXMN VFIZW VXZ W48 WH7 WK8 XJT XOL XSW Z7R Z7S Z7W Z81 Z83 Z88 Z8N Z92 ZMTXR ZOVNA ZUP AAPKM AAYXX ABBRH ABDBE ABFSG ABRTQ ACSTC AEZWR AFDZB AFHIU AFOHR AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION ADHKG AGQPQ CGR CUY CVF ECM EIF NPM PHGZM PHGZT PJZUB PPXIY PRQQA 7X8
ID	FETCH-LOGICAL-c458t-e1a84ba3a8b357b8924d080b06d55d58e7588a709b9b07bfcb9c56f2812329e43
IEDL.DBID	RSV
ISICitedReferencesCount	159
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000878463800002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1554-3528
IngestDate	Thu Oct 02 05:35:49 EDT 2025 Mon Jul 21 06:04:50 EDT 2025 Sat Nov 29 02:18:12 EST 2025 Tue Nov 18 22:23:34 EST 2025 Fri Feb 21 02:41:01 EST 2025
IsDoiOpenAccess	true
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	8
Keywords	Response bias Data quality Participant recruitment Test validity
Language	English
License	2022. The Author(s).
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c458t-e1a84ba3a8b357b8924d080b06d55d58e7588a709b9b07bfcb9c56f2812329e43
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ORCID	0000-0002-5480-5213 0000-0003-4396-4128 0000-0001-6598-2701 0000-0002-7523-5213 0000-0003-2403-6940 0000-0001-5353-7969
OpenAccessLink	https://link.springer.com/10.3758/s13428-022-01999-x
PMID	36326997
PQID	2731717232
PQPubID	23479
PageCount	12
ParticipantIDs	proquest_miscellaneous_2731717232 pubmed_primary_36326997 crossref_primary_10_3758_s13428_022_01999_x crossref_citationtrail_10_3758_s13428_022_01999_x springer_journals_10_3758_s13428_022_01999_x
PublicationCentury	2000
PublicationDate	2023-12-01
PublicationDateYYYYMMDD	2023-12-01
PublicationDate_xml	– month: 12 year: 2023 text: 2023-12-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York – name: United States
PublicationTitle	Behavior research methods
PublicationTitleAbbrev	Behav Res
PublicationTitleAlternate	Behav Res Methods
PublicationYear	2023
Publisher	Springer US
Publisher_xml	– name: Springer US
References	TukeyJWExploratory data analysis1977Addison-Wesley Gautam, R., Kerstein, M., Moss, A. J., & Litman, L. (2018). Understanding geolocations and their connection to data quality. [blog post]. Retrieved from: https://www.cloudresearch.com/resources/blog/understanding-geolocations-and-their-connection-to-data-quality TverskyAKahnemanDAvailability: A heuristic for judging frequency and probabilityCognitive Psychology19735220723210.1016/0010-0285(73)90033-9 OppenheimerDMMeyvisTDavidenkoNInstructional manipulation checks: Detecting satisficing to increase statistical powerJournal of Experimental Social Psychology200945486787210.1016/j.jesp.2009.03.009 CurranPGMethods for the detection of carelessly invalid responses in survey dataJournal of Experimental Social Psychology20166641910.1016/j.jesp.2015.07.006 Moss, A. J., & Litman, L. (2018). After the bot scare: Understanding what’s been happening with data collection on MTurk and how to stop it [blog post]. HauserDJSchwarzNAttentive Turkers: MTurk participants perform better on online attention checks than do subject pool participantsBehavior Research Methods20164840040710.3758/s13428-015-0578-z25761395 MottaMPCallaghanTHSmithBLooking for answers: Identifying search behavior and improving knowledge-based data quality in online surveysInternational Journal of Public Opinion Research201729457560310.1093/ijpor/edw027 Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ...Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.https://doi.org/10.1016/j.jesp.2013.10.005 JohnOPNaumannLPSotoCJJohnOPRobinsRWPervinLAParadigm shift to the integrative Big Five trait taxonomy: History, measurement, and conceptual issuesHandbook of personality: Theory and research20083Guilford Press114158 BuhrmesterMKwangTGoslingSDAmazon's Mechanical Turk: A new source of inexpensive, yet high-quality data?Perspectives on Psychological Science201163510.1177/174569161039398026162106 Moss, A. J., Rosenzweig, C., Robinson, J., Jaffe, S. N., & Litman, L. (2021). Is it ethical to use mechanical turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. https://doi.org/10.31234/osf.io/jbc9d KennedyRCliffordSBurleighTWaggonerPDJewellRWinterNJThe shape of and solutions to the MTurk quality crisisPolitical Science Research and Methods2020861462910.1017/psrm.2020.6 PeerEBrandimarteLSamatSAcquistiABeyond the Turk: Alternative platforms for crowdsourcing behavioral researchJournal of Experimental Social Psychology20177015316310.1016/j.jesp.2017.01.006 KaneJVVelezYRBarabasJAnalyze the attentive and bypass bias: Mock vignette checks in survey experimentsAPSA Preprints202010.33774/apsa-2020-96t72 Ryan, T. J. (2018). Data contamination on MTurk. Retrieved from: https://timryan.web.unc.edu/2018/08/12/data-contamination-on-mturk Ahler, D. J., Roush, C. E., & Sood, G. (2019). The micro-task market for lemons: Data quality on Amazon’s Mechanical Turk. Political Science Research and Methods, 1–20. https://doi.org/10.1017/psrm.2021.57 HauserMCushmanFYoungLKang-Xing JinRMikhailJA dissociation between moral judgments and justificationsMind & Language20072212110.1111/j.1468-0017.2006.00297.x LuttrellAPettyREXuMReplicating and fixing failed replications: The case of need for cognition and argument qualityJournal of Experimental Social Psychology20176917818310.1016/j.jesp.2016.09.006 RamscarMLearning and the replicability of priming effectsCurrent Opinion in Psychology201612808410.1016/j.copsyc.2016.07.001 StorozukAAshleyMDelageVMaloneyEAGot bots? Practical recommendations to protect online survey data from bot attacksThe Quantitative Methods for Psychology202016547248110.20982/tqmp.16.5.p472 HauserDJPaolacciGChandlerJJKardesFRHerrPMSchwarzNCommon concerns with MTurk as a participant pool: Evidence and solutionsHandbook of research methods in consumer psychology2019Routledge LitmanLRobinsonJAbberbockTTurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciencesBehavior Research Methods201749243344210.3758/s13428-016-0727-z27071389 SmithBCliffordSJeritJTRENDS: How internet search undermines the validity of political knowledge measuresPolitical Research Quarterly202073114115510.1177/1065912919882101 Mechanical Turk Replication Project (2021). #MTRP: Mechanical Turk Replication Project.https://www.mtrp.info/index.html JacowitzKEKahnemanDMeasures of anchoring in estimation tasksPersonality and Social Psychology Bulletin1995211161116610.1177/01461672952111004 PeerEVosgerauJAcquistiAReputation as a sufficient condition for data quality on Amazon Mechanical TurkBehavior Research Methods2014461023103110.3758/s13428-013-0434-y24356996 Bai, H. (2018). Evidence that a large amount of low quality responses on MTurk can be detected with repeated GPS coordinates. Retrieved from: https://www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-random KrosnickJAResponse strategies for coping with the cognitive demands of attitude measures in surveysApplied cognitive Psychology19915321323610.1002/acp.2350050305 LitmanLRobinsonJLitmanLRobinsonJConducting ethical online research: A data-driven approachConducting online research on Amazon mechanical turk and beyond2020Sage Academic Publishing234263 DennisSAGoodsonBMPearsonCAOnline worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening proceduresBehavioral Research in Accounting202032111913410.2308/bria-18-044 LitmanLRobinsonJLitmanLRobinsonJIntroductionConducting online research on Amazon mechanical turk and beyond2020Sage Academic Publishing126 NeckaEACacioppoSNormanGJCacioppoJTMeasuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participantsPLOS ONE201611610.1371/journal.pone.0157732273513784924794 Levesque, H., Davis, E., & Morgenstern, L. (2012). The Winograd schema challenge. In Proceedings of KR, pp 362–372, Rome, Italy. SchwarzNStrackFDoes merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizationsSocial Psychology2014454305306 LitmanLRobinsonJRosenzweigCThe relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical TurkBehavior Research Methods20154751952810.3758/s13428-014-0483-x24907001 GomilaRLogistic or linear? Estimating causal effects of experimental treatments on binary outcomes using regression analysisJournal of Experimental Psychology: General2020150470010.1037/xge000092032969684 ZhouHFishbachAThe pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusionsJournal of Personality and Social Psychology201611149350410.1037/pspa000005627295328 ChandlerJPaolacciGHauserDJLitmanLRobinsonJData quality issues on mechanical turkConducting online research on Amazon mechanical Turk and beyond2020Sage Academic Publishing. Thousand Oaks95120 Litman, L., Rosenzweig, C., Jaffe, S. N., Gautam, R., Robinson, J., & Moss, A. J. (2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. https://doi.org/10.31234/osf.io/wr8ds FaulFErdfelderELangAGBuchnerAG* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciencesBehavior Research Methods20073917519110.3758/BF0319314617695343 Litman, L., Rosenzweig, C., & Moss, A (2020). New solutions dramatically improve research data quality on MTurk. CloudResearch. https://www.cloudresearch.com/resources/blog/new-tools-improve-research-data-quality-mturk PaolacciGChandlerJIpeirotisPGRunning experiments on Amazon Mechanical TurkJudgment and Decision Making2010541141910.1017/S1930297500002205 Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015). Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. Kennedy, C., Hatley, N., Lau, A., Mercer, A., Keeter, S., Ferno, J., & Asare-Marfo, D. (2020a). Assessing the risks to online polls from bogus respondents. Pew Research Center Methods. https://www.pewresearch.org/methods/2020a/02/18/assessing-the-risks-to-online-polls-from-bogus-respondents GoodmanJKCryderCECheemaAData collection in a flat world: The strengths and weaknesses of Mechanical Turk samplesJournal of Behavioral Decision Making201326321322410.1002/bdm.1753 CliffordSJeritJCheating on political knowledge questions in online surveys: An assessment of the problem and solutionsPublic Opinion Quarterly201680485888710.1093/poq/nfw030 RobinsonJRosenzweigCMossAJLitmanLTapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant poolPLOS ONE2019141210.1371/journal.pone.0226394318415346913990 BerinskyAJMargolisMFSancesMWSeparating the shirkers from the workers? Making sure respondents pay attention on self-administered surveysAmerican Journal of Political Science201458373975310.1111/ajps.12081 ChmielewskiMKuckerSCAn MTurk crisis? Shifts in data quality and the impact on study resultsSocial Psychological and Personality Science202011446447310.1177/1948550619875149 BuhrmesterMDTalaifarSGoslingSDAn evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective usePerspectives on Psychological Science20181314915410.1177/174569161770651629928846 PermutSFisherMOppenheimerDMTaskMaster: A tool for determining when subjects are on taskAdvances in Methods and Practices in Psychological Science20192218819610.1177/2515245919838479 KE Jacowitz (1999_CR19) 1995; 21 JA Krosnick (1999_CR24) 1991; 5 MD Buhrmester (1999_CR6) 2018; 13 L Litman (1999_CR26) 2020 J Robinson (1999_CR44) 2019; 14 PG Curran (1999_CR10) 2016; 66 DJ Hauser (1999_CR18) 2019 1999_CR25 OP John (1999_CR20) 2008 1999_CR22 M Ramscar (1999_CR43) 2016; 12 L Litman (1999_CR27) 2020 EA Necka (1999_CR37) 2016; 11 1999_CR35 H Zhou (1999_CR52) 2016; 111 JW Tukey (1999_CR49) 1977 J Chandler (1999_CR8) 2020 1999_CR33 DM Oppenheimer (1999_CR38) 2009; 45 1999_CR34 R Kennedy (1999_CR23) 2020; 8 1999_CR31 A Luttrell (1999_CR32) 2017; 69 F Faul (1999_CR12) 2007; 39 R Gomila (1999_CR14) 2020; 150 1999_CR30 G Paolacci (1999_CR39) 2010; 5 L Litman (1999_CR29) 2017; 49 E Peer (1999_CR40) 2014; 46 A Storozuk (1999_CR48) 2020; 16 SA Dennis (1999_CR11) 2020; 32 M Chmielewski (1999_CR9) 2020; 11 S Permut (1999_CR42) 2019; 2 E Peer (1999_CR41) 2017; 70 JV Kane (1999_CR21) 2020 S Clifford (1999_CR7) 2016; 80 1999_CR45 1999_CR1 M Buhrmester (1999_CR5) 2011; 6 M Hauser (1999_CR17) 2007; 22 1999_CR2 AJ Berinsky (1999_CR3) 2014; 58 1999_CR4 L Litman (1999_CR28) 2015; 47 B Smith (1999_CR47) 2020; 73 A Tversky (1999_CR50) 1973; 5 DJ Hauser (1999_CR16) 2016; 48 JK Goodman (1999_CR15) 2013; 26 1999_CR13 N Schwarz (1999_CR46) 2014; 45 1999_CR51 MP Motta (1999_CR36) 2017; 29
References_xml	– reference: Ahler, D. J., Roush, C. E., & Sood, G. (2019). The micro-task market for lemons: Data quality on Amazon’s Mechanical Turk. Political Science Research and Methods, 1–20. https://doi.org/10.1017/psrm.2021.57 – reference: Levesque, H., Davis, E., & Morgenstern, L. (2012). The Winograd schema challenge. In Proceedings of KR, pp 362–372, Rome, Italy. – reference: BuhrmesterMDTalaifarSGoslingSDAn evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective usePerspectives on Psychological Science20181314915410.1177/174569161770651629928846 – reference: PeerEBrandimarteLSamatSAcquistiABeyond the Turk: Alternative platforms for crowdsourcing behavioral researchJournal of Experimental Social Psychology20177015316310.1016/j.jesp.2017.01.006 – reference: Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015). Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. – reference: MottaMPCallaghanTHSmithBLooking for answers: Identifying search behavior and improving knowledge-based data quality in online surveysInternational Journal of Public Opinion Research201729457560310.1093/ijpor/edw027 – reference: LitmanLRobinsonJLitmanLRobinsonJConducting ethical online research: A data-driven approachConducting online research on Amazon mechanical turk and beyond2020Sage Academic Publishing234263 – reference: StorozukAAshleyMDelageVMaloneyEAGot bots? Practical recommendations to protect online survey data from bot attacksThe Quantitative Methods for Psychology202016547248110.20982/tqmp.16.5.p472 – reference: Ryan, T. J. (2018). Data contamination on MTurk. Retrieved from: https://timryan.web.unc.edu/2018/08/12/data-contamination-on-mturk/ – reference: HauserDJPaolacciGChandlerJJKardesFRHerrPMSchwarzNCommon concerns with MTurk as a participant pool: Evidence and solutionsHandbook of research methods in consumer psychology2019Routledge – reference: TukeyJWExploratory data analysis1977Addison-Wesley – reference: LuttrellAPettyREXuMReplicating and fixing failed replications: The case of need for cognition and argument qualityJournal of Experimental Social Psychology20176917818310.1016/j.jesp.2016.09.006 – reference: BerinskyAJMargolisMFSancesMWSeparating the shirkers from the workers? Making sure respondents pay attention on self-administered surveysAmerican Journal of Political Science201458373975310.1111/ajps.12081 – reference: KrosnickJAResponse strategies for coping with the cognitive demands of attitude measures in surveysApplied cognitive Psychology19915321323610.1002/acp.2350050305 – reference: LitmanLRobinsonJRosenzweigCThe relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical TurkBehavior Research Methods20154751952810.3758/s13428-014-0483-x24907001 – reference: Moss, A. J., & Litman, L. (2018). After the bot scare: Understanding what’s been happening with data collection on MTurk and how to stop it [blog post]. – reference: JohnOPNaumannLPSotoCJJohnOPRobinsRWPervinLAParadigm shift to the integrative Big Five trait taxonomy: History, measurement, and conceptual issuesHandbook of personality: Theory and research20083Guilford Press114158 – reference: Kennedy, C., Hatley, N., Lau, A., Mercer, A., Keeter, S., Ferno, J., & Asare-Marfo, D. (2020a). Assessing the risks to online polls from bogus respondents. Pew Research Center Methods. https://www.pewresearch.org/methods/2020a/02/18/assessing-the-risks-to-online-polls-from-bogus-respondents/ – reference: OppenheimerDMMeyvisTDavidenkoNInstructional manipulation checks: Detecting satisficing to increase statistical powerJournal of Experimental Social Psychology200945486787210.1016/j.jesp.2009.03.009 – reference: ChandlerJPaolacciGHauserDJLitmanLRobinsonJData quality issues on mechanical turkConducting online research on Amazon mechanical Turk and beyond2020Sage Academic Publishing. Thousand Oaks95120 – reference: KennedyRCliffordSBurleighTWaggonerPDJewellRWinterNJThe shape of and solutions to the MTurk quality crisisPolitical Science Research and Methods2020861462910.1017/psrm.2020.6 – reference: GomilaRLogistic or linear? Estimating causal effects of experimental treatments on binary outcomes using regression analysisJournal of Experimental Psychology: General2020150470010.1037/xge000092032969684 – reference: RamscarMLearning and the replicability of priming effectsCurrent Opinion in Psychology201612808410.1016/j.copsyc.2016.07.001 – reference: FaulFErdfelderELangAGBuchnerAG* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciencesBehavior Research Methods20073917519110.3758/BF0319314617695343 – reference: HauserMCushmanFYoungLKang-Xing JinRMikhailJA dissociation between moral judgments and justificationsMind & Language20072212110.1111/j.1468-0017.2006.00297.x – reference: HauserDJSchwarzNAttentive Turkers: MTurk participants perform better on online attention checks than do subject pool participantsBehavior Research Methods20164840040710.3758/s13428-015-0578-z25761395 – reference: LitmanLRobinsonJAbberbockTTurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciencesBehavior Research Methods201749243344210.3758/s13428-016-0727-z27071389 – reference: Bai, H. (2018). Evidence that a large amount of low quality responses on MTurk can be detected with repeated GPS coordinates. Retrieved from: https://www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-random – reference: Moss, A. J., Rosenzweig, C., Robinson, J., Jaffe, S. N., & Litman, L. (2021). Is it ethical to use mechanical turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. https://doi.org/10.31234/osf.io/jbc9d – reference: KaneJVVelezYRBarabasJAnalyze the attentive and bypass bias: Mock vignette checks in survey experimentsAPSA Preprints202010.33774/apsa-2020-96t72 – reference: Gautam, R., Kerstein, M., Moss, A. J., & Litman, L. (2018). Understanding geolocations and their connection to data quality. [blog post]. Retrieved from: https://www.cloudresearch.com/resources/blog/understanding-geolocations-and-their-connection-to-data-quality/ – reference: TverskyAKahnemanDAvailability: A heuristic for judging frequency and probabilityCognitive Psychology19735220723210.1016/0010-0285(73)90033-9 – reference: DennisSAGoodsonBMPearsonCAOnline worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening proceduresBehavioral Research in Accounting202032111913410.2308/bria-18-044 – reference: NeckaEACacioppoSNormanGJCacioppoJTMeasuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participantsPLOS ONE201611610.1371/journal.pone.0157732273513784924794 – reference: GoodmanJKCryderCECheemaAData collection in a flat world: The strengths and weaknesses of Mechanical Turk samplesJournal of Behavioral Decision Making201326321322410.1002/bdm.1753 – reference: SmithBCliffordSJeritJTRENDS: How internet search undermines the validity of political knowledge measuresPolitical Research Quarterly202073114115510.1177/1065912919882101 – reference: RobinsonJRosenzweigCMossAJLitmanLTapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant poolPLOS ONE2019141210.1371/journal.pone.0226394318415346913990 – reference: CliffordSJeritJCheating on political knowledge questions in online surveys: An assessment of the problem and solutionsPublic Opinion Quarterly201680485888710.1093/poq/nfw030 – reference: PeerEVosgerauJAcquistiAReputation as a sufficient condition for data quality on Amazon Mechanical TurkBehavior Research Methods2014461023103110.3758/s13428-013-0434-y24356996 – reference: ZhouHFishbachAThe pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusionsJournal of Personality and Social Psychology201611149350410.1037/pspa000005627295328 – reference: Litman, L., Rosenzweig, C., Jaffe, S. N., Gautam, R., Robinson, J., & Moss, A. J. (2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. https://doi.org/10.31234/osf.io/wr8ds – reference: JacowitzKEKahnemanDMeasures of anchoring in estimation tasksPersonality and Social Psychology Bulletin1995211161116610.1177/01461672952111004 – reference: SchwarzNStrackFDoes merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizationsSocial Psychology2014454305306 – reference: CurranPGMethods for the detection of carelessly invalid responses in survey dataJournal of Experimental Social Psychology20166641910.1016/j.jesp.2015.07.006 – reference: PaolacciGChandlerJIpeirotisPGRunning experiments on Amazon Mechanical TurkJudgment and Decision Making2010541141910.1017/S1930297500002205 – reference: LitmanLRobinsonJLitmanLRobinsonJIntroductionConducting online research on Amazon mechanical turk and beyond2020Sage Academic Publishing126 – reference: Litman, L., Rosenzweig, C., & Moss, A (2020). New solutions dramatically improve research data quality on MTurk. CloudResearch. https://www.cloudresearch.com/resources/blog/new-tools-improve-research-data-quality-mturk/ – reference: Mechanical Turk Replication Project (2021). #MTRP: Mechanical Turk Replication Project.https://www.mtrp.info/index.html – reference: BuhrmesterMKwangTGoslingSDAmazon's Mechanical Turk: A new source of inexpensive, yet high-quality data?Perspectives on Psychological Science201163510.1177/174569161039398026162106 – reference: ChmielewskiMKuckerSCAn MTurk crisis? Shifts in data quality and the impact on study resultsSocial Psychological and Personality Science202011446447310.1177/1948550619875149 – reference: PermutSFisherMOppenheimerDMTaskMaster: A tool for determining when subjects are on taskAdvances in Methods and Practices in Psychological Science20192218819610.1177/2515245919838479 – reference: Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ...Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.https://doi.org/10.1016/j.jesp.2013.10.005 – volume: 11 issue: 6 year: 2016 ident: 1999_CR37 publication-title: PLOS ONE doi: 10.1371/journal.pone.0157732 – start-page: 1 volume-title: Conducting online research on Amazon mechanical turk and beyond year: 2020 ident: 1999_CR26 – ident: 1999_CR31 doi: 10.31234/osf.io/wr8ds – volume: 80 start-page: 858 issue: 4 year: 2016 ident: 1999_CR7 publication-title: Public Opinion Quarterly doi: 10.1093/poq/nfw030 – volume: 46 start-page: 1023 year: 2014 ident: 1999_CR40 publication-title: Behavior Research Methods doi: 10.3758/s13428-013-0434-y – volume: 5 start-page: 213 issue: 3 year: 1991 ident: 1999_CR24 publication-title: Applied cognitive Psychology doi: 10.1002/acp.2350050305 – ident: 1999_CR51 – ident: 1999_CR13 – volume-title: Exploratory data analysis year: 1977 ident: 1999_CR49 – volume: 66 start-page: 4 year: 2016 ident: 1999_CR10 publication-title: Journal of Experimental Social Psychology doi: 10.1016/j.jesp.2015.07.006 – ident: 1999_CR33 – volume: 16 start-page: 472 issue: 5 year: 2020 ident: 1999_CR48 publication-title: The Quantitative Methods for Psychology doi: 10.20982/tqmp.16.5.p472 – volume: 13 start-page: 149 year: 2018 ident: 1999_CR6 publication-title: Perspectives on Psychological Science doi: 10.1177/1745691617706516 – volume: 69 start-page: 178 year: 2017 ident: 1999_CR32 publication-title: Journal of Experimental Social Psychology doi: 10.1016/j.jesp.2016.09.006 – ident: 1999_CR2 – volume: 45 start-page: 867 issue: 4 year: 2009 ident: 1999_CR38 publication-title: Journal of Experimental Social Psychology doi: 10.1016/j.jesp.2009.03.009 – ident: 1999_CR22 – volume: 11 start-page: 464 issue: 4 year: 2020 ident: 1999_CR9 publication-title: Social Psychological and Personality Science doi: 10.1177/1948550619875149 – volume: 111 start-page: 493 year: 2016 ident: 1999_CR52 publication-title: Journal of Personality and Social Psychology doi: 10.1037/pspa0000056 – volume: 39 start-page: 175 year: 2007 ident: 1999_CR12 publication-title: Behavior Research Methods doi: 10.3758/BF03193146 – volume: 47 start-page: 519 year: 2015 ident: 1999_CR28 publication-title: Behavior Research Methods doi: 10.3758/s13428-014-0483-x – volume: 70 start-page: 153 year: 2017 ident: 1999_CR41 publication-title: Journal of Experimental Social Psychology doi: 10.1016/j.jesp.2017.01.006 – volume: 58 start-page: 739 issue: 3 year: 2014 ident: 1999_CR3 publication-title: American Journal of Political Science doi: 10.1111/ajps.12081 – volume: 5 start-page: 207 issue: 2 year: 1973 ident: 1999_CR50 publication-title: Cognitive Psychology doi: 10.1016/0010-0285(73)90033-9 – ident: 1999_CR35 doi: 10.31234/osf.io/jbc9d – ident: 1999_CR34 – year: 2020 ident: 1999_CR21 publication-title: APSA Preprints doi: 10.33774/apsa-2020-96t72 – volume: 29 start-page: 575 issue: 4 year: 2017 ident: 1999_CR36 publication-title: International Journal of Public Opinion Research doi: 10.1093/ijpor/edw027 – volume: 32 start-page: 119 issue: 1 year: 2020 ident: 1999_CR11 publication-title: Behavioral Research in Accounting doi: 10.2308/bria-18-044 – volume: 12 start-page: 80 year: 2016 ident: 1999_CR43 publication-title: Current Opinion in Psychology doi: 10.1016/j.copsyc.2016.07.001 – volume: 8 start-page: 614 year: 2020 ident: 1999_CR23 publication-title: Political Science Research and Methods doi: 10.1017/psrm.2020.6 – volume-title: Handbook of research methods in consumer psychology year: 2019 ident: 1999_CR18 – volume: 45 start-page: 305 issue: 4 year: 2014 ident: 1999_CR46 publication-title: Social Psychology – ident: 1999_CR30 – volume: 73 start-page: 141 issue: 1 year: 2020 ident: 1999_CR47 publication-title: Political Research Quarterly doi: 10.1177/1065912919882101 – ident: 1999_CR1 doi: 10.1017/psrm.2021.57 – start-page: 95 volume-title: Conducting online research on Amazon mechanical Turk and beyond year: 2020 ident: 1999_CR8 – volume: 22 start-page: 1 year: 2007 ident: 1999_CR17 publication-title: Mind & Language doi: 10.1111/j.1468-0017.2006.00297.x – volume: 49 start-page: 433 issue: 2 year: 2017 ident: 1999_CR29 publication-title: Behavior Research Methods doi: 10.3758/s13428-016-0727-z – volume: 2 start-page: 188 issue: 2 year: 2019 ident: 1999_CR42 publication-title: Advances in Methods and Practices in Psychological Science doi: 10.1177/2515245919838479 – volume: 26 start-page: 213 issue: 3 year: 2013 ident: 1999_CR15 publication-title: Journal of Behavioral Decision Making doi: 10.1002/bdm.1753 – volume: 48 start-page: 400 year: 2016 ident: 1999_CR16 publication-title: Behavior Research Methods doi: 10.3758/s13428-015-0578-z – ident: 1999_CR25 – ident: 1999_CR4 doi: 10.1016/j.jesp.2013.10.005 – volume: 150 start-page: 700 issue: 4 year: 2020 ident: 1999_CR14 publication-title: Journal of Experimental Psychology: General doi: 10.1037/xge0000920 – volume: 14 issue: 12 year: 2019 ident: 1999_CR44 publication-title: PLOS ONE doi: 10.1371/journal.pone.0226394 – ident: 1999_CR45 – start-page: 114 volume-title: Handbook of personality: Theory and research year: 2008 ident: 1999_CR20 – volume: 6 start-page: 3 year: 2011 ident: 1999_CR5 publication-title: Perspectives on Psychological Science doi: 10.1177/1745691610393980 – volume: 5 start-page: 411 year: 2010 ident: 1999_CR39 publication-title: Judgment and Decision Making doi: 10.1017/S1930297500002205 – start-page: 234 volume-title: Conducting online research on Amazon mechanical turk and beyond year: 2020 ident: 1999_CR27 – volume: 21 start-page: 1161 year: 1995 ident: 1999_CR19 publication-title: Personality and Social Psychology Bulletin doi: 10.1177/01461672952111004
SSID	ssj0038214
Score	2.6857338
Snippet	Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis...
SourceID	proquest pubmed crossref springer
SourceType	Aggregation Database Index Database Enrichment Source Publisher
StartPage	3953
SubjectTerms	Attention Behavioral Science and Psychology Cognitive Psychology Crowdsourcing - methods Data Accuracy Humans Psychology Self Report Surveys and Questionnaires
Title	Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
URI	https://link.springer.com/article/10.3758/s13428-022-01999-x https://www.ncbi.nlm.nih.gov/pubmed/36326997 https://www.proquest.com/docview/2731717232
Volume	55
WOSCitedRecordID	wos000878463800002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVPQU databaseName: ProQuest Central customDbUrl: eissn: 1554-3528 dateEnd: 20241209 omitProxy: false ssIdentifier: ssj0038214 issn: 1554-3528 databaseCode: BENPR dateStart: 20230101 isFulltext: true titleUrlDefault: https://www.proquest.com/central providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest Health & Medical Collection customDbUrl: eissn: 1554-3528 dateEnd: 20241209 omitProxy: false ssIdentifier: ssj0038214 issn: 1554-3528 databaseCode: 7X7 dateStart: 20230101 isFulltext: true titleUrlDefault: https://search.proquest.com/healthcomplete providerName: ProQuest – providerCode: PRVPQU databaseName: Psychology Database customDbUrl: eissn: 1554-3528 dateEnd: 20241209 omitProxy: false ssIdentifier: ssj0038214 issn: 1554-3528 databaseCode: M2M dateStart: 20230101 isFulltext: true titleUrlDefault: https://www.proquest.com/psychology providerName: ProQuest – providerCode: PRVPQU databaseName: Research Library customDbUrl: eissn: 1554-3528 dateEnd: 20241209 omitProxy: false ssIdentifier: ssj0038214 issn: 1554-3528 databaseCode: M2O dateStart: 20230101 isFulltext: true titleUrlDefault: https://search.proquest.com/pqrl providerName: ProQuest – providerCode: PRVPQU databaseName: Social Science Database customDbUrl: eissn: 1554-3528 dateEnd: 20241209 omitProxy: false ssIdentifier: ssj0038214 issn: 1554-3528 databaseCode: M2R dateStart: 20230101 isFulltext: true titleUrlDefault: https://search.proquest.com/socscijournals providerName: ProQuest – providerCode: PRVAVX databaseName: SpringerLINK Contemporary 1997-Present customDbUrl: eissn: 1554-3528 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0038214 issn: 1554-3528 databaseCode: RSV dateStart: 20050201 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3JTsMwEB1B4cCFfSmbjMQNIpo4TpwjoCIuLYhNvVneghAoRU2L6I3f4Pf4EsZZKiEEElwsJbETZ2bieRPPArBPtbZhEGpPo3R4qKFbHkfF5gUmlUFgKWeRKYpNxN0u7_WSyyooLK-93estyWKldnYlgtqj3Kehy6YcOFcCFzuPyHEG1R13BRuuru_q9ZfywA_L8Jgfxn1VQd9w5bc90ULVnC38b5KLMF9BS3JcysISTNlsGeYmK9x4BUS7yu2d3ZPTp_7I1H53H2_vOTl26cVfrCHF_ygicyJJLZkEsS2pis-4HK_EeZaSMiJzTPB652Y0eFyF27P2zem5V1VYQNYwPvSsL3moJJVcURYrjsaYQQipWpFhzDBu8Z24jFuJSlQrVqlWiWZRGnAHxBIb0jVoZP3MbgBBS0ynxlDFYrQw_VRKRHrK5beLIqZ03AS_JrrQVfpxVwXjSaAZ4mgnStoJpJ0oaCdem3AwGfNcJt_4tfdezUuB34jb-JCZ7Y9ygRDNR7MV59yE9ZLJk_vRCAFskuD8DmuOiuozzn952Obfum_BnKtTX_rBbENjOBjZHZjVL8OHfLAL03EvLlq-CzMn7e7lFR51gk7RXhStO4NS_gkCSvM3
linkProvider	Springer Nature
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV1LS8QwEA6yCu7F92N9RvCmxbZp2vQoy4qiLqKreAt5VcSlK9vtojf_hn_PX-KkjwURBT03bdOZSeabZuYbhPaJUibwA-UosA4HPLTrMHBsjq8T4fuGMBrqotlE1O2y-_v4qioKy-ps9_pIstipbVwJoPYo80hg2ZR9m0pga-cBOU4H4LEsY_71zV29_xLme0FZHvPDfV9d0Ddc-e1MtHA1J_P_m-QCmqugJT4ubWERTZl0CTUnO9zrMuKdits7fcDt_iDXdd7dx9t7ho8tvfjYaFz8j8IiwwLXlokB2-Kq-YzleMU2sxSXFZmvGK5f9vLh0wq6Pen02qdO1WEBVEPZyDGeYIEURDBJaCQZBGMaIKR0Q02ppszANzERubGMpRvJRMlY0TDxmQVisQnIKmqkg9SsIwyRmEq0JpJGEGF6iRCA9KTltwtDKlXUQl4tdK4q-nHbBaPPIQyxsuOl7DjIjhey4y8tdDC557kk3_h19F6tSw5rxB58iNQM8owDRPMgbIU5t9BaqeTJ80gIADaOYX6HtUZ5tYyzX1628bfhu2j2tHd5wS_OuuebqGl71pc5MVuoMRrmZhvNqPHoMRvuFLb8CYg47-Q
linkToPdf	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LS8NAEF6kinjx_ajPFbxpsMlmk82x1BZFLQWreFv2FRFLKk0r9ubf8O_5S5zNoyhiQTxnk2xmZne-yc58g9ARUcr4nq8cBdbhgIeuOQwcm-PpWHieIYwGOms2Ebbb7P4-6nyp4s-y3csjybymwbI0JcPTZx3bJU4A4J6mLvEts7Jn0wpsHT2gyFnfJtLbeP3mrtyLCfNcPy-V-eW-7-7oB8b8cT6auZ3W0v8nvIwWC8iJ67mNrKAZk6yihcnON15DvFlwficPuNHrj3SZj_fx9p7iuqUdfzEaZ_-psEixwKXFYsC8uGhKY7lfsc04xXml5hjD9evuaPC0jm5bzW7j3Ck6L4DKKBs6xhXMl4IIJgkNJYMgTQO0lLVAU6opM_BNTIS1SEayFspYyUjRIPaYBWiR8ckGqiT9xGwhDBGairUmkoYQebqxEIAApeW9CwIqVVhFbqkArgpactsdo8chPLGy47nsOMiOZ7Ljr1V0PLnnOSflmDr6sNQrh7VjD0REYvqjlAN0cyGchTlX0Wau8MnzSADANopgfieldnmxvNMpL9v-2_ADNN85a_Gri_blDlqwrezzVJldVBkORmYPzamX4WM62M_M-hNPZfjI
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluating+CloudResearch%E2%80%99s+Approved+Group+as+a+solution+for+problematic+data+quality+on+MTurk&rft.jtitle=Behavior+research+methods&rft.au=Hauser%2C+David+J.&rft.au=Moss%2C+Aaron+J.&rft.au=Rosenzweig%2C+Cheskie&rft.au=Jaffe%2C+Shalom+N.&rft.date=2023-12-01&rft.issn=1554-3528&rft.eissn=1554-3528&rft.volume=55&rft.issue=8&rft.spage=3953&rft.epage=3964&rft_id=info:doi/10.3758%2Fs13428-022-01999-x&rft.externalDBID=n%2Fa&rft.externalDocID=10_3758_s13428_022_01999_x
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1554-3528&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1554-3528&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1554-3528&client=summon