Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concern...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Behavior research methods Ročník 55; číslo 8; s. 3953 - 3964
Hlavní autoři: Hauser, David J., Moss, Aaron J., Rosenzweig, Cheskie, Jaffe, Shalom N., Robinson, Jonathan, Litman, Leib
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York Springer US 01.12.2023
Témata:
Attention
Behavioral Science and Psychology
Cognitive Psychology
Crowdsourcing - methods
Data Accuracy
Humans
Psychology
Self Report
Surveys and Questionnaires
Response bias
Data quality
Participant recruitment
Test validity
ISSN:1554-3528, 1554-3528
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. In a pre-registered study, participants ( N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk’s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1554-3528
1554-3528
DOI:10.3758/s13428-022-01999-x