The sceptical Bayes factor for the assessment of replication success

Replication studies are increasingly conducted but there is no established statistical criterion for replication success. We propose a novel approach combining reverse‐Bayes analysis with Bayesian hypothesis testing: a sceptical prior is determined for the effect size such that the original finding...

Full description

Saved in:
Bibliographic Details
Published in:Journal of the Royal Statistical Society. Series B, Statistical methodology Vol. 84; no. 3; pp. 879 - 911
Main Authors: Pawel, Samuel, Held, Leonhard
Format: Journal Article
Language:English
Published: Oxford Oxford University Press 01.07.2022
Subjects:
Advocacy
Asymptotic properties
Bayes factor
Bayesian analysis
Bayesian hypothesis testing
Case studies
Error analysis
Hypotheses
Hypothesis testing
Incompatibility
Null hypothesis
Property
Regression analysis
Replication
replication studies
Reproducibility
reverse‐Bayes
sceptical p‐value
Skepticism
Social sciences
Statistical methods
Statistics
Success
ISSN:1369-7412, 1467-9868
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Replication studies are increasingly conducted but there is no established statistical criterion for replication success. We propose a novel approach combining reverse‐Bayes analysis with Bayesian hypothesis testing: a sceptical prior is determined for the effect size such that the original finding is no longer convincing in terms of a Bayes factor. This prior is then contrasted to an advocacy prior (the reference posterior of the effect size based on the original study), and replication success is declared if the replication data favour the advocacy over the sceptical prior at a higher level than the original data favoured the sceptical prior over the null hypothesis. The sceptical Bayes factor is the highest level where replication success can be declared. A comparison to existing methods reveals that the sceptical Bayes factor combines several notions of replicability: it ensures that both studies show sufficient evidence against the null and penalises incompatibility of their effect estimates. Analysis of asymptotic properties and error rates, as well as case studies from the Social Sciences Replication Project show the advantages of the method for the assessment of replicability.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1369-7412
1467-9868
DOI:10.1111/rssb.12491