P-Value Precision and Reproducibility

P-values are useful statistical measures of evidence against a null hypothesis. In contrast to other statistical estimates, however, their sample-to-sample variability is usually not considered or estimated, and therefore not fully appreciated. Via a systematic study of log-scale p-value standard er...

Full description

Saved in:
Bibliographic Details
Published in:The American statistician Vol. 65; no. 4; pp. 213 - 221
Main Authors: Boos, Dennis D., Stefanski, Leonard A.
Format: Journal Article
Language:English
Published: Alexandria, VA Taylor & Francis 01.11.2011
American Statistical Association
Subjects:
Approximation
Bootstrap method
Errors
Exact sciences and technology
General topics
Hypothesis testing
Inference from stochastic processes; time series analysis
Log p-value
Mathematics
Measure of evidence
Null hypothesis
P values
Parametric inference
Prediction interval
Probabilities
Probability and statistics
Probability theory and stochastic processes
Random sampling
Reproducibility
Reproducibility probability
Sample size
Sampling distributions
Sciences and techniques of general use
Standard deviation
Standard error
Statistical significance
Statistical variance
Statistics
Stochastic processes
Values
Variability
Statistical contrast
Log p-value
Error estimation
Reproducibility probability
Prediction
Probability
Prediction theory
Statistical estimation
Probability distribution
Stochastic process
Prediction interval
Measure of evidence
Hypothesis test
Bootstrap
Jackknife method
Filtering theory
P value
Resampling method
ISSN:0003-1305, 1537-2731
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:P-values are useful statistical measures of evidence against a null hypothesis. In contrast to other statistical estimates, however, their sample-to-sample variability is usually not considered or estimated, and therefore not fully appreciated. Via a systematic study of log-scale p-value standard errors, bootstrap prediction bounds, and reproducibility probabilities for future replicate p-values, we show that p-values exhibit surprisingly large variability in typical data situations. In addition to providing context to discussions about the failure of statistical results to replicate, our findings shed light on the relative value of exact p-values vis-a-vis approximate p-values, and indicate that the use of *, **, and *** to denote levels 0.05, 0.01, and 0.001 of statistical significance in subject-matter journals is about the right level of precision for reporting p-values when judged by widely accepted rules for rounding statistical estimates.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
content type line 23
ISSN:0003-1305
1537-2731
DOI:10.1198/tas.2011.10129