Sustainable data analysis with Snakemake [version 2; peer review: 2 approved]

Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way...

Full description

Saved in:
Bibliographic Details
Published in:F1000 research Vol. 10; p. 33
Main Authors: Mölder, Felix, Jablonski, Kim Philipp, Letcher, Brice, Hall, Michael B, Tomkins-Tinch, Christopher H, Sochat, Vanessa, Forster, Jan, Lee, Soohyun, Twardziok, Sven O, Kanitz, Alexander, Wilm, Andreas, Holtgrewe, Manuel, Rahmann, Sven, Nahnsen, Sven, Köster, Johannes
Format: Journal Article
Language:English
Published: England Faculty of 1000 Ltd 2021
F1000 Research Limited
F1000 Research Ltd
Subjects:
Adaptability
Automation
Data Analysis
Data processing
Method
Programming languages
Quality control
Reproducibility
Reproducibility of Results
Software
Sustainability
Workflow
Workflow software
data analysis
reproducibility
transparency
workflow management
adaptability
sustainability
scalability
ISSN:2046-1402, 2046-1402
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for a data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.
Bibliography:new_version
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
No competing interests were disclosed.
ISSN:2046-1402
2046-1402
DOI:10.12688/f1000research.29032.2