Optimising parallel R correlation matrix calculations on gene expression data using MapReduce

Background High-throughput molecular profiling data has been used to improve clinical decision making by stratifying subjects based on their molecular profiles. Unsupervised clustering algorithms can be used for stratification purposes. However, the current speed of the clustering algorithms cannot...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:BMC bioinformatics Ročník 15; číslo 1; s. 351
Hlavní autori: Wang, Shicai, Pandis, Ioannis, Johnson, David, Emam, Ibrahim, Guitton, Florian, Oehmichen, Axel, Guo, Yike
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: London BioMed Central 05.11.2014
BioMed Central Ltd
Springer Nature B.V
Predmet:
Algorithms
Analysis
Benchmarks
Big data
Bioinformatics
Biomedical and Life Sciences
Cluster Analysis
Computational Biology/Bioinformatics
Computer Appl. in Life Sciences
Correlation
Gene expression
Gene Expression Profiling - methods
High-Throughput Nucleotide Sequencing
Humans
Information management
Life Sciences
Mathematical analysis
Microarrays
Optimization
Performance evaluation
Research Article
Snowfall
Software
Transcriptome analysis
Vanilla
MapReduce Framework
Data Block
Data Preparation
Hierarchical Cluster Algorithm
Vanilla
ISSN:1471-2105, 1471-2105
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Background High-throughput molecular profiling data has been used to improve clinical decision making by stratifying subjects based on their molecular profiles. Unsupervised clustering algorithms can be used for stratification purposes. However, the current speed of the clustering algorithms cannot meet the requirement of large-scale molecular data due to poor performance of the correlation matrix calculation. With high-throughput sequencing technologies promising to produce even larger datasets per subject, we expect the performance of the state-of-the-art statistical algorithms to be further impacted unless efforts towards optimisation are carried out. MapReduce is a widely used high performance parallel framework that can solve the problem. Results In this paper, we evaluate the current parallel modes for correlation calculation methods and introduce an efficient data distribution and parallel calculation algorithm based on MapReduce to optimise the correlation calculation. We studied the performance of our algorithm using two gene expression benchmarks. In the micro-benchmark, our implementation using MapReduce, based on the R package RHIPE, demonstrates a 3.26-5.83 fold increase compared to the default Snowfall and 1.56-1.64 fold increase compared to the basic RHIPE in the Euclidean, Pearson and Spearman correlations. Though vanilla R and the optimised Snowfall outperforms our optimised RHIPE in the micro-benchmark, they do not scale well with the macro-benchmark. In the macro-benchmark the optimised RHIPE performs 2.03-16.56 times faster than vanilla R. Benefiting from the 3.30-5.13 times faster data preparation, the optimised RHIPE performs 1.22-1.71 times faster than the optimised Snowfall. Both the optimised RHIPE and the optimised Snowfall successfully performs the Kendall correlation with TCGA dataset within 7 hours. Both of them conduct more than 30 times faster than the estimated vanilla R. Conclusions The performance evaluation found that the new MapReduce algorithm and its implementation in RHIPE outperforms vanilla R and the conventional parallel algorithms implemented in R Snowfall. We propose that MapReduce framework holds great promise for large molecular data analysis, in particular for high-dimensional genomic data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new algorithm as a basis for optimising high-throughput molecular data correlation calculation for Big Data.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ObjectType-Article-2
ObjectType-Undefined-1
ObjectType-Feature-3
ISSN:1471-2105
1471-2105
DOI:10.1186/s12859-014-0351-9