A History-Based Resource Manager for Genome Analysis Workflows Applications on Clusters with Heterogeneous Nodes

Bioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International journal of parallel programming Jg. 47; H. 2; S. 317 - 342
Hauptverfasser: Badosa, Ferran, Espinosa, Antonio, Acevedo, Cesar, Vera, Gonzalo, Ripoll, Ana
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York Springer US 01.04.2019
Springer Nature B.V
Schlagworte:
Algorithms
Bioinformatics
Clusters
Computer Science
Management systems
Multiprogramming
Nodes
Performance enhancement
Processor Architectures
Resource management
Resource scheduling
Servers
Software Engineering/Programming and Operating Systems
Theory of Computation
Workflow
Resource manager
Resource sharing
Makespan
Bioinformatics workflows
Multivariate regression prediction
Slowdown
Scheduling algorithm
ISSN:0885-7458, 1573-7640
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Bioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next. Resource management systems in clusters don’t consider these variations and subsequent needs. As a result, the computing power offered by clusters is not harnessed properly, compromising both application performance and resource efficiency. To tackle these issues, we propose a History-Based Resource Manager for bioinformatics workflows applications running on clusters with heterogeneous nodes. The proposed resource manager features a prediction model that generates multiple performance predictions for each job under different combinations of cluster resources. Furthermore, the proposed resource manager includes a scheduling algorithm that considers the degree of multiprogramming of the nodes, scheduling combinations of applications for simultaneous same-node execution upon their compatibility. To test the proposed resource manager, we process two workloads formed by different amounts of workflows made up by common bioinformatics applications. Results prove that for the given cases, the proposed resource manager improves the performance obtained with SLURM, using First Come First Served policy. The proposal shows an average workflow makespan improvement range between 28 and 35%, averaging 32%, an average workflow efficiency improvement range between 75 and 83%, averaging 79%, and an average resource usage improvement range between 96 and 101%, averaging 99%. Furthermore, the proposed scheduling algorithm can improve the average workflow makespan by a range of values between 26 and 36%, averaging 31%, compared to Max–Min and Min–Min algorithms.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0885-7458
1573-7640
DOI:10.1007/s10766-018-0600-z