Building a scientific workflow framework to enable real‐time machine learning and visualization

Summary Nowadays, we have entered the era of big data. In the area of high performance computing, large‐scale simulations can generate huge amounts of data with potentially critical information. However, these data are usually saved in intermediate files and are not instantly visible until advanced...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Concurrency and computation Jg. 31; H. 16
Hauptverfasser: Li, Feng, Song, Fengguang
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Hoboken Wiley Subscription Services, Inc 25.08.2019
Schlagworte:
Algorithms
Analytics
Anomalies
Artificial intelligence
Clustering
Computational fluid dynamics
Computer simulation
Data analysis
DataSpaces
Disks
Fluid flow
high performance computing
Machine learning
real‐time online data analytics
scientific workflows
Simulation
Turbulent flow
Visualization
Workflow
ISSN:1532-0626, 1532-0634
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Summary Nowadays, we have entered the era of big data. In the area of high performance computing, large‐scale simulations can generate huge amounts of data with potentially critical information. However, these data are usually saved in intermediate files and are not instantly visible until advanced data analytics techniques are applied after reading all simulation data from persistent storages (eg, local disks or a parallel file system). This approach puts users in a situation where they spend long time on waiting for running simulations while not knowing the status of the running job. In this paper, we build a new computational framework to couple scientific simulations with multi‐step machine learning processes and in‐situ data visualizations. We also design a new scalable simulation‐time clustering algorithm to automatically detect fluid flow anomalies. This computational framework is built upon different software components and provides plug‐in data analysis and visualization functions over complex scientific workflows. With this advanced framework, users can monitor and get real‐time notifications of special patterns or anomalies from ongoing extreme‐scale turbulent flow simulations.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1532-0626
1532-0634
DOI:10.1002/cpe.4703