Optimal scheduling of in-situ analysis for large-scale scientific simulations - Malakar, Preeti; Vishwanath, Venkatram; Munson, Todd; Knight, Christopher; Hereld, Mark; Leyffer, Sven; Papka, Michael E. (Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2015)
Today's leadership computing facilities have enabled the execution of transformative simulations at unprecedented scales. However, analyzing the huge amount of output from these simulations remains a challenge. Most analyses of this output is performed in post-processing mode at the end of the simulation. The time to read the output for the analysis can be significantly high due to poor I/O bandwidth, which increases the end-to-end simulation-analysis time. Simulation-time analysis can reduce this end-to-end time. In this work, we present the scheduling of in-situ analysis as a numerical optimization problem to maximize the number of online analyses subject to resource constraints such as I/O bandwidth, network bandwidth, rate of computation and available memory. We demonstrate the effectiveness of our approach through two application case studies on the IBM Blue Gene/Q system.