Earth System Models (ESM) utilize and also produce a vast amount of data. Besides the bottlenecks within codes, limitations imposed by model in- and output (I/O) as well as couplers in an increasing manner determine performance of computational climate studies. To address this, we apply existing workflow solutions (e.g., known from Grid computing) to boost the overall performance starting and make coupling of eScience tools easier.
Scientific workflows for glaciology modelling
A collaboration between University of Iceland, CSC Finland and Uppsala University, Sweden studying the scientific workflow of coupling a continuum ice flow model (Elmer/Ice) for simulating the sliding of a glacier (example video) and a discrete particle ice model HiDEM for simulating the calving of a glacier (example video) which comes from a project associated to the Nordic Centre of Excellence SVALI.
While some semi-automated workflow existed already, it was based on a more than 1000 line shell script which was hard to understand, hard to modify, and thus hard to port to a different high-performance computing (HPC) cluster. In addition, it was not possible to easily monitor the running HPC executions and to, e.g. re-start failed HPC jobs.
As part of eSTICC, a fully automated high-level workflow has been created by using the workflow engine of the middleware UNICORE.
The resulting high-level workflow addressed the above problems: the resulting workflow is easier to understand, to modify and thus easy to port to new platforms. In the course of this creating the high-level workflow, also inefficiencies in using the HPC clusters were addressed: the discrete particle ice model needs to a lot of CPUs, whereas the continuum ice flow model needs less CPUs, but still all the CPUs reserved for the discrete particle ice model were reserved, but to huge extend idle, for running the continuum ice flow model. As result of the workflow, it was even possible to use different clusters for the two coupled models, each optimally suited for the two different models thus saving precious HPC resources. Also other inefficiencies such as unnecessarily copying output files of one workflow step as input files of the subsequent step, were removed.
A video demo of the resulting automated workflow (based on the UNICORE rich client):
An example result of coupling the two ice models of sliding and calving:
Shahbaz Memon, Dorothee Vallot, Thomas Zwinger, Helmut Neukirchen.
Coupling of a continuum ice sheet model and a discrete element calving model using a scientific workflow system.
Abstract and talk at European Geosciences Union (EGU) General Assembly 2017 (session IE1.1/CR1.14/AS4.21/BG9.66 Atmosphere – Cryosphere interaction in the Arctic, high latitudes and mountains: Transport and deposition of aerosols, eScience and ensemble methods), Vienna, Austria, 23–28 April 2017. Geophysical Research Abstracts
Vol. 19, EGU2017-8499, Copernicus Publications, 2017.
Shahbaz Memon, Dorothée Vallot, Thomas Zwinger, Jan Åström, Helmut Neukirchen, Morris Riedel, Matthias Book
Scientific Workflows Applied to the Coupling of a Continuum (Elmer v8.3) and a Discrete Element (HiDEM v1.0) Ice Dynamic Model
Currently under review at Geoscientific Model Development (GMD), 2018.
Scientific workflows for remote sensing machine learning
The concept of scientific workflows is universal. In addition to ice modelling, we have also applied it to machine learning approaches used for remote sensing. The different steps of machine learning, such as cross-validation, training and prediction have been coupled together.
Shahbaz Memon, Gabriele Cavallaro, Björn Hagemeier, Morris Riedel, Helmut Neukirchen.
Automated Analysis of remotely sensed images using the UNICORE workflow management system.
Short paper at 2018 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), July 23-27 2018, Valencia, Spain, IEEE 2018 (to appear).
Shahbaz Memon, Gabriele Cavallaro, Morris Riedel, Helmut Neukirchen.
Facilitating efficient data analysis of remotely sensed images using standards-based Parameter Sweep models.
Short paper at 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), July 23–28, 2017, Fort Worth, Texas, USA. DOI: 10.1109/IGARSS.2017.8127797 IEEE, 2017.