Deploying Science Gateways with Apache Airavata Using SciGaP Hosted Services
Science gateways provide user interfaces to high performance computing, cloud computing, and other scientific research infrastructure for researchers who are unfamiliar with or need more capabilities than provided by command-line interfaces. Science gateways themselves can be developed from numerous high quality open source software packages, but operating the gateway remains a challenge. To address these issues, we present the Science Gateways Platform as a service (SciGaP.org), a hosted, multi-tenanted deployment of the open source Apache Airavata software framework for science gateways. SciGaP services are currently used to support over 40 active science gateways, connect with over 70 different High Performance Computing and cluster computing systems, and provide access to over 200 scientific applications. SciGaP-supported have delivered over 230 million computing hours since 2016 to scientists in fields including computational chemistry, biophysics, computational fluid dynamics, and bioinformatics. This tutorial will demonstrate the capabilities of science gateways built using Apache Airavata, progressing from demonstrations of the turnkey Django Portal for Airavata’s user and administration interfaces to exercises covering intermediate and advanced customization of the portal user interface using the Apache Airavata API. In working through the demonstrations and exercises, we demonstrate Apache Airavata’s remote job and data management capabilities, and its ability to enable users to save, clone, and share computational experiments with collaborators.
Securing Science Gateways with Custos Services
Science gateways and virtual research environments have become essential tools for many thousands of researchers, educators, and students. Cybersecurity services that provide best-of-breed solutions for managing users, groups, and access to distributed research resources are critically important to ensuring that gateways are and remain trusted by both users and resource providers to control and protect access to accounts, user information, scientific data, and resources. The authors present a tutorial on Custos, a cybersecurity service based on open source software that helps science gateways manage user identities, integrate with federated authentication systems, manage secrets such as OAuth2 access tokens and SSH keys needed to connect to remote resources, and manage groups and access permissions to digital objects. Custos is particularly intended to support scenarios that integrate the above features into end-to-end usage scenarios. This tutorial will offer an overview of Custos’s capabilities, provide hands-on exercises on using its features, demonstrate to science gateway providers how to integrate the services into their gateways with software development kits for the Custos API, introduce developers to the code and how to review and contribute to it, supply gateway providers with information on how Custos services are deployed for high availability and fault tolerance, and how Custos operations handle incident response.
Deep Dive into Constructing Containers for Scientific Computing and Gateways
In recent years, using containers has been rapidly gaining traction as a solution to to lower the barriers to using more software on HPC and cloud resources. However, significant barriers still exist to actually doing this in practice, particularly for well-established community codes which expect to run on a particular operating system version or resource. Additional barriers exist for researchers unfamiliar with containerization technologies. While many beginner tutorials are available for building containers, they often stop short of covering the complexities that can arise when containerizing scientific computing software. The goal of this full-day tutorial is to demonstrate and work through building and running non-trivial containers with users. We will containerize community scientific software, exhibit how to share with a larger community via a container registry, and then run on a completely separate HPC resource, with and without the use of a Science Gateway. The subject matter will be approachable for intermediate to advanced users, and is expected to be of interest to a diverse audience including researchers, support staff, and teams building science gateways.
Pegasus 5.0 Workflows
Workflows are a key technology for enabling complex scientific computations. They capture the interdependencies between processing steps in data analysis and simulation pipelines as well as the mechanisms to execute those steps reliably and efficiently. Workflows can capture complex processes to promote sharing and reuse, and also provide provenance information necessary for the verification of scientific results and scientific reproducibility.
Pegasus (https://pegasus.isi.edu) is being used in a number of scientific domains doing production grade science. In 2016 the LIGO gravitational wave experiment used Pegasus to analyze instrumental data and confirm the first detection of a gravitational wave. The Southern California Earthquake Center (SCEC) based at USC, uses a Pegasus managed workflow infrastructure called Cybershake to generate hazard maps for the Southern California region. In March 2017, SCEC conducted a CyberShake study on DOE systems ORNL Titan and NCSA BlueWaters. Overall, the study required 450,000 node-hours of computation across the two systems. Pegasus is also being used in astronomy, bioinformatics, civil engineering, climate modeling, earthquake science, molecular dynamics and other complex analyses. In 2020, we released Pegasus 5.0 that is a major improvement over previous releases. Pegasus 5.0 provides a brand new Python3 workflow API developed from the ground up so that, in addition to generating the abstract workflow and all the catalogs, it now allows you to plan, submit, monitor, analyze and generate statistics of your workflow.
The goal of the tutorial is to introduce the benefits of modelling pipelines in a portable way with use of scientific workflows. We will examine the workflow lifecycle at a high level and issues and challenges associated with various steps in the workflow lifecycle such as creation, execution and monitoring and debugging. Through hands-on exercises in a hosted Jupyter notebook environment, we will describe an application pipeline as a Pegasus workflow using Pegasus Workflow API and execute the pipeline on distributed computing infrastructures. The attendees will leave the tutorial with knowledge on how to model their pipelines in a portable fashion using Pegasus workflow and run them on varied computing environments.
GPU code optimization and auto-tuning made easy with Kernel Tuner: A hands-on, bring your own code tutorial
Graphics Processing Units (GPUs) are enabling and accelerating many applications in science and industry. The performance of GPUs for computing applications strongly depends on how the software has been optimized for the hardware. There are many different implementations and code optimizations to consider that can also be parameterized, creating a large, non-continuous search space that is infeasible to search by hand. Moreover, the optimal kernel configuration is often different for different hardware or when the application is executed on a different problem. As such, automated performance tuning (auto-tuning) techniques are crucial to optimize such applications.
In this tutorial, you will learn how to use Kernel Tuner, an easy-to-use tool for auto-tuning GPU codes using simple Python scripts. Kernel Tuner supports OpenCL, CUDA, C++, and Fortran. We will take a step-by-step approach to explain the auto-tuning basics, such as tuning GPU kernel thread block dimensions, building up to more complex search spaces with many tunable parameters and interdependencies.
We will cover a wide range of topics from the basics of GPU code optimization, parameterizing code, and verifying the output of kernels with many different parameterizations, to using advance search space optimization strategies to accelerate the auto-tuning process.
Participants are welcome to bring their own code for the exercises, but we also have plenty of code examples to work with. Experienced GPU programmers will join as mentors to assist participants with using Kernel Tuner on their code. Participants can use Jupyter Notebooks and Google Colab if they don’t have access to a machine with GPUs.
Search, manage, visualize and share your data with SeedMeLab
Data is an integral part of scientific research. With a rapid growth in data collection and generation capability and an increasingly collaborative nature of research activities, data management and data sharing have become central and key to accomplishing research goals. Researchers today have variety of solutions at their disposal from local storage to Cloud based storage. However, most of these solutions focus on hierarchical file and folder organization. While such an organization is pervasively used and quite useful, it relegates information about the context of the data such as description and collaborative notes about the data to external systems. This spread of information into different silos impedes the flow research activities.
In this tutorial, we will introduce and provide hands on experience with the SeedMeLab, a scientific data management system for teams struggling with intractable data organization and data access. SeedMeLab enables research groups to manage, share, search, visualize, and present their data in a web-based environment using an access-controlled, branded, and customizable website they own and control. It supports storing and viewing data in a familiar tree hierarchy, but also supports formatted annotations, lightweight visualizations, and threaded comments on any file/folder. The system can be easily extended and customized to support metadata, job parameters, and other domain and project-specific contextual items. The software is open source and available as an extension to the popular Drupal content management system.
Portable Programming of Heterogeneous Parallel Systems with SkePU
SkePU is a C++ based open-source framework for high-level portable programming of heterogeneous parallel systems using algorithmic skeletons. Skeletons are generic programming constructs based on higher-order functions, such as map, stencil, reduce, scan, that implement algorithmic patterns for which platform-specific parallel implementations may exist. Skeletons provide a high degree of abstraction and portability with a quasi-sequential programming interface, as their implementations encapsulate all low-level and platform-specific details such as parallelization, synchronization, communication, memory management, accelerator usage and other optimizations.
From the same high-level source code using SkePU skeletons and data-containers, the framework generates target code for multicore CPU execution, CUDA and OpenCL code for execution on GPU and multi-GPU systems (and other OpenCL devices such as Xeon-Phi), hybrid CPU-GPU execution, and on HPC clusters. SkePU transparently performs a number of run-time optimizations, including data transfer minimization, data locality optimizations and auto-tuned back-end selection.
The SkePU framework is organized as an include-only template library plus a light-weight pre-compiler based on LLVM clang; the cluster backend uses the MPI support in the StarPU runtime system. The development of SkePU started in 2010, and we recently presented the third generation of the framework. Beyond cooperative research projects such as H2020 EXA2PRO, we use SkePU since several years as a lab environment for teaching high-level parallel programming in our master-level course on multicore and GPU programming at Linköping University, Sweden.
In this tutorial we present SkePU with its skeletons and data-container types, guide participants through some hands-on programming exercises in SkePU, and also point out some of the more recently added new features such as multi-variant user functions (which can be used by platform expert programmers to exploit e.g. custom SIMD instructions without compromising the portability of the SkePU program). The source code of SkePU is available at https://skepu.github.io.
