Extreme-Scale Scientific Software Stack (E4S) Release 0.2

Summary

The second release of Exascale Computing Project (ECP) Software Technologies (ST) software, Extreme-Scale Scientific Software Stack (E4S) Release 0.2, includes a subset of ECP ST software products, and demonstrates the target approach for future delivery of the full ECP ST software stack. Also available are a number of ECP ST software products that support a Spack package, but are not yet fully interoperable. As the primary purpose of the 0.2 release is demonstrating the ST software stack release approach, not all ECP ST software products were targeted for this release. Software products were targeted primarily based on existing Spack package maturity, location within the scientific software stack, and ECP SDK developer experience with the software. Each release will include additional software products, with the ultimate goal of including all ECP ST software products.

For a detailed description of all ECP ST software see the recent Capability Assessment Report.

Interoperability Approach

While porting of individual scientific software products is challenging, achieving interoperability between packages is even more difficult. E4S uses a dual-pronged approach for achieving software interoperability: Spack and SDKs.

Spack

E4S uses the Spack packages manager for software delivery. Spack provides the ability to specify versions of software packages that are and are not interoperable. It is also a common build layer to not only E4S software, but also an enormous number of software packages outside of ECP ST. These features support achieving and maintaining interoperability between ST software packages.

Software Development Kits (SDKs)

An ECP ST Software Development Kit is a collection of related ECP ST software products (called packages) where coordination across package teams will improve usability and practices, and foster community growth among teams that develop similar and complementary capabilities. An SDK is more of a project than a product, although it involves several products. It can also be considered as an association of products and product teams. The activities that take place inside an SDK promote interoperability (where appropriate and logical) between products. The initial version 0.2 release of E4S contains member packages of one SDK - the Extreme-Scale Scientific Software Development Kit (xSDK). Future releases will incorporate additional SDKs that will be defined in the coming months.

ST Product Release Participation

At the time of the E4S 0.2 release, all ECP ST software products fall into one of three categories of release preparation.
  1. Full Release
  2. Partial Release
  3. Future Release

Full Release

ST software products in this category are available via Spack, and can be built in a common environment with the other products in this category. A specific, Spack-registered version for each product in this category is part of the 0.2 release.

List of Full Release Products (37)

  • Adios
  • Bolt
  • Caliper
  • Darshan
  • Gasnet
  • GEOPM
  • GlobalArrays
  • Gotcha
  • HDF5
  • HPCToolkit
  • Hypre
  • Jupyter
  • Kokkos
  • Legion
  • Libquo
  • Magma
  • MFEM
  • MPICH
  • OpenMPI
  • PAPI
  • Papyrus
  • Parallel netCDF
  • ParaView
  • PETSc/TAO
  • Program Database Toolkit (PDT)
  • Qthreads
  • Raja
  • SCR
  • Spack
  • Strumpack
  • Sundials
  • SuperLU
  • Swift/T
  • SZ
  • Tasmanian
  • TAU
  • Trilinos
  • VTKm
  • Umpire
  • UnifyCR
  • Veloc
  • xSDK
  • Zfp

Partial Release

ST software products in this category are available via Spack and may or may not build in a common environment with other products in the partial and full release categories. While a specific version of the products in this category has not been tested and selected for the release, the partial release designation makes it clear to users that these software products can be accessed via Spack.

List of Partial Release Products

  • ASCENT
  • Catalyst
  • Dyninst Binary Tools Suite
  • Flang
  • Flexible Computational Science Infrastructure (FleCSI)
  • libEnsemble
  • LLVM
  • mpiFileUtils
  • ROMIO
  • Visit

Future Release

The Spack package for ST software products in this category is either under development or in the planning stages. These products may or may not be available currently via other delivery mechanisms, but are not available via Spack at the time of the 0.2 release.

Obtaining E4S 0.2

The E4S 0.2 release can be obtained in one of two ways:
  1. Source release via Spack.
  2. Container-based binary and source release.

Spack Release

Spack contains packages for all of the products listed in the Full Release category above. General instructions for building software with Spack can be found at the Spack website. The Full Release packages can be built via Spack concurrently. For more information, see /usr/local/packages/ecp in the container referenced below. Additionally, all of the packages in the Partial Release category above support a Spack package. Questions concerning building those packages are deferred to the associated package development team.

Container Release

The container release contains binary versions of the Full Release packages listed above. A clone of Spack is also available in the container which can be used to compile the Full Release and Partial Release packages. Example Spack "recipes" (lists of configuration commands) are available in the container. See the README.txt file for more details. This release also includes an OVA file that has Docker, Charliecloud, Shifter, and Singularity preinstalled in it. The Docker container image is also available from Dockerhub: 

# docker pull exascaleproject/sdk:AHM19

Videos

Downloads

Docker

To use the Docker image, please install docker and download the ecp.tgz file from the link above: 
# wget http://tau.uoregon.edu/ecp.tgz
# gunzip -c ecp.tgz | docker load
# docker images
# docker run -i -t exascaleproject/sdk:AHM19 /bin/bash
Inside the Docker container: 
# which spack
# cd /usr/local/packages/ecp/apps/demo; cat README
# cd demo/trilinos/Zoltan; ./compile.sh; ls -l Zoltan
# ./run.sh

Singularity

# which singularity
# wget http://tau.uoregon.edu/ecp.simg
# singularity exec /home/livetau/ecp.simg /bin/bash --rcfile /etc/bashrc
# which spack
Replacing MPI On Theta at ALCF: Allocate two nodes: 
% qsub -A  -t 30 -n 2  -q debug-cache-quad  -I
% /projects/ECP_SDK/tutorial/run_job.sh
% cat /projects/ECP_SDK/tutorial/run_job.sh

module swap PrgEnv-intel PrgEnv-gnu
module swap cray-mpich cray-mpich-abi
export SINGULARITYENV_LIBWLM_DETECT=/opt/cray/wlm_detect/1.3.2-6.0.6.0_3.8__g388ccd5.ari/lib64
aprun -n 16 -N 8 singularity exec -H $HOME    -B /projects/ECP_SDK:/projects/ECP_SDK:ro -B /opt:/opt:ro -B /var/opt:/var/opt:ro /projects/ECP_SDK/containers/singularity/ecp.simg bash -c 'unset CRAYPE_VERSION; source /usr/local/packages/ecp/misc/bashrc; spack load   trilinos hypre parmetis hdf5 metis openblas superlu zlib netcdf matio boost@1.66.0 scalapack suite-sparse tau ;spack unload openmpi mpich ;  export LD_LIBRARY_PATH=$LIBWLM_DETECT:$CRAY_LD_LIBRARY_PATH:$CRAYPAT_LD_LIBRARY_PATH:$LD_LIBRARY_PATH   ; /projects/ECP_SDK/tutorial/demo/trilinos/Zoltan/Zoltan; '

Shifter

Replacing MPI with system MPI on Cori at NERSC: 
# shifterimg -v pull docker:exascaleproject/sdk:SC18
# shifter -E --image=exascaleproject/sdk:SC18 -- /bin/bash --rcfile /etc/bashrc
# which spack

# shifterimg images | grep exascaleproject

# salloc -N 2 -q interactive -t 00:30:00 --image=exascaleproject/sdk:AHM19 -C haswell -L SCRATCH
# srun -n 32 shifter  -- /bin/bash  -c 'unset CRAYPE_VERSION;  . /etc/bashrc ; spack load   trilinos hypre parmetis hdf5 metis openblas superlu zlib netcdf matio boost@1.66.0 scalapack suite-sparse tau; spack unload openmpi mpich; ./Zoltan'

Charliecloud

# wget http://tau.uoregon.edu/ecp-cc.tgz
# tar xf ecp-cc.tgz
# ch-run --bind=$HOME:$HOME ./ecp-cc -- /bin/bash --rcfile /etc/bashrc
# which spack
# cat /usr/local/packages/ecp/Acknowledgment.txt
If you wish to use the OVA file, please contact below for login information. The OVA file has Docker, Singularity, Shifter, and Charliecloud preinstalled.


Please contact Sameer Shende at sameer [at] cs.uoregon.edu if you have any questions.