High Performance Container Workshop

Virtual HPCW 2020

The workshop series assembled thought leaders to provide the audience with the ‘state of containers’ and the latest trends.
As ISC moved to a virtual event, we went virtual as well pre-recording a stream of information. At the time of the workshop we elease the videos on youtube.

Instead of having one long sitting, we are going to break the workshop into three sessions that will all have similar structure:

As tools we are going to stream the video on YouTube and use the #hpcw20 channel within hpc-containers.slack.com (Invitation Link if you are not registered already) for communication (live-chat, polls and collect questions) with the panelist and Q&A.
Plese do make sure that you add yourself to the slack channel.

Agenda

Runtime

Recorded: (6/16 - 5PM CEST)

The first segment starts building from the ground up by introducing container runtimes and why HPC did not adopt standard runtimes. Afterwards the leading project presented the current state of the art, the panelist concluded the segment by discussing the area with the community.

# Duration Title Speaker Company Links
1 5min Introduction and Scope Christian Kniep AWS Youtube
2 10min Container Runtime Overview Andrew Young Sandia Labs Youtube
3 10min The podman runtime Valentin Rothberg RedHat Youtube
4 10min State of Singularity Ian Kaneshiro CtrlCmd Youtube
5 10min State of SARUS Lucas Benedicic CSCS Youtube
  45min Live Panel / Q&A     Youtube

Build

Recorded: (6/16 - 7PM CEST)

This segment focuses on building of images as an artifact, how recepies look like and what the end-user might worry about when defining the artifact.

# Duration Title Speaker Company Links
1 5min Introduction and Scope Christian Kniep AWS Youtube
2 10min State of BuildKit Nicholas Dille Haufe.Group Youtube
3 10min Buildah, a tool to build OCI images Carlos E Arango RedHat Youtube
4 10min State of Singularity Build Greg Kurzter CtrlCmd Youtube
5 10min Build Tools (Spack/Easybuild/…) Kenneth Hoste Ghent University Youtube
6 5min HPC Container Maker Update CJ Newburn NVIDIA Youtube
7 10min End-user artifact to specify an image? Johannes Köster University of Düsseldorf Youtube
  45min Live Panel / Q&A     Youtube

Distribute

Recorded: (6/17 - 5PM CEST)

Once an image is build it needs to be distributed - this segment focuses on how that can be done in a scalable and secure manner.

# Duration Title Speaker Company Links
1 5min Introduction and Scope Christian Kniep AWS Youtube
2 10min OCI Image Spec Carlos E Arango RedHat Youtube
3 10min Singularity Image Format Ian Kaneshiro CtrlCmd Youtube
4 10min Skopeo Distribution Tool Valentin Rothberg RedHat Youtube
5 10min Trust when distributing images Brandon Lum IBM Youtube
6 10min Container Registries Phil Estes IBM Youtube
7 5min Findability of Bioinformatic Container Images Johannes Köster University of Düsseldorf Youtube
8 5min Biocontainers and Conda Björn Grüning University of Freiburg Youtube
  45min Live Panel / Q&A     Youtube

Orchestration/Scheduling

Recorded: (6/17 - 7PM CEST)

Starting with a simple scheduler like SLURM this segment builds up to more complex schedulers (K8s) and workflow managers (Nextflow, Argo, Airflow).

# Duration Title Speaker Company Links
1 5min Introduction and Scope Christian Kniep AWS Youtube
2 10min Simple Container Orchestration Holger Gantikow ATOS Youtube
3 10min Orchestration/HPC with Kubernetes Carlos E. Arango RedHat Youtube
5 10min Portable workflow orchestration at scale with Nextflow Paolo Di Tommaso Seqera Labs Youtube
6 10min Workflow Schedulers (e.g. Argo, Airflow) Bill Sparks HPE Youtube
  45min Live Panel / Q&A     Youtube

HPC Specific

Recorded: (6/18 - 5PM CEST)

Approaching the meet on the bone we discussed the particularities of HPC. Device integration, POSIX file-systems, MPI(/PMI) and scale in general.

# Duration Title Speaker Company Links
1 5min Introduction and Scope Christian Kniep AWS Youtube
2 10min HPC Device Integration (kernel-/userland) CJ Newburn NVIDIA Youtube
3 10min How do container scale? (hpc systems vs cluster) Bill Sparks HPE Youtube
4 10min High- and lowlights when moving HPC apps? Saber Feki, Saber Feki, Nagarajan Kathiresan, Mohsin Shaikh, Samuel Kortas, Amr Abdelazim KAUST Youtube
5 10min MPI / PMI Update Josh Hursey IBM Youtube
  45min Live Panel / Q&A     Youtube

Use-cases and Outlook

Recorded: (6/18 - 7PM CEST)

We looked back on 6 years of this workshop, got a glimps into how big centers run containers successfully and how HPC use-cases evolved.

# Duration Title Speaker Company Links
1 5min Introduction and Scope Christian Kniep AWS Youtube
2 15min 6y HPCW Containerization ReCap Burak Yenier TheUberCloud Youtube
3 10min NERSC Overview and Outlook Shane Canon NERSC Youtube
4 10min NERSC Overview and Outlook Lucas Benedicic CSCS Youtube
4 10min Orchestration for the Edge CJ Newburn NVIDIA Youtube
5 10min ECP Supercontainer Project Andrew Younge Sandia Youtube
  45min Live Panel / Q&A     Youtube
    - Trends and Projects to be aware of?      
    - What is next? Compute models?      
    - Why didn’t k8s take over yet?      
    - How can the HPC community schmooze the non-HPC?      

Workshop ReCap

Recorded: (6/25 - 5PM CEST)

# Duration Title Links
  45min ReCap Panel 1w after stream Youtube