HPC systems are increasingly used for data science, machine learning, and AI. These systems offer more computational power and memory than a typical laptop or workstation. However, there are some pitfalls to avoid when runnning such workloads, especially regarding I/O.
This session will cover a number of these pitfalls and how to avoid them. It will also cover some best practices.
Learning outcomes
When you complete this training you will
- have a better understanding of I/O challenges on HPC systems;
- be able to avoid common pitfalls on HPC systems;
- be able to make an informed choice on data formats;
- understand the challenges of running applications that load many packages/modules;
- know some best practices for running Python and R applciations on HPC systems.
Schedule
Total duration: 1 hours
| Subject | Duration |
|---|---|
| introduction and motivation | 5 min. |
| data formats | 40 min. |
| using containers | 10 min. |
| wrap up | 5 min. |
Training materials
Slides can be viewed directly on the web. Example code and benchmarks can be found in the GitHub repository.
Target audience
This training is for you if you need to run I/O intensive workloads on HPC systems.
Prerequisites
You will need a working knowlegde of the Linux command line and HPC systems. A working knowledge of Python is a plus.
Trainer(s)
- Geert Jan Bex (geertjan.bex@uhasselt.be)