From Laptop to Supercomputer: Building Production ML Pipelines with dagster-slurm

Bridge the gap between data science prototypes and production-scale machine-learning (ML) workloads on HPC clusters. Learn how dagster-slurm brings modern orchestration to supercomputing environments without rewriting your code and how this allows to use EU sovereign cost-efficient GPUs.

As machine learning (ML) models grow in complexity and datasets expand beyond what a single machine can handle, data scientists and ML engineers face a critical challenge: how to scale their work from a laptop to high-performance computing infrastructure without completely rewriting their code or becoming HPC experts.

This session introduces dagster-slurm, an open-source integration for running Dagster assets on Slurm HPC clusters. dagster-slurm lets you take the same Dagster pipelines from a laptop to a Slurm-backed supercomputer with minimal configuration changes. As an additional benefit you can profit from EU sovereign cost efficient GPUs.

We will explore:

• Why this matters: The gap between local development and HPC production, and why traditional approaches create friction
• How it works: A live demonstration showing the same pipeline running locally and on a real supercomputer (MUSICA with GPUs)
• Key capabilities: Environment reproducibility with pixi, distributed computing with Ray, monitoring, and automatic dependency management

The session includes live demonstrations:

• Transforming a simple Python script into a scalable HPC pipeline in minutes
• A production document processing workflow with Docling + Ray on GPU infrastructure for RAG data preprocessing

Ideally, prior python experience is available. But we'll introduce the concepts as we go. If you write Python and want to scale your ML workloads beyond a single machine, this session will show you a practical path forward.

Target audience

Everyone is welcome who wants to scale data science and machine learning workflows beyond a single computer and benefit from EU sovereign compute.

This webinar is particularly relevant for:

• Data Scientists & ML Engineers looking to leverage HPC resources
• MLOps Engineers seeking better orchestration for HPC workloads
• Research Software Engineers bridging academic and production systems
• Platform Engineers evaluating HPC orchestration solutions

This webinar is open and free of charge for all participants from academia, industry, and public administration from EU and/or EuroHPC JU member countries.

Entry level & prerequisites

Intermediate

• Basic Python programming experience is required.
• Familiarity with ML/data processing workflows is helpful.
• No prior knowledge of HPC, Slurm, or Dagster is required.

Agenda

10:00   Welcome & Presentation of AI Factory Austria AI:AT

10:05   Introduction to dagster-slurm

10:10   Live Demo 1: From Local Script to HPC Cluster

10:20   Core Concepts: Environment Management & Orchestration

10:35   Live Demo 2: Production Pipeline with Docling + Ray on GPUs

10:50   Questions & Discussion

Speaker

Georg Heiler (ASCII Supply Networks & T-Mobile Austria)     
Georg is a Senior Data Expert at T-Mobile Austria (Magenta) and an Research Software Engineer at Complexity Science Hub and ASCII Supply Networks, where he solves complex challenges with data at scale.    
At Magenta, Georg leads the cloud migration of enterprise data platforms, architecting modern data infrastructure for Austria's leading telecommunications provider. At ASCII, he tackles large-scale multi-modal ML-Ops challenges, building systems that process large datasets of supply chain data.    
Georg is the creator and lead maintainer of dagster-slurm, an open-source framework connecting modern orchestration with high-performance computing. He also contributes to metaxy, a toolkit for supply chain knowledge graph construction. His work bridges the gap between cutting-edge research infrastructure and production-grade systems, making supercomputing accessible to data science teams.

Connect & Project Links:

• Personal Website: https://georgheiler.com
• dagster-slurm GitHub: https://github.com/ascii-supply-networks/dagster-slurm
• metaxy GitHub: https://github.com/anam-org/metaxy
• Documentation: https://dagster-slurm.geoheil.com
• Presentation Slides: https://dagster-slurm.geoheil.com/slides/

Language

English

Date, time, and location

17.04.2026, 11:00 – 12:00 (CEST), LIVE ONLINE WEBINAR (Zoom)

Registration

Registration is required, the link to the registration form can be found on the top of this page in the menu on the left. Please register with your official institutional email address to prove your affiliation.

You will get the Zoom link in the automatic confirmation by email (subject starting with "[Indico] Registration"), please check your Spam/Junk folders). We will also send 1–2 reminders before the webinar.

Please do not hesitate to contact us at training@ai-at.eu if you have any questions.

Organizers

This webinar is jointly organized by AI Factory Austria AI:AT and Austrian Scientific Computing ASC (aka ASC Research Center, TU Wien).

Acknowledgements

AI Factory Austria AI:AT has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 101253078. The JU receives support from the Horizon Europe Programm of the European Union and Austria (BMIMI / FFG).