Foundations of LLM Mastery: Fine-tuning on multi GPUs

Scaling fine-tuning of large language models (LLMs) to multiple GPUs can unlock new levels of performance and efficiency, making it accessible for industries of all sizes.

In this 3.5-hour course, participants from start-ups, SMEs, and large enterprises will gain hands-on experience with powerful multi-GPU fine-tuning techniques, optimizing their LLM workflows for both speed and scalability.

Key topics include:

• Distributed Data Parallel (DDP): Efficiently distribute data and model updates across multiple GPUs.
• ZeRO with DeepSpeed: Scale large models while managing memory effectively.
• Fully Sharded Data Parallel (FSDP): Yet another technique to reduce memory usage by sharding.
• Hugging Face Accelerate: Simplify multi-GPU training with intuitive tools and configurations.

Through guided exercises and real-world examples, this course provides participants with the skills to fine-tune LLMs efficiently across multiple GPUs, preparing them to tackle complex, large-scale projects.

By the end of this course, participants will have developed a foundational understanding of techniques to distribute model parameters and states accross GPUs on several nodes.

Agenda & Content

For a detailed timetable and additional information, please see Agenda & Content in the left menu.

Entry level

Intermediate

Prerequisites

Basic programming skills in Python.                           
Having worked with Pytorch would be beneficial.

Target audience

Course for academia, industry, and public administration.

Course format

This course will be delivered as a LIVE ONLINE COURSE (using Zoom).

Hands-on labs

All participants will get a temporary user account on LEONARDO to do the hands-on labs.

You will use your own laptop or workstation to connect conveniently from your browser to a LEONARDO Jupyterlab session and do the hands-on exercises.

Lecturers

Simeon Harrison (AI:AT – AI Factory Austria)

Martin Pfister (AI:AT – AI Factory Austria)

Language

English

Date, time, and location

27.11.2025, 09:00 – 12:30 CET, LIVE ONLINE COURSE (using Zoom)

Prices and eligibility

This course is partly funded by the EuroCC 2 project. Therefore, the course is open and free of charge for participants from academia, industry, and public administration from EU and/or EuroHPC JU member countries.

Registration

The registration form and deadlines are available from the left menu. The registration form might close even before the deadline if the number of registrations has reached its maximum (and will open again if there are cancelations).

Please register with your official institutional email address to prove your affiliation.

You will get an automatic confirmation via email (subject starting with "[Indico] Registration"), please check your Spam/Junk folders.

Following your successful registration, you will receive further information approximately 1 week before the course.

Waitinglist

After the number of registrations has reached its maximum or the registration form has been closed, you may want to send us an email stating that you are interested to be put on the waiting list (vacancies may occur due to cancelations, etc.).

To be able to do the hands-on labs on the ASC systems please provide your full international mobile-phone number for the two-factor authentication required to login to the ASC systems.

Modification, withdrawal & no-show policy

Your registration is binding. Please only register for the course if you are really going to attend.

You can update your registration data or withdraw your registration anytime before the registration form has been closed via the link "Manage my registration" which you can find at the bottom of your automatic email confirmation (subject starting with "[Indico] Registration").

Alternatively, or after the registration form has been closed, please inform us about your cancelation or any change in your registration data (especially your mobile-phone number) via email (training@asc.ac.at).

No-show policy: If you do not cancel and do not show up at the course you will be blacklisted and excluded from future training events.

Organizers

This course is jointly organized by AI:AT – AI Factory Austria and ASC – Austrian Scientific Computing (aka ASC Research Center, TU Wien).

Acknowledgements

AI:AT – AI Factory Austria has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 101253078. The JU receives support from the Horizon Europe Programm of the European Union and Austria (BMIMI / FFG).