Data Parallelism Workshop: How to Train Deep Learning Models on Multiple GPUs

Please note: Those interested in attending this pre-symposium workshop are strongly encouraged to first register for onsite attendance at the Frontiers in NeuroAI symposium. Workshop preference will be given to in-person symposium registrants on a first-come, first-served basis.

About this Course

Modern deep learning challenges leverage increasingly larger datasets and more complex models. As a result, significant computational power is required to train models effectively and efficiently. Learning to distribute data across multiple GPUs during deep learning model training makes possible an incredible wealth of new applications utilizing deep learning.

Additionally, the effective use of systems with multiple GPUs reduces training time, allowing for faster application development and much faster iteration cycles. Teams who are able to perform training using multiple GPUs will have an edge, building models trained on more data in shorter periods of time and with greater engineer productivity.

This workshop teaches you techniques for data-parallel deep learning training on multiple GPUs to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.

Learning Objectives

By participating in this workshop, you’ll:

• Understand how data parallel deep learning training is performed using multiple GPUs
• Achieve maximum throughput when training, for the best use of multiple GPUs
• Distribute training to multiple GPUs using Pytorch Distributed Data Parallel
• Understand and utilize algorithmic considerations specific to multi-GPU training performance and accuracy

Certification

The hands-on portion of the workshop will be conducted using NVIDIA’s Deep Learning Institute (DLI) infrastructure. Participants who complete the workshop and successfully pass the assessment will receive a NVIDIA DLI certificate of completion.

Course Prerequisites

• Experience with deep learning training using Python/PyTorch (this course provides excellent preparation)
• Participants must bring a laptop computer capable of running the latest version of Chrome or Firefox. Each participant will be provided with dedicated access to a fully configured, GPU-accelerated workstation in the cloud.