Welcome to the UvA Deep Learning Tutorials!¶
Note
Interested in learning JAX? We have recently started translating the notebooks from PyTorch to JAX+Flax. Check out our new notebooks in the tab Deep Learning 1 (JAX+Flax) to learn how you can speedup the model training with JAX!
For this year’s course edition, we created a series of Jupyter notebooks that are designed to help you understanding the “theory” from the lectures by seeing corresponding implementations. We will visit various topics such as optimization techniques, transformers, graph neural networks, and more (for a full list, see below). The notebooks are there to help you understand the material and teach you details of the PyTorch framework, including PyTorch Lightning. Further, we provide one-to-one translations of the notebooks to JAX+Flax as alternative framework.
The notebooks are presented in the first hour of every group tutorial session. During the tutorial sessions, we will present the content and explain the implementation of the notebooks. You can decide yourself whether you just want to look at the filled notebook, want to try it yourself, or code along during the practical session. The notebooks are not directly part of any mandatory assignments on which you would be graded or similarly. However, we encourage you to get familiar with the notebooks and experiment or extend them yourself. Further, the content presented will be relevant for the graded assignment and exam.
The tutorials have been integrated as official tutorials of PyTorch Lightning. Thus, you can also view them in their documentation.
Schedule (Deep Learning 1, edition 2023)¶
Date |
Notebook |
Tuesday, 31. October 2023, 15:00-16:00 |
Tutorial 2: Introduction to PyTorch |
Tuesday, 7. November 2023, 15:00-16:00 |
Tutorial 3: Activation functions |
Tuesday, 14. November 2023, 15:00-16:00 |
Tutorial 4: Optimization and Initialization |
Tuesday, 21. November 2023, 15:00-16:00 |
Tutorial 5: Inception, ResNet and DenseNet |
Tuesday, 28. November 2023, 15:00-16:00 |
Tutorial 6: Transformers and Multi-Head Attention |
Tuesday, 5. December 2023, 15:00-16:00 |
Tutorial 7: Graph Neural Networks |
Tuesday, 12. December 2023, 15:00-16:00 |
Tutorial 17: Self-Supervised Contrastive Learning with SimCLR |
How to run the notebooks¶
On this website, you will find the notebooks exported into a HTML format so that you can read them from whatever device you prefer. However, we suggest that you also give them a try and run them yourself. There are three main ways of running the notebooks we recommend:
Locally on CPU: All notebooks are stored on the github repository that also builds this website. You can find them here: https://github.com/phlippe/uvadlc_notebooks/tree/master/docs/tutorial_notebooks. The notebooks are designed so that you can execute them on common laptops without the necessity of a GPU. We provide pretrained models that are automatically downloaded when running the notebooks, or can manually be downloaded from this Google Drive. The required disk space for the pretrained models and datasets is less than 1GB. To ensure that you have all the right python packages installed, we provide a conda environment in the same repository (choose the CPU or GPU version depending on your system).
Google Colab: If you prefer to run the notebooks on a different platform than your own computer, or want to experiment with GPU support, we recommend using Google Colab. Each notebook on this documentation website has a badge with a link to open it on Google Colab. Remember to enable GPU support before running the notebook (
Runtime -> Change runtime type
). Each notebook can be executed independently, and doesn’t require you to connect your Google Drive or similar. However, when closing the session, changes might be lost if you don’t save it to your local computer or have copied the notebook to your Google Drive beforehand.Snellius cluster: If you want to train your own (larger) neural networks based on the notebooks, you can make use of the Snellius cluster. However, this is only suggested if you really want to train a new model, and use the other two options to go through the discussion and analysis of the models. Snellius might not allow you with your student account to run Jupyter notebooks directly on the gpu partition. Instead, you can first convert the notebooks to a script using
jupyter nbconvert --to script ...ipynb
, and then start a job on Snellius for running the script. A few advices when running on Snellius:Disable the tqdm statements in the notebook. Otherwise your slurm output file might overflow and be several MB large. In PyTorch Lightning, you can do this by setting
enable_progress_bar=False
in the trainer.Comment out the matplotlib plotting statements, or change
plt.show()
toplt.savefig(...)
.
Tutorial-Lecture alignment¶
We will discuss 7 of the tutorials in the course, spread across lectures to cover something from every area. You can align the tutorials with the lectures based on their topics. The list of tutorials in the Deep Learning 1 course is:
Guide 1: Working with the Snellius cluster
Tutorial 2: Introduction to PyTorch
Tutorial 3: Activation functions
Tutorial 4: Optimization and Initialization
Tutorial 5: Inception, ResNet and DenseNet
Tutorial 6: Transformers and Multi-Head Attention
Tutorial 7: Graph Neural Networks
Tutorial 8: Deep Energy Models
Tutorial 9: Autoencoders
Tutorial 10: Adversarial attacks
Tutorial 11: Normalizing Flows on image modeling
Tutorial 12: Autoregressive Image Modeling
Tutorial 15: Vision Transformers
Tutorial 16: Meta Learning - Learning to Learn
Tutorial 17: Self-Supervised Contrastive Learning with SimCLR
Feedback, Questions or Contributions¶
This is the first time we present these tutorials during the Deep Learning course. As with any other project, small bugs and issues are expected. We appreciate any feedback from students, whether it is about a spelling mistake, implementation bug, or suggestions for improvements/additions to the notebooks. Please use the following link to submit feedback, or feel free to reach out to me directly per mail (p dot lippe at uva dot nl), or grab me during any TA session.
If you find the tutorials helpful and would like to cite them, you can use the following bibtex:
@misc{lippe2024uvadlc,
title = {{UvA Deep Learning Tutorials}},
author = {Phillip Lippe},
year = 2024,
howpublished = {\url{https://uvadlc-notebooks.readthedocs.io/en/latest/}}
}
- Overview
- Part 1.1: Training Larger Models on a Single GPU
- Part 1.2: Profiling and Scaling Single-GPU Transformer Models
- Part 2.1: Introduction to Distributed Computing in JAX
- Part 2.2: (Fully-Sharded) Data Parallelism
- Part 3.1: Pipeline Parallelism
- Part 3.2: Looping Pipelines
- Part 4.1: Tensor Parallelism
- Part 4.2: Asynchronous Linear Layers with Tensor Parallelism
- Part 4.3: Transformers with Tensor Parallelism
- Part 5: Language Modeling with 3D Parallelism
- Tutorial 2: Introduction to PyTorch
- Tutorial 3: Activation Functions
- Tutorial 4: Optimization and Initialization
- Tutorial 5: Inception, ResNet and DenseNet
- Tutorial 6: Transformers and Multi-Head Attention
- Tutorial 7: Graph Neural Networks
- Tutorial 8: Deep Energy-Based Generative Models
- Tutorial 9: Deep Autoencoders
- Tutorial 10: Adversarial attacks
- Tutorial 11: Normalizing Flows for image modeling
- Tutorial 12: Autoregressive Image Modeling
- Tutorial 15: Vision Transformers
- Tutorial 16: Meta-Learning - Learning to Learn
- Tutorial 17: Self-Supervised Contrastive Learning with SimCLR
- Tutorial 2 (JAX): Introduction to JAX+Flax
- Tutorial 3 (JAX): Activation Functions
- Tutorial 4 (JAX): Optimization and Initialization
- Tutorial 5 (JAX): Inception, ResNet and DenseNet
- Tutorial 6 (JAX): Transformers and Multi-Head Attention
- Tutorial 7 (JAX): Graph Neural Networks
- Tutorial 9 (JAX): Deep Autoencoders
- Tutorial 11 (JAX): Normalizing Flows for image modeling
- Tutorial 12 (JAX): Autoregressive Image Modeling
- Tutorial 15 (JAX): Vision Transformers
- Tutorial 17 (JAX): Self-Supervised Contrastive Learning with SimCLR
- GDL - Regular Group Convolutions
- GDL - Steerable CNNs
- DPM1 - Deep Probabilistic Models I
- DPM2 - Variational inference for deep discrete latent variable models
- DPM 2 - Variational Inference for Deep Continuous LVMs
- AGM - Advanced Topics in Normalizing Flows - 1x1 convolution
- HDL - Introduction to HyperParameter Tuning
- HDL - Introduction to Multi GPU Programming
- Tutorial 1: Bayesian Neural Networks with Pyro
- Bayesian Neural Networks
- Simulate data
- Getting started with Pyro
- Bayesian Neural Network with Gaussian Prior and Likelihood
- Define and run Markov chain Monte Carlo sampler
- Exercise 1: Deep Bayesian Neural Network
- Train BNNs with mean-field variational inference
- Exercise 2: Bayesian updating with variational inference
- Bayesian update
- Tutorial 2: Comparison to other methods of uncertainty quantification
- DNN - Tutorial 2 Part I: Physics inspired Machine Learning
- DNN - Tutorial 2 Part II: Physics inspired Machine Learning
- DS - Dynamical Systems & Neural ODEs
- SGA - Sampling Discrete Structures
- SGA - Sampling Subsets with Gumbel-Top \(k\) Relaxations
- SGA: Learning Latent Permutations with Gumbel-Sinkhorn Networks
- SGA - Graph Sampling for Neural Relational Inference
- CRL - Causal Identifiability from Temporal Intervened Sequences