Deep learning

Below I list various sources that I found myself or through friends for people wishing to follow pedagogical tutorials on modern deep learning techniques.

General resources

Artificial Intelligence is highly dynamic and hyped field: AI Index is a good place to start if you want to get a sense of the field.

TIP

Join a journal club, or start your own where you and your peers would review the latest papers to stay updated on the trends.

I recommend joining the mailing list of our journal club which brings together geophysicists interested in the applications of machine learning.

This one is quite nice but targets french speakers: CNRS-Fidle. They cover the basics and the modern tools of machine learning from A to Z with practical exercises and offerring a convenient docker or an option to use personal python environment. They also have a youtube channel with all the lectures.

Transformers

A gentle introduction by Jay Alammar into how multi-head self-attention mechanism was implemented in the original paper on transformers. The jupyter notebook that is available is a bit dated since it is based on tensorflow 1. You will find other useful links in this introduction such as pytorch anotated code that goes with the famous article Attention is all you need.

This course on Natural Language Processing is recommended if you want practical tips on how to pick up pre-trained transformers from Hugging Face 🤗 repository and fine-tune them for your downstream task.

Interestingly some argue that transformers are a version of Graph Neural Network (GNN): Source: Gradient website

Diffusion models

A success of AI art such as DALL-E and Midjourney have made headlines. But what are they based on? Find out more about probabilistic denoising diffusion models:

• I really recommend the following repository for people who are interested in Langevin dynamics (which is at the heart of diffusion models), a tutorial by Prof. F. Rousseau on Diffusion Models: part A, part B supplied with a jupyter notebook.
• Materials from the 🤗 Huggingface Diffusion Models course have been recommended to me by Redouane Lguensat.

Tutorials for physics applications

MOOC Machine Learning in Weather & Climate from European Centre of Medium Range Weather Forecast (ECMWF). An ecxellent introduction for Earth science practitioners willing to dive in to how geophysicists apply machine learning to their domain. From statistichal post-processing of weather forecasts to physics-guided data-driven parametrizations of climate.

This paper is also a bit dated now, but offers a brief intro to Machine Learning to Physicists.