Episode 96 · Engineering Deep Dives

Running the Heat Equation Backwards: Diffusion Models and Flow Matching

Sankalp Thakur · 67 minutes

Pipeline integrity engineer Wren Okafor watches a dye pulse smear out in a water line, and her friend Teodor Lindqvist, a machine learning engineer, claims his whole field runs that smearing backwards. Starting from the score (the gradient of log density, which makes the normalising constant vanish), the episode builds Langevin dynamics, Song and Ermon's annealed noise-conditional score networks, and Ho, Jain and Abbeel's DDPM (2020): the variance-preserving forward process, the noise-prediction loss, and its equivalence to denoising score matching. It then covers the score-SDE view (Song et al., ICLR 2021), Anderson's reverse-time SDE, the Fokker–Planck equation (advection plus diffusion, and literally the heat equation in the variance-exploding case), and the deterministic probability-flow ODE. Next come fast samplers (DDIM, DPM-Solver, Karras's EDM), classifier and classifier-free guidance, latent diffusion and Stable Diffusion, and diffusion transformers. The second half moves from noise to transport: continuous normalising flows and the continuity equation, flow matching (Lipman et al.), rectified flow and reflow (Liu et al.), stochastic interpolants (Albergo and Vanden-Eijnden), Monge, Kantorovich and Benamou–Brenier optimal transport and minibatch OT, then consistency models and distillation. It closes with the state of the art, each claim tiered: SD3's rectified-flow MM-DiT, FLUX.1, FLUX.2 and FLUX 3, video models, masked diffusion language models (LLaDA, Mercury, Gemini Diffusion), Diffusion Policy and pi-zero in robotics, RFdiffusion and AlphaFold 3 in protein science, and GenCast in weather. Builds on ep69 (PDEs), ep70 (functional analysis), ep15 (information theory), ep22 (probability) and ep34 (inference economics).

Listen to the show on your preferred podcast platform.