"They poured static on me until I forgot what I was, then taught a network to un-forget me one grain at a time. I came back as a photograph of a cat I had never seen. The procedure was undignified, I admit, but the results speak for themselves."
A Diffusion Model, Halfway Through Denoising
A diffusion model learns to generate by mastering the opposite of destruction: take a clean image, add Gaussian noise in many small steps until nothing is left but static, then train a network to undo one step of that corruption, and you can start from pure static and walk all the way back to a brand-new image. This single idea, learned iterative denoising, is the engine behind Stable Diffusion, DALL-E, Midjourney, and the video and 3D generators of the chapters that follow. It is also a direct descendant of the denoising you met classically in Chapter 7 and learned in Chapter 31; the difference is that diffusion denoises not once but dozens of times, each step nudging samples toward the data distribution. This chapter builds the idea from the forward corruption process up, shows the three equivalent views that explain why it works (the variational view, the score-based view, and the flow view), then turns to the engineering that made it fast and controllable: efficient samplers, guidance, and the latent-space trick that let the whole thing run on a single consumer GPU.
Chapter Overview
Diffusion models have a history that runs further back than their 2020 breakthrough suggests. The forward-and-reverse framework was theoretically proposed by Sohl-Dickstein et al. in 2015, framed as a non-equilibrium thermodynamic process: a Markov chain gradually corrupts data into noise, and a learned reverse chain restores it. The idea was principled but produced blurry results on small images and attracted limited attention. What made diffusion competitive was a reparameterization introduced by Ho et al. in the 2020 DDPM paper: instead of training the network to predict the clean image $x_0$ directly, train it to predict the noise $\varepsilon$ that was added at step $t$. This noise-prediction target yields a dramatically simplified training loss: the mean-squared-error objective that the whole field now uses, and the resulting models immediately outperformed the best GANs on log-likelihood benchmarks while being far more stable to train. The five-year gap between the 2015 theory and the 2020 breakthrough was not a failure of the idea; it was the discovery of the right parameterization.
The recipe is almost suspiciously simple. The forward process takes a real image and adds a controlled amount of Gaussian noise, repeatedly, over hundreds of steps, until the image is statistically indistinguishable from random static. This process has no learnable parameters at all; it is just a fixed corruption schedule. The reverse process is where the learning happens: a neural network, almost always a convolutional U-Net with attention, is trained to predict the noise that was added at a given step, so that it can be subtracted off. Run the reverse process from pure noise and you generate a sample. Section 33.1 builds both processes from scratch and shows the one beautiful algebraic shortcut, the closed-form jump to any noise level, that makes training tractable.
With the machinery in place, the chapter turns to understanding. Section 33.2 formalizes the denoising diffusion probabilistic model (DDPM): the noise schedule, the three equivalent parameterizations of what the network predicts, and the variational bound that justifies the simple noise-prediction loss everyone actually uses. Section 33.3 reveals that the same model is, in the limit of infinitely many steps, a stochastic differential equation whose drift is the score of the data distribution, the gradient of log-density you first met in the energy-based models of Chapter 30. That continuous view unlocks the probability-flow ODE, a deterministic path between noise and data that the fast samplers of Section 33.4 exploit to cut a thousand sampling steps down to twenty.
The final three sections are about making diffusion practical and steerable. Section 33.5 presents the 2022 to 2024 reframing, flow matching, rectified flow, and consistency models, that straightens the generative path and pushes high-quality sampling toward a single step. Section 33.6 covers guidance, the technique that lets you trade diversity for fidelity and, in its classifier-free form, is the mechanism behind every "prompt strength" slider in every image tool you have used. Section 33.7 closes with latent diffusion: instead of denoising pixels, compress the image into a small latent with an autoencoder and denoise there, a change that dropped the compute cost by an order of magnitude and put Stable Diffusion on laptops. By the end you will understand not just how to call a pipeline but why each piece exists.
The thread running through the chapter is the one promised in Chapter 7: denoising, introduced as a humble image-cleanup operation, returns here as the entire generative engine. The U-Net is the convolution of Chapter 3 made learnable and stacked; the cross-attention that injects text is the attention of Chapter 22; the latent space is the one from Chapter 31. Diffusion is less a new idea than a new way of composing ideas you already hold.
Prerequisites
You should have read Chapter 30: Foundations of Generative Modeling, especially its treatment of energy-based models, score functions, and Langevin dynamics, because the score-based view in Section 33.3 builds directly on it. Chapter 31: Autoencoders & VAEs supplies the variational lower bound that Section 33.2 reuses, the denoising-autoencoder intuition, and the autoencoder that Section 33.7 repurposes for latent diffusion. From the deep-learning part you need the PyTorch training loop of Chapter 18, the convolution and U-Net structure that the denoiser is built from, and the self- and cross-attention of Chapter 22 that conditions the network. Comfort with Gaussian distributions, the reparameterization trick, and basic stochastic calculus notation (you will see $dx = f\,dt + g\,dW$, but we explain every symbol) makes the derivations concrete. The classical denoising of Chapter 7 is the conceptual seed of the whole chapter.
Chapter Roadmap
- 33.1 Destroying & Rebuilding: The Forward & Reverse Processes The forward noising process with no learnable parameters, the closed-form jump to any noise level, and the learned reverse process that walks static back to an image. Both built from scratch in PyTorch, with a tiny trainable denoiser on a toy dataset.
- 33.2 DDPM: Noise Schedules, Parameterizations & the Variational View The denoising diffusion probabilistic model in full: linear and cosine noise schedules, the three equivalent prediction targets (noise, clean image, and velocity), and the variational bound that collapses to the simple noise-prediction loss used in practice.
- 33.3 The Score-Based View: VE/VP SDEs & the Probability-Flow ODE Diffusion as a stochastic differential equation, the variance-exploding and variance-preserving formulations, why the reverse drift is the score of the data, and the deterministic probability-flow ODE that shares the same marginals as the noisy SDE.
- 33.4 Fast Sampling: DDIM, Solvers & Step Distillation How to cut sampling from a thousand steps to twenty: the deterministic DDIM sampler, high-order ODE solvers like DPM-Solver, and progressive distillation that trains a student to take the steps a teacher needs many for.
- 33.5 Flow Matching, Rectified Flow & Consistency Models The modern reframing that straightens the generative path: conditional flow matching as a simpler training objective, rectified flow's straight-line transport, and consistency models that learn to map any point on a trajectory to its endpoint in one step.
- 33.6 Guidance: Classifier & Classifier-Free How to steer generation toward a class or a prompt: classifier guidance using the gradient of a noise-robust classifier, and the now-dominant classifier-free guidance that trains one network on both conditional and unconditional objectives and extrapolates between them at sampling time.
- 33.7 Latent Diffusion: Compress First, Then Diffuse The trick that put diffusion on consumer hardware: train an autoencoder to compress images into a small perceptual latent, run the entire diffusion process there, and decode once at the end. The architecture of Stable Diffusion, plus the modern diffusion transformer (DiT) backbone.
The engine of every diffusion system is a single training objective. The network $\varepsilon_\theta(x_t, t)$ is trained to minimize
$$\mathbb{E}\bigl[\|\varepsilon - \varepsilon_\theta(x_t, t)\|^2\bigr]$$
where $x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\varepsilon$ is the clean image $x_0$ noised to timestep $t$ using Gaussian noise $\varepsilon \sim \mathcal{N}(0,I)$ and cumulative noise schedule $\bar{\alpha}_t$. Section 33.2 derives this objective from the variational bound and shows that the full ELBO simplifies to exactly this mean-squared-error term. This noise-prediction target is not arbitrary: because $\varepsilon_\theta$ approximates $-\sqrt{1-\bar{\alpha}_t}\,\nabla_{x_t}\log p(x_t)$, the trained denoiser is simultaneously estimating the score (gradient of log-density) of the noisy data distribution, the connection Section 33.3 makes precise through the SDE view, which shows that the same model defines both a stochastic reverse process and a deterministic probability-flow ODE. The three views (variational (33.2), score-based (33.3), and flow (33.5) are lenses on the same denoiser; train it once with the objective above, interpret it three ways. Above the objective sits a step-count dial: the thousand steps of Section 33.1 are a deployment choice that fast samplers (Section 33.4) and straight paths (Section 33.5) cut to a handful, with classifier-free guidance (Section 33.6) trading diversity for fidelity and the latent trick (Section 33.7) moving the whole process into a cheap compressed space.
What's Next?
This chapter gives you the generative engine; the next chapter gives it a voice. Chapter 34: Text-to-Image Systems takes the conditional diffusion model of Section 33.6 and the latent backbone of Section 33.7 and asks how a sentence becomes a picture: how a text encoder like CLIP or T5 turns a prompt into the conditioning vectors that cross-attention consumes, how the major systems (Stable Diffusion, DALL-E, Imagen, and the SD3 and FLUX generation that adopted flow matching from Section 33.5) differ, and how to prompt and evaluate them. From there, Chapter 35 shows how to edit and control diffusion outputs with masks, edges, and inversion, and Chapter 36 extends the same denoising idea into time and three dimensions. Everything generative that follows is built on the seven sections you are about to read.
Bibliography & Further Reading
Foundational Papers
Recent Research (2022-2026)
Books
Tools & Libraries
diffusers. github.com/huggingface/diffusersTutorials & Explainers
diffusers, mirroring the build-then-shortcut structure of Sections 33.1 and 33.7.