I don't think his book is meant to be read from cover to cover. It's useful in many ways, as an outline, as a reference, or as an entry point to your niche of interest.

Let's say you are interested in diffusion models. You read the corresponding chapter (and play with the companion code).

If you are missing some background, you can also backtrack and skim through the initial chapters.