Author here! Just wanted to say that this is indeed in a good place to share, some very useful stuff, but is also very work in progress. I'm may 60% or so to my first draft. Said progress is coming every day and I happily welcome fixes or suggestions on GitHub.
As the other commenter said, R1 required very standard RLHF techniques too.
But a fun way to think about it is that reasoning models are going to be bigger and uplift the RLHF boat.
But we need a few years to establish basics before I can write a cumulative RL for LLMs book ;)
Author here, you can find more on my website: https://natolambert.com/cv
Have been building RLHF systems at HuggingFace since ChatGPT, with some other experience before.