Specifically, the requirement for completely custom GUI styling without writing his own render functions means it’s really a task of selecting easy, customizable GUI libraries rather than generic GUI work.
The requirements to be self-contained executables and under a 40MB limit also rule out a lot of options. The author admits that Qt could have met these requirements but the open-source licensing part wasn’t compatible with his goals and he didn’t want to pay for a license.
If you relax the requirements to tolerate some dependencies, allow larger download sizes, or to simply use built-in Windows GUI controls the situation is very different.
For writing a lightweight, completely custom GUI with no external dependencies and permissive licensing I could have guessed ImGui would be the answer before I started reading.