The biggest pain in doing "raw" X is the async nature of the protocol - to write a robust X client you really want an event-loop driven approach that embraces that like XCB does, instead of trying to paper over it (like Xlib did).
was a bit confused about the segfault stuff mentioned towards the beginning of the article. but got quite quickly disabused of that notion with gdb etc.
Most "from scratch" would be some sort of microcontroller - write directly to hardware, program registers directly. It's the best feeling when you know every single instructions on the CPU is under your control, there is no 20 million lines of kernel code that do something.
But the MCU needs special hardware (MCU itself, display, programmer) so it is not a good starting project. This brings us to good old MS-DOS. Sure, you need to call BIOS to set up graphics (mode 13h, "320x200x256 colors" is the best for beginners), but after that, you are talking directly to hardware.