Have you considered working off a commercial ARM microcontroller, say ARM M3, from STM, NXP, or Infineon? It's orders faster than a Z80, the documentation is world class, you can control every bit code that it runes, and there's a whole industry of people who spend their lives writing code for them.

I've written from bare metal: keys, display, networking, with hard realtime stuff (response times in millionths of second). It's not actually the bad.