The video config can be changed by the CPU in the middle of a scanline so you can't decode an entire scanline in a tight loop but need to run video decoding interleaved with the CPU (currently it's ticked after an entire CPU instruction though, not per cycle), and every tick of the video system needs to figure out all the decoding parameters again, because they might have changed since the last tick (and demo-scene demos use this mid-scanline-stuff a lot).
Also the display isn't driven directly by the 6845, HSYNC and VSYNC go through the Gate Array where a few ticks of latency is added, the same is true for the interrupt request that happens every 52 scanlines.
On top of that the CPC has an extra funky pixel bit pattern (http://cpctech.cpc-live.com/docs/graphics.html)
I haven't spent any time at all yet optimizing the video system though, for instance the pixel decoding might benefit from lookup tables. I first want to get it accurate enough for demo-scene gfx demos, and than will take care of performance.