Does it help if you have an APU? A processor with integrated graphics. Since scanout needs to happen at a regular rate every 16ms, does that allow reads along with refresh? In effect hiding or negating some of this?
Modern graphics systems are too flexible for this - you can reprogram the graphics memory layout in such a way that would break the refreshing. But this technique was used on early PCs (with CGA graphics) and many of the 80s microcomputers.