Then paint to a regular buffer and do a memcpy to the framebuffer that has no cache at the end of each frame, possibly only copying a region/tiles you want to update.

All the libraries that exist are designed to work like this.