There really just needs to be a terminal API for true pixel graphics. Something like sixel but modernized and cleaned up. There's no need to hack in graphics on top of Unicode block characters, which negates backwards compatibility anyway.
This is nearly what I'm talking about but it requires the terminal emulator to understand the image format. I guess that makes sense for transfer efficiency, and if you want a pixel-level API you could just transfer BMPs.