A sensible protocol to draw raster graphics would be nice, though. (And it might exist, I don't even know). I don't care whether I need to type "cat" or the name of a terminal-based image viewer.
We have an idea in that direction, where each program in a pipeline can optionally specify a file type, and the last file type hint in the pipeline determines how the stdout is rendered. So if you have "cat image.jpg", then "cat" can indicate a file type of image/jpg, causing stdout to be interpreted as an image. But if you have "cat image.jpg | grep foo" (just making something up here), then grep can indicate a file type of application/octet-stream because it cannot guarantee what the file type is.
The tricky part, from what I can see, is explaining the pipeline topology to the terminal, so that it knows that "grep" comes after "cat", even though command parsing happens in the shell only. That's still a big unknown. A previous project in this area (TermKit) mandated that applications specify a "Content-Type" header (like in HTTP) in their stdout, but that would break most existing programs, so it's not an option for us.
It's not a problem to have this handled in the shell such that the output is rendered in such a way as to please the terminal. You can do this manually (e.g. append a "| display-jpeg-in-terminal" command to the pipeline), or use a magic bytes based file viewer.
Look at "run-mailcap" or its shortcuts like "see". It does basically that, and can also be fed through standard input (which covers the piping situation).
Making plain `cat` work too should not be too big of a problem with some small support from the running shell. iTerm2 also knows when a command is starting to run/exits, parsing the output it retrieved from STDOUT in between by matching it against some magic numbers and displaying the content accordingly is not too difficult.
In fact it's not currently technically possible for a terminal emulator to know which program is writing into it. It maybe many programs simulatenously, or it may be the kernel asynchronously, etc... If you want to change that you would have to make the architecture tremendously more complicated and less flexible.
And the same goes for shells. You don't want to complicate the architecture just so that a lawyer could attest you that "it was cat" who drew the image und some weird interpretation. Nothing would be gained, and nobody would be able to understand what's happening anymore.