That exists, it's called GUI...
Improving on the text model is almost always premature optimization, though. Consider the incredible cost for an observer to "understand" what you are doing. Insane amounts of work (resulting in lock-in or a product never shipped) is not how the text model could become so successful.
Why can tools like grep or sed be so immensely useful? Precisely because they don't need to understand what goes in (other than a stream of lines), what does the pattern mean (other than being a regular expression), and what goes out (other than a stream of lines).
And what's the benefit of programs understanding "semantics of output" really? Everything is text. When a file is printed on the terminal, select it (e.g. double-click), type "vim " (or whatever), and middle-click. It's good enough most of the time. When there is some really dry repetitive work, bang out a command-line to automate that for you (along the lines of "while read filename; do vim filename; done").
A GUI can only help for the most common cases. If these are only a minority of cases, GUIs are only crippled tools preventing you from doing real work. (Personal anecdote: I use nothing but a browser and xterms on most days or even weeks).