I think it's also really easy to underestimate all the tooling built around text. As soon as you try out some other way, you loose out on version control, diffing, grepping, and a whole bunch of other general tools built for text.
I think the only way to get around this is to build generalized tools for working with binary data as a data structure. The problem with this approach is that now you need to maintain a database that describes every possible binary format, where text editors only really have to concern themselves with ASCII & unicode to be useful for most cases.