Text-oriented... what did you expect? Binary formats are tough to standardize. We have raster images, video, audio files, and fonts, but very few of those formats are "web" formats. Most of them are designed around getting the most bang for your bit to the extreme--an MPEG-4 decoder is a real beast, to be sure, and it's hardly a good model for vector graphics. Vector graphics follow more of a human authorship pipeline (capturing a series of drawing commands) rather than machine capture pipeline (sampling data at a certain frequency). JSON, XML, JavaScript, HTML, even HTTP are built on text-oriented protocols and formats because text-oriented systems are easier to standardize and extend. MPEG-4, MP3, JPEG, Vorbis, are binary protocols because they only exist to represent and compress narrowly-defined multidimensional data.
Resource-friendly for small systems--how would we reduce memory consumption? The alternative model for vector graphics is what, PDF or PostScript? They're not exactly models of restraint. SVG looks positively austere compared to PDF or PostScript.
The closest thing I can think of to your ideal "minimal binary" vector graphics is something like TTF, which is basically its own little nightmare compared with SVG. TTF has its own vulnerabilities, its own proprietary scripting system compiled to bytecode, et cetera.
For me, SVG is doing a great job of enabling really cool things like D3, and at the same time hammering more nails into the coffin that is Flash--something far more a resource hog than SVG. And we don't really need more 90s-era formats, we've got plenty of those, thank you very much. SVG is to images what HTML is to documents. You can build simple, static ones easily enough, and you can also make a little bit of magic happen if you throw in some JavaScript.
It looks like in the future we'll be embracing more binary systems. TLS, HTTP/2, and WebAssembly are some examples. But it took a long time to get the institutional expertise to invent such systems, and if we had rushed to use binary instead of text in the early days, we would have probably gotten it wrong in a bad way.
Besides... the hardware is getting bigger and faster anyway... we might as well deliver some good experiences with it. Most of your battery is probably going to power the backlight and GPS radio anyway.