While I think that ASCII including a specific record delimiter character (0x1E) that isn’t used is a missed opportunity, I actually think the lack of a set interchange format is a benefit.
No format will work for everyone and if there was one set format, we’d still have people complaining about how it doesn’t work for their usecase. We'd also have had to (finally) answer the question of how to send the schema with the data.
JSON (and XML and YAML and…) are all methods to try and adapt a text based data interchange format. And they all work roughly the same. You ultimately need some kind of processing library to full read them. But working with the data in any sort of structured context requires specialized tools (ex: jq).
Or you can still grep it, just like I do to manage my tab delimited data, and get part of the way.
I think the chaos of having no standard is kinda the point. Text is the lowest format that is easy for people to work with. We can visually parse it pretty easily, so the tools that support text can be pretty universal. Grep, sed, and awk can do a lot.
But to do anything more, you’re always going to need some kind of specialized parsing, even for tab delimited or CSV data. So, let the terminal work with generic text and be “good enough”.
As an example: you mentioned JSON and YAML and suggested a slimmed down YAML might be more powerful. I deal with columnar data with tens to hundreds of columns and thousands of lines. This wouldn’t work with either JSON or YAML… it’s too verbose. Instead I use (compressed) tab delimited text files for their ease of parsing in many languages. But those won’t work for many other use cases.
With data exchange, there is never going to be one right sized option for all needs. If you can accept that, then the “right” option is to choose none of them.