If JSON just originally had comments there wouldn't be such a proliferation of redundant configuration language, since JSON serves perfectly well 99% of cases.
Comments, trailing commas, multiline strings, and a real date type.
Comments, trailing commas, multiline strings, a real date type, and dropping the quotes on keys.
Comments, trailing commas, ...
JSON really isn't just the "simple tweak" away from being the perfect config language it is often presented as. There's this fun sort of error that only a group of people can make, where someone can stand up and make a statement like "JSON just needs a couple of tweaks to be the perfect config format!" and everyone can individually nod in agreement, and the group thinks it is in agreement. But it turns out every individual actually had a completely different interpretation of the statement and they don't agree at all. When you get a group of developers together to be clear about the changes to JSON you will discover not everybody has the same tweaks in mind.
(& I wrote this before seeing the sibling comment from enriquto who gives another thing necessary for perfection. I don't even 100% know what enriquto means by "standard" floating point numbers versus what is already in the spec, but I bet it's another thing that multiple people could nod in agreement to but in fact mean something quite different about if you nail them down to the spec level!)
JSON could be much better, and good enough for far more purposes, with the top 3 tweaks there.
I think it's not entirely unlike programming languages. If I sit down and write out all the features I want from a programming language, I can't have it. They straight-up contradict each other. As a simple example, I want simplicity and a rich type system. Nope. Not gonna happen. Intrinsically at odds with each other. And there are many other such places, more subtle than that but no less real.
Config has the same thing going on. I've made my peace with just doing whatever's convenient in the moment and not stressing about it. It certainly isn't worth bringing in a new syntax my local team has never heard about because it's just the perfect little config syntax. It's very hard for any config syntax to be better enough than what came before to justify adding another tech to the stack.
JSON with comments wouldn't be perfect, but it would be unacceptable a lot less often.
I'm not sure who made the decision at the time, but looking back at the hours we have collectively wasted because of this non-feature, I believe they would have reconsidered.
IMO, that would make it "good enough" for most use cases.
Having the application parse a date from a string isn't too bad. And dropping quotes on keys is just a nicety. And really, trailing commas aren't strictly necessary either.
But comments are absolutely necessary for a human readable configuration format. And multi-line strings are critical for any use case where you have strings that could be long. Like, say, a "description" field.
As a sidenote, the absence of multi-line strings is a major frustration I have with writing JSON schemas in JSON.
TOML on the other hand is also a clear indication that Tom did not like using (or hadn't learned about) a formal grammar.
And I don't thing you can validate JS with a regex? (Have I forgotten that much of formal language theory?)
There's just a couple inherent problems with the notion of a universal configuration language...
First not all software has the same type of configuration requirements. I don't mean different schemas, I mean some measure of "configurablity". You have things like nginx or varnish that are configurable enough that configs are almost (or actually) programs. You also have programs where you need to store a few KVs that will be inserted into strings at some point and that's it.
That wide range of "amount of possible configuration" is hard to capture easily in a single configuration language.
Next you have a wide range "flatness" - that is if the configuration needs to be a list of kvs (optionally w/ sections) or if you need a tree or graph shape, or something weird.
Finally there's the whole preference and "style" thing. Writing a config language has an appeal to it - it's a fairly straight-forward thing to do (at least seemingly, until you hit the edge cases), and it's infinitely bikeshedable. In a similar vein, it seems to devolve into this sort of thing quite often: https://xkcd.com/927/
This is one of those situations where it's nice to be "old" - I like that I only have to know a handful of config formats these days, rather than keeping track of each program's bespoke config format.