Well, optimizations need good abstractions to work with. The reason why a SQL query can run much faster than a naive for loop would is precisely due to SQL being very removed from lower-level details.

So in extension, optimization is terrible because we might be using the wrong abstraction level? E.g. it is hard to optimize a larger JS program, because if the developer specified some sort of detail it might throw out an entire class of possible optimizations.

There is actually an interesting presentation on the topic here: https://news.ycombinator.com/item?id=9396950 , which discusses among others whether a two-way communication between compiler and developer may help.

For sure. But not just "development by committee" also "development by hundreds of unrelated committees". By which of course I mean the full stack of dependencies that everything relies upon now, along with all the glue code in between that has to repeatedly transform data as it gets passed around through the system.

I've often seen some dependency module be the bottleneck, and an in-house replacement that does just what we need can perform an order of magnitude or two better than a generic 'all things for all people' module.

Also there are things that work well for dozens or hundreds of data pieces but become abysmal when you eventually scale to hundreds of thousands. But no one else has that use case, so it works fine for them. And for those that do, by then the frog is boiled and they're used to it being slow.