Re "ultimately the market" most people would point to 20th C experiments with command economies here. The hope was as you describe, a benevolent leader trying different approaches until one was clearly superior, and avoiding all the cost & waste of competitors doing the same work twice, not to mention the waste of advertising.
And it didn't work very well, even on purely economic grounds. Cars are a prototypical example to think about, and they were spectacularly awful for decades. It turns out the benevolent leaders are good at figuring out what's good for their careers, and less interested in making useful products (meaning safe, reliable, fast, efficient, etc.) Even in the west there was too little competition, but it turned out that duplicating almost everything GM did independently in Japan was a great investment to be making.
Another way to say this: If 2 competitors each develop the thing, then in the worst case they spend twice as much money as in utopia. But we can't get to utopia. And in situations where there is only one option (be it enforced by the state, or just a monopoly) it's easy to go wrong by a much larger factor than 2. If you don't like cars, think about drug companies, or academic publishes, who seem happy to increase prices by 10 or 100 times if they can get away with it.