See this[1] paper for more information. I think from the browser POV it is more about admitting that it just isn't possible to reliably mitigate Spectre and instead focusing on what can be done at the browser level. And at the browser level, it is possible to ensure that sensitive resources don't end up in processes running attacker JS.

Of course this could be fixed at the CPU level, but realistically very few people want that since that would drastically slow down modern CPUs which rely on speculative execution.

[1]: https://arxiv.org/pdf/1902.05178.pdf

Disclosure: I work at Google and am involved in deploying some of these cross-origin resource restrictions internally.