Chrome's design ensures that Spectre can only access resources that end up in an attacker controlled process. And this [1] post on "Post-Spectre Web Development" goes into detail about how a given website can ensure that its resources don't end up in an attacker controlled process. There are also a number of default protections against this like SameSite cookies and CORB that protect some resources by default.