The obvious downside is that the page contents are not private.
Chrome implemented something sort of like this with https://developer.chrome.com/blog/signed-exchanges. However this is very limited. It requires the linking site to cooperate. For example Google Search can link to a signed exchange rather than the original site. But this just moves traffic from the site's CDN to Google's. It also packages full bundles so shared resources need to be duplicated. Also any navigation inside that site will go to the origin and can't be cached.
Overall it seems like it probably isn't worth it. But I find it an interesting idea.