The reason prefetching is particularly difficult to do for regular pages is the privacy implications. The act of preloading the page is a network event that is observable by the server serving that page.
If the server is the same as the linking page, for example Google search result -> Google Cache, there is no new information transmitted. That server already knows the user did query X and it knows that the page was going to fetch cache page Y, since the query page X instructs the browser to do so.
If the server is distinct from the linking page, for example Google search result -> https://healthsite.example/, then the browser will make a request to the healthsite server without the user having clicked the result. The healthsite server will learn the IP of the user, and some information about the query from which page is loaded, all without the user ever "visiting" that site. This is a major privacy violation.
AMP Pages solve this by a) being loaded from Google's cache and b) Guaranteeing no off-cache subresources will be loaded before the page is navigated to. (a) requires Google's cache to serve the page. (b) requires that the document author gives up some control over scheduling resource loading in the prefetch.
Until recently, AMP was the only game in town that could achieve this. Chrome recently shipped with Signed Exchanges, which is a network-level technology that could allow prefetching arbitrary content from a cache. This would still involve a Google cache, but does not require coordination with the document loading. Google AMP now supports this (https://webmasters.googleblog.com/2019/04/instant-loading-am...), but it would also work with non-AMP pages.