Does this suggest an opportunity (maybe a business opportunity) for a service that, by crawling and scraping paywalled sites, presents to the human user what bots get to see? What if this service could somehow correlate all the socials you mentioned in a single, less horrible user interface?
Xcancel before it got cancelled and I doubt anyone would have paid for it anyway
archive.is, when google served cached pages you could get the actual content from the cache, not sure if other engines still serve the page they crawled as a cache.
This is one of the services the AI companies provide: presenting you a distilled, ad-free, version of the information they've scraped. At no payment to those providing it.
It's often not just based on User-Agent, but also the Google crawlers IP range. This makes it challenging to even get the bot view.
they would all try to sue you and you'd be fighting constant rate limits