Ah the good ol "whitelist the google bot IPs but pop-up paywalls for everyone else" trick. It'd be cool if google randomized their crawler's IPs and made their bot look like just another user, then we'd all get the same viewing experience.
This used to be a plague in the serps for technical error searches and Pinterest in the image searches.
The problem with the random ip proposal is that it is a cat and mouse game. If I own 1000 sites on various topics and cross reference visits via IP, well now it's not very random anymore. At one point there was even a services (fantomas) that kept an updated database.
This goes very deep in black hat SEO history with mosaic cloaking and other tactics.
I used this as a teen in the earlier days of Google to make decent pocket change, and that was in the early 2000s.
It goes even deeper when you get into parasite hosting tactics with paid backlinks and xrumer.
What's funny is all of these old school black hat seos have moved on to social media. PVA social accounts combined with AI rewriting are where the magic is now and don't even get me started on repurposed content via YouTube. It's no longer economical to tackle organic serps.
Heck I even have a bot that scans popular linked domains and monitors the expiration date on them for quick snatch with a nice grafana ui on it. And I've got quite a few nice domains with easy passive traffic and zero effort. All you do is restore the latest archive.org snapshots and setup a catch all email for leads.
Dall e and gpt 3 are going to turn these into legit businesses.
My point is if there is money to be made people will automate to the extreme. If you dig deep you'll realize an IP address doesn't mean much, hence recaptcha.
That's crazy. How does restoring an archive.org snapshot work, you just dump static html on the domain?
Are those mostly just content sites?
https://github.com/hartator/wayback-machine-downloader
You just dump and sync to s3 and use terraform to provision the route53 and bucket setups.
Yes they are mostly content sites. The hardest part is filtering adult domains assuming you don't want them. There are a staggering number of adult domains that expire every year and get huge traffic.
Can you grab assets like images or even video from archive.org?
Yup ! I find it MOST frustrating on "Google Discover" news-feed-stories on my Android phone.
Yet another actually-useful-for-users feature that got removed because profits.
I think the publishers should go ahead and create their own paywall content search engine but I would like the option to have my search results actually be available when I click them.