Shouldn't the solution be to gate bulk access for automated services for a price? Not just the wayback machine, any personal or corporate website?
My blogs are getting slammed and there are issues with cloudflare or captchas.
My blogs are getting slammed and there are issues with cloudflare or captchas.
Fine in theory but determined scrapers will use residential proxies in bulk.
The problem is that many of the people who are scraping this data doesn't want to pay. These are organisations who would rather not clone your git repo, and instead scrape every single page on your Forgejo installation. These are NOT nice people.
But maybe the problem is that they can't serve the data from the other websites like this, if they use it commercially. Right now they have non-commercial use, from what I understand.