> I don’t want to stop the bots from updating their knowledge based on all the content.
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”