mandatory link: https://github.com/TecharoHQ/anubis

It's an interesting project, I wish there would be better ways to do that, but I guess we are on war with crawlers for a while already.

The answer to your question is in the article.

Because there's a better way for bots to get the data via the wikipedia database dump. Sending some large zip archives is a lot cheaper than individually serving every page on Wikipedia.

Not at all costs though, including disrupting access to said free information.

And this free information is not free from rights to respect neither, it's under CC-BY-SA, which requires attribution and sharing under the same conditions, the kind of "subtleties" and "details" with which AI companies have been wiping their big arses.