Hacker News Reader

Comment by c16 | Hacker News Reader

Having created a data crawler too, the discussion about gated websites behind CloudFlare is painfully true. Unless you're a big player, you just can't enter this market. It's a real shame.

Supposedly stating you're a custom crawler [1] (like how Google has GoogleBot) and not a generic bot helps with this. I'm somehow very doubtful of this.

[1] JGC responded to a tweet a long time ago, can't find it now

View on HN

CloudFlare is cancer, they've blocked off most of the internet and deliberately block non-Google crawlers. There's been discussions on it here before.

It depends how their user's configure the settings. You can use Cloudflare and effectively turn all blocking off very easily. There is a reason their customers tend to leave it on, or in my case, increase the default security level. Sorry Tor users, but 100% of the Tor traffic I've gotten has been malicious WP vulnerability scanning

Okay, I understand, but I am so sick of identifying stoplights, trucks, and airplanes. It would be nice to have someone say, "Oh, you'd like to read my interesting information anonymously? Go right ahead!"

For webmasters, the damage these rogue bots, tor, hack attempts can just bring down the whole business. I think more and more smaller websites will be opting for cloudflare in coming days.

Webmasters? What is this, 2010?

But on a serious note - you can't exactly rely on Cloudflare to not bring down a business. For me, it is a very nice convenience to throw up in front of my WP sites for caching and to help limit the bad traffic I get, but that's about it

Reply on news.ycombinator.com