Hacker News Reader

Comment by SXX | Hacker News Reader

For instance there is no way for distributed search engines to work with CloudFlare. No, "contact me and we'll help" is not always a solution.

View on HN

Please explain the problem (here or via email to me).

I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1]

[1] https://en.m.wikipedia.org/wiki/Sealioning

I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.

https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results.

Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.

Thanks. I'll bring this to the attention of the bots and DDoS teams.

Yacy is 18 years old and not exactly obscure. If your bots team is unaware of it it's because they've chosen to be ignorant of it.

A search engine which is not run centrally by one organization on infrastructure in a known network, but rather something like YaCy where individual users run crawler nodes on networks that vary over time.

Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.

Yeah, I certainly don't want those crawlers anywhere near my servers. Block by default and allow site admins to unblock should they want to seems like the best way. It is also already how it works with Cloudflare.

It is great that you care and I guess others are already provided some examples, but I'll add my own 2c here. Obvious problem is that centralized service like CloudFlare do create entry barrier and make large players on search and data mining markets even more entrenched than ever.

Recently your company announced partnership with Internet Archive, but if CloudFlare want to continue play a role as behevolent party everyone should have equal access to this data. Yeah it means that some bad actors will be able to easily scrap the web too, but...

CloudFlare service can't prevent scrapping anyway. There are shady residential proxy networks, services to bypass captcha and scrapping software like Zennoposter. It's possible to make scrapping more expensive, but bad actors don't care because they have money. Unfortunately enthusiasts, open source projects and small companies don't have enough resources to do the same.

Making scraping harder definitely reduces scraping. Some bad actors will get through, but others will be deterred.

I think you might not understand that it's site owners like me who want to stop scraping. It usually comes from specific bad incidents, like copycat sites stealing our content and work.

Cloudflare wouldn't block scraping if website owners didn't want it. And website owners can easily disable this protection.

Scrapping protection is not a problem: defaults that CloudFlare promote are. Saying that website owners can disable it is akin saying website owners should go and whitelist Tor nodes. Most of website owners don't understand either of issues and they never gonna opt-out.

Also I'm talking from experience because I been on both sides of the fence: doing scrapping and implementing protection. So yeah your CloudFlare protection will deter 10% of bad actors, but will also cut off 99% of enthusiast / research efforts or users of niche software or browsers. Still anyone with $1000+ budget will scrap whatever they want.

>No, "contact me and we'll help" is not always a solution. reply

In fact it's a textbook definition of a non-scaling solution.

That response is just a way to move the discussion out of the public domain without actually addressing it. It’s a scam.

Calling CTO of a big company who come to talk with us "scam" is very counter-productive. Some of CloudFlare bad sides are certainly by design and cannot be changed, but they can still change their default filtering policies in a way that will help open web greatly.

jgrahamc is one of the most active HN users. He's in the top 20 on the "leaderboard". He has a good reputation. I'd hesitate to call this offer a scam.

That said, I don't think it's a good situation that this is the solution rather than a proper, documented position that people can work to.

I've never been able to "reach a human" at Google, Facebook and other web giants and I'm skeptical that you can at a place like Cloudflare. In fact, I'd be really astonished it was possible, because otherwise their business isn't scalable.

I personally love CloudFlare and (just like with e.g DigitalOcean) I always found a way to contact human there. Unfortunately it's doesn't fix fundamental issue of how they make internet much more centralized and easy to MiTM or apply censorship.

Did you contact anyone at Cloudflare for an issue not explained in the support docs and you got no response?

Cloudflare support has been exceptional to me as a website owner.

while I share your sentiments regarding some other companies, I was able to get in touch with an actual cloudflare technician (and not some outsourced first level support with standard boilerplate replies) in a timely manner even on their free tier when I ran into a problem with one of their system. every support case with them has do far been a real pleasure compared to what you experience with other companies. I only hope they will be able to keep this level...

The grandparent, jgrahamc, you're responding to is the CTO of Cloudflare.

If this doesn't at-least meet your definition of "reach a human at Cloudflare", I'm not sure what will.

Reply on news.ycombinator.com