Hyperscaling Have I Been Pwned with Cloudflare Workers and Caching
troyhunt.com
troyhunt.com
The road to hell is paved with good intentions.
Incidentally I first read the title as "I have been pwned by cloudflare workers and caching".
How would you know you have a problem if you outright ban non-conformant users? Is your customer support function not behind cloudflare, and accessible to users without an account?
You are making a lot of assumptions with that statement
Our security level setting is low enough that almost nobody would actually get blocked from the site. Anybody could access the contact page and email us or use the live chat
We use Turnstile in a couple of places and we have gotten a couple isolated reports about users being unable to perform actions behind Turnstile but it was always that they had some sketchy extension installed. And the extra security and bot protection we are getting makes those very low false positive rates worth it (we have tens of thousands of users so a couple reports in the last couple years is fine...)
You have literally replied to a thread in which we discuss how Cloudflare bans non-conformant users (who live in 3rd world countries, use linux and possibly other non-conformant computer practises according to Cloudflare's product managers). So you outright ban them by using Cloudflare.
-----
You also literally contradict yourself with the following two statements:
> I live on the opposite side of the globe and have no problems using Cloudflare. Also, my SaaS is deployed on Cloudflare and we have users in hundreds of countries who use it with no problem
and
> We use Turnstile in a couple of places and we have gotten a couple isolated reports about users being unable to perform actions behind Turnstile but it was always that they had some sketchy extension installed. And the extra security and bot protection we are getting makes those very low false positive rates worth it (we have tens of thousands of users so a couple reports in the last couple years is fine...)
Make up your mind, which is it? Do you have no problems using Cloudflare and your users in hundreds of countries use it with no problem or not?
-----
These being said, what percentage of lurkers actually contact random online services to let them know that something is wrong? Almost nobody does that.
Personally, I've only contacted Troy Hunt on haveibeenpwned and his blogs, letting him know on several separate occasions that his websites are inaccessible to some users, as far as I could tell, from 3rd world countries. He has deleted all of my comments, he probably deletes all comments critical of his service, since there's only praise allowed in his blog posts. To be able to contact him, I had to borrow a Macbook and use a US vpn, because all of his services are behind enless Cloudflare captchas.
How many website visitors of yours, not users, would be able or willing to do go to that length to contact you about your dysfunctional Cloudflare WAF?
And I'm sorry if you think that 2 users in 2 years having an issue when we have tens of thousands of paying users tips the scale of whether or not it is an overall net benefit for our company. If it wasn't for Cloudflare we simply wouldn't be able to provide the free versions of the software in the same fashion that we do now
It sounds like you're upset at somebody who improperly configured Cloudflare on their sites and now you are blaming the company and everybody that uses it without having a solid understanding of the tech
See, this is what makes Cloudflare's practises work. You are under the impression that 2 users in 2 years have had issues when, actually, 2 users in 2 years have bothered to jump through lots of hoops to finally contact you about your issues.
Your SaaS business seems profitable, so keep it up! But don't go around claiming only 2 users have had issues, you most definitely don't have a 100% support contact rate for Cloudflare related issues.
The WAF does not block anybody from accessing the site which I have been trying to explain to you.
You are not listening because you are taking one experience with one site and then projecting that on to me.
Perhaps I misunderstood. Is your contact page not behind cloudflare?
It can be both true that our entire website frontend (such as the contact page...) is behind Cloudflare and that nobody will be blocked. If you don't understand that it's not on me
I've already tried to tell you several times now that we only use TURNSTILE for a couple specific actions in the app and that otherwise nobody is going to be blocked or shown a captcha...
I'm not sure you realize just how flexible Cloudflare's security settings are and that if you are blocked it is entirely because the website owner set it up that way.
I guarantee you that you access a ton of sites behind Cloudflare and you don't even realize it
Nope, it would appear that you fail to comprehend that users which are banned from ever reaching your website would never contact you. You can't know what you don't know, it's a natural limitation, you just can't yet wrap your head around this concept. I suggest re-reading the thread.
You keep failing to understand that there are literally an infinite number of ways to configure Cloudflare and you are making way too many assumptions
I get where they're coming from, but I really hate it when companies take this kind of view. Back when I grew my software company, I cared about every. single. one. of my customers.
Also, I want to add that we only put Turnstile on the login page once we had tens of thousands of active users and not a single person had any issue logging into their account. These are paid users, so I can guarantee you they would have been emailing us
It's fine in Europe.
On Mac OS, if I used desktop Linux I'd probably get more.
Also, my work involves hearing about internet problems in parts of the third world, and captchas aren't something I hear about often, if at all.
Definitely not fine.
And depending on your definition of third-world country, I'm in one as well, and I don't have this sort of issue
But yeah, it's none of that just Cloudflare hates brown people or something.
Request URL: https://api-v3.amibreached.com/api/v1/cyble-it?SearchTerm=as... Request Method: GET Status Code: 500 Internal Server Error
I don't know what the situation currently is with HIBP, but Cloudflare does allow setting the security level. Maybe at the time it was still set to high/normal instead of "low" or "essentially off".
I like to write about these cases in my spare time, e.g.https://joshua.hu/losing-sight-vision-mission-of-your-role-p... and https://joshua.hu/losing-sight-vision-mission-of-your-role-p.... My all time favorite was when I was in hospital and couldn't connect to my travel insurance company's website because they blocked IP addresses from the country I was in (wasn't cloudflare though, I don't think: https://joshua.hu/losing-sight-vision-mission-of-your-role)
That's sad - and far too common. We're so conditioned to web sites and even apps being unusably slow and plagued by latency thanks to Electron and multi-megabyte JS bundle slop that the exception, software that is actually responsive and slim, is being judged as abnormal instead of an ideal to follow...
The majority of users seem to prefer slow animations for anything that changes, and flight/hotel search pages have used artificial delays for decades.
Tech conditioned people to this expectation. Tech could have also gone and say "no, screw you, we will not introduce artificial slowness Just Because" and in 5-10 years people would have adapted. Swim or die.
It's just the same with IT in general. In the Nordic and Baltic countries, even beggars have credit-card terminals because no one carries cash any more. Most if not all public service is done exclusively online - and yet we do not hear the horror stories of elderly people dying because they can't apply for social security that people are drawing up here in Germany.
People have the capacity to change and adapt, and one does not have to coddle adults.
Oh, haven't seen that here (Riga). Actually I was out in the capital few days ago and people play music on streets... I can't tip them because I have no coins and they have no terminal or QR code that would lead to a page that enables tipping.
Most places have cc terminals and buying stuff from hands also support sending money to bank account instantly using only sellers phone number, but some parts are still coins only. My wallet doesn't support coins unfortunately.
I'm from Poland (so close) and I find this true here too, but in 99% of the cases there is a human fallback. You can file your tax return online, but nothing stops you from driving to tax office and filing a paper form there, with a pen.
> yet we do not hear the horror stories of elderly people dying
Well, that's because 100% of the time, elderly people will use public and private healthcare by a phone call with a human. Even if apps and such are available.
The point of those fake animations or fake spinners is showing that "it worked" in the absence of "success" feedback.
I work with offline-first apps and we did some user testing. We have to be careful about things like navigating between pages, because if it's too fast the user will not register the change, and will assume it was an error.
Now THIS is the fault of tech industry, and where I agree that it's conditioning: a lot of tech products simply fail silently, or have very long timeouts, so users are conditioned to translate "lack of response" with "failure".
There are alternatives to animations, however: different designs between pages, changes close to the mouse pointer, or in the case of list refresh showing the "last refreshed 1 second ago"... or even showing a popup with "Successfully loaded". Often this is hated by designers (although the "success popup" is also hated by users), which is why people look for alternatives.
It’s a UI problem in how to make it update immediately but also have an indication that it’s updated.
It's frustrating how secretive this is all treated and how to get anything useful you have to go on telegram instead of there being an open way of checking.
Additionally, people don't just want a boolean answer of "was my email breached somewhere". They want a list of all the breaches that breached the email. So the returned data actually needs to be a list of emails and the list of breaches that each email was breached in.
>Via the public API. This endpoint also takes an email address as input and then returns all breaches it appears in.
The initial prefix check would probably reduce the amount of lookups necessary, as it would only be necessary to do a deeper search if the prefix matches.
Also regarding "you can get rid of a significant portion of requests at the edge with a bloom filter", Troy's existing design already gets rid of a significant portion of requests at the edge. That's why he says
>The response from each search was coming back so quickly that the user wasn’t sure if it was legitimately checking subsequent addresses they entered or if there was a glitch.
There are 5 billion emails in at least 1 breach and 16 million prefixes. Almost all if not all prefixes have at least 1 email in a breach. So almost all prefixes match. I don't see why it's useful to spend a bunch of effort optimizing the very rare case of a prefix not matching.
Now, if the bloom filter checked emails instead of checking prefixes, that would be useful. However, a bloom filter of 5B elements with a 10% false positive rate would be 2.8 GB, which is prohibitively large.
According to this calculator, a bloom filter for 16M elements with a 10% false positive rate would be 9MB.
I take it the k-anonymity thing essentially "chunks" your database. So rather than returning one (identifying) result for one email address, you instead take the first few characters of its hash, and return a few hundred results corresponding to all the emails whose hashes share the same prefix. (A bit like asking for the "L-N" section of the phone book instead of asking for a single line).
I'd be curious what sort of attack vectors or "side channels" you wargammed out when coming up with the scheme - eg. if a block wound up mostly comprised of email domains (providers) popular in one part of the world, and only a few from another geography, then you might be able to infer identity via timezones based on time of day queries come in.
I do appreciate how we've found a way to make hash collisions useful :-).