Subscription bombing and how to mitigate it
bytemash.net
bytemash.net
He ran the attack from midnight to 7AM, so there were no humans watching.
IPs were rotated on every single request, so no rate limiter caught it.
We had Cloudflare Turnstile installed in both the sign up form and in all credit card forms. All requests were validated by Turnstile.
We were running with the 'invisble' setting, and switched back to the 'recommended' setting after the incident, so I don't know if this less strict setting was to blame.
Just like OP, our website - to avoid the extra hassle on users - did not require e-mail validation, specially because we send very few e-mails.
We never thought this could bite us this way.
Every CC he tried was charged $1 as confirmation that the CC was valid, and then immediately refunded, erroring out if the CC did not approve this $1 transaction, and that's what he used. 10% of the ~2k requests went through.
Simply adding confirmation e-mail won't cut it: the hacker used - even tough he did not need it - disposable e-mail addresses services.
This is a big deal. Payment processors can ban you for allowing this to happen.
This is one of those levels of monitoring that only gets put in place after such an event. Eg whole subsystem analysis - the change card feature being used 1000s of times (well, proportional to scale) in 7 hours is a massive red flag
For a website, yes. But honestly the credit card people and their infrastructure should probably _also_ watch out for this. They'd be in a much better place to detect these.
JS can be reversed, you clearly see what data points they use for detection. Anything can be spoofed and it will look like human behavior.
And if everything fails, you outsource it to AI - Always Indian :D
It's easy to say that every site must add protections against every single type of attack, except it's impossible for site owners to be experts in fraud. While credit card processing vendors are expected to be experts in fraud. I ask you where in this situation would be the better place to implement fraud detection? Of the two places, whose more financially at risk?
I think we’re 100% in agreement: let the payment processors handle the fraud. Except payment processors unfortunately hold all the cards and will shut your site down if you don’t comply with their standards :(
After 12 hours, the pace of emails came to a halt, and then I started receiving emails to made up addresses of a American political nature on the same domain (I have wildcard alias enabled), suggesting that someone was perhaps trying to vent some frustration. This only lasted for about half an hour before the attacker seems to have given up and stopped.
Strangely, I didn't receive any email during the attack which the attacker might have been trying to hide. Which has left me confused at to the purpose of this attack in the first place.
Double check that there are no forwarding rules added to your inbox and add some protection against a SIM swap.
In my case, they didn't compromise any of my accounts but did attempt to open a new credit card so it would be worth double checking your credit reports.
The Internet was carefully designed to withstand a nuclear war and this approach, being adopted en masse, is slowly turning it into a shadow of its former self. And despite the us-east1 and multiple Cloudflare outages of last year, we continue to stay blind to this or even rationalize it as a good thing, because that way if we're down, then so are our competitors...
If they turn around later and do something evil, literally all I need to do is change the nameserver to a competitor and the users of my website won't even notice.
As soon as you turn on any other service they offer, you need to actively migrate away. It's an inherent issue of services that actually provide a benefit. If you're saying "I can just migrate to any other nameserver" then you're telling me you have no use for Cloudflare in the first place. Because if you did, you couldn't just not use it anymore.
Let's say you're using their WAF. Sure, you can just change your domain's nameserver and you've migrated away. But now you no longer have a WAF. Same for their CDN. Or their load balancer. Or their object storage. Or their CAPTCHAs.
Both are extremely useful and good products.
I assumed this is what GP was talking about when referring to the turnstile.
Cloudflare is an excellent solution for many things. The internet was designed to withstand a nuclear war, but it also wasn’t designed for the level of hostility that goes on on the internet these days.
In this case I assume the reason the attacker used the change credit card form was because the only other way to add a credit card is when signing up, which charges your card the subscription fee (a much larger amount than $1).
So the solution is don’t show the change card option to customers who don’t already have an active (valid) card on file.
A more generic solution is site wide rate limiting for anything that allows someone to charge very small amounts to a credit card.
Or better yet don’t have any way to charge very small amounts to cards. Do a $150 hold instead of $1 when checking a new card
As far as cloudflare centralization goes though, you’re not going to solve this problem by appealing to individual developers to be smarter and do more work. It’s going to take regulation. It’s a resiliency and national security issue, we don’t want a single company to function as the internet gatekeeper. But I’ve said the same about Google for years.
You can't block 100% of these attempts, but you can block a large class of them by checking basic info for the attempted card changes like they all have different names and zip codes. Combine that with other (useful) mitigations. Maybe getting an alert that in the past few hours or days even, 90% of card change attempts have failed for a cluster of users.
Whether every user notices it or not, attackers are looking for the ability to do small charges, and if you remove that they'll move on.
Attackers are going after small charges. That's the reason they're going after these guys in the first place.
>Site-wide rate limiting for payment processing? Too complicated, high-maintenance, and easy to mess up.
And then you give a solution that is 10x as complicated, high maintenance, and easy to mess up.
>You can't block 100% of these attempts, but you can block a large class of them by checking basic info for the attempted card changes like they all have different names and zip codes.
This is essentially a much more complex superset of rate limiting.
And these people weren't validating the email address on signup. To "reduce friction" i guess.
I routinely blocked by Cloudflare from viewing things and occasionally, I am blocked from buying things. Just this weekend, it was $100 worth of athletic wear. I just keep clicking the box and it never lets me complete the purchase. After the 7th or 10th time I go and find another vendor that would actually sell to me. I was more annoyed than usual because the website already had my credit card at this point – but as this article proves there are reasons to block an order even with a credit card.
How would you solve this at scale?
Another approach is to not ask for an email address at all, like here on HN.
But from a UX standpoint its a nonstarter
Youre asking users to
- leave the site/app
- open their email client
- compose a message or at least hit send
- wait for a reply
- then come back and continue
Thats a lot of steps compared to enter email -> click link. Each additional step is a dropoff point especially on mobile or for less technical users. Many people dont even have a traditional mail client set up anymore, they rely on webmail or app switching which adds even more friction
It also introduces ambiguity
- What exactly am I supposed to send
- did it work
- What if I dont get a reply
From the service side youre trading a simple well understood flow for a much more complex inbound email processing system with all the usual headaches (spoofing parsing delivery delays spam filtering)
In practice most systems optimize for minimizing user effort even if that means accepting some level of abuse and mitigating it elsewhere. A solution that significantly increases friction... no matter how principled...just wont get adopted widely
So while the idea is interesting from a protocol design perspective its hard to see it surviving contact with real users
Disagree. The UX would be pretty similar. Click a mailto link which opens the email client with to, subject and body precomposed. Click send. Server receives mail and the web page continues/finishes the sign up process. No need for an email reply. It’s different, but it’s not crazy.
Ignoring the fact that mailto won't work for most people (it opens my Mail app which i never used), "different" is enough to make your conversion rate tank. It'd be unreasonable for anyone in charge of making product decision to go with that
Mailto links are not that common these days.
- user click on the link
- their email client opens, with the To:, Subject:, Body: fields pre-filled
- user clicks "Send"
- a few seconds later a sign-in link arrives in their inbox
Anybody can send email with all of the dmarc stuff, how do you "be careful" with spoofed email?
You actually verify DKIM and SPF—you know, that “dmarc stuff”. That’s enough to tell you the mail is not spoofed.
Usually the very best you can do IRL is "probably fine" or "maybe not fine" and that's just not good enough to justify blocking customers. Email is an old tech and there's a lot of variation in the wild.
Every extra field in the sign-up form already lowers the conversion rate.
But also Anubis is a good alternative to slow bots.
Also, I doubt this is going to be pissing users off since they added Turnstile in invisible mode, and selectively to certain pages in the auth flow. Already signed in users will not be affected, even if the service is down. This is way different from sites like Reddit who use their site-wide bot protection, which creates those interstitial captcha pages.
It's not inherent, though! Easy, definite fix: Reverse the communication relation. If the user has to open their mail app anyway, you could simply require them to send an email to you, instead of vice versa. This would solve the problem completely. (If spoofing the sender could be done reliably, the service wouldn't be involved in the first place.)
Now, it would slightly increase friction and lower convenience. That's why it's not done. It's inherently incompatible with dark patterns, data collection and questionable new user acquisition, but this too could be solved through standards and integration - without making Cloudflare de facto infrastructure necessity!
Possible convenient, better solutions: Have the browser send this mail, either by passing a template to the mail app, integrating SMTP into the browser/addon, or instate a novel authentication protocol, which in fact may remove the human interaction completely.
As if 2FA security was the main motivation for asking for email, and/or phone anyway. Companies want user IDs, if possible UIDs, as soon as possible to increase user data value and gain marketing opportunities. I once had a "welcome mail" after typing in the address, before sending the form. Yeah...
But as I said, it's not about what's technically, or ethically mandated, but what's ensuring users won't get annoyed (getting bombed with mails is bad PR). Companies collect all these IDs for their (future) shareholders first and foremost. Asking for email doesn't alert people. Phone number would be more alarming, but that's still becoming the norm. They would ask for a picture of your passport too, but ... oh, wait!
Casually integrating Cloudflare into everything (incl. TLS termination lol), only makes data collection incentives greater. Let's not give in by declaring Cloudflare a fundamental necessity. Or do, but don't complaint about your disowned life as cattle.
How do you suggest to implement bot prevention that works reliably? Because at this point in time, LLMs are better at solving CAPTCHAs than humans are.
Until a few years ago, IP reputation was a good defence against this. The bad traffic almost entirely came from IP addresses in certain countries or from datacenter IPs we could block. Nowadays, that doesn't work due to the prevalence of VPNs, so many legitimate users appear to come from low reputation IPs.
Turnstile is a reasonable solution in its normal form, though the 'invisible' option still lets a lot of them through. Another thing that works, surprisingly, is looking for "webdriver" usage. Despite being easy to strip out webdriver fingerprints, we find that the majority of automated attempts do not bother to do this. Adding more steps, honeypots (with an immediate short term IP ban), etc. also have an impact. It becomes a game of piling up numerous defences in a sort of "Swiss cheese model".
The four horsemen of the infocalypse are always about power grabs, they're never about actually protecting citizens, or children, or securing a country or region.
It’s bonkers to me that there’s any developers out there working for these companies that never thought to implement simple email verification.
Also, people usually type their emails correctly, especially these days with auto-fill. So not sending confirmation emails is optimizing for the happy path.
I was once even sent all of the legal proceedings for a court case by a lawyer who was sent to the wrong address.
Absolutely 100 percent fraud and security proof.
However as you it is an extra step and I imagine many retailers have done the math between fraud and friction...
I think it would be quite annoying to have to verify my purchase everywhere, just like how I don't wanna sign up to every single merchant online. Let me purchase as guest without having to enter OTPs.
I have even had founder level emails that presumably are confidential sent to me because I share the name of someone operating in tech.
I respond or report when it's obviously some real person running a small group but for large monoliths there is very little to do except quickly reply to corporate email.
Really wish there was some kind of high level discussion about building something for this specific problem of non malicious wrong person same name errors.
Google could do it it's just not something that is monetizable at a scale they care about IMO and I have not been able to think of a way to make this work operating outside of email monoliths.
Would love to hear if anyone has ideas.
I have around 20 or 30 google accounts attached where i am the backup email address. Those people forget their passwords or stop using their accounts and i get email notifications about that. No confirmation from my side necessary.
I set up a new address that is less likely to end up with this problem. But migrating away from the old one is not easy…
I get so many other $MY_NAME emails, including bills (including multiple credit cards and things like Afterpay), deliveries, medical details/reports, family communications, etc, etc.
And it's very clear that quite a few online services blatantly don't verify email addresses, they just assume the email is valid and allow the person to start using it.
Does google not require a verification when you setup a backup email address?!
https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...
They are optimizing towards making it easy to purchase things on a whim.
They didn’t have a HIPAA coordinator by title, so I got to explain how to avoid this to their legal department.
Other POV: I simply do not use email — like at all (don't even have a SPAM account anymore) — if your website requires me to enter an email address just to buy something, I WILL find a burner/temporary email that will allow the transaction script to proceed ("10 minute email").
I am personally grateful when a store allows me to proceed with a guest transaction [i.e. non-login, non-email purchase] — even if that means I might need to "call in" should the transaction be flagged/delayed [usually isn't].
Intentionally not giving examples to avoid hacker/targetting, but many US clothing manufacturers offer these frictionless (and legitimate) purchase pathways.
Here are some interesting additional information from the attacks we analyzed:
* Email bombing as a service is a thing, where you can buy 10,000 credits for $10 and easily bomb target inboxes with over 2000 emails per hour.
* Most all email bombing attacks starts in the morning, between 8-10.
* Most common day of attack is Friday
[0] https://www.xorlab.com/en/
[1] https://www.xorlab.com/en/blog/from-chaos-to-control-insight...
>iCloud
Except by apple I guess...
CAPTCHAs were never meant to work 100% of the time in all situations, or be the only security solution. They're meant to block lazy spammers and low-level attacks, but anyone with enough interest and resources can work around any CAPTCHA. This is certainly becoming cheaper and more accessible with the proliferation of "AI", but it doesn't mean that CAPTCHAs are inherently useless. They're part of a perpetual cat and mouse game.
Like LLMs, they rely on probabilities that certain signals may indicate suspicious behavior. Sophisticated ones like Turnstile analyze a lot of data, likely using LLMs to detect pseudorandom keyboard input as well, so they would be far more effective than your bespoke solution. They're not perfect, and can have false positives, but this is unfortunately the price everyone has to pay for services to be available to legitimate users on the modern internet.
I do share a concern that these services are given a lot of sensitive data which could potentially be abused for tracking users, advertising, etc., but there are OSS alternatives you can self-host that mitigate this.
I suspect any company would take this trade off, losing some customers but significantly lowering fraud.
So, by your own admission, your solution doesn’t get around the “sufficiently advanced bot” problem.
I hope "LLM thinks your name is gibberish" won't become the new "your name can't include invalid characters".
Account notification emails don’t have unsubscribes while pretty much all junk does.
My conclusion is to move from WordPress software as fast as possible, every WordPress site I manage gets bombarded by bots.
> Verify your email address > Welcome to Suga > Reset your password > Three emails they never asked for, from a product they may never have heard of. We were just one of potentially hundreds of sites being hit at the same time.
@homelessdino
Why would you send welcome and reset to some victim that DID NOT verify?
Editing to add: almost 100% of these emails came from the same e-commerce product, I'll have to look up which. But every site i got an email from was running the same off the shelf template.
I stopped offering the trial since I figured it was not useful for a relatively cheap (19 EUR) app that has a 30-day money-back guarantee anyway. (It can also be used for 15 mins without a license, but requires a restart after that.)
> While the victim is drowning [...] the attacker is doing something else.
In the past months some personal mail accounts on a mail server I administer were victim of something that looked similar to what's described here.
Hundreds of mails apparently originating from various (legit-looking) random public web services, support requests, issue trackers, web contact forms etc. For example, a good part of them was from Virginia Department of Motor Vehicles (as in something like "thank you for filing a document #123 with us").
To make things even weirder, they were not sent directly to the address, but according to message headers were bounced through Google Groups (each time I checked the relevant group was already deleted). So as far as I can tell it was not the mail address hosted on my server that was being entered into those websites.
No phishing links, no attached malware, no short advertisements snuck into a text field etc. Just a huge amount of automated replies from "noreply@" legit entities.
I've seen several of these attacks and spent some time investigating them. To my knowledge these were not associated with any other malicious activity, like the author of the article mentions. If anything they were just a denial-of-service attack on a mail box (as in, making the human user trawl through garbage, the mail volume was far from saturating the server itself). What exactly would be a motivation for that I can't tell, except making the life of a small mail server admin even harder than it already is.
I got subscription bombed from about a dozen .gov sites a while back. Healthcare.gov, Social Security Administration, WTC Health Program, FDA, National Institute of Mental Health, Medicare.gov.
It's very easy for attackers to do this. Putting in effort to solve this for one site, and expecting any impact, is like trying to empty the ocean with a teaspoon.
Now, every mofo just wants a grant to ---- innocent kids in school.
Maybe I should just remove it from the sign-up form altogether and use it as a honeypot.
I haven't seen any proof that the big ones (Google, CF) can be easily and automatically bypassed, and would love to learn more if someone has evidence to the contrary.
Did you have to roll your own or was there some proof of work library you were able to use?
Update: Ah, found the code - https://withinboredom.info/posts/how-this-blog-actually-work...
What’s the alternative, though?
What if each service generates a link to a uuid url where all new message will be displayed? The user can rss subscribe to that.
So the user doesn’t receives anything.
If the email is targeted at a domain using mail servers from any major email provider (Gmail, Outlook), the user will probably find most emails into their spam folder anyway and the sending domain will get added to the spam list automatically. Especially if the attack hits multiple users on the same email service.
As a user, I would prefer no welcome email at all.
(cuu508, "you" in this instance does not mean you)
There was a time were you would have to select "sign me up for your newsletter" then you had to uncheck it. Then you had to check to not get an email and now you don't even get that choice.
And lately? You have to go dig through your email because you can't set a password (looking at you Claude), so you can't filter email.
What I find annoying is services that only send the welcome email once, and don't let you resend it if you never receive it.
We added a captcha and used a disposable email checking service to get rid of it.
Author, why can you not use your own words?
I am not sure what you meant to say, vs what is LLM garbage I could have prompted myself.
Which makes is even more annoying. Because you don't know which are the good bits where somebody is sharing his unique insight, and which is just taken from the LLMs world knowledge.
Why not accept that it is good, and forget about it being LLM?
> Because you don't know which are the good bits where somebody is sharing his unique insight, and which is just taken from the LLMs world knowledge.
The text probably have been based entirely on the internal notes and investigations and is very informative. Would it be better if the OP wrote it entirely by themselves? Not necessarily.
> New users were signing up but not doing anything, they weren’t creating an org, a project, or a deployment, they just left an account sitting there.
Surely the LLM version is:
> New users were signing up but not doing anything; they weren't creating an org, a project, or a deployment—they just left an account sitting there.
/s