Credential Stuffing | Hacker News Reader

Credential Stuffing

ciamweekly.substack.com

65 pointsmooreds43 comments

Oh boy this was a major problem at our budding fintech. Here's what DIDN't work:

1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.

2. Phone number 2FA. Significantly slowed legitimate user access but still didn't fully stop credential stuffers.

What did work:

3. rate limits and carefully tailored scripts that detected usage patterns and autobanned. Eventually they gave up on us guess wasn't worth the trouble. However I'm sure we lost a few legitimate users too in the process.

What I would try in the future:

- Passkeys as 2fa. Most browser automation platforms can't handle passkey auth inside a VM.

>They used advanced fingerprint-shifting browsers

I'm guessing this would be Firefox, possibly using in house extensions or userscripts designed to help further avoid fingerprinting?

There are a lot of dedicated anti-detect browsers, you can search for that term or fingerprint switcher, multi-accounting browsers, etc. Many of them are based on Chromium.

In my experience they're generally detectable by mismatches in various attributes compared to the "real" browser whose user agent they are spoofing (though of course, the ground truth of adversarial detection is always hard to know for sure).

I spent a year doing security for a highly targeted fintech-adjacent where credential stuffing was the primary security threat, and all non-phishing-resistant MFA was table stakes: all the real work was in combatting cred-stuffing attacks that had already defeated (usually through elaborate phishing) the MFA.

> 1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.

Don't you typically use that for valid users? As-in, you allow access when the fingerpint matches their existing fingerprint and when it doesn't you require additional information to be presented (i.e. security code).

So if somebody shifts their ip around they end up needing more information than just user+pass to login but somebody that doesn't (i.e. a normal person at home) does have the easy way to login.

Not sure why financial institutions still bother with passwords - every time i try to login to wise or something it requires email code/link. At that point just use the email auth.

Sending an email every time you want to log in sounds expensive.

Some years ago I researched the whole credential stuffing ecosystem for a course paper at uni.

Credential Stuffing is (or at least was) a gigantic market, and it is one of the biggest headaches for the biggest pay-walled services, like Netflix, HBO, Prime, etc.

The people that made a living out of it were stuffing millions or billions of credentials (sourced from database leaks) in the most popular services, hoping to then sell the accounts for small amounts of money, like a dollar for a Netflix account with a 10-day warranty. It's a numbers game at heart with a substantial technical aspect, where you need to optimize your checker code to essentially send properly formatted requests that can't be intercepted and don't arouse suspicion, and then you had an ecosystem of "methods" that are certain request-response chains that make your login request look like it's from a real person. People needed to figure out advanced methods to not invoke a CAPTCHA check, which is cost-prohibitive, but not impossible to solve automatically (AI wasn't a thing back then). You then have to buy millions of proxies that are able to route the requests from different IPs so that you're not sending millions of requests from a single IP. Checkers had reached a point where, depending on your proxies, were performing 10,000 or even 20,000 checks per minute. Multithreading was the cornerstone of these technologies, as a simple 2vCPU VM was already bottlenecked by proxy speeds.

Back when I looked into it, it was the wild west, as SSO and other technologies just weren't a thing yet. Companies would become fads of this credential stuffing scene, and it would take a dev team an entire sprint just for them to make a login page that was able to at least force a CAPTCHA check for each single request, and that's IF they had the proper monitoring tools to notice the gigantic spike in login requests. Having a valid account to a service like Ebay where you can then order whatever you want with the linked credit-card, you can understand how big of a security issue this is.

I haven't looked at it recently, but I assume that this has become vastly more difficult for the common-place services like streaming providers and digital goods marketplaces. SSO, IAM platforms like Keycloak, and advanced request scanning techniques have evolved. I'm guessing things have become substantially better, but it's always going to be a big issue for those smaller websites without a dedicated dev team or without at least someone maintaining them.

Is the paper public? Would love to review/reference it for the newsletter.

No unfortunately, and it's pretty old. It was a paper/report for a course during my undergrad, so not polished by any means.

Password managers and 2FA mostly solve the problem of credential stuffing but unfortunately millions of people still don't use it. 2FA is actually pretty good unless your account is a high profile account then bad actors might try to SIM swap[0] you and hijack your mobile phone number and essentially bypass 2FA. But there are others ways of 2FA too.

[0] https://en.wikipedia.org/wiki/SIM_swap_scam

I am so happy I am no longer responsible for these. We had a solid monitor and an analysis script that was quite good at dealing with the attacks.

Then the fun thing was that some lawyers concluded this is still a breach on success and that we should be responsible and report/mitigate these.

How? How do you stop your users from making dumb decisions? The only solution seems to be to "give up" and go passwordless, putting the credentials to the big boys in town.

For us, introducing a simple device and location validation system (track which users log in with which devices and from where), combined with breached password detection from HIBP, which both can trigger an email validation code flow, practically solved the credential stuffing issues we had immediately.

For the user it's kind of a a soft MFA via email where they don't have to enable it, but also don't always get the challenge.

Astonishingly, we had barely any complaints about the system via customer care and also didn't notice a drop in (valid) logins or conversion rates.

To me, that seems like a pretty reasonable approach... adding a password change at the end would probably be a good last add.

I tend to generate my passphrases for sites now, my only complaint is a password field should accept at least 100 characters. Assuming it's salted+hashed anyway, it's almost irresponsible to limit to under 20 characters. I'd rather see a minimum of 15 chars and a suggestion to use a "phrase or short sentence" in the hint/tip.

I wrote an auth system and integrated the zxcvbn strength check and HIBP as default enabled options. The password entry allowed for up to 1kb input, mostly as a practical limit. I also tend to prefer having auth separated from the apps, in that if auth fails via DDoS, etc, then already authenticated users aren't interrupted.

> a password field should accept at least 100 characters. Assuming it's salted+hashed anyway

There was recently a bug in bcrypt implementation where characters after first 64 were silently ignored.

Anyway, while it is easy to require long password it is almost impossible to detect password reuse. The only way to solve the issue is to not let users to choose passwords, if they want to change it then generate a new one for them. And that isn't happening unless sites are forced to do it by government.

As long as I can use a password manager for passwords... unfortunately, I have to login to the OS to get to the password manager itself.

I think there are plenty of other solutions, including 2fa, push notifications and likely more valuable than any of the previous mentioned bits would be to ensure that SSO works across an organization.

In general, simply requiring a minimum length of say 15 chars and the suggestion to use a phrase or sentence is enough. I've switched Bitwarden to the word generation option with capitals and numbers, which usually works, except when there's an arbitrarily small maximum length on the input field.

I switched because trying to type 20 random characters including special characters in under 20s (was a remote terminal limit on a VM I'd misconfigured and had no other way in) was pretty much impossible and had to run the reimage from scratch.

I remember dealing with a large credential stuffing attack at a marketplace right after we announced our series B ~2018. We developed some tools to keep them out through pattern matching, but it was not easy and it took some time to develop those tools.

Best companies to work with were spycloud.com and sift.com.

spycloud actually specializes in identifying leaked credentials, which are what attackers use in the credential stuffing list they go through, so you could identify "stuffable" credentials prior to the attack happening, which is nice.

sift was great at helping to just identify fraud in general, so if an account did quietly get compromised, we could identify it before the transaction was finalized.

More details on existing services or libraries for rate limiting would be nice. Still an informative article.

The author, Dan, is at FusionAuth, so that might be a good place to start.

I work for Stytch (another CIAM provider) on the fraud and security side and we do these too. I'd say you see credential stuffing defenses integrated into the auth provider rather than standalone rate limiting because so much of the relevant context is tied up in the auth side.

And, all the error messages end up being bad, as is the case for many security things. For our own features like Intelligent Rate Limiting https://stytch.com/docs/fraud/guides/device-fingerprinting/d... it's usually a bad idea to tell a user "You hit the limit, come back in an hour or contact support" because it gives an attacker information on how to improve. And we regularly see probing behavior where an attacker is trying to find the edges of a defense before starting a full-scale attack.

On the side topic of error messages - if you've ever seen "If your account exists, the password has been reset" that's another useless error message because "No account exists with that email" enables account enumeration.

Thanks for the feedback, appreciate it. I wanted a general overview of the problem and solutions first. Maybe I'll write a follow on article about services or libraries.

Love how the HN community is sharing rate limiting and other credential stuffing "war stories" here too.

at my first job I was able to identify a remote username enumeration vulnerability which, when combined with cred stuffing based on some publicly known cred dumps, netted me a couple dozen good user accounts on our web app. we ended up rate limiting login attempts and adding random delay to failed logins to mitigate the username enumeration vuln (tldr - if you sent creds w a nonexistent username the system immediately failed the login attempt, but if you sent w a good username and any password the system had to take time to hash and salt your password then compare it to the stored hashed passwords, so attempts w a good username have a much higher latency than w a bad one)

Yeah.. I started adding a random 1-4 second delay returning from failed logins regardless of the reason when creating auth systems. I really wish I would have been able to open-source the auth platform I wrote at a previous job.

It wasn't that complex, but pretty nice in that it had pretty typical features for auth, was simple to setup/configure and would simply generate a signed jwt passed to your app/url. I had db adapters to be able to use SQLite, MS-SQL, PostgreSQL and DynamoDB. It also had integrations for AD, Okta and Azure Entra.

As a person with two useless undergraduate and one equally useless graduate degrees I was scared to click the article.

One solution for credential stuffing is to generate passwords for users.

I guess ChatGPT doesn't know that one, yet.

> generate passwords for users

That would rub me the wrong way for sure. Maybe not to the point of abandoning your platform, but I'd still be irked.

Sure, and users are annoyed they don't get to use their go-to "letmein" password as well.

But this is worth it for certain platforms, especially where your users have something real to lose when they get pwned by cred stuffing (e.g. bitcoin casino).

Personally, I'd go a step further and say it's also worth it anywhere an account takeover can impact other people like Reddit and Twitter where cred stuffing lets you take over high profile accounts and then post a crypto scam or something more targeted, for example.

Mild inconvenience on a /register form doesn't sound like a serious counterbalance to the large cost of trivial account takeovers.

Maybe I don't understand.

How are you securely delivering the password to the end user? How are they securely storing a string you created? One that is presumably high-entropy?

Or are you talking about one time use passwords (OTP)?

Solutions exist that can completely block credential stuffing attacks most people just don't know about them, for example: https://layer3intel.com/tripwire

Good Article

Reply on news.ycombinator.com