How did LastPass master passwords get compromised?
palant.info
palant.info
> Our investigation has since found that some of these security alerts, which were sent to a limited subset of LastPass users, were likely triggered in error. As a result, we have adjusted our security alert systems and this issue has since been resolved.
Source: https://blog.lastpass.com/2021/12/unusual-attempted-login-ac...
Source2: https://twitter.com/troyhunt/status/1476296988001849345?s=21
If I tell to that the world appears to be shaped as a globe then that statement isn't vague just because I don't explain _why_ it appears shaped as a globe.
“SOME of these security alerts” “were LIKELY triggered” “HAS BEEN solved” (Emphasis mine)
How can the issue be definitely solved if you aren’t sure that they were actually triggered in error, if they were in error then it’s only some of them.
- Eng are still writing the postmortem
- Marketing want to put out a statement
- Eng know or suspect a bug exists that can trigger spurious notifications, but don't have sufficient logs to be able to reconstruct if that bug was in fact in play in production
- Legal advises not to say anything definitive that they can't stand behind later
I don't see any of that as particularly damning or malicious. "We aren't yet sure, but have a suspicion and are still investigating" can come out like the LastPass blog post when run through the PR filter.
"We have identified and fixed bugs that could result in incorrect masterpassword use notifications being sent but we have not yet been able to determine which if any of the recent wave of notifications were caused by those bugs. We are still investigating the issue".
Instead of communicating clearly around a serious security incident they are using mealy mouthed PR speak which does nothing to improve their image.
They are claiming they know what the bug was.
The problem is the year subs. To avoid wasting money you need to do it at the end of a year, but you also need to get your users trained up before the switch. We hit a complication and ran out of time and so had to re-up.
What I want to know is have I been compromised or not, the PR saves face at further expense of users (if they truly have been compromised).
They have been extremely clear that they have not found any signs of compromise. Did you miss that?
Of course no company can technically guarantee that they have not being compromised. If you are looking for someone telling you at any point they are 100% confident no user accounts have been compromised, then you will pick a company lying to you.
I also agree with you about the 100% confidence about not being compromised. Perhaps my previous statement was too black/white. I don't want PR or placating statements, I want a transparent status report without weasel words and which exhaustively covered the different cases (e.g. SOME of the messages were sent in error. what about the rest? Are the rest routine compromises that happen normally? Or was there a spike in compromised accounts?)
I get that direct evidence of a leak is difficult. However, a sudden surge of master passwords being known by third parties in uncorrelated accounts is a very good evidence that something happened. If that's not what happened, then what happened exactly? Was it really a bug in the notification system? Do they have evidence that the password used in the blocked login attempts weren't really the actual master password?
There is a lot of things they can do to show they are on top of things.
If you are a customer, and you received this message, you should definitely change your master password and probably rotate your stored passwords. You don't know if your email was real or not.
However, it explains why so many users were getting this message recently in a plausible way, that is not too hand-wavy except for their dodgy track record. Its not the level of transparency I would expect from Mozilla or even Reddit, but its par for the course.
You should probably migrate to another password store. I moved away a while ago for other trust reasons, but this particular incident on its own is not that concerning to me.
1. lie about it and the truth never comes to light
2. lie about it and get caught and the consequences are the same as if you came clean
When evaluating this kind of conspiracy theory, it's important to consider the number of people who would have to remain silent for the conspiracy to survive, and to consider how much it would cost to keep that many people silent. In this case, it's at least a few dozen so I think it's fair to assume that such a lie would not survive very long.
I’ve seen estimates as high as 10% of the population of east Germany were spying on their neighbors.
As far as examples of successfully kept secrets, I can’t give those, because I’m in on it.
All a conspiracy is, is a group of people keeping a secret.
As far as conspiracies not being successful long term, history tells us no such thing. Conspiracies with tons of people are successfully kept every single day, only to be discovered decades later when something is declassified for example.
You can never prove if a conspiracy to keep a secret is not successful if you never knew it existed.
Am I making any sense?
You sound paranoid.
Naming conspiracies that have never leaked is kind of hard, since we wouldn't know about them. So you have to look for examples of things that were talked about after they stopped being relevant, or that were uncovered without anyone blabbing.
This is a very unlikely expectation. Employees are under NDA so nobody will talk publically about it unless one of them feel so strongly about it to sacrifice their career (they'd certainly get fired, and being sued for breaching the NDA isn't going to make finding a new job easier).
Employees at all companies keep quiet about bugs like these all the time, that's the most common outcome.
NDAs are unenforceable against whistleblowers who report illegal activity.
Replace "eventually" with "maybe".
It certainly isn’t reassuring that they keep talking about credential stuffing, even though it’s quite unlikely to be the culprit here.
So then; Is the bug also responsible for pushing bad data to the users dashboards? If this is really a bug, it's a complicated one. I'd be curious if those IP's are still being shown on the users end.
Why would it need to be a complicated bug? It could be as simple as:
If UnkownIP OR InvalidMasterPassword Then LogAndSendNotication
Instead of:
If UnkownIP AND InvalidMasterPassword Then LogAndSendNotication
Please tell me why it needs to be a complicated bug.
The part of the site that shows the recent connections to your account, with IP's. People were sharing those IP's and thoughts in the linked thread.
That data came from somewhere; if it's related to this bug then this bug is also pushing incorrect data into other parts of the service. It's also entirely possible that it wasn't a bug, and instead a security issue, and that data is correct, and they're bending words to play it off as just a bug.
I can understand a bug triggering a warning system, but when it's also presenting related data in the account security logs; it's either a complicated bug or there's more to the story.
I showed you pseudo-code which could trigger this issue. It was trivial code which could cause it in practice. Yet you claim it must be complicated or more to the story. I have no idea why you feel that classifying a request in a certain way must be caused by a complicated bug - very strange.
I think you're into FUD-mode now because even after being shown wrong, you continue to spread misinformation.
Also, you are referring to LP functionality which to my knowledge doesn't even exist, and when asked you say you don't know the software being discussed.
Very strange behavior by you.
I'm viewing this through the lens of multiple days of differing social groups poking at this, and the crowdsourced information that's yielded.
> ...shown wrong, you continue to spread misinformation.
You haven't shown anything wrong: neither of us know what actually happened. nor am I spreading misinformation, nor making statements as to what happened; I'm questioning it. Is there some reason you're so accusatory?
> Also, you are referring to LP functionality which to my knowledge doesn't even exist, and when asked you say you don't know the software being discussed.
[1]. I haven't touched LP in, ehhh, 10ish years, and it was a feature even back then.
[1]: https://support.logmeininc.com/lastpass/help/lastpass-accoun...
I thought that was obvious given the context. You are spreading BS that this must be caused by some complex bug yet having nothing to back it up except for anecdotes. You have already said you don't know the feature set and you haven't used it in 10 years, yet you are making such a claim.
Why wouldn't I accuse you of spreading misinformation if you make claims without knowing what you're talking about?
> We then take that value, and use a salt (a random string per user) and do another 100,000 rounds of hashing, and compare that to what is in our database.
https://blog.lastpass.com/2015/06/lastpass-security-notice/
While it's true that the client-side hashing means that LastPass never sees your plaintext password, the first hash effectively becomes the password. Then it's on LastPass to treat it as such, which they claim to do by hashing it again.
Edit: another link describing the use of PBKDF2:
https://support.logmeininc.com/lastpass/help/about-password-...
Unfortunately the only way I know how to implement that is to have the server send JS down to the browser that instructs it to perform the hashing. For certain types of compromises server side, the attacker would just modify the JS to get the unhashed password. I'd also need to fall back to pure JS hashing for old browsers (5% users?), so there's a UX concern if I perform lots of rounds.
I kind of wish there was a different password HTML field that could run the client side hashing without JS, so the browser would manage that. Ideally using different UX so the user understands they are using a "safe" password field. The end result would be to deny access to the raw password, which is likely reused on multiple sites.
There is no 100% solution to this.
I would imagine most developers unfamiliar with encryption would assume that client hashing is sufficient and not bother with server side hashing which is the only one that ensures privacy in case of compromise on the server side (nothing really stops client side compromise).
You can certainly have client side JS produce a hash using some other credential as the salt and then encrypt the information server side. What you really lose is the ability to do low cost comparison, because most password hashing is done deterministically. Instead you would need to retrieve the record, decrypt with the server side key, and finally compare.
Extra hardware seem cool, but I've also rarely seen people using them. I'm guessing the added cost is a deterrent.
I'd like to see more password manager use, but changing user behavior is hard. Google suggests <25% of users do so.
https://services.google.com/fh/files/blogs/google_security_i...
* An attacker with access to the database will know they can reduce the "hashing algorithm" to two sequential hashing algorithms and still bruteforce a series of plaintext passwords to check to see if the hash matches what is in the database.
* An attacker with access to the plaintext network communications or app server can just store and replay the second hash to login
* An attacker with access to the client machine can grab the plaintext password still
Lastpass does this is for end-to-end encryption reasons, where it is useful, but for standard apps I don't think it would be.
However, it prevents the attacker from immediately trying the same raw password on other sites (i.e. credential stuffing). They would need to perform additional offline attack of the first hash. This adds cost to something that would have previously been trivial.
Given that about 65% of users reuse passwords and 76% don't even use a password manager [1], I think that slowing down credential stuffing attacks is important.
Protecting against some attacks is valuable even if you don't protect against all attacks. Layered security.
1. https://services.google.com/fh/files/blogs/google_security_i...
The web browser ecosystem makes this a pretty hard task, unfortunately. You need fallbacks that weaken the process to the point where it is basically useless in a lot of cases. But the goal of a client/server split in the KDF is actually quite sensible. The client side does the large amount of work to protect users from dictionary attacks. The server side does a relatively much smaller amount of work to protect itself from being easily exploited if its hashes are exfiltrated when the hashes aren't themselves vulnerable to dictionary attacks.
I would wonder though -- there's obviously a practical limit on user experience for waiting for computation, and is there really fast enough implementations available to the browser within that limit that would foil what a dedicated attacker could compute in proper hardware for a dictionary attack? You'd also have to consider if it's really that expensive to just throw more hardware at a server side hash. I guess it'd depend on the browser, whether the attack is targeted, and how big the attacker's budget is that you're considering.
Either way, if this was the goal and the team considered those factors then still felt it was worth the effort of doing so I would think it seems reasonable, though it's probably not something I'd make standard practice in my own apps.
But I can see using a split KDF for some high-security system where the clients all have a known minimum spec.
That is useless if a hash of the passphrase is sent by the client. The input space is evenly distributed over all hash values, so a dictionary attack is no better than sending all possible hashes directly (brute force).
A single round of server side hash suffices here.
Browsers have pretty good support for surfacing native code SHA family hash functions which you can use to speed up PBKDF2. It's called the Web Crypto API and it's available even in Internet Explorer 11. [1]
If you're willing to drop support for IE11 and 10+ year old phones like the iPhone 4S, then you get access to WebAssembly. With WASM you can get a bunch of custom algorithms to be quite fast. The Argon2 browser WASM library claims to be only about 10x slower than optimized native code. [2]
It's not perfect, but it isn't as bad as it used to be with just pure JavaScript.
--
[1] https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_...
Not sure the specifics of how lastpass implements this but this is a really common approach for end-to-end encrypted apps.
That makes complete sense, thank you for the answer.
As in, the idea is that it is used to save you from having two secrets which might be more or less easy/hard to remember.
It's a UX improvement (which might be a security imorovement on average too).
[1]: https://github.com/cfrg/draft-irtf-cfrg-opaque
[2]: https://blog.cloudflare.com/opaque-oblivious-passwords/
That logic doesn't really make sense. Malware might make hacking LastPass accounts unnecessary, but it would still be highly desirable (one target gives you everything else).
Frankly, it feels like OP decided on the conclusion before any analysis was done.
I don't think individual financial accounts would sell that well without really knowing if you have the necessary associated email accounts, etc to delay detection of a fraud.
I also don't think such groups are necessarily sophisticated enough not to have someone slip up at stage 2 of their plan, give away a few accounts to boast, etc, etc.
Malware doesn’t seem to fit to me.
Recent and related:
LastPass Login Attempted Activity Blocked – More Information - https://news.ycombinator.com/item?id=29731317 - Dec 2021 (12 comments)
LastPass says no passwords were compromised following breach scare - https://news.ycombinator.com/item?id=29723319 - Dec 2021 (68 comments)
LastPass users warned their master passwords are compromised - https://news.ycombinator.com/item?id=29716715 - Dec 2021 (313 comments)
Ask HN: How did my LastPass master password get leaked? - https://news.ycombinator.com/item?id=29705957 - Dec 2021 (508 comments)
> Our initial findings led us to believe that these alerts were triggered in response to attempted “credential stuffing” activity [...] We quickly worked to investigate this activity and, at this time, have no indication that any LastPass accounts were compromised by an unauthorized third-party as a result of these credential stuffing attempts, nor have we found any indication that user’s LastPass credentials were harvested by malware, rogue browser extensions, or phishing campaigns.
> Our investigation has since found that some of these security alerts, which were sent to a limited subset of LastPass users, were likely triggered in error.
[0] https://blog.lastpass.com/2021/12/unusual-attempted-login-ac...
This added bit hints that these emails were erroneously sent in response to the wrong password being attempted against the master account (which normally doesn't rate an email). It isn't fully spelled out tho.
LP spends most of the blogpost going on about bad user practices - which feels a lot like gaslighting in this context.
edit: I'm leaning toward accepting LP's incomplete, forced explanation and that hackery isn't in play here.
This seems likely. After the original HN post, I went to delete my LastPass account as I've been using Bitwarden for several years. I initially used the wrong password, but then successfully logged in.
I got the alert email in question, so either it was sent in response to the wrong password (incorrectly) or it was sent in response to the full login, which of course wasn't blocked. The former seems more likely to me.
It could also happen that there's always been a low frequency bug (e.g. 1 in every 50000 failed login attempts is mistakenly treated as "correct master password" and triggers email) but the nature of it was triggered more recently by some change in attacker behaviour.
e.g. imagine your bug is, if the failed login occurs just after the top of the hour, a variable somewhere has recently changed and this is mistaken as "correct master password" even when it isn't, so the email goes out. Somebody at LastPass finds this bug, it's happening maybe 2-3 times per month, no big deal, P3 give the new engineer trainee something interesting to look at in 2022.
But meanwhile bad guys discover the timeout they've been fighting resets hourly. Instead of one attempt every ten seconds per IP address in the botnet they're using to try phished credentials, they can do 360 attempts in the first 10 seconds of the hour, then do something else with the botnet for the rest of the hour. Now most of these attempts hit that bug, and suddenly dozens of spurious emails per hour are going out to your users. Ouch. Now, who is going to explain to the big boss that this is the same bug you triaged as P3 last month?
He spends most of the post dismissing user error. That looks very reasonable to me, as those are the most likely issues by a large margin.
It certainly isn’t reassuring that they keep talking about credential stuffing, even though it’s quite unlikely to be the culprit here.
It's possible the old accounts could be some old stock sold on a darknet forum and are being bundled in with the newer hashes/pwds. It's also possible that the entity harvesting the newer hashes/pwds isn't the same one who is amateurishly attempting to access the accounts.
Note: Lastpass's geolocation may be off (even more than usual for geolocation) as some of the IPs are in ownership dispute and all of them may be for VPNs.
Second: People only notice the failed login attempts. I don’t know what exactly this attack looks like, but I doubt that the point is triggering these alerts for as many people as possible. They rather want to log in successfully, meaning without any alerts being produced. Who knows how often this happened without anybody noticing?
Finally: We only know about people who were concerned enough about these alerts to write about it on Hacker News (or in some cases Twitter). That’s a tiny fraction of all LastPass users.
Normally, it isn’t such a huge vulnerability. TLS encryption is there, so nobody should be able to catch that hash in transition. And even if they did, the most sensitive data is encrypted so that you still need the master password. Still, this is rather suboptimal.
I am unsure if this is too small of a compromise for them to be able to a) care or b) spend resources on investigating or that they just don't know what's happening.
That their first reaction is to downplay rather than to admit that they don’t know much yet – that’s rather typical of LastPass unfortunately. Sadly, with this approach they aren’t exactly an outlier in the industry.
Thoughts on this as a possible root cause?
Well, or in the human-chosen passphrase. There are plenty of systems that can brute force an 8-character alphanumeric password run through PBKDF2 for 100,000 rounds.
Per https://support.1password.com/pbkdf2/, that costs...about $60k.
So keeping the ciphertext safe is in fact a very reasonable precaution, especially if you have a fairly short input passphrase or are not using a ton of rounds of key stretching.
I'm not sure how to read that table. Is that really the cost for a 100,000 iteration PBKDF2?!?
But yes, it matches my intuition--brute forcing human-strength keys is surprisingly cheap. (And I don't know if they're taking into account the discount if you have custom ASICs for this, defend against which is the argument made for scrypt instead.)
The problem is that, with web-based password managers, you are not only downloading the database, but also the code to decrypt it. A locally installed Keypass requires your PC to be compromised, whereas for LastPass it is sufficient for their servers to be compromised (while not avoiding the problem if you are compromised, either).
This seems par for the course, I can’t imagine any credible company or keen user actually using lastpass at this point.
*Not secure: It will always catch you off guard, and will require a lot of work, so you will postpone it which is, not secure.
- There's no way a flaw in an authentication protocol could compromise a master password (because the file sync software is completely detached from the password manager).
- Someone who compromised your master password can't get your passwords without first obtaining your database files.
That being said, I don't think online password managers are inherently insecure or anything like that.
Were those requests the ones that triggered the emails that many of us received, and were those requests made with the correct or incorrect passwords?
Do you have an explanation why some people changed their LP passwords, and then received another login attempt alert email after that? Is that a coincidence (i.e. it was just more incorrect credentials still being tried on the same accounts) or was the attacker aware of the password change? Did the attacker have access to the new password or not?
Many of us received the alert email that our passwords had been used (i.e. an attempted login with the correct password from a new IP), but swear that those were unique passwords (in my case, it was computer generated, locally stored in KeePass and never re-used -- many other cases like that). Did the attackers have our passwords in their possession, or no?
To be fair to them, I don’t think we’ve seen any reports of folks password DBs actually being compromised. Just a lot of presumed failed attempts.
If the e-mails were sent in error, then it’s all much to do about nothing. If the master passwords were actually compromised, then the system still successfully protected the clients password assets.
Could be some crazy coincidence but what Lastpass is saying isn't adding up.
https://www.securityweek.com/credential-stuffing-successful-...
The term "credential stuffing" is a bit counterintuitive for me as well, but I guess it fits the pattern of credential attacks. "Password reuse" has other meanings, so would be less precise as a technical term.
In the first, case, I'd feel less empowered to do something about it. In the second case I'd be more aware of all the databases with old passwords in it and never reuse a password twice. (if that's what's happening).
Researchers will sometimes modify services to log password attempts in plaintext in order to try to determine where stuffed credentials are coming from, but for obvious reasons it's not advisable to do this on a "real" service, so it's usually hard to know exactly what's going on---although the set of attempted usernames can sometimes give you a good hint, for example it's not at all unusual to see credential stuffing attacks where the attacker hasn't even enumerated users and is just trying common usernames. Some of these common username lists are kind of eccentric and you can recognize which one an attacker is using by some of the odder entries on it. I've had instances where googling a particularly weird username out of authentication logs just turned up exactly the perl script the attacker was using, uploaded to some compromised webserver where Google got wind of it somehow. I assume the username list it was using was originally from some real system but had been copypastad until it no longer had any relation to the original source. A lot of common password lists are like this as well.
The term "credential stuffing" should be viewed as a term of art and not used in communications to the public, but I do think it's useful because it describes a phenomenon which is different from, and may or may not involve, password reuse by users.
As a semi-related but amusing anecdote, there was for a time a fairly large-scale SSH credential stuffing effort by a botnet whose operator had made a mistake and mixed up the "username" and "password" fields in their credential list. Many SSH servers saw thousands of attempts with various usernames like "letmein123" and password "root" or "admin". I suspect the actual root of the problem was that they'd concatenated credential lists, not realizing they were in different formats. They may have even gotten the credential list that way, these things get passed around in really haphazard ways.
Note that 2FA with SMS is much less safe than a code generator app, since it is much easier for someone to get access to your phone number than to get access to the code generator.
The trouble is that the majority of their competitors isn’t great either. I used to recommend 1Password, but another knowledgeable security researcher isn’t really fond of them.
It's like data management in general: If you don't uniquely, personally control it, you don't really control it.
It is hackers using existing compromised email/password combinations on brute force attempts at Lastpass.
That is why your Lastpass password should be a password that you have not nor will ever use on any other site.
However, at the moment I'm not satisfied that a compromise has been demonstrated, either. As near as I can tell, nobody has reported a compromise, just suspicious emails. That's not enough evidence to prove a compromise.
LastPass' response, so far, adequately covers what we've actually seen.
That may be correct, because several persons have reported reproducing that issue before the LastPass fix - they have written that they logged on with incorrect password while using an IP from another country and still got the email that their master password was used.