I worked at LastPass as an engineer
twitter.com
twitter.com
Also, as far as i can tell with some very basic maths based on running hashcat on my 3070, if you have a big long password on your vault, even with very few rounds of pbkdf, your actual passwords should be safe.
I think 1Password's use of an extra "secret key" would be a huge save in a situation like this, where every encryption key is automatically very long
Some suggestions: https://blog.1password.com/where-to-store-your-emergency-kit...
Even if you write it down on a post-it note, you now need a physical attack to access it on top of whatever attack you needed for the master password.
It‘s a strictly additional barrier to the passphrase. Bitwarden and LastPass don‘t use one, so if you are fine with that security level, you could publish it on your blog, GitHub profile, or Wikipedia talk page and would literally not be less secure than those two.
It really only is defense in depth, but in a breach like this is where it could shine. A parallel brute force attack becomes infeasible if attackers need to also compromise cloud or local storage, or even trivially correlate semi-publicly but unstructuredly posted secret keys to accounts/vaults.
Thanks, folks, I should have jsut RTFA'd I misunderstood secret key as a failsafe, it is not:
https://support.1password.com/secret-key-security/
I installed both LastPass and 1Password and they make my homegrown key storage system feel so awkward by comparison.
The key is 38 characters long, which is certainly within the range of human memory, if you're inclined to memorize it.
Even if they were 100x the size of the next competitor, they would not get a free pass for the obvious technical failures of their implementation, which have nothing to do with the number of users. The entire vault should be encrypted, end to end. The number of PBKDF2 "rounds" should automatically have increased, even for old users. These are huge oversights that fundamentally undermine their credibility.
As far as coverups at other companies go, that would be some coverup to avoid any whistleblowers leaking things. Unless it was very recent, this is very unlikely. People take cybersecurity seriously, and counting on every employee to participate in a coverup of a serious breach is unlikely to go well.
If you store the secret key locally and only locally, the threat model should be the same as before.
This seems like business and product getting in the way of good decisions. Again.
That's obviously a design choice in order to extract data that can be monetized.
Kind of unclear but i think this implies that lastpass doesn't do some and reencrypt on login. Are they saying its just people who havent unlocked there vault in a decade, or are they saying all old users have this? If the latter this is really poor design.
> Lots of vault entries may be encrypted with ECB mode AES-256.
Wtf wtf wtf. This would be considered wildly insecure even by the standards of the 80s
Seems weird that they would have the number of rounds fixed from the time of account creation though. Surely it would be possible to increase the number of rounds over time.
Edit: I should add that I have logged into this account within the past year or two. So if they're supposed to increase over time, it ain't worked.
My account is very old, but this was hidden behind advanced settings. And I have used this account daily since I created it. Not having this increase from their end seems very unprofessional.
Of course, changing it now won't do any good with regard to previous leaks.
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.’”
Ideally they'd manage that number and prompt the user to change it over time.
Keep in mind that this is a trade-off, it requires more computational power AND battery, so giving users access to the leavers makes sense (e.g. power users with nice equipment can pump up the number).
That being said I do agree that it is up to the vendor to set AND enforce a new higher minimum. If someone logs in with 5K rounds, it should be changed to the new default (100k).
On a related note Bitwarden also allows users to configure this. I recently increased mine to 150K rounds.
99.9%+ of people aren’t that technical person. Safe defaults need to be used and kept up to date for them.
Old accounts that haven't been touched in years, sure. But active accounts being used daily? This is really unfathomable.
Well, No. In metacode, pbkdf2 works something like:
derived_password = pbkdf2(real_password, salt, 5000, ...);
Encrypt(vault, derived_password);
Point being, you can only change the number of rounds if you re-encrypt the vault, and you can only do that, if the users participates by giving their password to first un-encrypt it during that process. So it cannot be done silently in the background.
I've done this with normal login for several (legacy) systems I've worked on, migrating users from md5 or sha1 to better algorithms.
LastPass NEVER gets your master key server side. All encryption is done locally.
Also, hashing a password is extremely quick. To unencrypt the full vault and re-encrypt it does take an amount of focused time.
Might not be able to be done “silently” in the background, but obviously LP should have been forcing an upgrade on login.
The entire vault should not be encrypted directly with the user's password. That would go against standard best practices.
See this comment written hours before yours, right under the comment you replied to: https://news.ycombinator.com/item?id=34127993
There is no "big difference" here, and it could be done trivially during login. The fact that it would be done client side instead of server side is a very minor implementation detail.
Conceptually, it's not very different from how password hashes are updated, just that you need to re-wrap some other key as well simultaneously. Since the rest of the vault is unaffected, it's OK if the transaction fails: the old parameters will work with the old wrapped key, so all you need to do is ensure that the transaction is atomic. (Usually don't even have to do anything to ensure that if you just use the same database table/object.)
And it's probably a good thing the original design didn't have that — I'm pretty sure Crypto.getRandomValues() wasn't available on all browsers yet when LastPass was first release it's JavaScript version (not sure when that was — I've never been a LastPass user).
Of course, the format itself can also be (and should have been!) transparently upgraded. Interrupting legacy users just once for a 20-30 second upgrade of their entire vault is a no-brainer. The fact that LastPass hasn't proactively upgraded old vaults on login during all these years reflects very badly on their security.
I think there might be a way to update it without having to wait for the user to supply the master password, at least if their web login works like most, although it would require some additions to the database and their server side vault storage.
According to the documentation I found the master password you create when you make your account is the master for both logging in to your account and for encrypting your vault.
To handle its use as a login password they probably have a table somewhere that contains:
email, salt1, hash1(master_password,salt1)
To handle updating login hashing they could change that table to contain for accounts that have logged in or been created after the difficulty change:
email, salt2, hash2(master_password, salt2), NULL
email, salt2, hash2(hash1(master_password, salt1), salt2), salt1
When they do the change, they would change existing vaults to double encrypted, with derived_pass2 = hash1(master_password, salt1).
When someone logs in for the first time after the change, they supply master_password. The server would see from the login hash table that salt1 is not NULL meaning they are an un-upgraded account. It could then compute
temp = hash1(master_password, salt1)
pwhash = hash2(temp, salt2)
email, new_salt, hash2(master_password, new_salt), NULL
NOTE: the above is based on documentation of theirs that says you use the master password for logging in to your account. However other documentation of theirs says that the master password is never sent to their servers.
This allows an attacker to detect repeating plain text segments, e.g. reused passwords.
The Wikipedia article on block cypher modes illustrates this problem rather well [1].
[1]: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation...
Also, I've just learned that the quintessential file has been recently replaced. I understand the need for a higher resolution file, but I'm still a bit sad.
This is hardly a problem, any login form will also allow an attacker to do this.
You aren't wrong, if true that's certainly a mistake on their part. Practically speaking though as long as the passwords are randomly generated it probably doesn't matter, i.e. if you think about what the IV and chaining are doing in CBC. ECB is just as good for encrypting a random stream of bits; CBC just makes sure the bits are random before passing them into the cipher. Of course the optics are still terrible.
1. In real life, passwords aren't random. Password manager users are probably less likely than average to make this blunder, but I'd still be willing to bet _the majority_ of lastpass's customers do.
2. Even if you personally have all random passwords, would you really want to use a product made by people who made this level of a noob mistake? I used to pay for LastPass premium, now my account is deleted, my passwords are changed, and my money goes to Bitwarden instead - and every time I hear about Lastpass I don't regret this decision.
You need to know the specific order how to create the password for a new site. Otherwise it was just easier to plow through with your favourite memorised combination.
At least I found it hard to do with Lastpass on mobile. Now I’ve tested it with Bitwarden and 1Password 8, they seem to do a better job with it.
i have been using keepass(xc) since it's been around and i dont use random passwords. every year i create a formula that creates all my passwords and i put these in the vault. it also enforces password rotation. losing the vault i can still come up with any of my psswords just by remembering the formula.
for company server services it's a different story, i generate wholly random strong passwords and store them in the offline vault.
edit: typos
Obviously this isn't an endorsement of LastPass. Everything about their protocols is disappointing. Even if the encryption issues turn out to be of the well-that-was-a-dumb-choice-but-you-don't-need-to-panic variety, it's because they got lucky, not because they were good.
Maybe I wasn't obvious enough, but this is the problematic assumption. When I say "non-random password", that's what I mean: a reused password.
Almost everyone I know reuses passwords (sometimes with a slight variation, but 90% of the password the same), and my day job is a software engineer. I'd be willing to be at least half of Lastpass's users have at least 1 reused password.
The reason I suspect even non-random passwords might be mostly fine[1] is because there's no oracle and passwords are generally short. There's not much there for someone to hang their hat on. It's only a hunch, though; if tptacek for example says something different, I'd believe him.
1. Meaning: No need to cancel Christmas. You can deal with it later. Not meaning: ECB is as strong as CBC or whatever.
If the attacker doesn't have the valid password, the knowledge that two different sites share the same password isn't exactly game-changing. And if the attacker does have the password, they'll figure out where it works regardless of access to LastPass db.
To an outsider, hearing people talk about how LastPass used Very Bad ECB mode makes it sound like a fuckup with immediate and dramatic effect on the security of their passwords, that isn't exactly true.
Nobody should have been using LastPass, but we can still be realistic about the threats facing those who did.
> but the obvious problem here is the offline cracking of the vault passwords; this isn't worth arguing about.
Of course not, but that's a completely separate issue from their choice of AES mode.
If you have some attacker after specifically you, yes this information is of less use to them, because they will have tried every password of yours they have on every website already hoping you reuse passwords.
Kind of a long shot though.
Have they not heard of the first rule of crypto being "You probably aren't qualified to do it, use libsodium or TLS or SSH" yet? Why are they messing with cipher modes themselves?
a) LastPass is extremely old
b) Browser extensions used to be completely different. You needed to call out to native code to do tons of stuff because the JS apis were extremely limited and slow. So it was either "roll crypto in JS" or "make users install a binary", the latter stopped being possible over something like a decade ago.
CTR is not a very robust block cipher mode...
The point is you explain why you do this. Start the message with "We care about your encryption" and go from there to explain that they need you to upgrade because the don't know your master key, they need your help to do it.
Adding this kind of behavior should be their core business. Apparently it's not.
For this particular use-case? Hardly.
Since an attacker can easily use any login form of their choice as an oracle for password reuse, ECB mode can't really introduce significant issues here.
If the attacker has an encrypted vault, what is the login form there for them to use and detect password reuse?
Incorrect password attempts are essentially free, so ECB revealing reuse does not meaningfully help the attacker.
But for the attacker to try one of my passwords in a login form, they do need to have that password cracked already, which means that they already have my master password. But if they have that, they can decrypt all of my passwords, and the login forms are of no use.
Am I misunderstanding something here?
No, not at all. The point is that while ECB is a silly choice, it does not make it easier for the attacker to crack your vault. It does allow the attacker to see if you're reusing passwords, but does not reveal to the attacker what those passwords are.
On the other hand, If the attacker were to discover one of your passwords from another source, they'd be able to confirm reuse anyway by simply attempting to use that password on other websites.
- https://dropbox.tech/security/how-dropbox-securely-stores-yo...
- https://en.wikipedia.org/wiki/Pepper_(cryptography)
Their payouts for reported bugs under bug bounty program have been very low, for a service used at the scale of Lastpass I believe this is just not ideal.
> Points – $5,000 per vulnerability
Most people wouldn’t sell data for $40,000 if the bug bounty was $35,000. But that lastpass leak is worth many tens of thousands more than the $5000 (average payout of $450) offered.
I don’t know if he ever got into the details of why (maybe he couldn’t for contract reasons or something), but his judgement on it looks accurate.
I’d be curious what his thoughts are, or if the risks he was talking about back then we’re unrelated.
They do what any reasonable security company does, and call in vendors who have negative incentive to lie: https://support.1password.com/security-assessments/
I was especially impressed by the Cure53 ones, where they were provided access to the source code: https://bucket.agilebits.com/security/Cure53-1PW18-report.pd...
See page 10 [1] which explains how the secret key provides extra entropy (which makes it much harder to brute force a user who choses a weak password). Also see Story 1 on page 11 [1].
[1] https://1passwordstatic.com/files/security/1password-white-p...
From a large development team whose sole product centers around encryption? It seems strange to get it this wrong. What leads to that might make a good business case study.
At big companies, too often do the people in charge of the product seem to forget what core product really is.
They have been coasting for years. There are some serious bugs they never addressed.
I wish I could export my "password history" in some way
We may not be seeing the whole picture yet, but very concerning that the dev environment had access to Production database backups stored in a manner that was easily decrypt-able :/ The whole point of a Dev environment in a security focused company is that you reduce the surface area for access and failure.
> some source code and technical information were stolen from our development environment and used to target another employee, obtaining credentials and keys which were used to access and decrypt some storage volumes within the cloud-based storage service.
It sounds like they didn’t literally have backups in the dev environment (which would be absolutely terrible if they did).
I’m guessing they learned enough about the architecture from the dev environment to social engineer their way into getting someone to give them credentials to production.
> dev environment had access to Production database backups stored in a manner that was easily decrypt-able
Say what you like about old dinosaur companies like banks, but when I worked in the IT department for a bank back in the early 2000s developers absolutely couldn't touch production systems, data or backups. Ever. They always has to go through ops even during major incidents.
Of course banks aren't immune to fuckups, but at least on paper they tend to have sound security policies.
Startups are great at moving fast because there are no guardrails. Like lemmings, corps, banks and government entities externalize their risk to the third party because they recognize the name and logo. "Should we go with SecureCompany(tm)?". "Yeah, everybody uses them".
It's great for startups. Without red tape and heavy policy your lean team can create products that the hamstrung bigcorp could only dream of (just don't look under the UI!). They slowly build their customer base, and when they're ready to IPO, they run through some security and process certification theatre, and the investors and a handful of executives can make billions.
Overall, both sides seem to be happy with the situation. Bigcorp gets to cut tech staff and outsource, and they have a convenient finger to point when the regulator asks questions. The SaaS apologises and maybe loses a customer or two, but based on stock prices after major issues like this in the past, nothing bad really happens.
I'd be curious to read up on this. Would you happen to have a link to some coverage of this?
For comparison, BitWarden's report shows 2 [3] and 1Password's shows none [4].
[0]: https://www.tomsguide.com/news/lastpass-android-app-tracking [1]: https://reports.exodus-privacy.eu.org/en/reports/165465/ [2]: https://reports.exodus-privacy.eu.org/en/reports/com.lastpas... [3] https://reports.exodus-privacy.eu.org/en/reports/com.x8bit.b... [4] https://reports.exodus-privacy.eu.org/en/reports/com.agilebi...
OK higher is better sure, but is moving from 5K iterations to 100K per the recommendation a similar precaution to adding 1,2,3 digits to the master password?
The best precaution is to never use LastPass. If you used LastPass, the safe course of action is to change all your passwords that were in LastPass, and use a better password manager this time.
I'm not sure to whom the rest of your response is addressed, this was a technical question.
Adding a digit to a password can depend. Adding a 1 to the end of a password is such a well known thing that its basically useless because that's the first thing people try as its so common. Ultimately though, you are basically correct in a bruteforce scenario. However most password attacks involve a bit of knowledge about what the person likely choose, so the effect depends on what that knowledge is.
Keep in mind if you have a strong password (say 16 character randomly generated, never reused elsewhere) its not going to matter how many iterations as it will never be bruteforced even if the worst possible hash was used. This sort of thing protects people who use weak or maybe mid-tier passwords.
Personally, I store plenty of passwords in the browser, but they're all the passwords I care very little about. Things like e-commerce (where the CC isn't stored), forums, etc., where the password falling into the wrong hands would be annoying, but hardly life changing. For the accounts I really care about, they go in KeePass (backed up to cloud), but with a strong master password and keyfile protected by admin-only access in Windows. Yes, it's way more of a pain to 'sudo' KeePass and copy-paste passwords than use auto-fill, but I'm only logging into my bank accounts and stuff occasionally. If I managed large crypto wallets, those secrets would probably see their KeePass DB get another layer of security: air gap or similar. No way would I comingle those with my Grubhub password that I couldn't care less about.
The worst part was the reward. $1000 is a sick joke.
Unfortunately I believe that’s Safari-only and even then they seem to be aiming to move away from it - all non-Safari extensions are a “fat” client called 1Password X where the entire logic (and thus the sensitive data) is within the browser.
That information is long lived (unlike tokens which expire) and you need to use the real value (unlike the hash of a password). Are there resources/books which cover the (many) best practices to keep that information safe?
Things like: separate database/microservice, separate vpc?, encrypt the database, filter those fields from logs, etc etc etc.
Is a service like aws lambda actually more secure then using a paas (heroku/fly/render) vs ECS (where there might be slower turn around times to actually bump/patch CVEs than the aws lambda team would take?).
Administrator credentials are harder yes. Something like hashicorp vault can be made to fit any company (custom backend plugins). For most stuff it already has backend plugins available (ssh, any rdbms). The open source version I have seen can fit a large fintech startup no problem.
For example, let's talk about the BI tool, mode.com. As a customer, I login to mode.com and click the "New Database Connection" button, where I get a screen to input the connection information for my postgres database: host, username, password, port, etc. If you are a developer at mode.com, what are the best practices to keeping `user_database_connections.password` secure? I can't hash the `password` column because I need to use the raw `password` when my code connects to the user's postgres database.
I'm assuming I should encrypt the information, so an attacker who does a databse dump can't read the `user_database_connections` table, but what else should I do? Keep that information in a seperate database? seperate microservice? etc, etc.
Other examples of tools which connect directly to the database(s) of the customer, so they store db creds of their customers in the saas database: - hightouch.com - fivetran.com - explo.com
KDF is not a substitute for a good password.
Why bother with cracking the vault when you can just modify the javascript sent to report back the master password? Especially if you only do targeted attacks, and/or heavily obfuscate the phoning home, the chances of getting away with it is probably pretty high.
Yes, they might also use other techniques as well, for example, push a bad update if they have such privileged access. However, that’s relevant more in targeted attacks; you can’t really push a bad update to everyone before getting caught quickly. Also, often several people have to sign the code (in the case of non-JavaScript code), and there is log trail.
Edit: because I was (until very recently) a daily active user and had niter=5k. Didn't even know about this setting until the hack.
I can confirm because I am one such person. Honestly thought that would come across from context but I'll edit to clarify.
It's not like you need to work at LastPass or have deep expertise in the domain to look at your own account.
A properly designed online password manager is an extremely safe choice.
>"If you had a legacy 5000 rounds or 10000 rounds, the # of rounds can't just be increased server side. Changing that would mean a new key is derived. So older users likely still have a pretty low number of pbkdf2 rounds which makes them more vulnerable to brute forcing"
Can someone say about how long would it take for an attacker to brute force a pbkdf2 master key with 5000 or 10000 rounds?
Looking at their blog post from a few days ago they state:
>"The threat actor may attempt to use brute force to guess your master password and decrypt the copies of vault data they took. Because of the hashing and encryption methods we use to protect our customers, it would be extremely difficult to attempt to brute force guess master passwords for those customers who follow our password best practices. We routinely test the latest password cracking technologies against our algorithms to keep pace with and improve upon our cryptographic controls."[1]
Can someone say did they notify customers who had legacy accounts that used 5000 or 10000 iteration pbkdf2 master keys? They sure don't seem to mention anything remotely like that in their incident report.[1]
In the above paragraph "best practices" is a link to the URL that fails to mention anything about selecting or periodically increasing the number of pbkdf2 iterations.[2]. If this weren't bad enough you are interrupted from reading their important incident report by a pop up asking you to sign up for their mailing list. Is an incident report really an appropriate place to use for marketing purposes? Are they really this shit of a company?
[1] https://blog.lastpass.com/2022/12/notice-of-recent-security-...
[2] https://support.lastpass.com/help/what-is-the-lastpass-maste...
I ask because I presume LastPass put a heavy emphasis on security, yet this still happened. Is there a 0-day still out there in the hands of very malicious actors and not yet patched?
> some source code and technical information were stolen from our development environment and used to target another employee
How was the initial (src + docs) breach performed? How do they escalate that into a successful social engineering attack, particularly since the breach was known of?
Did the targeted employee have unfettered access to all this customer data?
Will be interesting to know the fuller story. I'm guessing that's only a matter of time.
I have a rather long pass phrase. That should theoretically mean that even with my vault, brute-forcing this is out of the question and everything is still business as usual?
If that's the case I'll sleep easier at night than the thought of trying to move everything yet again to another password manager and handle (what are strong) password resets at places like Google .... shudder.
Cryptographic libraries shouldn’t even expose it except in a subpackage or namespace called “hazmat.”
But then I'd have to say... why is a company doing a secure password saver not employing anyone who knows the very basics of how to use cryptography? :)
Because that isn't important for success. Perception is reality. Very few people can tell the difference between good and bad crypto work. So instead of wasting money on that, invest it into something that is understood by more of your customers. It's a sad reality.
Just exported my vault, and will see if there's anything relevant in there (I haven't used LP in 7 years, but I probably have some unimportant-ish passwords in there from back then)
Until now, I had forgotten that they still had my data :/
Switched to Microsoft’s authenticator. Never ever again trust a small company with my passwords.
May they say "F this, I quit" and we hire them.
I'd be more worried about a password manager that has never seen any security incidents - is it because there really aren't any, or that they haven't been caugh? Surely a security incident on a password manager serves as a major motivation to harden shit up?
It's owned by a Private Equity firm, which means cutting all costs and selling hard until there's no money left to squeeze out of the corpse.
They also have a development team that's very active. Lastpass had been coasting for years.
They've also listen to our bitching about requiring the master password every 2 weeks even with biometrics turned on, and rolled that back to a configurable setting
1pasaword is not perfect, but it’s far better than lastpass
I cannot fathom why anyone would look at the situation today and decide that the solution is more of the same.
Get an offline password manager and keep your valuables out of the cloud. I like Keepass, but there are plenty of FOSS solutions out there that allow you to actually own your data.
Just sayin'.
Get something that writes data in open format and does not interfere with how you store it. Eg. Keepass with some privacy focused file storage.
1Password has a fantastic track record.
Yes, tomorrow, their incentives might change. But the data is copied and available locally; if they start fucking with user data, we will see it coming.
Security is a lot about trust and trade-offs. This includes trusting someone who has shown themselves to be trustworthy, even if they might not be trustworthy tomorrow.
I remember too many SaaS companies with awesome reputation, that over time became sketchy (including Last Pass).
As for the teams - you shouldn't have shared accounts. Making introducing them convinient is bad idea. Where I work, we're audited for that and honestly it's very small issue if you give it a few minutes of thought.
Re teams, just because you shouldn’t have shared accounts doesn’t mean you can’t have shared secrets which end up in a password manager. Or more generally, the ability to have managed employee vaults. And regardless, shared accounts do happen, because some providers just suck. Making them as secure as possible is more important.
We work in Salesforce so lots of logins with login.sf.com. i have to always. 1 search credentials in plugin 2. Search credentials again to copy OTP.
Most my colleagues miss setting the seed of OTP correctly cause it feels like you are done when entering but you still have to save it at the bottom.
Lots of times it asks to overwrite (update) client X credentials with client Y
In our admin cobsole, half of the usernames are not visible, just white space, so when i Need changes i need to click around and find that setting page, which is poor UI but then gamle on every whitespace to give the right permissions to user X.
Moving to BW is high on my list for 6 months plus. But too busy, will make time to my now. Bye. Don't comment a lot but truly LP is one of the only pieces of software I really hate.
This is what you get for using services and not doing things yourself. Local, local, local.