LastPass says DevOps engineer’s hacked computer led to security breach in 2022
9to5mac.com
9to5mac.com
That said, LastPass is not deserving of any trust as a password product of any kind. That a password was captured by a keylogger on a Dev Ops home computer shows that they don't understand how to secure remote computers, the meaning of defense in depth, the importance of proper login authentication, or how to secure data at rest. Each of these points are close to the core of their business.
I don't wish them ill. I hope they recover from this, but they need to understand security to produce a security product.
The buck is supposed to stop at security, not at each employee's personal hygiene … if your game plan depends on the latter, it's game over.
There's more here than is being written, and I can only imagine because the truth probably stinks.
¹except TFA mentions MFA … but the mention of it doesn't really make sense.
I have to take security trainings twice a year that literally talk about the buck stopping at my digital hygiene and I better not fuck it up for The Company.
Companies have to understand breaches will happen, but preparing employees on how to spot attacks or understand when they've been breached is a huge component of their ability to repel or minimize attacks.
That's not on you as an employee, that's on the security team for not implementing compensating controls/defence in depth.
Yes you have a responsibility to detect phishing emails, not writing down passwords, inserting USB's etc. But if something happens to you completely behind the scenes during your normal business, it's not on the employee.
Most places force you to rotate the password so I would not say it is a responsibility to not write it down.
I do it. It is like there is a fixed number of passwords per life you can remember or something.
I know some places have rules like “must have at least N characters different from previous passwords”. However, depending on the exact rule, people can come up with easy-to-remember workarounds: e.g. the “blah” bit is ”foo” in even months and “bar” in odd ones.
An attacker can only hack the paper with physical access to my office. But if they have that, they might as well install an physical keylogger.
You can also combine a written down fragment of the password with a remembered one.
... and there are lots of unrelated people with physical access to your office. Cleaning staff, building maintenance, HVAC technicians, printer service staff... and all of these may not have the same level of background checks as your company has.
And even if you hire all of these yourself (which makes sense at a certain scale), that still doesn't protect you against marketing inviting a camera crew and walking around everywhere in one of these typical "life at the office" short films for Linkedin. IT staff offices seem to be very popular for such films since they're usually the most personalized rooms with lots of nerd stuff on the walls and desks.
Besides: swiping a photo of a post-it leaves no evidence, whereas installing a physical keylogger certainly does.
this is mostly so they can pin it on you when it inevitably happens (rather than the management)
In the case of LastPass, they're holding data (which they shouldn't have been to be fair) that's INCREDIBLY attractive to everyone from script kiddies to nation state actors. When it comes to keeping out nation states the threat model can get kind of wild and difficult to engineer for while building what's ultimately a consumer facing product. However, in this particular case it's fairly obvious that bad IT policies lead to an issue and LastPass got burned.
I would be more forging of LastPass but letting employees do BYOD means you have to trust everyone that ever uses that computer including spouses and children. It's just really dumb.
I guess there was a loophole in their MFA integration. Maybe they accepted the same TOTP twice - in a multi-region setup I guess this might be a trade-off that somebody might risk. With a keylogger one can theoretically steal a TOTP anyway or do other more sophisticated shenanigans.
> We enabled Microsoft’s conditional access PIN-matching multifactor authentication using an upgrade to the Microsoft Authenticator application which became generally available during the incident.
Maybe switching to push notifications with number matching is their mitigation for that (e.g. without affecting multi-region replication / performance).
cat ~/.aws/* | <some sort of curl to a pastebin>
On a devops/senior dev machine is colossal.
Anyone know of any such option? What I've come to use are separate env files that I source in various directories before running the commands that need crednetials, or a tool that decrypts a file, loads it into an subprocesses env vars and runs a program (something like mozilla/sops), but I still find that too cumbersome, I'd like it transparent and integrated with my shell.
A better way would be to not allow user accounts to deploy anything in any sort of prod accounts, instead only allowing this to happen through CI.
At this level of sensitivity you need to consider "should the employee's personal plex server be able to connect to this" and the answer might be a No. At this point you should be issuing them a PAW and a physical security key and auditing the shit out of it. It's not like lastpass can't afford it.
How much of the product can be salvaged?
They have a well-known brandname, but it is arguably radioactive now.
The product as software can be rebranded, but why go through this effort if the ubderlying software has proven faulty so many times in the past?
A similar effort can be invested in making open-source password managers better, so there is a clear opportunity cost to salvaging LastPass.
Plus a sale would surely only directly benefit those most responsible for LastPass' issues. It would mean they are directly rewarded for their incompetent execution..
All of the software engineers I have seen have a fairly unrestricted environment -- Linux machines, with sudo access, often with passwordless root access via "docker" group, and with non-intrusive "endpoint protection" system. It would be normal to for someone to run "npm install" on their machine, or check out a random github repo they read about and run code from it.
Such machine would be a prime target for malware. And endpoint protection I have seen seems to be really stupid -- basically hooking "exec" calls and checking for exact hash match (!). Any serious malware should be able to bypass it without much effort, and if it only stays on a single computer, the detection chance is pretty low.
(I have also seen some poor souls who were stuck on locked-down Windows machines.. but they usually ended up using their machines as remote terminals, doing most their actual work on some remote server. And that server is sudo-capable Linux with light/no protection, and see previous paragraph. I suppose if _that_ is infected, at least Lastpass might not be stolen... unless people start browser on server and log into lastpass there, I've seen this happen)
The developer uses MFA (TOTP, Push Notification, Yubikey etc) into a virtual desktop inside the organisation (Citrix, VMWare Horizon, etc).
From there, the developer can SSH / whatever into their development environment - which is hosted "inside" the corporate network, or their cloud provider, via internal links.
All code, and dev boxes live "inside" the corporate network, and only keypresses, mouse movement, and screen diffs are sent back and forth.
Most remote access packages can prevent clipboard, USB device, file transfer etc.
If you need a password manager for work purposes, then it lives on the corporate managed network - not your remote laptop/desktop - and to be really paranoid - you only ever "copy/paste" those passwords - you don't type them in.
If you really want to lock it down further, give the remote workers dedicated corporate equipment that they only use to access the remote desktops, so you can prevent some things like screen capturing, and really lock down the software to prevent things like keylogging software/malware.
You also should have the entire development environment segregated from the "business" corporate network as well.
It's only really an issue if you want to have offline developers - in which case I don't have any thoughts ready to hand - (but would expect it to be a very locked down machine, possibly with an even more locked down VM inside it).
As someone who regularly uses multiple layers of Virtual Desktop -> Virtual Desktop -> Remote Desktop, provided the network can handle it (on both your local network, and the corporate network), it works surprisingly well.
This is the key point, otherwise you'll also just get a linux machine with developer root access, but EVEN CLOSER to the corporate network.
Honestly, this HN thread is full of bad advice and factually incorrect patronizing. Okta-style system asking to accept every single permission would not have protected from an attack, because Okta caches and reuses authentication tokens. Clipboard snooping / keylogger detection wouldn't have worked because none of these solutions are robust against targeted attacks.
The only thing I can think of which would have (and should have) helped is alert SOC / incident reponse team. Good luck finding one though.
Developers have nothing to do with this. It's a common practice in companies that have "expensive" production environment (eg. VMs rented from AWS) that developers never get any kind of access to production environment. Ever. At all. No need to tie developers' hand by putting them behind a ton of unnecessary firewalls. They have no need for the sensitive information and shouldn't be burdened by protecting it.
The few people who do have access to company's "expensive" production environment are / should be very few people, most likely in the infra / DevOps department. These people do need to follow special protocol for communicating with the "expensive" environment, which, likely, doesn't happen all that often. Depends on the product, of course, but unlikely to be more than once a day, or even once a week.
----
PS. In many, many years of being in infra / system / automation I had never typed any passwords for any important services I had to use. They are usually difficult to type due to having all kinds of Unicode characters I wouldn't know how to reproduce w/o a little research. It's also very rare that they end up in system clipboard, since I usually end up using something like vi+tmux over SSH in Emacs' ascii-term to copy the password from somewhere and paste it somewhere else. So, stuff like AWS keys would have to be stolen by taking screenshots of my screen or something like that...
I mean, why on Earth would anyone deploy to production environment from their personal laptop? Normally, deployment is made from some sort of a testing / staging environment where the system was being tested / archived before shipping it to the next stop... It sounds like some kind of emergency / unplanned situation where a DevOps had to log into the remote system from their laptop.
You should check some solutions out.
Secure access to dev environment with multiple easy factors - MBP biometrics, yubikey, etc
Enforce strong password and FDE on the laptop.
Obviously doesn’t need to be a MacBook, but you stated an aversion to overly locked down windows laptops.
In general, just enforce strong multifactor creds when accessing internal resources.
You can also enforce device very requirements, so a stole device or yubi can’t be used either.
Then do what you want on your Linux instances in the cloud. They’re protected with their own controls.
The real answer is a zero trust network that implements:
- multi factor auth
- deployment approval gates
- end to end service encryption
- ALE for secrets and keys
- password managers
- WireGuard tunneling or equivalent
- read only production environments by default; major levers to pull in order to write
- fully partitioned environments, all of which partitioned away from the corporate network of laptops, printers, and security cameras
Yes. In general, it's a good idea to split state management from business logic.
In the simplest thing, that means that eg you have a database that's separate from the rest of your site. But the principle applies more generally.
Useful for keeping things simple.
To go further: if you want to log something, you send it to a log server that is super simple and can only write to one location. So if someone takes over your business logic service, they can't write arbitrarily.
It turns out you don't need administrative privileges for a lot of dev work (installing and running vs code, python, node, many databases, etc...).
My experience is that sudo apt-get install is a Linux Distro thing, most programs don't need special permissions as long they are installed in user scope.
So, answering your question, our devs are like regular users: when they need to install something that needs privileges they call IT. Surprisingly, that rarely happens.
You can sorta kinda harden these systems, but that would only work against common malware. And you generally can't isolate senior engineers in their own little DMZ, so any RAT on their machines usually leads to catastrophic consequences.
Especially as a "DevOps engineer", gatekeeping and providing least privilege access is in the job description. I understand getting lazy and relaxing the rules in some contexts but not when running a password manager on this scale, unacceptable.
Production access should only be allowed from a locked down system with no open ports and a very small whitelist set of software. Operations for said system should be simple. Deploy version x with necessary provisioning. Backup system. Restore system. View monitoring and logs.
You must minimize the surface area connected to production.
On the network side the system with the keys for production is also firewalled off from general Internet access. So potential malware can't phone home.
Isn't this like basic stuff where you would use encryption that only the end-user can bypass?
it's because the market doesn't actually pay for a secure product, but the appearance of one.
The end-user buying cannot really discern whether the company's product is actually secure. There's no third-party standard auditing (let's say, a gov't organization).
Banks are run securely, not because I personally audit them, but that the gov't mandates liability onto the banks for losses from their insecure systems. So the banks are secure, because they stand to lose a lot. The same must be mandated for all companies imho, or insecure companies would continue to exist and thrive.
Hard currency theft requires a physical attack and “digital currency” is just essentially a spreadsheet that requires a settlement mechanism such as correspondent banking to work.
Banks transfers are nothing more than messages going between different branches and banks there is nothing being transferred other than orders.
The attack surface on modern banks especially large ones is actually ridiculously small since you don’t only need to defraud or compromise a single bank but also the entire system and all other banks which are using it since once the offended bank notices some inconsistency it can issue a notice to reverse any offending transactions.
Also since bank transfers are often liabilities for most banks e.g. when an account in First Capital transfers $1M to someone in First Direct it means that First Capital now owes First Direct $1M which makes First Direct a creditor which is why it will likely quarantine the funds until the transaction is fully verified and settled and even then there still likely going to be a cooldown period to reduce the risk even further.
Most of the security within banks is designed to deal with internal threats since the entire banking system is essentially based on mutual trust which gives individuals even fairly low ranking branch employees the ability to authorize fairly substantial transactions.
i think you got the cause and effect wrong - banks are run securely because it's made to be very difficult to steal money. And stolen money gets tracked (if you did steal a large amount) by anti-money laundering laws, which makes it hard to spend it.
Why is banks' attack surface small? Why is all these other "systems" in place to make stealing money difficult?
Why isn't the same happening with stolen credentials, or data?
Bank customer data breach is not that rare.
Money in bank is "secure" because they are cross checked with counterparties. Their data leaks like everybody else.
https://www.bbc.com/news/business-64240140.amp
> He said Barclays told him it would do an internal fraud investigation which later resulted in Mr de Simone being held liable for all the losses.
> "They could not identify a point of compromise from the back end - to them it looked like the pin had been entered.
> "The only thing they could suggest was that someone knew the code therefore it's gross negligence on my part apparently.
In the olden days banks used to have decent security. Once you gain access to the account, to pay a new payee you'd need your bank card and pin number to do the 2FA. Now it's all the same phone.
> After eight months of evidence gathering and dealing with the police, an investigator at the Financial Ombudsman Service (FOS) upheld Mr de Simone's complaint against Barclays which now, if it disagrees with this, has the opportunity to ask the ombudsman to examine the case.
They are just trying to wear him down.
But here in true UK where we‘be had realtime, high volume transfers for at least a decade, it’s very possible to steal digital money and move it on before the bank notices.
Faster Payments in the UK are expected to be credited and spendable within 20min, normally it’s spendable within milliseconds. The actual settlements happen every few hours, and all of it is pretty much automated. Additionally because the money can be credited so fast, there is no recourse for a bank to recover a payment they’ve authorised, they have to settle it. The sending to the transfer message is as good as sending the money, once it’s sent, they have to pay, if they refuse, the money is taken from their collateral at the payment network to pay their debt, and they’re disconnected from the payment network.
If your account all of a sudden has 10’s yet alone of 100’s 1000’s or of transfers in it will be quarantined.
HSBC locked my account just last week for 7 or 8 transfers of circa £10 all being made all in the same day from colleagues from my work since we bought a gift for someone that was leaving and settled the payment.
Authorized push payment fraud still does happen but it’s fairly negligible in the big scheme of things.
> HSBC locked my account just last week for 7 or 8 transfers of circa £10 all being made all in the same day from colleagues from my work since we bought a gift for someone that was leaving and settled the payment.
I would be careful assuming that HSBC is representative of how banks in the UK work. They’re well know to be the most trigger happy bank when it comes to account and transfer freezes. HSBC was probably the number one cause of complaints related to transfers while I was working in the fincrime team of a different bank.
Banks run securely because they have serious and mandated change control processes, applied against a formal classification of the importance of systems.
Something you generally won't see outside the T100 of non-bank companies.
This means their IT evolves glacially slow, but it does keep things stable.
(I worked both in banks and at Google.)
However you are right that Google thinks they would lose a lot from being less secure.
Stealing cash from the vault: sure, that's 'real world' security.
Protecting their customers' data: nope, that's all digital.
I tend to disagree. The potential for any single employee to do substantial harm to any business is incredible and designing a system to make that not possible is nigh impossible.
It's neither the humans nor the institutions fault. It's just that systems involving humans are incredibly hard stuff. You are constantly weighing rigidity of the system with the potential to do harm within the system. How much way can it give to make it possible for humans to do their job in a complicated world?
If you go down the path of "we need a process for everything" you are going to end up with a lot of processes. The inherent problem with that approach is that (for most businesses that are not exactly amazon) a lot of processes can not not feasibly be systematically enforced and rely on being honoured by a human a juncture points, to an extent that makes you very uncomfortable when you consider what mechanisms you system has, for when they just don't.
As of now, most system simply rely on humans to do the right thing at the right time, for no other reason than it being the right thing and they also knowing that, for the whole world to not go up in flames.
They are also talking about a home computer. In my company, VPN access is limited to trusted devices; therefore, sensitive systems can only be accessed from a corporate machine.
Security at LastPass seems substandard for a company storing security credentials. Unfortunately, from my experience, this is relatively common, and regulators need to start issuing significant fines or prison sentences for this to improve. Unfortunately, it is too easy for CTO/CISO to find a scapegoat and avoid scrutiny.
Once you get password vault, it's very likely that you also get creds necessary to set up VPN. Besides, there are ways to bypass (poorly implemented) VPN and relying on VPNs isn't even the best practice nowadays.
I agree with you that a few CISOs getting sentences would be the fastest way to raise the bar across the tech sector, but that's never going to happen.
Until somebody pulls out their personal cell phone, and takes a photo of a screen containing highly confidential data, to then send it to someone else, because, dang it, they had to get something done NOW and it seemed very convenient.
Same with the Canadian ban on Tik Tok, why are they even allowed to have phones where they can install any software on?
BYOD and personalization is fine to a point, but only if you don't have high level access.
This isn't unheard of in the industry, Engineers using BYOD devices or similar to work from home. But with a company with a risk profile as high as LastPass this seems _incredibly dumb_.
You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!).
E.g., endpoint management, AV/detection, FIDO2 with Yubikeys to access AWS as an admin and otherwise. Just a few off the top of my head.
I'm surprised that LastPass's policies aren't at least that strict.
My company has what I think is a big hole in this policy in that we're allowed to use our own phone for email, a few corporate apps (like Jira) and our corporate password manager (not LastPass), but IT doesn't do any management of phones (other than being able to wipe them remotely if you're connected to the company email server). I suspect that the company doesn't want to spend the money on giving everyone a managed phone.
Waiting for the rebrand and the incoming lawsuits.
The question for future trust is: what's been / being done to prevent the same thing from happening again due to another single engineer?
https://support.lastpass.com/help/what-have-we-done-to-ensur...
TLDR;not much
Closing the barn door doesn't guarantee that the horses can't escape, but when you don't even have a barn door, it's hard to blame the stable boy when the horses get out.
It’s also possible that the stuff mobile devices can access are walled off from the internal network with a DMZ or firewall.
I assume your IT just has a whitelist for some stuff but I can’t imagine actually being a developed without super user privileges. Unless your doing some sort of very controlled software development.
You mean one of those company devices that is so locked down that they are close to impossible to work on?
Like if you need to install a new (part of) a toolchain, you need to go through IT which takes between 3 weeks and 6 months?
But your deadline is next week and your manager doesn't care about your IT "excuse", so just use notepad and CLI (or your own PC where you can do in 1 day what would otherwise take you months).
Those devices... yeah...
We also have two PC's per desk(plus consoles) so that our clients software isn't exposed.
I don't get how a company like LastPass didn't have basic security like this when a FQA with a whole bunch of semi literate (in IT security) has a ssytem setup.
The user had a keylogger installed on their machine, so the attacker could collect the master password, but how did they login to the vault on a new machine without MFA?
Did they get the MFA seed and login on a different machine, and nobody received a "You're using LastPass on a new machine, if this wasn't you..." message?
Did they exfil all that data to the compromised machine first, then out?
Unless I'm missing something, this doesn't seem right.
The 2FA part in the password managers (and least in the major players currently) is to get a copy of the encrypted vault from the server. The user did that part, and the encrypted vault was not easily accessible locally.
Even if the 2FA was used for decryption, it wouldn't really make you much safer, because malware can steal the decrypted vault out of memory right after you type in the 2FA. A HSM would solve this, as long as the HSM has some out of band way to communicate with the user, such as an approval button that malware can't press and a screen saying what password to release.
If the second factor is stripped for some arbitrary time, you don't have 2FA anymore. Your argument that "any adversary can read the vault from memory" is a weak one, we might as well not have passwords with that attitude.
The point of a second factor is that BOTH need to be present to get to the secrets. If one of those factors is stripped away for "convenience" we're misunderstanding the point of 2FA entirely. I can't make this any clearer.
It all comes down to threat model. If you don't have malware on your machine, 2FA and passwords are quite useful. If you do have malware on your machine, they're basically useless. This is basically the same for any service. Name one website or program that's safe even if you have malware on your machine.
>If one of those factors is stripped away for "convenience" we're misunderstanding the point of 2FA entirely. I can't make this any clearer.
It's not for convenience. It's because there's no practical way to implement encryption/decryption with 2FA. You seem to think there's some practical way to do it, but there isn't.
Lastpass 2FA protects against the threat model of an attacker who has stolen your password. In that case, the attacker cannot steal the contents of your database because the attacker can't get any form of the database, encrypted or decrypted due to not having the 2FA. Unfortunately now that an attacker has stolen all the encrypted databases by compromising Lastpass itself, this threat model is no longer realistic against this one specific attacker or any attackers that this attacker shares the loot with, because they now all have your encrypted database.
It is an implementation detail of the password manager itself. Any password manager can update their implementation to ensure the second factor is always needed when decrypting the vault. I'm not sure why you think this is an impossible feat. It's a choice that can be made.
The only thing I know that does encryption with 2FA is https://keepass.info/plugins.html#otpkeyprov . But I highly doubt it has much usage. It's going to be annoying typing in a 2FA every time you decrypt your password database (I decrypt my password database maybe 10 times per day). More concerningly, if you press the button on your 2FA device (this is HOTP, which requires you to press a button to get a new code) too many times, or typo the 2FA too many times, you can permanently lose access to your database because the HOTP device will advance past the point that the database supports.
So yes, it's a choice that can be made, but it has very major downsides.
And its not so much about protecting me, but to protect company interest right?
https://keepass.info/plugins.html#otpkeyprov
But this is beside the point I was making. My point was that even if the the database was encrypted with 2FA, right after you enter your 2FA malware can steal the decrypted database out of memory.
> right after you enter your 2FA malware can steal the decrypted database out of memory.
There's probably a creative protection here where each key is encrypted individually, but you'd still need some solution like the above HOTP trick or the attacker could scrape the key information out of memory, then decrypt each entry individually.
It seems to me like that would require a separate HOTP device for each password database entry, otherwise the malware can steal one HOTP token and use it against a different entry.
That would be a paranoid level of implementation. As it sits, the HOTP device is only _sometimes_ needed depending on the caching policy. Fix that broken implementation first, then we can figure out how to update the threat model to account for an adversary that has already infected your computer.
I don't understand what you mean. Are you talking about https://keepass.info/plugins.html#otpkeyprov or are you talking about LastPass? LastPass doesn't support HOTP AFAIK. HOTP isn't a very good form of 2FA (it's phishable, sometimes inconvenient, and it can become desynced), U2F is much better, but you can't encrypt a database with U2F.
KeepPass has a very customizable policy of when to lock the database. I have KeePass on my desktop set to lock if KeePass is inactive for 1 hour, or if my computer is inactive for 10 minutes, or if I lock my screen. Are you saying there should be a semi-locked state that requires a password but not a 2FA? Sure that's possible.
None of this protects you from malware on your computer though, so I don't know why we're talking about it.
I was trying imply that you need 2fa for the services itself so the passwort alone is useless.
Ironically office 365 has a nice implementation, where it for instance requires all admins to use MFA
This is my point, it is an implementation detail of the password manager to integrate OTP or another second with decryption of the vault. Any password manager can implement this.
From the perspective of the user, you are stripping a factor for some arbitrary period of time. It's a broken implementation.
I'm not sure why you are having such a hard time understanding this is an implementation detail of the password manager that can change at any time. You are treating this like it cannot be implemented differently. It absolutely can.
I'm using offline keypass. It's great.
A Hardware Security Module would avoid exfiltration of secrets.
(Off-line AND off-device)
From what I can tell, all of lastpass mfa options are based around some form of otp not webauthn.
We tested the above in our own environment, since we had control of the devices we did not need urls to do it. We just grabbed the data locally to confirm if it was true. At the time lastpass told us webauthn was in the pipeline so we stayed.
As far as I know, no popular password manager seriously includes "fully compromised local device" in their threat model. I don't think it can be done without hurting seriously usability (like having one 2fa verification each time you use a credential would work) and the predictable outcome of hurting usability too much is that people will find more handy insecure ways to store their passwords.
I wouldn’t mind tapping a YubiKey or my MacBook‘s Touch ID every time a password is accessed from the vault. That’s essentially how ssh keys work with smartcards or security keys as a second factor.
Also, as an aside. While correctly implemented Passkeys (without fallback auth methods) would make my life as a red teamer much harder, that would have only prevented this attack if the infected machine was engineer's private PC where they used corporate LastPass account and nothing else from their work. If the machine that's used for DevOps work gets infected, that's still and endgame because you're generating all sessions I need during your regular workday, so I don't really need the passwords / decrypted vault.
I'm interested in the technical idea here. You have a set of credentials encrypted with AES. So each vault item is encrypted with a symmetric key. How would you build a system to generate those keys using a rotating 2FA that isn't reversible by an attacker that can watch the entire process and can fake the timestamp or other elements on the computing device as needed?
I can see that you have code that enforces 2FA on each vault access, but that code is run on the local system, so it can be trivially bypassed by an attacker with root.
The product I'm working on have had for a long time an option to require a second factor for each login which works a bit as at described (encrypted data are stored locally but also encrypted with a key that's stored on our servers and protected by 2fa), but at the vault level, rather than at the credential level (it doesn't protect against device compromise, but prevents brute forcing of local data for exemple. You do lose offline access) and the UX is already annoying enough that in practice this feature is very rarely used and we are regularly considering dropping it.
I hope you find a product that works for you!
At Distrust, my security consulting firm, we train all our clients to build production systems that require a minimum of two engineers to mutate and that only pristine operating systems access production that have only signed reproducible used packages that have never been used for anything else.
Production environments need to be managed like careful methodical clean-room labs with strict accountability. Instead they are managed like collaborative art projects where everyone is trusted and nothing bad can happen.
The breach was preceded a couple of days by Plex corporate breach which devulged the engineer's credentials and home IP address.
This would have allowed the attackers to access Plex sever remotely, after which the source claims they used an RCE to install a keylogger (and probably a back door) on the engineer's PC.
The concerning part is that according to Plex devs (in their reddit sub), they have NO KNOWLEDGE of any RCE. They also haven't communicated with Lastpass, and no one reached out to them.
So if there's a Plex remote code exploit - it is still unpatched and actively being exploited - 8 months later!
Given that there is still no information on this Plex RCE, we should not assume that it requires authentication to function. So if you're using Plex, make sure to turn off public accessibility asap!
I think it's pretty dubious that they could pin it on a plex media server breach though. This is from the same company that sells logmein and goto; I haven't heard definitively that those pieces of software weren't installed on there.. Unless this machine was used for Plex and the occasional log in to work from home type thing, I doubt you could do the forensics on it with much reliability unless you got to it within days of the attack. If this is a normal devops guy? He's got 3 different chat apps, probably Steam, who knows what pirated crap is on there... Plex seems like a very convenient target; the plex corporate attack seems very very plausible as giving the attacker information though.
DevOps itself is a huge (and thin) attack surface. This is a feature of software as a centralized business venture.
If your company is storing user data, then it must be someone's job to do that. They need administrative server access to do that job.
The important takeaway is that the data itself can still be safe. So long as the company did not have a way to decrypt it, that data can rest anywhere. Guaranteeing that reality, however, is very difficult for a business - that is expected to be private about their implementations - to prove to its customers.
When this beach happened, LastPass should have focused on telling their users to never reuse the master password that they had set at that time. That's the biggest vulnerability: the content of their vaults (as copied by the attacker) was, and is, still kept behind that password. The need for each user to keep that specific password secret is the main effect of this situation.
This is a great reminder that you can't trust anyone to keep your data private. You can only trust math.
I'm listed as a janitor of where I work. Only those that know me, know what I really do.
I've tried to sell the policy forbidding employees from listing their positions or where they work on linkedin, each time management frowns and says no. One day they'll come around...
Forbidding employees from listing their role on LinkedIn would put them at a major disadvantage in job searching and recruiting.
Forcing employees to hide their role is unreasonable. The company doesn’t own the employee.
If you work at an organisation like LastPass in a privileged position, then you need to be aware that you are an enormous target. And it's not just your own or the companies security you potentially compromise, but millions of others arguably most sensitive information.
In Australia, if you have a security defence clearance, you are not allowed to display that in your social media networks (e.g. Linkedin), despite that potentially being important to your other job prospects in such industry. For those exact reasons.
If your LinkedIn said you were a DevOps engineer at LastPass, you know for sure that they're a prime target.
I'm not arguing the legality of it, just the problem it poses if you don't. Perhaps the solution is to tighten who can see your position and you diligently only connect with people you absolutely know and not have connections of connections on.
Linkedin doesn't even matter since you can buy the data any way. Email Signatures are mined to get role and contact info for databrokers. Anytime you email outside of the org, the CRM software could be grabbing that info and feeding back to a databroker. Not to mention people using addons to their mail clients. Why I don't use one at my current company even though it is company policy to have one from our HR department. Thats just email. There's also the credit report data that has your role on credit applications and also when you donate money to politicians/PACs that makes you list your role for compliance reasons.
My point being is that there are details you aren't allowed to put on your social media accounts, for the reasons we're debating.
*Please don’t pay this mob money, they are rent seeking, bottom feeding scum.
>Labor Code section 232.5 prohibits an employer from discharging or retaliating against an employee who discusses or discloses information about the employer’s working conditions.
https://www.dir.ca.gov/dlse/howtofilelinkcodesections.htm
I'm not sure if it applies, but I could see why lawyers might be nervous about forbidding employees from saying their role.
No employer may do any of the following:
(a) Require, as a condition of employment, that an employee refrain from disclosing information about the employer's working conditions.
(b) Require an employee to sign a waiver or other document that purports to deny the employee the right to disclose information about the employer's working conditions.
(c) Discharge, formally discipline, or otherwise discriminate against an employee who discloses information about the employer's working conditions.
(d) This section is not intended to permit an employee to disclose proprietary information, trade secret information, or information that is otherwise subject to a legal privilege without the consent of his or her employer.
Why would people wanna work somewhere you can't speak about their job role? I think that'd filter out tons of applicants.
Edit: Also wanted to mention 3 out of 4 incidents I am involved in is related to insider threat.
Do you get a lot of janitorial service headhunting spam?
>In December, we notified a subset of customers whose SCIM, Enterprise API, and SAML keys were stored in unencrypted form. This only affected customers who joined LastPass and used these services in 2019 or before.
This part just blew my mind.
>Important: Since resetting MFA shared secrets destroys all LastPass sessions and trusted devices for these users, these users will need to log back in, go through location verification, and re-enable their respective MFA apps to continue using the service.
I feel sorry for everyones internal helpdesk. This is going to be brutal.
This comes off as spin to me by LastPass or LogMeIn’s PR department. Even if this was the case, how is it possible for intrusion detection systems to not observe and report abnormally high egress traffic? Downloading every vault should be a rather noticeable event.
We installed security product X, job done, walk away happy![0]
Anyway, this company has had incident after incident. This will keep happening every few years for them like it has for the past 10. As will lack of transparency/ outright lying.
Some commenters are saying they wish the company the best. I don't. Use something else. LastPass needs to die.
So what title do you need to have to practice DevOps?
It's not a job title, it's an engineering practice. People who participate in DevOps include "software engineer", "network engineer", "IT operations engineer", "platform engineer", "cloud engineer", "quality engineer", "security engineer", "manager", etc.
I'm also one of the people who does "operations" but has a DevOps job title. I've grown to accept this as a second definition, it's quite common now.
"OPERATIONS ENGINEER" might work, but it raises another set of problems, e.g. does it imply operational responsibility (on-duty) work, which I don't think is a given in DevOps jobs nowadays.
There are many options that don't tack "dev" into your non-dev job titles.
If its a matter of gatekeeping the "developer" status, go read some actual job posts with the title DevOps in them. Its not uncommon to come across proficiency requirements in at least one programming language (e.g. Go/Python/Rust), used in automation libraries, cli tools or whatever, which the applicants are expected to "develop". Or is it just constructions that can be developed?
This is not just to do with LastPass. It doesn't make sense why one would put their personal, most valuable passwords in the hand of another party. Of course, it's handy on a team between people.
When it comes to our bank accounts, do we trust someone like that with the pin numbers to our credit and bank cards?
One positive that comes out of this is that it raises awareness of thinking about the difference between security and convenience to manage one's own passwords.
One solution? Locating a zero, or no knowledge file storage system which is encrypted at rest and transit and placing your own files on it is the first start.
Spideroak used to fill that slot nicely (not sure if it's available anymore), and others I have heard about are sync.com and syncthing which do this just fine. Are there any other solutions that would be reasonably teachable to the average user?
I have seen people create multiple files on a share that works fine. I wasn't too keen on it at first but it did seem to work OK.
The other problem is that there is stuff you want for yourself, stuff shared with your partner, stuff shared with all the family (children included) and stuff shared 1-on-1 with each child.
That gets messy quickly.
I am glad I went through all the pain.
So if you're running a Plex server, you should disable public access immediately.
I hope this involved something along the lines of: "This zoom meeting won't end until you update your router firmware".
>> the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity.
How does A => B?
>> The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.
The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine?
I understand it must be hard to try and come clean in a RCA without injecting some excuses and mitigating factors but you can't attempt soften the blow or the entire thing is a big, smelly mess. It should be a bunch of facts with no emotion, THEN the mitigation and lessons learned. LP just issued a bunch of press releases.
I think the idea here is that we want the cryptography operations to happen entirely locally, so that LastPass doesn't have any access to them. However, if you do that, someone with root on that system and the Master Password can replicate the operations the local system does on the vault. I'm not aware of any symmetric-encryption algorithm that includes a time-based un-replayable TOTP or HOTP in the key-generation process.
DevOps guy here is just running a Plex server and probably pirating porn on his work laptop because why not at that point. Can’t even bother updating this which Plex makes literally about as easy as they possibly could. It’s a one click fully automated process with zero interruption. Was he using internet explorer too?
Or was it his work laptop or did they just let him log in to work on any random computer?
Did he actually get any work done? I want to hear an interview from this guy. How did he get hired as DevOps while avoiding all the basics of fundamental basic work practices?
Kubernetes does MFA, all the Clouds do MFA and the company you work for can afford a "cheap android phone as key".
No matter if bare-metal, cloud or managed. If you habe ANY edit rights you need MFA.
In the kubernetes world it really is not so difficult
One key thing however is I'm not seeing Bitwarden in this list, big ups to them.
For myself, I just use google's password manager for non-critical passwords, and I use a single password for all my banking. I feel that is safer than using a password manager.
I have some AWS keys in some files that are used by terraform/packer. A hacker could easily get them.
Some other AWS keys are stored in the CI system and provided as env variables. Someone that can merge/push to the specified branches can just change the CI script an exfiltrate them.
How can I fix that?
I would need some MFA for both cases. I would imagine it would be a good idea that I have to confirm each action on MFA device, which will then generate temporary tokens that are invalid after a few minutes. I locked into some solutions like Hashicorp Vault but I was not able to build something in a short time. New features were always more important.
How do you do it?
You use aws-vault(https://github.com/99designs/aws-vault) and configure it with IAM and MFA with YubiKeys. You configure e.g. the profile jonsmith.
When you run
aws-vault exec jonsmith -- aws s3 ls
it will ask you, e.g. every hour to confirm with YubiKeys and cache the key for one hour. After that the temporary keys expire. Can you also store keys different from AWS?
One benefit is that it made me audit my online accounts -- I removed many.
So: Be prepared to switch if you want this type of service. New normal.
So LastPass, a security company, has people who are tasked with both delivering features as fast as possible using a variety of development tools (dev) and administer their production systems where critical data lies (ops). On the same machine, from home.
It seems that this breach is not really an accident and more a logical conclusion.
Ie. they would be a 'dumb' storage system for the customers encrypted data. The data would be encrypted by the customer before upload. And then decrypted again by the customer after download.
Users would be identified by a unique random ID, and users would auth by signing a challenge with a secret key known only to the customer.
That way, even if a bad guy worked for lastpass and had full admin access to all servers, they couldn't steal anything.
And, in fact, if this system was properly designed, it could all run with opensource server code, and the datastore fully open for anyone to inspect, to prove that the security is down to cryptography rather than trusted yet fallible humans.
Because the attacker did not need a separate __physical__ MFA (2nd F) after collecting the master password (1st F).
>Backup of LastPass MFA/Federation Database – contained copies of LastPass Authenticator seeds, telephone numbers used for the MFA backup option (if enabled), as well as a split knowledge component (the K2 “key”) used for LastPass federation (if enabled). This database was encrypted, but the separately-stored decryption key was included in the secrets stolen by the threat actor during the second incident.
Unless I am misunderstanding this, they mention to business users the need to reset shared secrets from OTP providers.
>For users of Duo Security, Symantec VIP, RSA SecurID, or SecureAuth, regenerate the shared secret for each respective MFA solution and paste the new shared secret into the respective MFA app configuration in the Admin Console.
A single engineer had both access to the prod database AND the data decryption values?
they know what bank you use, which adult sites you visit, what medical conditions you might have
I hear lastpass is pretty good, if he needs recommendations.
I think generally people consider it low risk, because "who cares if someone can see what I was watching", but if you can get RCE on a computer running Plex Media Server just by logging in to the Plex account, then of course that's a huge security surface and your Plex login should be a secure randomly generated password and have 2FA enabled.
Of course, this can be avoided by paying for the premium Plex subscription which allows you to create accounts for other 'non-admin' users, but for people trying to avoid spending money (a large number of the Plex user base probably), then anyone with the login can change settings/administer the server.