Updated Okta Statement on Lapsus$
okta.com
okta.com
The reason I think that is because if they really had keys for production machines it seems very unlikely they wouldn't have used them to produce some more damaging collateral than they've presented.
We get alerts when folks are sharing tutorial code with fake keys like DEADBEEF in them. It's nice to know DPL works, if you use it.
#Dick
#Harry
Of course, you still wouldn't have 8500 channels. That's a lot of incidents.
Once.
Thanks for bringing back the nightmares, friend.
Somewhere, there’s some project manager that says “Wow, I bet spooky23 would love to know about my spreadsheet sorting project”.
Much like the OP here I'm in hundreds of teams, and almost all of them do all their talking in #general and they wonder why people never respond to notifications.
The Teams UX and general paradigm is awful. Right now I'm in 3 group chats and two channels (in two Teams) discussing the Okta incident. Huge overlap of them, but not 100% so some people aren't getting all the information.
The context switch between a mostly useless teams tab and a needlessly full screen IM window is too heavy for me.
> Security Standards. Okta's ISMP includes adherance to and regular testing of the key controls, systems and procedures of its ISMP to validate that they are properly implemented and effective in addressing the threats and risks identified. Such testing includes:
> a) Internal risk assessments;
> b) ISO 27001, 27002, 27017 and 27018 certifications;
> c) NIST guidance; and
> d) SOC2 Type II (or successor standard) audits annually performed by accredited third-party auditors ("Audit Report").
I don't think storing AWS keys within Slack would comply to any of these standards?
They are not effective security controls and never will be and should never be a measure of that.
(Upon further review, it appears to be the more UK way of saying it! Ha!)
I’ve also been closely monitoring the responses from our CTO and VP of Security when someone from our DevOps team posted a link to the Verge article in slack this morning.
Which brings me to this inquiry: How are your orgs responding to this? We have a dependency on an Okta-like provider and my first thought when reading this news was “you know, wonder if we should give our shit a sanity check”, and someone beat me to this, proposed it in slack but the idea was turned down by our SecOps team.
More reasons to look elsewhere.
https://auth0.com is the "still cares about customers" vendor
I'm not affiliated with them, just traumatized by working in IT
> There is no impact to Auth0 customers, and there is no impact to HIPAA and FedRAMP customers.
Interesting. Does this agreement also works the other way as well (Okta can't just decides to terminate your account no matter the reason)?
There are no winners.
I’d look at stuff like FedRAMP as a starting point for the control environment and explore further.
The decision makers have absolutely no idea how any of this stuff works.
Mostly picking apart logical inconsistencies in the language and / or re-emphasising the already disclosed info. Does not seem like they were able to produce any hard collateral to contradict anything Okta stated which probably means it's at least ball park accurate.
https://de.catbox.moe/ovt7t7.jpeg
I remember healing a while ago that certain Telegram channels would be blocked on iOS devices due to Apple's content policy, that's what this seems to be about. Update: Yeah I can view them on desktop just fine.
Yeah, this is the fabled content moderation block for App Store distributed apps. The Mac App Store version of Telegram also blocks it, but the direct download dmg does not. I think the direct download apps are also updated more frequently, but I could be wrong on this.
If I steal your secure token and log into you through a cloned browser session and access your gmail have I compromised your gmail? It feels like it.
Maybe it is just a distinction without a difference.
The access is transient and you remain in ownership over the credentials and account, because the credentials were not compromised (just the programs/browsers with pre-existing auth). Though with physical access it's probably only a mater of time before local admin passwords are brute forced and access to keychain/browser saved logins is inevitable?
> If I steal your secure token
It's more like you just logged in using your secure token then someone stole the device and took advantage of that. I think we can all agree there's a difference between having access to the laptop and access to the account without the laptop.
There is a difference, but that difference is not the one you seem to be implying. An account can be compromised even it it hasn't been fully and permanently taken over. The temporariness of an account being compromised does not mean the account was not compromised.
You can make a distinction and clarify that the account was compromised but that the account credentials were not. However if you extend that to saying that the account was not compromised when attackers did have temporary access, then you are simply lying.
1. some kind of remote access to the support engineer's session on that laptop
2. physical access, no login
3. physical access as a different user
4. physical access, logged in as the support engineer
No full disk encryption or alike? Physical access should not be enough to access sensitive data, unless you have data unencrypted.
Now, JUST this week (presumably, in the last 72 hours to explain why disclosures have not already been sent) "we received a report from the forensics firm this week. The report highlighted that there was a five-day window of time between January 16-21, 2022, where an attacker had access to a support engineer’s laptop."
Okta writes "We are deeply committed to transparency" but shows none in their blogpost in contrast to CloudFlare, that doesn't write anything about transparency but displays a lot of it.
A bit like queuing for your internet providers customer support for 3 hour and hearing "We care about customer satisfaction" every five minutes.
Anyway, I likely misinterpreted yesterday’s comment..
https://twitter.com/eastdakota/status/1506148082194386949
I believe @eastdakota is saying that Cloudflare has their own homegrown SSO internally, but they do not make that available externally to their customers (it's not a public product that one can buy). I don't think that the tweet was saying that they use Okta externally.
I'd love to see more of this in the industry. CF is, IMO, industry leading here.
Especially considering their core proxying service doesn't have any requirement for a globally consistent datastore, there is zero excuse for a global outage.
How do you think they manage configuration for proxying millions of customer websites? Using a globally distributed data store.
Eg.
> Cloudflare reads the system Okta logs every five minutes and stores these in our SIEM so that if we were to experience an incident such as this one, we can look back further than the 90 days provided in the Okta dashboard. Some event types within Okta that we searched for are: user.account.reset_password, user.mfa.factor.update, system.mfa.factor.deactivate, user.mfa.attempt_bypass, and user.session.impersonation.initiate. It’s unclear from communications we’ve received from Okta so far who we would expect the System Log Actor to be from the compromise of an Okta support employee.
As far as I can see, there's a lot of cloudflare accounts visible in the screenshots shared by the group. Stuff like cloudflaretv1, etc..
Very ambiguous statement, not really fitting in with the whole "deeply committed to transparency" image they are trying to emit.
What does "facilitate" really refer to here? If it was just triggering it, they would have said so, presumably. And why is only passwords mentioned as what couldn't be obtained and not the tokens from MFA as well, does that mean they could obtain those tokens?
I wonder how it fits in with the groups own statements that they still have active access. Gonna be interesting to see what Lapsus$ replies to this statement.
The idea that support can just trigger a reset email makes little sense. Perhaps Okta has some complex mechanisms that I am not aware of, but if this was any system I’ve ever worked on, an employee could take over an account if they so desired.
My expectation (and experience of other similar systems) was that Okta would not allow password resets by anyone but the organization administrators. However, that doesn’t appear to match up with what has been disclosed here.
When organizations make the on-prem to cloud jumps they frequently are trading off oversight and experience. Many tenured internal teams have been broken apart by these types of migrations because they are ultimately sold to management as cost saving endeavors. These folks would make your observation about organizations admins controlling resets, but they are gone.
And why use the more general word of "facilitate" when they could have been specific and say "trigger reset password flow" or similar.
Hence their statement is ambiguous.
Regardless, "reset" can mean different things and it definitely seems like they're being cagey here by intentionally using imprecise language.
Ex - You have gmail gated behind okta using MFA, and the lost device is the user's MFA (ex - phone using TOTP).
If the user has no session currently logged in on another device, they won't be able to create a session without their MFA device, and therefore won't be able to access their email.
How likely you are to hit this depends on org settings, like gmail session time, okta session time, mfa requirements, etc.
Ideally - they'd have backup codes somewhere... but most users won't (or they'll do things like store them in their email...)
(And if I lock myself out, make it very very expensive for me to prove my identity to recover access)
Password reset is the soft underbelly of most companies.
It looked kinda successful though...
Yeah, it was.
I wouldn't have realized that unless I'd read far enough to see that Entity A did compromise Account B. Since the author didn't define what an attempt in this context means, I think it would be proper to assume that attempt meant the overall effort to compromise the account.
- unsuccessful login occurs
- successful login + screenshot + various nefarious actions
- (some time later)
- attacker's access locked down
Implying that they detected an unsuccessful login, but doesn't mean that they didn't prevent unauthorized access, which would make what they said technically correct but kinda useless for us users. That is something I did not originally consider, but wow ... maybe. I hope not. I mainly want answers about what is affected, and whether Auth0 was compromised.
"a customer support engineer working for a third-party provider"
actually means
"one of our customer support engineers, who happens to be an employee of another company, working for us under contract".
> Where that other processor fails to fulfil its data protection obligations, the initial processor shall remain fully liable to the controller for the performance of that other processor’s obligations.
GDPR Art 28(4): https://gdpr-info.eu/art-28-gdpr/
Oh okay then, pack up guys, everything’s fine
I really hate this kind of corporate bullshit
> There are no corrective actions that need to be taken by our customers.
Isn’t this an objectively false statement?
IMHO it's more of an empty statement. They never pin down what end-state is being discussed. So we can't evaluate if "additional steps are needed" to achieve that (unspecified) state.
It could be that the speaker is intentionally equivocating. I.e., knowing that the audience will assume some particular end-state. But if cornered, the speaker can claim (lie) that he implicitly meant a different end state.
despite an overwhelming preponderance of damning evidence from twitter (as well as the hacker themselves) you've somehow managed to find yourselves secure instead?
Christs whiskers thats some impressive doublethink. Its also an excellent opportunity to fall on a sword that gives future attackers --hat colour dismissed-- an immediate incentive to simply publish regardless as you dont appear to be acting with very much good faith. if this sort of an attack is a carrot, you've clearly shown a predilection for the stick.
I'd have far more faith in them if they were transparent about what had happened, what they're doing about it, and how they will make sure it can't happen again.
Instead, they are being weasels - I for one, will not be using their services again, and this behaviour is the reason why.
Here's another example from a couple of years back where they used some really weasely language to claim they weren't vulnerable to a CVE (spoiler: they were):
https://twitter.com/jonoberheide/status/1506280347306188805?...
There are pros and cons both ways. You're centralizing the risk, but also centralizing the effort for quality. Perverse incentives however causes loss of quality: saying you fucked up will lose you customers, and some of that sweet, sweet stock value.
The fact that you can work for these companies without security clearance seems a bit insane. The more they grow and centralize, the more of a national security risk they become. It's not hard to get hired at Okta, and when you're in, you're in. At what point is not solving a security bug at Okta that you've discovered and selling it for monero more lucrative than actually fixing the bug? I realize that not everyone is like me and has a strong values that prevents them from doing so, and that scares me. If you're talented, it's super easy to backdoor a system and get around code-review
I'm torn
Workers at organizations get compromised all the time. This doesn’t mean their systems/products are compromised.
Okta is not just a bunch of software, it's also staff and processes, and the result is a trusted service they provide to customers. If that service is compromised, it doesn't really seem to matter how?
I hear what you're saying, but the how does really matter, and will change how customers perceive the issue and make decisions about how to react.
e.g. "databases were open to the Internet and all data has been siphoned" lands quite differently than "a staff member abused their privileges but the scope of abuse was limited to xyz".
If I'm a customer, it tells me a lot about what Okta needs to do next, and how much I should freak out right now. It's still extremely problematic that a staff member (1st or 3rd party) could abuse such privileges, and I immediately have questions about how those privileges were abused and to what actual effect, but it's a fundamentally different problem than other types of breaches.
If I was a bank and claimed that I haven't been robbed, an insider just transferred billions of pounds out of the bank and then fled, I think everyone would rightly say "What are you talking about, you have been robbed!"
It doesn't matter if it was done by a guy in a black and white stripey t-shirt, or if it was done by a rogue internal employee, a bank robbery is a bank robbery.
In fact, the ability of an internal staff member to transfer lots of money out of the bank probably signifies a more significant and systemic issue - particularly if i've lost my money and the bank refuses to acknowledge they have been breached/robbed (it was just an internal rogue staff member, not a robbery! our security hasn't been breached!).
A bit of a stretched analogy - but i'm sure everyone gets the point. Security isn't just about technical security - it's the whole process involved in making sure these things don't happen. A banks 'technical' security might be great, but the bank would still be considered horribly insecure if a staff member can transfer any money out of an account. Equally an auth service might be 'technically' secure, but the ability of a single rogue staff member to impose a lot of damage suggests more systemic issues.
I have to respectfully disagree.
Yes, the end result may be the same, but even in a bank robbery, the how matters, and will drive different behaviors from everyone involved: the bank, law enforcement, and customers of that bank.
If as a customer, I learn that a guy in a stripey t-shirt holds a teller at gunpoint, my conclusion goes something like "that's a terrifying situation for the teller, and I hope they're ok". I'm probably not going to stop using that bank.
If on the other hand, I learn that there are systemic issues with bank security, and internal employees have been embezzling funds somehow, I'm probably going to think hard about whether this is a bank I want to do business with.
> Security isn't just about technical security - it's the whole process involved in making sure these things don't happen.
Yes, and when factors are involved that are out of the bank's control (e.g. a crazy person walks in with a gun), it might be fair to ask why the guy got inside to begin with, but the conclusions you draw about such an incident are far different than the conclusions you'd draw if internal employees were involved.
In case this wasn't clear from my earlier comment, I didn't mean to imply that an internal process issue makes any of this ok. But it does make it different than other types of breaches.
Bottom line: the how still matters, not because one type of problem is ok and the other isn't, but because the actions a customer should take / consider will be different depending on how the breach happened.
The how matters, I agree, but it doesn’t change the fact that a breach is a breach.
Different breaches can have different severities, but they are still both breaches.
My issue isn’t on the severity of the breach, it’s that Okta are saying there wasn’t a breach.
IMO, support agents also should not have the ability to view or access a customer's account without some form of time limited, auto-resetting-to-opted-out default confirmation that support can view the account from an existing logged in admin.
We should not glorify them.
This means they could have reset anybody’s credentials and logged in. There would a record of it if the audit logs are valid, but saying no action is needed may be a stretch.
Does it? It specifically says "but are unable to obtain those passwords," which reads to me like they are able to trigger a password reset email to the user, but are not actually able to set the password themselves.
It seems pertinent a support engineer could trigger a password reset if they were worried a password had been compromised for a user.
If you change a password, then you didn't "obtain" the previous password. Weasel words but there you go.
It still sounds to me like they're saying that they could set the password on an account to anything they wanted, but not retrieve an existing password.
This would be a big red flag to anyone who's observant and realizes that their password had been changed, but plenty of people would probably simply take it in stride and reset their password thinking they'd forgotten it. Either way the horse is out of the barn.
Anyway I agree it would be bad practice but that doesn't mean it's not done.
Really? That's insane. I'd like to hear more about that.
I have no idea what an Okta support engineer can or can't do but I know that a regular account administrator can set an arbitrary password for a given user via the Okta UI.
Then those users are hosed anyway? They could always trigger a normal user-facing password reset email without any access to support systems.
that and maybe going through JIRA and seeing if anything was of any interest?
It's pretty clear there is no super admin breach here. None of the screenshots show an admin interface - the app portal shot clearly shows a distinct lack of the "Admin" button. No user directory shots. There is a single image of a password reset confirmation, but those are also in documentation so it's hardly a smoking gun.
It seems far more likely that a support engineer had their laptop compromised and their limited access was used to try and make a mountain out of a molehill. Not like a ransomware group would have reason to over-exaggerate their access and ability, right?
Off the top of my head: direct database access, http server - could push compromised pages to all Okta users
The AWS keys really could be the keys to the whole proverbial kingdom.
The rest of the note literally show they have been breached. What?
I wonder how many passwords and other creds were harvested from tickets alone.
Lapsus$ went after Okta's customers and in some cases successfully, it appears.
Where does it appear like that? I have tried to follow this but the only thing I have seen shares is info from within Okta. I have seen email addresses of CloudFlare employee in screenshot of Okta, is that what you are referring to?
https://www.vice.com/en/article/y3vk9x/microsoft-hacked-laps...
Everyone's terrible with secops, but Okta employees were apparently dumping AWS keys into public channels.
New Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30774193 - March 2022 (24 comments)
Also:
DEV-0537 (LAPSUS$) Criminal actor targeting organizations - https://news.ycombinator.com/item?id=30774406 - March 2022 (0 comments)
Lapsus$ hackers leak 37GB of Microsoft's alleged source code - https://news.ycombinator.com/item?id=30763623 - March 2022 (117 comments)
In this case it reads as though Okta are obfuscating the truth, and that's not good.
There it is.
> Support engineers are also able to facilitate the resetting of passwords and multi-factor authentication factors for users, but are unable to obtain those passwords.
Those two things are at odds with each other, no?
[0] https://www.csoonline.com/article/3389138/how-onelogin-respo... (unsurprisingly the canonical article from their weblog is missing now)
Against "Someone that what has to lose at this point?"
What to lose? Their reputation?
Both sides have incentive to stretch the facts. But Okta has more accountability since if an Okta customer comes forward and says "We had credentials of several of our users maliciously reset during that time period and have the logs to prove it", then Okta is going to have a hard time of it.
If Okta comes up with proof that Lapsus didn't have the access they said they did, Lapsus is not going to have many "customers" complaining "you didn't crime that other company as much as you said you did"
I do enjoy the lies given by Okta.
1. We didn't compromise any laptop? It was a thin client.
2. "Okta detected an unsuccessful attempt to compromise the account of a customer support engineer working for a third-party provider." - I'm STILL unsure how its a unsuccessful attempt? Logged in to superuser portal with the ability to reset the Password and MFA of ~95% of clients isn't successful?
4. For a company that supports Zero-Trust. Support Engineers seem to have excessive access to Slack? 8.6k channels? (You may want to search AKIA* on your Slack, rather a bad security practice to store AWS keys in Slack channels )
5. Support engineers are also able to facilitate the resetting of passwords and MFA factors for users, but are unable to obtain those passwords. - Uhm? I hope no-one can read passwords? not just support engineers, LOL. - are you implying passwords are stored in plaintext?
6. You claim a laptop was compromised? In that case what suspicious IP addresses do you have available to report?
7. The potential impact to Okta customers is NOT limited, I'm pretty certain resetting passwords and MFA would result in complete compromise of many clients systems.
8. If you are committed to transparency how about you hire a firm such as Mandiant and PUBLISH their report? I'm sure it would be very different to your report :)
_________________________________________________________________________________________________________________________________________________________________________________________________________ https://www.okta.com/sites/default/files/2021-12/okta-securi...
21. Security Breach Management. a) Notification: In the event of a Security Breach, Okta notifies impacted customers of such Security Breach. Okta cooperates with an impacted customer’s reasonable request for information regarding such Security Breach, and Okta provides regular updates on any such Security Breach and the investigative action and corrective action(s) taken. -
But customers only found out today? Why wait this long?
9. Access Controls. Okta has in place policies, procedures, and logical controls that are designed:
b. Controls to ensure that all Okta personnel who are granted access to any Customer Data are based on leastprivilege principles;
kkkkkkkkkkkkkkk
1. Security Standards. Okta’s ISMP includes adherence to and regular testing of the key controls, systems and procedures of its ISMP to validate that they are properly implemented and effective in addressing the threats and risks identified. Such testing includes: a) Internal risk assessments; b) ISO 27001, 27002, 27017 and 27018 certifications; c) NIST guidance; and d) SOC2 Type II (or successor standard) audits annually performed by accredited third-party auditors (“Audit Report”).
I don't think storing AWS keys within Slack would comply to any of these standards?
> The potential impact to Okta customers is limited to the access that support engineers have
Well the implication here is that the support engineer only had some limited set of tasks, but if that Support Engineer also indirectly had access to AWS keys then it suggests that Lapsus could have had broader access than just what the support engineer had direct access to.
> Support engineers are also able to facilitate the resetting of passwords and multi-factor authentication factors for users
No breach at all here.
Wondering if this is related.
Is it common for attackers to just get bored and leave taking just a few screenshots?
Or was that all bluff and lapsus didn't have anything
If someone breaks into your house, and stays there for a week, would you say they didn't really break in because they didn't murder you?
>proof-of-breach
Maybe you're referring to a "proof of concept"? A "proof of breach" in the security industry is an incident to be investigated, and if necessary, involve law enforcement. It isn't meddling kids that didn't cause any harm. As a side note, I've never heard anyone use the term "proof of breach" in that context in the industry.
The fact that Okta didn't detect this earlier is concerning in its own right, let's not downplay the fact that the level of access that third party providers have is not a solved issue in the industry. The RCA/post mortem/follow up actions from Okta's side should not be "they got access but didn't do anything with it, we don't need to change anything".
https://www.okta.com/blog/2022/03/updated-okta-statement-on-...
I do enjoy the lies given by Okta.
1. We didn't compromise any laptop? It was a thin client.
2. "Okta detected an unsuccessful attempt to compromise the account of a customer support engineer working for a third-party provider." - I'm STILL unsure how its a unsuccessful attempt? Logged in to superuser portal with the ability to reset the Password and MFA of ~95% of clients isn't successful?
4. For a company that supports Zero-Trust. Support Engineers seem to have excessive access to Slack? 8.6k channels? (You may want to search AKIA* on your Slack, rather a bad security practice to store AWS keys in Slack channels )
5. Support engineers are also able to facilitate the resetting of passwords and MFA factors for users, but are unable to obtain those passwords. - Uhm? I hope no-one can read passwords? not just support engineers, LOL. - are you implying passwords are stored in plaintext?
6. You claim a laptop was compromised? In that case what suspicious IP addresses do you have available to report?
7. The potential impact to Okta customers is NOT limited, I'm pretty certain resetting passwords and MFA would result in complete compromise of many clients systems.
8. If you are committed to transparency how about you hire a firm such as Mandiant and PUBLISH their report? I'm sure it would be very different to your report :)
_________________________________________________________________________________________________________________________________________________________________________________________________________ https://www.okta.com/sites/default/files/2021-12/okta-securi...
21. Security Breach Management. a) Notification: In the event of a Security Breach, Okta notifies impacted customers of such Security Breach. Okta cooperates with an impacted customer’s reasonable request for information regarding such Security Breach, and Okta provides regular updates on any such Security Breach and the investigative action and corrective action(s) taken. -
But customers only found out today? Why wait this long?
9. Access Controls. Okta has in place policies, procedures, and logical controls that are designed:
b. Controls to ensure that all Okta personnel who are granted access to any Customer Data are based on leastprivilege principles;
kkkkkkkkkkkkkkk
1. Security Standards. Okta’s ISMP includes adherence to and regular testing of the key controls, systems and procedures of its ISMP to validate that they are properly implemented and effective in addressing the threats and risks identified. Such testing includes: a) Internal risk assessments; b) ISO 27001, 27002, 27017 and 27018 certifications; c) NIST guidance; and d) SOC2 Type II (or successor standard) audits annually performed by accredited third-party auditors (“Audit Report”).
I don't think storing AWS keys within Slack would comply to any of these standards?
"kkkkkkk" is the Brazilian way of typing "hahaha". Is Lapsus from South America?
> highlighted that there was a five-day window of time between January 16-21, 2022, where an attacker had access to a support engineer’s laptop
These are some impressive mental gymnastics!
> Okta detected an unsuccessful attempt to compromise the account of a customer support engineer working for a third-party provider
> highlighted that there was a five-day window of time between January 16-21, 2022, where an attacker had access to a support engineer’s laptop