Okta’s Investigation of the January 2022 Compromise
okta.com
okta.com
"In this post, I want to provide a timeline and my perspective on what has transpired, and where we are today with this investigation. I hope that it will illuminate why I am confident in our conclusions that the Okta service has not been breached and there are no corrective actions that need to be taken by our customers."
And then go on to write paragraphs of detail and a timeline that explicitly shows for a five day period an unauthorized user had full super user access to the service.
He explicitly says, "The report from the forensic firm highlighted that there was a five-day window of time between January 16-21, 2022 when the threat actor had access to the Sitel environment, which we validated with our own analysis."
This is some unbelievable double speak to try to claim that Okta was not breached. Any trust I had in this company is completely in the toilet, just based on this response. How is the CEO not responding to this too? The hole just keeps getting dug deeper the more they say.
At this point, it would be helpful for Okta to stop using the term "breached", because apparently they aren't using the word in the same way everyone else is, and it's a point of contention.
Okta is intentionally using their own definition to down-play what’s happened.
A breach is simply ‘to overcome defences’ (ie to get access to something you shouldn’t). In this case their defence against someone else accessing the super user application was the support employee and their credentials, but this defence was clearly overcome by the hackers.
I agree that they need to stop using the word.
The most likely reason they dont wanna use the common sense term "breached" is because it implies legally a breach of contract, which means liability and getting sued and having to show up in front of congress.
Very sad to see their response be so intransparent and flawed. I wish technical people would be more involved in writing these responses, not lawyers.
> the Okta service has not been breached
If you consider the Okta service as the infrastructure acutally providing auth services to customers - then it wasn't breached. Only Okta support services were breached and that did not (going with Okta's line here) actually allow for somebody to access the actual Okta auth services.
It was a customer service rep who could send password reset emails.
Systems listed were that app and SaaS offerings. The laptop was vendor owned and managed.
Given my experience, it probably had 0 access to anything “inside” Okta. There was no mention of vendor VPN or Virtual Desktop access. Or access to any internal system beyond what was a horribly named call center support tool.
They really wouldn’t have passed the necessary audits for their enterprise and government customer base if they allowed 3rd party devices “real” access.
Beyond a “hacker” getting access, there is very little trust in (likely) contract employees of a contract company.
The fact that the tool is named Super User is killing them more than anything here.
The only way one can reach this conclusion is by ignoring all the breaches / CVEs that happened in large companies during the past few years.
Nowadays I just assume every company is crap at security unless proven otherwise.
Yes, most companies are bad at security. Some aren't though. The problem right now is figuring out which ones are - there's very little signal.
We all know it is very uncommon to do a great job. Everyone has been breached sooner or later. Any anyone who has worked in engineering or security in tech companies knows how often security concerns are underprioritized far behind more visible but less important work.
Hedging is a good answer. Relying on a single point of failure that, if it ever fails open, will expose everything at once? Not a smart idea.
it sounds like they built the support tool (with its unfortunate name) such that it places limited trust in support contractors and the information technology that supports them. because of this they're able to identify potentially affected customers and even audit all activities that have taken place. in other words, the system worked as designed.
this is very different (and orders of magnitude less severe) than an actual compromise of the production service itself.
It doesn't matter if they did it by fooling or paying a low level CS rep to get access to their account vs. using their 'leet hacking skillz' to pwn the electronic defenses. A breach is a breach and the CSO of all people has to own up to that fact.
"This is an application built with least privilege in mind to ensure that support engineers are granted only the specific access they require to perform their roles. They are unable to create or delete users. They cannot download customer databases."
not being able to create or delete users seems a far cry from "near full admin access."
But at least changing passwords of existing accounts seemed possible according to the screenshots, perhaps disabling 2FA as well.
As long as Okta does not provide a list of what the user was able to do, we don't know
I can't see any place in the screenshots where they are setting a user's password, only sending reset links (which would do nothing if the support user does not have access to system email or end users email).
Also at my workplace, 2FA is not enforced by an IdP (like Okta), but by our own application (and therefore could not be disabled at the IdP level).
As others have mentioned, the slack access is potentially more concerning.
If Okta has implemented a BeyondCorp model, the attackers had access with known limits. It doesn't matter how much access if the machine they've accessed is itself untrusted. That's the punchline of an episode of Star Trek: Lower Decks where they allowed an evil computer to take over the lighting systems on a starship; while they repaired the ship, all the computer could do was flick the lights on and off angrily.
You can be a security expert and no tech company will care unless you cram leetcode non-stop.
On the plus side the lack of security focus makes for a lot of opportunities for the stock market when betting against overvalued tech companies with this problem!
one day there will probably be professional engineering certifications for software engineers, and those certifications will largely center around building secure software and systems. (with maybe some focus on safety critical systems as well)
it's funny when you think about how young the field is.
The idea that at a security focused company they are passing around secrets via public slack channels is wild. That’s the thing they need to address.
There is plenty concerning about their response to this situation, and this phrasing can be confusing, but from my POV in the industry this choice of words is understandable.
To simplify: The hackers had minimal access to stuff and couldn’t do much.
Not defending Okta's response here; I think it has been quite terrible.
"superuser" is a tool which can be used for various tasks. It is not the same thing as "full super user access". As I understand it, the tool superuser doesn't allow you to do whatever you want, like accessing any piece of data in the databases. It is used by their support engineers.
Not trying to say there was no impact, but "full super user access" traditionally implies more than being able to run one specific tool with the name superuser.
I think people who have written about this and looked at the screen shots has been sloppy and spread misinformation.
I like how it just glosses over access to all the other tools which often contain a treasure trove of data. Just Slack can give an attacker worst case credentials pasted into channels and best case loads of information for more targeted social engineering attacks. LAPSUS$ even stated they had access to over 8K channels.
From the article:
> The majority of support engineering tasks are performed using an internally-built application called SuperUser or SU for short, which is used to perform basic management functions of Okta customer tenants. This does not provide “god-like access” to all its users. This is an application built with least privilege in mind to ensure that support engineers are granted only the specific access they require to perform their roles. They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories.
Most commercial services allow unauthenticated password reset just by hitting a web form. It would send a password reset link to the correct place and the person would either ignore it, follow through and have a new password or report it. It doesn’t impact the service access at all.
Meanwhile creating an account would allow new access to the system. I’m having a hard time figuring out how that’s not a world of difference?
Support _Engineer_ that is using a prebuilt application to do stuff like reset MFA. In what world does this have anything to do with engineering?
* They stress that the compromised account wasn't able to "create/delete users or download customer databases", but not what it could do. Could it change passwords of accounts and add 2fa methods, allowing them to take over rarely/never accessed users? Disable 2fa? Change account permissions? List user accounts and metadata to build a user account DB for further attacks? The application is named "SuperUser" ...
* It took public posting of a screenshot to trigger an audit of access logs, two months after the compromise was detected!
* "Only" 2.5% of customers were accessed. That's supposed to be a good thing? Those were certainly the most valuable targets.
* Concludes everything is just fine and no corrective actions need to be taken, but affected customers might want to do their own analysis... Huh?
Sounds a lot like damage control.
Yup, the first line made my hackles stand up; surely there's no security incidents raised for a user-invoked operation like adding MFA to a subcontractor's account? Audit logging is fine, but that's a user-initiated operation that already requires (if they have their shit in order) username, password, existing MFA if applicable, and an e-mail confirmation. To the user. It's all by the user.
Does Github raise the alarms if I change something in my MFA settings?
2.5% of all customers were accessed by all Sintel employees for the period in question. How many customers did the particular affected Sintel employee access?
They assessed all the actions that took place by the Sintel employees. Were any of them sensitive?
The tool doesn't allow them to create or delete users. Does it allow them to modify users so that an attacker can take over an account?
The attacker had access to "Jira, Slack, Splunk, RingCentral, and support tickets through Salesforce". Doesn't sound bad on its own but have those tools been evaluated to ensure they don't have sensitive information? Do we know what the Sintel employee in question accessed from each of those tools for the time period?
The employee's machine was logged into using an RDP session. Was the employee's machine assessed to ensure they weren't storing other sensitive information? While not allowed, a lot of support engineers will quickly drop in "temporary" sensitive stuff into random notepads and what not. Was this assessed? Also, why does a support engineer have a machine that has the ability to enable an inward bound RDP session at all?? How was that not the first trigger since this is a known attack vector for support agents.
Given the possibly small blast radius of this issue, Okta had all the opportunity to turn this into a communications win for them and just build a multiple of trust really quickly. All these half communications have utterly botched it and that's just really disappointing. We'll still continue to use them because the switching costs just don't justify moving off Okta. But Okta has really taken a hit in their "trust bank".
Most likely, they don't write it because they may not know themselves. It is not certain that it gets logged perhaps?
They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories.
So they can likely arbitrarily access/impersonate or at least password reset existing users, and probably reconfigure the accounts in creative ways.
For transparency, these customers will receive a report that shows the actions performed on their Okta tenant by Sitel during that period of time. We think this is the best way to let customers assess the situation for themselves.
So they have no idea which of the actions were or weren't legitimate and make it their customers' problem.
Uh huh, makes sense
> Named SuperUser
Uhh...
It lists all the operations that it can't do, but not what it can do. Can they download a private SAML certificate? Can they impersonate a user? Can they configure SSO and MFA settings? Can they download audit logs?
Oh, that's a good one. Definitely something that the software should not allow, because I can't see a legitimate reason for this (allowing to download the certificate is fine, but not the key).
Solar Winds was the first known incident to escalate to so called "Golden SAML" attack. If the support staff had access to signing certificates, then that would open the door to a wide-scale exploitation of Okta's clients.
A shower of Golden SAMLs, if you like.
New Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30774193 - March 2022 (24 comments)
Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30769537 - March 2022 (220 comments)
Also:
DEV-0537 (LAPSUS$) Criminal actor targeting organizations - https://news.ycombinator.com/item?id=30774406 - March 2022 (0 comments)
Lapsus$ hackers leak 37GB of Microsoft's alleged source code - https://news.ycombinator.com/item?id=30763623 - March 2022 (117 comments)
It really is not analogous at all.
That RDP was enabled, let alone could be accessed from outside the network is worrying.
To me this would appear that, to save a buck, they outsource a lot of functions that then meant customer security was partly out of their hands, and relied upon another company having their security ducks in a row. I'm sure in their marketing materials they boast about state-of-the-art security, but that's only as good as the weakest point in the chain.
The scenario here is analogous to walking away from your computer at a coffee shop... with your computer unlocked and logged in to various things
It's really not the "see, this is something you might do! It's not so bad!" out they thought it would be.
It speaks volumes that their embarrassment is so important that it was the second sentence of the whole investigation, while there is literally not a single word of apology to the actual customers who were compromised.
A swift communication by Okta could have avoided this all together. It seems they care more about their shareholders than their customers.
isn't this how publicly-traded companies are supposed to work? I agree on critizicing that approach and capitalism model, but I don't understand how that isn't common knowledge here.
Okta appeared to have kept this under wraps to prevent a shareholders backlash (short term approach) However, as a result they achieved the opposite, as the share price is still down this morning. This may of course be a temporarily glitch, however, I can see it resulting in a temporary loss of revenue. If I would be evaluating Okta versus a different solution right now, this may well sway my decision.
Publicly traded companies can set whatever priorities they want; the law only requires certain levels of accurate reporting. If shareholders think a company is too focused on customers, or not enough on shareholders, their options are simply to complain or sell the stock, or both.
Pretty ominous name. I wouldn’t hand out “super user” accounts to support engineers from contracting firms for “basic duties in handling inbound support queries”.
Really don't like that they're still unwilling to actually say the name of the 'leading forensic firm'.
This whole communication from Okta is a total debacle. They are trying desperately to under play the event and deflect blame in the poor transparency after the event all while not once mentioning that they plan on making any remediations or changing anything in their day to day operations after this incident to prevent it from happening again.
This is an example of total non-ownership. He is the CSO. It should be "I" or "My" and not diffusing responsibility onto his team with "We".
In my book you should celebrate your successes as a team ("we", "our") but failures are ALWAYS on leaders ("I", "my").
I've talked about this before [4] but just having internal audit logs doesn't cut it nowadays, even in this case it took two months to check the audit logs, you should give your customers access to their audit logs, ideally in near realtime so they can do proactive monitoring.
[1] https://help.okta.com/en/prod/Content/Topics/Reports/Reports...
[2] https://developer.okta.com/docs/reference/api/system-log/
[3] https://developers.google.com/admin-sdk/reports/v1/appendix/...
[4] https://apptrail.com/blog/2022/03/07/internal-vs-customer-fa...
Their CSO writes, "Over the past 24 hours we have analyzed more than 125,000 log entries to ascertain what actions were performed by Sitel during the relevant period. We have determined that the maximum potential impact is 366 (approximately 2.5% of) customers whose Okta tenant was accessed by Sitel."
IANAL and these are only my opinions but it seems like:
(a) Their DPO chose not to notify (GDPR Art. 33) without having the full picture or thought waiting several months for sub-processor's report was a justifiable reason for delaying notification
(b) They failed to perform their own basic forensic activities in light of a "compromise" and only reviewed logs on March 22nd
(c) Have terrible taste in naming their support app "Super User"
In my opinion they are also down playing the importance of the data that may have been compromised. For example do the hackers now know which accounts have MFAs attached to and which don't. What the password policies are (e.g. strength, number attempts, etc.)
Apparently they had some internal issue with orphaned records or something and deployed a code "fix" that ended up deleting legitimate records in a way that wasn't auditable from the customer side.
I think it took about 4-6 weeks of trying to escalate through support and account reps until we actually got an answer. It was also very surprising to see role associations just disappear without a trace (we associated Okta groups with AWS IAM roles in the Okta integration)
I'm excited about this and in the future will be checking stock price first thing when I see these smaller publicly listed tech companies getting hit with big hacks.
For MSFT (and companies of that size) this type of news doesn't move the stock that much but with current volatility there will be plenty of great plays in the future for smaller companies.