Hackers stole access tokens from Okta's support unit
krebsonsecurity.com
krebsonsecurity.com
Okta published a blog post here: https://sec.okta.com/harfiles and kicks it off with
> Okta Security has identified adversarial activity
and no mention of the notification from the third-party. Nice transparency!
Genuinely curious, we use Okta and I’d like to understand why you are saying that.
What quantitive evidence have they ever demonstrated that shows they can stop the attackers who would like access to the billions of dollars of assets whose access they authenticate? A criminal enterprise can literally hire tens to hundreds of skilled hackers full time for years to target these systems and still turn a profit.
The default assumption is that systems are easily hacked. Claiming protection against even small teams of moderately skilled attackers, let alone organized crime, is a extraordinary claim. Where is their extraordinary evidence?
One can prove that a vault has been secure for the N last years, but not that it will be secure for eternity.
There is plenty of evidence you can provide to establish confidence that a certain degree of security has been achieved. Robust auditing, thorough review, formal methods, exhaustive testing, competent red teams exercises failing to find any vulnerabilities, etc. The only people throwing their hands up claiming security can not be evaluated have nothing useful to say about security because they do not even believe it is possible to know if they did anything.
Having red teams, having audits, having scans, etc is simply not enough for some folks but in Okta’s eyes, it’s enough for C-suite talks of taking Authentication/authorization off the plate of their IT department.
I firmly believe for every individual who thinks they are untouchable, there’s a hacker who knows more and is willing to throw it all away to prove a point.
So yeah, show me a red team exercise with 10 M$ of funding, they get a 30 person hacking team and 1 year fulltime, that failed to find any vulnerabilities and failed to gain access to any sensitive data, then we can talk about if they provided evidence of adequate security. I bet all they have is what everybody else has which is red team exercises that had 3 people for a month that reported 27 serious vulnerabilities, then another red team exercise that found a different 23 vulnerabilities, then another, then another, then another, always finding new ones because their systems are actually at the 100 K$ quality level. Those exercises do provide evidence and confidence in their security, you can be extremely confident their systems are grossly inadequate for their threat landscape. I have not looked, but the same can almost certainly be said about their certifications, audits, etc. since the gold standard that everyone aspires to is, when looked at objectively, grossly inadequate.
So that they never have to pay it. Up to them to decide how to upskill their red team now.
Barring such a bug bounty, there is no way to trust them.
And you can substanciate a claim that an an online service is secure without proving it.
I mean, do you seriously think that if person A says: “My vault is secure for 15 minutes against a human with a crowbar.” And person B says “Prove it.” That person A would ever respond with: “I can not because a vault is not a mathematical object and therefore proof is impossible, but I can substantiate it.” That would be ridiculous beyond belief. That is what you are doing.
https://blog.cloudflare.com/how-cloudflare-mitigated-yet-ano...
It's the same type of session replay attack (likely HAR) discussed in the original article, no?
It seems a reasonable expectation to assume that anything sent to Okta support isn't instantly available to attackers.
So, yes, valid session tokens were dumb. But also yes, Okta fucked up here too.
No that’s not a reasonable assumption. Malicious Okta employee is just as significant an attack vector as compromised Okta support tool.
If Okta employee is a high priority threat model... then the customer is better off not using Okta.
Not that it shouldn't be considered, but if Okta top-to-bottom penetration is expected and accepted, then that's taking Zero Trust to a whole new length.
It's literally a policy from Okta to investigate issues, which isn't Cloudflare's fault. The tool from Okta got compromised and all clients that needed support from Okta could/have been damaged as well.
Additionally, it was Cloudflare that SAW something was off and notified Okta. Cloudflare didn't get breached at all.
> The root cause is that Okta got compromised.
It's even suprising that Cloudflare's policies are so good in detection that they detected this at all, before Okta.
This is not an endorsement of the decision to use Okta, but I understand why.
also i sincerely believe theres a little bit of not minding they suck because they can just blame okta if something happens and blame is the worry.
Two things are corrosive from the executive ranks:
"Everyone knows this isn't working, but the director who authorized it is now an SVP"
"Not my idea, so let's do something different just so I can say it was my idea"
The middle way is healthier.
Because, if you are already using Okta, it costs budget to change that. Who is signing up to to do that? Didn't Family Circus used to have a ghost character labelled "Not Me"?
And, an bunch of people who didn't like Okta went with Auth0 and then wound up with Okta, anyway.
My experience has been: start with Google until it's too painful to continue, choose between Azure AD or Okta. Self-hosting for plenty of firms is just asking for worse scenarios. Is there some market leader I'm unaware of?
Horrible title, not transparent or direct, pretty lame. and I agree, it's dreadful that Okta did not even mention that BeyondTrust told them about it on Oct 2 (30 minutes after BT uploaded their HAR file to Okta Support).
Nothing like disclosing a hack to company that's supposedly an expert at cyber security and authentication (within 30 minutes!) only for them to tell you that you're wrong and not admit it for 15 days while a hacker runs amok.
Do better, Wylie.
https://www.beyondtrust.com/blog/entry/okta-support-unit-bre...
Of course the easiest solution is you shouldn’t voluntarily share HAR files for an active session.
> *Indicators of Compromise*
> ...
> Okta activity for a user without any clear indication that the user authenticated (e.g. a user.session.start event for that user from a similar geographic area)
This includes not only Okta but Auth0 (Okta acquired), Authy, and Duo.
Source: I see this communication at (large enterprise) workplace and so many other grand cloud migration pronouncements on internet by executives of F500 companies.
You actually can write and test software on a system that never gets connected. (Shockingly true!)
I also can’t tell if it’s satire, which goes to show maybe communicating mysteriously has ran its course.
Say what you mean, mean what you say.
1. All software has security bugs. This includes on-prem software.
2. Many IT groups do not have the expertise to run and secure a directory. You don’t just install some software, setup some accounts, and leave the box in the corner. You have to make sure the system survives disasters, you have to test backups, you have to figure how to determine if you were hacked, etc. This is very hard work.
3. No one has ever shown that on-prem IT security is better than cloud vender IT security. My experience has been that some on-prem IT departments have very poor security. Here are some examples:
- One IT department I worked with used a VPN which used an MD5 certificate. This was between 2010 and 2020. MD5 was cracked and it meant anyone could preform a man in the middle attack.
- One IT department I worked with deployed a web service which allowed unauthenticated users to access person customer information (national ID/tax number, address, name, phone number, etc.). Note the person doing this knew he was doing something unethical and did it anyway.
- One IT department I know of supples nurses with an add supported text editor to type up patient notes. Adware is not know to value privacy and should never be used to process sensitive information.
- One IT group I know of will not update Redcap to a supported version. The version of Redcap the IT department uses has known security bugs and the IT department was told about them. The official excuse for not upgrading it is the IT group does not like Redcap.
I think Redcap is a piece of software which stores information used in medical studies and used to calculate public health statistics. It contains lots of sensitive data.
My main point is moving from the cloud to on-prem may or may not improve security. It really depends on the capability of individual IT groups and many IT groups are not capable of running an identity service like Okta.
The other thing to consider is cloud venders typically have much larger budgets and can spread costs over many customers. This means they can afford more security specialists, spend more on securing systems, spend more on detecting breaches, etc.
Then there also is the question of cascading effects. E.g. if someone compromises a cloud provider and gains SSO access to Org A and Org B at the same time they might be able to do more harm
Running the same software as everyone else on-prem doesn’t make it more secure — it might even do the opposite unless you’re quick to do security updates.
And that's not 'on prem' but usually colocated hosting.
Whether it might improve security is definitely questionable. But it does limit the impact to only one org. Even if multiple orgs use the same product, the local implementations might differ and increase friction to scale the exploit over large swaths of companies.
Using Okta, every random support person at Okta is a super-admin within your organization able to grant themselves access to all of your stuff.
Sure, everyone deploying things on-prem will mean attackers will focus on the software instead of the cloud vendor. But that's A∨B vs. only B.
> - One IT department I worked with used a VPN which used an MD5 certificate. This was between 2010 and 2020. MD5 was cracked and it meant anyone could preform a man in the middle attack.
MD5's collision resistance has been broken. Corporate VPNs are often deployed by shipping the cert's pubkey on each client. In that case you're not using PKI which means the switcheroo attack involving collisions aren't relevant and you only need to rely on MD5's preimage resistance which is still ok.
So while using MD5 doesn't smell great it's not necessarily an open barn.
The default assumption is that they can all be breached, the burden of proof is on them to prove they will not get compromised. It would only seem prudent to wait for positive proof of the extraordinary claim that they can not get hacked rather than extending the benefit of the doubt to serial incompetents.
when a breach can be caused by an unknown CVE in the future?
That seems impossible to prove, I mean I understand you can patch, audit, encrypt, your way to a safer system, but proof (by definition) is a very high bar.
What is an acceptable demonstration of proof given unknown future events? How does one define proof in the sense “will not get compromised” (future tense is used in your phrasing so a proof would have to include all possible future outcomes)
To go a step further, you can apply this to any acceptance criteria. What evidence is there that any of these systems meet any meaningful acceptance criteria. What evidence is there that Okta has systems that can protect billions of dollars worth of assets from the teams of professional and state attackers that currently target their systems?
To then get to your direct question, if you really need “should never get hacked”, you could provide machine checked mathematical proofs of correctness, robust and exhaustive validations, and NSA penetration test reports showing zero identified vulnerabilities like what was done for the F-35. I mean, I guess that is only like 1000x better than prevailing commercial IT systems and not completely foolproof, but it is certainly a good starting point. You can probably think of some ways to evaluate even more robust security if you need more assurance than the US air force.
Sounds pretty stupid, right?
Okay, now replace slingshot with tank.
Now it does not sound so stupid.
Turns out you can learn something based on the effort needed to breach a defense.
As it turns out, the entire commercial IT industry is basically incapable of stopping small teams of moderately skilled attackers as has been demonstrated to death. A team with just 1-2 M$ of resources is basically unstoppable even with 100 M$- 1 G$ budgets. That is the definition of gross inadequacy.
It seems a problem with the provider. You're problem is probably not even going to be checked without fulfilling their request.
Okta should have revoked the token after the file was no longer needed.
Should I remind you that multiple customers were compromised because of this and that Cloudflare was probably the only one that wasn't breached AND notified Okta...
The goal is not absolute security, rather practically secure systems against attackers with moderate resources.
We can’t prove that a security company will not be breached. But once they have a significant breach, it might be time to move on. They likely have other problems. I’m looking at you LastPass!
Okta holds the keys to the castle. It has had a security incidence in the past. A compromise of the Okta systems will have a huge impact on its users. The margin for error is small.
This is why anyone even slightly concerned about this should use offline, self-hosted, or self-built alternatives.
A honeypot is a trap set to catch attackers. It does not include real data.
https://en.wikipedia.org/wiki/Honeypot_(computing)
Seeing this misuse a lot lately!
These services are attractive targets, but they are not honeypots.
Some of them might also run honeypots separately from their core service, but that is not under discussion presently.
Says a lot about security discussions around
Way to ignore my point, and nitpick about semantics.
No, this forms part of the definition.
In all seriousness, they seem to love getting pwnded. You would hope that the constant stock drops as a result of events like these would force change, but I guess not.
1. https://www.malwarebytes.com/blog/news/2023/01/okta-breached...
3. https://www.theverge.com/2022/4/20/23034360/okta-lapsus-hack...
3. https://techmonitor.ai/technology/cybersecurity/okta-cyberat...
On one hand this was not a core product breach.
On the other, every core product breach has an indirect non-core breach.
> Please note that this information is Okta confidential information and should not be shared outside of your organization.
Which is pretty hilarious, honestly.
In fact, until we get actual liability for breaches like that, an investing strategy based on "buying the dip" every time a company gets breached would be probably be quite successful.
https://www.beyondtrust.com/blog/entry/okta-support-unit-bre...
Less tongue in cheek, does anybody have a recommendation for a reputable centralized identity/auth provider? Like Mullvad is to VPNs as <> is to OAuth?
EDIT:
Update after looking through the article the important bit is
>In an advisory sent to an undisclosed number of customers on Oct. 19, Okta said it “has identified adversarial activity that leveraged access to a stolen credential to access Okta’s support case management system. The threat actor was able to view files uploaded by certain Okta customers as part of recent support cases.”
So it wasn't a full breach so much as someone was able to get in and view the support tickets customers have opened. Which although bad isn't nearly as bad as losing a bunch of people's keys. (again)
It is nearly certain that sometime in that 15 days a bunch of Okta customers got breached and are infiltrated now without their knowledge.
“Security” as a practice within a company comprises of technology, process, and culture. Should one of these pillars be subverted by leadership in anyway, the organization security stance becomes crippled.
In my experience, this security kneecapping happens because some bonehead exec looks at ways to cut costs or doesn’t like having to pull their phone out to login, and decided on a layoff or loosening of policy because, “we haven’t had any problems so far, so why do we have these staff members or this highly obstructive 2FA in place?” I mean, people used to die with greater frequency in car accidents prior to seat belts, too, maybe we can eliminate those next?
Are possibly even more important than the tech. A good culture of "always report suspicious things, no one will be upset over false alarms" and a process that follows up on each report with rigor and responds to the constant stream of false positives in an engaging and encouraging way[1] catches a lot of problems before they start and/or are in the early stages.
I've witnessed and heard stories of such cultures and policies catching pretty sophisticated targeted phishing campaigns with 0 breach.
Similarly even for actual compromise - having a "always tell us when you suspect something, if you click on a bad link and report it when you realize it you won't get in trouble" means that breaches can be stopped early (again I've seen this prevent incidents from turning catastrophic).
It's cheaper to spend a few $100K on more people to handle false positives than it is to lose millions in direct costs from a breach and even more millions on reputational loss.
[1] e.g. "We examined the email and link you sent. Thank you for reporting it - it does in fact look suspicious, but in this case we've verified it's the a safe and legitimate link. Please continue sending in anything you're unsure of!"
- customers wouldn't scrub sensitive details from support HARs, unless those details were required for the issue
- if customers weren't required to submit sensitive details, that Okta support wouldn't have a tool to scrub any accidentally included from the support system (looks like it's SFDC-backed)
- if sensitive details were required, that Okta support didn't have an extremely tight data retention policy, in terms of hours not days, with active enforcement
These kinds of things were hard requirements just as presales demands from fintech and healthcare customers, much less for certification.
This isn't going to be fun, but it's making more sense every day.
Also, Microsoft and Google have one big thing in their favour: they run consumer services with billions of customers that they get a ton of experience from. And which are a huge target which rarely gets breached.
Those consumer services are not running on the same IDP services e.g. "Entra ID" but as a company it does prove they know what they're doing. I have to give them that. I know MS got breached recently too but they have a decent track record overall. Saying this as a MS critic by the way.
For me, the biggest bummer was the lacking security culture that let a cert validation component to not properly validate certs, in at least two ways (didn't enforce tier separation; accepted tokens signed by an expired or revoked CA).
Nobody Gets Fired For Buying _Microsoft, I guess.
OR!!!?? just buy Microsoft 365
Though call huh
You work here too?
Though azure ad was actually a bit confusing. But usually their rebranding makes zero sense. Like from lync to Skype or msdn to my visual studio. Or sms to sccm to mem.
Switching over to "Entra ID" is probably the only actually useful rebranding Microsoft did in the past 30 years.
BRB, gotta step out to say "I told you so"
What’s going on with these people? Is there any other SAML provider that gets compromised as often. They’re like the LastPass of SAML providers…
Both are bad at the very thing they’re supposed to provide and be good at.
I am sympathetic to the idea of not reinforcing bad behavior at a vendor, but also mindful that sometimes you're going to risk accept and move on (and revisit when the contract is up for renewal). Time and resources are finite.
EDIT: With all the above said, Okta would not be my first idp choice, considering all available information.
(head of security @ fintech, I own customer and internal IAM, thoughts and opinions my own)
Understatement of the year.
Also, this is inexcusable for an $11B company.