Cloudflare mitigated yet another Okta compromise
blog.cloudflare.com
blog.cloudflare.com
Surely instead of "recommending" a better security posture, they should tell Okta that they won't continue to use them as a vendor unless they follow a stricter security policy?
Second, moving venders is VERY hard. It can take years to do it. I have worked at another cloud provider and moving from an old system to a new one is a huge undertaking. You do not just say, “See you, I am out of here.”. It can take years of work and planning.
Third, you do not know what CloudFlare is doing internally. My guess is they have already spoken to Okta several times about the problems they have had. At this point, CloudFlare may actually be moving off Okta or thinking about moving off of Okta.
Edit: Looks like a fairly strong down day in general so maybe it was just a coincidence. When you look at 1Y of data it's clear that OKTA swings around a lot. There's precedent for moves like this, in other words.
Okta is different—the problem it solves is squarely in the domain of their customers' infosec teams, and its one job is to be safer than alternative authentication methods for large enterprises. It makes sense that investors would worry about them taking a larger hit than ${random_SaaS_product} would.
There might be/come a longer form technical write-up with more detail of how those products helped, if that's what you're after, CF often seems to follow that approach, sort of announcement + detail, or press + technical.
>There might be/come a longer form technical write-up with more detail of how those products helped
There won't be, they just used their blog to pressure Okta to improve certain things, which is their right I suppose but it makes for very boring reading.
So I was thinking “but they did share their safeguards: they’re using the features of their product that can let you do X, Y, or Z.” Which is an entirely orthogonal point.
I expect that their proprietary techniques underlying all of these capabilities are mostly derived from their insane eyeballs on Internet traffic and their ability to fingerprint and correlate devices and network traffic at scale. I agree it would be nice to hear more about that.
In this case:
> In fact, we contacted Okta about the breach of their systems before they had notified us.
Cloudflare probably shouldn’t be posting their tokens anywhere to anyone so it’s hard for me to think they’re some how in a moral high ground.
I think it's pretty fair for Cloudflare to place this in Okta's court. Okta customer support knew what they were asking for and should have had greater controls in place for dealing with those files safely.
> Okta explained that when it is troubleshooting issues with customers it will often ask for a recording of a Web browser session (a.k.a. an HTTP Archive or HAR file). These are sensitive files because in this case they include the customer’s cookies and session tokens, which intruders can then use to impersonate valid users. [2]
[0] Old content: http://web.archive.org/web/20230207011818/https://help.okta....
[1] New content with updated warning: https://help.okta.com/oag/en-us/content/topics/access-gatewa...
[2] https://krebsonsecurity.com/2023/10/hackers-stole-access-tok...
Sure, it’s still not a good idea for them to give Okta their tokens. But the above, combined with the fact that forgetting to redact a HAR is an incredibly easy mistake to make, makes me tend towards giving them a pass on this one.
I also used the 2FA app because I thought it was the newest thing to do. But, it gave me admin rights, and back door access. Ect. And nobody believes me.
If you want to communicate with okta you send them an email, you don't shame them publicly.
- 2nd time Okta was breached
- Cloudflare shared/detected? it earlier. How long till more Okta customers were breached because they were silent?
- Cloudflare considers authentication as very dangerous. There was a post recently: "hackers login" ( can't find the post anymore?) which means they are very consistent about these types of attacks.
How does a session token end up in a support ticket? Are they putting session tokens in URLs or something?
The tone of this is fine. What isn't fine is that Cloudflare isn't dropping Okta. I might consider dropping Okta.
First, Okta should get credit for publicly acknowledging the compromise.
Second, expecting any company or organization to never get hacked is unrealistic. Organizations which are transparent and acknowledge their security breaches should get credit for it. Organizations which cover things up should be avoided.
Third, no one knows how to build secure software or services. The closest anyone has gotten is Open BSD but it can still be hacked once you install software on it.
Forth, the number of publicly acknowledged breaches does not tell us anything about Okta’s security. Here is why:
- We do not know how many breaches each authentication provider has had
- We do not know how many breaches were never detected by the authentication provider
- We do not know why the breaches occurred. Breach causes can range from gross incompetence to “WOW, that attacker was really clever and found a new class of security bugs”.
- We do not know how Okta responded to the breach. We also don’t know how its competitors respond to their breaches.
My main point is security is hard and measuring security is also hard. We cannot use simple metrics to determine if an organization is a good organization or a bad one.
What????
They would reasonably face civil suits if they didn't.
[0] - https://www.justice.gov/usao-ndca/pr/former-chief-security-o...
Where "out there" is Okta. You are basically saying: Don't leave sensitive data with Okta.
That is not an argument one would want to hear from a company whose only reason for existence is being good at security.
The security bugs are not there because of incompetence or stupidity. They are there because security is really hard and it is even harder to get every software engineer to care about security.
If the best organizations in the industry make security mistakes, what makes you think the rest don’t either?
Reality is often unpleasant. It is better to acknowledge it than to pretend it does not exist.
Perfect security is impossible, but better than 50% odds of never getting critically compromised are reasonable to expect.
If their security is not good enough, their value quickly becomes negative. Not only are they already a gigantic target and a single point of failure, but by being visibly bad at security they are standing in the spotlight with "hack me" on their back.
(If Okta was zero-dayed, IMO we'd have heard about it. Great way to shift blame.)
"In early October 2023, Okta was notified of a breach resulting in hackers stealing HTTP access tokens from Okra's support platform by BeyondTrust. Okta CTO Charlotte Wylie denied the incident for a number of weeks, but later recognized that a breach had occurred"
https://en.wikipedia.org/wiki/Okta,_Inc.#cite_note-33
"Okta’s Wylie declined to answer questions about how long the intruder may have had access to the company’s case management account"
https://krebsonsecurity.com/2023/10/hackers-stole-access-tok...
I would also suggest that each of the companies that publicly posted have something to gain from doing so. We also don't know if they are telling the full story. Using the "we told okta on X date" as assuming that starts the clock on okta not disclosing a breach to the public is a pretty ridiculous take.
We still don't know the full circumstances of how this happened and Okta has not yet publicly commented on their side of this story. Presumably because the investigation is ongoing. But reading the details in the cloudfare post, access for the breach stopped on the 18th and okta told customers on the 19th. Is okta supposed to alert customers of every single report of a breach, every report that might have some credibility, etc...? Maybe there are process improvements to be made in the review process, but we have no visibility into the current level of effort they are making.
Meanwhile, other companies have had known issues for months/years (keyword is KNOWN) before disclosing. I don't want to be a victim of this more than anyone else, but I think we need to be more reasonable in our "hot takes" to these situations even if we are calling for continued improvement.
If they keep undermining the 'security' bits they will 100% get ditched. Assuming there is a credible alternative. But I'd argue there isn't a credible alternative at the moment (on either the security or non-security front).
If you listed all of the 'security forward' SaaS companies you can think of. I guarantee 3/4 use Okta. I also bet all would be keen to switch as soon as an alternative was reasonably 2x better on either the 'security' or 'non-security' fronts. Even given the massive pain in the bum it would be to migrate. No one loves Okta or their sales team.
The dumbest guidance your security architect could have given was "Okta got hacked by actors that don't care about us. We should move to ${SOLUTION} that is objectively worse for users and probably worse for security"
No they won’t. This is my point - the ‘security’ bit isnt even on the radar of 3/4 of those companies. Compliance is
We were using a different solution at that point and planning a move TO okta. It wasn’t just about the fact it was hacked (happens to the best of us) but how they got hacked, how they found out and how they responded all of which made immediately clear what a nightmare it was/is
Seems a bit haughty to publicly chastise another company. The tone of this article is a bit off-putting for me personally.
I think this makes more sense for strategic business partners. In the Cloudflare-Okta case I'd wager that their relationship is fairly transactional.
Also, I think CloudFlare’s blog post was very good.
"Take any report of compromise seriously and act immediately to limit damage; in this case Okta was first notified on October 2, 2023 by BeyondTrust but the attacker still had access to their support systems at least until October 18, 2023."
It is good to call Okta out here as it impacts Cloudflare's business as well and if you can't fix a critical issue for 16 days, that is bad. Remember we are talking about Auth here. A breach impacts everything.
Hopefully they sink their teeth and give out a nice fine for this insane negligence, but I suspect okta is in for a strongly worded letter.
If Okta does not want its customers to publically complain about its actions, Okta needs to improve and do better. In particular, if someone says they have been hacked, listen to them and keep digging until you find the problem.
If a threat-actor stole bank my password because I stored it in a file on my laptop how is it my banks fault?
That said, security is a really hard thing to get right 100% of the time. And there is no guarantee that another security vendor won’t have other issues.
But Okta should’ve been more forthcoming communicating this.
Compared to other businesses I've worked in, they seem to be having too many preventable security issues, and they're in the security business.
Cloudflare's incident report here is what I'd consider forgivable and sounds like they take security very seriously, but they can't be perfect. Okta looks pretty bad.