Hacker News Reader

Comment by templapsusread | Hacker News Reader

Lapsus has responded https://img.guildedcdn.com/ContentMedia/e4149dc99f447074cb2c...

they edited and added more content https://img.guildedcdn.com/ContentMedia/372280f522049aa0b0eb...

To note in some of the earlier screenshots you can see they have the EC2 Instances menu open in their tabs - that's a bit concerning, why does a support engineer need AWS EC2 access?

The LAPSUS$ post suggests that they queried the AWS keys out of Slack. So the support engineers just have access to Slack, and Okta engineers were dumb enough to put those keys in Slack.

I'm incredulous an auth focused company would do this without someone freaking out? Even my much smaller SaaS companies would react quickly to stop and rotate these if this happened.

looking closely at that I'm suspicious that lapsus$ is playing up what were perhaps trivial or ephemeral keys that were non-sensitive ... eg: test / dev / debug instances etc.

The reason I think that is because if they really had keys for production machines it seems very unlikely they wouldn't have used them to produce some more damaging collateral than they've presented.

The things that go on at small SaaS companies would horrify many. One SaaS company I worked for had a "god-mode" password hard coded into the source code. It was visible in plain text in the source, and you could login as any customer if you knew this master password. Of course, this was not logged anywhere. When employees left, they'd change the password and deploy a new build.

DPL scanners in Slack, anyone? Figured Okta would definitely have those.

We get alerts when folks are sharing tutorial code with fake keys like DEADBEEF in them. It's nice to know DPL works, if you use it.

Can you open the web console with just an access key? My impression was you could only use that to act through a CLI tool, at least officially you need to have powers or act as a user with powers to use the web console directly?

Not using access keys- although using the cli you could create a user which does have the ability to login to the portal.

Untrue, any valid access key pair set can call “GetSignInToken” and access the aws console.

You can add users with console access from the IAM API if you have enough reach.

Frequently, support teams have lab environments. Hopefully, it'd be an account per engineer, but at least isolated from production. If software engineers access these for reviewing issues which made their ways to bugs, then potentially this is a vector that organizations should be concerned about. Attackers will be happy to make 20+ pivots, so even an isolated AWS account for a support engineer is a nice base.

If it's just a tab heading, we don't know if they have access or not. You can always open that page as long as you can log in. But it may show "you don't have permissions to display the instances". If you're thinking instead "why does a support engineer need AWS access" - cloudwatch metrics/logs come to mind.

8600 channels? Wouldn't that overwhelm you? I'm trying to think up scenarios where an org would need so many, but I can't. Is this normal?

There's probably a support channel for each customer and agents tasked with floating amongst them.

I suspect lots of small channels with only a few people in them. They have 5k employees, it adds up.

#Tom

#Dick

#Harry

You jest, but the last two companies have been at have had channels named like "Steves" for all the Steves in the company.

Support tickets, or incidents, or escalations might require creating a new slack channel

It's not a bad idea to create channels for incidents. Ever tried to keep an incident timeline in a Jira ticket? It's absolutely horrendous. We used to automatically create a slack channel whenever a Jira incident ticket was created, and post it into the engineering channel with a message like "SEV1 incident, please join channel #INCIDENT-21". Slack is real nice for posting graphs, links, screenshots, going off on different investigation threads, pinning a status that gets constantly updated, calling out for people to join the channel using @you-hoo and running automations using bots. Jira absolutely sucks for live incident handling.

Of course, you still wouldn't have 8500 channels. That's a lot of incidents.

Ever tried to keep an incident timeline in a Jira ticket?

Once.

Thanks for bringing back the nightmares, friend.

You're not going to be even aware of the majority of the channels. In my experience having ~1.5 channels per employee sounds right across the company. I've started channels for: specific incidents, personal huddle, lunchtime game organisation, limited notifications target, etc. Also we don't know if they count archived channels or not.

Yep, same here. I create many topical channels to discuss some narrow-ish topic and archive them after the matter is concluded. Some of those channels just live a few days or even just hours - other live for several weeks but with several days of silence in-between messages. If I need to look something up I can grab the channel from the archive and won't have messages for unrelated topics in-between.

This blew my mind as well

Channel sprawl is very normal. While my day job has a broad scope in the org, I’m a “member” of over 900 MS Teams. I’d guess that 3% are active and i interact with 0.

900 teams? That's a lot, are you sure you don't mean 900 channels (or whatever Teams calls them) instead?

Nope. 900 teams.

Somewhere, there’s some project manager that says “Wow, I bet spooky23 would love to know about my spreadsheet sorting project”.

A lot of people want to have full control so rather than create a channel called "Project Phoenix" in a team called "Operations" they'll create a whole new team, and use the general channel.

Much like the OP here I'm in hundreds of teams, and almost all of them do all their talking in #general and they wonder why people never respond to notifications.

The Teams UX and general paradigm is awful. Right now I'm in 3 group chats and two channels (in two Teams) discussing the Okta incident. Huge overlap of them, but not 100% so some people aren't getting all the information.

I forgot to mention that the teams interface is truly awful. I do what I should be doing in a “Team” in one of a half dozen pinned group adhoc chats.

The context switch between a mostly useless teams tab and a needlessly full screen IM window is too heavy for me.

At a previous job a new slack channel was spun up for every incident, no matter how minor by automation. So yeah a few thousand doesn't sound that weird to me.

We have 70k channels in our slack at my current company. A large portion are deserted.

They have access to that many channels. There's likely many that are public (including history), but the user is not a member.

A more poignant elegy to the modern landscape of compliance theater I have never seen:

> Security Standards. Okta's ISMP includes adherance to and regular testing of the key controls, systems and procedures of its ISMP to validate that they are properly implemented and effective in addressing the threats and risks identified. Such testing includes:

> a) Internal risk assessments;

> b) ISO 27001, 27002, 27017 and 27018 certifications;

> c) NIST guidance; and

> d) SOC2 Type II (or successor standard) audits annually performed by accredited third-party auditors ("Audit Report").

I don't think storing AWS keys within Slack would comply to any of these standards?

Yep. All these standards are tick boxing for liability. Nothing more.

They are not effective security controls and never will be and should never be a measure of that.

I don't know if tick boxing was a spoonerism or intentional or a real thing but I love it and am stealing it.

(Upon further review, it appears to be the more UK way of saying it! Ha!)

Yep UK here. Normal here :)

We’ve been monitoring this internally, as customers of an Okta-like service.

I’ve also been closely monitoring the responses from our CTO and VP of Security when someone from our DevOps team posted a link to the Verge article in slack this morning.

Which brings me to this inquiry: How are your orgs responding to this? We have a dependency on an Okta-like provider and my first thought when reading this news was “you know, wonder if we should give our shit a sanity check”, and someone beat me to this, proposed it in slack but the idea was turned down by our SecOps team.

Sounds about right. Here there will be a staff security training symposium that runs everyone through a training course bought in from the lowest bidder that is tangentially related to the issue followed by a self-congratulatory management meeting and that will be the whole issue resolved to satisfaction.

I moved over to Azure AD this morning (we only have a few devs and were already using Azure DevOps so this was doable). I requested that Okta cancel our account and let them know the reason was the potential data breach and their CEO's response on Twitter. Okta's response was that we signed an MSA agreement and that cancelling isn't an option, nor termination of fees.

They sound like they're running the organisation like a dating site.

More reasons to look elsewhere.

Or like Adobe: https://news.ycombinator.com/item?id=30222165

Having just paid for Lightroom for a year this one really annoys the shit out of me.

You’re buying Okta when you get Adobe too. :)

Okta is the Oracle of identity management.

https://auth0.com is the "still cares about customers" vendor

I'm not affiliated with them, just traumatized by working in IT

Auth0 was acquired by Okta (https://auth0.com/blog/okta-acquisition-announcement/), although Okta claims in the post that

> There is no impact to Auth0 customers, and there is no impact to HIPAA and FedRAMP customers.

Auth0 is run as an isolated subsidiary in its own infrastructure, with the old CEO still overseeing operations. Due to the massive difference in Okta & Auth0's implementations I don't see that changing anytime soon.

That's what ASCAP does too! Scummy!

> Okta's response was that we signed an MSA agreement and that cancelling isn't an option, nor termination of fees.

Interesting. Does this agreement also works the other way as well (Okta can't just decides to terminate your account no matter the reason)?

And yet Okta is the ultimate in box-ticking technology. They are bought to tick the boxes. So what happens now that the box tickers are not ticking the boxes?

Usually a mass exodus to a similar service with the same guarantees resulting in months of capacity problems as they try and scale out from customer influx.

There are no winners.

They aren’t security controls at all. Just puffery.

I’d look at stuff like FedRAMP as a starting point for the control environment and explore further.

A lot of compliance theater comes from the requirements of insurance companies.

The decision makers have absolutely no idea how any of this stuff works.

tbh it is a reassuringly weak response.

Mostly picking apart logical inconsistencies in the language and / or re-emphasising the already disclosed info. Does not seem like they were able to produce any hard collateral to contradict anything Okta stated which probably means it's at least ball park accurate.

What happened to point 3

This physically hurts to read now that I understand the official position of Okta. I am eagerly looking forward to the unfolding of consequences.

What telegram channel is this?

It's https://t.me/minsaudebr.

I could read the posts, but when I clicked to read the comments I got this:

https://de.catbox.moe/ovt7t7.jpeg

I remember healing a while ago that certain Telegram channels would be blocked on iOS devices due to Apple's content policy, that's what this seems to be about. Update: Yeah I can view them on desktop just fine.

> Update: Yeah I can view them on desktop just fine.

Yeah, this is the fabled content moderation block for App Store distributed apps. The Mac App Store version of Telegram also blocks it, but the direct download dmg does not. I think the direct download apps are also updated more frequently, but I could be wrong on this.

Reply on news.ycombinator.com