Hacker News Reader

Comment by scriptsmith | Hacker News Reader

It's crowdstrike: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...

> 7/18/24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.

> SCOPE: EU-1, US-1, US-2 and US-GOV-1

> Edit 10:36PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...

> Edit 11:27 PM PT:

> Workaround Steps:

> Boot Windows into Safe Mode or the Windows Recovery Environment

> Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

> Locate the file matching “C-00000291*.sys”, and delete it.

> Boot the host normally.

View on HN

Right after you enter the bit locker recovery key.

You do have your bit locker recovery key, right? .....right?

I doubt most of the clients who use CS know what BitLocker is, let alone how to back it up, assuming it wasn’t backed up automatically by Windows.

Doesn't that get backed up automatically to the Microsoft account?

That's opt-in.

In Enterprise setups the key should be backed somewhere in Active Directory.

Yes you should be able to pull it from your domain controllers. Unless they're also down, which they're likely to be seeing as Tier 0 assets are most likely to have crowdstrike on them. So you're now in a catch 22.

Log into hypervisor, rollback VM

Microsoft shops gonna be running Hyper-V. Probably also got hosed.

Rolling back an Active Directory server is a spectacularly bad idea. Better make doubly sure it's not connected to any network before you even attempt to do so.

In theory. I've seen it not happen twice. (The worst part is that you can hit the Bitlocker recovery somewhat randomly because of an irrelevant piece of hardware failing, and now you have to rebuild the OS because the recovery key is MIA.)

I know it does for personal accounts once linked to your machine. Years ago, I used the enterprise version and it didn’t, probably because it was “assumed” that it should be done with group policies, but that was in 2017.

Most people installed CrowdStrike because an audit said they needed it. I find it exceedingly unlikely that the same audit did not say they have to enable Bitlocker and backup its keys.

I can confirm this. EDR checkbox for CrowdStrike, BitLocker enabled for local disk encryption checkbox. BitLocker backups to Entra because we know reality happens, no checkbox for that.

This was particularly interesting (from the reddit thread posted above):

> A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest").

> His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.

I find that hilarious

Me too, as I am also not affected. But I do pity those guys who now try to solve that deadlock.

> but they can't access any of the documentation to do so, because everything is down.

One of my biggest frustrations with learning networking was not being able to access the internet. Nowadays you probably have a phone with a browser, but back in the day if you were sitting in a data room and you'd configured stuff wrong, you had a problem.

Nobody, not one person, thought that documentation should be stored in hard copy?

I'm guessing someone somewhere said that "it must be stored in hard copy in a safe" and the answer was in the range of "we don't have a safe, we'll be fine".

Or worse, if it's like where I worked in the past, they're still in the buying process for a safe (started 13 months ago) and the analysts are building up a general plan for the management of the safe combination. They still have to start the discussions with the union to see how they'll adapt the salary for the people that will have to remember the code for the safe and who's gonna be legally responsible for anything that happens to the safe. Last follow-up meeting summary is "everything's going well but we'll have to modify the schedule and postpone the delivery date of a few months, let's say 6 to be safe"

Not just financial / process barriers. I worked for a company in the early 90's that needed a large secure safe to store classified documents and removable hard drives. A significant part of the delay in getting it was figuring out how to get it into the upstairs office where it would be located. The solution involved removing a window and hiring a crane.

When we later moved to new offices, somebody found a solution that involved a 'stair-walking' device that could supposedly get the safe down to the ground floor. This of course jammed when it was halfway down the stairs. Hilarity ensued.

Any chance you have a link to that comment?

Didn't bookmark it or anything and going back to the original reddit thread I now see that there are close to 9,000 comments, so unfortunately the answer is no...

Here you go. <https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/commen...>

Just hope there is no mutual recursion, i.e. recovery key A is stored on machine B, recovery key B is stored on machine A!

Isn’t that what office safes are for? I don’t know the location, but all the old guard at my company knew that room xyz at Company Office A held a safe with printed out recovery keys and the root account credentials. No idea where the key to the safe is or if it’s a keypad lock instead. Almost had to use it one time.

BitLocker for Business stores the bitlocker key centrally. Still, it is a huge manual undertaking fixing every system.

Absolutely correct. Unfortunately, there is no other solution to this issue. If the laptops were powered down overnight, there might be a stroke of luck. However, this will be one of the most challenging recoveries in IT history, making it a highly unpleasant experience.

Yeah in context we have about 1000 remote workers down. We have to call them and talk through each machine because we can't fix them remotely because they are stuck boot looping. A large proportion of these users are non-technical.

How fortunate the phone system is not vulnerable to CrowdStrike...

Man, talk about a mass-phishing opportunity.

I heard the central system was on Azure, running CrowdStrike.

MS Windows Recovery screen (or the OS installer disk) might ask you for the recovery key only, but you can unlock the drive manually with the password as well! I had to do that a week ago after a disk clone gone wrong, so in case someone steps on the same issue (this here is tested with Win 10, but it should be just the same for W11 and Server):

1. Boot the affected machine from the Windows installer disk

2. Use "Repair options"

3. Click through to the option to spawn a shell

4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT.

5. In the shell, type: "manage-bde -unlock C: -Password", enter the password

6. The drive is unlocked, now go and execute whatever recovery you have to do.

Good luck.

On my corporate Windows 11 22H2 "manage-bde -unlock C: -Password" does not unlock the disk with the user key. I guess it needs recovery key as well.

Don’t you need more options if the key is in a TPM, or there is a password but it’s only part of the key?

Can you even get the secret from the TPM in recovery mode?

> Can you even get the secret from the TPM in recovery mode?

Given that you can (relatively trivially) sniff the TPM communication to obtain the key [1], yes it should be possible. Can't verify it though as I've long ago switched to Mac for my primary driver and the old cheesegrater Mac I use as a gaming rig doesn't have a hardware TPM chip.

[1] https://pulsesecurity.co.nz/articles/TPM-sniffing

TPMs embedded in the processor (fTPM) are pretty popular and it's a lot harder to sniff communications that stay inside the cpu.

yea I don't need an attack on a weak system, I mean the authorized legal normal way of unlocking BL from Windows when you have the right credentials. Windows might not be able to unlock BitLocker with just your password.

I don't know how common it is to disable TPM-stored keys in companies, but on personal licenses, you need group policy to even allow that.

Although this is moot if Windows recovery mode is accepted as the right system by the TPM. But aren't permissions/privileges a bit neutered in that mode?

help for if you have bitlocker turned on (because of course you do).

https://x.com/attilabubby/status/1814216589559861673

https://x.com/nathanmcnulty/status/1785094215749476722?s=46

GPO to fix:

https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217...

Saved to my desktop? How does that help? /s

Happy weekend to everyone who works there.

Can you post a summary? We're affected but I don't have access to that portal.

Someone posted this in the thread, but I also can't log in to verify

> Summary

> CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

> Details

> Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.

> Current Action

> Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.

> Status updates will be posted below as we have more information to share, including when the issue is resolved.

> Latest Updates

> 2024-07-19 05:30 AM UTC | Tech Alert Published.

> Support

> Find answers and contact Support with our Support Portal

They've bumped this support info to a blog post that's linked from their home page: https://www.crowdstrike.com/blog/statement-on-falcon-content...

It includes PDFs of some relevant support pages that someone printed with their browser 5 hours ago. That's probably the right thing to do in such a situation to get this kind of info publicly available ASAP, but still, oof. Looks like lots of people in the Reddit thread had trouble accessing the support info behind the login screen.

"Start your free trial now." Hahahahah you have got to ne kidding me :)

They had me at "crowdstrike engineering"

So engineer-like.

Reply on news.ycombinator.com