Hacker News Reader

Comment by _yb2s | Hacker News Reader

Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates?

This is just basic IT common sense. You only do updates during a planned outage, after doing an easily reversible backup, or you have two redundant systems in rotation and update and test the spare first. Critical systems connected to things like medical equipment should have no internet connectivity, and need no security updates.

I follow all of this in my own home so a bad update doesn’t ruin my work day… how do big companies with professional IT not know this stuff?

View on HN

CrowdStrike let's you create update strategies and rollout groups.

This update bypassed all of those settings.

Well that context makes it make a little more sense... I still wouldn't be trusting a service like that for mission critical hardware that shouldn't be connected to the internet in the first place.

The question with these types of services is: is your goal to keep the system as reliable as possible, or to be able to place the blame on a 3rd party when it goes down? If it's a critical safety system that human lives depend on, the answer better be the former.

How do you keep an airline ticketing system offline? How would anybody book tickets without access to the databases?

Whitelist the persistent store?

you don't need to airgap it. just limit the access to the specific APIs/access to the database and block everything else.

CrowdStrike won't be able to upgrade itself through your database API...

If you must trust the software not to do rogue updates then I have to swing back into the camp of blaming the operating system. Is Linux better at this?

I've noticed phones have better permissions controls than Windows, seemingly. You can control things like hardware access and file access at the operating system level, it's very visible to the user, and the default is to deny permissions.

But I've also noticed that phone apps can update outside of the official channel, if they choose. Is there any good way to police this without compromising the capabilities of all apps?

Microsoft has tried pushing app deployment and management platforms that would make this kind of thing really possible, but it constantly receives massive pushback. This was the concept of stuff like Windows S, where pretty much all apps have to be the new modern store app package and older "just run the install.exe as admin and double click the shortcut to run" was massively deprecated or impossible.

you wouldn't be trusting it.

But that's besides the point in any enterprise environment. Or even in a SMB where third parties are doing IT stuff for you. Your opinion doesn't matter there. Compliance matters. Paper Risk aversion matters. And they don't always align with common IT sense and, as had been proven now, reality.

> Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates?

Because it lets them "scale" by having fewer and cheaper offsite IT and contractors to manage vs hiring pesky onsite employees.

You do that for antivirus definition updates?

Probably, implicitly. Have automated regular backups, and don’t let your AV automatically update, or even if it does, don’t log into all your computers simultaneously. If you update/login serially, then the first BSOD would maybe prevent you from doing the same thing on the other (or possibly, send you running to the other to accomplish your task, and BSODing that one too!)

But yeah this is one reason why I don’t have automatic updates enabled for anything, the other major one being that companies just can’t resist screwing with their UIs.

I’m not an IT professional, but I don’t use antivirus software on my personal macs and linux machines- I do regular rotated physical backups, and only install software digitally signed by trusted sources and well reviewed Pirate Bay accounts (that's a joke :-).

My only windows machine is what I would classify as a mission critical hardware connected/control device, an old Windows 8 tablet I use for car diagnostics- I do not connect it to the internet, and never perform updates on it.

I am an academic and use a lot of old multi-million dollar scientific instruments which have old versions of windows controlling them. They work forever if you don't network them, but the first time you do, someone opens up a browser to check their social media, and the entire system will fail quickly.

Yes. In an environment where you have so many clients that they can DDoS the antivirus management server, you have to stagger the update schedule anyway. The way we set it up, sysadmins/help desk/dev deployments updated on day 1, all IT workstations/test deployments updated on day 2, and all workstations/staging/production deployments on day 3.

what happens if there's a 0-day RCE? 72 hours of your production systems hanging out in the open...

The schedules are shockingly easy to adjust.

What people aren’t understanding is MOST of the outage isn’t caused by a crowdstrike install itself, it’s caused because something upstream of it (a critical application server) is what got borked, and that’s having a domino effect on everything else.

Reply on news.ycombinator.com