Hacker News Reader

Comment by nimbius | Hacker News Reader

i work for a diesel truck maintenance and repair shop and its been hell on earth this morning.

- our IT wizard says the fixes wont work on lathes/CNC systems. we may need to ship the controllers back to the manufacturer in Wisconsin.

- AC is still not running. sent the apprentice to get fans from the shop floor.

- building security alarms are still blaring, need to get a ladder to clip the horns and sirens on the outside of the building. still cant disarm anything.

- still no phones. IT guy has set up two "emergency" phones...one is a literal rotary phone. stresses we still cannot call 911 or other offices. fire sprinklers will work, but no fire department will respond.

- no email, no accounting, nothing. I am going to the bank after this to pick up cash so i can make payday for 14 shop technicians. was warned the bank likely would either not have enough, or would not be able to process the account (if they open at all today.)

View on HN

> our IT wizard says the fixes wont work on lathes/CNC systems

Why, whY, WHY...are these things connected to the internet?!

And why are they running windows? And why are they running Crowdstrike? WTF

If they are offline it should not matter which OS they run, maybe a RTOS for the control software but anything goes for the ux.

If they are online, well...

Not sure what the OS has to do with CrowdStrike's fuckup. CrowdStrike also runs on Linux and macOS.

And yet, of course it happens to Windows.

This thread has multiple anecdotes of the same happening on their Linux version earlier.

The boot loop / BSOD issues are Windows specific.

It's probably the remote computer that's running Windows and currently affected by Crowdstrike.

CNC literally stands for "Computer numerical control". They're like the OG 3D printers, they just work subtractively than additive, and at much much much better precision.

You absolutely need computers to control them and loading up models via USB sticks becomes annoying rather fast, so naturally the control computers are network connected.

"Network connected" or "conveniently programmable" !== "Internet connected"

It was a rhetorical question. I'm sure the GP knows what the machines are and why they might need some kind of convenient data supply.

Both manufacturers and on-site IT teams have simply gotten cavalier about internet connectivity, network isolation, automatic updates, etc -- convincing themselves that the catastrophic risks that come along with these processes will either not happen to them or will only happen when someone else can be blamed.

> WHY...are these things connected to the internet

Because the manufacturer makes sure they don't start up if they're not. Otherwise how else would they be able to spy on you?

And charge you

Source?

https://slate.com/business/2017/04/a-garage-door-company-s-r...

That’s not a lathe nor a CNC system. Again, which CNC manufacturers are installing windows + crowdstrike on their machines just so they can spy on their customers? You’re all just spreading conjecture. This attitude isn’t at all as widespread (nor profitable) in low(ish) volume B2B hardware sectors.

These industries have terrible track records wrt security and even software robustness, but they don’t routinely spy on their customers for weird marketing reasons. If there’s remote connectivity it’s for real reasons (eg remote maintenance, updates etc).

The suggestion that CNC machines run internet connected windows+crowdstrike just so the manufacturer can spy on their customers strikes me as pretty ridiculous and your garage door story doesn’t really relate. Much more likely that they do it for (possibly bad) non-malicious reasons.

some of them even have GPS. To prevent selling to sanctioned countries or reselling in general.

Why, whY, WHY...are these things connected to the internet?!

It's so that the support engineer at the manufacturer can log in to troubleshoot. And then company IT support sprinkles a layer of antivirus on top. That's how we got here.

>> Why, whY, WHY...are these things connected to the internet?!

Because SCADA systems. It's worthwhile to have an overview of an entire plant up in the main office. You can easily see what's running, what's not and what's got problems that need fixed.

Now for a small shop running jobs individually, they should definitely NOT be connected to the internet or even the LAN. But hey, some people think a thermostat needs to be on the network so there's that...

So the manufacturer can sell you a "cloud connected service plan" where they change the font once every six months.

Remote monitoring, analytics and diagnostics have a significant impact on uptime, utilisation and profitability. You're thinking in terms of a single machine, but the managers of machine shops are thinking in terms of a complex process across many machines and often across many sites. Some of that functionality could be delivered using an airgapped network, but a lot of important features essentially require an internet connection.

An embedded controller can deliver all of this information over a serial line to a central hub.

For our entertainment in times like this, of course!

grabs a bucket of popcorn and takes cover

Tinfoil hat: Government might want to track/limit/<remotely brick> CNC machine usage someday to say prevent weapons manufacture and encourages this behavior in a similar manner to the way it encourages social media platforms to censor speech. Some of the really advanced CNC machines have GPS in them and won't work in "bad" countries.

Why are these things running Windows!?

Almost every system you interact with in the world has some critical thing in its innards running windows.

Most stuff is fine in France.

Not according to Air France. https://www.connexionfrance.com/news/air-france-flights-and-...

No Crowdstrike salespeople in France?

We are using OpenOffice as firewall

https://www.numerama.com/politique/12508-albanel-le-minister...

I like the idea that technology is so unreliable in star trek because the computers are all centuries of software accretion with Windows way down the stack somewhere.

The late great Vernor Vinge explored this in A Fire Upon The Deep. One of the characters is/was in a former life a programmer-archeologist. The idea being that so many thousands of years in the future every relevant program has already been written, so his job was to comb the archives for the right mix of codes and integrate them, rather than right something new.

“So we've got this CNC controller written in Rust from 2036, and, ah, here is a GUI for something like that written in late 90's Visual Basic 6… Just combine those and…”

“So uhm, you do know what you are doing, right?”

“Sir! I am a programmer-acheologist! Oh this is fascinating… Hold on, I must unearth and preserve this beauty of a BAT-file before we can go any further.”

More specifically why are they running extra endpoint management software that receives automatic updates from the Internet...

This is basically the IoT apocalypse scenario (AC is down??) but ironically not affecting many IoT devices, I assume.

A lot of industrial machinery is just $x00,000 of equipment strapped to a windows pc. Hell, a lot of it is strapped to a version long EOL

Can confirm this is the norm in machine shops. I encounter systems running dos, 3.1, 95, 2k and mostly XP constantly. I rather prefer the old dos systems of the obsolete stuff. Less variables. It is easier and more reliable to freeze the tech in time than it is to manage updates.

My last CNC job was just a 98 pc that dropped into dos to load programs, this must have been right around when win10 came out. Sneakernet and floppies made it secure enough, but the main network where all the orders were handled was... terminal based.

OS is irrelevant in this case, and CrowdStrike deserves all the blame. They literally brought down Linux systems earlier this year. https://news.ycombinator.com/item?id=41005936

It's still a valid question, just not directly related to the crashes.

Because almost everything industrial runs windows because that's what the devs of those companies were most familiar with since MS-DOS days and evolved organically over time to modern versions of Windows due to great backwards compatibility and platform familiarity.

Right but typically embedded systems run Linux, because while Windows has great compatibility on x86 it's virtually worthless outside of that.

Those aren't embedded systems though, but mini PC computers. And embedded systems often run bare metal C code, not always Linux, especially for spindle/servo control where they get their commands from that PC.

The automation world runs on windows

Because Microsoft is giving away licenses to unis, esp in developing countries. IT jobs are seen there as a way to earn a good living and you get hordes of people who know nothing but Windows. That's how you get into the situation where most of the toolchains for embedded systems run on Windows, software for embedded systems is written on and for Windows, and so on. And then, one botched update fucks up everything.

There are a lot of things running Windows because it's pretty straightforward to write a user-mode driver to interact with custom hardware compared to Linux, where every driver needs to be in the kernel and built with the kernel. Yes, there's DKMS, but it's still more of a faff than the relatively plug-and-play mechanism that Windows offers, especially since Vista.

props to your IT guy for setting up in house phones. I believe these should be kept and expanded upon.

Feels like there is some sort of lesson we should learn as a society.

Not trust Microsoft to be the operating system of the modern life?

Yes, monocultures are disasters in the making.

Your IT wizard is probably wrong. There is a fix that involves booting into safe mode and deleting a file.

Unless you have an encrypted file system this should be a relatively trivial fix.

CNCs might not allow direct Windows access for end-users and require on-premise support from the manufacturer. Our cnc can be remotely serviced… if Windows boots.

If you’ve got physical access to the machine it’s your machine. All you need is a USB port.

I’d expect that the manufacturer puts out their own fix which basically copies crowdstrikes suggestion. I’d even suspect it by the end of the day today.

The fix is really simple, and luckily also very simple to automate. It’s going to be a lot of running around for IT staff (if deputized helpers!) but this should all be over by the weekend.

No it really is not. If you have a service contract, you do not touch it.

Nobody is going to try physically tampering with the HMI attached to their 50k$+ machine when you have a support contract indeed

> If you’ve got physical access to the machine it’s your machine. All you need is a USB port.

You're a few years out of date here. Physical access is not the end like it used to be. We live in an era of hardware-backed anti-tamper and signed loaders/kernels.

If you have a way around it, I suggest you start reaching out these companies because you could make a lot of money.

Fair enough and I've been out of IT for a while. I wish I was still in it though, I'd love to be working on this!

So the CNC manufacturers pay for crowdstrike licenses? That's crazy

They probably have to due to CrowdStrike lobbying and fear-mongering requirements for their kind of software into export-controlled hardware.

Yes because compliance requirements say “EDR must be installed on all machines”.

Out of curiosity, what set of requirements would that be for?

I’ve seen comments mention banking, or privacy references maybe when handling SSN and birthdates together (airports, hospitals?).

SOC2, PCI, FedRAMP, cyber insurance. Just about any cybersecurity related compliance will have "All machines must have EDR."

Machinery shipped to users usually do not allow for the users of the machinery to "boot into safemode". Thank John Deere and the anti-"Right to Repair" crowd for that.

I doubt it was shipped from the factory with CrowdStrike, and if they had enough access to install it, they have enough access to fix it.

These things are "cost optimized" and don't feature the kind of remote management iDRAC/openBMC/piKVM that would allow it to be remotely fixed. Embedded windows connected to the internet is super ***.

You can doubt, but you literally don’t know.

Reply on news.ycombinator.com