Hacker News Reader

Comment by Tharkun | Hacker News Reader

This is mind boggling. Who installs these systems? Who maintains them? Surely this is supposed to be done by someone with at least a certain amount of clue? Enough clue not to hook up your insecure gear to the internet? No?

I don't even get how this happens. Surely these things are not just plugged in to a modem? There has got to be some kind of LAN involved. If there is, then there should at least be an edge firewall. Or even a simple garden variety gateway with NAT, which would already prevent all of those open ports from being accessible. So what gives? Are people deliberately hooking this gear up to the internet, deliberately exposing ports without taking security into consideration? That is patently insane.

View on HN

How much do engineers specializing in these systems get paid? I interviewed at a couple of places that do industrial-style systems and in one case, I got an offer that clocked in far below the requested rate (not a negotiation tactic, they declined to revise it), and at another, the recruiter couldn't hang up fast enough when she heard salary requirements.

It's something like the Idiocracy Effect. In Idiocracy, the present-day world's best minds devote all their energy to eliminating hair loss and prolonging erections, eventually leading to a collapsed society. The real world is not too far off. Our best minds are dedicated to improving ad targeting and keeping users in vanity-gratifying loops on social media, while our industrial control systems that literally run the modern world wallow in the 70s.

Everyone has some culpability in that. How many of us yearn to work for the local power utility (or one of their upstream vendors) rather than Facebook, Google, or the latest VC pump-and-dump scheme -- err, I mean, hot SV unicorn? Even if they did compensate competitively, are there enough hackers with a sense of [realism/duty/patriotism/INSERT_OTHER_SOCIAL_VALUE_HERE] to heed the call?

There's definitely a role for some type of regulation or standards here. Industrial controls should be considered vital infrastructure that require serious and immediate investment. A brief visit to Shodan will show thousands of exposed industrial resources, and that should scare your pants off.

Don't know about the US market but in EU salaries in the embedded/automotive/industrial control are crap compared to frontend/app kiddies.

It's definitely a tragedy of the commons, idiocracy style.

Lots of embedded work related to critical infrastructure is now dumped to Eastern Europe or Asia as the good people in this field are mostly gray beards which are deemed too expensive by management.

I used to work on those sorts of systems. I'm now making 3x as much at FAANG. I love working on systems where you own it top-to-bottom, but not going to turn down extra piles of cash every month.

How many of us yearn to work for the local power utility

I actually would (or a vendor as you say) if it didn't mean earning 1/3 or less as much money. I mean, what better way to give back to the world than to apply one's hard-earned skills to improving the technologies (water, gas, power, manufacturing) that make modern society possible? It's like real life Ark or Infinifactory.

But the pay is just not viable, and there are often stricter credential requirements than in valley software engineering.

As someone who worked in this industry -- it's peanuts, but they'll tell you its gold.

I walked out of my IA job with the boss screeching about how I was "making the biggest mistake of my life", how "nobody can match our prestige"... after a four-year pay freeze. Nearly a 20% pay rise right off the bat, and that was the initial offer.

The other thing about IA is there's no career progression for engineers unless someone higher-up quits (which usually triggers a deckchair shuffling or a hiring). If you're good at sales, you can progress into that -- otherwise your only option is likely to be management.

If you want to make cool things, IA is not the industry for you.

If you're happy making the same thing day-in day-out, just with a different coloured box? You'll fly.

If you can play politics and ruthlessly spike your colleagues, you'll get all the way to the boardroom.

May I ask how did you get out of industrial automation and where did you go after?

It’s not deliberate in the sense they want people to be vulnerable.

It’s the path of least resistance in action.

If you make it significantly more work to do something than not then people simply won’t do it.

As the visible difference between secure and insecure is invisible to management it slides.

As an industry we’ve failed all horribly at making secure the easy default option.

And that’s going to haunt all of us.

> As an industry we’ve failed all horribly at making secure the easy default option.

This will never and can never happen. Security and sharing are at odds.

If I don't want connectivity, I don't even buy a connected controller as it is more expensive. So, the fact that a connected controller got bought means someone wanted connectivity.

Of course, the person who wanted connectivity didn't allocate any actual money to the person actually doing configuration, maintenance, auditing, updating, etc.

Only at this point, do we have the fact that configuring a secured device is a pain. So, the default is to turn off all the security to get it up and running. After that, if someone can't tweak the software to turn the security back on in a couple of hours, it gets left off (remember--nobody actually allocated budget so these are unbillable, wasted hours).

All of this together collides into a nice big pile of fail.

> > As an industry we’ve failed all horribly at making secure the easy default option.

> This will never and can never happen. Security and sharing are at odds

I don't think the GP was suggesting that products be made so secure to be functionally useless. There's a wide spectrum between completely open and completely locked down. But right now the gradient to go from open -> secure is so steep that most people aren't willing to put in the effort.

For example, have you ever tried using OpenSSL to create a self-signed certificate? The defaults are probably not what you want. Adding on stuff like the proper certificate extensions is even more work. Products should be designed so that it's not a herculean task to add security when you're ready to take that step.

> It’s the path of least resistance in action

This. Combined with the fact that the handful of people who have the knowledge/experience to properly secure the systems are probably too busy/overworked to expend the time to improve the situation. Note that the time to improve the situation includes not only the actual time to implement the changes, but also the time to play politics and get funding/resources.

At least at the company I work at, management cared about security for about a month after Wannacry happened. Once all the PCs were patched for that specific issue, it was back to business as usual.

"This. Combined with the fact that the handful of people who have the knowledge/experience to properly secure the systems are probably too busy/overworked to expend the time to improve the situation"

That's not true. There's tons of people that could setup VPN's or whatever for them that are better than what we see. The actual problem is management doesn't care about security with no financial damages for that in most cases. So, they don't try to secure the project. I've known a few engineers doing embedded systems telling me how their customers almost always ignored warnings if dollars invested was somewhere above zero.

The above commenter said:

> Note that the time to improve the situation includes not only the actual time to implement the changes, but also the time to play politics and get funding/resources.

That's the problem. Explaining to the beancounters why this is needed and why Susan can't just RDP in from home anymore. The implementation of these things are fairly trivial, the hand holding takes forever. Add the fact that lots of us got into IT to avoid playing these games and you've got the current state of affairs.

I work on a safety critical piece of software. To give QE more than a day or two to test it is a fight every single release. After many unsuccessful attempts I stopped swinging at windmills. We pulled about half of the software releases we made last year due to patient safety issues discovered by customers. Management still wants "agile"

"Agile" and safety critical, that is terrifying to me.

It's common nowadays. The new edition of IEC 61508 that's being worked on is supposed to incorporate agile workflows.

Not sure if your comment was aimed at agile vs. "agile".

Agile and safety critical is fine. It works.

"Agile" is a problem, safety-critical or no.

> The actual problem is management doesn't care about security with no financial damages for that in most cases. So, they don't try to secure the project.

Nor should they, if they act rationally.

To be honest, I've started thinking that having a lot stronger regulation for punishing companies is the only way to improve the situation. GDPR is a good start in the EU, although it is privacy-focused.

The free-market solution to this is that users (be they consumer or B2B) start shunning the companies and products with the worst security records, but the problem is that security (and privacy, it seems) are so hard to grasp that it's just not happening. Companies exist to make money, and it's sad how many of them won't actually care about anything else. So if that is the only thing that they care about, the only option to make them care about security is to make insecurity a bad road business-wise.

It was painful to admit as much myself in a counterpoint I wrote to Bruce Schneier:

https://www.schneier.com/blog/archives/2011/06/yet_another_p...

I used to work in the industrial controls industry. The systems are often designed by application engineers working for the industrial control equipment’s manufacturer or distributor. In the case of distributors, the engineering work is often provided for “free” and paid for with the markup the distributor applies over their cost to purchase the components direct from the manufacturer. Those same application engineers will be involved with helping to make the sale. If a customer asks “can I connect this to the Internet?”, any response other than “of course!” is liable to result in a talking to from the sales manager for that account.

Few of the incentives in that industry lead to good security. More details here: https://news.ycombinator.com/item?id=3260127

> Surely this is supposed to be done by someone with at least a certain amount of clue? Enough clue not to hook up your insecure gear to the internet? No?

The way I understand it (and what my brief experience in writing software for industrial use confirms), it's the usual: the people making purchase decision have little to no clue, which - in absence of regulations - allows companies selling these products and services to not care, employ people without clue, and/or purposefully trade off quality for sales points (think of how many pseudo-features you can add to a PLC once you hook it up to the cloud).

I don't know about ICS, but CCTV cameras, HVAC controls, parking gates are too often installed by technicians with little to no IT knowledge. They ask the local IT to open port (ie. port forwarding) for remote management or servicing without knowing the implications. When offered a VPN they often refuse - they do not want or know how to use it.

My (not vast) experience with this stuff is that it’ll often get plugged directly into an LTE-Ethernet modem. Or, depending on the vintage of the equipment, the chain looks more like LTE-Ethernet modem connected to a “serial port concentrator” with Ethernet on one side and a bunch of serial ports on the other side. Sometimes the installers will turn on IP Whitelisting on the inbound ports, but sometimes that gets skipped.

I can't say much about Switzerland, but on a slightly related note, in Poland we have automatic doors in trains that don't bother detecting if someone's in the doorway before closing. I'm honestly surprised that these didn't kill a single child yet.

Security is not mentioned in the specification for the control system so the company building it does nothing.

When asked customer does not want to pay any more for secure remote access.

Reply on news.ycombinator.com