Hacker News Reader

Comment by waste_monk | Hacker News Reader

I've said it before and I'll say it again. The only way critical infrastructure should be connected to the internet is via data diode [1]. Otherwise it should be air-gapped.

It still lets you remotely monitor systems, but if you want to adjust the operation of the critical infrastructure you have to physically go there and do it in person. People often complain that this is inefficient, but if your own workers are unable to remotely affect a system then neither can an attacker - given the consequences of an attacker compromising a critical infrastructure I think the inconvenience is a fair price to pay.

[1] for those unfamiliar, this is typically implemented using two computers connected via fibreoptic cable, such that there are only transmitters (lasers) on one side and only receivers (photodiodes) on the other side. Thus the information can physically only travel in one direction. The computers at each end handle conversion from connection-oriented protocols (e.g. TCP) to connectionless (e.g. UDP) while maintaining reliability.

View on HN

I work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving to middle of nowhere to make a change, and no way those customers are going to pay me for three days what otherwise takes a few hours.

Your customers probably would be more willing to pay if they had to be insured against damages caused by hacking.

It’s an interesting question i was contemplating while writing the post, is it possible to cause damage by hacking the plants I work on? Exceeding conditions set forth in Environmental certificates, yes, destroying the machinery ... would be possible only because some modern equipment is used and connected to the network, if the Ethernet ports on the devices containing the sync-checks were disconnected, or an older dumb device where any setting adjustments are made with a screwdriver, then it wouldn’t be possible to damage about half of the plants. The other half with oil pumps you could cause some significant damage by telling it the oil pump is running when it is not and suppressing all the bearing temperature protection.

I wonder what if any cyber security requirements are imposed by their insurance companies.

> would be possible only because some modern equipment is used and connected to the network [...] containing the sync-checks

"How 30 Lines of Code Blew Up a 27-Ton Generator: A secret experiment in 2007 proved that hackers could devastate power grid equipment beyond repair—with a file no bigger than a gif": https://www.wired.com/story/how-30-lines-of-code-blew-up-27-...

YouTube video of demonstration: https://youtu.be/LM8kLaJ2NDU

But you’re forgetting second order damage. What if somebody takes out a large enough fraction of power plants in the middle of winter? Or in the middle of a heat wave in Texas? Casualties could be rather significant if the plants can’t be restored in time.

> Your customers probably would be more willing to pay if they had to be insured against damages caused by hacking.

This is actually the other way around. Insurance is a moral hazard. If you're paying for the insurance anyway then taking steps to reduce the insurance company's risk rather than your own is throwing money down the drain.

Meanwhile the insurance company as a third party has less control over your actions than your own company and is less able to make efficient trade offs, so any requirements they impose tend to be either ineffective or unnecessarily burdensome or both.

The problem is that publicly traded or municipal utility companies have pretty much the same problem where the employees/managers don't pay personally for damage and the people who do are too diffuse or far away to understand their exposure well enough to mitigate it.

It’s the premium you pay that quantifies the risk.

Until there’s no insurance company that quantifies the damages and makes a quote to insure it, it’s all “Snowden cranks waving arms” nonsense.

Once that is in place a new excel sheet magically blips onscreen and big maneuvers begin to minimize that premium and comply to the minimum standards the insurance will not classify as negligence.

> It’s the premium you pay that quantifies the risk.

But then what? You pay the premium. Then you comply with whatever "make sure your Linux servers have antivirus" rules the insurance company comes up with. Then you stop doing anything that costs a penny more than that because the risk is now on the insurance company, which makes the problem worse. And when everybody does that, everybody's premiums go up.

My experience was that the insurance company had reasonable and effective (if costly) requirements - the “install antivirus on Linux” requirements tend to come from payment processors who would be liable for exactly nothing if you got hacked.

One of my colleagues preaches - for the case of fire/damage etc insurance - to get a quote; do everything they require (as these are almost always things that have or would have saved them payment before, e.g. keeping beige boxes above floor level and under a desk - which mostly mitigates water damage from both floor flooding and ceiling leaks). And after they would insure you, once it is clear they are confident they have positive expectation from insuring you (and they know what they are doing) - that’s the time to reevaluate if you want an external insurer, or your own piggy bank.

With respect to hacking, this advice is less applicable, however.

In an efficient market (I know...) another insurance company would start offering lower rates if you implemented more effective measures.

Well, if the insurance is not looking for an excuse to go bankrupt they’ll survey the existing setup and either refuse to insure or ask an astronomical premium. The infrastructure operator will then negotiate a compliance progress plan to reduce the premium down to an affordable amount.

I’m not saying there aren’t other mechanisms to pervert this model, but as a first approximation this is how’s it supposed to work: as long as the operator is forced to buy insurance, and there’s no way for the insurance to get off the hook, parties are forced to develop a credible risk assessment and mitigation.

Here's the failure mode: The insurance company comes up with some prohibitively expensive and not cost effective at all requirements that would work (but bankrupt you) if you actually implemented them. Someone at the company feigns compliance with the rules.

Then, if there is a claim, the insurance company investigates whether you followed the rules, finds that you didn't, and denies the claim. The insurance company doesn't go bankrupt and doesn't care that this happened. But most of the time there isn't a claim so nobody on either side actually notices that this is happening.

I will take the free government money thank you very much. Maybe even hire another person to help make these physical site visits. If we don't take the government's money, they'll just spend it on weapons.

Private and public companies own power plants too.

Cool. Let's lobby for regulations that force those companies to hire us. We're gonna make so much money.

You’d make more money as a plumber

You're essentially you're willing to trade security of critical infrastructure (and potentially risk lives) for the convenience of being able to do your job from home.

The plant owners have made that call.

Is a vpn secure?

If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?

>The plant owners have made that call.

Arguably it should not be up to the owner entity as to the security requirements they need to meet. At least down here in Australia we're in the process of legislative reforms that significantly expand both the scope of critical infrastructure (most of which is privately owned) and the responsibilities and requirements of critical infrastructure operators (in terms of risk management, security plans, mandatory reporting, etc.)

>Is a vpn secure?

Teamviewer (which I understand was the vector used in the water system breach) or a VPN that hasn't passed the extensive testing required of secure cryptographic devices is not acceptable.

IF the industrial control systems were treated as a classified information system, were linked using an approved and properly managed [1] secure cryptographic device (on both sides e.g. back to a secure control centre, not to your house so you can work from bed), and generally had the rest of the protective elements (e.g. requiring personnel working on it to hold a SECRET or equivalent clearance proportionate with the value of the system) in place, that might be OK. But otherwise my position remains the same - it should not be remotely controllable via the internet.

>If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?

That might brick some cars on-board computers or "only" kill perhaps a few dozen people before people caught on and it was recalled - still tragic, but small in the grand scheme of things. Critical infrastructure is called that because if it goes wrong it can harm or kill people on a vast scale, so it should be held to far higher standards.

[1] e.g. keymat issued by whatever your equivalent to the NSA is e.g. our Australian Signals Directorate, and otherwise verified to be secure.

I could imagine a malicious defect in Tesla firmware that would cause all Tesla vehicles on the road to make an aggressive manoeuvre that would most likely result in a crash at the same time. So I think the potential scale for death and injury With remotely updatable vehicle firmware is large.

Can you recommend any VPNs that are up to the required standard for critical infrastructure?

Although since the power plants I work on would struggle to kill even one person and damaging them would only cause financial hardship for the owner or an insignificant exceedanve of environmental regulations, so they must not be critical.

Wireguard is the thing I would use. The fact that non other than Linus Torvalds called wireguard "a piece of art" may or may not be a selling factor.

It is small, lightweight, reliable and does what it is meant to do. It uses existing, trusted network tech in the right ways and due to the low complexity you reading the source code is an realistic option.

Does wireguard support storing keys in a hardware device, so that the cannot easily be exfiltrated if the machine is compromised? You want that for security critical infrastructure.

AFAIK implementation in Linux kernel not yet support any hardware-backed handshake, but it's mainly because WireGuard developers didn't consider code stable until March of 2020.

But there are userspace implementations and I guess it's should be doable to make them talk to a hardware device.

>I could imagine a malicious defect in Tesla firmware that would cause all Tesla vehicles on the road to make an aggressive manoeuvre that would most likely result in a crash at the same time. So I think the potential scale for death and injury With remotely updatable vehicle firmware is large.

It's pretty low risk AFAIK, generally cars infotainment and CAN bus are isolated, and I understand Tesla vehicles can't update while driving. Who knows, I'm not familiar with their architecture, but I'm not too worried about it.

There are attacks but IIRC they rely on GPS spoofing, or tricking machine vision to cause the vehicle to swerve or accelerate, you can't just drive someone off a cliff over 4G (at least without implanting hardware into the car).

>Can you recommend any VPNs that are up to the required standard for critical infrastructure?

Approved devices vary by country, but this list is a decent example: https://www.ia.nato.int/niapc/Category/IP-Encryption_25

Unfortunately it's hard to provide info when these kinds of products are not generally available for purchase outside of the defence community, and the user manuals and related documentation are themselves restricted. But generally speaking if it's acceptable for use with classified info then it should be adequate protection for critical infrastructure.

>Although since the power plants I work on would struggle to kill even one person and damaging them would only cause financial hardship for the owner or an insignificant exceedanve of environmental regulations, so they must not be critical.

They might be relatively harmless in isolation, but it could be really harmful if an attacker could compromise a dozen of these small plants and disable or destroy them at the same time so there's less reserve capacity in the grid to pick up the slack when you launch your main attack. You don't even need to have that big of an impact on industry or what have you, just causing a blackout on a hot day and killing a bunch of elderly people via heatstroke would whip people into a panic.

Or perhaps an attacker would compromise one plant and use it as a vector to attack someone like you next time you connect, gain a pivot point via your workstation to set up a persistent backdoor and steal confidential documents and email so they know your schedules and organisational weaknesses, steal your VPN credentials and attack multiple power plants at once when it suits them, etc.

Thanks for the VPN link.

>They might be relatively harmless in isolation, but it could be really harmful if an attacker could compromise a dozen of these small plants and disable or destroy them at the same time so there's less reserve capacity in the grid to pick up the slack when you launch your main attack.

I suppose it depends on what is considered a small power plant. I would say anything less than 20MW is small, greater than 30 MW is medium, and greater than 100 MW is large. Simultaneously tripping a dozen 20 MW power plants would result in a loss of 240 MW of generation, about the same as 2 large units. There is 258,000 MW of nameplate generation on the western north american grid and a peak load of about 150,000 MW. The link below [0] shows that transmission level operators are looking at 1000's of MW of lost generation to have a measurable impact on the grid but losing that amount of generation causes operational headaches and not automatically an outage.

[0]: https://www.wecc.org/Reliability/2013-10%20WECC%20JSIS%20-%2...

Do you have more info on the changes to Australian infrastructure legislation?

It hasn't been passed yet, but there's some info here:

https://www.homeaffairs.gov.au/reports-and-publications/subm...

https://www.aph.gov.au/Parliamentary_Business/Bills_Legislat...

The CI cybersecurity reforms are part of the overall 2020 cybersecurity strategy (https://www.homeaffairs.gov.au/about-us/our-portfolios/cyber...), there's a lot of changes happening in the sector at the moment. If you have an interest in cybersecurity now would be a good time to grab some certifications and be ready to jump on board, as there's expected to be a big demand for cybersecurity professionals over the next few years to support these initiatives.

This looks like a decent summary of the proposed changes: https://www.ashurst.com/en/news-and-insights/legal-updates/m...

> If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?

OTA updates to car firmware is widely considered to be a stupidly irresponsible idea. I don't think we should justify doing risky things by pointing at other people doing risky things, who only get away with that because of clout and regulatory wheels turning slowly.

> OTA updates to car firmware is widely considered to be a stupidly irresponsible idea.

On one hand, it opens the door to hacking, pushing a malicious update, etc (stuxnet much?). On the other hand, if it is never updated, then we will end up with a zombie network consisting of lightbulbs, fridges, and teapots (yes!)

Edit: sorry I could not resist adding teapots (HTML error 418)

I love the concept of an electric car. Self-driving not so much.

OTA updates, and car security in general, is a joke it's so bad.

Teslas are great cars in many ways, but cybersecurity is not one of those ways if you go by pentest reports. I stopped at listing five below, but there are quite a few more out there.

https://www.forbes.com/sites/leemathews/2020/11/23/researche...

https://www.forbes.com/sites/daveywinder/2020/02/19/hackers-...

https://electrek.co/2020/06/10/tesla-hacker-unlocks-performa...

https://www.zdnet.com/article/tesla-car-hacked-at-pwn2own-co...

https://insideevs.com/news/356769/video-tesla-model-3-hacked...

If there's an emergency at a power plant and you need the sysadmin to personally fix it, then that has the potential to be just as big of a safety issue as a malicious actor is. It's trading one form of security for another.

I think this is even scarier, that there can de emergency where sysadmin should manualy fix it. There should be fallbacks and redundancies with muliple levels if this is realy a crytical system. These would should give enough time to transport whatever personnel you need on site to fix things.

Thanks for speaking up, rhodozelia. The currently prevailing voice of HN appears not to be on your side, but you have 30 customers who are on your side, and you probably represent the views of many. We need to hear more of your side.

I wish more attention (money) was given to cyber security on infrastructure projects, and all projects. Cyber security is a mess everywhere. How Many organizations in the USA have not been hacked or are Unhackable? The only one I would guess is google and maybe amazon? Banks and financial markets seem competent as well.

Banks generally have terrible track of record for security across the globe. But they usually have large and agressive legal departments so no sensible white hat ever gonna touch them with a ten-foot pole. At least they wont ever be allowed to publish results of security audit.

I worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.

Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on.

If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.

> Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on.

Don't power plants have all kinds of potentially dangerous mechanical systems under computer control? Overpressure on a fuel line causing the pipe to burst and fill the building with natural gas, things like that?

There aren’t nearly as many things as you might think that have unbounded failure modes like that. At the mechanical and electrical engineering level you have fuses, breakers, pressure relief valves, etc that make all of these failure scenarios just “the machine shut down”.

And then the machine shuts down and you have to fly there anyway to fix it.

This sounds like turning "an attacker may destroy power generation infrastructure and maybe kill people" problem into "an attacker may repeatedly DoS the power generation systems, likely damage it, and still probably cause loss of life by second-order effects". Doesn't feel all that different.

To use an analogy: it's like saying that attackers accessing the company servers remotely aren't a problem, because the best they can do is to keep them in a reboot loop...

Your analogy made me chuckle.

the western grid has 258,000 MW of generation. renewables go up and down like yo-yos, power plants go on-line and offline all the time and a large and always increasing fraction of it is unscheduled and uncontrolled.

Of course it depends on the size and location of the power plant, but there are many more small plants that can go offline without causing any significant effect to the grid than there are large ones, and the large ones probably have the resources to be manned 24/7 either on site or from a remote site over a dedicated and thereby secure fibre link.

One take-away from this thread is that not all infrastructure is 'critical'.

Crappy analogy. There is a huge difference between being able to shut down some power generation that is a drop in the bucket on a grid vs overloading some equipment and killing people.

One is life and death, another is lost revenue for a power plant.

If you think infosec people are paranoid, boy have you not met a good mechanical engineer yet.

Down in the analogue world, random overpressures, voltage spikes, simple mechanical fatigure, etc happen all the time, so you're going to have to deal. I've written reports on why, regardless of what a sensor said, a simple static beam made of nonflammable materials neither a) spontaneously combusted, b) broke Clausius's principle and moved heat from a cold body to a hotter body. But we still checked.

I agree on this, if something bad happened most critical system are designed to break somewhere to limit the catastrophe, or other times they are over engineered 1,000x because sometimes you can never tell what nature can do to your system.

I wish to believe that. I'd expect it of good engineers.

However, I don't trust systems to remain as robust as initially designed, after decades of undergoing constant pressure to reduce costs, make things more (fiscally) efficient, "cut out fat", etc.

Even the loss of institutional knowledge can degrade the safety of a system over time.

Ever had to start maintaining a codebase after the greybeard who made it retired? Did you have any documentation to go off of? Anyone remaining in the company regard it as anything other than a production critical black box?

Now imagine that codebase is an industrial process, with 20 years of growth, add-ons, changes. Probably some mismatching control systems, and god only knows if the as-builts or red line plans for the additions are all correct or rectified with each other..

Certainly the systems are designed to. But over the operational life, things happen. Systems degrade, maybe maintenance lapses a bit.

Maybe, for example, a mechanic tired of trouble shooting an issue replaces the 9th blown burst disc in a row with one he just cut from the mud flap on his shop truck. NBD, the system was only running 5% over pressure anyway and he wants to go home.

Later, this overpressure state backs up through the system and overheats a pump which, considering that it is pumping liquid explosives, isn't ideal. Maybe this trips a temperature sensor and the jury rigged burst disc is discovered. Maybe it doesn't, or the sensor was bypassed or silenced, and the pump explodes.

I'm rambling, but this is a real world example. My point is that these various factors: designed safety monitors like sensors or fail-safes play hand in hand with over-engineered systems as well as the security of the control systems. Any one part of the system should not be allowed to lapse or be less secure with the idea that another level of the system will catch the problem before it becomes life endangering.

Mechanical, chemical, regulating temperature, regulating flow, regulating speed. Imagine chemical X reached 10C over the recommended and starts a chain reaction. Or imagine sensors going offline. I was once working on a food-producing factory, and we had our shares of accidents, though not lethal. Any type of machinery with fluctuating power supply may be dangerous unless there are automated killswitches, release valves, etc. in place.

Power failures (especially long ones) can kill people

If the power plants would shut down or be severely damaged, wouldn't the resultant lack of power cause all sorts of damage, including things like accident victims not being able to reach emergency services and all the other classic ripple failures? I can imagine there wouldn't be any direct casualties in the plant, but I can't imagine power plants that upon unplanned shutdown would not cause severe damage, including physical harm and death (albeit indirectly). Even power disruptions of a few hours cause deaths.

Syncing a generator out of phase is exactly what was trialled in the Aurora Generator Test: https://en.wikipedia.org/wiki/Aurora_Generator_Test

Didn't go well for the jenny. (Though a turbine might have fared better)

Does your power plant deal with any steam? Gag a few safety valves and find out the hard way.

So did anyone of your deployments feature on one of the presentations of @viss - https://youtu.be/hMtu7vV_HmY

so if the customers would pay you to drive to the middle of nowhere to make the change you still wouldn't do it?

I'm a bit more concerned why one person is critical for 30 power stations. Surely there should be at least a few others that could be called. What if you were on holiday, incapacitated or just quit?

SCADA systems can kill people, any computerized system that can kill people must be protected as a weapons platform, therefore SCADA system must be protected as if they were weapons platforms.

Wait, isn't that a bit over-dramatic? I mean sure, that's true of nuclear systems. But I mean, my home thermostat? My car's smart key system?

As for the thermostat - it's winter below freezing and a hacker turns your heat off during the night. Sure someone might wake up, and it would have to be very cold for you to freeze, but if you have someone old or a child then it could be life threatening. Depending on the heater it might be able to make it cause a fire.

Many car's smart key system is on the same network in the car as everything else, including the brakes. Read more in https://money.cnn.com/2014/08/01/technology/security/most-ha...

So no, it's not being overly dramatic to categorize any SCADA system as a weapons platform. It might be, as they say, 'an abundance of caution', but I am fine with erring on that side than erring the other way.

Three Mile Island can be viewed as caused because one gauge was not being tracked correctly (where it was supposed to be was being tracked, not where it actually was). What if remote access was enabled on systems in a nuclear power plant, so that people didn't have to drive all the way there? A hacker might manage to replicate some version of Three Mile Island. I can hear the 'but it's secured' response from some already, but any system that can be accessed can be hacked, given enough time and resources.

The key thing people should realize is that cybersecurity does not prevent hacking, it attempts to delay it until a real person can take corrective action to protect the system. This is why you have defense in depth and rings of security. Essentially, it is a door lock. The better the lock, the more skilled class of thief it will dissuade by being 'too hard to bother'. A skilled thief who has state of the art lock picks, or is willing to use a door ram, is not going to be deterred by even the best of locks (think organized crime or state actor hackers here). But if our imaginary door only opens one way no matter what, to let people out, and can physically never let people in - then even the greatest thief is SOL. That is the purpose of a data diode.

The other other concern, and it's not minor, is that any time you start measuring safety in terms of cost effectiveness you are putting a value on human life. It is saying that is is cheaper to pay the settlements and/or fines resulting from deaths due to insufficient safety precautions than it is to make it safe to begin with.

It seems like a big jump from the kinds of idiotic security lapses we’re seeing to a full air gap. What’s wrong with proper authentication and encryption?

It’s not a nuclear power plant. There’s no crack team of snipers to kill anything that gets too close to the fence. Physically these places are certainly not wide open, but neither are the fences, doors, and locks any more intense than what’s on the local high school.

I think scale is the answer to that. These systems are being scanned for day and night for vulnerabilities from attackers and once they have infiltrated can lie in wait to do something whenever they want. A foreign actor can do that at massive scale undetected (both when successful and when not).

Same isn't true for physical security.

Well, put too much inconvenience and some "smart" guy will circumvent it anyway.

I have not personnaly worked in industrial environment, but from people I know, security wise, it is very scary at some places. Here is one such example of a "smart guy":

The IT crew of a factory have a sane policy of deny all traffic, except some well thought out rules.

A supplier would like to open a remote connection for maintenance of the equipment. Since the process for opening some traffic for a short duration is so convoluted, they finally install a 3G card inside the equipment (without telling their customer, and they even make it a habit for future install) so that they can do maintenance whenever they want.

The IT crew still thinks all is safe from outside...

That's pretty much sabotage.

Did someone in this story report it to the company management? This sounds like a clear lawsuit (or at the very least, contract termination) waiting to happen, and one that would scare other vendors away from such "clever" workarounds.

I don't know about management, but the guys on the factory floor were aware and happy with it. It made their job more convenient too, and the supplier was marketing its solutions as secure, so don't worry ;-)

> the guys on the factory floor were aware and happy with it. It made their job more convenient too,

Oh, I don't doubt that :).

> the supplier was marketing its solutions as secure

... but I very much doubt the truth of that marketing :).

I think it's still a ticking liability timebomb. At some point some external auditor will discover the backdoor to the plant and heads will roll.

The 3G thing is extremely common. Even with HUGE suppliers. Including Siemens Energy. Their steam turbine systems come with a 3g connection as standard. I even think they won't do 24h/7d support without it.

This, this right here is the correct approach. Unfortunately, it is rarely taken, as economics are directly opposed to it. Usually you have a mixture of company wide LAN, with directly connected machines, accessible on demand via VPN from service-contractors.

My fallback approach is, to insist on machines incapable of self-destruction and Whole System BackUps. Its easier & cheaper, to unplug, replace some spare-parts, reinstall some acronis images and go. Sad world, but real world

The data diode seems like an interesting idea, but couldn't that still enable a hacker (one who knew very well what they were doing) to increase the amount of lye in the water?

If you have the diode like:

plant < diode < internet

you could control it but not get any feedback. I meant

plant > diode > internet

so you could monitor it but not control it from the internet.

oops, duh, thanks for clarifying.

Reply on news.ycombinator.com