Hacker News Reader

Comment by ac29 | Hacker News Reader

In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks.

I regularly work with these sorts of water districts (larger, better funded ones as well). In reality, some of these small districts may only have 2 or 3 SCADA operators on staff. Sending them home with a pager, a tablet, a VPN password, and some overtime pay is a lot easier to get past the city council then taking on another two employees to cover the night shift for those rare events that need to be handled ASAP.

I could share some real horror stories, but it wouldnt be professionally appropriate. Suffice to say, this story did not surprise me at all.

View on HN

"In a perfect world, maybe there would be unlimited budgets for small rural water districts to have 24/7 onsite staff and run highly secured networks."

I reject this line of thought.

A small rural water district can run with looser tolerances and looser guarantees - and have done so for decades.

They should spend half the time (and a quarter of the money) setting up systems that fail safely and revert to known states and operate with looser tolerances.

As for telemetry ...

I am not joking at all when I say that a green light on the building that turns red and everyone in the county knows to call either Jed or Billy if that light is red is a completely reasonable system. It's a small rural water district (your words) after all, right ?

re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting.

I like your thinking and I try to espouse it myself (keeping things as simple as they can be-- keeping "technology" out of voting, not connecting things to networks that have no business being connected, etc). Short of a Battlestar Galactica-type "our machines rise up and try to kill us" event, though, I don't think the average person will ever understand the vulnerability inherent in networked computers or the risk/benefit tradeoff of connected vs. disconnected systems.

Even down at the level of local politics in a rural setting the "optics" of bringing technological solutions to bear on problems is seen as forward-thinking-- particularly when it "saves" the taxpayer money. I can't imagine trying to convince a local water board that moving away from a PLC-based system with a remote support vendor world fly, even citing this example.

This event will be another opportunity for more security vendors to cite in case studies justifying their products. More layers of garbage will build up on a foundation of protocols and design philosophies that grew up in an era of disconnected systems with lower stakes and a less complex threat model.

I don't see big money to be made in providing sensible levels of connectivity and security to this kind of infrastructure. I don't see industry stepping-up because of that. Maybe regulation is the answer, though I'd just expect regulatory capture to take over, and have it become another "PCI". Maybe a lot of people have to die before society takes it seriously, as has been the case with so many other safety codes over human history.

It makes me really sad, embarrassed for our industry, and more disappointed in human nature.

> re: alerting telemetry - I'm not finding photos to cite right now, but I've absolutely seen "infrastructure" buildings (here in rural Ohio) with warning annunciators (lights and bells) on their exteriors along with signs reading "If this light is flashing call xxx." It's definitely a viable system for alerting.

I can confirm these exist, as growing up my father's phone number was on a some of those signs. Fancy systems would automatically call an answering service, but for smaller ones, he'd get a call (or page, given the era), usually from a neighbor or something.

It wasn't limited to rural areas either. I distinctly remember him taking service calls in ultra-wealthy neighborhoods on the weekends. One time, the gentleman who called in let my sister and I watch movies in his personal movie theater while my dad fixed the system.

There is little problem on pushing telemetry into the internet.

The largest issues is that you must gather the data from sensors that can't interfere with the thing you are measuring, and that you must process it with computers that don't connect to the ones controlling the process. The first one is really just good engineering practice, and the second is already cheap and getting cheaper by the day.

Also, whatever you do at your process control, you should have some emergency overrides that set when the conditions get too abnormal. Those should be simple (AKA, no computers if possible) and stand-alone. Looks like they got this one right.

Plus the benefit of setting something up to get telemetry as you described, is that someone won't later be tempted to use teamviewer (as in the article example) to open up the whole control system just to view some of that telemetery.

I invite you to become a pentester for a year and see if you still reject this line of thought.

You misunderstand - I am saying they need neither the computers nor the networks.

By running with loose tolerances and loose guarantees (and keeping systems as simple as possible) they remove the need for these tools - and their attack surface.

pen testing often involves walking into a building looking like you should be there. If no one is around to let you in it also can involve half a minute of getting the door open with some primitive every day tools. No computers or software required.

You just switched the attacker model from "script kiddie somewhere in the world, changing stuff for the fun of it" to "physical presence with specific malicious intent". Those are not in the same ballpark, not even the same country.

You underestimate the stupid things local kids can get up to for fun if they thought they wouldn't get caught. So you can at least throw out the "specific malicious intent".

Ok, but how many stupid local kids are there and how many worldwide script kiddies?

Kids who are trespassing are not stupid. You have to take the risk, observe the behaviour of the security and then behave reasonably so that people would forgive you if you get caught.

The action is stupid, but trespassing idiots get caught quickly - that's just "survival of the fittest" mechanics.

If you never trespassed in your life then you were probably not smart enough to get away with it?

Security is a big word, we are talking about large facilities with like 1 or 2 people on site.

As a kid I've tresspassed plenty, it often takes 'security' hours to even spot you

I’m not particularly worried about a group of kids contaminating a single rural water supply, I’m more worried about coordinated effort by a malicious nation state to accumulate security cracks and then use them in a coordinated effort during a war time scenario.

I think it would be quite possible to have the control systems completely offline, while installing a reputable alarm system that is connected to the internet. If those two things have no network connection, then you could monitor the premises, but even a remote hacking of that system would not enable changing of chemical levels.

But you have this problem as well, even if you use networked systems. Now you need to mitigate both.

You must be a pentester, because the commenter above you wasn't talking about pentesting at all.

His comment says that small rural town water treatment plants don't have to run for 24/7. Not sure what you thought he was saying.

>You must be a pentester, because the commenter above you wasn't talking about pentesting at all.

This sounds like it should be a punchline to one of those funny programming jokes https://news.ycombinator.com/item?id=25850739

It would make waay more sence for the entire state / country to buy software for their water systems all at once and set up a department of a few dozen people that travel the country and make sure its setup properly. Why would you not have all water treatment in the country on the same software platform?

A lot of these problems with inefficiency comes down to "muh state's rights", and less federal involvement. I think my stance shines through, but I understand it's debatable per case.

You parent is saying: "just never put your stuff in the Internet". I suppose it really is "that hard" to tell clients this and we can expect more problem down the line.

How did they operate before computing became a mainstream thing? Utilities as a comcept are significantly older than computing itself, and certainly mainstream, internet-connected computation.

They had to pay more for employees who could be on site at unreasonable hours.

Perfect, unemployment is a major problem because of Covid19 anyway.

COVID19 is also decimating budgets

If all it takes to prevent the poisoning of an entire city's water supply is two employees, I certainly hope my governments are choosing to hire those two employees.

I will make an assumption here that you don’t live in a rural city...

Or, let's face it, in the United States.

Of all the commentary I’ve read on this issue so far, this might be the scariest anecdote so far.

Reply on news.ycombinator.com