Hacker News Reader

Comment by amenghra | Hacker News Reader

from https://twitter.com/zackwhittaker/status/1358868187656388611:

    I can't immediately verify the veracity of the claims made by the sheriff but,
    the fact that the authorities *set up* a public-facing and/or remotely
    accessible system that allowed someone to change the water chemical levels is by
    far the bigger issue here.

View on HN

I worked at a water treatment facility for a few summers, and the SCADA system there was on a physically separated network. Actually, there were two SCADA networks, one for each of the plants, with the distribution system (the water towers and pumping stations randomly scattered throughout the service area) attached to one of those networks. I don't know how secure those remote links were, but I suspect they were the easiest ingress into the network.

A couple computers did bridge the two networks, but (IIRC) they were simple embedded systems doing read-only access (for compiling reports). I know when they did a pen-test, the pen-tester could compromise most of the corporate network (including service accounts), but they couldn't punch through to the SCADA systems.

I'm familiar with the systems you outline, and yes, those are more difficult to penetrate. However, those systems are significantly more expensive and more complex than the simpler ICS systems. Oldsmar Fl doesn't sound like a place that could afford such a system. Of course, can they afford not to have higher security systems is an open question?

The biggest cost of having physically separate networks (or at least network separated) is the HR cost of increased staffing and on-call requirements due to not being able to support the system remotely.

For a small city, it's non-trivial.

There has been for quite a while a big concern that industrial control systems are accessible, often poorly hardened (and by that I mean to the extent of having default passwords), and quite vulnerable to attack.

The only thing surprising about this is that we don't hear about it tenfold more.

Absolutely.

Meanwhile, we live in a world where VPNs are sold to the casual user while critical systems are left on internet facing networks.

I've never understood why, if these critical systems need remote access, it's not all done through a VPN of some sort. VPNs are not infallible, but it significantly increases the bar for entry from script kiddie to nation state real quick (depending on choice of crypt), while choosing a well supported implementation ensures long term bug fixes and security patches.

In all honesty, why is a system so critical on the internet at all? People say, ease of administration, but there are other methods of achieving the same thing. Up to and including running your own network. On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone. On the other, the people in your community won't be put in harms way.

> On the one hand your engineers and chemists won't be able to fiddle with the aeration stage using their brand new, whiz bang, iPhone

Maybe not at home, but couldn’t they have a local 802.11 network set up for this?

No idea why this wasn’t. Often times they’re not for this reason _and_ because the hardware itself is too difficult or impossible to get online and can’t be upgraded. Forget networks for a moment - a system running Windows XP is way less risk than an upgrade to Windows 7. Plenty of companies have older systems running vital hardware that if it went offline could cause massive outages, revenue loss being only one if the impacts. So air gapped networks are pretty common in ICS environments as a result.

After meeting enough SAP consultants in the ICS space, all I can say is I'm shocked it doesn't happen every day.

> The only thing surprising about this is that we don't hear about it tenfold more.

If you're someone who stands to gain from disrupting a nation's infrastructure... you don't tip your hand until it most benefits you.

If it really is the case that large parts of the infrastructure are very unsecure, expect to hear about it all at once, instead of little by little.

Water seems like a really weird system to sabotage though - power can bring businesses offline in a serious way but a city reservoir likely isn't supplying any businesses with a real need of water for any sorts of industrial needs... It's more of an inconvenience. Messing with chemical balances in particular seems like a prank or someone really twisted trying to give a bunch of folks long term health complications.

Sometimes attacks are probes and discoveries meant to determine or validate efficacy of a set of attack vectors including but not limited to human assets. Other uses are for distractions from other efforts. And yeah, sometimes they’re pranks. It’s not clear with the given facts what’s really going on.

You can't cause a rash of serious short term problems that increase the load on your health care system? That would be pretty compelling from a terrorist perspective or for nation states trying to demoralize/reduce trust in the current government.

Depends how long you can shut it down for. Even if only a few hours, an _unexpected_ shutdown of water across an entire city would certainly cause a panic with people descending on any stores open to buy water.

To be honest, these days I just take it as a given that all critical US infrastructure (the power grid, hospitals, and now apparently water treatment plants) is riddled with time bombs, and that if we were to ever get in a shooting war with the countries which put them there, they'd all go off at once and we'd be in a world of hurt. I hope government/military planners are making the same assumption.

It would be sort of darkly amusing if we've done the same thing to other countries, and so time bombs in infrastructure essentially replace nuclear weapons as the guarantors of Mutually Assured Destruction.

Seems to me that the real issue is lack of security, not the fact this system exists at all. Eg Every cell tower has remote access protocol and we rarely hear about those being hacked.

There's probably 100x more cell towers than there are water plants. The impact of hacking a cell tower isn't direct loss of human life (granted, knowing off a large number of cell towers would be very disruptive). The answer to the question "should it be online" and "how much $$$ should we spend securing it" is going to be different in these two cases.

I think there's also a fair question of "ownership of damages" here - cities get sold water treatment management systems and want them online as cheaply as possible - city councils end up owning the mistakes in misconfiguration but companies selling the systems are incentivized to make those default bad configurations possible - even while, in bold lettering, mentioning that you should not use the default authentication.

Cell towers are a really integral part of carrier's business - I'm not certain whether most are owned by providers or other companies, but either way the folks that put the tower up owe the customer (be it a phone user, a phone provider or some subcontactor of the provider) an explanation and pay the costs of bad configuration... I'd also assume that making sure these towers stay up is someone's fulltime job (likely multiple people) - while there won't be an employee constantly monitoring city water systems since it would take so little of a single person's time.

I'm not sure I agree that this is /wrong/ per se - the issue arises from the city council's disinterest / lack of expertise (which itself comes from disinterest) in these systems. If the issues are disclosed clearly, and the city council continues to sign off on the implementation (due to disinterest, cost pressure, whatever) without consulting knowledgeable third parties, then it's only realistic that the blame falls on the ultimate decision-maker (in this case, the city council).

The issue is that that strikes me as being incredibly socially inefficient. This town is probably going to be suuuper careful with water system security from here on out but the next town over might hit the same issue a few years down the line. There probably aren't more than a few dozen vendors of this type of service nationally and it'd be easier to learn the lesson at that consolidated level.

impact of hacking a cell tower isn't direct loss of human life

Not a direct loss, but plenty of opportunity for indirect loss. Disrupting emergency systems is the first that comes to mind. Covert hacking and surveillance could also be used for assassination plots.

Cell towers are generally going to be actively defended though - they tend to connect to private backhaul circuits, or link by IPsec to the security gateway in the mobile network.

The difference here is that nobody takes responsibility for a water treatment works in the same way a mobile operator looks after base stations - most operators aren't putting their base stations anywhere near the public internet. When they do it's under very careful control, like with femtocells.

Reply on news.ycombinator.com