CISA Alert: Water Sector PLC Targeting
censys.com
censys.com
Describe the network security of the industrial automation industry and their customers in a single statement. Lol.
How do you lockdown something that may have not been taken offline for decades because it will cost downtime or harm. Or something that can’t be locked down without tossing new tech around it that may not be compatible with the protocols etc.
Tunnel your dialup within your obsolete 3g network, and then tunnel that obsolete 3g network within something modern. The obsolete technologies are not the issue here.
Also the thing about dialup is that it's point to point so the attack surface isn't even remotely comparable to exposing a port on the open internet. I should generally be able to trust the link that my phone company provides. Faxes are still used in many secure settings in preference to email.
The dial up part is far more involved, especially when they have auto answering connected directly to PLC or HMI, mostly with shared passwords. Also, you can’t trust the network operator either, insider threats and rogue employees are a threat. Additionally, dial ups are less monitored compared to modern network, and usually you end up with duct tape solutions like jump server to have strong authentication and continuous logging in firewall and such, plus proper encrypted tunnels so even physical wiretapping isn’t possible, and the assumption of air gap isn’t there because it’s reachable through public telephone, and the worst part, these dial ups are usually connected to windows XP Scada developers machines.
To add, obsolete is bad too, when your device cease to have vulnerability patches, you are screwed regardless of whatever configs you put.
And the problem has actually gotten better over the years:
https://trends.shodan.io/search?query=tag%3Aics+rockwell
The situation used to be worse with things like the Lantronix password recovery service (i.e. a UDP port that would just send you the device password without any auth). It's still not ideal and takings things offline isn't easy (https://blog.shodan.io/taking-things-offline-is-hard/) but it's getting better (slowly).
I much prefer the non-vendor perspective on this. Andy Krapf, co-chair of the Water ISAC, has a great breakdown about the status quo systemic problems that water faces today.
The aim of a water network is to:
1. Take water from a water source (elevated dam -- strongly preferred, river, ocean) and as much as possible, gravity feed it to a treatment plant.
2. Treat the water using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. This is where availability and safety engineers would design equipment to not be dependent on software and instead use mechanical or analogue electronics control.
3. Pre-position treated water as much as possible at ~50-90m hydraulic head (~500-900kPA) above the water faucets where people want to use the treated water and provide a buffer for X days of usage. Any pumps between the treatment plant and elevated storage therefore only need to operate intermittently to refill the buffer.
Sewerage networks have similar aims:
1. Let sewage flow as much as possible downhill to the treatment plant via gravity. Where a rising main (elevation gain) is required, place a large enough sump for X hours/days of usage and pump up to higher elevation from the sump.
2. Treat the sewage using processes that are simplified/fail-safe as much as possible and could be operated manually by humans if necessary. For example, a compressor used for aeration can be manually switched on/off with a mechanical switch and plugged into a diesel generator, and not require someone logging in with multi-factor authentication to a laptop to issue a command to a PLC to turn on the compressor.
3. Design overflows into the system for emergency release of partially or untreated sewage, and practice this process as part of disaster recovery exercises. This is generally an aim arising due to risk assessment process that says building a $1bn sump with 8 independent pumps is cost prohibitive versus the 1-in-200 year chance of untreated sewage messing up a downstream river for a few weeks.
Ultimately a lot of the cybersecurity risk comes down to government appetite to accept 1-in-1000 (or whatever) year failure modes. Is it worth investing now in triple modular redundant automated control systems (mostly used in safety-critical sectors such as aviation and space), or installing a just-in-case diesel generator at every one of 500 pumping stations across a region, or building $10bn of sewage sumps to hold sewage for up to a month, or building 2 treatment plants instead of 1 and using different technology for each, or hiring and training more humans to regularly exercise manual control and operation of a network, etc? Or just accept that once every 1000 years, some water rationing may be required, or a downstream river will be polluted for a few weeks?
CISA and its predecessors have been warning utility operators about critical infrastructure vulnerabilities for what, 15 years at this point? That goes back to the first Obama administration.
Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. You cross a threshold where you're being deliberately careless.
When you are putting more effort into securing your Plex server on your home network then public utilities are taking on machinery that dumps chemicals into the local water, something is not right and finger-pointing isn't going to fix it.
I work with PLCs. Default passwords of not, the idea that such weakly secure devices are being made accessible from the public internet boggles my mind.
Public sector has always paid low. But the problem is widespread, almost universal, and they've had a 15 year head start of the federal government telling them to get their shit together.
At some point it just became standard industry practice is my guess.
Government is supposed to respond to these things and create incentives to correct. Telling a small municipality to do something without a carrot ir stick does nothing.
In this instance someone else will be providing the stick.
The one and only exception is the military, because lives are literally on the line in a way that is not at all abstract.
Paying skilled people highly does actually incentivize people to do better work, especially if they are actually embedded into the community they are essentially working for. If being a civil servant was as "glorious" as being a techie is SF there would be a very different attitude around the work.
The people who climb to the top aren’t the ones who took a risk and got the reward. Quite the opposite, they’re the ones who learned to play the game and didn’t upset the power structure by rocking the boat. No one is going to tell the emperor he has no clothes when the path to power is political and has no grounding in reality.
Highly conscientious, intrinsically motivated people will do the right thing in any environment. And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
>In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing.
Federal employment is merit based. Advances are earned, not doled out to the teacher's pet or the ass-kisser with the most perfect pucker. You have no idea what you're talking about.
I have no idea where you or anyone else got the idea that a government employee has no responsibility or incentives and works strictly for political ends and even worse, where none are willing to speak up. You clearly have no experience in that space and should focus your comments on things that you better understand through direct experience.
>And those people don’t last long in political/bureaucratic environments where the incentives are misaligned.
This is bullshit and you are grossly misinformed. Plenty of people fight the polarization of the system on political lines from within the system. They use the laws that are in place in order to prevent systemic abuses. They may not win every internal fight but they do go to battle for the right reasons.
You just have no idea.
Notably everything in the comment you replied to applies equally to the public and private sector. They are neutral observations about systemic motives and the associated perverse incentives.
Also it's not clear to me that any of the utilities in question have anything to do with the federal government so I'm not sure why you dragged them into this.
I'll make a note of that. Any comment that I make on this site is made in the hopes that there will be discussion generated. Sometimes that happens and other times it doesn't. I don't spend any part of my life wondering why people choose to or choose not to engage. Most of us have busy lives and this (HN) for us is an opportunity to catch up on interesting things.
>You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks.
I am not sure that noting that a poster has no experience in a subject and has no idea how it really works in practice becomes an uncharitable assumption. It may look like a personal attack but it is based entirely on the content of the user's own posts, which support the conclusion that I reached.
>They are neutral observations about systemic motives and the associated perverse incentives.
From /u/moscoe original comment in the thread:
>Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist.
These two sentences here demonstrate that /u/moscoe does not have any experience working in the public sector for federal, state, or local government entities yet somehow they feel qualified to speak confidently (incorrectly) that those who have worked in that capacity are ham-strung by the system and disincentivized to make waves. That is false.
I mentioned federal employees specifically because they are civil servants (referenced by /u/moscoe above) and someone close to me has worked a long career in the federal government under several administrations from both political parties.
I think you have your opinions and you are entitled to those opinions. I am also sure that you are wrong about the content of my post. It was not uncharitable nor were there any assumptions. The conclusions that I drew were supported by the statements made by /u/moscoe. If it ends up looking like a personal attack then there was a basis for that personal attack.
It's possible that you read my comment without reading the full thread so some of the context was missing.
There are better run governments than we have in the US.
The contempt for the state is a self-fulfilling prophecy. The state is incompetent because many of us believe it is inevitable that it will be. Compensation is just a part of it; coherent administration with continuity is even more important.
Our original bill pay (ran from 2006 to 2019) stored passwords in plaintext in flat files. Concerned citizens noticed because our password reset would just email you your own password. Instead of fixing it, they just removed the ability to recover an account without coming into an office. Our CTO knew, he wrote the whole thing!
We had a fun one, Outlook was sending employee passwords to our bill pay system and ending up plaintext in our logs due to some quirky fallback default behaviors around DNS and VPNs. Reported and ignored, of course.
Regarding TFA, we had an insurance requirement to properly air gap our PLCs - which we didn’t do. We (the CTO) just put them on a separate subnet and lied to insurance.
There’s not much you can do to an organization that has no real oversight here, especially once the “coast to retirement” types infest the place. We need something like HIPPA or PCI-DSS with real auditors and real teeth for utilities.
> failed to anticipate not only these infrastructure breach
They've been warning them for close to two decades.
Minnesota chose the path of no locks on their front doors and are now crying that someone walked in without knocking first.
Passing the buck to the Federal Government is not understanding the problem.
What does that audit look like and how frequent does that happen? It’s a security assessment? A quality assessment? A risk assessment?
I’m open to the idea, but I will repeat, I don’t think the problem is well enough understood for a “make the feds do it” type of comment.
> make the feds do it
National Security has always been a federal government responsibility. You make it sound like I’m expecting the federal government to take on some new responsibility. If the federal government starts a war with another country they’re absolutely responsible for minimizing by collateral damage at a fucking minimum.
NSA doesn’t do IT for the DoD/W, DHS doesn’t do IT for the government, CISA only gives guidance where they can. And IT does not equal OT. The issue comes down to actual skilled people hours to do the work and resource constraints to do so.
I agree that in theory this would not be a stretch if the stars aligned, but these are for the most part, not federal government funded entities nor government controlled even at a state level. They are usually clooged together by 100 years of paper maché. And that’s just water. What about Energy? Data Centers? Pharma? Regulation is way too far behind to just instantly drop a silver bullet.
And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI.
I'll say, you should see the hours I have to work sometimes. Honestly I've rarely seen IT jobs pay OT.
It comes down to incentives. If you want broad security you have to do more than hope that every water utility will both hire good people and also allow them to do their jobs properly.
> But what does that fundamentally mean for boots on the ground?
How does any regulation look on the ground? How does the federal government regulate banks and airports?
> these are for the most part, not federal government funded entities nor government controlled even at a state level
Neither are banks or airports
> What about Energy? Data Centers? Pharma?
Energy and pharma are already regulated. Maybe data centers will be eventually if they are deemed sufficiently critical.
> Regulation is way too far behind to just instantly drop a silver bullet.
I don’t know what this even means in the context of securing our water system. Do you mean to say that regulation can’t ensure that these software systems don’t use default passwords and so on?
> And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI
What is new is that we started a war with a country with a respectable technology competency without doing anything to shore up our defenses.
It is a top down initiative that needs to happen in a more meaningful way. It’s adjacent in a similar way to the issue of air traffic controllers. Talent shortage, old equipment, old hats, burn out and the like.
It needs some real backing and effort to make it happen.
Top down regulation drives the systematic change, just like it did with the banking sector.
> It is a top down initiative that needs to happen in a more meaningful way. It’s adjacent in a similar way to the issue of air traffic controllers. Talent shortage, old equipment, old hats, burn out and the like.
I'm not sure how this analogy works. What's the ATC equivalent to leaving default passwords on critical infrastructure?
Banking is resourced well enough to absorb that regulation and stand up a compliance team. I bring up ATC because of the consequence. Default passwords are a symptom, not the failure mode.
ATC is already federal, with the FAA running that. 20 years of regulation has not fixed the problems that still plague that industry: outdated equipment, short-staffed, a small niche talent and training pipeline, and people dying as a consequence of those systemic problems. That's even closer to my point. Making something regulated doesn't change the inherent problems inside the industry.
OT does not equal IT
> finger-pointing isn't going to fix it.
Your entire comment was finger pointing…
Passing the buck to the Federal Government is not understanding the problem.
That's a join in an existing database.
Where's the database and what's the code for this case?
> What's the code for this case?
Are you seriously asking me to provide a program?
AFAIK the most robust data they have on industrial sectors is within the national labs, and even then its considered sparse.
"I think Minnesota is behind it."
The first quote makes it the state's responsibility to secure local water systems, which I'm not sure that it is. The second makes it at least sound like the state of Minnesota is the entity running the attack on local water systems within their state, which is off in paranoid conspiracy territory.
Trump was absolutely wrong.
Which is one of MANY problems OT faces. IT best practices don’t suffice in OT and even when they do, most of these orgs are too resource hamstrung to do anything about all of the fires they have to put out.
Not to mention all of the OT vendors who flooded the market with tools instead of people being taught the boring process driven work.