Drinking water systems for 26M Americans face high cybersecurity risks
scworld.com
scworld.com
Cybersecurity is unfortunately not “sexy” enough for the common American voter to get behind.
The issue is unlikely to be money, nor is it likely to be technical. If throwing ever-increasing amounts of money at the problem isn't fixing it, maybe it isn't all that crazy to try the opposite.
> In a statement to Recorded Future News, an EPA spokesperson confirmed that the memorandum – handed down in March – was being withdrawn due to lawsuits filed by attorneys general in the States of Missouri, Arkansas, and Iowa as well as industry groups American Water Works Association (AWWA) and National Rural Water Association (NRWA).
There is no point in trying to solve what there is no will to solve. Less money, more money, they just don’t want to have to do it or be liable.
https://therecord.media/epa-says-litigation-from-republicans...
https://www.iowaattorneygeneral.gov/newsroom/attorney-genera...
https://content.govdelivery.com/attachments/IACIO/2023/04/18...
This statement from the EPA is the crux of it: "Most cybersecurity practices can be implemented at minimal cost."; a statement that anyone involved with software/cybersecurity knows has never been true about any software system, ever. It feels very reasonable to me that some of these dirt-poor counties could look at a new set of cyber requirements and say, we physically cannot pay for this; and while the EPA goes on to list sources of funding they offer, I don't know much about those; its possible that given they've already vastly underestimated the cost of securing an industrial software system, they're also vastly under-provisioning the grant funding available to do it.
No one wants their water systems insecure. Republicans aren't a comic book villain; and to help empathize with how they think, right or wrong, consider this: What if we took a good chunk of the EPA's budget and distributed it to the local water utilities directly (in other words: Your federal taxes go down, your county taxes go up). The EPA seems really good at drafting memorandums they have to redact and publishing reports about how insecure our water systems are; Republicans would argue, the money we spent on those things did nothing to help actually fix the problem, so maybe the solution is "less EPA".
I'm not saying this is right, and I'm not saying its even representative of a cogent reality. I'm just saying, this is the line of thinking.
> Republicans aren't a comic book villain;
Agreed, they’re much worse. “We’re going to spend nothing, and when something happens to you, too bad, so sad” under the guise of fiscal responsibility. But if that’s what you vote for as a voter, who am I to get in your way? This isn’t a problem to be fixed, this is an indicator of electorate intent.
> What if we took a good chunk of the EPA's budget and distributed it to the local water utilities directly (in other words: Your federal taxes go down, your county taxes go up).
This sounds like a recipe for fraud and waste due to the sophistication of those responsible at the local level, with no accountability.
1) What, Republicans don't need to drink water now?
2) I find it amazing. For decades the small government people have been arguing that centralising power to the national level is a bad idea because it creates a single point of failure. It is a real tell that people are play-acting with their concerns about the Republicans that they currently control all the major organs of the US government AND are led by Trump, but this is not enough to provoke a "gee, maybe centralising all these critical functions is a bad idea" response. The answer is still to concentrate power towards Trump and the Republican party, just to demonise them more vociferously.
I mean gee. If there are a group of people who are so terrible at securing clean water, don't give them control over the water systems ~50% of the time. Let it all happen at the local or state level. Is this such a radical position?
The feds get involved when there are environmental issues (EPA) or something directly under the federal prerogative comes up like federal lands, standardisation, the army, or interstate cooperation.
Those functions could be wholly coordinated by the states or direct communication between relevant bodies by the way.
> Here's a wild idea, how about we return to the concept of actual governance for the first time in a few decades...
These two ideas are actually somewhat contradictory, I recommend picking one. You can't return to good governance by ignoring governance every time you think something needs to be done.
And you don't need the EPA to be involved to defend against these sort of risks. In fact, standardising the IT approach through the EPA will might increase the risk since any vulnerabilities will be more likely to be standardised. Highly heterogeneous systems are expensive and difficult to cripple. If an adversary wants to attack there is a single point of failure they can target - the EPA process.
This is not a safe assumption to make. As an example: Standardized patching programs are very basic and reduce a lot of this risk. Without them (or incentive/budget for them), it’s typical for most private and public orgs to end up with pretty bad security posture.
This is a known failure state that results in many known vulnerabilities that can be trivially exploited, regardless of how heterogeneous these various networks are.
Sadly, from what I’ve seen in the news and in various reports, I don’t think many regional organizations have the funding, expertise, or incentive to run effective security programs.
If it was a more centralised system, there'd be a 90% chance it was all working and a 10% chance that the incompetent local officials were incompetent EPA officials instead, standardising everyone on insecure systems with nobody able to overrule them. It'd make a bad situation into an all-or-nothing gamble with substantially worse worst-case performance.
> You're proposing local and state governments spend orders of magnitude more taxpayer money on duplication of effort.
Doubt it. The expenses are going to be mostly capital and operating ones that can't be avoided.
Is 30 million people’s water being at risk in one of the richest countries on the planet a success? Is it even good enough?
Percentages like that aren’t as meaningful for government as they are for SaaS products.
This suggests to suspicion you've never worked in or with local or state government agencies and have spent zero time in a water treatment facility. I've done both so here's a couple of things that might help form a more robust picture of the situation:
1. These organizations are typically operating with budgets below the minimum requirements necessary to maintain the systems they're responsible for. This is why municipal bond initiatives are so common for water system maintenance and expansion projects.
2. Private contractors created this situation. Local and State government agencies contract out all of their IT projects and do not have personnel on staff capable of performing the work under discussion.
Point 2 is underdone, the question is basically should knowledge be in the private or public sector. Given that the private sector is generally more efficient than the public it seems a bit of an assertion to say that contracting out projects will decrease efficiencies. It could easily make the situation better. The US Federal government is absolutely known for being captured by contractors and consulting firms because it is a big, obvious & lucrative target and its bandwidth is completely overwhelmed to the point where very few people vote based on EPA policy. Odds are it'd be much more efficient to handle that at a lower level of government.
> This suggests to suspicion you've never worked in or with local or state government agencies and have spent zero time in a water treatment facility.
This is a silly point. I might as well ask "Are you one of the people who are managing water operators in the US incompetently?". If the debate is that the managers are clearly not up to the challenge, a lack of expertise is not a downside because the process the managers are using is being challenged.
I have had a lot of experience on the subject of organisational design. If you want better results, centralising into a single unaccountable bureaucracy isn't usually the way to get there. It is a bad strategy.
I anticipate you saying voting, so I'll preemptively observe that voters can vote for the government to switch to a different private provider too. Voting doesn't lose any of its potency if the government uses contractors, although it runs into the same problem as always that the EPA is realistically never going to be high enough on the priority list to change an election's outcome.
The whole "overstepping their mandate" thing is perceived as a very real and present threat among MAGA people, for this reason specifically. This is the whole notion of "The Swamp" Trump was going to drain; unelected career bureaucrats.
There’s a balance that needs to be struck between new blood with fresh perspectives and old guard with decades of experience navigating Washington politics.
So if, as toomuchtodo points out, Republicans are having to sue the EPA to stop them from managing water systems it is clear that the EPA is overstepping their mandate.
If they do in fact have a mandate to manage water systems and the Republicans aren't simply and obviously in the correct moral and legal position here, then this idea that they don't have a mandate must be discarded and they are in fact managing water systems at the federal level.
As can be seen from my earlier posts, I agree with the Republican lawyers & what appears to be your position - the EPA shouldn't be managing water systems. Leave it to the state & local authorities, the results will be better overall. If they are alarmed over something they can report it all they like, but they shouldn't be the ones to try and fix it.
If you have questions on the extent of the EPA's mandate and why, you'll probably find the Wikipedia page a better resource than me.
> They have a mandate to monitor and set standards for water systems.
So we've found a point of disagreement, because that is pretty much my definition of management. IMO, to a reasonable approximation, management is the process of monitoring a situation and setting standards that people must meet.
What do you think management involves, and what would separate the work of, say, senior management from what the EPA is doing? If the power to monitor and set standards isn't the heart of management, what is?
I don't understand why you think an agency nationally coordinating all the local governments' efforts and making reports to Congress should exist at any level below the federal.
I mean, if there was a software engineer designing systems, working with vendors, setting his own standards that were higher than company standards, collecting monitoring data, managing permissions, yada yada, we would still have to acknowledge that he was not the manager in the region, he is an employee and the manager is someone else. He is managing some aspects of the situation, no doubt, but if we ask "who is managing all this" the conventional answer isn't the developer, but the developer's manager. "Management" is typically reserved for the senior managers in the area monitoring the outcomes and setting high-level standards. By analogy, the EPA is attempting to manage water systems at the Federal level.
Do parts of that chain of logic seem questionable to you? Is it possible that when you say "managed" you mean what I would call "executed"?
> I don't understand why you think an agency nationally coordinating all the local governments' efforts and making reports to Congress should exist at any level below the federal.
I recommend asking questions if you don't understand something. It works well for me. This (drinking water) is a critical system. Centralised management is too risky, if something goes off the rails then it'd affect drinking water across an entire country. It is better to have disparate systems where failure is more localised.
Right now, the headline suggests <10% of the US is at risk. Trading that for a situation where there is a <10% chance that all of the US is at risk is actually making the risk profile worse. The chance of a problem is the same because nothing has been done to lower the triggers, but the consequences are substantially worse because the system is trying to centralise under the EPA. The EPA shouldn't be in a position where local authorities feel they have to sue them to keep the system decentralised, they shouldn't be trying to manage local operations.
The article states that drinking water systems for 26 million Americans are vulnerable to significant cybersecurity risks. What that obviously implies is that the drinking water systems for... 308 million Americans face an at least mitigated and acceptable level of cyber-risk. Are you stating that 308 million Americans live in counties led by Democrats, as they're the only ones capable of securing their water systems? Clearly that isn't the case. Are you crediting all of the good things about the good water systems to Democrats, and all the bad things about the bad water systems to Republicans, somehow? That feels unlikely; though I'm sure the federal government funds a significant amount of our local water systems' budgets, and currently the Republican party is the party of anti-federal government, obviously past and current Republicans also had a hand in creating some of that federal infrastructure and no one is saying it should entirely disappear.
> This sounds like a recipe for fraud and waste due to the sophistication of those responsible at the local level, with no accountability.
Its actually fascinating how Republicans would say the same thing of some of our federal institutions: That they've become so big that they're impossible to hold accountable. There's some truth in both extremes: but the Republican view is that the voting populace can more-easily hold their local government and utilities accountable than we're able to hold the federal bureaucracy accountable. Given the measurable performance and efficiency of some of these federal institutions; I do believe that Republicans are, at least, operating in good faith, even if I tend to side with Democrats directionally. Of course, we've become a culture of extremes: you're either in the party of "don't touch it, touching it is anti-federal and anti-democrat" or "tear it down and salt the earth it stood on", there's no in-between take that maybe these institutions just need an overhaul (well, there was [1], but it never panned out).
[1] https://www.washingtonpost.com/climate-environment/2022/05/3...
You’re pretty close to the mark on the line of thinking. Boiling down complex issues to pablum and creating problems is a hallmark of modern conservatism. Sorta like I set your house on fire, then launch a campaign attacking the firemen for not putting it out fast enough.
Cyber is seen as a risk and most municipal utilities and auditors are treating it as such. The private companies… not so much unless there’s a clear financial benefit or mandate.
Government info-sec jobs suck too. Crap pay, red tape, onsite only. Also, alot of security people have ethics surrounding privacy, data security, etc. Why work for a culture that spies on its own citizens, its allies, and engages in global terrorism? The NSA can attract some decent mathematical minds but lacking on the security front.
The issue is the US Government is hundreds of agencies, and each state in turn has hundreds of agencies as well.
Each of these agencies has their own IT that manages that agencies's infra and security AND they are very limited funds wise and salary wise.
For example, back when I was a PM, a customer of mine was the de facto CISO of a several hundred person agency yet only earned around $120k a year and had a less than $1M budget for all IT spend.
The agency could not build it's own hiring pipeline (having to resort to USAJOBS and the department it was a part of) nor was there any truly unified security platform.
While the naive answer would be "have everyone use a single platform", just about every presidential administration in the past 20 years has tried that and failed.
Isn't that literally the job of a National Security Agency, instead of spying?
Ever since WW2, the NSA's primary role has been Signals Intelligence, not Internal IT.
Offensive Security is an entirely different mission from Defensive Security.
If you're working government info-sec for drinking water systems, it's going to be mostly for municipal water systems, often agencies that are a similar geographic scope as counties or cities but sometimes independent from them. Water districts aren't spying much (unless you have strong feelings about water meters, or lawn watering restrictions), and they rarely take part in acts of violence outside the districts they serve and the sources of their water.
Some of these systems are private companies, and who knows about ethics there, sure.
I get having monitoring systems for it that are accessible in a way they could be hacked and disrupted, but why is the core operational infrastructure that way? Command and control should be isolated and be using 50-70 pneumatic tech to control it. Building in such a way to allow it to be disrupted remotely is the core problem here.
Just because you can, doesn't mean you should.
Actually it’s very easy to isolate that part. One way network equipments with physical isolation have existed for decades. An optic fibre with only an emitter on one side and only a receiver on the other will do the trick.
If you want to do that - and you do because having someone next to your equipment all the time to monitor things is both a waste of time and money and very error prone - you will need a data diode.
If you're a small facility servicing a few thousand people, you can't afford to have that kind of monitoring, and so you have to economize in various ways. One of the popular ways is pooling together with other small facilities so that you have one person doing that monitoring for several sites at once, which requires some form of remote operation.
Furthermore, when I worked at a large water company, all of our network, even the telemetry to the various pumping stations dotted around the service area, was on a private network airgapped from the internet. But there's also economy of scale here; a large company servicing 1.5 million people in a large metropolitan area can afford to do custom fiber backhaul in a way that even a bunch of small companies in the rural Midwest cannot, and so the control systems end up being Internet-accessible because it's too expensive for them not to be.
Small cities are quite willing to economize even if it means X risk some attacker will muck with the system.
But the thing I'd take issue is the "can't afford it" part.
Of course these cities could afford onsite worker. As gp pointed out, these districts operated long before the Internet and they could provide water then. But the neoliberal paradigm has appeared and suddenly the constant claim is no organization "can afford" not to do any given automation measure, no matter how illogical or dangerous. And so a key thing calling organizations on this baloney.
The actual functional control for ensuring a critical service like water is working should remain as an analog computer with something like pneumatics, or other such technology. These are robust and can continue to operate even when electronic circuits have failed.
Loss of visibility, and loss of service should be separated. This should be the same even for power stations as well.
The ones not on VPN were owned by big enough utility to have their own private fibre networks.
I am ripping out analog electronics from the 70s that nobody alive wants to support and installing PLCs with Ethernet.
The attention paid to cybersecurity goes up with the size of the facility as downtime is more expensive. Failsafes preventing damage or loss of life are seperate processors from the plc and independent often redundant systems, but still digital.
They don't want to pay a 24/7 on-site ops center. They take their chances and bolt-on security, and that's how the incentives work today.
Our water, power, and Internet are all delivered by local co-ops. We actually do get a direct say in how the money is spent on our infrastructure.
It's one of the reasons why I have fiber Internet whereas the closest town (managed by for profit entity) is still fighting to roll it out years after we had ours run to us.
I also got reimbursed by the co-op for the water line to my house when we built the place.
I also lobbied the power board to prioritize tree removal near lines for a more reliable service.
The definition of rural to me is around 12k residents in a county that is around 1000 square miles. That's the size of where we are. The largest town is around 3k people. Two counties over is a city of about 40k. The 3k town is part of our co-op, so they have fiber. The 40k had less than 15% on fiber the last I knew, but that was two years ago. Since then they haven't run anymore lines, but have added customers on their existing lines.
Also, due to past shenanigans with vendor lock-in schemes the Engineering Managers often have a valid concern for cryptographic/locked infrastructure and maintenance cycles. Ironically, right-to-repair legislation may slowly improve the situation.
It is not a technical problem, but a bureaucratic one =3
Also 55 gallons of water is ~450lbs, so it's not going to be terribly easy to move with an [appliance] dolly. You probably want pallets and a pallet jack (and a smooth concrete floor).
Personally I'd suggest getting an RO filter for your every day drinking water needs, and setting up a rain barrel collection that you can routinely use for outdoor garden/plants. Then if you suddenly need drinking water, you should be good just boiling the rain water. And if there is some large scale catastrophe with some kind of chemical/radiological contaminant in the rain, you can run it through the RO.
I can also drain it in < 5 mins with a cheap siphon (I let it drain into a sump basin). So swapping it out every few months is easy.
But I agree, a RO filter is great, esp for everyday use.
One concern I have with rain water barrel is how its collected. If you have an asphalt roof, there is some nasty stuff in it, and not sure if boiling helps in that situation. Need to do more research there. But its a great idea for watering a garden/yard.
But, similar to not having any smoke detectors in your home when something catches fire at night, being in a pandemic with him running the health agency is not going to turn out well for you.
(It’s all about spin)
Of course, an example statistic like 99.9% of airline passengers surviving a flight is not all that great…