What’s interesting about the Florida water system hack is that we heard about it | Hacker News Reader

What’s interesting about the Florida water system hack is that we heard about it

krebsonsecurity.com

440 pointstodsacerdoti181 comments

I don't have any specific knowledge of this event but I did work for a smart cities company for a while and can explain some of what hypothetically leads to messes like this.

Most municipal infrastructure automation is run by SCADA systems. SCADA systems are designed to be closed ecosystems by default, which means they don't have access to the internet. Cables are run and point to point connections are made to transport these signals. If you've seen a big shiny aluminum box on the side of the road, that's a SCADA system.

Consumer demand has changed a lot since SCADA was first thought up. These days people want to be able to log into a portal and see their billed water usage by the day. Cities want to be able to turn off a meter remotely at the click of a button rather than rolling a truck. They want pressure loss, contaminate, and fault detection. They'd like to balance pressure in the water system dynamically. They want to perform detailed reports on total water loss, which means instrumenting pumps, retrieving telemetry data, auditing that data against every home usage, identifying faults, and producing a report out of it. All of these things don't work very well with closed ecosystems as they generally require a certain amount of external information. In response, cities have either crafted their own devices because they don't have the money to hire an integrator to design and produce them a device that will survive extreme weather conditions, be fault tolerant, make minimal use of power, and employ standard security measures or they hire an integrator. Even certain integrators can do shotty work.

The takeaway from this isn't that SCADA is bad or the infrastructure version of "the sky is falling!". It's that we desperately need to be thinking about how different generations of systems can be modularized and be built upon to minimize cost while maximizing security, capability growth, and reliability.

For the folks talking about "who tests what", I don't know what I can talk about in regards to specific programs but I feel confident that I can say that the FBI tracks this vigorously and works with companies that work in public infrastructure.

SCADA vendors have been way too caviler about collecting data from the "plant floor" for the "front office" with technologies that weren't designed with a threat model that even remotely approaches a run-of-the-mill LAN, let alone the Internet. Never mind protocols that have no provision for authentication or encryption-- we're talking about embedded devices that fall over and die when port-scanned or when they receive "too many" broadcasts. Network segmentation is usually difficult in these environments too. Inevitably, there's a piece of software and the associated devices it controls that have to layer 2-adjacent to "discover" each other and communicate. The SCADA vendor sets the expectation that a "front office" computer should be able to interface just fine with the "plant floor" and blames the meddling IT department with their subnets and default gateways and ACLs for things not working. (I may have some pent up frustration on this topic.)

Aside from the technical limitations of some of these platforms there's a cultural issue. Controls engineers aren't trained in Infosec. The software and device manufacturers are developing better products, but I don't feel like there's a "secure by default" initiative. It still takes skills on the part of the installer.

I still see a huge lack of consideration for security in the people actually installing gear (at least in the small-to-medium sized manufacturing companies and small municipalities I work around). The biggest concerns are making sure vendor's support personnel can troubleshoot remotely because frequently the on-site personnel have limited skills. That's how we end up with TeamViewer, VNC forwarded thru routers, etc.

> The biggest concerns are making sure vendor's support personnel can troubleshoot remotely because frequently the on-site personnel have limited skills.

That's an excellent call-out. City salaries will almost never employ the type and level of personnel to do anything beyond maintaining the status quo typically.

It's more than just a lack of salary. There simply isn't enough technical work to keep highly skilled engineers busy at your typical industrial or infrastructure sites.

Even if you had the funds to employ highly trained engineers, most sites couldn't give them enough work to keep their skills sharp.

A lot of the failure is just the basics, and it's not always technical incompetence. It can be managerial.

For instance, most SCADA and IT vendors can do a lot better than (probably not patched) Windows 7 with TeamViewer. However, there's two prevailing attitudes that interfere:

1. We're not big like Boston or New York, therefore we're not going to be targeted and are fine, nobody cares to harm us.

2. When you have a working SCADA system, nobody wants to run updates or make security policy changes when that might lead to not having a working SCADA system. Security vulnerability remains the constant invisible bogeyman, you never know if you did enough until it's too late.

> 1. We're not big like Boston or New York, therefore we're not going to be targeted and are fine, nobody cares to harm us.

If enough events keep occurring (and why would we suspect they won't), this attitude will HAVE?? to change. Kids will learn about it, and do it for the lulz. The city of Dallas had its tornado sirens hacked, and at least in the local areas, it brought some attention to the subject. Eventually, the bored teenagers in the small towns are going to do it just because.

Yeah, I think every time an incident like this occurs, it provides a new cautionary tale where people can see how mistaken that attitude is.

> Eventually, the bored teenagers in the small towns are going to do it just because.

Not just locally. If you're trying to mess up an industrial system to see it mentioned in the local newspaper few days later, you may just pick a random system from Shodan, note its location, and check their local newspaper.

I've said it before and I'll say it again. The only way critical infrastructure should be connected to the internet is via data diode [1]. Otherwise it should be air-gapped.

It still lets you remotely monitor systems, but if you want to adjust the operation of the critical infrastructure you have to physically go there and do it in person. People often complain that this is inefficient, but if your own workers are unable to remotely affect a system then neither can an attacker - given the consequences of an attacker compromising a critical infrastructure I think the inconvenience is a fair price to pay.

[1] for those unfamiliar, this is typically implemented using two computers connected via fibreoptic cable, such that there are only transmitters (lasers) on one side and only receivers (photodiodes) on the other side. Thus the information can physically only travel in one direction. The computers at each end handle conversion from connection-oriented protocols (e.g. TCP) to connectionless (e.g. UDP) while maintaining reliability.

I work on about 30 different power plants for which I have written software in the control systems. I can access all of them remotely, no way i’m flying/Driving to middle of nowhere to make a change, and no way those customers are going to pay me for three days what otherwise takes a few hours.

Your customers probably would be more willing to pay if they had to be insured against damages caused by hacking.

It’s an interesting question i was contemplating while writing the post, is it possible to cause damage by hacking the plants I work on? Exceeding conditions set forth in Environmental certificates, yes, destroying the machinery ... would be possible only because some modern equipment is used and connected to the network, if the Ethernet ports on the devices containing the sync-checks were disconnected, or an older dumb device where any setting adjustments are made with a screwdriver, then it wouldn’t be possible to damage about half of the plants. The other half with oil pumps you could cause some significant damage by telling it the oil pump is running when it is not and suppressing all the bearing temperature protection.

I wonder what if any cyber security requirements are imposed by their insurance companies.

> would be possible only because some modern equipment is used and connected to the network [...] containing the sync-checks

"How 30 Lines of Code Blew Up a 27-Ton Generator: A secret experiment in 2007 proved that hackers could devastate power grid equipment beyond repair—with a file no bigger than a gif": https://www.wired.com/story/how-30-lines-of-code-blew-up-27-...

YouTube video of demonstration: https://youtu.be/LM8kLaJ2NDU

But you’re forgetting second order damage. What if somebody takes out a large enough fraction of power plants in the middle of winter? Or in the middle of a heat wave in Texas? Casualties could be rather significant if the plants can’t be restored in time.

> Your customers probably would be more willing to pay if they had to be insured against damages caused by hacking.

This is actually the other way around. Insurance is a moral hazard. If you're paying for the insurance anyway then taking steps to reduce the insurance company's risk rather than your own is throwing money down the drain.

Meanwhile the insurance company as a third party has less control over your actions than your own company and is less able to make efficient trade offs, so any requirements they impose tend to be either ineffective or unnecessarily burdensome or both.

The problem is that publicly traded or municipal utility companies have pretty much the same problem where the employees/managers don't pay personally for damage and the people who do are too diffuse or far away to understand their exposure well enough to mitigate it.

It’s the premium you pay that quantifies the risk.

Until there’s no insurance company that quantifies the damages and makes a quote to insure it, it’s all “Snowden cranks waving arms” nonsense.

Once that is in place a new excel sheet magically blips onscreen and big maneuvers begin to minimize that premium and comply to the minimum standards the insurance will not classify as negligence.

> It’s the premium you pay that quantifies the risk.

But then what? You pay the premium. Then you comply with whatever "make sure your Linux servers have antivirus" rules the insurance company comes up with. Then you stop doing anything that costs a penny more than that because the risk is now on the insurance company, which makes the problem worse. And when everybody does that, everybody's premiums go up.

My experience was that the insurance company had reasonable and effective (if costly) requirements - the “install antivirus on Linux” requirements tend to come from payment processors who would be liable for exactly nothing if you got hacked.

One of my colleagues preaches - for the case of fire/damage etc insurance - to get a quote; do everything they require (as these are almost always things that have or would have saved them payment before, e.g. keeping beige boxes above floor level and under a desk - which mostly mitigates water damage from both floor flooding and ceiling leaks). And after they would insure you, once it is clear they are confident they have positive expectation from insuring you (and they know what they are doing) - that’s the time to reevaluate if you want an external insurer, or your own piggy bank.

With respect to hacking, this advice is less applicable, however.

In an efficient market (I know...) another insurance company would start offering lower rates if you implemented more effective measures.

Well, if the insurance is not looking for an excuse to go bankrupt they’ll survey the existing setup and either refuse to insure or ask an astronomical premium. The infrastructure operator will then negotiate a compliance progress plan to reduce the premium down to an affordable amount.

I’m not saying there aren’t other mechanisms to pervert this model, but as a first approximation this is how’s it supposed to work: as long as the operator is forced to buy insurance, and there’s no way for the insurance to get off the hook, parties are forced to develop a credible risk assessment and mitigation.

Here's the failure mode: The insurance company comes up with some prohibitively expensive and not cost effective at all requirements that would work (but bankrupt you) if you actually implemented them. Someone at the company feigns compliance with the rules.

Then, if there is a claim, the insurance company investigates whether you followed the rules, finds that you didn't, and denies the claim. The insurance company doesn't go bankrupt and doesn't care that this happened. But most of the time there isn't a claim so nobody on either side actually notices that this is happening.

I will take the free government money thank you very much. Maybe even hire another person to help make these physical site visits. If we don't take the government's money, they'll just spend it on weapons.

Private and public companies own power plants too.

Cool. Let's lobby for regulations that force those companies to hire us. We're gonna make so much money.

You’d make more money as a plumber

You're essentially you're willing to trade security of critical infrastructure (and potentially risk lives) for the convenience of being able to do your job from home.

The plant owners have made that call.

Is a vpn secure?

If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?

>The plant owners have made that call.

Arguably it should not be up to the owner entity as to the security requirements they need to meet. At least down here in Australia we're in the process of legislative reforms that significantly expand both the scope of critical infrastructure (most of which is privately owned) and the responsibilities and requirements of critical infrastructure operators (in terms of risk management, security plans, mandatory reporting, etc.)

>Is a vpn secure?

Teamviewer (which I understand was the vector used in the water system breach) or a VPN that hasn't passed the extensive testing required of secure cryptographic devices is not acceptable.

IF the industrial control systems were treated as a classified information system, were linked using an approved and properly managed [1] secure cryptographic device (on both sides e.g. back to a secure control centre, not to your house so you can work from bed), and generally had the rest of the protective elements (e.g. requiring personnel working on it to hold a SECRET or equivalent clearance proportionate with the value of the system) in place, that might be OK. But otherwise my position remains the same - it should not be remotely controllable via the internet.

>If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?

That might brick some cars on-board computers or "only" kill perhaps a few dozen people before people caught on and it was recalled - still tragic, but small in the grand scheme of things. Critical infrastructure is called that because if it goes wrong it can harm or kill people on a vast scale, so it should be held to far higher standards.

[1] e.g. keymat issued by whatever your equivalent to the NSA is e.g. our Australian Signals Directorate, and otherwise verified to be secure.

I could imagine a malicious defect in Tesla firmware that would cause all Tesla vehicles on the road to make an aggressive manoeuvre that would most likely result in a crash at the same time. So I think the potential scale for death and injury With remotely updatable vehicle firmware is large.

Can you recommend any VPNs that are up to the required standard for critical infrastructure?

Although since the power plants I work on would struggle to kill even one person and damaging them would only cause financial hardship for the owner or an insignificant exceedanve of environmental regulations, so they must not be critical.

Wireguard is the thing I would use. The fact that non other than Linus Torvalds called wireguard "a piece of art" may or may not be a selling factor.

It is small, lightweight, reliable and does what it is meant to do. It uses existing, trusted network tech in the right ways and due to the low complexity you reading the source code is an realistic option.

Does wireguard support storing keys in a hardware device, so that the cannot easily be exfiltrated if the machine is compromised? You want that for security critical infrastructure.

AFAIK implementation in Linux kernel not yet support any hardware-backed handshake, but it's mainly because WireGuard developers didn't consider code stable until March of 2020.

But there are userspace implementations and I guess it's should be doable to make them talk to a hardware device.

>I could imagine a malicious defect in Tesla firmware that would cause all Tesla vehicles on the road to make an aggressive manoeuvre that would most likely result in a crash at the same time. So I think the potential scale for death and injury With remotely updatable vehicle firmware is large.

It's pretty low risk AFAIK, generally cars infotainment and CAN bus are isolated, and I understand Tesla vehicles can't update while driving. Who knows, I'm not familiar with their architecture, but I'm not too worried about it.

There are attacks but IIRC they rely on GPS spoofing, or tricking machine vision to cause the vehicle to swerve or accelerate, you can't just drive someone off a cliff over 4G (at least without implanting hardware into the car).

>Can you recommend any VPNs that are up to the required standard for critical infrastructure?

Approved devices vary by country, but this list is a decent example: https://www.ia.nato.int/niapc/Category/IP-Encryption_25

Unfortunately it's hard to provide info when these kinds of products are not generally available for purchase outside of the defence community, and the user manuals and related documentation are themselves restricted. But generally speaking if it's acceptable for use with classified info then it should be adequate protection for critical infrastructure.

>Although since the power plants I work on would struggle to kill even one person and damaging them would only cause financial hardship for the owner or an insignificant exceedanve of environmental regulations, so they must not be critical.

They might be relatively harmless in isolation, but it could be really harmful if an attacker could compromise a dozen of these small plants and disable or destroy them at the same time so there's less reserve capacity in the grid to pick up the slack when you launch your main attack. You don't even need to have that big of an impact on industry or what have you, just causing a blackout on a hot day and killing a bunch of elderly people via heatstroke would whip people into a panic.

Or perhaps an attacker would compromise one plant and use it as a vector to attack someone like you next time you connect, gain a pivot point via your workstation to set up a persistent backdoor and steal confidential documents and email so they know your schedules and organisational weaknesses, steal your VPN credentials and attack multiple power plants at once when it suits them, etc.

Thanks for the VPN link.

>They might be relatively harmless in isolation, but it could be really harmful if an attacker could compromise a dozen of these small plants and disable or destroy them at the same time so there's less reserve capacity in the grid to pick up the slack when you launch your main attack.

I suppose it depends on what is considered a small power plant. I would say anything less than 20MW is small, greater than 30 MW is medium, and greater than 100 MW is large. Simultaneously tripping a dozen 20 MW power plants would result in a loss of 240 MW of generation, about the same as 2 large units. There is 258,000 MW of nameplate generation on the western north american grid and a peak load of about 150,000 MW. The link below [0] shows that transmission level operators are looking at 1000's of MW of lost generation to have a measurable impact on the grid but losing that amount of generation causes operational headaches and not automatically an outage.

[0]: https://www.wecc.org/Reliability/2013-10%20WECC%20JSIS%20-%2...

Do you have more info on the changes to Australian infrastructure legislation?

It hasn't been passed yet, but there's some info here:

https://www.homeaffairs.gov.au/reports-and-publications/subm...

https://www.aph.gov.au/Parliamentary_Business/Bills_Legislat...

The CI cybersecurity reforms are part of the overall 2020 cybersecurity strategy (https://www.homeaffairs.gov.au/about-us/our-portfolios/cyber...), there's a lot of changes happening in the sector at the moment. If you have an interest in cybersecurity now would be a good time to grab some certifications and be ready to jump on board, as there's expected to be a big demand for cybersecurity professionals over the next few years to support these initiatives.

This looks like a decent summary of the proposed changes: https://www.ashurst.com/en/news-and-insights/legal-updates/m...

> If Tesla can update the software in a fleet of vehicles remotely why can’t I update the software in a power plant?

OTA updates to car firmware is widely considered to be a stupidly irresponsible idea. I don't think we should justify doing risky things by pointing at other people doing risky things, who only get away with that because of clout and regulatory wheels turning slowly.

> OTA updates to car firmware is widely considered to be a stupidly irresponsible idea.

On one hand, it opens the door to hacking, pushing a malicious update, etc (stuxnet much?). On the other hand, if it is never updated, then we will end up with a zombie network consisting of lightbulbs, fridges, and teapots (yes!)

Edit: sorry I could not resist adding teapots (HTML error 418)

I love the concept of an electric car. Self-driving not so much.

OTA updates, and car security in general, is a joke it's so bad.

Teslas are great cars in many ways, but cybersecurity is not one of those ways if you go by pentest reports. I stopped at listing five below, but there are quite a few more out there.

https://www.forbes.com/sites/leemathews/2020/11/23/researche...

https://www.forbes.com/sites/daveywinder/2020/02/19/hackers-...

https://electrek.co/2020/06/10/tesla-hacker-unlocks-performa...

https://www.zdnet.com/article/tesla-car-hacked-at-pwn2own-co...

https://insideevs.com/news/356769/video-tesla-model-3-hacked...

If there's an emergency at a power plant and you need the sysadmin to personally fix it, then that has the potential to be just as big of a safety issue as a malicious actor is. It's trading one form of security for another.

I think this is even scarier, that there can de emergency where sysadmin should manualy fix it. There should be fallbacks and redundancies with muliple levels if this is realy a crytical system. These would should give enough time to transport whatever personnel you need on site to fix things.

Thanks for speaking up, rhodozelia. The currently prevailing voice of HN appears not to be on your side, but you have 30 customers who are on your side, and you probably represent the views of many. We need to hear more of your side.

I wish more attention (money) was given to cyber security on infrastructure projects, and all projects. Cyber security is a mess everywhere. How Many organizations in the USA have not been hacked or are Unhackable? The only one I would guess is google and maybe amazon? Banks and financial markets seem competent as well.

Banks generally have terrible track of record for security across the globe. But they usually have large and agressive legal departments so no sensible white hat ever gonna touch them with a ten-foot pole. At least they wont ever be allowed to publish results of security audit.

I worked on software that interfaced with industrial control systems on some large machinery that could definitely kill people (and sometimes did, anyway). We absolutely flew out to every location to make changes, around the world. Yes, it was a great cost of time and money. But it's just irresponsible to allow connections from the outside when something going wrong would kill people and cause a lot of damage.

Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on.

If the machines are killing people sometimes that is probably a whole other level of risk and I agree not something I would want to touch remotely.

> Other than syncing a generator totally out of phase I can’t think of anything that could potentially hurt someone at the power plants I work on.

Don't power plants have all kinds of potentially dangerous mechanical systems under computer control? Overpressure on a fuel line causing the pipe to burst and fill the building with natural gas, things like that?

There aren’t nearly as many things as you might think that have unbounded failure modes like that. At the mechanical and electrical engineering level you have fuses, breakers, pressure relief valves, etc that make all of these failure scenarios just “the machine shut down”.

And then the machine shuts down and you have to fly there anyway to fix it.

This sounds like turning "an attacker may destroy power generation infrastructure and maybe kill people" problem into "an attacker may repeatedly DoS the power generation systems, likely damage it, and still probably cause loss of life by second-order effects". Doesn't feel all that different.

To use an analogy: it's like saying that attackers accessing the company servers remotely aren't a problem, because the best they can do is to keep them in a reboot loop...

Your analogy made me chuckle.

the western grid has 258,000 MW of generation. renewables go up and down like yo-yos, power plants go on-line and offline all the time and a large and always increasing fraction of it is unscheduled and uncontrolled.

Of course it depends on the size and location of the power plant, but there are many more small plants that can go offline without causing any significant effect to the grid than there are large ones, and the large ones probably have the resources to be manned 24/7 either on site or from a remote site over a dedicated and thereby secure fibre link.

One take-away from this thread is that not all infrastructure is 'critical'.

Crappy analogy. There is a huge difference between being able to shut down some power generation that is a drop in the bucket on a grid vs overloading some equipment and killing people.

One is life and death, another is lost revenue for a power plant.

If you think infosec people are paranoid, boy have you not met a good mechanical engineer yet.

Down in the analogue world, random overpressures, voltage spikes, simple mechanical fatigure, etc happen all the time, so you're going to have to deal. I've written reports on why, regardless of what a sensor said, a simple static beam made of nonflammable materials neither a) spontaneously combusted, b) broke Clausius's principle and moved heat from a cold body to a hotter body. But we still checked.

I agree on this, if something bad happened most critical system are designed to break somewhere to limit the catastrophe, or other times they are over engineered 1,000x because sometimes you can never tell what nature can do to your system.

I wish to believe that. I'd expect it of good engineers.

However, I don't trust systems to remain as robust as initially designed, after decades of undergoing constant pressure to reduce costs, make things more (fiscally) efficient, "cut out fat", etc.

Even the loss of institutional knowledge can degrade the safety of a system over time.

Ever had to start maintaining a codebase after the greybeard who made it retired? Did you have any documentation to go off of? Anyone remaining in the company regard it as anything other than a production critical black box?

Now imagine that codebase is an industrial process, with 20 years of growth, add-ons, changes. Probably some mismatching control systems, and god only knows if the as-builts or red line plans for the additions are all correct or rectified with each other..

Certainly the systems are designed to. But over the operational life, things happen. Systems degrade, maybe maintenance lapses a bit.

Maybe, for example, a mechanic tired of trouble shooting an issue replaces the 9th blown burst disc in a row with one he just cut from the mud flap on his shop truck. NBD, the system was only running 5% over pressure anyway and he wants to go home.

Later, this overpressure state backs up through the system and overheats a pump which, considering that it is pumping liquid explosives, isn't ideal. Maybe this trips a temperature sensor and the jury rigged burst disc is discovered. Maybe it doesn't, or the sensor was bypassed or silenced, and the pump explodes.

I'm rambling, but this is a real world example. My point is that these various factors: designed safety monitors like sensors or fail-safes play hand in hand with over-engineered systems as well as the security of the control systems. Any one part of the system should not be allowed to lapse or be less secure with the idea that another level of the system will catch the problem before it becomes life endangering.

Mechanical, chemical, regulating temperature, regulating flow, regulating speed. Imagine chemical X reached 10C over the recommended and starts a chain reaction. Or imagine sensors going offline. I was once working on a food-producing factory, and we had our shares of accidents, though not lethal. Any type of machinery with fluctuating power supply may be dangerous unless there are automated killswitches, release valves, etc. in place.

Power failures (especially long ones) can kill people

If the power plants would shut down or be severely damaged, wouldn't the resultant lack of power cause all sorts of damage, including things like accident victims not being able to reach emergency services and all the other classic ripple failures? I can imagine there wouldn't be any direct casualties in the plant, but I can't imagine power plants that upon unplanned shutdown would not cause severe damage, including physical harm and death (albeit indirectly). Even power disruptions of a few hours cause deaths.

Syncing a generator out of phase is exactly what was trialled in the Aurora Generator Test: https://en.wikipedia.org/wiki/Aurora_Generator_Test

Didn't go well for the jenny. (Though a turbine might have fared better)

Does your power plant deal with any steam? Gag a few safety valves and find out the hard way.

So did anyone of your deployments feature on one of the presentations of @viss - https://youtu.be/hMtu7vV_HmY

so if the customers would pay you to drive to the middle of nowhere to make the change you still wouldn't do it?

I'm a bit more concerned why one person is critical for 30 power stations. Surely there should be at least a few others that could be called. What if you were on holiday, incapacitated or just quit?

SCADA systems can kill people, any computerized system that can kill people must be protected as a weapons platform, therefore SCADA system must be protected as if they were weapons platforms.

Wait, isn't that a bit over-dramatic? I mean sure, that's true of nuclear systems. But I mean, my home thermostat? My car's smart key system?

As for the thermostat - it's winter below freezing and a hacker turns your heat off during the night. Sure someone might wake up, and it would have to be very cold for you to freeze, but if you have someone old or a child then it could be life threatening. Depending on the heater it might be able to make it cause a fire.

Many car's smart key system is on the same network in the car as everything else, including the brakes. Read more in https://money.cnn.com/2014/08/01/technology/security/most-ha...

So no, it's not being overly dramatic to categorize any SCADA system as a weapons platform. It might be, as they say, 'an abundance of caution', but I am fine with erring on that side than erring the other way.

Three Mile Island can be viewed as caused because one gauge was not being tracked correctly (where it was supposed to be was being tracked, not where it actually was). What if remote access was enabled on systems in a nuclear power plant, so that people didn't have to drive all the way there? A hacker might manage to replicate some version of Three Mile Island. I can hear the 'but it's secured' response from some already, but any system that can be accessed can be hacked, given enough time and resources.

The key thing people should realize is that cybersecurity does not prevent hacking, it attempts to delay it until a real person can take corrective action to protect the system. This is why you have defense in depth and rings of security. Essentially, it is a door lock. The better the lock, the more skilled class of thief it will dissuade by being 'too hard to bother'. A skilled thief who has state of the art lock picks, or is willing to use a door ram, is not going to be deterred by even the best of locks (think organized crime or state actor hackers here). But if our imaginary door only opens one way no matter what, to let people out, and can physically never let people in - then even the greatest thief is SOL. That is the purpose of a data diode.

The other other concern, and it's not minor, is that any time you start measuring safety in terms of cost effectiveness you are putting a value on human life. It is saying that is is cheaper to pay the settlements and/or fines resulting from deaths due to insufficient safety precautions than it is to make it safe to begin with.

The data diode seems like an interesting idea, but couldn't that still enable a hacker (one who knew very well what they were doing) to increase the amount of lye in the water?

If you have the diode like:

plant < diode < internet

you could control it but not get any feedback. I meant

plant > diode > internet

so you could monitor it but not control it from the internet.

oops, duh, thanks for clarifying.

It seems like a big jump from the kinds of idiotic security lapses we’re seeing to a full air gap. What’s wrong with proper authentication and encryption?

It’s not a nuclear power plant. There’s no crack team of snipers to kill anything that gets too close to the fence. Physically these places are certainly not wide open, but neither are the fences, doors, and locks any more intense than what’s on the local high school.

I think scale is the answer to that. These systems are being scanned for day and night for vulnerabilities from attackers and once they have infiltrated can lie in wait to do something whenever they want. A foreign actor can do that at massive scale undetected (both when successful and when not).

Same isn't true for physical security.

Well, put too much inconvenience and some "smart" guy will circumvent it anyway.

I have not personnaly worked in industrial environment, but from people I know, security wise, it is very scary at some places. Here is one such example of a "smart guy":

The IT crew of a factory have a sane policy of deny all traffic, except some well thought out rules.

A supplier would like to open a remote connection for maintenance of the equipment. Since the process for opening some traffic for a short duration is so convoluted, they finally install a 3G card inside the equipment (without telling their customer, and they even make it a habit for future install) so that they can do maintenance whenever they want.

The IT crew still thinks all is safe from outside...

That's pretty much sabotage.

Did someone in this story report it to the company management? This sounds like a clear lawsuit (or at the very least, contract termination) waiting to happen, and one that would scare other vendors away from such "clever" workarounds.

I don't know about management, but the guys on the factory floor were aware and happy with it. It made their job more convenient too, and the supplier was marketing its solutions as secure, so don't worry ;-)

> the guys on the factory floor were aware and happy with it. It made their job more convenient too,

Oh, I don't doubt that :).

> the supplier was marketing its solutions as secure

... but I very much doubt the truth of that marketing :).

I think it's still a ticking liability timebomb. At some point some external auditor will discover the backdoor to the plant and heads will roll.

The 3G thing is extremely common. Even with HUGE suppliers. Including Siemens Energy. Their steam turbine systems come with a 3g connection as standard. I even think they won't do 24h/7d support without it.

This, this right here is the correct approach. Unfortunately, it is rarely taken, as economics are directly opposed to it. Usually you have a mixture of company wide LAN, with directly connected machines, accessible on demand via VPN from service-contractors.

My fallback approach is, to insist on machines incapable of self-destruction and Whole System BackUps. Its easier & cheaper, to unplug, replace some spare-parts, reinstall some acronis images and go. Sad world, but real world

> Consumer demand has changed a lot since SCADA was first thought up.

I find this hard to believe. In this modern world of connected convenience, I’d say what I want from the water company is 99.99999% safe water and 0.00001% real time billing. The idea that you would sacrifice the former for the latter is ridiculous.

Nothing in your comment makes me think SCADA is bad, it just makes me think connecting it to the internet is dumb.

A big difference is employee and organizational expectations too. People expect to be able to monitor things from home, not pay people to sit at a desk and watch it.

That, I believe. But don’t try to wrap that in some nonsense about customers wanting it. What we want from water is safety. Period.

I would also point out if you sell and support SCADA systems, the village is your customer/consumer, which I believe the parent post also alluded to.

You are not the customer. You do not buy the SCADA system.

You are the consumer. You want pure water. That I believe. The people buying the SCADA system want to be able to look at a dashboard while staying at home in their slippers during COVID-19. Worse yet, the SCADA system might have already been bought some time ago -- as is very common -- and adding "Hey I want to look at this from home!" might have happened a decade after the purchase.

As I noted, I worked for a smart cities company, the consumer for a SCADA and complimenting systems are both city employees and residents. You would also be surprised at what people desire out of their local water company. Portals and online payments were very much in demand. People don't like writing checks or having to go up to the municipal buildings to pay their bills.

> What we want from water is safety.

This is a different subject altogether.

That said, the kind of pressure and pump monitoring systems I mentioned are all part of keeping water safe.

- You can have a website that takes payments without putting the infrastructure online. Take what you get from the human meter reader, put it in a database.

The real negative impact of meter reading isn’t the lack of the online billing, it’s that the water company might not be able to bill as often and might have to sometimes bill on estimates, so people can get saddled with surprise bills

- I don’t believe the two are actually related but if I believed that setting up a payment website required introducing huge security vulnerabilities, I’d rather mail a check. And I hate mailing checks.

- As for the sellers viewing the water companies as their customers: I agree; that makes sense. But the water companies are public utilities and if they’re putting their own convenience over the safety of residents, it should be addressed with regulation.

> You can have a website that takes payments without putting the infrastructure online. Take what you get from the human meter reader, put it in a database.

This has been done since the dawn of water management. Manual meter reading is the source of most errors in a system. SCADA systems generally don't get directly tied in with meter management and billing, but they are required somewhere in many processes. Sorry if I didn't make that clear.

> I don’t believe the two are actually related but if I believed that setting up a payment website required introducing huge security vulnerabilities, I’d rather mail a check. And I hate mailing checks.

It doesn't. What I said is about how these things happen. It doesn't mean the only way to achieve a "smart city" is via TeamViewer. Outfitting your system with devices designed with the SCADA security model in mind has proven to work.

> But the water companies are public utilities and if they’re putting their own convenience over the safety of residents, it should be addressed with regulation.

I'd caution you against routinely arriving at maximalist interpretations. This has very little to do with convenience and more to do with efficiency, safety, and resource management which makes your water cheaper, safer, and less likely to be lost.

> These days people want to be able to log into a portal and see their billed water usage by the day.

there's lots of safe ways to export data from a secured/highly controlled system in a manner designed for one way information flow. even if it's as simple as putting data in a pipe-delimited CSV in a cron job.

first get the data out of the secured system's boundaries, nothing sensitive, and then do whatever is needed to present it to customers in a friendly web GUI.

The notion that SCADA systems are air gapped, closed ecosystems is 40 years out of date.

I've had hundreds of plant engineers and SCADA operators tell me that their systems were "not connected to anything" and every single one of them was wrong.

Whether it's an old 2400 baud modem on a spare port on the RTU or "serial" comms being multiplexed across IP or MPLS networks or the unlicensed TeamViewer install for vendor support, there is always a "well except ..."

Supervisory Control and Data Acquisition. Haven’t they always been able to remotely gather telemetry data and control devices? How was this achieved with network access?

Great response.

Sometimes they don't even craft devices, even shoddy devices would be better than other options some take. Sometimes it's just a matter of someone in the muni saying 'hey, we can fix this, let's put this on a password protected wireless network'. Sometimes it's not even password protected.

Sometimes it's not the muni, but the vendor, who decides for remote management and puts the device on the internet.

I agree with the other poster who said they thought data diode was the way to go. Make it so the companies and munis can get data, but cannot send data to the devices unless they are physically on site. Of course, that means the site has to be secured - a lot of times security through obscurity is relied on by putting in remote locales, fake locales, etc. This also includes securing the device - it should detect tampering and default to a 'tampered state' if tampered, one that is encoded at manufacturing so it cannot be changed. While I'm on it, I would love to start seeing TPM 2 chip and Sel4 OS a mandated minimal requirement for all SCADA devices. The cost would go through the roof - strike that, the atmosphere - but it is so necessary.

My wife is a sewer commissioner for a tiny sewer district up in the mountains. It is a state of the art facility from a water treatment perspective since the treated water exits into a mountain stream. However, they recently needed to update their control systems because they all currently run on Windows98 and can no longer get spare parts from their vendors etc..

At her request I sat in on a meeting with their vendor to ask some stupid questions (but less stupid than what the commissioners would have asked) about their proposal for updating the control systems.

Some crazy things discovered:

* it's all windows, all the time. I don't know why but I find this disturbing, but apparently all the drivers for these systems only work on windows. I guess Linux is only just starting to penetrate this market.

* yes, remote access is a thing (even in the old system), especially in this case where the plant it under 20 feet of snow half the year. They claim the system can only monitor things, not control things, but I wonder how locked down that really is. Security is via a VPN.

* the vendor at play here seems to be a tiny shop of a few people that basically serve a few dozen sewer treatment plants in the area. They deal with replacing the hardware and programming these interfaces in.. kind of a crazy local cottage industry I just wasn't really aware of.

Having a background in web services / unix left me pretty unprepared to really do much more than ask some stupid questions around security / obsolescence, but the vendor had reasonable answers to my concerns. Just a very different world than what we think about every day.

Omg that really is crazy and disturbing that a small development shop would hitch their wagon to the most popular desktop OS in the history of the world.

Does the popularity make their jobs easier?

Is that a serious question? Is selling Windows software that a user can install in 5 minutes easier than selling Linux software which likely requires a municipality to buy all new computers (and all the local government budgeting that involves), teach all its employees a new OS, standardize on things like distros and window managers, and then maybe be able to run it? Yes, it's much easier.

Where it's sketchy to find Windows is inside the chassis of some piece of specialized equipment with its own control panel. You may not even realize that what you're working with is a full-screen Win32 app until you get a bluescreen or taskbar popop or something. Then you learn that in addition to the domain-specific software/hardware and what you're trying to use it for, you've got a whole Windows desktop worth of reliability risk in there.

If it's just a touchscreen or hardware buttons, you may need a key to open a panel where the keyboard/mouse are, or to bring your own keyboard/mouse, in order to navigate the Windows part of the interface to fix the problem.

Obviously if it's intended to be networked to existing desktops for control over a LAN, the client software will be Windows, but even then you hope the server isn't.

Everybody knows Windows a bit. Even my retired mom who isn't a native english speaker could click few times on Next button, which might be enough to deploy controlling software with default settings. Good luck with Unix.

Outside our tech bubble working on Unix is still perceived as hollywood-style hacking of the bank and folks really have no clue, unless its done in GUI just like... you guessed it, Windows. Folks wouldn't even touch it.

OMG did they really pick an OS that's so known for its unreliability that it coined "the blue screen of death" and that people now expect it to update without their consent?!?!

Windows is an awful choice for any system that must be reliable. I know, because the industrial control system at my workplace used to be windows, and cost us mega bucks every year in down time. Now it's Linux, and we haven't had ANY downtime in over two years.

Desktop OSes are for browsing the internet and editing documents... complex, rapidly evolving, not very reliable or sensitive to reliability. It's terrifying enough to involve a Mac in a stage performance.

Re Windows all the time: I don't think Linux is the right alternative. GPOS use should be as minimal as possible for safety-critical systems like this.

This exactly. I've come across linux systems doing control work in pentests before, and they can be just as miserably protected as windows systems -- possibly more so, because they're on some forsaken SoC that needs a custom kernel and userland to work, and it hasn't been updated in 15 years, there are privilege escalation attacks galore and the vendor has some backdoor user account enabled.

All this to say, Windows isn't the worst thing ever necessarily, and ideally you want a system that's purpose built and only does what is required with the least amount of code/systems possible.

Is this akin to the "everyone gets pen tested eventually, whether they want to or not" joke?

Also I would guess whether we hear about it or not is more likely to be a political decision. Can the news be leveraged for whatever change we want to see? Or is it too embarrassing and works against us? Sometimes it doesn't need hiding, so then we find out. (Cynical, I know.)

This is negligence on behalf of the water authority.

10+ years ago I used to do demonstrations of this specific attack for govt and other critical infrastructure owners. In our demo, we had a two small vats of coloured "chemicals," with pumps controlled by a bank of PLCs, an HMI and a network of a bunch of vulnerable VMs. We would walk people through how to exploit trivial web vulnerabilities that would get them to the HMI and cause the pumps to reverse and cause a chemical spill from the vats. The look on their faces when they caused a vat to overflow themselves with very little skill, was their "scared-straight" moment.

It was 90's, even 80's level hacking. There is no way the managers didn't know this was possible and likely. They're only acting surprised. What this really looks like to me is a test run for something more coordinated and worse.

Why would you ever hook that shit up to computers that are online? The purpose of the utility fees is to employ people at those sites to govern the operation of those resources.

"Online," being operative term. The demo showed a standard vulnerable web app whose back-end provided access to the corporate network, which had a network link to the HMI workstation, which controlled the PLCs that actuated the pumps. You didn't even need to do stuxnet level PLC code injection, you just logged in and worked the HMI.

Not sure what kind of filters protect against this kind of thing, but a pH sensor with an alarm might mitigate the family level risk somewhat.

The EPA should hire pentesters to go through each utility and expose problems with this. Seems like it would do well in exposing the problem

You're assuming that the goal here is to achieve security. It isn't. The goal here is to achieve the appearance of security. And that is not an entirely unreasonable approach because appearing to be secure can discourage people from even attempting an attack. Your home security system won't actually keep anyone who knows what they are doing from robbing your house if they decide to rob your house. What it could do is make them decide to rob your neighbor's house instead.

I think the analogy falls apart because clearly trying to appear secure isn’t deterring Russia/China from hacking US infrastructure.

That's true, but this almost certainly was not a state-level actor. When Russia or China decide to do damage to our infrastructure, they will almost certainly succeed. The deterrent in that case will be retaliation, not the appearance of security. But in no event can our civilian infrastructure be actually protected. It's just too hard. There's too much of it and there are too many people who legitimately need access.

>When Russia or China decide to do damage to our infrastructure, they will almost certainly succeed.

What if there are 100 attacks per year, with a 99% success rate?

There'd be 99 hacks and they missed just one.

I don't get what you are asking.

What if there were lots of of successful hacks and this just happened to be the outlier because an admin was online and looking at the system at the exact moment the hack occurred?

Any large adversary would compile a huge number of ways to disable infrastructure, then use them all at once, rather than one at a time.

The quote "When Russia or China decide to do damage to our infrastructure, they will almost certainly succeed," already made that point though.

Maybe they are. But we can estimate that they're not too impactful - otherwise we'd notice either the attempts or the effects. If there are ongoing successful attacks, they do not cross the very important threshold, past which US government officials would start making public noises in the direction of nuclear chain of command.

Russian government has quite successfully attacked our information infrastructure and goaded us into attacking ourselves. They didn't need any hacks; they walked right in the front door of Twitter x Facebook, et al.

> Your home security system won't actually keep anyone who knows what they are doing from robbing your house

Sure but it might keep out all the people who don't know what they are doing. Like some clueless junky.

Fun fact: addiction is a disease and not a choice. It is strongly related to trauma, ptsd, income inequality and other factors individuals don’t have a choice in.

Classifying someone as a “junky” adopts a cruel and unrealistic worldview blaming the victim and absolving a number of powerful actors of their role in contributing to addiction on mass scale.

A junky is a victim of powerful actors contributing to addiction? Presumably if we're talking about someone who got addicted to precipitation opioids and then turned to heroin, yes. You do have a point there. It's like the partner that cheats because she feels neglected. Yes that was a contributing factor, but at the end of the day nobody put a gun to their head - it was a (bad?) choice that they made. Yeah physical addiction to prescription drugs is a little different, a little closer to having a gun to your head, but still.

What about those who were recreational drug users that went down a dark and dangerous path? Presumably in this case it's fair to put the blame on them. Yeah, maybe they had issues with their parents and maybe they had some other problems. But nobody put a gun to their head. They made a bad choice and paid the price. That's how life is.

You can't absolve personal responsibility either.

> What about those who were recreational drug users that went down a dark and dangerous path?

What about the rev users who didn’t go down the dark path?

This precisely is the difference of having genetic predisposition to addiction, trauma etc.

No, it’s not a choice more than obesity / food addiction is.

If you’re interested in expanding your thinking see the work of Gabor Mate.

> This precisely is the difference of having genetic predisposition to addiction, trauma etc.

Maybe. Or maybe just the culmination of a series of bad choices.

> No, it’s not a choice more than obesity / food addiction is.

That's a choice. Nobody is making you buy the potato chips and cheesecake. Nobody is holding you back from the gym. It might annoy a lot of very fat Americans, but you've got effectively the same DNA as everyone else - but they're not as fat. It's a cultural thing and a choice.

Yes I know I'm glossing over some nuance here, but personal responsibility remains. Ignoring that seems like just resigning yourself to be blown wherever the wind takes you.

Take responsibility for your decisions, take a good honest look at where you are, what you don't like about that, make a plan to change that, and then execute it.

You're arguing against the clinical consensus on addiction, which is backed by decades of evidence. You're going to have to do much, much, much better than just saying, "well, I think that's wrong." You're no more justified in your belief than a climate change denier.

A clinical consensus on whether addiction is a choice? Perhaps you are thinking of a clinical consensus on how best to treat addiction? This is often glossed over by the "addiction is a disease" crowd. The efficacy of treating addiction as a disease does not make it any more "true" as an explanation for the circumstances leading up to it.

Are we taking about drug addiction or food "addiction" here? Because despite using the same word, those are physiologically very different things.

I think I'm familiar with the consensus views. What am I saying that you feel is unjustified?

> No, it’s not a choice more than obesity / food addiction is.

I agree that obesity/food addiction and drug addiction are both on a similar level, but to me they both sound like a choice.

Mostly because if we decide to apply the same standard you use for qualifying whether something is a choice or not, then literally nothing is a choice.

A man who physically abuses his wife and children? Not a choice, he is doing it simply because he had some experiences in his life that made him that way or he is just simply genetically predisposed. Someone cheats on their partner? Not their choice, they were predisposed to do it or their partner was just not making them happy enough. Harvey Weinstein? Not his choice, he was just a victim of his own circumstance who was also genetically predisposed to do what he did.

You get the idea, but I honestly do not believe that most of those things are anything but choices, unless we decide to pretend we believe in predetermination and complete lack of free will. In which case, I would agree with you, those decisions were not choices, because choices don't exist in the context of belief in predetermination.

Mind you, I am not saying there are no circumstances in which those situations would not be a choice. There are indeed some scenarios like that. However, and that's purely subjective based on my personal observations, those scenarios are a minority.

The logical trouble you are having is trying to fit an analogy multidimensional phenomenon into a binary model.

To be fair, I simplified, and I agree with your critique of what I said.

I don't think it is a binary choice vs. not choice. If speaking frankly, I do believe that any of those things discussed feature both a choice component and a predisposition component. The proportion of each of those two components varies heavily based on a person and a situation.

However, even with this in mind, it is difficult to say that the choice component is ever at zero.

We can discuss to which degree humans have a choice in their actions, because it is definitely debatable, since they all agree on the ground truth that there is always some degree of choice. But that discussion cannot coexist with a belief in total predetermination, since that removes any ground truth completely.

> to me they both sound like a choice

Addiction Is a Brain Disease, and It Matters: https://science.sciencemag.org/content/278/5335/45

Ok but I still don't want them impulsively robbing me when they are coming down and need money for the next fix.

The article mentions that a huge number of water treatment facilities are controlled by publicly-accessible web portals.

Maybe the goal isn't to make every single facility 100% secure to all threats, but it's totally defeatist to say that we couldn't somewhat increase the actual security of many systems. VPNs alone would provide a great first line of defense, and even for a government-run utility, the barrier to installation is very low there.

I have a fair level of skepticism around security theatre, too, but I think it's unfairly cynical to suggest that we don't have a goal of making ourselves more secure.

There ought to be a second principle here:

https://en.wikipedia.org/wiki/Kerckhoffs's_principle

In addition to assuming the adversary knows how everything is designed (as opposed to security-by-obscurity) there should also be an assumption that security is obtained through real implementations, not theatre.

> Your home security system won't actually keep anyone who knows what they are doing from robbing your house if they decide to rob your house. What it could do is make them decide to rob your neighbor's house instead.

What happens when everyone in my street buys the same security system? Seems like we all just spent a lot of money buying something that now offers us no benefit.

Short of purchasing a honeypot property with (seemingly) no security system and going full Home Alone on the inside of it, it seems like you're going to need actual security at some point.

They target a poorer neighborhood

There's an old tale about this where everyone switched from brooms to vacuum cleaners to save work, which then caused a coal plant to be opened for the increased power demand, which proceeded to dirty everyone's homes so everyone was working just as hard for a slightly dirtier home than when they were using brooms.

At least for the home example, you're usually better off reducing the factors that would make a break-in happen in the first place rather than trying to win an arms race against a determined attacker; there are high trust societies where people don't bother locking their doors and people don't get burgled. But obviously the threat model is different than a utility.

> Short of purchasing a honeypot property with (seemingly) no security system and going full Home Alone on the inside of it, it seems like you're going to need actual security at some point.

You mean men with guns stationed 24/7? There is no "actual security" against determined burglars available on the market for regular people. There's delaying action (e.g. laminated glass), noise making and deterrence (cameras). Each can be worked around.

This is also a general point on security: there's no perfect security, there's only adequate security - where "adequate" means it costs the attacker more to breach it than they can expect to profit. From this view, appearance of security is almost as good as actual security measures - you make security look expensive to breach, which achieves the same effect, as long as nobody is willing to call your bluff.

I think you're vastly overestimating the budget the EPA is given and mistaking the goals that that money is meant to accomplish. Making some vendors that give politicians money look bad isn't a desired outcome.

Politicians don't run low level operations, and security contractors can give money to politicians too.

You need someone like the US Digital Service doing this with DHS. This gives you the force of federal law enforcement but the competence of skilled infosec practitioners on tours of duty short enough to enable high (“market rate”) comp but also results driven engagements. Bring back Chris Krebs, have this executed as part of CISA.

Yeah, I hate to agree with throwing DHS at something, but this is exactly the kind of thing they should be doing.

This sounds like the best use of DHS I've heard of since their inception.

The refrain continues that we automate more than we're able to secure.

Its all fun and games until somebody wants to kill people and they suddenly can.

The FDA has gotten more serious about pacemaker security due to some incidents that were entirely accidental in nature but implanted medical devices having any sort of wireless networking capacity scares the willies out of me.

It's extremely easy to kill a lot of people in myriad ways. The only thing protecting us is that most people aren't interested.

Most of them require you to be on the same continent as the person you're trying to kill. The easier you make it to kill, the more abstracted it becomes, the more disconnected the killer is from the act itself, you increase the amount of killing and the number of people who will be willing and able to do it.

Florida sunshine laws are the reason we heard about it. The press conference was a good sheriff getting in front of news that was coming out sooner or later in the investigation.

(This is also thought to be the reason "Florida Man" is a thing: https://www.oxygen.com/florida-man-murders/crime-news/why-ar...)

What I still haven't figured out is, why does a municipal water system, intended to provide drinkable water to a city or cities, even have a treatment system with a setting for "poisonous concentration"?

Shouldn't the system notice when the "knobs are turned up too hi" and disallow that?

a la "The front fell off":

Well, Um, these water plants are build to very rigorous webscale cyberstandards: They’ve got to have a ON/OFF switch. We used a bounded model checker to show nothing bad could happen in a maximum number of transitions.

What’s the maximum number of transitions?

Oh,… one, I suppose.

Plants still have skilled human operators that know better what they need at that exact moment than whoever programmed the system ages ago, like a pilot should be able to fly a plane according to his inputs, so unless you are a system programmer that wants phone calls from all of the projects you have ever worked on saying “the system won’t let us do x” you give the owner all of the tools they might need to do whatever they might want with their equipment.

Confirmations to ensure setting is entered accurately and logging of changing of settings and by whom are good ideas though.

> you give the owner all of the tools they might need to do whatever they might want with their equipment

Are you quite confident that is how engineering works?

The 737 Max fiasco to me is a sign that yes, this is how engineering should work.

For example, a well engineered plane allows the pilots to control the fuel flow to engines, but also does not allow pilots to dump fuel into the passenger compartment. Much engineering goes into, how to isolate the toxic chemicals of fuel or other fluids/gases away from the passenger compartment. So you have freedom, but you also have constraints.

Likewise, a well engineered plane probably should allow pilots to fly the plane according to their needs, rather than overriding their inputs as they fight to point the nose up while the plane decides that no you really want the nose down right now. Of course the pilots should've disengaged MCAS, but in a crisis it's bad engineering to force operators to remember which unhelpful/murderous system needs to be disabled in order to avoid catastrophe. The system took too much agency away from the operators, and violated the principle as espoused by that comment you're quoting.

You're conflating "owner" with "operator". The airlines bought the 737Max because they calculated it could increase profits. The operators – pilots – were not sufficiently involved in designing, implementing, or documenting MCAS to even know they should be advocating for agency.

Do you have a different approach to providing equipment to utilities?

If "building in safety mechanisms to protect human life" is a different approach than what's done now, then yes. There's a reason the US never built and licensed commercial nuclear reactors like the RBMK, and it's not because the utility companies said they preferred the design that's more expensive to build and operate.

Sure plants have safety mechanisms to protect human Life. So does your car, but it will still let you do stupid things with it. Sometimes they are ok: person in the car is injured and I need to drive in a way that might normally be unsafe. The machine can’t make the judgement calls about when it is ok to operate outside of the normal desired operating envelope. Machines have physical limits and usually they are built to protect themselves from damage, but you can still crash a plane or a car or put a brick on your gas pedal and leave the car in neutral and destroy your engine.

In this specific case I imagine the plant wasn’t physically capable of dosing at that rate - and there should be alarms if the outgoing water is outside the desired operating parameters.

Sometimes such things are needed. Let's say that system gets contaminated with E. coli or like. Only effective way to combat this is to inform the consumers and turn certain chemical concentrations high enough that it is know they are effective in solving the issue in the network.

Something like that should be part of a separate and independent "emergency response" system that is offline normally. The day-to-day machinery for normal operations should have limits. If something like E. coli outbreak happens, you go "break the glass" on the emergency controls. You don't have a knob that goes from off to normal to deadly.

> “The system wasn’t capable of doing what the attacker wanted,” said Joe Weiss, managing partner at Applied Control Solutions, a consultancy for the control systems industry.

The UI will be designed for people who know 11,100 ppm is a stupid level, so it's probably light on fancy error messages.

You might as well ask why Google has a setting where I can type "' SELECT * FROM NSA_SECRETS;--" into the search box.

> The UI will be designed for people who know 11,100 ppm is a stupid level

Therein lies the problem: assuming your users will never be malicious or make mistakes.

Note also that you can type whatever the hell you want in a google search box. I could type "launch all nuclear forces at Russia", but that isn't going to actually happen.

>But Marcin says Teamviewer would actually be an improvement over the types of remote access systems he commonly finds in his own research...

I agree with this.

I imagine that we will find out that the attacker was indeed someone who was given access (at some point in time) and not an attacker from the internet

I briefly worked for a company making a web-based product where the security was abysmal.

What was shocking was that I had to explain to the lead engineer the "never trust the client" security concept. A few hours later he found a major exploit.

I suspect that there are many small software companies with similar problems.

You dont need to hack Florida water from my experience.... the quality went down monthly in my ex-city... I had a well and was unaffected but the city need to flush the pipes almost monthly because the people were getting brown water....

It was probably Teamviewer or something similar.

It has been widely reported that Teamviewer was indeed used.

It has been widely reported that if you (two) would read the article, it mentions TeamViewer was used.

But more interestingly, the article mentions it was widely reported.

It's pretty scary that without even reading the article we already know which piece of software was responsible - yet somehow some brightspark thought it was a great idea to:

1) have the computer running this stuff on a network which faces the internet 2) install teamviewer on it

I mean.. what did they expect?

Yes, KrebsOnSecurity reported that TeamViewer was used. Can’t find the article right now though.

It is interesting to try and work out what would have happened without our heroic operator/QC present.

Yes, obviously - or perhaps not, to some - no designer of the physical plant in their right mind would consciously spec the ability to raise levels so high so quickly. The thinky box can ask for whatever it wants but that don't make it so.

That said, for that sort of town it should only require a tonne or so per hour NaOH to raise to that level. Which is throughput you can get from any generic .

Depends on the tank size, and how much NaOH is in reserve, I can see one of the outcomes being "yo, why's the hydroxide tank empty?". Or "customers reporting funny tasting [but not harmful] water". I'm not convinced that dangerous levels were impossible, however, and I was expecting to be.

Good Ars article, by the way. "Breached water plant employees used the same TeamViewer password and no firewall"

https://arstechnica.com/information-technology/2021/02/breac...

I used to live in Oldsmar, that’s frightening

Surely there's mandatory reporting on events like that?

Couldn't the involved staff be jailed if they tried to conceal it?

Normally they jail people for releasing "state secrets" not the other way around

The United States will not be a part of the “developed” world in 20 years; too much aging infrastructure is being neglected because the country is too corrupt to actually govern effectively.

I’m curious what aging infrastructure you’re referring to.

Where I live, things are very nice and it’s maybe fairly unique to my exact county.

But if you are referring to overall infra like highways or something- I don’t share the sentiment it’s in anyway a problem where we live.

Im not plugged into interstate trade or things like that. So maybe I’m not totally aware of the problem. But it hasn’t seemed locally problematic.

Edit: downvote, really? I’m trying to know more about crumbling infrastructure.

Didn't downvote you, but its a pretty well known issue https://www.infrastructurereportcard.org/wp-content/uploads/...

tl;dr the USA earns a D- in infra.

I've driven cross country a bunch of times. I'd give the roads a lot better than a D. I have to wonder if this institute is biased in some way.

That’s like your broker saying your investment strategy is D- because you don’t have enough money in his firms managed funds.

The D- is a bogus rating meant to increase spending, not improve value.

https://www.strongtowns.org/journal/2011/12/12/best-of-blog-...

https://usa.streetsblog.org/2013/03/20/is-the-asce-failing-t...

One of your sources is a decade older than the link, and the other is quoting a guy critiquing their methodology but "Marohn said he could be convinced that more spending is justified, but not before he sees a smarter analysis based on the return communities get on their investment."

You probably could find better ones without much effort, as I also didnt use a ton to find mine in the first place - I was just referring to a document that was pulled up when the bridge collapsed in my home town.

If you want to quibble about specifics sure, but that's different than the parent's "I dont know what the conversation is about"

There's a ton of literature out there on America's infrastructure problems.

This is a good summary from a neutral source: https://www.cfr.org/backgrounder/state-us-infrastructure

These studies aren't nearly as dire as the grandparent comment that claimed the United States wouldn't be part of the developed world in 20 years.

Much of that report focuses on things like traffic congestion and delayed flights from busy airports:

> According to Petroski, the delays caused by traffic congestion alone cost the economy over $120 billion per year.

Ironically, the abrupt shift to remote work and work-from-home due to COVID has significantly reduced traffic burdens in many cities.

I'm all for improving infrastructure where necessary, such as the ~10% of bridges that are structurally deficient or improving access to cheap broadband. I'm not enthusiastic about rallying cries to encourage more car traffic, though.

Water technology infrastructure is not what I'd really call "aging". Most municipal automation systems are quite old, but they're very functional. The underlying technology is called SCADA, which is still used today.

SCADA is an offline ecosystem by design because the innards of the system are mostly just signals flying around. There's no encryption or IAM in place to defend against bad actors. More recently the cities want their municipal infrastructure that runs on SCADA to integrate nicely with other information systems to either retrieve data which is useful for generating reports, like the delta between water pumped and water served which indicates total loss. The other use case is to perform actions. If I know there's a fire on one side of the city that is a four alarm fire, which will put pressure on service lines as multiple firetrucks begin to demand water, I can decrease pressure on one section while increasing pressure on another. You could perceivably do this in an automated fashion, but there's no such capability in any SCADA systems I know of. Hence, municipalities end up finding integrators who will build them custom devices to interface with their systems.

So, this is less an "aging infrastructure" problem and more "new capability demand".

We hearing about it is just what a water company such as Nestle would want as a ploy to get more customers buying filtered water..

> Gualtieri told the media that someone (they don’t know who yet) remotely accessed a computer for the city’s water treatment system (using Teamviewer)

You can't just hack Teamviewer.

Possible by a national actor, not from a script kiddie.

National actor is obviously not going to change to a value that sounds dangerous but will be quickly found and does nothing.

So it's a insider job, I'd wouldn't rule out someone sat on a keyboard. I'd even put it out there perhaps no one remoted in.

Reply on news.ycombinator.com