Hacker News Reader

Comment by cratermoon | Hacker News Reader

What I still haven't figured out is, why does a municipal water system, intended to provide drinkable water to a city or cities, even have a treatment system with a setting for "poisonous concentration"?

Shouldn't the system notice when the "knobs are turned up too hi" and disallow that?

View on HN

Plants still have skilled human operators that know better what they need at that exact moment than whoever programmed the system ages ago, like a pilot should be able to fly a plane according to his inputs, so unless you are a system programmer that wants phone calls from all of the projects you have ever worked on saying “the system won’t let us do x” you give the owner all of the tools they might need to do whatever they might want with their equipment.

Confirmations to ensure setting is entered accurately and logging of changing of settings and by whom are good ideas though.

> you give the owner all of the tools they might need to do whatever they might want with their equipment

Are you quite confident that is how engineering works?

The 737 Max fiasco to me is a sign that yes, this is how engineering should work.

For example, a well engineered plane allows the pilots to control the fuel flow to engines, but also does not allow pilots to dump fuel into the passenger compartment. Much engineering goes into, how to isolate the toxic chemicals of fuel or other fluids/gases away from the passenger compartment. So you have freedom, but you also have constraints.

Likewise, a well engineered plane probably should allow pilots to fly the plane according to their needs, rather than overriding their inputs as they fight to point the nose up while the plane decides that no you really want the nose down right now. Of course the pilots should've disengaged MCAS, but in a crisis it's bad engineering to force operators to remember which unhelpful/murderous system needs to be disabled in order to avoid catastrophe. The system took too much agency away from the operators, and violated the principle as espoused by that comment you're quoting.

You're conflating "owner" with "operator". The airlines bought the 737Max because they calculated it could increase profits. The operators – pilots – were not sufficiently involved in designing, implementing, or documenting MCAS to even know they should be advocating for agency.

Do you have a different approach to providing equipment to utilities?

If "building in safety mechanisms to protect human life" is a different approach than what's done now, then yes. There's a reason the US never built and licensed commercial nuclear reactors like the RBMK, and it's not because the utility companies said they preferred the design that's more expensive to build and operate.

Sure plants have safety mechanisms to protect human Life. So does your car, but it will still let you do stupid things with it. Sometimes they are ok: person in the car is injured and I need to drive in a way that might normally be unsafe. The machine can’t make the judgement calls about when it is ok to operate outside of the normal desired operating envelope. Machines have physical limits and usually they are built to protect themselves from damage, but you can still crash a plane or a car or put a brick on your gas pedal and leave the car in neutral and destroy your engine.

In this specific case I imagine the plant wasn’t physically capable of dosing at that rate - and there should be alarms if the outgoing water is outside the desired operating parameters.

a la "The front fell off":

Well, Um, these water plants are build to very rigorous webscale cyberstandards: They’ve got to have a ON/OFF switch. We used a bounded model checker to show nothing bad could happen in a maximum number of transitions.

What’s the maximum number of transitions?

Oh,… one, I suppose.

> “The system wasn’t capable of doing what the attacker wanted,” said Joe Weiss, managing partner at Applied Control Solutions, a consultancy for the control systems industry.

The UI will be designed for people who know 11,100 ppm is a stupid level, so it's probably light on fancy error messages.

You might as well ask why Google has a setting where I can type "' SELECT * FROM NSA_SECRETS;--" into the search box.

> The UI will be designed for people who know 11,100 ppm is a stupid level

Therein lies the problem: assuming your users will never be malicious or make mistakes.

Note also that you can type whatever the hell you want in a google search box. I could type "launch all nuclear forces at Russia", but that isn't going to actually happen.

Sometimes such things are needed. Let's say that system gets contaminated with E. coli or like. Only effective way to combat this is to inform the consumers and turn certain chemical concentrations high enough that it is know they are effective in solving the issue in the network.

Something like that should be part of a separate and independent "emergency response" system that is offline normally. The day-to-day machinery for normal operations should have limits. If something like E. coli outbreak happens, you go "break the glass" on the emergency controls. You don't have a knob that goes from off to normal to deadly.

Reply on news.ycombinator.com