Hacker News Reader

Comment by kevan | Hacker News Reader

There's definitely some advantages to using age-old practices. It's easy to explain how trust and verification works in a system when 2 people have separate physical keys and you need both to perform an action. It's a lot harder to explain things like modern crypto to most people. Until a critical mass of the decision-making chain understand the tech we may be better off not using it for mission-critical things.

Alternatively, this could also be a misguided effort that holds back smart-grid tech in America by a decade. It's really hard to tell.

View on HN

My intuition from experience is that in sufficiently complex systems (basically all modern hardware + software systems), things break from humans misunderstanding or not knowing the rules far more often than from physical failure.

If there's a reason for using an electronic system, by all means.

But for God's sake air-gap it, compartmentalize functionality with clearly defined interfaces, build simple fault-tolerant monitoring, and make final controls hand-on-metal.

When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.

>When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.

My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.

I wonder what would happen in an actual battle with the fog of war when people are pushed beyond exhaustion. What happens when senior command is dead or missing?

There exists a very distinct line between routine/exercise and war.

You may have missed the opportunity to experience this distinction first hand, but I can assure you with the utmost confidence, the meta does indeed change.

What event was this?

Sounds like the US Navy destroyer that collided [0] with a cargo ship.

[0] https://en.wikipedia.org/wiki/USS_Fitzgerald_and_MV_ACX_Crys...

Based on google results, it appears to be an incident where the USS McCain collided with the Alnic MC near Singapore. She was getting off course, and poor UI made it so hard to steer her back that she crashed. Injuries and deaths resulted.

https://medium.muz.li/ui-confusion-almost-sunk-a-us-destroye...

Though one could say that this was due to too little automation rather than too much -- the default behavior could be for the ship to steer itself away from another vessel or shallow water unless overridden (setting off alarms that this action is being taken)

My $30K car does a fair job of nudging the steering wheel if it thinks I'm running off the road, surely a $2B battleship could have some basic collision avoidance capability too.

Unfortunately military ships are expensive and have strict schedules. You may be able to drive your car to a shop any day to diagnose a sensor error but military vessels dont have that luxury.

To my knowledge the detection systems (for detecting ships ect) were either not calibrated/failed making a ship impossible to detect. There may have been a serious lack of training spotting sensor failure too. The ship was due for repairs and had been put off. Also, according to SOP there should have been someone making rounds and looking out and identifying ships. Because honestly how do you miss a huge cargo ship lit up like a Christmas tree

There were multiple critical failures leading to the crash. Bad training, poor maintenance, and lax procedural compliance.

The problem with automation is that it works perfectly until it runs into that edge case which nobody thought about. The problem with the real world is that there are infinity edge cases.

I'd like to use this in my .signature.

> there are infinity edge cases.

this is simply not true. there are finite particles in discrete space so there's no infinity here.

I would call it O(N!) with N increasing over time opposed to infinite and essentially something everything technically faces automated or not - it is just those lacking agency cannot change deliberately.

Not because it isn't effectively beyond linear counting or coverage but because it describes how it grows - in different permutations. Especially with actual other actors involved and literal arms races. Sure your expensive plate armor may stop blades, arrows, and even current musket fire but would it stop cannonfire?

All of which 95% of those edge cases are most likely programmed leading to saving humans countless times in the most situations to the point they don't even realize it.

Why do you think flying is so safe? It's because of automation and redundant systems.

Do things go wrong? Yes. But imagine how many things would go wrong if you gave people a 15,000 line check list.

I didn't mean for the automation to be primary - I meant for it to be a llast-ditch collision avoidance system, like the steering wheel nudge on my car.

Humans should still pilot the ship, but if they are going to let a collision happen, then the ship should try to avoid it even without human intervention (unless they press the "I know what I'm doing and want to do it anyway" button after the ship sounds the alarm saying it's taking evasive action)

So the system doesn't have to handle all of the edge cases, just the "We're on a collision course with that ship ahead" case.

A lot of problems with automation vs manual controls is that the line between those tends to be blurry.

It's not that automation wouldn't work in the absolute majority of cases, and that we couldn't define the cases where automation will have to know it cannot make it.

It's not that humans would always be better than automation: humans make mundane mistakes and casual accidents happens, and humans make bad judgements also in extreme situations.

Looking at the catastrophes in aviation and shipping from the recent years the problem seems to lie where automation and human controls interact.

In the typical case the automation is not completely switched off while the human still tries to take control of the system and some override functionality still triggers the automation without the human realising that.

Maybe there's a sensor failure that causes the automation to misbehave. Then the human controller turns off the autopilot and begins to apply the proper fix but the electronic system, through its faulty sensors, sees it differently and concludes that the human is doing something way too dangerous and doesn't let him. Then a tug of war begins with the human controller where the automation is thinking that the situation is dire and the ship must be saved while the human realises that something is not working properly but in the heat of the moment doesn't have time to diagnose or narrow down to shutting off automation and just keeps applying more.

If only the automation knew it can't "see" properly and the human is already trying to fix things, or if only the human knew part of the automation is still active and secretly, with no indication, countering the proper corrections by the human controller.

Sounds like these systems need a little red killswitch under a plexiglass cover. It's baffling to me that anyone who's ever designed an automated system thinks there won't be situations where a human has more complete awareness of what's happening and needs to unambiguously wrest all control.

It's usually not that simple. The majority of complex systems have substantial portions of their operation effectively invisible to us.

Electricity, radiation, remote or sealed mechanical systems, deeply layered software.

In all of these, you need systems to communicate actual state to a human to effect even a manual resolution.

Operators may cause slight SOP deviations in emergency situations. Anticipating this in design between human controls and automated function is going to be very hard. I hope we can get voice input or other advanced input devices setup for communication between systems.

The input isn't the problem IMHO, it's the output.

If a human is in the loop at all then it's because they may be expected to make a decision. And that decision is based on information. And that information needs to be presented unambiguously.

One of the smartest things I read on feedback here was to start with a balance-in, balance-out counting system.

It's unambiguous: if the number of things being processed and output is different than the number of things input, that tells me something.

And that's where automated systems get dangerous. It's incredibly risky if instead of informing and shrinking their response space, they expand it in an attempt to auto-correct the issue.

More than one large power outage has been caused by a human operater manually grounding live transmission lines by mistake.

Sounds like an interesting situation, got any good links on the topic?

It's an extremely common situation with interesting solutions. If you've experienced the power going out, then coming on a few seconds later, then going out, and coming back on a few seconds later, then going out and staying out you've experienced a "recloser" attempting to clear a power fault (tree falling on power line perhaps, the recloser goes through the motions in an attempt to burn the thing away and clear the fault).

A quick google turned up this fairly lengthy page https://www.electricaltechnology.org/2018/03/electrical-prot... but the protective systems are covered in depth in basic power transmission tech texts. Also here https://en.wikipedia.org/wiki/Recloser

Cool, thanks!

Nearly this exact scenario has been bugging me for a couple months. I started thinking about it due to deployment issues.

Why don't we have a proverbial two key system for certain activities like deploying to production or deleting user accounts, or backups?

We make do with things like pull requests, but everywhere we've ever had retros, I've run into situations where someone did something very stupid, somebody else rubber stamped it, and now we have a mess. Approvals on an action do not approximate the ritual sobriety that is embodied in two independent humans deciding to turn a key.

There are some people for whom these two actions can be considered equivalent, but we tend to bag on them for their meticulousness.

I think I'm wanting a tool where I type "deploy 1.0.12345 to production" and it sits there waiting until someone else types "deploy 1.0.12354 to production", and then it stops us because one of us transposed some digits.

Someone did release a Pam module? That required confirmation before someone sudo-ed. I thought it was Stripe but I can't find it right now.

Hey, that was me! See the sister comment.

It's actually a plugin for sudo (surprisingly yes, sudo has plugin capabilities[1]) and not PAM. I had originally developed it as a PAM module, but the sudo plugin API allows for the neat trick where the TTY is mirrored.

[1] https://linux.die.net/man/8/sudo_plugin

We didn’t “release” it, but I worked on a dual-control PAM module: https://github.com/cjdev/dual-control

It's not necessarily for deploys (though you certainly could make a workflow that uses it to do so!) but I developed a dual-control plugin for sudo while at Square. We use it extensively.

https://github.com/square/sudo_pair

In action: https://raw.githubusercontent.com/square/sudo_pair/master/de...

Doesn’t that encourage using sudo in interactive mode rather than elevating for just the required commands?

I suppose you're right, but the reason that's generally advised against is so people don't run stupid stuff without thinking while root. I'd guess you're less likely to do that if a peer is watching.

And to stouset: that's a seriously cool project!

> Doesn’t that encourage using sudo in interactive mode rather than elevating for just the required commands?

It might, but policy could specify not allowing interactive mode for situations that dictate a two-man rule. Violation of said policy would necessitate a review and possible HR-related action depending on how serious the company wants to treat things.

https://blog.cloudflare.com/red-october-cloudflares-open-sou...

we have some of these systems at AWS where you need two people to approve a commit, or a type of change operation. Basically, introduce friction at various points in the name of safety.

Yeah I specifically called this out as a stopgap not a solution. For reasons of ergonomics and suggestibility (see also the primacy effect) you are more likely to make a bad judgement call when prompted into it than if you have to reason it out for yourself.

I have seen a contactor SCADA controller (high voltage physical relay) for a high voltage transmission line at a Florida substation on the public internet (no airgap, no vpn) during an audit.

Apply brakes while we work on the steering. This is good policy. “Internet Of Shit” is no joke.

Industrial controls at least work most of the time. Can't quite say that about consumer gear.

That does not surprise me in the least. Some of the stuff I came across doing DD would make your hair stand on end, after 130 companies I'm more surprised by the ones that do it right than the ones that have no clue.

Agree entirely.

In a hacking group I'm part of, we had an address that ended up being a passwordless VNC to what looked like a centrifuge controller. It was at a place whose IP tracked to CERN.

We reported it as fast as we fucking could. It took about 6 hrs for them to get it off the public internet with no password.

Thank you for reporting this!

We do what we can.

From our assessment, it looks like that was a complete mistake. And those do indeed happen. But what if someone were to have found it and were malicious, and started randomly mashing buttons?

That's why our jobs as sysads are never done. We occasionally do boneheaded stuff. Users also do. And so do contractors. And I'll do what I can when I see something wrong. Professional courtesy.

Why not create a closed government only internet or intranet where all grids connect to each other but none have access to the main internet. No computer in the network can connect to the internet and the govnet, maybe even have a different protocol and other unique features. They could also use ai to determine if a command to say shut something important off makes sense if not notify someone in charge before following through on suspect actions.

The “smart grid” we are working towards does things like scheduling your AC and dishwasher to turn on/off depending on supply and demand fluctuations. To that end, every node, both consumers and energy producers, needs to be connected. To do this completely independent from the existing internet would amount to doubling the cost of our communication infrastructure. It would also defeat your purpose, because suddenly everyone on the grid has access to the network again, and could try to attack it.

You also need some connection, somewhere, to the “regular” internet. As just one example, consumption data is the basis for billing, which needs to connect to financial infrastructure.

And, finally, even air-gapped systems are vulnerable. See Stuxnet.

Logistics and attendant expenses most likely - especially with practice to gain any actual benefits.

Say everyone has a registered nonrepudiatable cryotographic certificate - a decent idea in theory. However apply it to every accessible grid communication part including ones which may cycle in and out and could have secrets extracted. Suddenly it doesn't provide as much benefit if a hacked SmartMeter can get in anyway and there is deniability. Better than nothing but you could have likely gotten better bang for your buck with strict firewall rules or hiring a swarm of consultants.

Similarly vetting commands has two problems - one is that reaction time may be critical. Second the judgements themselves are complexity and may be a source for errors - especially when they are "triage" ones that cause a shutdown or damage to avoid greater damage

> Why not create a closed government only internet or intranet where all grids connect to each other but none have access to the main internet.

An air gap didn't help Iran against Stuxnet, or the US DoD against agent.btz:

* https://en.wikipedia.org/wiki/2008_cyberattack_on_United_Sta...

It wouldn't hurt, and it's probably worth doing anyway as a risk mitigation, but keeping it completely sanitary would be impossible.

Does smart-grid tech work any better than smart-home tech? So far most 'smart' tech hasn't had a good track record on security or reliability, certainly not at the level you would trust your major infrastructure to it.

Keep in mind most "smart home" technology is marketed at consumers who have no idea what SSL/TLS is (nore do many care sadly even though they should), and are meant to be as cheap as possible.

While I have no clue what the security or reliability looks like in industrial applications, I would hope they evaluate their solutions with reliability in mind.

Most industrial application also done know or care what ssl/TLS is (however this is starting to change). I have had industrial control systems reboot by just doing an nmap scan.

This

It's barely better than iot stuff. And some "older" systems probably still need NT4 to run (I mean, if you're lucky)

Hah. My hope died with the 30 ICS-CERT emails I’m bombarded with each day.

>While I have no clue what the security or reliability looks like in industrial applications, I would hope they evaluate their solutions with reliability in mind.

Keep hoping... The number of industrial systems deployed that are connected to the internet with the default passwords unchanged or no passwords is nauseating.

https://threatpost.com/exposing-scada-systems-shodan-110910/...

Serious question. I've been hearing about industrial applications/medical etc. having default passwords exposed on the internet for years.

How have they not been exploited/fixed already? Shodan's made it clear for years, of course anybody can use zmap/masscan.

People that know the passwords are generally ethical. Plus the risk reward isn't there. Say you have a hack for a pacemaker and you kill 1000 people. Then what? Literally every nation on Earth will be looking for you and you will either rot in prison for life or be put to death. And if the wrong country gets you first there will likely be a lot of torture along the way.

Without having hacked a database of patient to device mappings, it's not like you can ask for ransom.

Frame someone you don't like, short the vendor's stock, or separately hack a hospital to get the patient records. You don't even have to kill them for that, just a recordable, scary event.

Also, you may already want someone in particular dead and are just looking for a vector. This seems almost untraceable.

I've seen several talks where they demoed showing access to hydroelectric dams, etc.

I'm thankfully not this person, but at a basic level I could see some bored teen turning off a dam just for fun.

Honestly the only reason I don't is because it would be "wrong" to do, but the internet is full of people, it's certainly not a stretch that somebody would want to.

It can happen by accident. Some teen has a bot that scans around bricking devices with weak security; bricks critical infrastructure.

For the utilities, absolutely. The utilities have hard requirements of delivering power to all customers while maintaining grid and frequency stability, so the ability to shape demand and supply quickly is at least one very valuable knob.

Yes. The modern sensors and controls being built in to US electric transmission and distribution systems have resulted in a far better run grid: more stable, more reliable, more efficient, fairer, and enabling things like real-time market making which enables rapid improvement using new technologies such as grid storage batteries, etc.

It does come with some vulnerabilities. I'm not sure what the best way forward on those is; I suspect this bill's effects, if any, will not result in a more resilient or less vulnerable grid system. Attackers will just find different leverage points to attack.

Somewhat related anecdote:

I live in the Finnish countryside, and since much of the power grid out here is still ancient, there are power outages rather often.

However, if the power goes out, there's always something that tries to power everything back on automatically after exactly 30, 60, 120 and 240 seconds. If something's still wrong, the power will simply flick on and then off within a few seconds. If an outage lasts more than four minutes, it usually takes much longer to get it running again (I assume it's because it then triggers an alert that will be handled by a human operator).

I would assume there's a lot of similar stuff going on in the US power grid, considering it's immense size what I suspect is a somewhat similar mix of new and very old equipment.

And yes, surge protectors are mandatory here.

> there's always something that tries to power everything back on automatically after exactly 30, 60, 120 and 240 seconds

Sounds like a recloser, which tries a number of times to restore power under the assumption that most faults are transient (for instance, a branch falling in the wires and then sliding to the ground). After a set number of retries, it gives up and stays open; at that point, a human has to locate the fault, clear it (for instance, poking the fallen branch with a stick until it falls to the ground), and then tell the recloser to close the connection again.

Yes, that sounds to be it. I can imagine such systems takes care of quite a lot of incidents in large power grids.

> If an outage lasts more than four minutes, it usually takes much longer to get it running again (I assume it's because it then triggers an alert that will be handled by a human operator).

I read a statistic at some point (which I can no longer find the reference for) that power outages tend to be bimodal: either they last a short time (e.g. <5min) or they last a long time. There's hardly anything in between.

In the US, utilities report outages longer than five minutes:

* https://www.eia.gov/todayinenergy/detail.php?id=35652

It is just because ICS/IoT vendors do not care/know a thing about cybersecurity. If you push security development practices, it should really improve the situation. Just prevent an access to the market until they pass the red team audit. Would surely motivate them enough.

It probably will yield good results for the next decade but will set a bad precedent for the world in general and back us into a corner as confidence in the security for this stuff improves and the next decades passes.

Software does not appear to be on a trajectory of increased security or reliabilty. It has always been hard to verify correctness, let alone measure how vulnerable and easy to exploit software is. Testing software has not been a suitable replacement to a large technical user base constantly tweaking and improving said software.

If software were to become reliable, I'm sure those pioneers in reliable software can pave the way to reasonable use in mission critical, high value espionage targets.

Until then, legislation like this will pave the way back to what has worked for centuries.

Reply on news.ycombinator.com