Hacker News Reader

> Engineers aren’t allowed “one small fuckup” when building bridges and buildings. Doctors aren’t allowed “one small fuckup” in the OR or when calculating dosages.

I do agree with the rest of the post but this is definitely NOT true. Your house probably has small issues here and there if you are willing to pay for an inspector. Doctors fucking up things is not news too, plus you never know if there is "one small fuckup" as long as you don't feel particularly bad afterwards.

The thing is I think you guys regard real world engineering too high. Small fuck ups happen all day.

Exactly. Threat modelling is a thing outside IT secutiry too.

Youtube channel "MentourPilot" [0] has some excellent examples of how civil aviation was built to withstand "small fuckups": with "Swiss cheese" model. [1]

[0] https://www.youtube.com/c/MentourPilotaviation/videos

[1] https://en.wikipedia.org/wiki/Swiss_cheese_model

I think most of the quibbling is over the semantics of “one small fuckup”. It’s in quotes for two reasons 1) because the comment I was responding to worded it that way it and 2) I’m deliberatly calling that characterization bullshit: one _truly_ small fuckup obviously doesn't leave you vulnerable to ransomware. Systems need to be able to withstand of small things, have redundancies, engineers build in tolerances, yada yada.. of course! Were talking about *things that leave your entire business in the hands of ransom ware criminals* or *things that kill people* or *things that crash planes* or *things that cause bridges to collapse*. Those things can’t be tolerated by definition because they are fatal failures.

That’s my point. We would not tolerate companies that build bridges that collapsed all the time and keep letting them build bridges so why should we tolerate companies that get hacked repetitively by ransomeware groups? Insurers are running because it’s not good business to insure these risks. Sounds like the market is working just fine.

"Small fuckups" in home construction which have no material consequence are acceptable.

"Small fuckups" which result in catastrophic failure, property damage, injury, or death, are not.

For the latter, you'll find all manner of standards, codes, practices, maintenance, and inspections.

What we're discussing with regards to infosec aren't "small fuckups", they're repeat major catastrophies, bearing little or no risk to the firms and organisations perpetrating them.

A large component of the risk resides simply in collecting and retaining data, as well as exchanging it with other organisations. That turns out to be a very high-risk activity, though the risks are externalised. The point is to internalise those costs, and to sharply reduce the incentives for the practices which present those risks. This includes incentivising the IT products and services sector toward architectures which can be more adequately protected.

You can sue the doctor for malpraxis. You cannot sue MS because , funny, they have an army of lawyers and their product does not claim to be usable and secure.

> Pilots aren’t allowed small fuckups when flying planes.

Pilots are allowed small fuckups all the time when flying planes. I've got around 1500 hours and haven't had a perfect flight yet. When I stop flying, I expect to have 3000 hours and still to not have had a perfect flight.

Aviation is designed around having redundancies, tolerances, and contingencies for these small fuckups.

> Aviation is designed around having redundancies, tolerances, and contingencies for these small fuckups.

Are there lessons for software engineering to be learned from aviation?

SREs I know talk about aviation incidents all the time, spinning it into a “what lessons do we learn?”

- Aviation incidents are investigated and the investigation is typically bent on discovering root causes, not assigning blame. Do the same thing with bugs & security breaches, you’ll learn more.

- Pilots spend time in simulators practicing unusual situations like water landings or engine failure—do the same thing in software, simulate failure scenarios and intrusion events and see how your engineers and technicians handle those events. You develop new training materials and redesign systems based on what you learned in simulations.

- Pilots rely heavily on checklists rather than memory to ensure that all steps have been completed. This translates well to QA, deployment, various tasks people you need to do in production like migrations or whatnot.

- Pilots require good UX to do their job and cannot be overloaded with too many tasks. Sometime an incident will be traced to dangerous things that don’t seem dangerous when you look at the UI (like thrust reversers), or traced to systems that assigned too many duties to pilots (who get overloaded). For example, if you have a button that deletes a database, it should require a serious confirmation step, like a molly guard on a plane.

- Tasks must be explicitly delegated to individual people (which is what happens on an airplane). For example, in the Gimli Glider incident, the flight engineer role was eliminated and the engineer's responsibilities were not explicitly delegated to the pilot or copilot, but assigned to “both”. Same should be true for tasks in software engineering and incident response.

We also talk about how medicine seems uninterested in learning these lessons, for various reasons.

I think so. I’ve enshrined an adaptation of FAR 91.3 into our technical operations handbook (which is incorporated into our SOX compliance policies). The intention was to make sure that a specific group is clearly, unambiguously, and completely in charge. They are the final authority as to the production operation. (We call this group “Problem Management”, though in the ITIL framework this is more akin to critical incident management)

“Problem Management is exercising emergency authority here” clears away any ambiguity and the confidence this gives them means they are free to act to resolve an incident rather than wonder who they need to call to give the blessing.

https://www.law.cornell.edu/cfr/text/14/91.3

When onboarding new members to that group, we go over this, including the origin story as part of helping them understand how (and that) we expect them to use this.

In other areas? Probably. Aviation tries not to bet lives on a single piece of equipment working perfectly. Where that’s unavoidable (like the Jesus nut and pin on a helicopter), inspection and maintenance procedures are more stringent.

For software reliability, many of them. A large number are mainstream. If you take a distributed systems book, you will see many, and test suites are an evolution of the aviation checklists.

For security, I don't think there are any. Aviation isn't great on security anyway.

Doctors don't get blamed when someone they treat subsequently gets shot. Civil engineers don't get blamed when a saboteur blows up the bridge they designed.

This is not a good analogy. Civil engineers would be blamed if a force they were supposed to compensate for, like a storm, destroys a bridge they designed.

The grey area is if attackers built a weather machine to create exactly the perfect storm tailored to stress a small design defect until it fails. Even more so when the probability of such a storm occurring naturally is 0.

The thing is, our current understanding is that such a weather machine doesn't exist so it isn't necessary to design around it.

In the cyber security world such machines absolutely do exist so we should be expected to design around them.

That's a great analogy.

Have you ever driven under the base of a large bridge (you can’t it’s restricted)? I’m sure you’ve seen the road spikes and the anti car bomb barricades put around high profile buildings… when something is valuable and a possible target of attack you protect it and make it less vulnerable.

> The problem is that companies don’t want to invest the time in good security because it’s less profitable. Or, they are old and decrepit. If the company runs critical infrastructure, then peoples’ safety is at risk.

Most of the time, it's that a non-technical decision maker decides to take a badly mis-calculated risk. "Bob in sales can't access the files he needs." "IT give him admin rights! Now! no... I don't even understand or care to understand that access control mumbo-jumbo you are talking about." A week later, "Hey, you know we just got bitlockered. Bob's laptop got virused, and because we gave him admin rights his machine encrypted all the things AND the backup server too."

> Engineers aren’t allowed “one small fuckup” when building bridges and buildings. Doctors aren’t allowed “one small fuckup” in the OR or when calculating dosages. These mistakes are grave and should be costly. Pilots aren’t allowed small fuckups when flying planes.

That's wrong.

Engineers aren't allowed to fail. They can have as many "fuckups" along the way, as long as they manage to compensate for it and maintain safety it's not a failure. Doctors... well, this profession has a group mentality of shielding one another from publicly acknowledging and documenting mistakes.

Truth is ransomware it utterly useless against a simple back-up and recovery scenario. Any competent engineers can and will deliver one if paid to. It's that companies don't see the point and don't care. As long as the insurance premium is less than what it will cost to implement such a thing they will keep getting ransomed.

Engineers design for resilience to physical failures, whose types are known and can be estimated, predicted, and modeled. Cybersecurity has to deal with human attackers, which cannot be modeled.

>>Cybersecurity has to deal with human attackers, which cannot be modeled.

It absolutely can, it is called Zero Trust, as has been a cyber security model for many many years now

The problem is most organizations have their entire business workflow around the Ring Fence model, and organizations is extremely prone to "we have always done it that way" when it comes to business processes so they expect IT to buy some software, or some widget to bolt on to the ring fence that will protect them

Businesses need to completely rework their internal methodologies, and processes which is something most companies refuse to do

Human behavior is absolutely modelable - usefully, even ("all models are wrong but some are useful") - and people in security do it all the time.

Moreover, engineers also design for resilience to human "attackers" in physical systems all the time.

If anything, cybersecurity is easier because you can leverage the work of tens of thousands of other people in your tooling. You wanna re-write ssh or fail2ban yourself?

We model human attackers all the time, it's probably one of the core components of the role.

> Doctors aren’t allowed “one small fuckup” in the OR or when calculating dosages.

Yes they are. You should talk to an actual doctor before making such claims.

Bad anology because the system is broken all the way down. If what you are saying is true a bridge engineer wouldn’t even be able to build a bridge because their are no materials in the world to build a safe bridge out of.

As someone in the healthcare vendor side of things.... Doctors make a lot of mistakes. There is plenty of software, processes, etc around trying to catch and mitigate those mistakes.

Engineers, doctors, and pilots are not dealing with an equally clever human on the other end trying to beat them. They wouldn't do well if someone were casually changing a number in their calculations or swapping bottles or screwing with the instruments.

Doctors are allowed many small fuckups and some major. Docotors have insurance and medical associations with the best lawyers.

What if I told you that an adversary can place explosives at different parts of the bridge and set them off to see if the bridge goes down? and they can try over and over again until they succeed once, and then you are screwed.

this is the asymmetric nature of the threat. The governments of the west need to align and go after the offenders or sanction countries that give them refuge. It's impossible for a single company to defend itself against adversaries that can keep trying over and over again with impunity.

> Pilots aren’t allowed small f*ups when flying planes.

I don't blame you if you don't take a strangers word on this, but I would strongly encourage you to look at this YouTube channel VASAviation (https://www.youtube.com/user/victor981994) or watch the excellent movie, "Charlie Victor Romeo", which cover in extreme detail the routine, regular, and on-going mistakes that are made in aviation. Mistakes happen in aviation ALL THE TIME; they just don't cause death or crashes in every event.

I can't speak for Engineers and Doctors, but for: pilots, ground crews, flight mechanics, and ATC I have a tiny soapbox. Each of these jobs routinely on a daily basis have mistakes that are made. Sometimes the errors are minor. Other times they are major, and aren't discovered for months or years after the fact, which either led to the loss of millions of dollars worth of equipment, or worse.

Sure, CEOs and such are usually held responsible for those mistakes, but the aviation community focuses more on the ideals of, "Every rule, written in blood" mentality. Things are permitted, until they aren't. This is precisely because it caused a disaster. Things are safe(r) now because they were significantly less safe just a few years ago. The aviation community is focused on preventative actions as a result of the ongoing looming threat of failures. There is no way to avoid them entirely. There is an understanding that when you are in the air, there is NEVER certainty. You are NEVER safe. You should constantly be concerned about what can (and will) eventually go wrong. It's only a matter of how bad the failure will be.

TLDR

Aviation isn't safe because they "don't allow mistakes" but because they have a rigid and foundational system around how to deal with failure.

"Engineers aren’t allowed “one small fuckup” when building bridges and buildings. Doctors aren’t allowed “one small fuckup” in the OR or when calculating dosages. These mistakes are grave and should be costly. Pilots aren’t allowed small fuckups when flying planes."

What a beautiful dream.

I’m not saying it doesn't happen. I’m saying when it does there are grave consequences. Malpractice insurance is a thing and a similar analog to ransomeware insurance.

>Engineers aren’t allowed “one small fuckup” when building bridges and buildings

Yes they are. When estimating the forces you need to handle you can round up. This means that if you made a small mistake it like won't matter since you planned to handle much more than necessary.

>in the OR or when calculating dosages

Dosages don't need to me exact. There are some tolerances and it's easy to imagine a system to eliminate most of human error from this process.

>Pilots aren’t allowed small fuckups when flying planes.

Sure they are. A mistake doesn't mean your plane will crash or fall out of the sky. Planes can already fly themselves.

>Why should we tolerate small fuckups and negligence that only serves a company's shareholders’ interests?

Because hindsight is 20/20.

“Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands.” - Unknown

> Engineers aren’t allowed “one small fuckup” when building bridges and buildings. Doctors aren’t allowed “one small fuckup” in the OR or when calculating dosages.

But engineers aren’t held responsible if a malicious actor drivers a bulldozer into one of the bridge’s columns and brings it down. Same with a doctor. If someone is maliciously messing with the meds, then they are not responsible for over or under dosing.

With these security incidences, we are dealing with malicious actors trying to being down the system. I don’t think you can use analogies from civil engineering or medicine for that.

>Engineers aren’t allowed “one small fuckup” when building bridges and buildings. Doctors aren’t allowed “one small fuckup” in the OR

These are really poor analogies. A better analogy would be engineers who then had to protect those bridges from attack via a number of different modes, including drone and cruise missle strikes; and seemingly legitimate travelers on the bridge who are actually malicious.

Likewise, a better analogy would be doctors who had to contend with nurses who are bad actors, substituting saline for needed drugs or deliberately administering higher than prescribed dosages, etc.

> Engineers aren’t allowed “one small fuckup” when building bridges and buildings. Doctors aren’t allowed “one small fuckup” in the OR or when calculating dosages. These mistakes are grave and should be costly. Pilots aren’t allowed small fuckups when flying planes.

That's BS and you know it. Any system that relies on humans being perfect fails rather quickly.

While major incompetence is generally punished, all of those systems you describe have ways built in to deal with routine human failures.

You wrote a long diatribe I admit I didnt read but the first paragraph, but just to say this: this is not profitable because competitors dont do it. If competitors have to do it because no insurance, suddenly the cycle starts and everyone starts doing it.

It s a risk assessment, as we pile crap on top of crap in deeper and deeper systems, security holes grow more numerous to a point it will be profitable to cover them better than the competition.

A bridge is O(10^6) less complex than a F500 IT setup.

I think your first paragraph is bogus for two reasons, which I’ll leave at the end[1] because I’d rather respond to you main point than the silly reactionary thing you wrote at the top.

I feel like you aren’t really responding to the claim that having good security is extremely hard and expensive and that it seems bad for every company to have to spend a lot of money on a problem which is likely far from their core competency.

The thing you’re actually asking for is for a lot of businesses to go bankrupt trying to be perfect at something that is not their traditional domain and then you make some claim that they have the audacity to try to create value for their shareholders instead? Isn’t it outrageous that companies might try to do literally the thing companies are meant to do instead of being experts in computer security.

I suppose you would counter that it is easy or cheap or imperative for them to be secure but, in the world where hacking is relatively easy, cheap, and profitable ($1-10mm sounds like a lot to a person but not to a country or reasonably sized business) and there are myriad ways to make trivial-seeming configuration errors in software that leave you open to attack, I think it is neither easy nor cheap. And frankly I think it is not imperative for businesses either because their purpose is to make money and while cybersecurity is an important risk to worry about, they can only invest in it so long as they are making money from the thing they are actually meant to be good at.

[1] regarding the first graf:

1. Mostly small errors by engineers or doctors or pilots are… small errors. Bridges won’t collapse if the concrete was mixed slightly too wet, and bigger cracks are usually noticed and mitigated in good time. Doctors get dosages wrong all the time (which is why there are attempts to train nurses to question everything the doctors say) and while malpractice is definitely a thing, there are lots of cases where things go wrong at a small scale and are either fixed or unnoticed. If you ever ask a doctor about this in a social setting they will probably have plenty of examples. And if you get something wrong in a plane you are unlikely to immediately crash. Even landings can be aborted quite late. Big planes crash very infrequently and I think it is silly to therefore assume that the pilots are generally infallible. So I claim that your direct statement that these are not allowed is false.

2. I also claim the analogy is bad because the examples you give are not adversarial. The patients are generally not trying to trick their doctor into making the wrong prescription and Athena geography isn’t going to shift and change to try to exploit weaknesses in the bridge built across it. The pilot flies high above the ground with separation from other planes and various safety systems. Even in an unlikely event like an engine stall, there are possibilities to recover. In computer security the threats are fast and numerous so any small error can be quickly magnified.

Yet doctors have insurance

Which generally covers civil lawsuits up to the point where it’s determined that the doctor was grossly negligent. Then they aren't allowed to practice anymore.

You just jumped from "small fuckup" to "grossly negligent", come on now.