Hacker News Reader

As an engineer I found if I didn't lie and pretend to support the false reality of the managers above me, I wouldn't have that job long.

Once we had a very large effort, creating a new hardware and software platform for the companies products. The CEO demanded it be done in 9 months, our estimate was accurate almost to the day-- 18 months. Politically it was easier for the PMs to agree with 9 months and then have 9 one-month delays it took, instead of getting fired for insisting on a good faith estimate.

What do you think the security in that product was like?

Security directly conflicts with the business model of cutting every corner possible and loading engineers as much as possible. A real security program would have to start with reasonable hours and goals.

I got curious about what legal authority the NTSB has. Here is what I found:

>like a cop, the NTSB can secure an accident scene and keep others away. It can examine the aircraft wreckage. It can have aircraft parts tested to help determine why the accident happened. It can even subpoena evidence. . . . If there is a lawsuit concerning the crash, the NTSB will not get involved. Not only is the NTSB’s report of the accident’s probable cause inadmissible, but the NTSB investigators are prohibited by law from testifying in court, even if they are served with a subpoena.

Source: https://www.aviationlawmonitor.com/2009/04/ntsb/the-ntsbs-li...

>The NTSB may issue a subpoena, enforceable in Federal District Court . . . For purposes of the Health Insurance Portability and Accountability Act of 1996 (HIPAA), Public Law 104-191, and the regulations promulgated by the Department of Health and Human Services, 45 CFR 164.501 et seq., the NTSB is a “public health authority” to which protected health information may be disclosed by a HIPAA “covered entity” without the prior written authorization of the subject of the records.

Source: https://www.law.cornell.edu/cfr/text/49/831.59

I care about the issues (as an angineer), but my experience with raising any security issues is that it results in a lot of pain for me personally.

You report something to the ‘security’ team, and suddenly you’re responsible for doing all the work, as well as the prime suspect in an investigation ‘why didn’t you fix this before, since when did we know about this?!’

I’m absolutely incentiviced to just let any issue lie until it is discovered, because then it actually is the security teams problem.

I was expelled from a university for reporting security flaws, the solution we all seek is the simplest one and for administrators it's easiest to hurt the people making noise which commonly results in the noise going away. fight-or-flight response at it's finest.

Edit: reminder that for the "common folk" these "security issues" are not a 5-minute fix, they are fundamentally different realities which require every machine on the network to be re-checked before they can be used again. There is a clear communication failure between the ones who want security and the ones who want "security".

Would like to hear the story behind this one.

It’s common, a friend almost got expelled for reporting a flaw in the universities ID card system. That friend did not brake anything, they did not sneak into any protected spaces. Just discovered and validated a flaw and then reported it.

A high school friend (2 decades ago) told a teacher that the system keeping track of student grades was vulnerable to attack. The teacher asked the student to demonstrate by attacking the system and adjusting one of his test scores down by 1 point. My friend obliged, and the teacher reported the vulnerability to the administration. An administrator threatened my friend with expulsion, but when he proposed to go public with his story in response, they decided they wouldn't expel him. The resolution was "please don't tell anyone", and the vulnerability was never fixed.

Which university? People should know so that they can avoid it.

Once in a past life I discovered that the asset tracking system I was maintaining had no actual security on any of the data it returned. Once you'd logged in as one user you could just change the URL to access whatever user's data you wanted.

I raised the issue with my manager, we had a meeting about security, I explained that I wasn't a security expert but that this was a big problem and we needed to do a full security audit. I guesstimated 3 months to go through the codebase looking for security holes (in hindsight, a huge underestimate) and they said it was too expensive. Talking to fellow ex-employees the issue was still there 10+ years later.

Not to scare you, but I know a power plant operator who has that issue on its plant operations system. It's pretty amazing.

I know another company where anyone in the company, anyone at all with a login into account in the Active Directory SSO of any privilege at all, even a janitor, can sign any code they want with the release keys.

Nobody cares.

In a way this is similar security to what you get at a single office with no internal security. For the vast majority of history it worked.

The wonders of a high trust environment. The vast vast majority of people will not exploit security holes.

Jup.

To summarize, engineers mostly don't live security not because they don't care, on the contrary, but because doing so is the fastest way to sabotage yourself in the workplace and make enemies.

Two of my own examples

1) A coworker had the task of updating aged signal generators, because we had reported a bug which was then fixed. The signal generators worked without memory, a USB Stick was used as storage memory. As he plugged said USB stick into the work laptop, which he had never touched before, a virus warning popped up. We decided to do the right thing and tell our IT department about it and ask for further steps.. What followed were ridiculous hostile accusations, threats and pressure by the IT people, instead of a sensible reaction.

2) We had a safety instruction at work and it was pointed out anyone should follow safety rules and remind colleagues to spread awareness. Few days later I entered through a security door and two unknown colleagues followed right behind me. As I noticed, I politely asked them to show their access credentials. I was met with hostility and disgust. Later I learned those two were higher ranged managers, anytime I met one of them later, they remembered and treated me badly, for doing exactly what was asked.

I even went so far and told my supervisor of both instances to "raise awareness" of the reality, but only got shrugs and a blank face.

I think you are overestimating the importance of "revenue impacting incidents" to company employees.

If the company makes a couple of million extra or less this year it doesn't effect the majority of workers. Their bonus isn't going up or down etc. And remember this incident has already happened.

By contrast if a report comes out blaming the loss on a worker, department or division then that could have major consequences. No matter how "blameless" it is, come next round of bonuses, promotions or layoffs everybody knows it'll be factored into the decisions.

So people don't have an incentive to make themselves look bad and unlike with the NTSB there is no legal powers or fear of causing deaths behind the investigation.

I understand, but it sounds like we are digging ourselves into the same hole as USSR workers who were not incentivized to deliver working products, when we do that. It’s a civilizational peril. How do we solve cooperation at large scale? Is the only way to watch large companies accumulate bored employees and constantly recreate “the small guy”, the startup, which will finally make things right, until they become too big to be incentivized?

That's exactly how it works, and why big companies do not grow until they consume the entire world. They become complacent and unable to change, and a startup eventually takes their business away.

It's funny that people expect management to function better because it's employee-owners of a capitalist corporation, instead of feudal lords or members of a communist central committee.

The feudal owners in a startup situation are the marketplace owners who organize all startups to compete.

The day we lose is the day all of those marketplaces themselves are cannot be replaced.

i.e. VC's and other providers of capital.

Well they would except often startups are prohibited from competing via regulatory capture or excessive funding costs

That comparison is apt. A big corporate entity is a mini command economy, and fails in similar ways.

There is an inflection point where the focus of the organization shifts from mission focus to control focus. Usually it’s when growth slows. A fast-mover industry like tech is a great example, but it happens almost everywhere.

> Is the only way to watch large companies accumulate bored employees and constantly recreate “the small guy”, the startup, which will finally make things right, until they become too big to be incentivized?

Yes, and you have just illustrated why the USSR struggled economically in comparison to the United States.

I didn't say "employees" so much as "executives"; and the executives I'm referring to go beyond owning P&L. They actually do care about revenue, which is why everyone lies to them.

> Do you know what executives do care about? Revenue. Lost deals.

This is often why engineering needs aren't covered. Things are presented as risks and expense, instead of in terms of revenue. A $10,000 expense actually wipes out $10,000 in net profit, so you need to generate revenue sufficient to create $10,000 in net profit. Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it. Finally, there is how risk frames what you present. If you come at me with "this might happen" the other side of the coin is "this might not happen", and most managers will avoid the certainty of expense for the possibility of an expense.

If you are working with a CEO, valuation is where it's at. Try to understand the swing in company valuation based on profit. Present to the CEO like this: "X is highly likely to happen within Y months, and it will impact $Z in net profits, leading to a change in company valuation of up to $N." Make sure $N is enough to matter. You just took 98% of the arguments against taking action off the table.

> Most companies have really rosy gross margins, but really tight net margins, so a $100K expense will take $2.8 Million in revenue to offset it.

I don’t think you are interpreting things correctly here. This is both counterintuitive and wrong.

I'm pretty sure I'm not wrong. Source: I own three companies. The $100k and $2.8M were just examples (very thin net profit). It's equally likely you could have $100k in expense and need $200-$400K in sales to offset it if you are in a very high margin business. Regardless, what matters is that the unexpected expense will reduce profit, every time, and by framing risk in terms of profit you will help your CEO make better decisions.

You’re not “framing the risk in terms of profit.” You’re talking about a set of equations in your head, that only you know, that come up with weird results like “it takes way more revenue to ‘offset’ an expense” because you’re holding “net” or “gross” or whatever margin constant. Which is really confusing and surprising and not how anyone I’ve ever met talks about this stuff, and they’d be equally puzzled and I don’t think would experience an illuminating or aha moment or whatever.

Personally I found it a little interesting, I get what you are going for, but it’s so strange and not really useful or true.

I think you are looking for deep insight and the point that was being made was shallow. If I make $3000 in profit, at 30% margin, I have to have $10000 in sales. It follows that if I have an unexpected expense of $300, it will require $1000 in sales to have generate enough profit to cover the expense.

> X is highly likely to happen within Y months

How do you quantify "highly"?

I think that one way or another, most people actually try to evaluate things this way.

If they knew with 100% certainty that some breach would happen, they would probably invest the time and money to fix the situation. Just as if the probability for the breach to happen was known to be 0%, they would be justified in not fixing it.

You frame this as CEOs being pragmatic, and I agree. But then, the other side of the coin is that if you're regularly wrong (there's a high risk of a breach happening — then nothing happens) they'll probably stop listening to you after a few times.

Also don't forget the reverse-tinkerbell effect: if you do things right and nothing bad happens for a while, they may be tempted to consider that perhaps you're being overly cautious and perhaps a doing things a little bit faster/cheaper would be ok anyway.

And depending on the cost of something bad happening, they may be right!

Well, the idea that fools consistently occupy exec positions is a bit foolish, isn't?

Given 90% of execs spend their entire lives doing nepotism and politics, I don't think it's as foolish as you frame it.

people can do and be two things at the same time

I also wonder if "blameless postmortem" culture perhaps actively works against preventing these kind of incidents. It doesn't seem that anyone in IT is ever responsible for damage they cause.

But yes, lying, "not seeing" and covering documentation is pretty much standard corporate behaviour I've seen around plenty of companies as well.

I no longer believe in blameless post mortem as a general rule. I have, through experience, come to believe that the contexts where blameless post mortems work are the contexts where literally anything works because they are organizations that have high hiring bars and high expectations. My current employer is not one of them; we are a mountain of mediocrity and all blameless post mortems do is act as an excuse to avoid raising the bar.

The principle of blameless postmortems is not supposed to absolve anyone of the responsibility to change anything, it’s supposed to foreground that serious failures are organizational failures first and foremost, because it’s the organization that has an obligation not to fail, not individuals, who fail all the time as a rule.

Exactly this.

I spend a bunch of time reading accident reports from agencies like RAIB and MAIB, but my real jobs have been closer to the Web PKI and thus m.d.s.policy

Back in 2015, Symantec's CA issued some certificates that shouldn't have existed, including for names owned by Google. What's wrong there? Well, a blameless postmortem would probably tell you that your processes and procedures are bad, you are creating bogus certificates to "test" a real CA whose certificates are actually trusted in the real world. Need better processes, training, oversight to ensure things improve. What did Symantec do when they were caught? They fired the low-level employees who conducted the tests and wrote a blog post "A Tough Day as Leaders" which blamed the fired employees for getting it wrong. Some leadership (the blog post of course no longer exists although I assume it's archived somewhere).

Less than two years later, Symantec is back in trouble again because an RA they've worked with has been issuing certificates, using their CA infrastructure (and thus, from our point of view they were issuing these certificates even if they unaccountably believed this isn't their fault) that should not exist. This time Symantec's bosses blame not only low-level employees, but also auditors, bosses at the Korean RA, and anybody else they can think of... except themselves.

This is a gross failure of leadership. Once upon a time a US President said "The buck stops here", but Donald Trump was very clear, "The buck stops with everybody" and "I take no responsibility" and it seems Symantec's leadership were made in that image. They quit the CA business rather than do what it would take to fix the problem.

If you conduct a "postmortem" after an incident, then "Nobody was to blame and nothing needs to change" is almost certainly just as much the wrong outcome as "It's Jane's fault, fire her".

I mentioned I read MAIB reports. One MAIB report sticks out, after many years, in the following way:

Unlike every other MAIB report I've read, this one has No recommendations. Someone died, and yet there is nothing to recommend. Why not? Well, the cause is very simple, two men on a fishing boat took a lot of heroin, and their boat crashed, it sank and one died. No need to recommend that you shouldn't take heroin while operating a fishing boat since heroin is an illegal drug already and operating a vessel under the influence of drugs or alcohol is already a crime too.

If your next "blameless portmortem" doesn't have any recommendations, ask yourself, was what happened already a crime? Are the people involved dead or in prison and so either way beyond the value of recommending a different course of action next time? No? Then we need to recommend how to actually avoid it happening again.

> They fired the low-level employees who conducted the tests and wrote a blog post "A Tough Day as Leaders" which blamed the fired employees for getting it wrong. Some leadership (the blog post of course no longer exists although I assume it's archived somewhere).

"A Tough Day as Leaders"

https://archive.is/Ro70U

And the MAIB report: https://assets.publishing.service.gov.uk/media/54db3fd940f0b...

>If your next "blameless portmortem" doesn't have any recommendations, ask yourself, was what happened [in IT services] already a crime?

Imagine the amount of training needed for a government worker to be able to decide that and be familiar with law AND not have a political/financial lean to be able to do that. The US government has already decided they're not going to train their population with relevant marketable skills globally at competitive ages. They've turned education into a private/public dating program to preserve social class norms.

The US government is still too ill equipped, corrupt, and disinterested in regional/global IT regulations. They seem more interested in weaponizing the IT realm to remain relevant across the globe. It seems to me like they're consistently deluded in thinking they can maintain systemic supremacy given movements in india, china, russia, and europe.

> and all blameless post mortems do is act as an excuse to avoid raising the bar

"Well, there's your problem, right there."

The entire point of doing blameless post-mortems is to correctly identify problems for resolution. If management doesn't drive changes in response (process, training, communication, whatever), you have a different problem to solve before they'll do any good.

I imagine blameless postmortems sometimes happen because people want to avoid blame. So they argue that all the cool places do "blameless". We should try that too!

And then the organization doesn't understand the actual concept behind the idea (nor did the suggestor want that). Instead the organization learns "we have decided not to blame anyone". And then everyone involved is satisfied.

A post-mortem should not necessarily blame the individual, but blame the circumstances the individual finds themselves in.

Yes, a hard-coded password is bad practice. But does the company have a bad culture of keeping configs in repos? Maybe management thinks it easier to commit configs with sensitive data, than to set up proper deployment shit. And after all, the repos are private, so it should be fine yeah?

Bad code ending up in production is something you'll see often. Does the company have nice test suites for everything? Continuous integration pipelines? E2E tests? Or is upper management pushing everyone to their limits, because "fuck it ship it"?

A post mortem should absolutely also assign individual responsibility for grossly negligent behaviour.

If management thinks that skipping testing and implementing insecure controls is the way to go, get that in writing.

Collectively, developers need to show a higher degree of professionalism in this regard.

In my negative experiences, "blameless" turned in to "nobody did anything wrong" which, of course, undermines the whole point of finding out what actually happened so we can see if there is a thing we can do to reduce the likelihood of it happening again.

Sometimes, the root cause is indeed someone with the privilege but not the good sense ignoring warning signs. If we can't identify that problem, then we can't improve our odds for the next time.

A valid blameless answer then is "remove the privilege" and yes, despite whatever objections you'll raise, this is possible. Difficult, but possible.

Like, even in the case of the extreme example of someone deciding to intentionally harm the company. You fire them, but then what, how do you prevent the next person to go rogue from causing equivalent harm?

This is what slows big companies down. They accrete so many of these rules and restrictions that people can't do their work.

I believe more of these incidents should conclude: "this is better to accept as the cost of doing business than to try to spend money to fix".

Or people start ignoring the rules in order to do their work. Which has it's own problems (particularly as rules normally aren't divided into "important" and "unimportant".

This is the thorn in my foot.

I refuse to ignore company rules, and also comment on gross negligence, which more often then not means I am the quarreler, not the good engineer in the eyes of coworkers and bosses.

> They accrete so many of these rules and restrictions that people can't do their work.

I hear this from software engineers, from time to time. What do you mean they aren't allowed to SSH into prod anymore? How will they debug, update, or maintain anything?

Sometimes this is your standard-issue hostile reaction to change. The old approach is what they are used to and they don't understand the need to change it (and "ask to understand" mostly so they can try and negotiate). This new world just seems to get in the way for no clear reason. Management neither appreciates nor understands the reluctance and just pushes on.

Usually what needs to happen is a series of changes across the org. You roll out the change with references to policy to support it. Workflows get updated and reworked so that SSHing into prod is not, in fact, the way to update systems or view logs or whatever.

Most of all, educational materials are provided. Often I find that people object to changes because they don't know another way to work. If all you've ever known is SSHing into prod to read logs, you've probably never heard of Kibana or used OpenTelemetry.

Public policy is just waking for this, and private administration is way too unprofessional as a group to formalize the problem (thus some people understand it, most don't), but rule creation requires a cost-benefit analysis.

Any new rule you create is a chance to analyze the entire set, and maybe redesign it.

That kind of result may mean you need a Molly guard. The original Molly was a toddler who reportedly pushed the Big Red Switch on an IBM 4341 twice in one day and so they put a cover over the switch to put an end to that sort of outage. Occasionally people need firing but even in that sort of circumstance there's an organization problem that needs addressing.

Let's not conflate security and safety here. The boards hailed by the article are all about investigating safety failures, and so is the advocated invetigation board. Security is a different beast. It's sub-par in many airplanes/power plants/... too.

Do bureacracies essentially provide complexity in which dysfunction readily hides?

That's very disappointing. I've always brought up everyone up to think that of all the professions, you can always trust an engineer. They deal in facts. They don't lie. Perhaps it's as simple that in the real world, engineers can't lie. Because the world wouldn't work is they did.

> There are no penalties, but really, for things as egregious as hard coded passwords

Is this sentence missing a word or phrase?