Hacker News Reader

I have a similar experience from waaaaay back (now near 30 years). We had a water leak in the basement of the institute of physics. Nothing terribly major, ran for a few hours shorted power downstairs eventually (that's how it was found). Once fixed, pumped out, all that remained was a musty smell for a few days.

Except four days later network connection failed in the building. Tracing the cabling eventually lead us to the basement; some under-stairs / slightly below floor level comms cabinet, empty other than the building's outbound router. 48V Telco, no mains connect, had sat under"water" (well some rather murky stuff) for almost a week and done well. Once we got it washed (with clear water) and dried, it continued fine (till replacement a few weeks after). We were a little amazed, to be honest, physicists or no. Brave device, that.

I wonder although wasteful, isn't this better than accelerated failures taking the defense radar offline at random times in the future?

Reminds me of a friend of mine who somehow blew a shift in his car and did something bad to his engine. Instead of pulling over and stopping, he just drove it home and ended up destroying the engine.

So he got a whole new engine instead of having insurance pay to dismantle the engine and replace a few valves. It might even have been a similar price, but a much more robust solution.

But the action that has actually been taken just guaranteed the prompt destruction of the radar equipment, and a prolonged outage to replace it.

Unless an attack was underway that the radar was tracking, the right course of action was to power it off immediately, and start cleaning it. After the cleaning, it would go online promptly, and then parts of it could be assessed for damage, taken offline when the replacement is ready, and replaced, likely without even stopping the operation.

This kind of paralysis and fear of making decisions, especially when time is of essence, would be exactly what I'd like to instill in my enemies.

It would explain a surprising amount of events at least here in Sweden if BCG and McKinsey were part of a conspiracy to weaken the crisis preparedness of states worldwide.

If your friend did a money shift the damage happened at the shift moment and driving home didn't change the outcome.

I reread the comment a few times and I think he agrees with you.

His friend got the better outcome by not stopping immediately is what he was implying.

It's quite impressive it kept working for that long, and even more amazing someone was capable of saying "no, let's not cut the power, there might be a problem if we do"

Honestly, that doesn’t surprise me at all. This seems perfectly in line with “I don’t care what it is, or how well I perform it, as long as I have a job.”

Few things in my career have granted me more power faster than taking ownership of decisions.

Some people are so terrified of having to say "I was wrong" that they either will do nothing, or just happily follow decisions that someone else makes. It's one thing when that's a line employee behaving that way, but there are managers so terrified of making a mistake that all they really are is scribes, tracking who said what and who decided what. These weasels don't lead. They don't contribute anything. They just have a quick answer when someone above them asks how this fuckup happened. Well that was Jameson. He decided to do X so clearly it's his fault things broke.

I was in a room once with a guy who was so caught up in the idea of catching me in a gotcha that he had no idea what to do when I said, "yep, that's wrong, and this is how we fix it." His face went through the same roller coaster of emotions that happens when you drop your ice cream cone. I got to say most of what I needed to say before he even recovered. I think I was proudest of keeping a straight face during the entire conversation.

Where has intellectual honesty gone that just being transparent verges on a superpower? I feel like I'm taking crazy pills.

Taking ownership of decisions in this way breeds another problem: people love to police when they perceive other people taking authority that's not rightfully theirs.

I had your view until 7 years at Google. Showing this to 80% of people there would garner tsktsking and quips about delusions of grandeur and seeming unbalanced. In performance reviews, you could count on this person saying you collaborate poorly and don't take others concerns into account, because they felt humiliated and retconned a reason why it's your fault. (To be fair, given your vivid understanding of their face at the time, it sounds like you understand they wouldn't feel well after you did this)

In the higher rooms, I saw at least a couple times everyone living this worldview and it ending up in bickering and extreme insistence the other person was missing fundamentals while nothing got done for months: which in the more Machiavellian parlance, is still good if you're good, because most people won't do the work, just argue over the 20% of the problem they understand, and someone has to win eventually

I keep editing this comment because I get the vibe it disagrees with you, but it doesn't, at all. If anything, your framing helps sharpen why it felt like an odd place, in a bad way, to me from day 30.

I've known enough people at Google that I've gotten a decent view into the broken-ass culture there, from the outside looking in, and I'd say that's a pretty decent summary of some of their worst problems.

One of the things that impressed me at my time at Google (2011-2015) was the culture of blameless post-mortems, and of general transparency of many (not all) things.

Sadly, this culture must have deteriorated; my friends who are still at Google say that it's become literally the stuff of Dilbert comic strips :-(

Hah, somewhat perversely, I'm glad to hear that: it's exactly how I sum things up in a sentence in my head, then I think I'm being overdramatic / personalizing it, because I kinda grew up learning about tech via Dilbert, I was a big fan of Dogbert at 7 years old.

But I'm not.

I did Aug 2016-2023. Of course it's a complicated story, and you've heard a lot already I'm sure, but when you mention blameless postmortems, it remind me of my pet peeve: promo committees done within teams (basically 2 levels up, i.e. skip is doling out promo, with no recourse or real outside input) was a horrible, wretched idea.

There was this odd obsession with saving time on perf, but they kinda just needed to tell people to chill and recognize that it really is important for management. With all promos being judged by people next to you, what we might call "being agreeable" became a necessity.

I distinctly remember in 2018 being told if things were how they had been, that I should just do a blameless postmortem, but with things as they are, that was a quick way to end up on the bench for the rest of my career.

Interestingly, I was at Google during I think the very end of its golden years (2011-2016). I loved it and when I heard about how sclerotic it had gotten was pretty surprised.

Looking back, IMO the seeds of its downfall had already been planted by the time I was there, including the culture of blameless postmortems.

There was this idea that any problem—any outage or whatever —reflected not stupidity on the part of the perpetrator (Google hires the smartest people after all) but a process failure. Blameless postmortems avoided blame by ending with a process recommendation.

I’ve worked at startups since, and I do think that creating an environment where people are comfortable taking risks is just a fundamentally hard culture problem, and you don’t even have to be big to have it (though it doesn’t help). I’ve seen 15-person startups where engineers wouldn’t fix something because “I’m just an engineer and this is really a product decision” while the PM says “we need to get the engineers together in a meeting and come to a consensus on this.”

IMO Google’s hack of giving people space to take risks by letting them pass off failure as a process problem did sort of work, to their credit, until they got to where it sounds like they are now, where you can’t do anything without being accused of bypassing “good process,” which means going to endless lengths to get everyone’s opinions and placating everyone who disagrees with you.

It seems like at Meta they solved the problem by regarding failure as good and just. At my current company, to do something risky, you have to write a design doc and have a comment period, and then you’re free to do whatever the doc proposes. Idk what other approaches there are—I’d pay to read a book about it, though.

It seems this is a common problem with larger organizations, that everything turns (corrupts) into a process. I've even seen processes for following processes.

To make the problem worse, nobody dares to remove from a process as that would mean they would be accountable for the consequences of the "missing steps", and at the same time adding to a process is almost always safe - another step in a process to clarify or verify can't really fail and having contributed to the process work in a process driven organization looks fantastic on your yearly review. So we get more and more processes that are perpetually growing in scope.

The long term damage that heavy processes do, what I've seen first hand, is extreme stagnation. Even simple and straight forward improvements are avoided if it means that you need to produce ridiculous documentation that nobody will ever need, or seek approval from a (very busy) group of people who contribute nothing other than checking that you followed the processes.

When I remove steps I brag about it. I don’t know or recall if I ever consciously reasoned that as trying to normalize that activity, but that is certainly what I’m doing.

Shorter processes in clock time matter more than step count, but it’s hard to do one without the other.

For what it’s worth, in most cases, an organization that overly focuses on process is superior to one with a naive individualism that throws employees under the bus by default.

I think I’m pretty good at Five Whys, for two reasons. One, I have enough of a chess player’s mind where I can see at about the third Why that someone is steering us toward a toothless solution and steer us toward a better one.

But the other one is that blameless doesn’t mean we don’t talk about the person who fucked up. The entire point of the NTSB’s version of the process is to get pilots and ground crew to be vulnerable. To explain how they pushed the wrong button or signed off on a maintenance deferment or whatever. I was tired, I got asked tangential questions during a critical moment. I misinterpreted the text, etc.

I’m neurodivergent. I spend way too much of my day analyzing my errors. But the way my brain works on a Tuesday is the way everyone else’s works in the middle of a high stress situation. For me it still feels like a bad Tuesday. The tools and steps I build for keeping me from looking stupid also work for the team, once they give them a chance. And during Five Why’s I just have to say the quiet parts out loud, and I can speculate rather well on whether other people had similar experiences.

I regularly take authority and you’re right about how people can feel, but that’s where your social skills come to play.

Your average person doesn’t even take authority.

The next “level” per se (not that you have to take authority — only if you want to) is the person taking authority.

Then the next level above that is the person that takes authority but understands how it looks to other people, so they sure everyone else understands that they are taking authority to make things move along, not because they are entitled to it. They attack the problem head on.

My goal is to get good engineering done while also making sure everyone feels included and good about being at work.

If you’re about good engineering and respect for other people, you can get things done however you want.

I’ve worked at too many places that only promote you once you’re doing 99% of the job title. It’s a joke at some places. “I thought you already were <new title>”

If you don’t act like a lead dev you’re not going to get much practice at being one. And I’m not talking about the bossing people around part, but the fostering teamwork and taking care of people parts.

If anything people would fault me for saying “no” so much. It’s part of why I tried my hand at security for a bit but found it exhausting. I was never more “the adult in the room” than in that position.

I’m not saying own all the decisions. With a largish team you want at least three people who will stand by a decision even when it’s not a unanimous vote. (Don’t ever do four or six unless you have a good boss. Jesus Christ that was awful.) When I do say something will definitely work, I’m almost always right.

What I left out and thought of later is that it’s more important to own the cleanup than the decision. Once you do that people trust you more, and the trust leads to other opportunities. So it’s not really, “I was wrong” although that already filters out way too many people. It’s the “here let me help you fix it.” That makes it stick.

Whilst I fully agree - you have to remember a line employee has limited incentive to take said risks as there's little to gain and a lot to lose.

I don't see how that manager expected to have a job by deciding not to cut power to submerged equipment, and have it flushed of the brine and cleaned, ASAP. It looks completely idiotic/incompetent from where I'm sitting.

There are plenty people who never changed a lightbulb in their life, and some of them manage other people.

Different industries prioritize different things. I'm speculating but being the military they may consider that the enemy is planning an incursion exactly when some critical detection systems are expected to be down. So they'd rather kill some equipment while they mobilize a contingency measure. No idea what was in the manager's head or if the threat model makes sense but things aren't always super obvious and don't lend themselves to generalization.

Potentially, but if your intelligence is so terrible you have no idea of that attack beforehand, or your radar coverage so sloppy that you can’t take any of them offline without losing that coverage, you probably failed already.

Because that's what companies incentivize for, sadly.

The handbook or SOP somewhere probably does call for a risk assessment.

So they have two choices. Choice A is to do what's obvious, cut power and get to work. That has two branches. One branch is saving the day and potentially millions of dollars. For what? A side mention at the next all hands, maybe. Branch 2 is something goes wrong, and the company says you didn't follow the handbook, and you're fired or worse, found liable.

Choice B is following the handbook, knowing everything will be ruined. But, you keep your job, and maybe the handbook gets updated.

Both choices provide no reward. Only one provides no risk.

Ding ding ding! As an employee you're not there to do good work, you're there to what you're employer asks of you which is sometimes good work. When its your own electronics you can do whatever you want but as a rando employee you're not in a position to accept the risk.

It is only impressive coming from the consumer grade electronics.

Microchips, capacitors, PCB's and other stuff like manufactured for defence have much wider and higher tolerances. One won't find the cutting edge 2 nm processes (depending on the specific area of defence, 200-300 µm would still be cutting edge today for them), materials used are different, plus all sorts of chemical and physical hardening is employed. Most of the time, the chips can't use the consumer grade cutting edge designs and processes as they have to withstand not just abuse but also stronger EM fields (in radar equipment) or EM pulses (e.g. a nuclear explosion).

After electronics have been manufactured, they proceed onto the QA phase where chips are literally baked in ovens for a day or a few, then retrieved and tested. They may get irradiated, soaked for periods of time etc. Most people would call it abuse – from a civilian perspective. The surviving ones are deemed to have passed the QA and are used to make the actual equipment.

> 200-300 µm would still be cutting edge today for them

Sorry, I think you're off by a few orders of magnitude, do you mean 200-300nm?

No, I am not, although 200-300 µm is not a commonplace.

Smaller processes result in chips either becoming very unstable or plainly kicking the bucket when exposed to strong electromagnetic fields and electromagnetic radiation. Radar equipment generates such strong EM radiation that is lethal to humans and to microelectronics no matter how well shielded the chips are. Then there are highly charged particles released during a nuclear explosion, so the electronics have to survive at least for some time at the battlefield.

Chips produced for defence and military use some of the most conservative design and manufacturing processes, and even the analogue processes may still be used somewhere (I have not checked recently though).

The navy used to throw the boards in the cleaner and repairs the failures. Now we have better cleansing solutions, but damn if that isn’t nightmare fuel. I would have pumped and dehumidified the room until parts could be pulled/repaired. These days they just scrap everything. It’s terrible.

I also worked at a very large multi-billion dollar defense radar, once upon a time. Though by your description maybe a different one.