Comment by duxup | Hacker News Reader

Comment by duxup | Hacker News Reader

One of the fist things my boss told me at my first "real" (good career type money) job was:

"Everyone screws up, you will too, just be honest about it and tell me and it will be good."

It was a job working with this memory mapped wonky hardware tied to mainframes. There was no "undo" and as soon as you wrote to memory there it was. It was inevitable that you would typo something sometime in an important system.

Finally 3 years later a buddy is talking to me over the cube wall "Hey was that #3 you were working on yesterday?" I of course am typing away while talking and say out loud "3? Um..."

So I type something like -RESET SYSTEM 3-

I meant to type 9 a non critical system. 3 tied to a data replication system that was absolutely critical that it be running otherwise all transactions would stop (well for a bit while backups take over).

So if you couldn't use an ATM for a huge bank for a little while decades ago (fortunately it was the middle of the night), that was me ;)

I went to my boss the next morning and told him what I did, and he says "This like your first in 3 years, that's a record or something, it's usually a few within like 6 months. Nice job!"

It was a great place to work, no finger pointing, if you screwed up no big deal, everyone stuck around working with that team for decades.

When there were conference calls it was rarely stated (if ever) who actually did the thing. It was just accepted that it happened and we could discuss how to prevent it and such. "The engineer" or "the support team" and such were common phrases.

Inevitably folks would ask "who was it" and the answer usually was something like "it doesn't matter".

I’m at a place with a similar policy: you can screw up, just own it as soon as you realize you screwed up, do your best to mitigate the damage ASAP, and see that the fix gets prioritized appropriately.

I still screw up, of course, but this attitude has made me feel less paranoid about performance and less like an idiot when I do mess up. The amount of trust a team needs to have to make this work is really the key,IMO.

This approach has been adopted by the SRE movement under the "blameless postmortem culture" alias: https://landing.google.com/sre/sre-book/chapters/postmortem-...

That's a good read.

Blame is such a complicated thing and the results of any mess are always way complicated even without the human factor.

I'm not convinced in any complex system outside of a guy running through a datacenter with a hammer... that anyone can actually "properly" assign blame or fault.

Humans are going to dork up, but no point in coming down on someone(s) for all the complexity that leads to a problem..... helping everyone avoid it on the other hand has a lot of benefit.

How do they convince people not to use failures as political capital against other teams? It's all well and good to say "We promise we won't do it!", it's quite another to actually not call out the other team's failures when they're competing with you for budget/headcount, or against other individuals when stack-ranking.

Ideally by not creating systems where people within the same organization view each other as (more than friendly) competitors, and by weeding out the sociopaths who can’t help but view life through that lens.

You do it by management not making things zero sum, and by correcting/penalizing people for blaming behavior (e.g. saying a root cause is that a team/person made a mistake, vs a system not preventing or mitigating the mistake)

One of the first things I tell new graduates/analysts/data scientists is that you're not a real one until you've broken something important & valuable.

It's the breaking of things and seeing it go wrong in the real world that often boosts one above the level of academic coding and model building, because "holy shit, all that engineering and principals stuff actually matters and the real world is hard and there's consequences".

But I expect everyone to do it also, because without curiosity and an urge to push the boundaries, you'll be a very mediocre data scientist :p

how did you like working on mainframes?

It was actually equipment connected TO mainframes. It was a weird world, very old, with old school people / processes.

It wasn't where I wanted to be, but it was also my first real job for a guy who dropped out of college (I was a poor student / shouldn't have gone when I was that age)... so I didn't mind at all.

But it was weird, lots of old east coast customers with old ways. One of our supersmart engineers was a woman. She would be on a call and would tell them their mainframe was configured wrong and straight up some guys would ask for her manager every time.... every time.

So she would come get me, a lowly n00b with a male voice and I'd pretend to be her manager looking things over and tell them the same thing, word for word, and they'll belive me. Weird world.

For whatsoever little it's worth, I have fond memories of my mainframe days.

One of the wonderful things about them is that they never change. You can sit down and, in a period of weeks or months, read all the black binders containing all of the (usually excellent) documentation, and then you'd know everything about it, forever. Close enough to everything, anyway.

Some people might find this horribly boring, but it meant I could easily partition my time between "the familiar thing that always works the way I expect", and "the experimental stuff on the side". Today, those lines are super blurred; I rarely get the opportunity to become a proper expert on anything at all in the web world, so 100% of my time is "omg something's broken and quick, find the quirky part in the experimental thing that's running in production."

And, speaking of documentation, we just don't have anything like mainframe documentation in modern software. O'Reilly books often come sort of close, or used to, but Unisys for instance had something like a 30-volume set of binders, about 500 or so pages each, containing extremely deep, carefully edited documentation on every single system call. Imagine if your favorite web framework had an entire wikipedia, with a big team dedicated just to testing and reviewing every page for accuracy.

And finally, with modern queues, databases, file systems, and -- much as I'm loathe to admit it -- systemd, we're finally just now catching up to the disaster recovery procedures that were standard for mainframes in 1995. You could literally walk into the data center and yank the power cord for the mainframe while it was in the middle of running payroll, plug it back in, and it would pick up where it left off without skipping or double-counting a single record.

They had gigantic limitations too. TCO was astronomical by modern standards, so the hardware never got upgraded, so you'd never be able to do any of today's big data stuff on a mainframe. Software development, such as it was, was done in COBOL or JCL or WFL or, maybe, Algol or FORTRAN, and git isn't a thing and a lot of that software has decades of history behind every single line of code.

But it wasn't all bad.

Reply on news.ycombinator.com