Hacker News Reader

It's sort of like how it's better to be a firefighter than a maintainer, which is guess is the point. The ease may be similar, but coming in to save everyone from an explosion gets you more points than oiling the valve every day so pressure doesn't build up in the first place. It's a cultural issue that is very hard to combat.

Indeed, thank your (figurative and literal) janitors and maintenance staff! :)

Indeed. Sometimes the saviour is the one responsible for the disaster but companies also need their heroes or at least that's how I interpret certain management decisions. Been on both sides of the fence.

God, this hurts. I pride myself in avoiding deadlines by looking to the future and getting things done before they're needed, so when the time comes, I can focus on polish and integration. But then some dick in management notices that I did this thing in a day, doesn't count the prep-work, and hands me an impossible deadline that's already promised to a customer...

In my fantasy world, the developers are the ones who make promises to the users and thats it

Hell, I'd settle for managers simply being held responsible when the impossible deadline they set wasn't met.

Perhaps you can staff your firefighting crew with the effective maintainers. They've been oiling their valves, so they have time to spare helping fight the fire. This becomes doubly visible.

it's better to be a firefighter than a maintainer

Oh this is very true. I've achieved a pretty good reputation in my org getting called in when things have gone wrong somewhere. Not so much for typical day-to-day work.

Probably has to do with our brains preferring instant gratification over long term payoffs.

Reminds me of moments I had early in my career where I was the hero for fixing problems I caused. :) That company didn't have code reviews and my bosses weren't very technical, so it was pretty easy to pretend like I wasn't at fault for anything. Sometimes they'd figure out that I caused the problems and even then it didn't seem to really matter. If I had the skills I have now 7 years later, my bosses would probably think they're overpaying me because it wouldn't seem to them like I'm doing anything more of the time.

If you don't get enough praise for not breaking things, get a days since a workplace accident sign, and update it every day.

Set it to zero when you break something.

Edit to add: you should be able to get some praise (or at least self-praise) at milestones like 30/60/90/180/365 days without an incident.

Recently one of my clients noticed something we had broken on their website. When they noticed it, we fixed it in a matter of minutes. They were so pumped on the response time they then left us a very glowing public review.

Thanks for connecting these dots! You've put words to a situation I've experienced before but never fully grokked

true, the problem is when the same person is both the arsonist and the fire fighter. It becomes a habit for some people to show "heroic" effort. No effort is done to see who caused the issue, the spotlight is usually on who fixed it. These "arsonists" also have higher velocity in delivering features since they don't care about quality, so they get into the good books of upper management.

I remember John Sterman doing some system dynamics modelling on Oil Refinery maintenance vs actual fire fighting[1].

The results are just as you suggest.

Personally speaking I'd like to work at the software equivalent of Toyota. Where the whole organisation is constantly looking for feedback about development processes and how barriers to quality can be removed.

I think the problem is that how to get credit for the not having problems in the first place.

1. http://web.mit.edu/jsterman/www/SDG/maint.pdf

"but coming in to save everyone from an explosion gets you more points than oiling the valve every day so pressure doesn't build up in the first place. It's a cultural issue that is very hard to combat."

The "hero" part of the firefighter comes from the fact, that they actually go towards real danger as part of their job, they often do unpaid and voluntarily. (A big fire out of control, is a very scary thing)

While the maintainance worker is usually not in danger, if he does his job right.

" It's a cultural issue that is very hard to combat."

So I do not really see a issue here to combat (when using this metaphor). The actual firefighters deserve their praise.

And if some companies are too shortsighted and praise the people heroically fixing their own mess, then I would use a different metaphor there, to adress this.

What metaphor would you use though?

Clearly there are companies praising heroic fixes, while not praising too much regular maintenance.

I guess firefighting is the right metaphor, to fixing big errors in live production environments: it takes a different skill and mindset, to do things under pressure, nothing wrong to praise that.

But I think it is wrong, to talk down the job of actual firefighters, because:

"Clearly there are companies praising heroic fixes, while not praising too much regular maintenance"

that is a seperate issue.

the fire metaphor still works: we have arsonists (that build up tech debt or just bad practices) that turn firefighters (by fixing in emergency mode).

Of course not all tech debt is bad, the trick is know when to use it to good effect, not building it up everywhere only to pay it back at the costliest moment. It's all a balancing act.

When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks about bravery?

- Sun Tzu

I will say as someone who has to wrangle a lot of junior devs -- don't have imposter syndrome on this. The hard part is sticking with the issue and seeing it through to the logical conclusion. Many people have difficulty even conceptualizing the steps that need to be taken in order to solve a problem. If you are smart enough to get help from someone in devops and understand the advice they gave, you're doing the same stuff senior devs and CTOs do on a daily basis. We're all frauds lol

computing is a fraud, a CPU is literally a rock that we tricked into thinking

This gave me a great belly laugh

If a CPU is thinking, then all rocks are constantly thinking their rocky thoughts, contemplating the immeasurable eons of their existence. A CPU is, then, merely a rock which we have made to think some thoughts which are to our benefit.

You’re oversimplifying.

We didn’t just trick a rock into thinking - we beat on it until it was really thin, shoved a little bit of lightning inside of it, then tricked it into thinking.

That's pretty much it. There's a certain category of people with a certain attitude towards debugging. You have to be able to break it down into steps - sometimes from "first principles". Then work up and identify where the issue is and how to resolve it. Sometimes it requires looking at logs and simulating production conditions and other times you may have to understand low level network stack.

I still remember the time trying to reproduce and fix a certain issue. We had a 3-node cluster mysql database setup with one main and two replicas. The replicas are read-only. If one replica goes down, reads shouldn't be affected at all. We had an application layer on top of this to coordinate and enforce these requirements. Every which way you look at the code, there's no reason why this shouldn't work. And yet, when one node went down in production under a very particular circumstance with the network, it took down the entire cluster - not that the cluster was not reachable - it wouldn't allow reads and just hang.

Tracing it took me down the hole of understanding the precise network issue (fading memory - I think the machine wasn't responding back and tcp kept retrying) + linux tcp setting (I think tcp_delack_min) + a java networking setting + how thread safety was setup in the application (do not use synchronized at the method level). It took a week to diagnose.

Recently, there was a minor issue with adapting a sample Spark UDF to work with our setup and it took the developers 3 weeks and they kept giving up after every try.

People in the category you describe are few and far apart in my experience.

To me solving a problem like the one you describe here is one of the biggest joys of working with IT.

Though it seems to come at the expense of being able to design large systems. Not sure if there is a way to be good at both.

How does skill at deep-diving into complex unknown technical problems hurt the ability to design large systems?

I’d guess that taking the time to deep-dive a problem takes away from time to design large systems.

I'm a senior dev I say this only because I have to.

The problem with encouraging developers to speak up when they're stuck is that the developer that is stuck isn't really ever sure if it is something they really should know about the library, framework, or language or if it really is something hard. In some sense senior developers have a real leg up here because they know that they're capable of building systems for large companies.

I've seen many junior developers fired because "they require too much hand holding" and I don't think the advice should be "speak up" but rather "make at least one friend as soon as you possibly can that you can privately ask questions to so you don't look like an idiot" and I know this advice is better for the individual than the company, but it's the reality. The selection function for new, junior employees is "can they get the things done with minimal hand holding" not "do they ask questions" and I'm not ok with that, but it is the reality and we shouldn't push juniors into situations that would get them fired before they're up to speed.

I really don't think I agree with this. The thing that's always frustrated me the most is when someone new doesn't ask questions when they don't understand something or aren't sure if they're doing the right thing then powers ahead anyway only for me to inevitably have to come along and waste even more time fixing the problem than it originally would have took had they just asked first.

Whatever the work is, the important thing is that it gets done, usually as efficiently and as good as possible. If it takes me explaining the same thing 3 times in a row to get something done properly the first time I'd rather do that than have to go back and fix things later.

And honestly, everybody starts somewhere, the best way to learn is by asking questions. If someone is asking me questions, it tells me they recognize they don't know everything, care enough to learn and are humble enough to ask for help.

Such qualities are rare in people and i've found every time, the people like that make the best employees and coworkers.

> The thing that's always frustrated me the most is when someone new doesn't ask questions when they don't understand something or aren't sure if they're doing the right thing then powers ahead anyway only for me to inevitably have to come along and waste even more time fixing the problem than it originally would have took had they just asked first.

Until you get the person who questions everything, draining all your time. Then they become reliant on you, and stop thinking for themselves.

This can usually be solved by coaching, not only for this person, but for the teammates who are spending more time answering questions.

For coaching the questioner, I ask them how they approach finding information, and recommend they time box their exploration, and only ask their question if they've given it a good try. I I also will describe my personal workflow to find answers when answering their questions.

For coaching the rest of the team, I often suggest they ask "what have you tried so far?" or "where have you looked so far?" before just giving an answer. This can help show knowledge gaps, and oftentimes the questioner will take another look and discover the answer themselves.

And yes, if they never get away from this behavior even after getting feedback multiple times, then that turns into a different sort of conversation.

I guess maybe I could clarify. I do agree there's a difference between a person who asks a lot of questions and one who just asks questions out of laziness. There's a difference. If a person is asking a lot of questions, especially the same ones over and over without any clear signs of gaining understanding, yeah there's definitely a problem, that person is dragging down work then.

But I do think there's a difference. If a person generally asks a lot of questions, but there's clear signs of improvement and they're trying to pull their weight, then I really don't see it as a bad thing.

I agree both as a junior and as a senior human.

That is what I believe you believe and I understand where it is coming from. I've managed juniors and I know they make mistakes.

That said, if a junior brings up a question like "why does this error for some tests, but not others?"

    records = some_orm_thing()
    some_model_ids, other_ids = map(list, zip(*records))

The answer is obvious. Some tests return no records for the ORM thing and that doesn't unpack. If you ask this type of question it makes you look bad. I'm not saying it's the end of the world or anything, but we should at least be realistic about how juniors are treated. A private, friendly slack message is much less likely to get them fired than a post in a channel or pull request.

My wording was maybe a little strong, I did try and clarify a bit in another comment below but, I was maybe a little bit unclear and think I might not have fully understood the point you were making. I admit, I did take it as juniors should just shut up, which i admit probably wasn't the most good faith way to interpret it.

I agree there's a difference between the kind of question like the one above vs. something like 'Why isn't there a check in there before unpacking a record?'

The first question is basically 'Teach me stuff I should know' the second is more along the lines of 'Why do we do it this way?'

And there does come a limit, if after a month they're still asking the same questions, not really seeming to have learned anything, it's probably not going to change.

I dunno, really I guess it usually comes down to the person and I hate to say it but, how annoying I find the questions and the person's behaviour.

I'm really happy you responded because I fully agree with you.

I think my larger point was that smart juniors should seek to get clarification privately after exhausting their own skill, but you're absolutely right that they should raise issues as they come up in a thoughtful way to accelerate the team and their own growth.

I agree. Whatever level a developer is, there is a mountain of tedious work to be done. Asking is more like getting directions, not taking a free ride. Good developers grow in a method negotiation skill, not in a complete autonomy, because exact methods and their details may and will be very important for future integrations. In a complete isolation, no plural number of seniors will create a good system. It’s strange to expect that from noobs.

Of course there is always a level of knowledge too low to participate constructively, but that’s another scope (fire your HR along with yet another zero-skills guy, etc).

I am in a senior(ish) position and honestly I prefer talking things out with at least one other person before making larger decisions. Even if they're a "junior" person, explaining my ideas to someone helps me see the spots where I don't really understand the problem before I get to solving it. I especially like it when people disagree with me because it forces me to construct a real argument in support of my approach.

My intuitive response as I started to read this was a sort of "gentle-art" thought: Would it be helpful for someone to start their question: "I don't know if I should know this [already]..." ?

If the answer is [in context] indeed "yes", then that could help tune their threshold for when learn-by-self vs learn-by-others will be appropriate [again, in that context].

I think speaking up is fine. It is when ANY engineer is lazy and doesn't bother to:

  a) do any research before asking the question, 
  b) trying to understand the answer given or 
  c) doesn't follow the answer given.

I will help anyone that asks but if they are just using me for quick answers and never learning anything then eventually they will use up their credits with me. I cannot do their job and mine at the same time...

> The problem with encouraging developers to speak up when they're stuck

I'm not actually encouraging that. I'm encouraging them to not have imposter syndrome when they actually manage to fix something, regardless of their methods. The ones who ask stupid questions, in my experience, don't end up fixing the issue ever because they don't understand what they're doing enough to ask the right questions, so they wouldn't be at this point.

I've never worked at a place that treats juniors like that, so my experience is opposite of yours. It's totally fine for the more senior dev to push back and say, "I think you kind figure this one out, maybe look into X", but juniors should be asking questions, it's one of the primary ways to become un-junior.

I've never even heard of a junior fired for "asking too many questions".

> is that the developer that is stuck isn't really ever sure if it is something they really should know about the library, framework, or language or if it really is something hard

It's also important to know if it is a bug or incomplete implementation of a feature. Sometimes the code is over engineered or there are no docs to help understand its intent.

If junior devs get fired for asking questions, or because that makes them look like idiots, I'd take a hard look at whether they have the support needed to set them up for success. Maybe the solution is actually "don't hire Jr devs" because the organization isn't able to support them.

IMHO the rule of thumb is - no stupid questions, if repeated refer to previous answer, if repeated consider firing an idiot, if it came from above you - consider promotion or giving a notice.

> developers fired because "they require too much hand holding" and I don't think the advice should be "speak up" but rather "make at least one friend as soon as you possibly can that you can privately ask questions to so you don't look like an idiot" and I know this advice is better for the individual than the company, but it's the reality. The selection function for new, junior employees is "can they get the things done with minimal hand holding" not "do they ask questions" and I'm not ok with that, but it is the reality and we shouldn't push juniors into situations that would get them fired before they're up to speed.

Thank you for saying this. I haven’t heard it being said anywhere else, but knowing people that you can DM and ask about things first is “socially” a better strategy than an ask tons of questions in a public thread. What’s also worked for me is to have a private channel meant only for developers (no manager, PM etc. Just devs, free to ask any question without judgement).

People instinctively assume the junior person asking questions is less competent.

It varies widely by company culture. At a previous job we had a Slack channel called #dumb-questions, specifically for exactly this type of thing; a no-judgement zone where everyone could share the job of filling in each other's mental gaps (one problem with finding just a single friend to ask questions of is that you start to worry about bothering them too much; this way no one person was a choke point)

> It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard.

I think that's why he's putting scare quotes around "easy", because it's referring to what managers or stakeholders think rather than the real level of challenge.

He explicitly says this in the article. I think the majority of commenters only read the title.

> I think the majority of commenters only read the title.

I've lost count of how many times this has happened.

In one case, I went through and read several dozen comments on an article. No exaggeration: all commenters (100%) said nothing whatsoever about the article's text. They were all just holding forth on their own opinions and experiences related to the headline.

Maybe if the article was right here instead of a click away, more people would read it.

I don't think so.

Just look at YouTube, lots of people comment on videos without actually watching the video despite it being right there.

I've gotta ask : How do you read these quickly ? It'd take me so much time to read even half of the articles from threads I'm somewhat interested in, I can't help but feel that trying to get the gist of it by reading the comments is the most efficient way for me. Well, it would've been, if the people commenting actually read the article.

I read the article, doesn’t mean I can’t leave an anecdote and reiterate the same point.

> if management/product view something as easy or hard.

This applies not just to work done, but work to be done.

Once the business people asked me to do something that turned out to be extraordinarily difficult. Afterwards we were talking about it and they mentioned that they considered asking me to do it a different way that would have achieved the same goal, but they thought the second option would be harder, so they only asked me to do the first option. The second option turned out to be extremely easy and I could have finished it in 5% of the time option 1 took.

This applies outside of programming as well. One time I was building kitchen cabinets for a remodel. I gave my wife 2 options for the doors. She went with the more difficult option. After they were done she admitted that she preferred the other option, but thought it would be harder so she went with the less desirable option to save me work. Facepalm.

I think this is a perfect example of why it is necessary to effectively communicate the relative tradeoffs and costs of option A and B when someone asks you to do something.

And to also also questions to dig into what they are really trying to accomplish, and why they are trying to do it that way.

this is what the article talks about, you want to work on things you know are easy, but others think is hard

In the last 5 years I have probably written 100k+ lines of production code. My most celebrated feature was about 100 lines to store a single flag in a per user store.

Way to describe the engineer/business disconnect.

You are proud of the complex problems which you have solved over the years. The business is proud of the little widget that made everyone's jobs a bit easier.

I've seen people take this lesson to heart and go full Wally.

>I've seen people take this lesson to heart and go full Wally.

Who is Wally?

https://en.wikipedia.org/wiki/Wally_(Dilbert)

It was a little widget, just needed a bit of state persisted across sessions. The pain point was at page load, so about as far as most business pays attention. Also I do feel like it caused me to go full Wally without realizing it.

but if management/product view something as easy or hard

Yep. My job involves a fair amount of coding but it's geared toward data analysis. I might create something in a day with dozens of data points and look like a hero, while another project might take a week and the output is a single number.

Actually, yesterday it was two numbers, X & Y. (Upper & lower bounds on something). Deceptively simple in appearance to anyone consuming the information, but behind the scenes it took an order of magnitude more time to make it a dynamic model based on latest available data instead of a static hand-crafted analysis.

> It matters, I think, not if the task is easy or hard but if management/product view something as easy or hard.

That's eye opening, thank you! I've once worked on a "hard" statistical model and all I got from my boss' boss was a "who cares" face. Then he was so happy to see our (to my view) dumb web app, he didn't even realized it was the model behind that app that consumed 80% of my time/energy :/

I've had this plenty of times, where praise and accoaldes come down tot he person who made the prettiest or shiniest thing. It's kind of why I moved into frontend development, I always have something so show every week, and management are always happy, since I always look like I have made amazing progress. It's gaming the system, but I would rather this, than what you experienced, where the developer looks like they are wasting time in the eyes of management.

Those two problems sound like they could differ substantially in perceived business value. Maybe it’s not how hard management think it is, but how valuable?

If management thinks it’s easy and it has big business value - they’re going to understaff it and expect it done tomorrow. Which is gonna hurt someone, and it usually isn’t the execs.

I feel your pain and I have been/am in your shoes.

Unfortunately, you will not get the kudos for things you build unless you spend time selling those projects. I personally hate this as I want to spend my energy on building things that make a difference.