Hacker News Reader

Comment by raldi | Hacker News Reader

I agree with your assessment that the primary reason we write unit tests is to be able to quickly make changes in the future without fear of breaking something: You do some extra work today so that you can save a tiny bit of work (and worry) tomorrow, and over time that accumulated future benefit outweighs the penalty paid today.

However, many in the industry forget that this is the underlying reasoning, as seen just two days ago here on this site. Read through the top-rated comments on this post: https://news.ycombinator.com/item?id=13119138

They describe an emergency situation where a single "3" needed to be changed to "4" ASAP or people would lose their jobs, and everyone's applauding the gatekeepers who insisted on significant refactoring and the creation of additional tests before the change could be approved.

I agree with those who say those improvements should maybe have been demanded immediately after the fire was out, but those who would have delayed the firefighting out of blind allegiance to the rules seemed, to me, to have forgotten that the rules are there to serve the programmers (particularly, their ability to quickly ship working code), and not the other way around.

A rule that's failing to do that should be changed or ignored.

View on HN

I wonder what would have happened if he had initially told the CEO it would take 6 days for the change?

"We can do that, but it will take us 6 days, otherwise we risk taking the plant down and aggravating the issue."

I wonder if the CEO would have just said ok thanks.

In my experience that's the case. The engineer in that link got himself in a bad spot, because he didn't know what was involved for the change when he communicated his estimate. And most of his back and forth that slowed him down would have been avoided had he known beforehand how to properly do it. Even with everyone's feedback it sounds like a 1 day code change. That seems to me like the reason the change was slow is more ramp up time for him working on a code base he doesn't normally work on.

I agree that the best possible answer to management would have been, "It'll take six days by the book, or if you approve some emergency rulebreaking and the issuance of technical debt, I can have it done in an hour. And then it'll take six days to clean up."

Then the boss can make an informed decision.

Yes, this. Provide options to management, with risk analysis, so they can make a decision. Management's priorities are not always the same as development's priorities.

The important thing here is to provide the risk-assessed alternative in writing. This covers the asses of both sides! If something blows up and management was not warned about the possibility, they're well within their rights to rake the engineers over the coals for it. But if engineers warn management of the potential consequences and management chooses to take the risk, if it blows up, you have your CYA right there - they were warned in writing.

This works if you trust your manager to take responsibility for the decision, which you often (usually?) can't.

If you can't trust your manager to take responsibility for the decision, then it's better to make a decision you're going to be held accountable for than to let your manager make the decision and then hold you accountable.

I've seen other engineers in the position where they give a risk analysis and warning in writing, and then shit hits the fan and they get fired. Maybe it doesn't happen immediately, maybe the reason they were fired isn't explicit, but the change in the manager's attitude toward the engineer traces back to when they did what the manager said.

There are also mangers who won't follow through on the tech debt part because they don't trust their engineers even if their engineers are trustworthy. When they discover that they can bypass testing by pulling the emergency lever, they'll start pulling it all the time because they see it as a way to get what those lazy engineers to do their jobs faster. And when tech debt catches up with you, bugs abound, and development slows to a crawl, the engineers get blamed.

Maybe you have a boss you can trust to take responsibility for their risky decisions. Maybe your boss trusts you when you say that paying down tech debt is necessary. But maybe your boss and their boss don't have the same relationship, and the shit rolls down hill.

Yes, I want to work in a trusting environment where my interests are aligned with doing what's best for my company, but an at-will employment capitalist economy doesn't always work that way. It's every man for himself at a fundamental level, and exceptions to that are too rare to make a blanket claim that people should just do what's best for their company.

If you can't trust your boss to not scapegoat you when you provide a risk analysis in writing, you need another job.

Eh, it's not all that big of a problem of you know how to deal with it (protect your own interests, not the company's, by making conservative choices as I described above).

I work 35 hours a week, make $125k/year, have good benefits, like most of my coworkers, and am doing relatively interesting work for a company that isn't completely evil. Sure, my boss can scapegoat me if I let him make technical decisions, but I don't. That's a tradeoff I'm okay with. I'm always on the lookout for something better but I'm happy where I am.

And besides, most people who think their bosses won't scapegoat them if something goes wrong are naive. If it's your job or theirs and they have the power, you're screwed. It's better to avoid the risk and not put yourself in that situation.

To be clear, I'm not saying don't take risks. I'm saying wroth risks on your own behalf.

"Yes, I want to work in a trusting environment where my interests are aligned with doing what's best for my company, but an at-will employment capitalist economy doesn't always work that way."

What economy does work that way?

Employee-owned cooperatives sidestep this issue pretty well by empowering workers. I don't know of any entire economies, though.

I agree with you. I think most often then not, you need to withhold the inferior alternative, or be very careful in how you presented it. If you downplay the risks, or you don't emphasize them enough one way or another, it will still end up being your fault, even in writing.

Trade offs that are business related can be delegated and should be delegated to the business to make. But saying something like: "if we do A, it could make us vulnerable to X,Y,Z technical problems, but would allow you to have what your business needs 7 days early" can be dangerous, because it assumes the business stakeholders truly understands the dangers of technical issues X,Y and Z, which they almost always do not.

As an engineer, you should advise the business towards the proper way to do things, and you are responsible to make sure your advice is clear and loud. It is sometimes appropriate to suggest the less ideal scenario, but if you do, be ready to assume full responsibility and ownership on it.

You should never have an advice that goes like: "Yes we can do it, but it'll cause other problems." and expect for these other problems to not be your responsibility to fix and mitigate as they appear. The implicit of all your suggestions is always: "I can make this work." So better be sure whatever you suggest you can actually have it work.

It's important not to understate the fact that both options are a false choice. you can have it the right way Plan A, or the right way Plan B. There was no offer of a Plan C where we hack stuff and then just hope for the best.

Plan C is what the Strategy guys always want to go for, because they can't recognize when the repucussions happen. They just tell themselves it's those good for nothing engineers screwing up again.

But too often there's been one guy ithe engineering group who values accolades over stability and will offer to be a hero, only to wander off without finishing anything. Later in life I've realized I should have suffered through more of the issues at companies where the team at least had solidarity. I knew I wanted that in a team, but didn't know that I needed it.

That's why you don't offer a Plan C. There's a fast-but-dangerous Plan B, with a risk analysis. You've warned them what can happen, in writing. Now, if something goes wrong that you didn't warn about, there could be other repercussions, but that's still better than not warning them at all, which easily makes you the "But no one told me this could happen!" scapegoat.

Plan C doesn't differ from Plan B in terms of technical approach. It differs in terms of the consequences of failure - not the technical consequences, but the political consequences.

My first really interesting project that led me down the sort of DevOps/Agile trail was in the mid-1990s, when I helped write a risk management and contingency planning process for the mid-sized company where I worked. One of the features of the process was that management could not reject a risk analysis on any grounds except incompleteness. They couldn't say "I don't want that written down!" They had to sign off on it, in writing.

This turned out to be very popular with both teams and management. Teams felt they were finally getting the opportunity to cover their asses, and management finally felt like they were getting straight answers from the teams about actual risks. In the earlier lack of process, teams were at least passively discouraged from honest, written risk analysis, as naysayers who were resisting business opportunities. Worse, if a warning was given and then the problem happened, the people who warned were blamed for not preventing the problem!

When we beta-tested the analysis process, the first project that tried it actually turned down business because it was too risky. That had never happened before. We probably saved the company many millions.

Manager agrees, rulebreaking then fix later.

Later...never arrived for tomorrow is another Very Important Thing that must be done now!

Somehow management continues to forget the critical resource: time to do things right

Yeah, but the problem is that managers are not engineers, they want "results" and do not have sufficient software engineering knowledge to be aware of the effects of making this type of decisions over and over again.

Eventually the software engineering department has a code base that is riddled with tech debt, to a point where changing a 3 into a 4 ACTUALLY takes 6 days.

Then management asks "WTF? What did you do? Why does it take 6 days to turn a 3 into a 4?", next comes frustration and SWEs leave the company.

I've seen this happen at 3 different companies in 3 different industries, a huge company (hundreds of devs), a medium company (50 devs), and a small company (2 devs) [1].

Every time it happened, it was because a SWE team was constantly delivering under pressure by management that disregarded the cost of writing code "that works" without ever addressing tech-debt, no matter how much the SWEs warned management.

[1] To be fair the small company only fell into that anti-pattern because neither the engineers nor the managers knew any better.

Perhaps the reason you've seen this phenomenon everywhere is that the companies that don't do it tend to fail, whereas the companies that do tend to succeed and grow and hire people like you.

This was one of the most important lessons I learned at my first job out of university. Beautiful, well structured, clear, simple code is nice, but the customer doesn't give a shit about it. All the customer cares about is that your product solves their problem.

By all means, if you have additional resources, invest them in refactoring and improvements and additional testing and continuous integration and all those things that make our days enjoyable and our product quality high. But your first priority has to be making sure that your product solves your customer's problem.

I disagree with this. Management doesn't need to micro-manage everything, that's certainly not their job role. This was a simple change in an internal system. Yes, maybe this minor super isolated change could have broken something in some parallel world, but the stuff it would have broken is not an airplane computer or a heart machine. Bloody hell, sometimes it is important to put things into context. Engineers get paid a hell lot of money and you would expect that someone with that qualification and level of experience would be capable to make smart decisions independently without being micro-managed by a manager all the time. After all the engineer is the real expert of his/her code and can make a much better assessment of the impact of the change than the manager, so freaking do it! That's what you get paid for! Seriously, there was no excuse for this to take 6 days.

This handshake adds yet another day to the process though.

Increasing the backlog by a month is not so urgent that 6 days is a problem. It's useful to have a process in place for more urgent hotfixes in production, but I don't see why that would be needed for this particular case. And rushing changes for a mission critical system can be very dangerous.

Already discussed: https://news.ycombinator.com/item?id=13131116

More accurately: by the parent of that comment.

If you read closely, it was only 6 days because the CEO did some rulebreaking at the end. I imagine it would have been several weeks otherwise.

At some point organisations forget that the process is there to serve us, and not us to serve the process. Moving variables to parameter files, renaming legacy variables, these all seem like much more risky things than simply changing the value of the variable.

> I agree with those who say those improvements should maybe have been demanded immediately after the fire was out, but those who would have delayed the firefighting out of blind allegiance to the rules seemed, to me, to have forgotten that the rules are there to serve the programmers (particularly, their ability to quickly ship working code), and not the other way around.

I think it depends on the type of product you are working on. Certain domains require very strict adherence to policy - for very good reason. Just because your shiny new aircraft's deadline is tomorrow, doesn't mean you can skimp on the required testing.

I think the linked thread is good example, changing 3 to 4 can seems minor, but i think even with rules in place, it is not a burning 'fire', it should be easily acceptable to run this with rules enforced in a day or two. Letting people to override rules is more damage in long run for sure. Maybe they would override next time when they shouldnt

Sorry, but you don't get to make that call; it's the CEO's job, and the CEO said it needed to be done immediately, not in "a day or two".

Having read that article, I do agree - but the requested change, while being "one line", could have unforeseen consequences that could also have a very negative effect on the business. I'm thinking that the CEO wouldn't take the blame for that though.

But if people's jobs were truly on the line, I'm inclined to agree with the "screw it, push it through" approach.

I wholeheartedly disagree. In fact, I believe that's the difference between an engineer and a programmer.

If you are hired as a programmer, then yes, just do whatever we ask of you.

But if you are hired as an engineer, everything the business asks of you comes with an implicit: "and make sure it's done in a proper way that won't break anything, or slow us down, or cost us too much, or limit our ability to gain a competitive edge."

You don't just change a 3 to a 4 because the CEO wants you to. You have to make sure the change doesn't come with unforseen impact that would put the company at risk, and you have to make the change in a similar way. That's what the CEO expects also. If you did the change, and it had caused impact to the business, that you had not pointed out, and for which the business believe is more harmful then having waited a few more days, you and only you are to blame, and you will be. You can't say, but CEO told me, you're the engineer, you're the person they hired to know this stuff and prevent these issues from happening, not the CEO.

I'm not sure we're actually in disagreement. If Captain Kirk tells Scotty, "We need warp 9 in five minutes or we lose the ship" and Scotty cuts corners that cause a decompression explosion that kills twenty redshirts, then yes, that would probably be an example of bad engineering management.

But if Scotty delivers on time at the cost of overloading an expensive piece of equipment that, after the battle is won, requires a week in drydock to replace, that's probably a successful execution of exactly the kind of call a senior engineering manager is expected to make.

It always amazed me how little Star Trek used robots, even just remote arms, to do things. (Yeah, it's a side-effect of it being made for TV)

Trek wasn't made for logical consistency. They could duplicate Riker with a transporter accident, but they couldn't make another Data that way, they had an entire episode about deconstructing him, then another about him constructing a 'child' robot.

Could be we don't, most disagreement is miscommunication.

In your example, Scotty knew what he was doing though. He didn't say, wow, what Kirk wants me to do could kill twenty redshirts in the process, I'll just take the gamble since he seems to want me to. He knew exactly the impact, and made it knowing he would easily be able to contain it.

Which is often not the case in Software and in practice. You have to do something to know the impact, because most problem we solve is always new. Its not something we did many times before. If that variable was often changed, then it would be completely different, because he'd known, just like Scotty, that its something they can do. In that case you can make the choice to say, lets change it, and later handle the tech dept of the less maintainable code.

Also, in software, its almost never the case that people can't wait a few more days.

If Scotty didn't confirm that it's important enough to risk serious damage, he didn't do his job. His job is not to blindly trust the captains omniscience, his job is to inform the captain of the technical consequences. If warp 9 in 15 minutes is preferable to taking damage, than that's the better option.

In the example with the line of code that took 6 days, there was no dramatic emergency in production that required cutting corners. If it had been an emergency, of course the code refactoring demanded in code review should have been postponed; those changes increased the impact of the change, and therefore the resting requirements.

And if it really is an emergence that requires people to drop what they're doing, then someone with sufficient authority should be directly involved in order to override all the usual procedures.

But you don't just drop all procedures just because somebody claims somebody said something. That would be dangerously irresponsible.

The point is that decisions should be made by the people who have the proper information to make them.

If the CEO lacks the understanding of the technical consequences of a change that may blow up the company, the engineer should make the decision. If the engineer lacks understanding of the business consequences of not making the change - like losing an important client, or suffering a wave of negative PR, or facing a lawsuit - then the CEO should make it. Ideally, both sides should be communicating these consequences so that both of them have all the relevant information and would ideally make the same decision. Then the decisions can get made at the lowest level that has all this information, and the CEO doesn't have to get involved.

In practice, there are many cases where the CEO can't communicate all of the relevant business realities, eg. if you're facing a lawsuit if you don't make a change, it's often better not to worry the rest of the staff or make them subject to depositions, and simply to ensure that the change gets made. That's why the CEO is the decider by default in organizations, and also why it's usually expected that employees will obey a direct order from the CEO or be fired.

> If you are hired as a programmer, then yes, just do whatever we ask of you. But if you are hired as an engineer, everything the business asks of you comes with an implicit: "and make sure it's done in a proper way that won't break anything, or slow us down, or cost us too much, or limit our ability to gain a competitive edge."

This is a distinction that is a very thin line and most people with "engineer" in their title would not sign up for.

True, most engineers, and others, will commit crimes including criminal negligence, when asked firmly enough by a powerful corporation that can fire them. See Flint, Michigan, the great robbery of 2008 etc, etc. But the idea of a profession is that you don't, that you have professional ethics.

Yeah the term engineer is thrown around pretty casually in the software world. You can apprentice and take the PE (Professional Engineer) test, but it's often hard to find software engineers with their PE certificate. It's much more common in other fields like Mechanical, Electrical, and primarily Civil.

It all my years (15) of professional experience, I've only worked under one PE, an Electrical Engineer.

In some states you literally can not have "engineer" in your job title unless you have a PE certificate/accreditation/whatever.

This seperation of programmers to mindless programmers and somehow superior engineers is bullshit. You have to follow the best practices in whatever you do.

Some people know how to program. Other people dedicated years to the study of computer architectures, first order logic, distributed systems, fault tolerant systems, etc ...

In my opinion, computer science and engineering is not about just making the code you're told, it's about questioning whether that code needs to be done in the first place, and if so, how.

Most software development is not done, or meant to be done as an engineering discipline. It is far more often a craft. That said, it depends on the project, environment, company and legal requirements.

For that, hire a programmer. Need a dependable architecture for mission critical software? You get the point ...

Define mission critical? Can be down for a 15 minute update once a week? Most have transparent updates? Should be up most of the time? Must be up durring East coast business hours?

It still depends.

Exactly. As an engineer, no matter what your boss says, you are responsible for the consequences of your actions. If the CEO tells you to do something which you know is unsafe, you must decline (and explain why, preferably).

Sorry, the CEO's job is not to talk about nitpicky technical details. He's got no voice in what a code or a test should be.

Ok this is the problem with story telling tbh:

David: It's for Philip. It we don't do this right away, we'll have to have a layoff.

and

Judy: OK, then I'll fill out that section myself and put this on the fast track. ----- 2 days later. ----- David: What's the status of 129281?

"It we don't do this right away, we'll have to have a layoff" and "2 days later", making an impression of this is an "a day or two" task, not a "fire/emergency"

I don't think if this change finished in 2 days anyone will be unhappy. But if you took this to production, and somehow failed, everyone would blame QA/testing

The ideal solution is neither always enforce the rule nor always permit lapses. Are you saying you don't think human beings are capable of coming close to that ideal solution?

In my experience, no, they cannot exercise judgment and an absolute system has to be in place.

In theory, could informed, intelligent, rational actors without ulterior motive do so, sure. I'd sleep on a couch and eat ramen for the chance to be part of such a team, but I haven't met them.

Those rules that were overridden were basically QA bikeshedding about variable names in something that has never changed before, and insisting on tacking on refactoring while trying to put out fires.

The engineer did their due diligence, wrote tests to make sure it had the desired behaviour, and got it done in minimal time. Clearing technical debt in old modules should be done, I agree, but not while trying to put out fires. It adds considerable risk to a change which should not have any impact except for the request.

> everyone's applauding the gatekeepers who insisted on significant refactoring and the creation of additional tests before the change could be approved.

I certainly didn't applaud the gatekeeper because I believe they opened themselves up to a great risk. If you are doing an emergency patch to production you should minimize the amount of changes. All the refactoring of code was an unnecessary, reckless risk. The refactoring should move to the next scheduled release as the top priority since part of it is already in production.

The scenario in the linked post was not a "fire", and it makes sense that it would require going through the regular process. If it were a fire then where was the CEO after step one? Why didn't he (or the 'operations manager') send out an email to everybody informing them to bypass the regular process?

A fire is something like, "the site is down for XX% of customers!" (for a large value of XX) or the "the software is routing product to the wrong place!" No doubt a real fire would have been handled differently.

Everything in moderation.

including moderation.

That's rather extreme.

I prefer a slightly different translation of that Ancient Greek saying: nothing too much

For a moderate amount of everything.

I think most people here have had the experience that the agreement will be made that the fixes will come "later". And then later never comes.

Reply on news.ycombinator.com