Hacker News Reader

Comment by mi100hael | Hacker News Reader

I generally agree with the author, but as a software developer, I try to stay much less emotionally attached to the end-result. If I have concerns about the long-term stability and viability of a code base, I'll raise my concerns with the project manager or whomever's making the decisions on what to prioritize. But if they don't want to, that's their prerogative. They're higher on the chain of command than I am. I'm certainly not going to lose sleep over it. When the house of cards does come crashing down and it takes two weeks to add a simple feature due to tech debt, or worse, some bug surfaces that takes down a production deployment and takes more than an hour to fix, my ass is covered. I have an email trail raising concerns that went ignored.

View on HN

This is the same CYA attitude that I have been assuming the last 2 years because it's such a frequent occurrence.

The only issue is that I still get blamed. I don't know how many times I've repeated this scenario:

1. I highlight gap / issue in code.

2. Team says, "Need to ship; we'll accept the risk"

3. During go-live, fecal matter hits fan because of aforementioned gap in code.

4. Everyone acts surprised.

5. I point out that I mentioned it 2 months ago.

6. Everyone makes excuses / claims they don't remember it that way / moves on to burn down another project.

7. I'm left to clean up a mess / have taken a hit to my reputation.

Not sure what I'm doing wrong...

Next time you raise an issue, point out what happened last time. Then fix it.

Nothing...

In the country of the blind, the one eyed man gets locked up in a lunatic asylum.

> Not sure what I'm doing wrong...

Perhaps you failed to build a paper trail (i.e. an email trail) to document the fact that you predicted the disaster, and who it was that decided to ignore your warnings. If you do that, when the fingers start pointing in your direction, you can send everyone copies of the emails that prove who is responsible for the mess.

No, that's not it, because usually management doesn't give a shit about that. Even if you have the paper trail, they're still going to say, "Quit whining; it's 3 AM and stuff doesn't work. Fix it."

I too want to get paid for being irresponsible.

If they want us to be responsible, then they need to give us what we need when we say we need it. You're saying they should be able to make us cut corners and then support it. You don't get to have your cake and eat it, too.

You have to win people over by making their lives easier or by making them look good. This does not necessarily have to be related to code or technology in any way.

Then, after people have a visceral positive feeling about you, make some tiny suggestion for improvement. Make the improvement, and whoever approved it, do something to make sure they associate your suggestion with a positive experience.

Slowly repeat. It is a slight selling of your soul. It is effective.

IMO there's nothing you can do to fix this culture. They're determined to ignore the inconvenient problems and then conveniently forget they were told about them. Even if you attempt to document these things, you'll just be labeled some negative name like "whistler blower" or "anal" and the same reputation problems will exist.

FWIW, 2 things mitigate this. First, some companies do actually remember people who fix things and get things done, so that's on your side. The other is that it's not likely you'll be there long-term anyhow. Companies that do this also generally fail in other departments, like paying you what you're worth as your worth increases. So you'll probably look for a new job eventually anyhow.

I know it's hard to see from where you're standing, but I wouldn't fret about it. Just keep doing your job the best you can and warning them, and let the chips fall where they may.

Perhaps you're doing what I'm about to recommend already, but I can't be sure based on what you wrote. It's in regard to "2. Team says, "Need to ship; we'll accept the risk".

If the Business Unit, for whom the software is being written, has a representative on your team, then make sure you explain the risk in purely business terms, not technical ones. A BU rep's ears are going to perk up with statements like "invalid posts may get written to the general ledger" or "1 in every 20 orders may get dropped" rather than "universally scoped variables are being overwritten by invalid data in some cases" or "uncommitted nested transactions may cause a core dump."

If your team doesn't include a BU rep, then you could always get friendly with one and explain the potential risks over coffee or lunch. Perhaps even suggest some questions the rep should ask the team before the mod goes live.

In my situation, a BU rep is always part of the team and I rarely have other developers on the team. The reps and I have lunch together regularly anyway, so they get my opinions and recommendations that way.

9 times out of 10 it's the "BU rep" that's responsible for rushing things out the door yesterday.

The only thing you're doing wrong is pointing out that you mentioned it two months ago. That's the last thing an organization faced with a problem wants to hear. What they want is for everyone to act like the problem was entirely unpredictable and have everyone make 'heroic efforts' to make the fix.

Thereby perpetuating the culture that ignored the issue, and providing no downside to the individuals that ignored it.

>1. I highlight gap / issue in code.

>Not sure what I'm doing wrong...

probably you're doing it too serious and taking it too personal. After 20+ years in the industry, I got desensitiveized enough to bother that much about small things, and when i point to the issue i usually do it in the "constructive" "spirit-sharing" and "team-building" way - "Ha! Just imagine how those suckers - customers or downstream devs - would be hitting that bug, completely lost at what to do, they would have no chances to make it work so we should give out a Grand Prize to the unlikely one who would make it through... " I noticed that various leaders/management/etc. have hard time to full-heartedly subscribe to that vision of future. Not that that gets the thing fixed or anything like this, mind you. Yet it helps with the blame part if/when it still comes my way - "Wow! when _we_ (me and you, the manager/etc.) were laughing at it _we_ did look into the crystal ball! _We_ are that smart!" The management somehow don't enjoy sharing in on that smartness :)

I know this is how it works. But it troubles me. When did we abdicate responsibility for the systems we built? Does a bridge Engineer find it acceptable to give notice "The bridge will fall in high winds". Then build the bridge anyway when management decides its an acceptable risk?

At some point (agile?) we explicitly gave up all moral responsibility for our creations, to people with money.

To be fair, for the vast majority of software projects it's more like "The plastic canopy on this toy firetruck will crack if a child steps on it, which they will", and management saying "nope, use that cheaper plastic anyhow".

In that case, the mechanical engineer would certainly acquiesce, just like most of us do when management prefers speed/cost to scalability.

Yep, exactly. I think most of the time management isn't totally blind to the risks of crap software, it's just a matter of weighing the time & cost vs risk and deciding what the acceptable ratio is. Obviously I want to put out the absolute cleanest, best possible application, but one that gets the job done without crashing is all the business can justify time & cost-wise.

I can understand that, but they seem oblivious to the long term cost too. They usually have no idea how seriously technical debt will affect them 3 years down the road and by the time it gets to the point they can't even hire anyone decent willing to work with the code.

> Does a bridge Engineer find it acceptable to give notice "The bridge will fall in high winds".

Not any more.

But in the 19th century bridge failure was quite frequent, until things like the Tay Bridge Disaster happened, and then some standards were imposed, and engineers and others were empowered to enforce high standards.

In any field you can have fast development, or you can have safe development, but you really can´t have both. It remains to be seen quite what level of software disaster it will take to bring this lesson home in our industry.

If last Friday events do not, I am not sure what will.

Bringing down Twitter for part of the country for a few hours likely won't be enough. Unfortunately it will probably take people being physically hurt or shutting off most of the internet for days at a time.

The tragedy is that, with non-critical software that can't kill anyone, users are happy with most bugs most of the time. Therefore we will never have legal quality standards for most types of software.

I realize that you're talking about "quality standards", but depending on the industry, there are plenty of legally mandated software standards out there. You're likely to run into some of them if you haven't already:

- 508 / ADA compliance

- HIPAA

- AICC / SCORM / cmi5

- PCI / DSS

Some of those are not very good i.e. not quality standards. Is there a quality standard for quality standards or does the buck stop there?

Well, that's just the thing. Whether (we) programmers like it or not, the only relevant measure of quality in a program is whether it does the thing that the user needs it to do. In other words, "functional standards" is the only type of quality standards that you could ever expect to have for software.

Things like code quality and inner loop optimization are largely irrelevant for the people who are using the software.

> At some point (agile?) we explicitly gave up all moral responsibility for our creations, to people with money.

You hit the nail on the head. It happened when we stopped financing our own work.

Agree.

We would have dramatically less surveillance programs, automated enforcement and bombs if there wasn't as many people ready to blindly write the software for them.

We should be thinking hard about what we are building and how it will be [ab]used, especially in government.

Bit of a jump there, no? The ethics of building crap commercial software is quite a bit different than the ethics of building mass surveillance or munitions software.

No.

"I try to stay much less emotionally attached to the end-result".

This is the mindset that drives most of military, police and totalitarian policy. You don't drop a bomb or violate the constitution without a bit of emotional divestment.

The mindset may be the same. The ethics behind adopting that mindset are very different. No one dies if Tinder crashes because it tried to load too many images at once.

I believe there clearly is a difference when lives are at stake. Often that is not the case with software (but it can be!).

You don't know that, and cannot know that.

Our industry loves to write CYA statements about the product not being intended to be used in critical situations and bury it in the fineprint of licenses we know customers do not ready anyways. Then, we act surprised when the same customers go and use the product in life or death situatioins.

This is not theoretical, just to highlight the most well known example: Our current generation of social media was created for the purpose of allowing college aged people in first world nations to share trivial details about their personal lives with friends and aquaintances. But now we see activists using the same media to organize civil resistance against governments; specially governments with huge stains in their human rights record.

> Does a bridge Engineer find it acceptable to give notice "The bridge will fall in high winds". Then build the bridge anyway when management decides its an acceptable risk?

Maybe they should. There's a reason agile won; it was and remains more efficient than the alternatives.

More efficient in producing crap solutions that can be sold by reinvesting time and money into sales and marketing, yes.

Better from the market's POV != better for the people who use it.

Which is still an improvement on taking months/years to produce a crap solution that also doesn't do what the client wants, as was the case before.

Upper-case ‘Agile’ won because it's micromanagement without the blame falling on the managers.

"If builders built buildings the way programmers write programs, the first woodpecker that came along would destroy civilization".

It wasn't that we gave up responsibility; it was that we were told "ignore those problems and build it poorly anyway, because if you don't we'll just replace you with someone who will."

We abdicated responsibility when management decided to stop giving us the time we said it was actually going to take.

I don't know about you, but if I'm not given what I need to do something, there is no way in hell I'm going to take responsibility for it.

EDIT: Not to mention that, if I do hold my ground and refuse to do something until I have what I need, the company is going to fire me and find someone who doesn't give a shit. And quite frankly, I like being able to buy food.

Have you ever had a project manager who has software development skills and experience?

It seems like there is a class division now, even at startups where the devs are on the frontlines and all actual decisions are kicked up a hierarchy.

If there's a quality issue, the frontline devs should be able to blow the whistle and get the problems resolved because we have the technical knowledge to identify the problem and to fix it.

CYA, Cover Your Ass, is a good call though since the tech companies aim to keep developers down.

I have never had a project manager who did not have software development skills and experience. It really doesn't make a difference: they still need to trust the team they're leading.

Developers need an appropriate level of authority.

Building something safety-critical? Every dev needs to be able to "pull the rope and stop the train" if something isn't right.

Building a toy? Kick the "this is fragile, should we do it?" questions up to the PM.

I don't want my devs deciding to arbitrarily gold-plate every trivial feature, but at the same time, if there's a latent quality problem, I need to know about it.

They're not interested in your email trail. They just want a scapegoat. You're not put on trial first before you're fired; you're just summarily shown the door. Since you're not given any chance to defend yourself, your documentation doesn't matter.

Blaming an engineer for supposedly poor engineering is a lot more politically viable than blaming management for poor management -- because they're higher on the chain of command than you are.

¯\_(ツ)_/¯ It's worked for me thus far. Actually, most of the time the situations have improved. If you go about raising your concerns tactfully, people start taking your opinions more seriously because you accurately forecasted problems in the past.

It can also work to just take a little longer on a feature and do some cleanup/fixing without getting permission. If anyone bothers you about it, just say "I noticed this issue and wanted to avoid running into problem xyz again since it took us offline last week." Again, most of this boils down to tact. Project managers don't like looking bad to upper management. Usually you just have to point out (tactfully!) how dumb they will look if something crashes in prod on a weekend.

I did this at a previous company, and got blackballed for having a "negative attitude". 6 months after I told my manager and the technical architects that it wouldn't scale, the project got cancelled because it didn't scale. No one suffered except for me because I had the balls to open my mouth.

I've been employed at places where that email trail is ignored or used as "evidence" against the engineer.

This sucks. Like 'hawleyal wrote, survival requires it - but it's painful, because you can't get attached to the product you're working on. If you do, you'll be fighting with the management, customers and/or cow-orkers.

>... cow-orkers ...

That's insulting for the cows !

Funny... but surely you should be thinking of the orks as well.

Agreed. Survival requires it. Still feels like a shame to knowingly spend a lot of your time making garbage for money. MLIA

Not to pick on you especially but, uh, you are evidently one of the toxic co-workers the guy was ranting about: you are enabling the fuckery.

Do you get that?

Things shouldn't be this way, that's the whole point of the article.

Maybe to some extent I am. But I have enough senior/architect experience to have dealt with all of the issues the author raises. Inheriting tech debt, bad developers writing code that needs to be immediately re-written, lazy developers that just don't give a fuck, etc. With a perfect team of developers and a perfect management team, there would be no problems. Unfortunately, we all have to deal with idiot developers or idiot managers or idiot customers at least occasionally. I don't intentionally enable fuckery, but I've learned long-term career happiness depends on picking your battles wisely.

Wow, you're good. ;-)

I apologize.

I'm frustrated at work, I don't understand why all these intelligent, capable people wind up building such a haphazard, nearly spastic organization.

I should not fault you for coping, even thriving, in a world not of your own making.

The problem is, those problems with the code end up becoming the fires that you then have to fix on your off time. You think you have your ass covered, but management isn't going to give a shit that you have this paper trail. They're going to see that shit isn't working, and you're going slow and bringing up ancient history.

Reply on news.ycombinator.com