If you succeed you will fail
boyter.org
boyter.org
Some years ago, I worked on a system not quite as bad as what the author described, but close. We released a new product with a known, quite bad security vulnerability (I'd made sure our product team was _extremely_ aware of this), as well as no monitoring to speak of. The deadline had been communicated for around one year, but nobody had ever really discussed the significance of the date, other than it was what we were all death-marching to, and we needed to deliver.
What did that date turn out to be? The head of product management's birthday, which was revealed to the rest of the company on the highly-celebrated the launch date. People were just kissing ass. I left several months later.
It feels unconscionable to me that a company could have launched an incomplete, insecure, customer-facing product just to give a birthday gift to a leader, but I suspect this sort of thing is common.
I work alone with no deadlines. Sometimes I have to release something or I'll never finish. Perfect is the enemy of the good, as the saying goes. I often realise that half the "blockers" are actually nice-to-haves that I'm willing to complete much later.
However I don't consider the thing finished, only released. In a dev team, the post-release bug-fixing and refactoring never takes place. Managers immediately move on.
Come what may. If your product was to be released somewhere in that half of the year, this date it was.
Non-technical managers have a hard time with software engineering projects because they're opaque and unpredictable. Opaque in the sense that no matter how many progress reports they get, the manager will never understand what progress is actually being made (because they can't read the code). And unpredictable because development is inherently unpredictable.
By setting a hard deadline managers seek to control the project. Engineers will come to them with "we can't release this by the deadline because $technical_gibberish". But the temptation is always to just stick to the deadline and refuse to extend it, because once you extend it once then everyone knows it isn't actually a hard deadline and the deadline will get extended again and the thing will never get released.
My worst case was one project where the management team decided that we could skip the testing phase and just test on prod after launch, because that would allow us to meet the deadline. Obviously the application completely failed because there were thousands of bugs in it that hadn't been caught. The post-mortem was a total blamefest. The project got canned and people got fired. All that effort for nothing, just because they wouldn't extend by a few weeks. But we met the deadline!
Currently we have an issue where we are telling the customer that we need a 4 week pilot to make sure that when real data comes in it behaves as we expect, and so we can get back to providers on how to fix their data. If you go live with only synthetic data ever going through your product then you are asking for trouble.
They are trying to negotiate that down to 2 weeks. Why? so their KPIs are met? Maybe it's because we are external, and if it fails they can blame us for the problem rather than them nickle and diming over a couple of weeks.
I want to consult in a different sector after this :/
Personally, I've seen so many "super duper important must-meet" deadlines which in the end turned out to be completely unimportant, that I've decided long ago that I'm not gonna do any more overtime to make any deadlines.
It's quite a sobering experience as a developer to stress over making a heroic effort to meet a deadline only to see after the deadline passed that actually it wasn't really so make or break after all, and that there was plenty of time to fix the issues afterwards. It's really not worth it to abuse your body and your mind by working overtime and spending too much time in a bad posture behind a computer just to make someone else happy.
Of course, I'll still do my best (I always do), and I might even agree to moderate amounts of overtime where it's warranted (as in: not because it is necessary because of shitty planning), but I'm not putting any emotional energy into deadlines, ever again.
Bo. No they did not. The goal was to deliver a working product by some date. They missed the "working" part of that.
India delivers on a fixed price contract, on time. The bank are happy. Launch day gets near. They begin loading customer data into the live system in preparation. At first things are OK but then it starts slowing down. Slower and slower. By 100+ customers the system has slowed to a crawl. Investigation shows ORM abuse to be the root cause. There's no quick fix: the developers didn't understand ORMs or data model design and the whole codebase is just FUBARd performance wise. The project has to be completely written off.
In the wake of this, said large bank becomes terrified of technology projects. They no longer want anything to do with them. They decide to buy a platform from a competitor and rebrand it. Internal execs warn that this strategy is suicidal as it ensure they can never differentiate, but the warnings go unheeded. It apparently took over a decade for the organization to recover its confidence enough to try software projects again.
You see this everywhere outside the tech sector. Non-techs can't abide technical people and the way we do things. It's low key culture war. They see tech people as weird and unreasonable (sometimes true, but not moreso than the other way around). Nerds like Star Trek and get passionate about their tools and board games and the right way to interview people, and many other things that non-tech people find baffling or sometimes outright offensive (especially hiring). Non-tech can't tell the difference between BS and not, so constantly fall for yes men and grifters who just tell them what they want to hear. Then projects randomly explode and nobody can understand why or how to stop it happening again.
One thing they will not countenance under any circumstances: learning enough tech to be able to manage technical projects. If it's ever brought up the idea will be laughed off. Programmers will learn banking/logistics/medicine/tax law/etc if necessary to understand a project, but the reverse is not true and frankly deep down that seems to scare them.
These days most non-tech orgs have given up in my experience. They just want to buy a SaaS and if it means they can't differentiate from their competitors, oh well, too bad so sad but unless a tech startup takes an interest in your line of business it's not like any of your competitors are going to do it better.
Some organisations can do this, but I've see plenty that might outwardly try to avoid pointing fingers, but you can tell that despite warnings given, they do blame.
I told several leaders that one of their systems was literally on the brink and we were fighting fires on it every other day, and the processes in place were horribly broken.
12 months later shit hit the fan, my feedback was that I wasn't proactive enough, and they basically threw me and my team under the bus.
This despite budget under cutting, limits in hiring, enormous optimisation, education of other teams, research. Just getting 'No' or no real outcomes all the time on any escalation.
The leaders did this serially across the business, and lacked ownership on this aspect to even give people the resources and autonomy to do anything about it, yet would come looking when it came time, to throw other teams under the bus like this
It's very common.
There was probably a good portion of that army of ops and developers that felt the same way. And even if you can move mountains to point fingers and clean out all of the bad people, train the average ones, and keep the good ones around, you will likely get little compensation out of it, so no one does that. Even then, it's hard to blame incompetent people getting jobs and holding on to them. There's a lack of standards for skill and training so naturally people slip through the variety of bespoke vetting processes that companies come up with. There's also some economic reasons why people hold on to jobs for far longer than they should, even if they know they're bad at it.
Large amounts of bad complexity slowly erodes morale, but bad decisions like this one can set it on fire. Accruing bad complexity may not have any solid failures to trigger reassessment, which is why it often becomes a problem and stays a problem. It's an obvious problem for new people with fresh perspectives, while incumbents might have gotten used to it.
Encountered a similar situation a few years back while working for a government client. My advice to the team was that we needed to “let the train wreck happen.”
Otherwise the people who are accountable for the failure will say “oh who could have seen this coming” and refuse to learn anything for the next instance.
I think the reason for this is that observed production failure is _certain_. Hard fails are undeniable. They obviously need to be addressed and obviously deserve resources. The amount of deserved resources can be clearly calculated by projecting the concrete, observed costs of the failure forward in time.
Before prod goes down, there is much more uncertainty:
- even an expert engineering assessment has some level of uncertainty
- engineering may not have a full appreciation for the business context of the work, and might over-weight technical issues relative to other concerns
- if the engineers are contractors, or otherwise organizationally distant from the experience owners, that inserts a trust gap which further increases uncertainty
- the business owner’s projections are themselves uncertain. Is the expected launch volume really that high, or is it aspirational?
- the costs of failure are uncertain too… if the system goes down, how hard will it go down? What will that actually cost in lost revenue? Fuzzier stuff like brand reputation is even harder to quantify.
Meanwhile the costs of paying the contracted development team another 2 months on the same project are quite concrete. The team already spent significant political capital to force a change on an incumbent team. Now they’re saying they want more money because it still doesn’t work??
The big open question is - what was the cost of the failed launch? How long did it take to get the system back up and running at scale? What did it cost in terms of user loyalty? How does that compare to the concrete cost of holding launch until the auth system was upgraded?
Different people will answer those questions in different ways. What matters is how the customer answers those questions, whether their bosses believe that answer, and their bosses judgment of the overall situation.
What does work though is having a prepared refactoring plan ready to go when it all blows up. “Do that risky refactor we’ve been avoiding because of downtime?, well it’s all down now isn’t it”
- They failed to get the internal developer to buy in to the project or really work with the other team.
- They insisted on launching even though it was explained that the system wasn't ready.
- Insisted on having a launch party when failure was expected.
Developers, QA, product can all be incompetent, but when a manager is incompetent, he’s “evil”. In all seriousness, incompetence should never be rewarded and bad behavior should be called out, but in nearly every case we’re all really trying to do our best.
Proving that management is incompetent is a fairly large task in and of itself, and the people that can hold them accountable are all peers who generally wouldn't dare to "rock the boat" so to speak.
I've termed this whole set of phenomenon as a "managerial crisis", and it's been written about extensively since the last 60's.
If you sitting in the same boat this is a reasonable approach most of the time. Even then you may have to deal with an occasional irrationally acting person. However if you are in another boat e.g. providing services or products there are a lot fewer barriers for misbehavior and some may be even rewarded. Win-win and win-loose are different games and deluding yourself in what game you find yourself in a quick way to fail big.
That's the "evil" part. The unethical "I'll save myself" if things wrong and having extra power to sell that narrative due to hierarchy.
You may be trying to do your best but saying nearly every case is pretty false for anyone who has observed the power dynamics for a long time. Hey, the same happens with developers. Depending on the org, your best for the team and your best for the career rarely meet and it's harder to have them meet as a manager.
Your work isn't just the product, it is navigating an imperfect structure to achieve what you need to, despite the fact that the imperfect structure is the one contracting the product/service, and is (mostly unwittingly) standing in the way of that.
It's not fair, it's not fun. It is what it is.
One possibility that comes to mind is that they’ve heard dire warnings before from other engineers which have turned out to be exaggerated.
Engineers do have Chicken Little tendencies sometimes, and the long term results can be similar.
Tells me that management is incompetent. What exactly are they managing if they don't understand engineering estimation and consequences?
When warnings are not given and a disaster happens: clearly the team was incompetent and should have seen it coming.
When warnings are not given or ignored and no disaster happens: they were overblown and pointless, why do we even have a disaster recovery team.
When warnings are taken seriously and a disaster still happens: it's the worst of both worlds.
You can't win in this situation. No combination of warning/not warning and disaster/no disaster results in success. If you can't trust your managers to cogently weigh the risks and costs, there's nothing you can do.
My (engineering) boss is more risk-tolerant than I am, but after 30 years in the business I respect his judgement that a particular risk is worth running.
I don’t think it was a great look for them that you were brought in to build this, and once you start pointing out flaws in other people’s systems you become a walking target. I’ve made this mistake too many times in the past, determined not to do it in the future.
Step back and find an outcome where your work doesn't harm their reputation/security. So how do you make them into allies? Or how do you not matter to them? Figure that out...
People really are like that, yes: Monday the boss says "X is now responsible for Y" and lo and behold, Tuesday EVERYONE already says "Y is not working well, X, do something, it's your fault!".
> Step back and find an outcome where your work doesn't harm their reputation/security. So how do you make them into allies? Or how do you not matter to them? Figure that out...
I have figured out part of that and I was genuinely asking if you have additional insights.
This is what I use at work to keep myself sane. If the highly paid VPs, directors, managers don't care about something, why should I?
This leads to a lot of friction with manager where they want to put blame on individuals, months after they ignored the warnings. But realistically, it is their fault and they will get fired soon enough if they try to make everything a problem of engineers.
The only way out is for engineers to spend more time recognizing a manager falling off the cliff and switching teams/companies. Doing actual work is not enough to protect one from politics.
The impedance mismatch is usually too great and speaks to some serious deficiency at the architecture stage.
My guess is that the Auth system, which became the bottleneck, authenticated everywhere at once, which would be super slow.
Not sure what lambda had to do with this. The whole SAP thing reads like a legacy nightmare.
The point was that the launch was predictably going to fail, it was predicted, and then it failed. And this was due to the kind of arrogant, disconnected management decisions that are sadly very familiar to many of us here at HN.
Failure is envitable and to succeed at something you have had to have had experience encountering a problem. If you succeeded first time encountering a problem it is likely you'll never know the solution to the problem and thus it is envitable failure will eventually become of it.
It is enough to say success is built on a mountain of failures. Only difference between something being a failure and it becoming a success is that attempts made to produce a desired result.
There are times where things have to fail in order for change to happen, but employees also generally want to make things that are meaningful, and work. It makes the process to inevitable failure really frustrating / demoralizing for the employees.
Good, cheap, fast, pick at most two. Maybe one. Sometimes none. Occasionally negative.