Edit: I’m thinking primarily about technical decisions, though of course these are made in a wider context (eg choosing tech that’s impossible to hire for).
The company abandoned the rewrite, acquired a competitor, and rebranded it as the company's new product. It turns out the competitor's code base had serious deficiencies (couldn't scale to the company's size, had important bugs), and was very unpleasant to develop against. In the 6 months following the acquisition, well over half of the engineering team left the company (heavily skewed towards the highest-caliber engineers) because they couldn't stand what their jobs had become. Many long-time customers threatened to leave because of user facing technical issues in the acquired product.
A telling incident: the company needed to deploy an emergency fix to the mobile app. The only machine authorized to publish to the app store was a laptop that the owner of the acquired company had left at home overseas. To complete the deploy, he needed to call his wife and walk her through the publication process over the phone.
The company managed to fix enough of the problems that they're still around today, but for a while there was a lot of uncertainty around the company's future.
Wow, that’s doomed to fail hard.
Where I work at, we’re slowly rewriting a legacy Perl code base to Go HTTP services, and the best decision we made was to allow interoperability between the two systems. This way we can migrate the legacy parts by parts with minimal impact. This is a slow process (in 5 years we migrated maybe 3/4 of the code) but very reliable. And we actually have something to show for it.
- Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring anymore in this country, even when team members quit. It took awhile but I figured out that the plan was to let natural attrition take over and for every person who quits from here, hire their replacement there.
- Different company: after being acquired, the acquiring company communicated for eight months that no changes in staffing would take place, and that once the acquisition closes, everyone will be put on exciting new projects and a new bright future will emerge. Then, after the acquisition closed, laying off a third of the company immediately. Another third chose the exit soon after. Eventually, the company died.
- Third company: after being acquired by a large competitor, the director of technology for this third company promised them a new product for four years. I'm led to understand by those closer to the situation than myself that this director then proceeded to coast on vaporware demos for those four years, claiming the need to "pivot" or "reboot" the product as necessary, and promising more and more pie-in-the-sky fantasies until finally the gig was up and he was fired. That subsidiary also officially closed its doors eventually.
Just a few from my own experience!
You can imagine the general reaction - many left, those with vacation time are taking it, progress ground to a halt, motivation and morale through the floor - so basically, burning three months of salary instead of doing exactly what you were bothered by - hiring new teams in the new locale, scaling down work in the old one, and then after things are stable in the new home, shutting down the old one.
It's truly baffling to me given that they're moving to a more expensive location, they're confused by higher costs of hiring, slowed progress and doubling of expenses.
End of the day it seemed like the best way to set several million dollars on fire - I imagine an executive saved their bacon somehow, but I really don't imagine there will be much of a bonus after the math is all done and tallied. I'm really curious how the new teams will react, realizing that there was this massive talent dump and nobody seems to know where anything came from - the transition is being handled as professionally as possible by those that remain, but I'm pretty sure I wouldn't take an FTE offer from them at this point given the track record.
I have heard some companies hide the truth of relocating to another city and announce it right before the move. As employee, you either move with the company or find a new job.
It's probably better to build a team in the new location then announce the move in advance. It will be a shock either way.
Honestly, if the company has decided to move, that seems like a good way to do it, no? Nobody gets laid off, they avoid the bad will of axing a team outright or asking you to train your replacements; it becomes a simple gradual turn-over.
The way I would do it, is tell people that we need to move their team to {{other country}} and therefore we would like to offer them either a severance package to find a new job _or_ work out a plan for them to transition to some other team that is in their location.
Nothing can 10X sink a product faster than bad engineering management.
I once joined an established Enterprise software company where leadership had decided they wanted to make a new product. A lot of the justification was they saw the need with existing customers, and they could make $200M+ ARR by simply getting their existing customers to use it. They invested a lot of money in marketing, created a lot of hype, and built up a huge worldwide org.
One of the premises of the product was that if you had a fleet of machines using this product, it would be a force multiplier for extracting value out of a particular broadly defined use case.
Except, engineering and product management was so busy getting the engineers to build something that could scale to infinity, they never bothered to improve the basic stuff that let somebody get value during a trial period. Customers would struggle to deploy and integrate this thing onto one machine. And when they did, they didn't see the value.
Leadership always parroted the same thing. Slides about once you have thousands of machines using this, the value is there. But over 3 years nobody really implemented a nice way of integrating 5 machines, let alone 1000.
3 years later the org went from 3000 employees worldwide to a few sales people in every region and development mostly being done in India. Comically enough, the lack of financial resources meant that the new management had to focus on things that actually mattered to customers. So the product actually got a lot better. But by that time all the momentum and trust was gone.
as i recall part of the slogan was "long term thinking"
Companies failing for "big" single decisions - like rewriting their code from scratch, or a poor tech stack choice, or a poor initial architecture - are much more rare, for the simple reason that most experienced technical leadership knows that these are risky decisions and tends to put a lot of thought into them, and then there's a lot more organizational commitment to following through on the decision once it's been made. Also, if done from a position of market leadership, you can usually recover from them - many startups start with a poor tech stack and poor architecture, and just assume that they'll rewrite everything once they get lots of funding and a suitable lead over competitors. The insidiousness of the "creeping tech debt" scenario is that you often don't realize you're screwed until you've fallen behind the competitors, which means you enter the "rewrite and tech switch" scenario from the position of being market laggards, which can kill you.
The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language.
This lack of in house skills also meant that oversight of the code the contractors produced was poor. Several of us raised alarms over this but we were ignored. Eventually, the day came when the first paying customer was signed for the new platform. It was a complete disaster. The code was very unstable and full of bugs. The deadline kept getting pushed farther and farther out.
The new owners concluded the entire division was a failure. They fired the head of engineering who had been in charge of the disaster and his boss too. Then they started work on a new version of the platform using entirely different people. The ones left behind on the old platform (in the old language) had nothing to do but provide support for the dwindling number of open contracts.
Millions of dollars went down the drain, years of time were wasted, customers were badly served and a lot of people were incredibly frustrated, all because one executive decided to ignore the in-house talent and experience and follow his own inflated ego instead.
FWIW, he was fired after about 18 months on the job (for reasons unknown to me).
Ignoring for a moment the rest of it: What was the end state supposed to be? Like, pretend the rewrite went perfectly - now what? You still have nobody in-house that knows the language, let alone being familiar with the codebase!
One day they vanished and owed us nearly $800,000. It was enough that our parent company just shut us down a few months later. Oddly enough we had a really good group of developers and likely could have rebuilt the whole app ourselves, but our parent insisted on trying to recover the money instead of just taking the IP.
My guess is that the software is so boring that it attracts weak engineers. But the complexity is really high due to the regulations. So in the end it’s a bad mix.
It’s both an operational and technical decision because it results in inadequate tooling and specials.
A service I know well had an issue, and as a result they started doing what modern services do - nuking things going it will clear as clean versions come up.
But unknown to the ops team, they nuked a bunch of custom stuff that no one knew how to build or really what they contained. Developers with direct access to production had rolled them out.
7 days of partial outage later they covered this up by hiding parts of the app until they could straighten things out.
> The most common engineering mistakes can be fixed with more money.
That applies to pretty much all mistakes.
> Typically some other factors is what makes or breaks a company.
Tech is still pretty much at the top of that list, especially for "tech" companies.
Also Netflix didn't have a head start because of the quality of their streaming tech. They had a head start because of the business decision to invest in streaming before their competitors did. That early decision led to the quality of their library which is what attracted customers which is what allowed them to reinvest in their tech.
There are plenty of other examples. Twitter and Reddit come to mind as companies that had seemingly awful technology that would regularly fail. They succeeded through that. Companies like Facebook have certainly produced some good tech, but almost all of that has come after the company became a success. That is similar to Microsoft who got their big start selling someone else's tech.
- Choosing load-bearing tech that you're unsure will meet 100% of your needs based on hype. For example there are tons of companies with marketing websites that for one reason or another can't have useful user analytics software attached or run A/B tests.
- Letting the engineers define the entire product. This leaves you with a "perfect" solution which then can't be explained let alone sold because the customer/user perspective was not properly considered. I've seen more than one innovative (and desperately needed) startup with patented tech fail this way despite having a groundbreaking solution at hand. Product design matters.
- Dividing your org into "good" teams and "bad" teams by funneling productive engineers towards important problems and not redistributing them once those problems are solved. This "good" team eventually spends all their time fixing the broken parts of the system that had been relegated to the "bad" teams (because they are so degraded that they are now the most important problems). This then causes those "good" engineers to quit because they don't like pure maintenance work, and the resulting rapid loss of knowledge cripples the business.
- Wasting large quantities of dev hours on things that won't ever make your cost of labor back. Obvious examples include companies that spend more to support IE than they bring in from IE users (I've observed this regularly doubling implementation times on a per-task basis).
- Native code avoidance. Everybody I know that has spent > 2y on a React Native project eventually switches to native code and wishes they had started that way. This is a sample size of 10+ real $MM projects. I've seen the same for many Electron-style apps. The resulting "stop the presses" rewrite is almost always started too late to save the day thanks to simple sunk cost fallacy.
... the list goes on an on. Statistics don't lie, the road to failure is wide and welcoming ;-)
IMHO the refactoring sounded very reasonable since the old code base was unmanageable. But there was a culture of extreme pressure from the CTO, making everyone haste through tasks preventing everyone to do "proper engineering". At the time the CTO was in some sort of permanent absence already. Afterwards the team lead left because of burnout, I also left and later on I heard they closed down the company.
Actually I've seen similar destructive refactorings at other places. At one they also had a lot of subtle problems leading to many user complaints. It could be fixed but by then it was already too late.
IMHO refactorings are great but it's always necessary to keep regressions in check all the time and really understand the design decisions of existing code.
How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.
(If 1 account gets compromised the other is still available)
Perhaps 2nd level is by using cloud specific features - such as AWS S3 Object locks (even owner can't delete the object till the retention period).
my 2cents
The problems stated are bad, but did something actually happen as a result, like a rogue script deleting things and the company no longer in business?
Never bet the future of your company on metrics that are entirely out of your control.
The business justification seemly makes sense to have a single development train. But the execution was horrible. Bugs, horrible performance. Inconsistent features copied over from the various native apps, etc.
It was released as a major upgrade, but it was almost alpha grade in reality.
That org probably saved a lot in engineering, but the customer backlash was immense and likely cost them more.
(It was a very popular and beloved note taking app. that tried to become an all-in-one day planner/calendar/todo/note platform).
The lesson is that, if you are just planning to make an MVP make sure you have your scaling figured out. The blue ocean gets red faster than you can imagine.
After a while, business stagnation from constant pivots and new initiatives resulted in attrition and a downward spiral of budget cuts and reduced morale. Attrition and knowledge loss caused velocity to drop, which caused more of the team to leave, which made the business stagnation worse, which caused morale to go down more and budgets further cut, etc. Eventually the company couldn't recruit the same level of talent to replace people who left, hiring standards dropped dramatically, and it was now impossible to pay off tech debt or even really run the tarball reliably anymore (a Ruby monolith running an ancient, unsupported version of Rails with a million security holes and bugs).
Engineering leadership made a mass exodus, the few people that were left ended up on a death watch as the move to an outsourced engineering org from India was implemented on the way towards a full migration to a 3rd party vendor platform. Software engineering was completely eliminated from the company and the lesson the company leadership took from this was that "we never should have built the platform in the first place" along with a dose of "external business factors outside of our control caused the decline in revenue, forcing us to make hard decisions".
Lots of issues historically, and recently for that.
This is a mistake because:
1. A lot of the data at big orgs is garbage or only understandable within a certain context by specific people. 2. Internal politics within lots of organizations prevents access to this data. 3. The AI cannot just figure it out. You need tons of humans in the mix which brings you back to #1 and #2.
It lets the right people get access to, and make sense of, their companies shit data.
Intelligence analysts like to use Communities of Interest "COIs" to keep track of things. These are kind of like wikis. A COI on the Middle East may have groups like Al-Qaida and ISIS, and people like MBS, and countries like Saudi Arabia and links between them. They can also have hierarchies, like "planes" split into "prop" and "jet" which in turn splits into "military" and "civilian," think decision trees.
One problem with these is they can take a while stand up. COIs need at least a few hundred objects in them before they can be useful. Palantir came along and said, "well, we can help stand them up with our AI."
They did their work, apparently didn't properly test it and just did things in production (maybe SOCOM doesn't have a dev environment, who knows?), and when they pushed run it created hundreds of thousands of objects throughout the COIs, with levels in the hierarchy that didn't make any sense. Like, imagine "747" falls under "Helicopter" which falls under "Cessna" which is under the "Nation" category.
It took weeks for SOCOM to rebuild things. I'm not sure if they had problems with backups or maybe they didn't properly have things backed up.
Apple seems to have jumped the ship successfully by switching to ARM, their keyboard and OS are still unusable though. Wonder how their China adventure will turn out.
Intel could be bought by Nvidia or AMD, but since the government invested so much into their backdoors there, they will be kept alive. But no chance that their architectural problems can be solved at all.
Boing is unfixable since its unfriendly McDonald-Douglas management takeover and technical decline since. Now even their stable flagship product line fell down in a straight line.
Maybe the first idea hit a niche that wasn't being satisfied by the market, and the second attempt tried to break into a heavily saturated market. Maybe they just lucked out by being in the right place at the right time, or having the right connection that could bring in a multi-million dollar contract, or...
From a broader perspective, I would say that the biggest and most common mistake that people make (engineers too!) is not to spend time examining the hows and whys of the success that you've had so far. Were you successful because you had brilliant ideas, or was it because you had a mediocre idea that filled an underserved niche market? Were you successful because you used Postgres or Ruby or Kafka or Elasticsearch? Were you successful because you created a culture of innovation and learning and team players? Or were you successful because you happened upon a fantastic solution for the specific problem at hand but can't generalize it to larger problems?
If you don't know why you were successful in the first place, it's hard to continue to be successful.
TL;DR: lack of introspection and evaluation of success criteria over time
2) Solving theoretical scaling problems with every possible technology du-jour.