I would love to hear more about this. I'm guessing that's a cost of at least $4M? How was this approved? How did they allow it to continue for four years?
The OP's assertion that it was primarily due to slavish devotion to InfoSec's insistence of getting off PHP-based MediaWiki, is incomplete and misleading. The team itself opted into the migration, and by their own admission in their own postmortem document, they wanted the opportunity to get off MediaWiki (killing two birds with one stone, both as a technology upgrade and as an InfoSec compliance thing), but they fundamentally vastly underappreciated the scale and complexity of the project (some top line #'s: ~4M total documents, 1.7M unique pages visited daily, 600K unique MAU, etc.). What probably doesn't come through in talking about this, is that the wiki is far from just a simple collection of text-based wiki pages in the standard sense; it serves more like something akin to an unholy abomination of a Wordpress-like platform with endless amounts of plugins and macros and templates, and gives you just enough programmability to be dangerous.
The primary reason it took 4+ years, was essentially down to going through multiple rounds of failed migration attempts, including failed attempts at producing automatic translations of wiki's from one platform to another (which proved to be incredibly complex, due to the nature of how customizable MediaWiki was and how many teams had gone and done so many interesting/advanced things with it, which was great... except it made automatic translation/migration near-impossible).
I don't disagree that InfoSec was a driving force behind the project, and may have given it an unhelpful level of urgency (i.e. pushing for the team to make early decisions/plans too quickly, which anyways resulted in this banned PHP thing lasting 4+ years, seems counterproductive). But even if InfoSec was not a factor, the team knew that staying on MediaWiki was not sustainable, and some kind of migration would've happened eventually.
So I guess what I'm saying is, IMO the perceived pressure from the InfoSec mandate was poorly managed by this team, and ended up exacerbating the problem, but fundamentally the choice to migrate to a different wiki platforms was something the team wanted to do (just maybe on a different schedule or with different plans/priorities, in an alternate universe). Migrating off MediaWiki was not done against their will.
https://twitter.com/patio11/status/1255371954443505665
It's always satisfying to read reports from the trenches demonstrating the exact opposite.
So while they do their best to not have _every_ initiative go this way, large companies - even the mythical FAANGs - will inevitably have moments like this wiki debacle, and they need to be able to absorb them without material impact on the bottom line or having to fire a bunch of people for trying and failing.
I would love to have empirical data about this whole business. I have no idea how you would collect it.
- Production operator: filling boxes, washing tanks, hook up and unload raw material delivery tankers
- Maintenance Technician: Overhaul and install new equipment, maintenance on equipment.
- CPS ID Laboratory Chemist: Ensure all raw and packaging materials and finished product comply with Company technical specifications including process water, ensure ingredients have all pertinent documentation before their final release.
Meanwhile, you look at Uber's jobs page and you see this (descriptions are quoting from their page)
- Business Development: We strengthen Uber’s position as the world’s leading mobility platform through creative and mutually beneficial commercial partnerships that unlock outsized value today and for many years to come.
- Data Science & Analytics: We transform data into magical experiences.
- Product: We create the vision for the future of urban mobility: deeply understanding our customers to solve their transportation needs with innovative technology.
Sure, Coca-Cola probably has some fat that can be trimmed. But at the end of the day you can't fire the guy testing if the coke water is poisonous, you can't do without someone to load boxes, you can't do without the mechanic. Uber, however, can clearly use substantially fewer people creating "creative partnerships" or "transforming magical experiences" or "creating the future of urban mobility".
Uber has around 30,000 employees to run a ridesharing business. (Didi, which does twice as many rides per day, has 10k.)
Coca-Cola produces 110 billion bottles and 3,900 different beverages around the world. They own their anchor bottler in North America and they produce syrup for pretty much every country in the world! They're so big they used to own Columbia Pictures.
I think Coca-Cola is a pretty bad example.
As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed.
Maybe I'm being dense, but this still doesn't register for me. If 4+ calendar years of work for a single team continues with no noticeable progress, then there's an organizational issue, no matter the size of the company.
As far as I could tell, the "reason" given by the OP for this waste was attributed to Amazon's scale. Could you explain how company size and broken corporate structure are correlated? Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure?
That there is an organizational issue. Of course there is an organizational issue. There will always be organizational issues.
> Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure?
Oh, it's definitely an organizational issue here, that's just not the main point. If you're willing to call this a "broken corporate structure," then you should revise your opinion of every company with more than 50 employees downward sharply. This is a normal corporate structure, not a broken one.
> Could you explain how company size and broken corporate structure are correlated?
When there are enough people, you will always end up with conflicting incentives somewhere. A larger organization will have more of these. In this case, it seems to be a clear case of competing priorities: IT wants to secure systems, and developers want to be maximally productive. I'm sure some folks in IT view Amazon paying ~$7M as a fair cost to remove known vulnerable tools from their infrastructure.
And after all the internal calculations were complete (which factor in way more than technical effort by a small dev team), they intentionally stayed the course.
This is probably an extreme example because an internal wiki that is used company-wide will have an incredibly complex and hairy stakeholder matrix (both business and technical), which means consensus is very challenging.
At some point the cost of the meetings required to work out a solution (both in time spent, and in time not spent on core work by the stakeholders) probably exceeds the cost of just letting a small team brute-force their way through a bad solution.
Cost here isn't just money, it's focus, morale, disruption, conflict, risk, etc..
Keep in mind that you have the benefit of hindsight, and the comfort of looking at it from the outside, with a very incomplete picture of all the constraints and pressures at play.
It is just not true that someone always runs the math and then decides by weighting pros and cons.
But having worked in and around these kinds of teams and companies for most of my 25-year career, I can tell you that it's "often", in my opinion/experience.
This viewpoint doesn't really match up with how enterprises operate.
It's possible to function at amazon's scale without meticulous accounting and financial operations. The reason for this is that is extremely easy for wasteful spending to propagate throughout the enterprise if unchecked.
Waste only appears after a project fails...it's easy to point out waste after the fact, but it's downright impossible to call it out as it's happening.
Getting documentation off google docs on the other hand seems impossible from a behavior standpoint :(
I can tell you that Amazon is pretty great in most ways, but they have a lot of really old school tools. For example, they still use majordomo to manage hundreds of thousands of email lists. They've customized and extended it in so many ways that it probably looks nothing like open source majordomo, but at it's core, it's still majordomo. They just wrapped 1990s majordomo CLI tool with a 1990s static HTML page that authenticates you with oauth and lets you create and manage email distribution lists.
So many tools at Amazon are like that. But they actually function amazingly well once you adapt to their 1990s->early 2000s quirks.
One comment I'll make is this: every environment will have preferred "paved" roads - use X language/toolchain and Y relational database, Z cloud/metal provider. Ultimately these details, other than X and a subset of Y, shouldn't matter if you build the right abstractions. And by "the right abstractions" I mean you should just be able to declare something like this: relational_database: mydb: size: 100GB readers: - auditservice readwriters: - application1 - application2
("how hard can it be" should be on your Words That Prefigure Expensive Disasters list. It's the business equivalent of "hold my beer and watch this")
I mean, it gave a lot of people something to do (and get paid to do) for 4 years. There's your answer, most of the time.