Why are large companies so difficult to rescue (regarding internal technology)
smashcompany.com
smashcompany.com
Her method? Ruthlessly purge the region of the old data system and install the new (web-based API to a central, new data system). Investigate every regional difference and consolidate into one model.
Before deployment day, get all the regional Directors in one room and tell them it was going to happen. Tell them there was no going back, no push-back would be permitted, and have the CEO in the room to confirm this.
It all went well, with her team of a dozen professionals that did a surgical strike to each region. IT equipment installers, trainers. Even one lady who's job was to ferret out the hidden PC somebody was squirrelling away in a closet (there was always that person) and rip it out, take it to her car and lock it in the trunk.
All went well, except for one region. That regional Director had missed the big day, and tried (vainly) to push-back for all the familiar reasons: special circumstance, retraining was going to keep them down for too long, and on and on. 100 years of data systems gone in a year (including paper systems) and replaced with a web API.
Sounds like an unrealistic expectation.
> ...No months of stuff crashing or having a horrible interface that impedes work or being intolerably slow.
Once you've got that kind of thing going on, only retirements and deaths can effect good change at an organization.
An easy example is if you have an application that does a bunch of different calculations based on random rules and data sets. Instead of trying to code every calculation and random exception that the users throw at you, you start with the top 80% (or 50% or whatever) and give them a robust way to add a "manual calculation". Then tell them to handle their own exceptions by doing whatever crazy crap they need to do and just stick the result in the system.
It's difficult because if you're not familiar with the industry then you won't even be able to imagine the places where exceptions can occur. And the users will complain about having to do stuff manually. If you automate 80% of what they are doing right now and they still have to do 20% manually, stuff that they are doing manually right now, that manual work suddenly becomes a critical blocker that prevents the whole system from being useful. So you need someone with a strong personality who will not just accept that whatever the users are saying is right.
Some of the worst turmoil happens when suddenly 'business as usual' for customers/users you didn't recognize the full value of disappears, and the front line people have to say things like "the new system doesn't let us do that any more".
You're absolutely right that they'll complain and drag their feet no matter what and that rock-solid leadership is needed to overcome this, but if you at least give them a way to do all the good-for-business stuff they did before in a similar time frame (with an overall faster/better system in the long run, of course), the majority of the sane ones will eventually come around.
The tech part is easy, relatively speaking.
The hard part is each plant can have hundreds of workers whose entire jobs depend on the way things are set up. It can be hugely expensive, time-consuming and disrupting for months to try to replace processes with new ones, particularly if you expect things to continue running without interruption. It's one of those rebuild-the-airplane-while-flying-it situations.
Maybe this was a uniquely lucky situation, because it doesn't sound like a generalizable magic-bullet to me.
It wasn't painless for everybody. In each region some good ol boy would resign the week before changeover. When they'd come in to do inventory some room or shed would be locked and nobody would have the key. They'd break the lock and whattaya know, the room would be empty. Supposed to be full of inventory.
I'm sure that's what the parent company thinks is happening. Whether that's what actually is happening at the plant level is another question. So is whether the improvement came with a hit to productivity or any other hidden cost. Obviously I know nothing specific about the company but speaking generally I've seen a lot of work arounds and inefficiencies in my day.
Doubters abound. Even now a decade later, folks meet this story with disbelief and doubt.
Worker: "Steve from Acme Co. needs a doohickey with a widget today but we don't have a SKU for that"
Boss: "Corporate takes 2 days to issue SKUs so just mark it as a doohickey and charge him 10% more for the widget"
Or maybe the customer has to go somewhere else because their turn around time can't be met.
>Doubters abound. Even now a decade later, folks meet this story with disbelief and doubt.
Because it's not our first time at the rodeo. If massive IT projects were this easy we'd all be doing it like that. They're generally not. Steam rolling the end user may have worked this time but other times it's ground the business to a halt because those individual plant level differences were there for a reason.
I agree they generally don't go well. Because they're run by weak Directors and VPs who are 2nd-guessing IT and getting in the way.
Plant differences are all well and good. But so is "IT not collapsing under 20 years of cruft". It costs something to bring all the plants in line. But it also profits e.g. they're still in business after 10 years.
Here's how the money part went. Once the tracking went online, they realized (region by region) they were losing $1M per month. 2nd month: losing $0.5M. Third month: break even. 4th month: 1M in the green.
Why? Because they now knew (for the first time) what the hell they were selling, how much to whom, and for what price. The VPs quickly became totally addicting to this instant-knowledge web API and used it to turn the business around.
They did an awful job. For example they scanned an item and then saw there were 10 items behind it and just multiplied the first item by 10 on the scanner (think of thumbdrives where the first one is a 512 MB drive and towards the back there may be a 256 MB drive). Afterwards, our inventory was worse than before. Corporate of course couldn't know this. I know your story is different, but they could still be getting duped.
Because something is difficult doesn't mean it is impossible, or shouldn't be done. The critical part is deciding you are going to do it, and not go back.
Of course, now they've lost the people who designed that system, and are hard up to keep it updated. Like most large corporations, they 'outsourced' IT years ago. Which means, they lost their IP for their business processes to a revolving door of contractors.
My sister is retired, but predicts the collapse of the US business sector due to critical process failures within a few years. Its a pretty dismal outlook.
I'm sure this is great when it works, but it's also reminding me of things like the TSB switchover disaster which lost them hundreds of millions of pounds and a lot of customers when their systems were down for a week. Apparently rollback wasn't planned for.
"No pushback" also means "do not tell us how this is going to go wrong, we aren't going to listen".
By this time your sister had hopefully moved sideways with "another successful migration" added to her resume.
> All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log.
This sounds crazy. I don't know any large companies that have successfully implemented it. This is basically arguing for a giant central database across the entire company. Good luck getting the 300 people necessary into a room and agreeing on a schema.
P.S. I don't know if he's using database as a shorthand for a service. If he is then you can ignore everything I've written.
Coming from a clunky, overused but underdesigned monolithic structure, the work to standardize inter-team/inter-department communication is really hard.
In general, the easiest thing is to overspecify those APIs and fight anyone who wants to simplify them. Complicated work looks good on a CV, after all.
So now you have hard-to-implement, full-of-cruft API designs. At that point, teams realize that the easiest thing is to work around them.
And off you go into splintered components which ignore the standard APIs as much as possible. Turns out that's much easier, and you deliver results with higher velocity!
From far enough away, you can just see "more standard APIs => higher velocity", so obviously you keep doing that, right?
An internal API is just as much a revenue driving product as an external one.
So a micro service is 2 pizzas.
Maybe for just a plain service you can have up to a a pizza party's worth of people.
But you can only have up to 10 nano services per sandwich before they become pico-services?
Technically they're harder to manage and often come at a high technical cost that you should never attempt to pay if you don't have to, but it's an effective way of managing the often extreme coordination costs that exist in large, dysfunctional companies that simply don't exist elsewhere.
I used to rail against microservices for years (working in startups) until I finally worked at a large dysfunctional corporation trying to run a lot of complex processes that spanned a lot of tightly coupled departments and then I started to see how they made sense.
It also gave me an appreciation of the problems Bezos recognized when he tried to decouple the organizational fiefdoms under his control.
At startups you don't need Conway's law, not enough people to be relevant.
At large companies software and people organizations tend to match or in the case you and parent are talking about need to match.
I'm willing to believe that things don't necessarily have to work this way, but I haven't seen it.
PS: I am not saying that’s my take from your post, just something I observe in general. Heck, even I use the term political to signal dysfunctional work. :/
I was more focused on saying "the furthest possible thing from a technical problem".
The only thing you need to do to fix this is run all the services on the same DB.
> This sounds crazy. I don't know any large companies that have successfully implemented it. This is basically arguing for a giant central database across the entire company. Good luck getting the 300 people necessary into a room and agreeing on a schema.
You don't need every service to use the same schema. You only need transactions that span all services. They can use any data schema they want. A single DB is only used for the ACID guarantees.
I dont think that will help. The fundamental problem is the amount of work necessary to begin producing results. Take the POS talking directly to the DB. What you want to build is a unified API but to do that you first need to write a new POS system and replace a bunch of physical devices. The problem is how do you convince the business folks to spend years and millions on something that will save/produce the company 0 dollars?
The path to do something like what's in the article is to first spend a year or so digging into code and finding every single issue like the POS one. Then you spend years fixing all of those issues. Then you spend more years actually building what you want.
As a consulting firm if you come in and bid $100 million and 7 years while everyone else does $30 million over 3, well, you lose. So thats what you bid. Who cares if it goes tits up? You collect your fat bonuses for 18 months and then move onto your next firm/position. The inevitable failure can be blamed on the person who takes over.
The issue of trust is also real, but I noticed that if you build a strong team with a strong "internal brand" you can drive a lot of the execution and technical decision making from there (the world used for these teams in enterprise is "CoE's" but I mean something a bit more substantial by that than what you usually read around). The thing is that in reality your CEO doesn't care about your technical implementation details. This is a blessing and a curse. The blessing part is that if you offer a team with a brand and a vision that has collected some internal trust from multiple parties over a couple of years, you might be able to take your own technical decisions independently from IT or other departments around the world.
As for the technical part of this, I am not an expert, but security, RBAC and friends are hard problems, I'd be surprised if you find a vendor that can do this for all your edge cases. I'd be for a fully internally built solution. But again, no expert.
A thing is a core competence if, and only if:
* It makes a difference to your customers.
* It is difficult for your competitors to replicate.
* It provides access to a wide range of markets.
Janitorial services, as the OP says, do not tick any of these boxes (unless you are a restaurant or something, in which case it ticks the first). Black and Decker's core competence is not selling cheap hand tools, it is making small electric motors. Think about it. https://en.wikipedia.org/wiki/Core_competency
IT frequently ticks all three: if your IT goes down your customers may know about it before you do. Your IT is going to be difficult for competitors to replicate (as long as you haven't outsourced the whole thing), and you can use that IT in lots of different markets.
However there are too many senior managers who have heard the words "core competence", not read the article, and assumed that since they are not running an IT company it follows that IT is not a core competence.
https://www.theregister.co.uk/2019/04/23/hertz_accenture_law...
-- Female CEO
-- 100+ years old
-- rentals
-- Has quietly bought up it's competition
Only thing which doesn't line up is employee numbers.
And they had Accenture do Angular app development.
Consolidation and “reuse” is often the problem, not the solution.
An "API" is the answer they are told they need - easy. Millions of dollars later after much time has past, the project is of course in trouble.
The author is asked if he knows anything about APIs. He says sure, they're easy. Just expose the database through an API. How hard can it be.
My prediction (guarantee) is that a lot more time and money will go down the drain before this API effort is finally abandoned. The issue is the chain of naive and poor decisions being made, which are exacerbated by the advice of the author.
Really? what advice is being given here that would exacerbate anything? From what I see, the author is carefully enumerating the problems which need to be carefully addressed before this kind of change can be made.
He is deliberately holding up the 'You just need an API, how hard can it be?' perspective as a naive one and looking at why it isn't that simple.
I found this an interesting and useful argument, particular the element about trust and loss of control
In the past I have worked on one of the Big car hire sites and in my opinion Accenture and Hertz where just incompetent.
Probably Non Culpable Incompetence for Hertz, Accenture not so much.
There seems to be a lack of translation to "what should actually be built." Management wants a unified API, but my first (and probably incorrect) idea would be to develop an ORM + set of functional libraries which can be implemented independently by each of the involved parties. That has its own risks, but getting everyone speaking the same language is a good place to start.
History is absolutely one part of it, but the other part is that there must be a benevolent dictator (person or small group/council) with the authority and courage to make technology decisions. Without that role, bikeshedding can often drain valuable inertia — sometimes fatally — from the decision-making process.
Large, mature organizations have a lot of inertia, and that inertia is difficult to overcome except by perhaps the most charismatic individuals. I’ve rarely seen a leader dictate or bully their way to effective, positive change. Resistance is extremely likely for various reasons, and it’s not likely to be in the form of vocal mutiny. Instead, it typically appears in the form of excuses or lethargy. A waiting game is played, usually with success, until the musical chairs shift the management once again.
Tef's "How do you cut a monolith in half?" is relevant:
https://programmingisterrible.com/post/162346490883/how-do-y...
I swear, there are so many exceptions and special cases embedded in a large company that it’s impossible to replace any one product without also overhauling the rest, oh and overhauling customer expectations, because if they’re used to a static website that takes 10s for each page load, then whatever replaces it better takes at least 10s to load.
Unfortunately getting the actual requirements after the 10 layers of indirection in the enterprise means you just hear the Product Owner say that the requirement is the page load should take 10 seconds.
By that point any justification has been long lost.
one solution I have seem similar to the story's problem was to adapt an edi like solution, there was one group whose entire job was to transform data from group to group and each group in turn had to meet the interchange formats. that can be a political fire storm in itself because the other truth of big companies is that there are many fiefdoms within it and some have a lot of pull.
For example the botched SEO changes ASOS made recently - that actually made the front pages of the financial press in the UK
This is my persona opinion and does not imply any connection with my employer.
Part of working with large systems is accepting that it's okay if some of the components suck as long as the objective is still being achieved.
The problem is that all pieces are interdependent. So you cannot just replace only a single part (of course that’s not always true, but often enough that it might as well be the rule).
The only way to get any traction is to get VP level people involved to prioritize which systems to work on. But some VP level people might see you reaching out to them as incompetence - "you can't prioritize your own work?! What did I hire you for!", etc.
At the end of the day it's really about the people in the org and how altruistic they are when considering the constraints of the IT dept.
The companies and institutions I've encountered all essentially say, "We want success but without change," and then wonder why their initiatives fail. The point about "Agile" being a euphemism for "trust," is huge. Startups can be farcically naive (or cynical to the point of evil) about how trust in organizations works.
C&C enterprise structures don't scale, and they become immensely vulnerable to challengers as a result. This article reminded me of the opportunity that large companies create, like stored potential energy in the form of opportunity costs. It's like a kingdom spread so widely that a small raiding army can feed itself on what these companies left undefended while planning campaigns.
How much of this article is true for the rest of the Fortune 500?
It's worth reading, although your feelings may get hurt if you're not already sold on functional programming. He's not very diplomatic.
Though I wonder, if large companies are routinely this inefficient (which matches my limited experience as well), how do they survive? Naively put, why isn’t every large company killed by a startup next week?
Because every other large company is as equally inefficient. When it's par for the course, it's not seen as an issue (or even seen at all). Efficiency is also usually not the metric that gets optimized towards - other dimensions such as predictability, reliability, longevity, consistency, stability tend to take precedence. Take the loyalty program vendor mentioned in the article - sure it's limiting their flexibility for what they want to do now, but it still exists. So depending on your viewpoint, that was a pretty solid choice in vendor.
Also, large companies have the scale to absorb the costs of their inefficiency with fairly minimal impact on their unit economics. And if a startup does pop up and gain traction with a much more efficient operational model, BigCo can write whatever check is necessary to gobble them up before it becomes a risk.
I think there are commonalities in pathologies. After all: humans are humans and large enough groups of people will exhibit regression to the mean.
But the exact details can vary so much that I am often reminded of Tolstoy's observation that:
> Happy families are all alike; every unhappy family is unhappy in its own way.
Making a big organization work well probably requires a bunch of highly-paid people to take on personal responsibility and make judgement calls where the buck stops with them. No-one wants to do this. It's personally risky and there's a kind of game everyone plays where they know everyone's avoiding this and it's considered fine as long as you fake Doing Manager Stuff and bring the Big Four in when you can't avoid making a decision, so you can blame them if something goes wrong. The same attitude infects the entire organization, unavoidably. If the big contracts are rolling in anyway (see above) there's little incentive to take personal risks.
It's an example of success breeding success, purely for its own sake.
If the CEO does not want to be a tech company then I’d investigate options for spinning out a service company to serve the rental market, ideally in collaboration with some competitors.
There are exceptions like... IBM!
Good grief, IBM are the Mongols. It all makes sense now.
Are you refering to the idealism, the guerilla dynamics, or how cool it looks on a t-shirt?
Near the end of the article the author mentions that 5 people can look each other in the eye and trust each other, while 11,000 can't. This suggested a question to me: Where does that stop? How many people do you have to have before a 'local culture' isn't enough and trust breaks down? I've read things in the past about Dunbar's Number which suggests this number is around 110. That seems to be about the number of people that individuals can 'keep in mind'. I have often wondered whether organizations might benefit from taking that limit into mind and structuring things so that no level or group would be permitted to grow beyond that. To grow larger, build a hierarchy where groups of about 100 choose a representative whose task is to know the concerns of everyone in the group and represent those concerns when discussing with reps from other groups, etc.
It can send emails or even order a person to make a phone call if necessary. Whatever "primitive" means the "Invented-here" API can abstract that functionality, so that the internal API devs can access and update the data from the external company.
I wish the author had explained this more. What is it about agile that requires more trust than anything else? Can't you write lies in any format equally easy?
And then there is Half-Arsed Agile: https://www.halfarsedagilemanifesto.org/
The author meant the former. Writing lies seems to point to the latter. Writing lies is not "valuing individuals and interactions".
Short term goals. Narrow focuses of executive team. Lack of technical executives. Brain drain.
I think the core logic of outsourcing is sound, but it's like many simple ideas: simple, but not easy. It's actually a restatement of economics 101 -- comparative advantage and gains from trade. Even if you are strictly better at doing everything yourself, it still makes sense to trade with others. At Pivotal we often discuss this as "the value line", the point in the stack where, we argue, it is more effective to delegate your technology problems to us (or -- this is how I feel personally -- to a comparable vendor, like Red Hat, over pure DIY) and to turn your own technology efforts towards the things that add value for your customers. Even if you could do a better job than any of the vendors, it might not make sense for you to do so.
That value line isn't static, of course, and it's differently located for each customer. And it remains necessary to retain enough inhouse expertise to sensibly assess vendor solutions. Enterprise software and software development is a complex "experience good". This isn't limited to software, it's true of many complex purchases -- I have a book in my collection called Industrial Megaprojects which contains a fascinating parade of such examples.
All that said, transformation is possible in my experience. The key is, as the article describes, that it takes a long time, a lot of work and a lot of money. It most of all requires enough time for cultural norms to resettle and for trust to solidify. I've worked with companies operated on a fear basis and basically, the best we could do was to create insulated bubbles of agility. We've worked with other companies which have more or less transformed themselves once we brought a spark. Sometimes we will be the true vanguard of a revolution, sometimes we will be the fad-of-the-week that the hardened veterans will shrug and mouth the slogans for while continuing with the actual way things get done. Sometimes, because many of our customers are so vast, we will be heroes to one division and villains in another.
The best that I and my peers can do is to help each client as honestly, empathetically and fully as we can to transform themselves. But, as the article points out, that is hard.
I think that 'Big Companies' need to deal with Ethics before trying to deal with Agile/Scrum. xD
> In terms of the best integration architecture, what seems to me the only long-term solution is something like the unified log architecture that Jay Kreps wrote about back in 2013. All incoming writes need to go into a centralized log, such as Kafka, and then from there the various databases can pull what they need, with each team making its own decisions about what it needs from that central log.
Then, from the linked Jon Kreps post, 2013 [0]:
> In order to allow horizontal scaling we chop up our log into partitions. Each partition is a totally ordered log, but there is no global ordering between partitions (other than perhaps some wall-clock time you might include in your messages). The assignment of the messages to a particular partition is controllable by the writer, with most users choosing to partition by some kind of key (e.g. user id). Partitioning allows log appends to occur without co-ordination between shards and allows the throughput of the system to scale linearly with the Kafka cluster size.
In practice, wouldn't the end "centralized log" be essentially a collection of the logs from each original/logical database, now managed by a specialized log team? To read the log, the interested consumer team would have to create the appropriate read-only replica of the original database, using the respective database technology / binaries / schemas. As such, isn't the overall system better described as "database replica as a service", as opposed to the finer grained implication of the "unified log as a service" of the article / Jon Kreps post?
I'm trying to figure out if the implication of such decisions are organizational, "who's responsible for the logs, who's responsible for the replicas", vs. architectural, "where does data bit X go, how does it maintain consistency with data bit Y, how do we read its current state?".
[0] https://engineering.linkedin.com/distributed-systems/log-wha...