Hacker News Reader

And then the devs moved on to something greater, the support was offshored and nobody could get the software to work as they wanted any more. 5 years later someone has the great idea to empower each plant by having different versions and the cycle continues.

It’s impressive that this worked. In my experience this doesn’t work because there’s so much “process” built around the tool to handle tons of undocumented edge cases that deploying the new tool just doesn’t work so it’s extra hard to get it up and running successfully.

The key to being successful at something like this, whether it's internal LOB software or a B2B enterprise product, is to provide a way for users to handle any exceptions out of band. If you try to code for every process exception you will (a) never go live and (b) create a horrendous monstrosity of a system. But if you can find the right spots to put in manual hooks then the users can be empowered to solve their own process exceptions.

An easy example is if you have an application that does a bunch of different calculations based on random rules and data sets. Instead of trying to code every calculation and random exception that the users throw at you, you start with the top 80% (or 50% or whatever) and give them a robust way to add a "manual calculation". Then tell them to handle their own exceptions by doing whatever crazy crap they need to do and just stick the result in the system.

It's difficult because if you're not familiar with the industry then you won't even be able to imagine the places where exceptions can occur. And the users will complain about having to do stuff manually. If you automate 80% of what they are doing right now and they still have to do 20% manually, stuff that they are doing manually right now, that manual work suddenly becomes a critical blocker that prevents the whole system from being useful. So you need someone with a strong personality who will not just accept that whatever the users are saying is right.

What a great summary, I could not agree harder.

Some of the worst turmoil happens when suddenly 'business as usual' for customers/users you didn't recognize the full value of disappears, and the front line people have to say things like "the new system doesn't let us do that any more".

You're absolutely right that they'll complain and drag their feet no matter what and that rock-solid leadership is needed to overcome this, but if you at least give them a way to do all the good-for-business stuff they did before in a similar time frame (with an overall faster/better system in the long run, of course), the majority of the sane ones will eventually come around.

> Tell them there was no going back, no push-back would be permitted, and have the CEO in the room to confirm this.

I'm sure this is great when it works, but it's also reminding me of things like the TSB switchover disaster which lost them hundreds of millions of pounds and a lot of customers when their systems were down for a week. Apparently rollback wasn't planned for.

"No pushback" also means "do not tell us how this is going to go wrong, we aren't going to listen".

Or, be diligently prepared for it to go right. After one or two regions succeed, you've got a pretty good idea of how the rest will go.

But what about training and processes?

The tech part is easy, relatively speaking.

The hard part is each plant can have hundreds of workers whose entire jobs depend on the way things are set up. It can be hugely expensive, time-consuming and disrupting for months to try to replace processes with new ones, particularly if you expect things to continue running without interruption. It's one of those rebuild-the-airplane-while-flying-it situations.

Maybe this was a uniquely lucky situation, because it doesn't sound like a generalizable magic-bullet to me.

It would definitely be interesting to hear the other side of this anecdote. My guess is that there's a lot of workarounds happening at the plant level that are being hidden.

The plant now tracks each individual product, line, customer, contract in realtime with complete transparency to the parent company. They swipe barcodes and use online tools. Their processes were streamlined and paper removed.

It wasn't painless for everybody. In each region some good ol boy would resign the week before changeover. When they'd come in to do inventory some room or shed would be locked and nobody would have the key. They'd break the lock and whattaya know, the room would be empty. Supposed to be full of inventory.

>The plant now tracks each individual product, line, customer, contract in realtime with complete transparency to the parent company.

I'm sure that's what the parent company thinks is happening. Whether that's what actually is happening at the plant level is another question. So is whether the improvement came with a hit to productivity or any other hidden cost. Obviously I know nothing specific about the company but speaking generally I've seen a lot of work arounds and inefficiencies in my day.

Yes, every single item on every single line is tracked. Nothing (can) get sold without being in the system.

Doubters abound. Even now a decade later, folks meet this story with disbelief and doubt.

>Yes, every single item on every single line is tracked. Nothing (can) get sold without being in the system.

Worker: "Steve from Acme Co. needs a doohickey with a widget today but we don't have a SKU for that"

Boss: "Corporate takes 2 days to issue SKUs so just mark it as a doohickey and charge him 10% more for the widget"

Or maybe the customer has to go somewhere else because their turn around time can't be met.

>Doubters abound. Even now a decade later, folks meet this story with disbelief and doubt.

Because it's not our first time at the rodeo. If massive IT projects were this easy we'd all be doing it like that. They're generally not. Steam rolling the end user may have worked this time but other times it's ground the business to a halt because those individual plant level differences were there for a reason.

They're massive because they're made to be that way. Technology transition takes will and backbone. If that's missing, there's where $25M gets spent down the drain.

I agree they generally don't go well. Because they're run by weak Directors and VPs who are 2nd-guessing IT and getting in the way.

Plant differences are all well and good. But so is "IT not collapsing under 20 years of cruft". It costs something to bring all the plants in line. But it also profits e.g. they're still in business after 10 years.

Here's how the money part went. Once the tracking went online, they realized (region by region) they were losing $1M per month. 2nd month: losing $0.5M. Third month: break even. 4th month: 1M in the green.

Why? Because they now knew (for the first time) what the hell they were selling, how much to whom, and for what price. The VPs quickly became totally addicting to this instant-knowledge web API and used it to turn the business around.

I was a consultant and ran IT security years ago at a company that was bought in a hostile takeover for ~$1B which was a significant premium in order to get some expertise for a special market they didn't have in house. One of the first things the purchaser did was tell the acquired company that they needed to change all their SKUs to match the corporate standard. At least some management at the purchaser knew that the acquired company used SKUs to track batches so they could detect, track and troubleshoot manufacturing and design defects, but apparently thought that standardization was more important. That was far from their only bad decision but within 5 years they'd run their 1B investment down to just the residual value of the IP about 100M. Fitting things into a procrustean bed (standardization) isn't always a good business strategy and inventory management in particular can be more complicated than it looks.

I used to work in retail at Circuit City. Our inventory was a mess, so corporate hired a company to fix that. At 4 AM a bus rolled up with people that spent all morning scanning items. We had to monitor them for theft and such (yes we saw several attempts).

They did an awful job. For example they scanned an item and then saw there were 10 items behind it and just multiplied the first item by 10 on the scanner (think of thumbdrives where the first one is a 512 MB drive and towards the back there may be a 256 MB drive). Afterwards, our inventory was worse than before. Corporate of course couldn't know this. I know your story is different, but they could still be getting duped.

I've seen this before with customer satisfaction. In data collection efforts driven by executives you can easily get bad results because they have zero visibility into the data collection process and its abuses and problems. I call it garbage in, gospel out.

They had all those problems, and solved them by being prepared. Training happened before changeover with a separate team moving from region to region.

Because something is difficult doesn't mean it is impossible, or shouldn't be done. The critical part is deciding you are going to do it, and not go back.

I did the same thing a few years ago in a healthcare place about faxing. I just issued a decree that we were done with paper fax. One my my team had identified a great product, my team ran with integration, we staged everything, tested, did training, then on rollout day we switch over all 45 locations, and terminated analog lines in each location (unplugged them and took the cords if we had to). A few people bitched but most people loved it. Sometimes you have to be the benevolent dictator.

Call me a pessimist, but all went well for a couple of weeks until they all realized they needed reports that don't exist in the new system and all the users individually realized the new system didn't enable the thousands of workflows that people had adapted their old systems to do that the managers didn't know about.

By this time your sister had hopefully moved sideways with "another successful migration" added to her resume.

Nope. Still running after a decade; still selling tires. Still adapting to requests for reports and workflow issues, as is possible with web-based tools. Because they are so much better than the old C-coded static tools and static reports running on a PC (not to mention paper). That's why we use web tools after all.

That's not at all like any of the "big bang" migrations that I've been involved in unfortunately.

I was reading the article thinking that I would take an approach just like that. I don't think there is any other way to ensure success.

It requires you to have complete and unwavering backing from top management even when some of the guys whose toes you're stepping on haven been golf buddies with said top management for 20 years.

Sounds like an unrealistic expectation.

It also depends on you having a crack IT team who can get it right the first time. No months of stuff crashing or having a horrible interface that impedes work or being intolerably slow. You have to be confident enough to push back against the exec who says "oh, we'll just hire $massive_consultancy_firm to do it" and takes years to roll out a half baked hackjob on top of their stupidly expensive CMS.

She insisted on a hand-picked team

    > ...No months of stuff crashing or having a horrible interface that impedes work or being intolerably slow.

Dude, that's the not a temporary transition state. That is normal _all_ _the_ _time_ for much of enterprise systems. The crashing might stop, eventually, but the byzantine intractable complexity maintained by a small army of information-hording corporate battle-axes... that stays forever, regardless of whether you go to a single "web-api" or not.

That sort of nepotism can be a problem. It's why Amazon's search sucks so bad. In the book 'The Everything Store', it tells the story of when a team at Amazon responsible for something else saw how bad search was and decided to fix it. So they built a prototype with Elasticsearch and whatnot. It worked great. The dude at the head of the search team, good friends with Bezos, got all territorial and angry. Bezos solution? Have a test between the two solutions to determine which was best. Bezos idiocy? He let the head of the search team decide who won. So, no surprise, the head of the search team said the search team won. So the improved search was shut down and the garbage search we all have to deal with regularly got to stay.

Once you've got that kind of thing going on, only retirements and deaths can effect good change at an organization.

It is, in the sense that I wouldn't expect it to happen, at least to me. But it's not unrealistic that some CEO would actually listen to you about the matter and put his foot down to make it happen. These are not matters difficult to comprenhend, my guess is that people use to take IT stuff for granted and that's it. I've also known guys who didn't give a fuck when they saw upper management screw up tech-wise so as not to get into arguments. That also cripples healthy feedback.

This sounds amazing. How did it turn out?

Still running a decade later.

Of course, now they've lost the people who designed that system, and are hard up to keep it updated. Like most large corporations, they 'outsourced' IT years ago. Which means, they lost their IP for their business processes to a revolving door of contractors.

My sister is retired, but predicts the collapse of the US business sector due to critical process failures within a few years. Its a pretty dismal outlook.

Can you go into any more detail about that or have any related resources about such a potential collapse?

https://www.youtube.com/watch?v=pW-SOdj4Kkk seems similar (somewhat melodramatic title, perhaps)

What would the "collapse of the US business sector" look like?

Can't sell product; can't manufacture or source parts; schedules start extending out to infinity?