I love these kinds of war stories of incredibly capable people. That was great planning for the unknowns.
So that is well known. What's not well known is this: there are pockets of brilliant people elsewhere, including Bentonville, Arkansas.
Especially in the years running up to 2000, I had the pleasure and honor of working with some of the most talented people in the world, even though we were all looked down on by many people on the coasts.
At that scale, everything had to be almost perfectly automated.
And, to be clear, this was not a static, boring network. Stores, home offices, distribution centers, you name it, were being added, moved, expanded, deallocated 24 hours a day, all over the world.
And the device configs were far from cookie cutter. An eye-wateringly complex heuristic was required to dynamically create router, switch and access point config that would allow a given store to function.
And, one quarter in the early 2000s, we (Network Engineering) achieved six sigma (99.9999%) global uptime.
Things did start going downhill around 2004 or so, a few years after Kevin Turner became CIO.
Ill have to look into kevin turner though, i dont know who that is....
Anything you’d care to share/point at?
I don't have much time right now, but I'll share a moment when I saw him interacting with some people during a crisis.
We got hit really hard with https://en.wikipedia.org/wiki/Code_Red_(computer_worm) even though only a minority of our systems were running Windows at the time. There were, at the time, a pair of Windows servers in each store, and each store was connected via a 56k frame relay circuit to the home office, along with a modem connected to a standard phone line for dial backup in case that was down.
Anyway, Code Red rapidly spread through all of these servers, and together they created enough traffic to basically bring down the frame relay network. We were, at the time, slightly over-subscribed at the T1 termination points, so if most of the stores on a termination leg maxed out their individual 56k links at the same time, the whole leg would be effectively down.
(Apologies for the unexpected dive into old networking stuff...it kind of just came out. :)
So most of our stores were down hard, meaning that most card transactions would not work. There was a facility to allow credit cards to go through without verification with a relatively low charge ceiling. But especially back then most transactions were NOT credit, and so weren't working.
This was a bad, bad situation, and it persisted for a number of days. It might have garnered some national news attention.
There were a number of war rooms going on. My team was in one, and we were working towards a solution by using a transient dial backup connection to reach into the store router and apply a draconian ACL.
ANYway, back to Kevin Turner. We saw him talking to a group of VPs and directors outside of our war room. He dismissed most of them, except for one, with whom he continued to have a very heated conversation.
Turner was definitely the aggressor; he got closer and closer to this person, got louder and louder. He ended up backing him against a wall and shouting in his face for a few minutes before walking away, jabbing his finger in his chest.
Just one story. We didn't like KT very much.
I had to recover the password on one when i inherited that network. I will never forget that password: Feet4Monkey.
Ill also never forget my ix.netcom.com password that was initially assigned to me....
Nice, that was a blast from the past. I used netcom.com for a few years in the early 1990s. Good times.
May I ask which one? Just idle curiosity.
I once worked in Point of Sale (POS) at Darden and several POS vendors and consulting companies. Even 25 years ago, your average fast food or mid-range (e.g. Red Lobster) restaurant had a network of ~10 or more computers running various POS, inventory, ordering, scheduling, and other processes.
I would argue the caliber of people I worked with there, from no name universities, or known names, but not equivalent to many people (Georgia Tech vs Berkeley, say), was generally better there. It of course varied person to person, but the teams were generally stronger, senior devs were more senior, juniors tended to be hungrier to learn, etc. Maybe it was just the lack of the FAANGs sucking up the best people.
I was like "WTF? They've probably never heard of Waterloo either!"
(For anyone unfamiliar, these may not be household names, but they are definitely top tier CS schools.)
Edit: we also hired a few idiots out of MIT. Remember one instance when an intern from MIT sent an email to a wide audience something to the effect "your API is breaking my code, fix your API". This was in C++. They were asking for something that didnt exist and then blindly dereferenced the nonexistent first element causing a segfault. My response was to the effect of "youre dereferncing an empty vector. API is fine. Fix your fucking code."
The interns original email was sent to just my team, but CC'd nearly all of front office. In my reply I added all of backoffice so they were aware of the idiot. Never heard from them again...
The coasts tend to have clusters of many top caliber technical colleges.
It's funny, because for the first 2/3 of my tenure there, WalMart Information Systems Division did move extremely fast....and yet almost never broke anything.
It was an amazing thing to see.
(You're almost certainly already aware of http://old.reddit.com/r/talesfromtechsupport - but just in case. And hint hint.)
I was aware of /r/talesfromtechsupport but haven't frequented it much. I will take another look.
My contacts over the past 11 years have said that the division has continued its slide into bullshit.
Given how fantastic things were, it makes me very sad.
It is hard to contemplate just how destructive the "cost center" / "profit center" distinction has turned out to be. For us, though, it means we had better stay the hell out of cost centers.
A business performing the duties of another's cost center, where costs may be amortized over multiple customers, can shift identically same tasks into a profit center.
What a lot of Silicon Valley people don't get is that there are people just as smart as them everywhere.
Walmart has had a pretty savvy tech presence in SV for decades...