Hacker News Reader

Comment by jpatokal | Hacker News Reader

Just another reminder of how systems that you'd think are rock solid often aren't.

In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and then editing the configs by hand. Didn't take too long until a typo in a chmod command took down the whole thing (= the node couldn't take down a network interface anymore, so failover stopped working), and they spent several weeks flying in people from all over the planet to debug it... and they still didn't learn their lesson!

View on HN

I heard similar stories from a friend working for a big telco.

The other day he was describing a strange bug which was triggering in the field of a large telco. It was strange because it used to get triggered after exactly 85 days of deployment.

It turned out to be a debug script that was pinging a development server and timing out (because the development server was not accessible from the field). The series of retrials and timeouts totaled to exactly 85 days, after which the rest of the script would activate and uninstall crucial dependencies!

Legacy telcos are the epitome of large big institutions where many of the best talents leave. After just a few years of working with them, I couldn't believe they could get a dial tone. How did they produce so much great R&D?

wining and dining college grads and telling them all sorts of great stories then laying the workload on them until they burn out or quit.

> How did they produce so much great R&D?

By having negotiated profit margins. Bell Labs was meant to soak up excess money. What we are now nostalgic for would never have happened without an interval during which a telco was considered a "natural monopoly."

But didn't Bell Labs have the telco mentality? Or how did they protect themselves from it?

Reminds me of a colleague who RDPed into each of our 140 subsidiaries to change a config file. He had a list of servers on his desk and ticked off every server. Took him the whole day to apply the changes.

Two words: job security.

(Also: lots of folks really don't enjoy learning new stuff. Or new ways of working. No, they really don't. Even if the new techniques are vastly better and more efficient. Put this down to a human cognitive bias favouring the tried-and-trusted over new-and-untested. There's a lot to be said for that when you're a neolithic hunter-gatherer or an iron-age peasant -- if you try something new and it fails, you maybe get to watch your family starve next spring -- but it's a bit less useful as a rule of thumb in the data centre.)

Totally true. Even when the path is seamless, initial costs are nearly zilch... learning new things and applying them is anathema to many devs.

You can get blamed for changing something! The power if inertia....

I did this with nagios configs in my first job :(

When the service guy in a former job quit, the databases at the customers' production servers started to be corrupted. It appeared that he had manually logged into the database at every single customer regularly and fixed the problem on site. I don't know if had told the responsible developers (I worked on a different product), but after it was apparent that the system was not stable.

Ha, probably no one ever got fired for manually SSHing into each machine. It seems like a joke, but it really isn't.

They can't fire that person. He's the only one who knows the state of the servers.

Considering that I've watched a new automated deployment process nuke an entire network due to a mistake in the config... Sometimes slow and plodding and potentially losing a node (as opposed to 100 nodes) is seen as preferable to the PTBs.

Of course intelligent, incremental deployment works too, but let's not confuse those poor suits...

Is that job security?

What about SSHing into each machine not manually, but automatically with OKing all the changes before reloading the configuration?

SSH is great at least in the sense that it's configuration-format-agnostic, mostly-OS-agnostic, client-side-automation-agnostic, and many other things like that.

We often get frustrated with strict change management processes, excruciating verification etc .. but there is a reason why the most mature (not always the biggest..) operators have these processes in place.

the guy responsible for deployment doesn't even know the repository version that is running in production... I'm really tired of fight against this jerks...

Reply on news.ycombinator.com