Hacker News Reader

Comment by freeqaz | Hacker News Reader

That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least.

I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options.

Blows my mind, but that's reality (apparently)!

View on HN

> _can't reboot their servers_ because they simply don't know how to bring the app back up

A hundred years later IT priests will demand human sacrifices to appease the server gods

So firms will need Ceremoniously Executed Offerings, eh?

Server uptime used to be a point of pride. Now people try to shut them down as often as they can. Funny how things change.

My server had 80kbyte bandwidth and doing very little in a much slower time.

Kernel patches were not on top of my head 15 years ago.

We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.

Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)

To be fair, it also used to take sometimes an hour+ to reinit a system. All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...

That seems to go hand-in-hand with the "Cattle not pets" concept, which I'm a huge fan of.

I would run screaming from such an organization.

I run screaming towards such organizations. Higher bill rates!

This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!

I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful.

Reality is a messy thing.

Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.

In your mind, do hosted wikis and KBs like Stack Exchange, Wikimedia or Confluence address the discovery problem? If not, is there anything close?

Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc.

We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.

In cases like these, I'm a strong proponent of turning it off and seeing who yells.

In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again.

There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"

future reference: next time just firewall the network and see what blows up rofl.

A bit of network analysis to identify services that are being called before turning things off would probably be wise, too.

The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.

the pyramids of our time

Reply on news.ycombinator.com