Hacker News Reader

Comment by moe | Hacker News Reader

I love how he mentions "water damage" in the list of things that can happen to a server. That made me snicker... who doesn't hate those leaky datacenter roofs!

View on HN

At work, a server that I am responsible for the software on, but not the general IT-type management, was becoming increasingly flaky. It kept powering down at odd times, once managing to corrupt a MySQL database pretty badly. Shortly after a firmware update, it ceased coming up. I honestly did assume it was something in my software update, as updating this server was often the next step after QAing the software. Once I realized that I couldn't even ping the device (which I am 2000 miles from), I had to send in the local IT personnel to log in physically.

This is when they discovered that the device would no longer physically turn on. Next step is to de-rack the device, at which point some strange discoloration was discovered in the holes in the case. Next step was to open the case, where it was discovered that the entire machine was full of mold. The roof was leaking, and this was on the top of the rack directly under the leak. In Silicon Valley, leaky roofs can apparently take a while to discover, but they will eventually make their presence known.

Now, this was in a local office, albeit a relatively well-equipped on that holds in the hundreds of rack machines, not a colo. But you know what? Shit happens. And you don't really have a guarantee that shit won't happen just because someone has magically slapped the word "data center" on a building.

once managing to corrupt a MySQL database pretty badly.

Rare as a Himalayan Snowcock spotting!

How about a sprinkler system? Or the fire department? Or a flood? These sorts of things are actually more common than you may think.

See: http://www.youtube.com/watch?v=ttcQy3bCiiU also, http://www.datacenterknowledge.com/archives/2007/12/04/rains...

Yeah, I know it does happen occasionally. It just amused me that he listed it in first position, no less (and I'm sure he did it for that very reason ;) ).

As common as water damage may be, it is probably a rounding error compared to less spectacular failures such as dying disk drives and PSUs.

Regardless, that datacenter flood video is absolutely priceless. Thanks for that!

Hey, it happens. One company I worked at moved into a new office (after I left) near the top floor of a highrise. There was a crack in the roof under one of the building's cooling units, and their computer room was near the plenum. During a big rain storm, water leaked down from the roof, through the plenum, ran along the ceiling and dripped all over the racks. Oddly, many of the computers remained running despite having water pooled inside of them.

Reply on news.ycombinator.com