brettproctor

Assuming dwaxe replies claiming to not be a robot, how would we go about verifying? :P

Can someone explain why people hate this reply so much? This is a perfect subreddit describing this very scenario, and my submission there is now #2, indeed making this relevant r/nottheonion material. Is this just generic reddit hate or what?

That is awesome.

Also relevant: https://www.reddit.com/r/KarmaConspiracy/

As well as: https://www.reddit.com/r/HailCorporate/

Relevant: https://www.reddit.com/r/nottheonion

I always thought the smartest thing was keeping the frontpage unappealing enough to scare off anyone who hadn't already been told by a good friend it was worth checking out. It is sort of this pseudo invite-only filter.

Pipes was the first service I ever owned from an ops perspective (circa 2008). We had two clusters, one on each coast. I thought we had capacity to take one offline to do a full upgrade, turns out we didn't.

By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.

Fun times.

Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.

This quote from the article talks about data verification. Although we all know that a backup isn't really a backup until you've tested restoring from it :-)

""" To tackle this, we built a background “anti-entropy” process that detects data aberrations by periodically scanning all data on all the drives and reporting any detected corruptions. Given the inexpensive drives we would be using, we calculated that we should complete a full scan of all drives every 30 days or so to ensure we would be able to re-create any lost data successfully.

Once an error was found and reported, another process would take over to read enough data to reconstruct the missing pieces and write them to new drives elsewhere. This separates the detection and root-cause analysis of the failure from reconstructing and protecting the data at hand. As a result of doing repairs in this distributed fashion, we were able to reduce reconstruction from hours to minutes. """

Yes, the idea is to be as general purpose as possible. That was one of the motivations for avoiding tape.

No. We published an article of doing raid in hdfs in june 2014: https://code.facebook.com/posts/536638663113101/saving-capac...