Pipes was the first service I ever owned from an ops perspective (circa 2008). We had two clusters, one on each coast. I thought we had capacity to take one offline to do a full upgrade, turns out we didn't.

By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.

Fun times.

Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.