Your comment reminded me of rachaelbythebay.com. While her site seems to be down at the moment, she has shared some useful war stories from debugging similar scenarios. I understand systems better after having read her work.