I was going for a PhD in Math and dropped out for pretty much the same reason. The field was so specialized that nobody really understands anyone else's research, which makes it mostly not-merit-based (politics) who gets hired where. Also, it really wasn't fun. I figured it would be easier to work as programmer and study Math (or other things) in my spare time.
It seems like a waste to spend 5+ years on a PhD and 3-5 years on postdocs just to get a tenured position at the #200 ranked university or to be a permanent-adjunct.