While there are certainly advantages to Anaconda, I've never encountered any troubles installing Pandas, NumPy, SciPy, or scikit-learn on any OS X or linux system. In my experience, getting GCC up and running is far more of a pain in the ass (and it usually isn't even that bad).
I use pyenv[1] and pyenv-virtualenv[2] to easily keep track of Python versions and virtual environments. I keep one virtual environment for each project I'm working on, and things prettymuch Just Work.