There are now wheels (binary distributions of Python modules) of numpy, scipy, and scikit-learn for both Linux and Mac OS X, so it should just be a matter of `virtualenv /tmp/v` (or wherever), `source /tmp/v/bin/activate`, `pip install -U pip` to make sure your pip is new enough to handle wheels, and `pip install numpy scipy scikit-learn`. Nothing gets built from source. Unfortunately the same can't (yet) be said for pandas; `pip install pandas` at this point will try to compile it.
Actually acquiring virtualenv on OS X seems not to be super straightforward, especially if you want to use the system Python interpreter. The best option seems to be to `brew install python`, and then `pip install virtualenv`. Or download Python from python.org, and maybe get Python 3 while you're at it.