Solid summary of some powerful tools, no wonder python is the new default for academic data analysis.
It also has very nice matrix and vector manipulations features built in to the language and is very easy to bind to C code.
Matlab,Ocatave,R,S are great but if you need to be closer to the metal, Lush offers a very good compromise.
Personally, I am experienced with Matlab but not so much with Python, so I am not able to judge. I definitely hate the fact that Matlab is proprietary and partly closed source. Also, I think Python syntax is much more pretty while Matlab is not even designed to be a real language. But alas, it comes with very powerful functionality out of the box.
Note, that the OP mentioned in the introduction that he had no access to Matlab over his employer or university and hence dismissed it.
[1] https://www.coursera.org/ml (one of the first videos)
If you're only coding the core of an algorithm (rather than a full-featured library with lots of plumbing), and your logic fits naturally into Matlab's native array operators, then using Matlab is a joy.
http://www.scipy.org/NumPy_for_Matlab_Users
IMHO, it's best to prototype in Octave and then build in python. I find that the Matlab/Octave syntax is too focused on linear algebra, so it's better for small prototypes (and for people coming from non-SW fields). For big projects, I prefer the 0-based arrays, more than one function per file, and all the rest of the python goodies. I estimate that 70% of my time is usually spend preparing the data (e.g. parsing xml, or some other files, etc), for which I find python more suitable.
In fact, I usually work with them side by side, testing ideas in Octave, then implementing these pieces into a large python project.
Edit: this has also been discussed here before, e.g.
Never trust academics when it comes to programming. :)
Seriously, Matlab might be a tiny bit better for scientific programming than Python. But: if you start building an eco system around your machine learning code (distributed evaluation of models, email reporting of results, web reporting of results, online tracking of training progress, data base related things, web services for other people, proper documentation, ...) you are happy if you chose python.
Also, there is theano for python which has auto diff, transparent GPU/CPU use and symbolic optimization. It makes your life easy if you are using complicated models.