Octave sits at an awkward half-way point between MATLAB and Python/R/Julia/etc. You get the shitty syntax of MATLAB at not quite MATLAB's speed and miss out on support as well as various incompatible toolboxes. So unless you have hard dependencies or lots of MATLAB code sitting around, Octave isn't that attractive an option.

It's great for matrix/vector maths, though. If that's all you do -- go for it. Everything over and above that is a royal pain in MATLAB and, conversely, in Octave.

I have used Octave but only for the Coursera ML Class, not actual production use. My understanding is that it is the open source version of Matlab capable of running many Matlab programs.

I learned Octave first, then R, then Python.

Octave to me feels like Python + NumPy. I'd say Octave has more in common with R than with Python.

Given the choice between Octave and R, I'd choose R for the more robust user community and incredibly diverse and thorough selections of libraries.