GNU Octave [0] is mostly compatible with MATLAB. GPLed and implemented in Fortran and C++, it's pretty much hack-able to your needs.
> MATLAB language is extremely awkward anytime you want to work with something that isn't a matrix.
From my limited experience, that's true but rarely of relevance. You can, and want to, use matrix operations all the way. Less bugs (simplier code), faster execution.