At the birth of Coursera in 2012, R and Python were already clearly established in the field of data science. R was the dominant open-source language for data science, with Python very close behind (and already gaining ascendancy among folks who identified with "machine learning" rather than "data science"). I remember Matlab/Octave being more associated with academics/students in EE (signal processing, wireless communications, and the like); if you want clear insight into matrix operations, Octave is great.
I think Ng made a very conscious decision at the time to eschew built-in functions and not get distracted by trendy languages - hence the use of Octave to learn how to implement algorithms at the most basic linear-algebra level.
Even at the time his decision was not well understood nor popular - way back then I remember people asking "Why Octave instead of R or Python?"
In those classes that did use Matlab, there were quite a few people sticking to Octave, though it wasn't 100% compatable. And when I got to Ng's course, I (and I imagine a whole lot of others) were really happy to see he went 100% with Octave. Had he gone the Matlab route, the old course would be pretty worthless now.
We'll never know what happened behind closed doors, but I think Matlab was sponsoring some courses in order to get new people hooked on Matlab and it just didn't pay off.