Personally, it's a bit puzzling to me why the ML community ran with Python and why there isn't a popular alternative for Ruby. It's very capable in terms of making DSLs.
My hunch is that the number-crunching libraries were better in Python (numpy etc).
1. NumPy is really fast and has a lot of features. Even if they don't need all of them they feel more at ease knowing that they can use them should the need arise.
2. Python is very familiar to them, their professors know it, they know it, it's easy to teach to new people in the team so everybody just goes with the flow.
As much as I think the bigger IT area is in a desperate need of innovation... I can actually agree with those two points. Tinkering should be left to us the programmers and sysadmins. Everybody else has a job to do and they cannot and won't spend a lot of time evaluating all tech possibilities.
That's okay. It's our job to tempt them with something irresistible. :)
Maybe the watershed moment was when MIT switched from Scheme to Python? I don't know.
But I believe (can't prove) that you are correct on the outset: some big org switched, then a lot of people went like "oh well, Org X can't be wrong", and the rest is history.
With the scientific python stack you entered a new world with unlimited seats, you could at least try to run the same experimental/exploratory code with minimal changes in a production environment, etc.
So in short it didn't need some tentpole user/customer/exemplar. In almost any situation it was free and it started making money as soon as it was downloaded installed and run.
At the same time, I saw no substantial interest in persuing the same dedicated effort at the same scale in any other ecosystem. Rails was what everyone did in Ruby (aside: recalling that the ffi changed between 1.x and 2.x and it wasn't a trivial API change, and how the only documentation seemed to be the code, would have broken any prior work... I can't imagine putting long-term effort in this direction), java and jvm hosted languages didn't seem to gain any traction from users for these problem spaces (maybe the inability to easily dynamically allocate more heap on demand in user code made this unfeasible?)
Looking back I think the fact that numerical work has found success being built on a huge foundation of stable libraries that are glued into a dynamic interpreted language and which is then often treated as a finished artifact, I can't think of another major language that is more appropriate than python.
Specifically relative to Ruby: at its core Ruby is matz's place to experiment about ideas in programming languages. If you are going to build your business around a few million lines of glue code that relies on the internals of a language, you want to have strong guarantees about the stability of what you're building on.
Looking back I think the python 2 to 3 transition period actually made this better for python by providing a very stable foundation in python2 while experiments and destabilizing changes were mostly pushed to 3, which kept ahead. When finally transitioning to 3, the needs of the scientific python community and ecosystem were so well known and represented that it was unlikely to simply get forgotten, left out, or left behind.
edit: Which lines up with my recollection, Ruby was synonymous with Rails and web dev while Python wasn't type cast yet.
It's 1995. Python has been around for 5 years. Guido is working with the scientific community and adding better complex number, array, and matrix support to Python. Ruby has just started and everything is in Japanese. Scientists and Engineers at US universities and FFRDCs (Federally Funded R&D Centers) keep plugging away at Python for scientific purposes. Finally in late 2000, the first book about Ruby is published in English. Ruby is still pretty unknown. In 2005 Rails hits the scene and the English-speaking world notices Ruby. But this is 10 years after the scientific community starts investing in Python and 15 years after the Python language came out. By this time, the community had Numeric, numarray, and finally NumPy to unify their efforts.
Ultimately, Python's BDFL started working with the scientific/numeric community before Ruby existed. It would be 10-15 years after Python that Ruby would start making inroads in the English-speaking world and by that time there was already good numeric/scientific tools in Python. Scientific communities weren't looking to learn new languages for the sake of new languages. Many still just use MATLAB.
MIT hadn't switched to Python when a lot of this happened. I think it's more that Ruby wasn't widely known to exist until 2005 and was "brand new" to the English-speaking world then. Python had been plugging away at numerical libraries for a decade by then and NumPy (which had learned from previous numeric/scientific libraries) came out basically when Rails came out.
Once all these numeric libraries exist for Python, are you getting grad students excited to work on an open-source numeric system for Ruby? Are you getting FFRDCs interested in creating libraries for a language that's quite similar to Python without any significant advantage? How do you justify that work? How do you get people excited about it when the reaction of most users is going to be "we're already happy with what we've got"?
Rails burst onto the scene at a time when people were often unhappy with their tools. Rails was a breath of fresh air with clean separation of concerns and a lot of helpful stuff. This is back when Java meant XML sit-ups, C# was closed-source and this is before most of what the community knows today existed, Scala had just appeared, Python was still in the Plone/Zope era (remember when an object DB in Python seemed good?), PHP was pretty dominant but often ended up a mess and didn't have the same framework that Rails was offering. Rails offered something compelling and new which meant attention. Many other communities followed Rails - Django, .NET MVC, JAX-RS, etc. Still, Rails was there first and got a lot of mind-share.
Similarly, Python was there first in scientific/numeric computing and people haven't provided enough to compel people to move off it.
I vaguely recall that Python documentation was generally much better at the time, and especially so the parts on writing extension modules. With Ruby, it felt like the English reference docs were largely an afterthought - you pretty much had to get a book on it to get far, and the more advanced sections (like extensibility) were especially lacking.
After a few different options such as Octave, numerical computation in the free software space coalesced around Python.
The Python ecosystem - it's change and release processes (PIPs etc) - thorough API documentation - make for a good target language, over other languages that simply do not have that infrastructure.
I don't know what the state of Ruby documentation is these days for instance, but back in the day, it was simply incomplete.
>It's very capable in terms of making DSLs.
Which is a double edged sword. Too many DSLs means you have to learn and remember a lot more to write code.
ISTM (supporting a system written in Ruby at the time) that a large number of the libraries were broken when used in concert with each other -- and that language experts knew how to monkey patch their way around these issues, but didn't bother documented these fixes.
I've been told that things are much more mature and stable now -- but that experience put me off the language.
I am not trash-talking that phenomena but I also found it off-putting and eventually moved away from Ruby/Rails after 6 years with them.