The counterintuitive rise of Python in scientific computing (2020)

The important part wasn’t the exact number, but rather the discovery that the ratio is pretty stable.

I don’t follow why its array syntax is easier than python though. They’re mostly very similar and the numpy developers seem to come from a Fortran background.

When you are being exploratory even waiting a half second to a few seconds for a build is enough time for many brains to forget aspects/drift from why they pressed ENTER. When you are being careful, it is an acceptable cost for longer term correctness/stability/performance/readability by others. It's the transition from "write once, never think about it again" to "write for posterity, including maybe just oneself"..between "one-liners" and "formatted code". There are many ways to express it, but it underwrites most of the important "contextual optimizations" for users of all these software ecosystems - not just "speed/memory" optimization, but what they have to type/enter/do. It's only technical debt if you keep using it and often you don't know if/when that might happen. Otherewise it's more like "free money".

These mental modes are different enough that linked articles elsewhere here talk about typeA vs typeB data science. The very same person can be in either mode and context switch, but as with anything some people are better at/prefer one vs. the other mode. The population at large is bimodal enough (pun intended) that "hiring" often has no role for someone who can both do high level/science-y stuff and their own low-level support code. I once mentioned this to Travis Oliphant at a lunch and his response was "Yeah..Two different skill sets". It's just a person in the the valley between the two modes (or with coverage of both or able to switch "more easiliy" or "at all"). This is only one of many such valleys, but it's the relevant one for this thread. People in general are drawn away by modes and exemplars and that represents a big portion of "oversimplification in the wild".

This separation is new-ish. At the dawn of computing in the 50s..70s when FORTRAN ruled, to do scientific programming you had to learn to context switch or just be in the low-level work mode. Then computers got a million times faster and it became easier to have specialized roles/exploit more talent and build up ecosystems around that specialization.

FWIW, there was no single cause for Python adoption. I watched it languish through all of the 90s being largely viewed as too risky/illegitimate. Then in the early noughties a bunch of things happened all at once - Google blessing it right as Google itself took off, numpy/f2py/Pyrex/Cython (uniting rather than dividing like the soon after py2/py3 split), a critical mass of libs - not only scipy, but Mercurial etc., latterday deep learning toolkits like tensorflow/pytorch and the surrounding neural net hype, compared to Matlab/etc. generally low cost and simplicity of integration (command, string, file, network, etc. handling as well as graphics output) - right up until dependency graphs "got hard" (which they are now), driving Docker as a near necessity. These all kind of fed off each other in spite of many deep problems/shortcuts with CPython design that will cause trouble forever. So, today Python is a mess and getting worse which is why libs will stay monoliths as the easiest human way to fight the chaos energy.

Nim is not perfect, either. For a practicing scientist, there is probably not yet enough "this is already done for me with usage on StackOverflow as a one-liner", but the science ecosystem is growing [1]. and you can call in/out of Python/R. I mean, research statisticians still tell you that you need R since there is not enough in even Python...All software sucks. Some does suck less, though. I think Nim sucks less, but you should form your own opinions. [2]

[1] https://github.com/scinim/ [2] https://nim-lang.org/

If so, I'd consider switching (as Matlab does that better than vanilla numpy). Julia is pretty great in theory. It is still a very new language for my uses, which means the documentation and community are orders of magnitude smaller than Matlab or Numpy/python.

- get more traction

- I'm less able to focus on the software and have to focus on business development

- I'm going to need to hire someone who is a competent programmer but can also deal with the mathy bits

- The OOP features of Ruby are consistent and ubiquitous (everything in Ruby is an object); Python depends on manual patterns to do OOP (self as the required first argument). Python also depends on special decorators to indicate what functions are instance vs static. Ruby does have a difference in definition, but it's simpler and more obvious (and requires one line fewer of code to define)

- Only finally has Python gotten a switch statement, and surprisingly it has adopted some Elixir-ish pattern matching features. Incidentally, some in the Python community are strongly against this new thing. "Why would you need that!?" Prior to 3.10, you would need more complicate if/else if structures in Python to do the same thing you could do in a concise and clear Ruby case (switch).

- Operations on collections: this is often described as functional programming, but it really is just "doing stuff on collections of data". And in that story, Python's list comprehensions are arguably less readable and less logical than Ruby's. Many of the tools you need in Python must be explicitly included from functools module.

- Ternary operator: many languages have `expression ? do_true_path : do_false_path`. It's a very common pattern which is concise and honestly quite clear. "This thing is true? then do this; else do that". But in Python you break that up into "do_true_path if expression else do_false_path".

- Everything in Ruby has a return value, but not so in Python. So in Ruby you can make assignments (or return values) from the result of if/case. For example, assume you want to return a specific value based on some series of conditions, such as handling an error and returning some enriched data based on the error code. In Python you will have to define a local variable and explicitly set that variable equal to some value in each branch of the conditional. Then afterward, you can use the value of the local variable. Or you would have multiple returns, one in each conditional branch. In Ruby you can simply do x = case ..., or because the last statement of a function is the return value, you wouldn't even have to return it. You just have 'case ...', and the value of the branch is what is returned.

There's a lot more. Some of it is subjective, but my belief is that once someone really knows both, they will prefer the Ruby way. And the more languages you know, the more you develop refined tastes. Ruby still holds up well after knowing 10 languages for me.

(added): the whole whitespace as code thing of Python. It does have one common pitfall, and that is when a line of code gets accidentally indented or unindented below a block which was indented. That line changes scope, likely changing the runtime result; but it may be technically valid, so the developer may not notice the mistake. This is just not a problem with languages that have { } or begin/end delimiters.

The significant dangers in Ruby lie in the metaprogramming, which probably scientist devs aren't exercising.

Bioscience / environmental science programs will find micropython good enough for their needs. The tool itself is just the means to the end if real science. Micropython let’s you deploy in lower power applications without having to learn much beyond what you already know from Jupyter notebooks.

I really don’t know any PhD students or post-docs in microbiology/environmental sciences who have the time to learn embedded C or similar languages.

I suspect the main reasons are:

1. Python's guiding philosophy of "There should be one-- and preferably only one --obvious way to do it". With later additions to the language, this is getting less true (3 different ways to format strings, asyncio, type hinting, etc). Some libraries also don't conform to this (matplotlib). That said, it's a lot better than the Ruby code I've encountered, which is like the wild west.

2. Python's syntax is reasonably simple to teach. The object model could be condensed into something very simple if you don't need a lot. With very basic knowledge, you can go a long way. Ruby's a bit more chaotic with things like inheritance, extend, and include; proc, block, lambda; having to use attr_accessor; syntax things like a = b could be a function call or not; if/unless; and many more things that are confusing.

3. Even basic things like loops in Ruby is not idiomatic as it wants to apply a function/block instead. Beginners, especially those with a bit of background, like their loops better than functional programming.

As I've spent many years working on Ruby code base I still get lost all the time. Python in my experience has been a lot better, although recent Python versions have regressed a bit as it introduced more syntax to do the same things.

Lots of comments in this thread from people who's Ruby experience is only from the post Rails era after ~2008, and don't understand that the post Rails culture wasn't really a thing when Python was first gaining momentum for scientific computing.

While van Rossum drew from that experience when making Python, the initial driving goal was as a scripting language for system admin tasks in the Amoeba distributed operating system.

https://docs.python.org/3/faq/general.html#why-was-python-cr...

People attracted to Ruby are mostly of an "artistic mindset", they want to be expressive, write code that doesn't look like programming code and using "magic" like dynamically created methods, monkey-patching, etc is accepted or even encouraged.

On the other hand, Python attracts more people with "engineering mindset", they like straight forward code that's readable, clear and understandable, even if it's not as expressive. "Magic" elements are frowned upon: for example, imports are explicit and always included in each file.

Obviously, I'm exaggerating it, but I think is a clear differentiation between the communities.

My guess is that the "Python mindset" got into creating better integrations for "engineering applications", like NumPy or SciPy, and that created some positive feedback in certain environments. The main strength of Python is its rich ecosystem of third party packages. There's a compounding effect, making it grow faster and faster.

Around 1993 I got hooked on Perl. I read the Perl book and it was great. But 1) I couldn't figure out how to handle complex data structures (this was Perl 4), and 2) I couldn't embed it into other projects.

More specifically, worked on a molecular visualization program called VMD. It had its own scripting language. I wanted a language to embed in VMD that was usable by my grad student users. This is when I first learned about Python, but I chose Tcl because it fit the existing command language almost perfectly.

At around the same time, UCSF started embedding Python for their molecular visualization package, Chimera, so it was already making in-roads in structural biology.

I later (1997) went into more bioinformatics-oriented work, where I did a lot of Perl. I tried out one implementation (a Prosite pattern matcher) in Perl - which took me reading an advanced Perl book to learn how Perl 5 objects worked. I then tried the same in Python, a language I wasn't as familiar with. And it was just so much easier!

At this time Perl was THE language for bioinformatics, but I thought it was a difficult language for complex data structures. (Bioinformatics at that time was mostly string related, plus CGI and databases - Perl was a great fit.)

I then moved over (1998) to cheminformatics, working more directly on molecular graphs. Python was a much better fit for those data structures than Perl. I started using Python full-time, and it's been that way since.

We used a third-party commercial package for the underlying cheminformatics called the Daylight toolkit. It had C and Fortran bindings. Someone else had already written the SWIG configuration to generate Perl, Python, and Tcl bindings, but these still meant manual garbage collection.

I was able to use __getattr__, __setattr__, and __del__ to turn these into a natural-feeling high-level API, hooked into (C)Python's reference-counted garbage collector.

I presented a couple of talks about this work, got an article in Dr. Dobb's (!) and got consulting work helping companies which either had existing Python work, or were moving to Python.

By contrast, I don't think I heard about Ruby until 2000 or so, years after Python started entering structural biology/cheminformatics. [1]

I wasn't particularly cutting edge - others had already developed tool like SWIG, which was because Beazley and others were using Python at LANL. Numeric Python started in part because of work at LLNL and other research organizations. The concept already firmly established was that Python would be used to "steer" a high-performance kernel.

And Python in turn changed, to better reflect the needs of numeric computing, in particular, the "..." notation in array slices was added to make matrix operations easier. (This was 20 years before '@@' was added to simplify matrix multiplication.) I believe the needs of numeric computing also influenced the changed to "rich" comparisons.

This all took place around the time Matz started developing Ruby. Python had a clear head-start. And except for bioinformatics, Perl never had much presence in the fields I worked in.

So:

> why did python's use in scientific computing keep expanding, and not ruby's?

Because Python was in-use several years before Ruby, and already rather visible as one of the three main languages to consider in that space (Tcl and Perl being the other two).

> Why was python already taing over many use cases by the 2000s, but not ruby?

Because people didn't really know about Ruby, while Python already had a pretty large user community. Probably also because Python's work was all in English, while a lot of the Ruby community was using Japanese.

> Why is Perl attrition relevant, when ruby was in fact explicitly based on Perl?

Perl attrition started before Ruby was much known. The complexity of the language, and the cumbersome need to roll-your-own OO, made it difficult for me to recommend to the typical software developers I work with - grad students and researchers in the physical sciences with little formal training in CS. Python by comparison which easier to pick.

So a language which explicitly based on Perl also picks up that negative impression.

(FWIW, I think Tcl is an easier language to start with than Python.)

> why did NumPy happen on python, not ruby?

Numeric computing in Python started before Ruby was much known. Quoting https://en.wikipedia.org/wiki/NumPy

"""In 1995 the special interest group (SIG) matrix-sig was founded with the aim of defining an array computing package; among its members was Python designer and maintainer Guido van Rossum, who extended Python's syntax (in particular the indexing syntax[8]) to make array computing easier."""

Quoting https://en.wikipedia.org/wiki/Ruby_(programming_language)

"""The first public release of Ruby 0.95 was announced on Japanese domestic newsgroups on December 21, 1995. ... In 1997, the first article about Ruby was published on the Web. ... In 1999, the first English language mailing list ruby-talk began, which signaled a growing interest in the language outside Japan."""

[1] Ha! I found a comment I made in 2003 saying I had looked into Ruby "a few years ago", at https://groups.google.com/g/comp.lang.python/c/xBWUWWWV5RE/m... . I also wrote:

""" I think my criteria for selecting Python over Perl is still true for Python over Ruby, in that it has too many special characters (like @ and the built-in regexpes), features (like continuations and code blocks) which are hard to explain well (I didn't understand continuations until the Houston IPC), and 'best practices' (like modifying base classes like strings and numbers) which aren't appropriate for large-scale software development."""

One of Python's original use cases was as a macro/script language you could import into your C application. Adding python to your C app took a day or so, and a side effect often was you'd make a Python library out of your app's library and suddenly, you could write standalone Python that called your app's code. Because it was so easy to write a python wrapper for an existing C library, by 1995/96 when Ruby hit the scene, Python already had quite a bit of importable functionality. The first serious web framework for Python was Zope, and it came out in 1998. I think Rails was around 2005/2006, and it was really cool, but one of the rubs was Ruby didn't have the libraries that Python did. In reality, it's amazing how good Ruby and Python have been.

Open ipython3

    In [3]: class Test:
       ...:     def __getitem__(self, index):
       ...:         print(index)
       ...: 

    In [4]: Test()[1, 2, 1:3, ..., :]
    (1, 2, slice(1, 3, None), Ellipsis, slice(None, None, None))

Numpy has existed for a long time. Its needs have been taken care of in upstream Python, to a big extent, and other libraries can use the same features.

If someone is wrong and you know more than they do, you can share correct information without putting them down.

Never needed Conda, but the packaging tool proliferation is an embarrassment, IMO.

Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.

* why distinguish between Series and DataFrame? just give me an interface for m x n matrices or even higher dimensions.

* pure vs. in-place operations. not such a big fan of having multiple versions of the same function, e.g. a more pythonic

    df[“my_col”] = series

    df.assign({“my_col”: series})

That brings me to another point: if we keep everything purely functional, then python’s syntax is making things a bit awkward. Where in something like JS you could just put every function call with its dot on a new line without the need to assign, in Python this requires putting line break characters or wrapping it in round brackets. This is one place where a language with explicit assignment terminators (semicolons) are a bit cleaner to work with.

All that being said scipy is still a great choice to have both system programming and numerical business logic in one language.

Little things, like some functions want to be called with a tuple of dimensions,

    np.zeros((rows, cols))

    np.random.randn(n, m)

When you slice a column from a matrix, and get a not an nx1 vector, but a 1d array, it makes me want to shell out $10,000 for matlab (yes, I know I can get a column vector with the slice A[:, [2]], but I shouldn't have to).

This problem leaks out into the ecosystem. For example, when you try to use scipy to integrate an ODE, and pass it an initial condition vector that is nx1, the scipy integrator will silently coerce your vector to a 1d array, pass it to your RHS function, which then either blows up, or more likely, produces silently wrong result because of numpy's insane array broadcasting rules.

This problem further leaks into the ridiculous function hstack. If you just used the function vstack, which made a 2x3 matrix from 2 1d 3 element arrays, you might imagine that hstack would produce a 3 x 2 matrix. But no. It creates a 1d 6 element array. For what you wanted, you actually need np.column_stack.

I think the way Eigen handles this is the most intuitive. You do linear algebra with 2d objects, and cast to arrays for elementwise operations.

There is also a huge inconsistency between what numpy exposes as an object oriented interface vs a "functional" interface. What I mean by this, is that I can call x.sum() on an array, but not x.diff(). For that, I need np.diff(x). There seems to be no pattern to what is exposed as a method vs a function.

The array slicing api is also really inconsistent. For instance, given a 3 element array x,

   a = x[5]

   a = x[2:5]

Next time I find myself manipulating numerical data, I'll definitely take a look at Pandas!

The tidyverse ironically ends up feeling more Pythonic, with more of a "there is one obvious way to do it" vibe.

Any feedback for us? What can we do to improve Ploomber?

Npm, as much as it annoys me, is light-years ahead of anything in python. Quicklisp is rather pleasant, now. Ruby has had gems for a long time.

I grant that it is a hard problem. I am not griping that it is not solved. More that the option community has largely failed to even pick a direction. The most used dependency methods are, in the modern spirit of python, deprecated already.

>>Python has a lot of great libraries. [...] But there should be a capstone level class for data/ai scientists before they can graduate which shows other languages

I didn't downvote your gp reply but your answer just pushes the question to an earlier point. Why did early earlier 1995 programmers at science labs like David Beazley and Jim Hugunin (who already knew "other languages" such as C Language, assembly, Fortran, etc) ... choose Python as the scripting wrapper for their C code? See my other comment about their earlier history: https://news.ycombinator.com/item?id=30813528

The "Python having a lot of great libraries" wouldn't have been a compelling reason for David Beazley since those earlier creators of scientific packages for Python chose Python before it had a lot of scientific libs. They were among the very first.

Here are some bullets from another deep link at a different point in the video[1] :

- David's first attempt writing his own homegrown scripting language.

- he also looked at alternatives like Tcl/Tk and Perl and they weren't as appealing as Python.

- David mentioned Python had a more powerful REPL.

- Python was also open source C code so he could easily modify it to run on the Thinking Machines CM-5 computer[2] in the physics lab

- wanted a language & runtime that encouraged the wider community to build more science tools

In your opinion, what was the superior programming language that David Beazley and Jim Hugunin should have chosen in 1995 that checks all the bullet points above?

[1] https://youtu.be/riuyDEHxeEo?t=42m43s

[2] https://www2.cisl.ucar.edu/supercomputer/littlebear