Array Programming with NumPy | Hacker News Reader

Array Programming with NumPy

nature.com

290 pointshardmaru111 comments

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely useless". It was also always unpublishable, as methods are never publishable in my field. Numpy has (obviously) vastly more respect and impact than my work, but the general problem remains.

Articles like this are a _huge_ deal for that reason. It's an immense delayed recognition for over a decade of work from a lot of folks.

> my work was "utterly and completely useless

I can never understand the arrogance of folks who would say something like this

I used to maintain a code beautifier/diff tool and hear the same things about how the idea was useless only to see those same people shortly there after use my tool or a close competitor. Once you see that pattern a few times you learn to ignore it in it’s entirety.

It’s hard to tell why people behave like that. I presume it’s because many people have a great fear of originality and require social validation.

Asymmetry between complexity of principle and power.

Ie., we are easily persuaded that something very complex will be very powerful (eg., a smart phone) -- but we intuitively regard something simple (eg., a hammer) as under-powered.

Hard to say how well this actually holds, but I'd guess in both cases we arent really enumerating use-cases in our head, we're just using explanatory complexity as a guide to practical power.

This is probably more extreme in cases where people have a specific notion of complexity in mind, eg., in academic environments where "tool A" is as simple as "tool B" if they use the same theoretical basis.

ie., Tool C is worthwhile if it includes a more complex theory, as therefore it is more powerful.

Academic administrators judge performance based on publication counts, journal impact factors, citations/h-index, and fundraising. Working on tooling doesn't fit in those buckets, so it's broadly-speaking "useless" to a researcher vying for promotions (eg tenure). It's an imperfect method of measuring true impact.

And to be fair, that was the context of the comment. It was meant to be harsh but true advice. It was quite arguably, at the time, worthless in the context of advancing my career. It turned into my career, eventually, but that wasn't the goal then.

as a former phd student, it's pretty systemic - labs sometimes get funding as a direct mapping with how many papers get published by that lab. If you do cool work but that does not land you a paper, you are literally wasting the budget that your employer spent on you. Once you solve that (and thanks to things like this numpy papers, things may be starting to evolve !)

I recall a story where a friend was unable to publish a paper in which he wrote an alternative to a very commonly used commercial tool (that virtually everybody used) with roughly 10 times better performance. He open sourced it and all, it was extremely useful, but there was no new methodology, it was simply very well implemented.

At a talk of his it lead to a very heated discussion where an older professor accused him of wasting government money on such nonsense.

Been there. A few years back I got a government scholarship for my PhD (which is still in progress, due to my follow up work). I basically built the foundation upon which to establish a new field for my university, and the region where I live. There are some professor who think that scholarship (and the little money it gave me) was wasted on my because I chose to build all of that from the ground up, instead of rushing through my PhD.

By the way, those of that opinion are all professors who wanted me on their labs, but I turned them down...

For every story like this, I believe there are many more in which the student simply writes their own implementation due to not invented here syndrome or engineering as a form of procrastination.

Yes, it is easy to be sidetracked on writing software. Not that software is a bad thing, but research is something different.

If you talked to me about my PhD for a few minutes you would surely put me into your "had to reinvent the wheel for no reason" category.

As indeed, I wrote an analysis framework for my data (of a gaseous detector used for axion search) [0] instead of using an existing framework used by my predecessor. However, things are always more complicated than they seem. Many of those not talked about students who rewrite stuff probably have reasons!

In my case the existing framework [1] was a monster that was bent to allow it to work with the kind of data we have in the first place. In my case my detector had several additional features, which fit _even less_ into the existing framework. It would have been a hack and still a significant amount of work to make it work well.

To be fair, when I started this I expected it to be less work than it ended up being. But that's the story of software development.

The advantages now are significant of course. I know the whole codebase. It does exactly what I want. I can extend it easily as I see fit.

That doesn't mean I didn't also partly procrastinate writing software. Far from it. Hell, there was no reason to write a freaking plotting library (a sort of port of ggplot2 for Nim) [3]. But again, this means my thesis will have plots created natively using a TikZ backend while at the same time provide links to Vega-Lite plots for each and every plot in my thesis (which of course will include the data for each plot!).

Finally, the most important point: A university / professor who only pays me for 20h a week does not get to tell me how I do my PhD.

[0]: https://github.com/Vindaar/TimepixAnalysis [1]: https://ilcsoft.desy.de/portal/software_packages/marlintpc/ [2]: https://github.com/Vindaar/ggplotnim

I certainly have experienced similar things, particularly been acused of reinventing wheels. Flexibility and performance are two big reasons, but also "it's fun" or "I want to understand X" also have a good weight when we do this kind of "useless reinvention".

Given the current publish or perish culture, I doubt any student serious about publication can afford to waste time.

Maybe I wasn't clear enough. When I started my PhD, I was working on leading edge, basically 3 people in my country knew that we were talking about (and I was one of them). It certainly wasn't NIH-syndrome. Still, instead of "bailing out" on the easy path (present a paper here, work with that professor in That Other Thing That Doesn't Interest Me, etc) I chose to keep doing what I love.

End result so far? I'm quite respected, still one of the leading researchers in my country on my specific topic, but since I don't have a PhD (because of the aforementioned delays, and some grumpy professors actively pushing against me) I'm starting to lose access to grants and programs.

I'd still do it all again, but with a few tweaks here and there, you know hindsight always helping.

It is obvious that we need good software, however from the point of view of science the old professor may have reason. If you are receiving a grant, you're not being paid to write software, in the same way that an engineer is not paid to write novels. As useful as the software may be, the person in question should be spending time on research (by definition new subjects), not writing again an existing software.

> If you are receiving a grant, you're not being paid to write software

In my (albeit limited) experience, software is a pretty common deliverable from a grant, at least in computational biology. This has also been my experience with more alternative funding sources like CZI and DARPA.

Taken more broadly, I think there is a huge disconnect between what academics are paid to do, and what takes most of their time. Review is unpaid. Grants are not dependent on which journal the results go into, but time could be saved by aiming lower. A salary can be payed from a research grant, while the investigator still has to teach.

What if that piece of software increases research output across the entire field? Often, a good piece of scientific software advances research more than what you're calling "research."

Writing software is sometimes necessary to achieve the objectives of a grant (even though this is not necessarily explicit). It’s not writing software that’s a problem, it’s reinventing the wheel; you should not focus on “scientists should not write software”, because that is obviously far from the truth.

For a scientist, writing useful software is a good way to get exposure, build a reputation and get citations. It’s an opportunity to do some different kind of problem solving than usual. It’s also a way of understanding how the software really work (which assumptions are built in, which methods are used, and how does it affect the software’s results?). This does help improve the quality of subsequent results.

A grant typically (there are exceptions, of course) lists things that are going to be studied. How the studying is done is typically down to the people doing the work. It certainly isn’t for grumpy old professors who hear a talk at a conference to judge.

We usually manage a Somethinginformatics journal publication to detail infrastructure work. At least Zenodo etc. And annoy users of the software to cite (a doi isn’t much to add to a manpage or log output).

> for over a decade

Probably two even if NumPy came out in 2005

props to your work and similar to numpy, i assume it has been immensely useful for loads of people.

but 'building the underlying infrastructure that tons of people use' is not science. in my department we had to fail a phd student because 90% of his work was just implementing bunch of existing methods as a python library. useful, yes; science, no. wasn't his fault, had a shitty supervisor, but making useful tools is not the same as undertaking scientific research.

That's quite the wrong way of approaching science. The scientific method is based on building on the shoulders of giants. Those giants aren't the professors in the direct vicinity nor are they only the papers you cite. The whole of the process is science and if we need further specialization for building better tools (hey, maths and statistics are scientific tools as well) I would classify that as science to a large extent.

What could be useful is openness in used tools and software and a way of getting citation counts for software used. It's nothing more than a table. That way the hotness of publication could start to flow for the underlying tools.

> The scientific method is based on building on the shoulders of giants. > The whole of the process is science

no. scientific research is proposing a useful model of an observable phenomenon. this is what you train for during a phd, at least in natural/life sciences: you learn how to test a hypothesis, not an easy skill.

refactoring code or transforming bunch of C++ into a python library is useful, but it's not science.

> What could be useful is openness in used tools and software and a way of getting citation counts for software used. It's nothing more than a table. That way the hotness of publication could start to flow for the underlying tools.

agreed 100%

Maybe we should take all this "not science" software away from the scientists and see how much science they can do without it.

If you write code that allows science to be done that couldn't be done otherwise then that is science. As a high profile example, a large amount of specialist software was developed for the LHC to allow it to process all the events coming from the detectors.

It sounds like the refactoring here was not really that useful in the first place.

yes, in 2020 you mostly cannot do science without software, electricity, desks and chairs and buildings, printers, pick your own irreplaceable tool. yet building these things to enable research is emphatically not itself scientific research.

doing a phd -> training to be a scientist.

Printers? Desks? Take your strawmen somewhere else.

Since you brought up LHC, here is an anecdote.

I worked on software development tools used directly for LHC as part of an internship.

That experience was of zero use when I tried to apply for a PhD later. It did get me several $BIGN internships though.

Make what you want of this story.

I'm trained as an economist so might have a different view. But what I think I know from physics is that, say, the people actively involved in engineering things like matter collidors do get authorship or at least appreciation for their role in furthering science.

For me, our discussion is mainly in where to draw the line around "the process of science". The chair, laptop and coffee machines aren't science. The statistical methods, papers and engineering are. You seem to cut parts of the engineering out, namely the non-novel parts. There's a lot to say for that. But a PhD is proof of apprenticeship as well. I wouldn't grant someone a PhD if all of his work is 'mere retooling'. But in a mainly research papers based PhD-application I wouldn't feel some retooling couldn't be allowed. One could demonstrate scientific craftsmanship in retooling.

oh i completely agree that a binary distinction between 'tools' and 'science' is not useful. it's also extremely hard to be a good scientist without being very good at 'tools'. however, editors at peer-reviewed academic journals or people awarding degrees mainly need to ask whether the work has advanced our knowledge on X.

if X is e.g. microbiology then it's fair to ask whether (1) some python library proposes something in terms of microbiology, and (2) bunch of biologists should make that decision.

this is why refactoring code is mostly dismissed as 'doing science' by most phd supervisors. sure counts as 'developing skills', which certainly should feature prominently as part of your training, but it cannot be all there is to a project.

I am really sorry but most “science” coming out of even the top institutions these days are sadly uninspired garbage regurgitations.

most of anything is 'uninspired garbage'. not sure what it's to do whether a particular phd should be awarded.

no-one is proposing that numpy isn't useful or people developing / maintaining tools aren't doing gods work. they have my endless gratitude and try to donate regularly.

however, phd training in my field -- natural/life sciences -- has a specific remit: you learn how to build and test a hypothesis, from start to end. optimising libraries is emphatically not it. as a scientist you should care whether you have a useful model that explains something about the world. this is orthogonal to how neatly you have implemented your linear algebra in python.

This comment displays a remarkable ignorance of scientific history. Why do you think Ramon y Cajal shared the Nobel prize, for discovering neurons, with Golgi, who 'simply' invented the staining method?

Inventing a method isn’t the same a re-implementing an existing method! Again, useful re-implementations are a good thing that people should be rewarded for somehow, but they are not new science.

It's very true that it's not scientific research. (And I actually generally agree with the premise behind "methods shouldn't be publishable -- they're for appendixes, not papers".)

However, there's increasingly a role for folks focused more on the scientific computing and methods side. E.g. "how do we constrain X parameters given Y observations" (yes, I just described inverse theory -- that's deliberate). The science isn't solving the problem, it's figuring out what models to use and what the inverted parameters mean. However, solving the problem correctly requires a lot of rather novel work and is very easy to get wrong.

It's similar to many other research staff positions. It's standard to include the person who operated/designed/etc the instrument you're using as an author on papers. Is it that crazy to include the person who developed the numerical methods and implemented the solution as well? For example, I have quite a few friends that stayed on as staff to run the lab or key pieces of equipment. They have tons of "middle author" publications as a result.

However, numerical methods and computing infrastructure and work is much less frequently recognized. This is a step towards changing that.

I know Travis' and Paul Dubois[0] work at Livermore was immediately recognized as of towering importance almost immediately. I was down the highway at LBNL porting shitty Mathematica, IDL[1] and Fortran Diffraction Grating code to Numeric or whatever they called Numpy back then almost as soon as it was released. People probably don't remember their history, but pretty much the only open source intepreters of the day were things like Perl (whose math capabilities at the time were pretty lousy). Scientists paid for a shitload of Maple, Mathematica, IDL, Matlab and Igor[2] licenses; and there still weren't enough licenses to share code with your friends, because nobody had licenses for them all.

Python 1.5 was the first non-mentat tier open source interpreter available that didn't get in your way as a scientist, and Numeric/Numpy was the first and still the most elementary piece that made it usable to science and numerics people. Might not have been letters to Nature tier back then, but Nature ain't what it used to be anyhow.

[0] Since a lot of folks have actually forgotten Paul: http://www.pfdubois.com/bio.html

[1] https://en.wikipedia.org/wiki/IDL_(programming_language)

[2] https://en.wikipedia.org/wiki/IGOR_Pro

I wholeheartedly agree.

I think journal editors have a responsibility here too in promoting references to software libraries used in the articles they publish. I almost never see these in my field (astrophysics), even though they are readily available and very easy to include.

This so much. I cannot stress enough what an impact various python developments have had on the economy as a whole. Sometimes the right tools can inspire people and that’s exactly what happened.

Looks like there's a new citation for NumPy in town.

"Citing packages in the SciPy ecosystem" lists the existing citations for SciPy, NumPy, scikits, and other -Py things: https://www.scipy.org/citing.html ( source: https://github.com/scipy/scipy.org/blob/master/www/citing.rs... )

A better way to cite requisite software might involve referencing a https://schema.org/SoftwareApplication record in JSON-LD, RDFa, or Microdata; for example: https://news.ycombinator.com/item?id=24489651

But there's as of yet no way to publish JSON-LD, RDFa, or Microdata Linked Data from LaTeX with Computer Modern.

As someone who cut his teeth on bioinformatics before eventually just completing a full computer science degree, I was a bit worried that this article wouldn’t address the applied scientific audience well. Pleasantly surprised, though, at how well this article evangelizes NumPy to that exact community.

Many labs are gaining access to or creating physical tools that create data analysis over experimental design problems. Biologists transitioning from running gels to detect the existence of a gene to running sequencing or flow cytometry to segment populations, for example. I remember seeing I think an RNA-seq experiment that tracked the full lineage of hematopoietic (blood) stem cells by a professor - my jaw was on the floor at the level of insight.

One remaining step is transitioning many bioinformatics courses from applied tooling to general program design and open-sourcing. I’ve noticed a few labs have done that really well for years, but it is not often found as a field-level skillset.

The coolest thing I read last year was about single cell rna-seq trajectory analysis of the differentiation for different blood cells.

You might like the mass cytometry reconstruction of the human haematopoietic system [1]. I've done a bit of scRNA-seq and mass cytometry, and the issue I have with scRNA-seq is the tiny dynamic range it has compared to flow/mass cytometry, which can make identifying populations much harder. Not to mention the cost!

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3273988/figure/...

Let's not forget to give at least some credit to Perl Data Langauge (PDL). It pioneered a lot of these ideas 10 years before NumPy existed, and is still a pretty great tool today:

http://pdl.perl.org/index.php?page=FirstSteps

Er... array programming and statistical programming languages pre-date both by decades.

APL from 1966, I believe, is the key lang for array programming.

And statistical languages like S from 1976 come to mind: https://en.wikipedia.org/wiki/S_(programming_language)

At a quick glance, it seems PDL is just a variation on S.

R (https://en.wikipedia.org/wiki/R_(programming_language) ) is kind of the successof of S. In certain communities (not only statistics, but for instance also biogenetics), there is quite some concurrency between R and (scientific) Python for data science.

For my understanding, the numpy syntax most closely resembles what would be possible in matlab. And matlab again seems to have roots from Fortran. Thanks to that, young folks nowadays can switch so easily between Fortran and Numpy, the syntax and call structures can easily be made to almost fit to each other.

> there is quite some concurrency

i think you meant "competition" here :)

(in polish, my native language, it's "konkurencja", but it's a "false friend of the translator"; i'm guessing you're in a similar boat)

Similar false friends in Dutch: Concurrentie

Or French : concurrence :D

I was referring to wrapping typed arrays in "normal" scripting languages, allowing integration with a larger eco-system, which is I believe why NumPy is more commonly used today than APL.

NumPy’s direct predecessor Numeric was released in 1995.

... and then in 2001 or so, numarray came about and was supposed to be better - but instead it sort of split the community and wasted resources.

Then Travis started Numpy which somehow magically was backward compatible with both numeric and numarray - and managed to get the community united again.

I didn't know that, thanks! It looks like these ideas were being tried in perl and python roughly in parallel.

For some reason this struck me as inappropriate for the outlet. It's a nice piece as an introduction to array programming with numpy, but seemed out of place to me.

If, going forward, 5% of all papers that use NumPy to get their results actually cite this paper, it will be one of Nature's most cited papers every year.

There's an interesting trend of what content gets published in peer-reviewed journals vs. blogs/github/etc. I suspect there is an audience segment that strongly values peer reviewed pieces that are equivalent content wise to introductory material in a variety of formats.

I wonder if github should add a "Review" feature to provide a similar content authoring experience.

It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research.

There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, since (assuming consistency) isn't math just a DAG anyways (and therefore isomorphic to a neural net, as are all things).

Traditionally that sort of stuff goes in tech reports, dissertations, or text books.

What's missing is the dissemination piece. Somehow people will absolutely refuse to take seriously the job of citing code they use, even when their main result is obtainable by "and then I ran something from scipy/numpy/pytorch/etc."

You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo.

If you have repo2docker REES dependency scripts (requirements.txt, environment.yml, postInstall,) in your repo, a BinderHub like https://mybinder.org can build and cache a container image and launch a (free) instance in a k8s cloud.

Journals haven't yet integrated with BinderHub.

Putting the suggested citation and DOI URI/URL in your README and cataloging citations in an e.g. wiki page may increase the crucial frequency of citation.

A Linked Data format for presenting well-formed arguments with #StructuredPremises would help to realize the potential of the web as a graph of resources which may satisfy formal inclusion criteria for #LinkedMetaAnalyses.

The issue is that none of the citation count engines (Google scholar, scopus, Web of Science...) count citations on those DOIs. So for a researcher who needs to somehow demonstrate impact through citation counts, it does not really help unfortunately.

We could reason about sites that index https://schema.org/ScholarlyArticle according to our own and others' observations. Google Scholar, Semantic Scholar, and Meta all index Scholarly Articles: they copy the bibliographic metadata and the abstract for archival and schoarly purposes.

AFAIU, e.g. Zotero and Mendeley do not crawl and index articles or attempt to parse bibliographic citations from the astounding plethora of citation styles [citationstyles, citationstyles_stylerepo] into a citation graph suitable for representative metrics [zenodo_newmetrics].

bitcoin.org/bitcoin.pdf does not have a DOI, does not have an ORCID [orcid], and is not published in any journal but is indexed by e.g. Google Scholar; though there are apparently multiple records referring to a ScholarlyArticle with the same name and author. Something like "Hell's Angels" (1930)? No DOI, no ORCID, no parseable PDF structure: not indexed.

AFAIU, Google Scholar does not yet index ScholarlyArticle (or SoftwareApplication < CreativeWork) bibliographic metadata. GScholar indexes an older set of bibliographic metadata from HTML <meta> tags and also attempts to parse PDFs. [gscholar_inclusion]

Google Scholar is also not (yet?) integrated with Google Dataset Search (which indexes https://schema.org/Dataset metadata).

FigShare DOIs and Zenodo DOIs are DataCite DOIs [figshare_howtocite, zenodo_principles]; which apparently aren't (yet?) all indexed by Google Scholar [rescience_gscholar].

IIUC, all papers uploaded to https://arxiv.org are indexed by Google Scholar. In order for arxiv-vanity.org [arxiv_vanity] to render a mobile-ready, font-resizeable HTML5 version of a paper uploaded to ArXiV, the PostScript source must be uploaded. Arxiv hosts certain categories of ScholarlyArticles.

JOSS (Journal of Open Source Software) has managed to get articles indexed by Google Scholar [rescience_gscholar]. They publish their costs [joss_costs]: $275 Crossref membership, DOIs: $1/paper:

> Assuming a publication rate of 200 papers per year this works out at ~$4.75 per paper

[citationstyles]: https://citationstyles.org

[citationstyles_stylerepo]: https://github.com/citation-style-language/styles

[gscholar_inclusion]: https://scholar.google.com/intl/en/scholar/inclusion.html#in...

[figshare_howtocite]: https://knowledge.figshare.com/articles/item/how-to-share-ci...

[zenodo_principles]: https://about.zenodo.org/principles/

[zenodo_newmetrics]: https://www.frontiersin.org/articles/10.3389/frma.2017.00013...

[rescience_gscholar]: https://github.com/ReScience/ReScience/issues/38

[arxiv_vanity]: https://www.arxiv-vanity.com/

[joss_costs]: https://joss.theoj.org/about#costs

[orcid]: https://en.wikipedia.org/wiki/ORCID

Owning to the distributed nature of git, and the properties of the hashes it uses, it is probably enough to put a full commit id in a paper to securely reference a software project, regardless of its hosting platform.

We'd just need a dedicated search engine, and a way to automatically extract those from papers, to clone and archive repos.

> the properties of the hashes [g]it uses

Git uses SHA-1, a hardened version since 2017, and are now doing per-repo upgrades to SHA-256 [0]. Lots of repos are presumably still on SHA-1 (and users on older versions of git).

As of 2020, chosen-prefix attacks against SHA-1 are now practical. [verbatim from 1] But I don't think second preimage attacks are practical yet.

Linus Torvalds argued in 2006 basically that it's irrelevant whether git's hash function is second preimage resistant. Selective quoting:

> remember that the git model is that you should primarily trust only your _own_ repository [2]

> [a malicious] collision is entirely a non-issue: you'll get a "bad" repository that is different from what the attacker intended, but since you'll never actually use his colliding object, it's _literally_ no different from the attacker just not having found a collision at all [2]

All that is just to say: git originally chose its hashes for the above mentioned "git model", thus didn't 100 % care about second preimage resistance. For your suggested search engine, depending on how the database is collected you might not be able to trust "your own repository" (if it's crowdsourced I could register another codebase with the same hash as Linux). A second preimage resistant hash function would be a requirement for the suggested use case.

[0]: https://git-scm.com/docs/hash-function-transition/

[1]: https://en.wikipedia.org/wiki/SHA-1#cite_ref-8

[2]: https://marc.info/?l=git&m=115678778717621&w=2

Maybe they're trying to promote a move from Matlab (which is close to exclusively used in a lot of academic fields) to Python, and not to Julia. Or maybe that's too far? Either way yeah I also find it slightly weird that this was somewhere in Nature.

Yeah, this is a review article. Nature is not the right spot for it.

Considering Nature labelled it as a review article and still published it, it does seem like Nature is an applicable venue for publishing a paper like this.

Nature has numerous review articles every year since 1974 (right on their website): https://www.nature.com/nature/articles?type=review-article&y...

Seems to me like it's just a way for the journal and the authors to collect a gigantic number of citations to win the academic citation game. Not that there's anything necessarily wrong with that; NumPy deserves it of course.

I think this is great. Nature is really a way for scientists to score points, not a publication that you read cover to cover that needs stylistic consistency. Right now the academic citation-count scoring mechanism doesn’t give enough incentive for people to work on the important infrastructure pieces like Numpy. So this is a good step towards putting scientific priorities in the right place.

Yep, especially when the author are the people who wrote numpy, I have absolutely no problem with that. It's about time they be recognized for their contribution to the tools of science.

I definitely think things like the infrastructure don't get enough credit. I also mean no criticism of numpy. But is Numpy per se conceptually that innovative, from a computer science perspective? I guess to me this just seemed unusually introductory, about a specific library for a specific language.

Put another way: if I was going to cite numpy, would I cite this? Probably not. Would I cite this paper for any of the more general concepts it covers? Probably not. I'd probably even argue someone shouldn't cite it for that latter reason, as those concepts supercede numpy (and appear in other languages under other names).

I forget who, probably Hamming, said his most cited paper ever was just an intro to statistics for biologists. This is not a new thing and it's not a bad thing. Papers are meant to spread information. If a field isn't aware that another field has solved a problem of theirs than even a 101 level paper is worth writing.

While NumPy may be old news to the programming community, I have experienced a surge of programming capability into wet bio labs that was not there a decade ago.

This kind of article heralds its adoption to mainstream biology. It's now well known enough to interest biologists in general!

That's a good point.

At first I was like, NumPy's fame -- owing to the rise of Python in scientific computing and data science circles -- has gone far beyond that of most things ever published in academic journals, so this hardly seems necessary.

But I see your point: Nature has always been held in high regard among natural scientists, and though most recently minted natural scientists have at least a passing familiarity with Python, the generations of scientists before them probably don't. Nature at least has enough of a cachet to grab their attention.

Plus having a publication in Nature does open doors. Travis Oliphant and some of the already-famous co-authors probably don't need these doors opened, but I'm sure others on that list would benefit.

It also leads the readers down the dead-end. Python is incapable of parallelism, the only way to badly emulate it is to launch several runtimes. Writing any scientific software in a badly design language when better alternatives exist is wasted time and effort.

It would be nice to have a faster, parallel Python but the best scientific ecosystems are Python and R.

For most scientists, programming productivity matters the most, and plenty of programs are embarassingly parallel.

For instance it's no trouble at all just launching a single threaded Python program once per sequencing sample, and it plays nicely with the supercomputer queuing system.

> the best scientific ecosystems are Python and R.

That is debatable

BLAS and LAPACK were originally written for FORTRAN, and FORTRAN is particularly suited to numerical computing and array programming.

https://modelingguru.nasa.gov/docs/DOC-1762

Aren't BLAS and LAPACK just for the linear algebra bits? Numpy is so much more than linear algebra.

Yes, there's a lot more to it than that.

For instance, the interface and array/matrix types make vector operations really natural and efficient in python.

And quite modern with recent language revisions, including OOP, generics, modules and first class support for C interoperability.

Indeed, FORTRAN has traditionally been used for vector and matrix math and NumPY adds the equivalent functionality to Python. The simple linear algebra abstractions for manipulating these structures are fundamental to each of the popular data science platforms.

MATLAB was created as an interface to LINPACK/EISPACK without having to learn FORTRAN. The importance of this comment is the emphasis on the core fundamentals shared by all the data science platforms rather than the different tradeoffs inherent in each ecosystem.

Young programmers are slowly but steadily reaching the power that Fortran programmers had 40 years ago. That's good news!

That statement only feels true to me if you interpret the word "power" as an exact synonym for "performance." Which is a definition that is valid, but also just about perfect for leading someone to miss the point.

Numpy approaches the performance you could get with Fortran. Mostly because its core is written in Fortran. What Numpy offers that Fortran never did, though, is leverage. The article mentions, but doesn't really do justice to, the sheer volume of interoperability that Numpy has enabled. It's not just that all these libraries were built on top of Numpy. It's also that their common Numpy substrate makes them all deeply interoperable with each other. And that works both above and below the boundary. You can swap out BLAS and LAPACK for something else - say, CUDA, or a distributed representation - and as long as the replacement also speaks Numpy's language, you can plug it into existing libraries that were originally written against Numpy.

In short: Fortran gets you performance. Numpy gets you that, and also productivity. I would argue that that actually makes Numpy more powerful than what was possible with just Fortran.

Try D language, with its numerical library you can get both productivity and performance that is even better than the OpenBLAS (Numpy and Julia library are based on) [1].

In D you get the consistency of a single unified language semantic unlike the impedance mismatched approach that is inherent in Python and Numpy programming combination.

[1]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

Numpy array syntax inconsistent to the degree it can be considered broken. For instance, to take elements number 4,3,2 of an array in Python, one writes A[3:0:-1], but for the elements 3,2,1 one has to write A[2::-1], because someone decided that making A[-1] refer to the last element of an array is "intuitive", and so A[2:-1:-1] will return empty slice.

Now if you want to use array comprehensions, the first case looks similar: [A[k] for k in range(3,0,-1)] but the second now has to be [A[k] for k in range(2,-1,-1)].

Further, for some reason, array and matrix are different types and one has to convert back and forth between them.

You don't need the matrix type in Numpy, it's just syntax sugar over a 2-dimensional array

The matrix class is now officially deprecated, FYI, so just use array everywhere.

I guess the generic way to write that would be A[bottom:top+1][::-1]. But the blame there goes to Python, not numpy, since the same is true of lists.

Ah, just the right time to publish about numpy - right when everyone is moving over to Julia because of numpy's warts.

I don't think that's accurate:

https://trends.google.com/trends/explore?date=today%205-y&ge...

*For certain definitions of everyone.

Me on a Thursday afternoon when I'm bored.

I'm interested in Julia, but if numpy has warts, I suppose Julia has not yet had time to even grow its skin.

Theoretically Julia is better for scientific computing, the only issue is its package ecosystem isn't as mature as Python's. But it's growing incredibly fast and there have already been libraries available for a few years that would be really impractical to write and maintain at such a level in C++ for Python. I assume that for science at least Julia will catch on a lot.

> Theoretically Julia is better for scientific computing, the only issue is its package ecosystem isn't as mature as Python's.

That's not a small issue. The ecosystem is probably the reason people choose NumPy over MATLAB, for example. NumPy is not inherently superior to MATLAB, and most academicians that adopted NumPy in the 2000's already had a MATLAB license, so cost was not a concern either.

Well that could have been said (and was said) about Numpy/Scipy when it started, "oh R has so many more packages, what numpy can do I can do in MATLAB ...", yet here we are.

You probably don't realize it, you are agreeing with me :-)

It depends on the definition of 'people'. There were many who adopted numpy much before the ecosystem had had time to catchup. But I would readily concede that Dr. Jones @national_lab didnt at that time, in fact he probably hasnt even now.

I do disagree strongly with the opinion that Numpy is no better than MATLAB :). MATLAB has adopted some Numpy features after Numpy came out (broadcasting for example) but Numpy offered some genuine and unique advantages, both technical (broadcasting, no need for a MEX compiler that I have to pay through my nose for, not restricted to weird naming conventions, nature of parameter passing, ...) and legal.

I just don't like the BASIC derived syntax of Julia (and Ruby.) I wish there was a language that was typed, had python like classes, subroutines and lambdas but JS like anonymous functions that was fast like Julia or at least close to numpy in number crunching without needing a module written in C.

Have a look at Nim, I was presently surprised when I recently tried it out. Now if there was just a better way of integrating with numpy it would be my goto language for writing computation intensive modules for python.

I started using python and numpy/scipy back then because it was vastly easier to deploy on a server or supercomputer. The matlab compiler meanwhile is clunky and adds new bugs and additional steps. Julia doesn't really match python in this regard either.

For more pure research and prototyping things both can do, I still think matlab is better though I rarely use it. I just like the idea of being able to easily deploy the code later somehow. Kind of an entrepreneurial feature.

In Julia, there is one package manager and it gets things right. https://docs.julialang.org/en/v1/stdlib/Pkg/ It's super nice to have no fragmentation when it comes to packaging. In Pkg, package states are immutable, always reproducible, and quick. Julia packages that have binary dependencies usually build them all for every platform using the binary builder infrastructure (https://github.com/JuliaPackaging/Yggdrasil). It makes cross platform installation robust and testable, and suuuper quick. Pkg really is the rolls royce of package managers.

I also like that package environments are integrated in Julia and thus there is only one way handle environments.

I think Numpy made it easier for people to integrate with a ton of open source libraries. Since everything is proprietary and the users are in a few specific niches, Matlab can't be as versatile. Also, each of the Matlab add-ons are another expensive license people are reluctant to pay. Sure, there are a bunch of contributed libraries for specific tasks, but comparatively the community is pretty poor.

> That's not a small issue.

In fact, it's not an issue at all since Julias ecosystem is a superset of that of Python: with PyCall you can use Python libraries and Julia libraries in one program without issues.

Julia interop with Python is insanely easy.

using PyCall

np = pyimport("numpy")

res = np.fft.fft(rand(ComplexF64, 10))

You just called numpy fft from Julia.

Yes, and the interface is pretty fast and without any noticeable overhead:

julia> data = rand(ComplexF64, 1024^2);

# python fft from julia:

julia> res = @btime np.fft.fft(data);

  78.613 ms (39 allocations: 16.00 MiB)

# python fft in ipython:

In [11]: %timeit res = np.fft.fft(data)

89.3 ms ± 1.65 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

As expected, julia has its own fft package (based on FFTW):

julia> res = @btime fft(data);

  61.540 ms (33 allocations: 16.00 MiB)

> its package ecosystem isn't as mature as Python's.

Python packages are either interfacing external libraries (something that is much easier to do in Julia) or if they are pure python, badly designed and buggy.

(and package management in python is broken beyond repair)

Interestingly, one of the Julia creators is one of the named reviewers.

Julia and the NumPy/SciPy community get along pretty well (or at least they did while I was still doing this stuff). In fact, I may have first heard of Julia when someone presented it at PyCon.

You can roughly divide people with strong opinions on Julia vis-a-vis Python into two groups: People who earnestly want them both to win, and people who are largely just watching from the sidelines.

Julia is a fine platform for data science but so are Python/NumPY, R, and Matlab. Few of us have the luxury of building greenfield projects that are independent of our previous choices and real-world constraints. Understanding NumPY within this larger context is important, even if you are committed to the Julia ecosystem.

Reply on news.ycombinator.com