Hacker News Reader

Comment by jofer | Hacker News Reader

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely useless". It was also always unpublishable, as methods are never publishable in my field. Numpy has (obviously) vastly more respect and impact than my work, but the general problem remains.

Articles like this are a _huge_ deal for that reason. It's an immense delayed recognition for over a decade of work from a lot of folks.

View on HN

I recall a story where a friend was unable to publish a paper in which he wrote an alternative to a very commonly used commercial tool (that virtually everybody used) with roughly 10 times better performance. He open sourced it and all, it was extremely useful, but there was no new methodology, it was simply very well implemented.

At a talk of his it lead to a very heated discussion where an older professor accused him of wasting government money on such nonsense.

Been there. A few years back I got a government scholarship for my PhD (which is still in progress, due to my follow up work). I basically built the foundation upon which to establish a new field for my university, and the region where I live. There are some professor who think that scholarship (and the little money it gave me) was wasted on my because I chose to build all of that from the ground up, instead of rushing through my PhD.

By the way, those of that opinion are all professors who wanted me on their labs, but I turned them down...

For every story like this, I believe there are many more in which the student simply writes their own implementation due to not invented here syndrome or engineering as a form of procrastination.

If you talked to me about my PhD for a few minutes you would surely put me into your "had to reinvent the wheel for no reason" category.

As indeed, I wrote an analysis framework for my data (of a gaseous detector used for axion search) [0] instead of using an existing framework used by my predecessor. However, things are always more complicated than they seem. Many of those not talked about students who rewrite stuff probably have reasons!

In my case the existing framework [1] was a monster that was bent to allow it to work with the kind of data we have in the first place. In my case my detector had several additional features, which fit _even less_ into the existing framework. It would have been a hack and still a significant amount of work to make it work well.

To be fair, when I started this I expected it to be less work than it ended up being. But that's the story of software development.

The advantages now are significant of course. I know the whole codebase. It does exactly what I want. I can extend it easily as I see fit.

That doesn't mean I didn't also partly procrastinate writing software. Far from it. Hell, there was no reason to write a freaking plotting library (a sort of port of ggplot2 for Nim) [3]. But again, this means my thesis will have plots created natively using a TikZ backend while at the same time provide links to Vega-Lite plots for each and every plot in my thesis (which of course will include the data for each plot!).

Finally, the most important point: A university / professor who only pays me for 20h a week does not get to tell me how I do my PhD.

[0]: https://github.com/Vindaar/TimepixAnalysis [1]: https://ilcsoft.desy.de/portal/software_packages/marlintpc/ [2]: https://github.com/Vindaar/ggplotnim

I certainly have experienced similar things, particularly been acused of reinventing wheels. Flexibility and performance are two big reasons, but also "it's fun" or "I want to understand X" also have a good weight when we do this kind of "useless reinvention".

Maybe I wasn't clear enough. When I started my PhD, I was working on leading edge, basically 3 people in my country knew that we were talking about (and I was one of them). It certainly wasn't NIH-syndrome. Still, instead of "bailing out" on the easy path (present a paper here, work with that professor in That Other Thing That Doesn't Interest Me, etc) I chose to keep doing what I love.

End result so far? I'm quite respected, still one of the leading researchers in my country on my specific topic, but since I don't have a PhD (because of the aforementioned delays, and some grumpy professors actively pushing against me) I'm starting to lose access to grants and programs.

I'd still do it all again, but with a few tweaks here and there, you know hindsight always helping.

Yes, it is easy to be sidetracked on writing software. Not that software is a bad thing, but research is something different.

Given the current publish or perish culture, I doubt any student serious about publication can afford to waste time.

It is obvious that we need good software, however from the point of view of science the old professor may have reason. If you are receiving a grant, you're not being paid to write software, in the same way that an engineer is not paid to write novels. As useful as the software may be, the person in question should be spending time on research (by definition new subjects), not writing again an existing software.

> If you are receiving a grant, you're not being paid to write software

In my (albeit limited) experience, software is a pretty common deliverable from a grant, at least in computational biology. This has also been my experience with more alternative funding sources like CZI and DARPA.

Taken more broadly, I think there is a huge disconnect between what academics are paid to do, and what takes most of their time. Review is unpaid. Grants are not dependent on which journal the results go into, but time could be saved by aiming lower. A salary can be payed from a research grant, while the investigator still has to teach.

What if that piece of software increases research output across the entire field? Often, a good piece of scientific software advances research more than what you're calling "research."

Writing software is sometimes necessary to achieve the objectives of a grant (even though this is not necessarily explicit). It’s not writing software that’s a problem, it’s reinventing the wheel; you should not focus on “scientists should not write software”, because that is obviously far from the truth.

For a scientist, writing useful software is a good way to get exposure, build a reputation and get citations. It’s an opportunity to do some different kind of problem solving than usual. It’s also a way of understanding how the software really work (which assumptions are built in, which methods are used, and how does it affect the software’s results?). This does help improve the quality of subsequent results.

A grant typically (there are exceptions, of course) lists things that are going to be studied. How the studying is done is typically down to the people doing the work. It certainly isn’t for grumpy old professors who hear a talk at a conference to judge.

> my work was "utterly and completely useless

I can never understand the arrogance of folks who would say something like this

I used to maintain a code beautifier/diff tool and hear the same things about how the idea was useless only to see those same people shortly there after use my tool or a close competitor. Once you see that pattern a few times you learn to ignore it in it’s entirety.

It’s hard to tell why people behave like that. I presume it’s because many people have a great fear of originality and require social validation.

Asymmetry between complexity of principle and power.

Ie., we are easily persuaded that something very complex will be very powerful (eg., a smart phone) -- but we intuitively regard something simple (eg., a hammer) as under-powered.

Hard to say how well this actually holds, but I'd guess in both cases we arent really enumerating use-cases in our head, we're just using explanatory complexity as a guide to practical power.

This is probably more extreme in cases where people have a specific notion of complexity in mind, eg., in academic environments where "tool A" is as simple as "tool B" if they use the same theoretical basis.

ie., Tool C is worthwhile if it includes a more complex theory, as therefore it is more powerful.

Academic administrators judge performance based on publication counts, journal impact factors, citations/h-index, and fundraising. Working on tooling doesn't fit in those buckets, so it's broadly-speaking "useless" to a researcher vying for promotions (eg tenure). It's an imperfect method of measuring true impact.

And to be fair, that was the context of the comment. It was meant to be harsh but true advice. It was quite arguably, at the time, worthless in the context of advancing my career. It turned into my career, eventually, but that wasn't the goal then.

as a former phd student, it's pretty systemic - labs sometimes get funding as a direct mapping with how many papers get published by that lab. If you do cool work but that does not land you a paper, you are literally wasting the budget that your employer spent on you. Once you solve that (and thanks to things like this numpy papers, things may be starting to evolve !)

I know Travis' and Paul Dubois[0] work at Livermore was immediately recognized as of towering importance almost immediately. I was down the highway at LBNL porting shitty Mathematica, IDL[1] and Fortran Diffraction Grating code to Numeric or whatever they called Numpy back then almost as soon as it was released. People probably don't remember their history, but pretty much the only open source intepreters of the day were things like Perl (whose math capabilities at the time were pretty lousy). Scientists paid for a shitload of Maple, Mathematica, IDL, Matlab and Igor[2] licenses; and there still weren't enough licenses to share code with your friends, because nobody had licenses for them all.

Python 1.5 was the first non-mentat tier open source interpreter available that didn't get in your way as a scientist, and Numeric/Numpy was the first and still the most elementary piece that made it usable to science and numerics people. Might not have been letters to Nature tier back then, but Nature ain't what it used to be anyhow.

[0] Since a lot of folks have actually forgotten Paul: http://www.pfdubois.com/bio.html

[1] https://en.wikipedia.org/wiki/IDL_(programming_language)

[2] https://en.wikipedia.org/wiki/IGOR_Pro

props to your work and similar to numpy, i assume it has been immensely useful for loads of people.

but 'building the underlying infrastructure that tons of people use' is not science. in my department we had to fail a phd student because 90% of his work was just implementing bunch of existing methods as a python library. useful, yes; science, no. wasn't his fault, had a shitty supervisor, but making useful tools is not the same as undertaking scientific research.

That's quite the wrong way of approaching science. The scientific method is based on building on the shoulders of giants. Those giants aren't the professors in the direct vicinity nor are they only the papers you cite. The whole of the process is science and if we need further specialization for building better tools (hey, maths and statistics are scientific tools as well) I would classify that as science to a large extent.

What could be useful is openness in used tools and software and a way of getting citation counts for software used. It's nothing more than a table. That way the hotness of publication could start to flow for the underlying tools.

> The scientific method is based on building on the shoulders of giants. > The whole of the process is science

no. scientific research is proposing a useful model of an observable phenomenon. this is what you train for during a phd, at least in natural/life sciences: you learn how to test a hypothesis, not an easy skill.

refactoring code or transforming bunch of C++ into a python library is useful, but it's not science.

> What could be useful is openness in used tools and software and a way of getting citation counts for software used. It's nothing more than a table. That way the hotness of publication could start to flow for the underlying tools.

agreed 100%

I'm trained as an economist so might have a different view. But what I think I know from physics is that, say, the people actively involved in engineering things like matter collidors do get authorship or at least appreciation for their role in furthering science.

For me, our discussion is mainly in where to draw the line around "the process of science". The chair, laptop and coffee machines aren't science. The statistical methods, papers and engineering are. You seem to cut parts of the engineering out, namely the non-novel parts. There's a lot to say for that. But a PhD is proof of apprenticeship as well. I wouldn't grant someone a PhD if all of his work is 'mere retooling'. But in a mainly research papers based PhD-application I wouldn't feel some retooling couldn't be allowed. One could demonstrate scientific craftsmanship in retooling.

oh i completely agree that a binary distinction between 'tools' and 'science' is not useful. it's also extremely hard to be a good scientist without being very good at 'tools'. however, editors at peer-reviewed academic journals or people awarding degrees mainly need to ask whether the work has advanced our knowledge on X.

if X is e.g. microbiology then it's fair to ask whether (1) some python library proposes something in terms of microbiology, and (2) bunch of biologists should make that decision.

this is why refactoring code is mostly dismissed as 'doing science' by most phd supervisors. sure counts as 'developing skills', which certainly should feature prominently as part of your training, but it cannot be all there is to a project.

Maybe we should take all this "not science" software away from the scientists and see how much science they can do without it.

If you write code that allows science to be done that couldn't be done otherwise then that is science. As a high profile example, a large amount of specialist software was developed for the LHC to allow it to process all the events coming from the detectors.

It sounds like the refactoring here was not really that useful in the first place.

yes, in 2020 you mostly cannot do science without software, electricity, desks and chairs and buildings, printers, pick your own irreplaceable tool. yet building these things to enable research is emphatically not itself scientific research.

doing a phd -> training to be a scientist.

Printers? Desks? Take your strawmen somewhere else.

Since you brought up LHC, here is an anecdote.

I worked on software development tools used directly for LHC as part of an internship.

That experience was of zero use when I tried to apply for a PhD later. It did get me several $BIGN internships though.

Make what you want of this story.

It's very true that it's not scientific research. (And I actually generally agree with the premise behind "methods shouldn't be publishable -- they're for appendixes, not papers".)

However, there's increasingly a role for folks focused more on the scientific computing and methods side. E.g. "how do we constrain X parameters given Y observations" (yes, I just described inverse theory -- that's deliberate). The science isn't solving the problem, it's figuring out what models to use and what the inverted parameters mean. However, solving the problem correctly requires a lot of rather novel work and is very easy to get wrong.

It's similar to many other research staff positions. It's standard to include the person who operated/designed/etc the instrument you're using as an author on papers. Is it that crazy to include the person who developed the numerical methods and implemented the solution as well? For example, I have quite a few friends that stayed on as staff to run the lab or key pieces of equipment. They have tons of "middle author" publications as a result.

However, numerical methods and computing infrastructure and work is much less frequently recognized. This is a step towards changing that.

This comment displays a remarkable ignorance of scientific history. Why do you think Ramon y Cajal shared the Nobel prize, for discovering neurons, with Golgi, who 'simply' invented the staining method?

Inventing a method isn’t the same a re-implementing an existing method! Again, useful re-implementations are a good thing that people should be rewarded for somehow, but they are not new science.

I am really sorry but most “science” coming out of even the top institutions these days are sadly uninspired garbage regurgitations.

most of anything is 'uninspired garbage'. not sure what it's to do whether a particular phd should be awarded.

no-one is proposing that numpy isn't useful or people developing / maintaining tools aren't doing gods work. they have my endless gratitude and try to donate regularly.

however, phd training in my field -- natural/life sciences -- has a specific remit: you learn how to build and test a hypothesis, from start to end. optimising libraries is emphatically not it. as a scientist you should care whether you have a useful model that explains something about the world. this is orthogonal to how neatly you have implemented your linear algebra in python.

I wholeheartedly agree.

I think journal editors have a responsibility here too in promoting references to software libraries used in the articles they publish. I almost never see these in my field (astrophysics), even though they are readily available and very easy to include.

We usually manage a Somethinginformatics journal publication to detail infrastructure work. At least Zenodo etc. And annoy users of the software to cite (a doi isn’t much to add to a manpage or log output).

> for over a decade

Probably two even if NumPy came out in 2005

This so much. I cannot stress enough what an impact various python developments have had on the economy as a whole. Sometimes the right tools can inspire people and that’s exactly what happened.

Reply on news.ycombinator.com