Hacker News Reader

You seem to attribute certainl motivations to the PIs based on some misconceptions of the incentives and constrains for doing research. It is true that scientific software engineering is undervalued and this is one of the many issues in modern science. However attributing this to PIs wanting to do everything themselves is completely off the mark. One of the biggest issues in science today is that PIs can't do anything themselves, they, after having been trained as scientist for years, become glorified managers and administrators.

The reason why they can't and don't want to hire software engineers to build well designed software is because they don't have any money for it and it is very mismatched with their incentives. If a PI gets a grant for a project, they typically get a constrained budget to achieve the scientific goals of the project. There is usually no money (and it would not be approved) to engineer well designed software, because that doesn't produce scientific outcomes.

Now in the situations when there is budget for this, which are the super large institutional research initiatives, like e.g. CERN, they build incredibly well designed software. The SKA for example will have the largest storage cloud in the world.

If one wants better designed scientific software one needs to seriously overhaul the scientific funding process and quite significantly increase funding.

PI here (Research Professor of AI, former industry R&D director, so I have seen both sides). The money will never be the same in academia, and rewards are outside of the control of the PI in most cases (e.g. in Germany, it's a government-level decision and everyone is paid the same), so there are two options: work with average people or find smart & idealistic ones. So...

...please excuse this shameless plug: I'm interested in people who would like to work with me to build research software (both on-premise and AWS cloud) using best practices, in particular using Rust. If you are interesteed in machine learning, natural language processing and information retrieval and would like to work with me as a research software engineer, please do get in touch (leidner at acm dot org). Our new 2 TB RAM dev research group server running Ubuntu 22.04 LTS is waiting for you (CPU/GPU cluster ressources also available) to set up (a VM for) your CI infrastructure. Your software will be used to push forward the frontier of research in machine learning and its applications to language (summarization, question answering) and information retrieval (vertical search engines, learning to rank, unsupervised topic modeling).

Edit: academia can be poorly funded and bureaucratic. The former can be fixed by grant applications, the latter must be tolerated until politicians/activists fix it.

The fact that you included a job posting in your comment might be the best evidence in this thread that it's hard to hire RSEs. Good luck of course :)

I recently quit a job as a software engineer on a research project for all the reasons mentioned. They want grunts who don't care about project management or good engineering practices and they will belittle you as not worth the money even when you took a (theoretical, as I am overemployed) massive paycut to go and work for them. I made 1/3 there compared to the jobs I am interviewing for to replace it.

These people desperately need software help and have worthwhile projects that need software help. They also need to learn to collaborate over just parsing out obscure goals and being upset when what they get matches their three sentence email and not their vision in their head.

Yea, it's frustrating because there are indeed some super cool projects just waiting to have interesting software written for them, especially in the more hardware/physical oriented R&D areas. There are some areas that take software seriously, but in general, I have found it really difficult to get scientists and research engineers excited about anything that they don't know about. I don't think its malice. I think its part of their education and institutionalization couple a little with arrogance and somewhat blind view towards areas outside their expertise. I have given presentations before, and I usually only got interesting feedback from non-PI level workers. I always tried to paint a picture that we were doing state of the art technology development, so why shouldn't state of the art software be developed to support that?

It's very interesting that you say you find it difficult to get scientists excited about anything, because I am told pretty much the opposite. To give you the perspective, I'm a scientist and my partner is a GP. I obviously have many friends who also work in science and one thing that my partner noticed quite early after we met is that the scientist friends (and myself) would ask a lot of questions when she was talking about her work, and really tried to understand reasons, mechanisms etc.. She noted that this was very different to almost everyone else.

I can definitely tell you that if I talk about my work non-scientists become quickly uninterested.

In my experience, scientists do not like anything that they perceive as "soft". I like to say that the hard sciences are actually the easy ones and the soft sciences are actually the hard ones. (As it turns out, Herbert Simon also said this.) The point is that when you discuss things like design, complexity, architecture, systems thinking, etc., I've found that this does not stick, so to speak. Scientists do not see beauty in form equaling function. In my experience, scientists like learning about things they consider science-y enough for their tastes.

This is all painting with broad strokes, but in my experience, the most interesting intellectual conversations I've had have been with people with the least amount of degrees. I always feel when I talk to someone with a Ph.D., they're afraid to let it be known someone might know more than them and if that happens, then it's suddenly something that's not important.

The book Disciplined Minds gets at this some. It's part institutionalization and other things. My gut feeling is that it's exacerbated by publish or perish culture and the push to always be right. So people that are brought up through that are intellectually risk averse.

bmitc 1 day ago | parent | context | flag | on: Science needs more research software engineers

> In my experience, scientists do not like anything that they perceive as "soft". I like to say that the hard sciences are actually the easy ones and the soft sciences are actually the hard ones. (As it turns out, Herbert Simon also said this.) The point is that when you discuss things like design, complexity, architecture, systems thinking, etc., I've found that this does not stick, so to speak. Scientists do not see beauty in form equaling function. In my experience, scientists like learning about things they consider science-y enough for their tastes.

In my experience what you say is true for engineers (and software engineers are amongst the worse in this respect), but much less so for scientists from the hard sciences (physicists, chemists, biologists). You can also see that from many famous physicists being involved in philosophy, while many engineers consider philosophy "irrelevant talking about nothing".

> This is all painting with broad strokes, but in my experience, the most interesting intellectual conversations I've had have been with people with the least amount of degrees. I always feel when I talk to someone with a Ph.D., they're afraid to let it be known someone might know more than them and if that happens, then it's suddenly something that's not important.

Maybe we converse in very different cycles, but generally for people I know the "higher up the foodchain" the more they ask even very basic questions. In my experience you can often distinguish who is the most accomplished scientists in the room by listening who is asking the most basic questions.

All this is obviously anecdotal evidence, but is reflected also by famous accounts like e.g. Feynman's account of the challenger commission.

On the flip side of that, many experiments run for decades or longer, and "state of the art" software is often a poor impedance match for that.

State of the art doesn't necessarily mean React. Just something with unit tests and more than one giant 5000 line file.

If the requirement is that it lasts 30 years, you can choose more timeless languages like C, over say Rust.

I guess a better word for that might be "not terrible" :).

And yes, way too much research code reads like somebody's first coding project.

I'm not convinced of that. Because what I've seen (several times) on a somewhat small scale perhaps, is someone's code, sometimes written decades ago, never maintained upgraded or properly developed used for years on a system and eventually it gives out, forcing basically a rewrite. And state of the art can mean different things for differently scaled projects.

I've certainly had to resurrect research code written 10+ years before I got to it. It's usually not too hard as long as it's in C/C++/Fortran and doesn't have too many dependencies. Something like Matlab, on the other hand, is often a nightmare...

You have too look at the big picture though. How much time would have been needed to train the PhD student (might have been even the PI) to write more maintainable code 10 years ago, and maintain that code to "best practices" over the time, compared to write that it works once and need to rewrite once after 10 years?

Moreover, people tend to forget that scientific software evolves much more slowly than commercial software and the number of people who work on this are much fewer, so why always adopt the newest method, if the code is still working. Show me commercial projects that have had the stability of something like LAPACK or BLAS, those essentially had a stable API for > 50 years now, with very few bugs.

It's okay for things to start off rough, especially depending on the context. But there's often not much excuse for things to be taken seriously and evolved.

But to be honest, I've seen this in industry as well from software developers. So maybe it's just a law of software development that's amplified in scientific software.

Are you saying that you hold multiple full-time jobs?

> They call it "codes"

They/We have probably been calling it that since before you were alive. I wouldn’t be surprised if it comes from the 60s or 70s.

Every group of people has their own terminology for stuff. There is no right or wrong.

Yeah. Considering how much time we spend developing and using the damn things, we’ll call them the way we like, thank you very much. We don’t really need condescending software types to just come and explain to us how we’re saying it wrong. That is particularly rich, considering the tendency of CS people to cargo cult and borrow physical terms without understanding them.

Always annoying to hear machine learning people speak of "tensors"...

I'm still waiting for std::pseudovector<T>

It aggravates me too, however, considering that the definition of a tensor according to mathematicians is an element of the tensor product of two vector spaces (or whatever other objects you can tensor together), and according to physicists would be an object which transforms like a tensor, I’m somewhat sympathetic. Neither definition sheds any light on what a tensor is to anyone who doesn’t already understand what a tensor is, and I’m convinced that the moment one understands what tensors are they lose the ability to explain what tensors are.

I have an academic mathematics background and I'm perfectly OK with this.

Yes, please stop naming your software projects after elementary particles :).

Sure, there's no absolute right or wrong on terminology, but it is a sign that they're cut off from the software engineering world, which leads to a whole host of dumb practices, described elsewhere in the comments here.

As someone with a mathematics background, whenever I encounter someone in another community using mathematics (like economics or accounting or finance or whatever) using mathematical terminology that seems silly to me, it's a bad sign.

I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense. Where have you seen that used?

I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense.

Why not? "A code" is a software package which solves some kind of problem computationally. Plural, codes. I'm sure it's been in use almost as long as the programmable computers. Here's[1] a link to an article from 1968 which uses the term in the singular ("The MENE Neutron Transport Code".)

[1] https://www.osti.gov/biblio/4819611-mene-neutron-transport-c...

That article appears to be written by a non-native English speaker. I don't think I've ever heard anyone refer to a software package as "a code."

I don't think I've ever heard anyone refer to a software package as "a code."

Then you probably haven't been around physicists, or read enough physics-related material. A non-scholarly reference which can't be easily checked online is George Dyson's "Project Orion," where Freeman Dyson (author's father) is quoted as mentioning this or that "code" as significant for the design of various Orion parts. (That should be a native enough English speaker.) The time-frame is late 1950s to early 1960s.

Actually I have been around physicists. In my experience they use “codes” for any amount of code. If A wrote some software, B might say “I’d like to use your codes.”

That reference you give doesn’t appear to be available online but I’ll take your word for it.

There are plenty of words in English that have the same singular and plural form.

I've seen it a lot outside of physics.

Obviously, what word they use for it is arbitrary but it's indicative of an actual problem - they're all totally cut off from the software engineering / CS community. Lots of them think programming is like learning to juggle or ride a bike or something, a skill that anyone just learns and then they're done. Then, obviously, they make major mistakes that invalidate their conclusions and nobody notices or cares because it's all student code, all the time.

It’s used fairly often in the US national lab community (Not just among physicists).

I think the point is that the physicists were among the very early adopters of computing systems. I’m not sure about the history of the term “code” used this way, but it might literally predate compiled software since Ulam and von Neumann invented the Monte Carlo method in 1948 during the Manhattan Project.

Personally I think it just sounds outdated now, but it’s pretty common to hear people call their software “a hydrodynamics code” or “a molecular dynamics code”

Now I do wonder about the history of the term.

The tendency is to call computational science software “codes”, but I’ve never heard a scientist call, ie a web browser, a code.

There’s a different connotation. Like how theres a different connotation between a program, an application, and and app. The are largely synonymous, but have slightly different uses.

It's pretty widely used in "traditional" engineering, at least in mechanical/civil/nuclear/aerospace. My impression is that it's probably a holdover from the good old days FORTRAN-IV.

See for example Code_Aster (as in, "a code"), which is one of the largest projects for open source FEA (and it mostly rooted in academia).

All mechanical engineers. All of them. Even that guy, who corrects himself when you're in the room - as soon as you're gone, they're back to the codes.

I'd encourage you to have a look at a well-known European physics research laboratory's (rhymes with Bern) library and engineering departments (not IT, unless they've started using version control and tests since I worked there). The salary is not as bad as academia, and there's plenty of interesting stuff going on. Most of it is accessible to anyone interested in a way I've not seen outside of academia.

Certainly cool stuff but out of the question due to location. I've worked for or met with people from similar size (at least in ambition) projects. It's a tough world to break into and especially difficult to bring a different outlook to.

I can understand not wanting to move very far, but the location is in many ways great: French alps within driving distance, Jura within walking distance, the lake, old architecture, lots of fields and forests. World-leading public transport. Geneva itself isn't very exciting, to be fair, but you're in the heart of Europe with easy access to all the rest.

As for "breaking in," I dunno. The hiring process seems very fair and international (very few locals work there in a technical capacity AFAICT), and there are plenty of people with flexible ideas. Not everyone, for sure, but then there are so many different groups there you couldn't possibly blanket everyone with that statement.

Have you considered US's national labs if you are in the US?

Hard to get into, have to move to one of five places, usually requires security clearances (huge ordeal), extremely clean lifestyle, if your PI is evil expect zero protection, post doc can be considered entry level qualifications, can be asked to work 7 days a week. Ultimately, usually still pays less than entry level software positions offering full time remote work...

> They call it "codes"

This is not really the core issue here, but yes, I've encountered this many times, and I have to admit that in some professional contexts I use "codes" as a shibboleth to identify people who aren't really in touch with the software engineering world.

I always assumed it came from big international physics experiments where English is most people's second language. People who learn to program in an expat community are going to pick up some jargon that will sound weird to native English speakers in California.

I wonder if this used to be the more common term, and at some point we switched to “code”. Physicists were among the first serious computer users, but their software culture may have bifurcated from ours at some point.

I think you accidentally revealed the problem when you said "best practices", because asserting such introduces a presumed total order on expertise, and that introduces conflict. thus ends the discussion between a PI and someone telling them they know better?

It's pretty easy for me to acquiesce all design decisions to their expertise and/or role in addition to being excited by what they do. I'd just like that to be reciprocated.

See this infamous discussion: https://www.reddit.com/r/MachineLearning/comments/6l2esd/d_w...

Tldr: They view software as just a tool and don't see how they'll benefit from following best practices.

Oh wow, looking at that Python code they linked to brings back memories of my ML course in college. We had skeleton code given to us to fill in for the assignments, and often times it took longer for me to understand what the code they gave was even doing so I'd know what to fill in than it took me to actually write the code. Occasionally there would be random semicolons at the end of the lines, but the author seemed to understand that wasn't necessary in Python since they weren't on most lines, so I guess they were just careless. They also used a very amusing "OO" pattern where they would pass in every input needed to the constructor of objects, then have one method pass all of those instance variables directly into other methods which ignored the fact that they had a `self` parameter and just used the copies of the instance variables (with the same names) passed in as parameters. The biggest confusion I ever had was in one method where they used the typical `n, d = matrix.dimensions` (I can't remember the exact incantation, but it was used in all the code to store the dimensions in `n` and `d`, immediately followed by `d = d + 1`, which I took to assume that the leftmost column of the matrix was just filled with 1s and wasn't needed (which happened in some of the algorithms), but then down in the implementation of the algorithm, the only time `d` was used was as `d - 1`. Because I had mentally skipped over the boilerplate at the top of the function, I spent a decent amount of time trying to figure out why they had subtracted 1 from d until I finally saw the increment above. To this day, I still struggle to decide if the author legitimately did not notice this and just used `d - 1` to try to fix a problem that they didn't know the source of, or if they knew and thought that it was a reasonable way to implement things, and I'm also still conflicted about which would be worse.

You can write FORTRAN code in many languages

Good craftsmen care for their "just tools", this is delinquency.