Hacker News Reader

Comment by greydius | Hacker News Reader

I do this kind of work. It's frustrating how awful most of the code is. You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems. I can spend a week just trying to get some software to build.

I could keep complaining, but I don't want to give people the impression that I don't like what I do. It beats the hell out of writing CRUD apps.

View on HN

> You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case.

Another way of looking at it: People doing STEM PhDs are extremely busy. They have to master the content in their field, keep up with new research, conduct their own research, write papers, go to conferences, give talks, teach, keep up with whatever department service responsibilities they have, etc.

They're smart people so they're going to be able to slap together some code that does what they need it to do. But to expect them to learn git and how to use it well, to learn about managing dependencies, about unit tests, about build systems, about best practices for documenting code, about programming abstractions or OOP whatever else... that's a lot to ask.

(And I say this as a STEM PhD who did learn all that stuff, more or less.)

When I was writing code as part of my PhD/research, even if I knew how to do it well, I didn't have the time to do it properly. I still remember writing a specific algorithm to compute approximate rotation numbers and slapping it in the middle of the code that had to use it, because I didn't have time to refactor it to do it properly in an easy way (would need to pass around too much data). It did what it needed to do... and luckily, I didn't need to get back to it, since it would be a steaming piece of code (in C).

If I had to do this kind of things again, I might do them properly, but I'm not 100% sure. Back then, I had more pressing things to do than writing excellent code, since I was writing code to drive the research (as in, numerical investigation), not as proper research (the end results were theoretical results)

I would say if unit tests, build systems, documentation, abstractions, modern languages, and so on are valuable for productivity, reduced bugginess or ease of access for the next developers then the incentives exist for academic researchers to learn and use these tools.

I think the real problem in academia is the same as the problem in some organizations that also don't follow best practices: the people empowered to make the decisions about whether to invest people's time into improving testing, building, technical debt, or documentation do not actually believe that an increase in productivity, increase in access or decrease in bugginess will be achieved.

> I would say if unit tests, build systems, documentation, abstractions, modern languages, and so on are valuable for productivity, reduced bugginess or ease of access for the next developers then the incentives exist for academic researchers to learn and use these tools.

The time to learn all of this is almost like a field in itself say Software engineering.

I just need to get shit done.

I had a project to do bayesian hierarchical modeling. I wrote it from scratch in R without using a MCMC framework.

Yeah it's ugly, yeah it's slow. But I got initial results to satisfy my mentor and within the time limit of the summer internship. This is on top of learning two semesters worth of Bayesian statistic in 3-4 weeks.

Time isn't expendable and I can't devote my life to every single thing while neglecting other aspect (love, health, etc..).

Yeah your theory is nice only if you don't have a deadline, a life outside of work, etc...

>Yeah your theory is nice only if you don't have a deadline, a life outside of work, etc...

Not all academic disciplines are as hectic as yours. When I was in a top grad school, most of my colleagues were not under the stress you describe. And yet their code was still crap.

Incentives are a bigger reason than workload.

> When I was in a top grad school, most of my colleagues were not under the stress you describe.

Serious questions: Where was this amazing place? When were you there? Was it a STEMish PhD? Really.

I've never heard of a grad school in STEM these days that is anywhere near like that. I'd love to know because the way STEM PhDs are going in the US these days, I can't recommend them in general, only specific PIs and labs.

>Serious questions: Where was this amazing place? When were you there? Was it a STEMish PhD? Really.

Yes. Engineering. I don't like to reveal too much about my past, so I won't say the name. Suffice it to say that it's ranked in the top 10 for pretty much all engineering + many of the sciences. EE, CE and CS are all in the top 5.

It was a decade ago, though.

Workload all depended on your advisor and funding landscape. Some were very demanding, and they had miserable lives (mostly pointless, too - most of academia seems to be converging towards a race to the bottom). But not all professors were that way. Of course, standards still have to be met, and laid back professors' students usually took longer to get a PhD because they were more relaxed.

To give you an idea, two of my fellow group mates are now faculty members. And the one whose work was most like what you describe, and who did the best work IMO, never was able to secure a tenure track position.

The lifestyle you describe is really not worth the PhD you get in the end. The ROI is just not there - financially or otherwise.

It really does seem like luck is the thing, even in something like the ivory tower. It stinks, but all that hard work is the price of the lotto ticket; it still ends up being just plain luck in the end.

I also feel like it's a race to the bottom now. It used to be 'publish or perish', now it's 'funding or famine'. The paper numbers, impact, and truthfulness have nothing to do with anything. I see so many PIs that essentially end up as one income households, their spouse's, not their own. The 'good ones' just get pushed out or leave, an evaporative effect, leaving only the liars. The academy is in real trouble these days. It isn't on life support yet, but it's not able to climb the stairs anymore.

But most academic projects get thrown away after the paper is published. These things help software maintenance. But if I never need to maintain this code, why worry?

Even in computer science a lot of this holds true, and you're lucky if you find someone who both knows the specifics of their niche of the field well and knows how to write clean code.

I once had a quite respectable lecturer working for me, and his code was awful. But that was ok - we knew that. He was hired for his conceptual skills and domain knowledge, and we paired him with a junior developer that could take his raw output and turn them into cleaner code.

If you don't have the time to do it right, how are you going to find the time to do it all over again?

In academia, you don't have to. Make the prototype, write the paper, present it at the conference, job done.

It's some engineer's job to do it properly in an actual production setting.

> Make the prototype, write the paper, present it at the conference, job done.

No. One's point is to communicate one's ideas. Code is the only specification of a system detailed enough to reproduce the desired results. You wouldn't present an unfinished paper written in the equivalent of crayon so that particular attitude is the worst kind of lazy bullshit for people actually trying to make use of your hard work.

> It's some engineer's job to do it properly in an actual production setting.

Fuck this attitude so damn hard. As an academic, one _is_ the _first_ engineer who has a duty to lead one's colleagues - to mark as clearly as possible the path for those who come after. This "fuck you, got mine" attitude is unacceptable and unsustainable.

Don't worry about it, the vast majority of the code won't see light of day, there is no engineer who has to fix it for production, there is no production.

The idea is everyone writing everything the "right way" from the get-go would be premature optimization.

> The idea is everyone writing everything the "right way" from the get-go would be premature optimization.

The right way in this case doesn't mean AdapterFactoryBeanFactoryService-land. It means taking care that your code is clear, concise and correct for others skilled in the art. The fact that academic code exists at all, as an example, means that it _is_ in production. There's no excuse for shit code that isn't written for other human beings first and computers second.

> People doing STEM PhDs are extremely busy

We're all busy. How long do you think it takes for time spent learning good software development practices to pay for itself? Surely not longer than the length of a PhD.

> We're all busy.

Sure, but if the core of your job, the key skill you bring to the table, is writing software, you are (hopefully) going to find the time to learn best practices for writing software.

If that's not the core of your job, if your key skill is something else and writing software is just an occasional means to an end... maybe the incentives aren't there.

> How long do you think it takes for time spent learning good software development practices to pay for itself?

If you almost never write a program longer than 1000 lines and rarely use a program six months after you've written it, it may never pay for itself.

Don't get me wrong, I personally have an interest in good software development practices. I just understand why not everyone can or will take the time to learn them.

Academics writing sketchy code is like software engineers writing sketchy docs. We all agree it's not ideal but we can understand why it happens.

Not like a STEM PhD, you're not.

When is your first and last meeting of the work week? 6am Monday to 7pm Friday? Have you taken more than 3 weekends off in the last 12 months? Do you measure your daily coffee consumption in gallons? And yes, I'm serious.

PhD-land is crazy, perhaps this is why sooooo many drop out.

I moved from CS into Earth science a decade ago and I was similarly appalled at the state of affairs in the beginning. First, this doesn't have anything to do with how smart people are, it's a question of time and other resources. Our job simply isn't to write pretty code, it's just a means to an end. If you know how to do something in FORTRAN, why would you invest into learning how to do it with <lastest trend>? It's a huge cost! Second, the evolution of such code occurs often over decades; as the article states, with students and post-docs hammering along until it finally does what they need to get their project to move forward. The code usually begins with "Do X" and then the rest of the alphabet is tacked on later. We all know it's bad. But in the end, it's usually only your own lab group using this stuff; maybe a handful of people around the world. The cost of polishing something that started out as hastily written code to get something done in time for {conference,paper,proposal} a decade ago is simply to high. It works. Done.

I wish the state of affairs was prettier, but I doubt it will change any time soon. And frankly, I don't think it has to. Software engineers will always find something to complain about others' code (coding style discussions come to mind). If there's value in commercializing scientific prototypes, experts should rush in and do it. There's no need for us to invest in perfect code just in case someone may want to reuse it. Having said that, NSF is starting to fund initiatives to improve the state of affairs. For my field, it's EarthCube: https://www.earthcube.org/ but this seems to be fumbling along; all of their goals would need to be adapted by scientists outside of that community (standards, etc). Without the right incentives (funding, publications), it's just not going to happen.

So, lots of low hanging fruit.

Can you identify (say) five things that academics can do with an existing code base that will make your job easier (and therefore cheaper to the downstream users)? Perhaps pitch the top five pick at the abilities of a computer science student looking for a project?

Can you identify (say) three key practices to adopt when starting a new project given the likely skill levels of postgrad students in your domain?

You could get an ebook out of those I imagine.

I don't have time to give a fuller answer right now, but if I were to write an ebook as you suggested, the recurring theme would be "design for collaboration." Make your project easy for others to use and contribute to. And, contrary to what many people have stated in this thread, I think this aligns with the incentives of academic research. If someone else can easily extend your project with some new ideas, then you've just co-authored another paper. In addition, projects that gain traction beyond your own research group are more likely to get funding.

1) test suite - this shouldn't require explanation

2) documentation - both top level docs with examples and well-commented code. keep track of units

3) one-step build - I should be able to download the project, type 'make' (or something equivalent), and start using it

> design for collaboration

You have a different view of how academia works than most academics.

Most PIs these days do NOT collaborate and are actively hostile towards it. It's a shame, but it's true.

Also, the niche-ing of academia is real, in many fields, there may be only 3 other people on the planet that understand what you are actually doing, and many of them may not speak English and you may not know they are out there until 2 years from now (publishing takes a long time).

Most code is written by grad-students that barely know what a for loop is, let alone how to use git, and they only write it to do it once for a specific paper. Look into MatLab, that is the most used language in bio, by far. It's basically psuedo-code that compiles, and it's still a mystery to most grad-students.

> So, lots of low hanging fruit.

Well, yes and no. Speaking as someone who has done this as well: if it's UI stuff, or gluing processes together, it's fine. But sometimes they want you to fix up a custom algorithm they implemented. Which can be one of those cutting-edge scientific things that actually require a PhD-level of knowledge and insight into the topic. So you can only get half-way with untangling the spaghetti mess without sincerely not knowing for sure if you broke something or fixed a bug, because you don't know what the correct output should be.

1. Use version control

2. Have significant test coverage

3. Have at least a rudimentary form of Continuous Integration, even if it's just running a script on every check in.

Proper version control would already be a huge gain.

You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems.

Good developers get paid enough, that it indicates that maybe our job isn't quite that easy to learn.

Our basic programming abstractions and software engineering practices are things that we as a field developed over time, based on experience; it's not like they're something you just automatically know without having to sink time into studying if you're above some IQ level. They also depend to varying degrees on how big your program is, which parts are expected to change more or less rapidly, what kind of program it is, etc.

Unit tests assume you know what the code is supposed to do before you write it, that the code will change significantly more often than the required functionality, and that the tests are easier to read and check than the code being tested.

Agree with everything except for unit tests, they're something of a cargo-cult fad. Any of my developers caught wasting their time on these would get a stern talking to.

As a physics PhD spending most of my time on writing code for numerical simulations, I generally agree that unit tests are a waste of time for most things. Unfortunately, most people nowadays seem to think you either fetishize unit test coverage percentages or don't care about code quality at all.

In practice, at least for simulation code, functional, integration and regression tests are useful when employed judiciously. Most importantly, you verify your results using published benchmarks in the scientific literature or analytic results where possible. Obsessively covering every trivial bit of code with a unit test of its own has always struct me as rather a weird fad.

"Obsessively covering every trivial bit of code with a unit test of its own has always struct me as rather a weird fad."

The advantage unit tests have over regression tests is that the time between the moment developer implemented a change and finding out they broke something is as small as possible. When tens of people work on the same codebase this saves enormous amount of effort.

Fethisizing over some rule for it's sake is silly of course. My org would waste a lot of money and time without unit tests.

Another feature unit tests provide: Think of the unit tests as a living documentation. Breaking something leaves a breadcrumb trail to the invariant which was sullied.

I'll happily grant the point that unit tests are, to a considerable extent, the price you pay for putting more developers in charge of the same parts of the codebase. In my line of work that's usually not a concern, but I can see merit in the idea for large teams. I also think that statement scales down in the sense that unit tests rarely make sense for code written and maintained by one or two people.

"I also think that statement scales down in the sense that unit tests rarely make sense for code written and maintained by one or two people."

I am maintaining several end user critical projects related to data transform and transport, mostly by myself. I have no idea how I could develop them as efficiently without unit tests to verify my changes don't brake some corner case as new features are added.

The cadence for changes can be fairly low - I might return to some component after doing something else for six months, and I probably need to add some feature without breaking something in the process.

Sure, we have integration testing and smoke testing, but the further a bug travels the release pipeline from the developer the more expensive it is to fix. This cost cascade is quite easy to visualize - the work of whomever caught the bug is stopped, and depending where they are located in the product ecosystem their stall can cause lots of work for other people. If they are a tester they need to file a report. If they are a customer, their work is interrupted, they contact local sales, who then contact global helpdesk, who then identify and log the bug.

Much simpler and easier if there is a unit test to catch the bug in the first place.

Now, there can be domain specific flavors to this. My domain is computational geometry and transport of 3D modeling data between domains. But in my domain anyone not securing their code with unit tests is wasting their employers money and setting their end users at risk by increasing the likelihood of bugs.

There is lot of cargo cult nonsense in software engineering. Unit testing is not one of them. It saves time and effort by catching a range of bugs not caught by e.g. compiler for statically typed language and it secures the program logic for future changes.

Equating unit tests with 100% code coverage is a strawman. I don't think most people care about full coverage, and in some kinds of code it would be impossible.

> Any of my developers caught wasting their time on these would get a stern talking to.

Assuming this is serious, FWIW I'm a STEM PhD and I find that even in the smallest scripts, even just web-scraping stuff, if I don't write unit tests then I get it wrong and later discover I need to do it all again. Or worse, I figure out later, after I've built lots of code over this, that there is an error and the subsequent code all needs to be redone.

In short, I find unit tests to be a big time and effort saver. And I get the right answer, for a change. :-)

I test API's and code I expect to reuse a lot but I agree on CRUD type stuff. If the code is being used in one place and bugs aren't all that consequential its a huge waste of time to test everything.

I've found that unit tests are less important when you have good CI and your codebase isn't shit. On high quality code it takes so little time to fix bugs and redeploy that unit testing is a hard sell.

Some unit tests are redundant and only waste time. Others are very useful.

Do not write unit tests just because the methodology requires it. Write unit tests when the unit needs to be tested. Typical examples:

* Smoke tests (does this code work at all?)

* Tests for known edge conditions

* Tests for known bugs (to catch regressions)

Interesting - so how else do you test?

End-to-end tests comparing output to hand-computed results on small examples (or perhaps the output of an earlier, simpler, prototype)?

(For predictive models) evaluate the output on a test set? Depending on what you're trying to do, the contents of the black box may not matter so very much if the results on an agreed test set are good (...and you're confident it's independent from any training data...)

Manual eyeballing and sanity-checking of output? (in my experience important however many layers of automated testing you're using, and undervalued by people who focus on software as an engineering process more than an art-form).

...and, circumstantially, probably a bunch of others. I don't think this is a field where making lots of rules is especially helpful (except, perhaps, the "always eyeball" one...)

Thanks, useful. I never seem to be allocated enough time to write tests and always worry as a result, since use of automated tests seem to be close to dogma for many. Along with agile it seems to be one of those things that people cling on to for security, appropriateness or otherwise notwithstanding.

Really? We use unit tests extensively in LibreOffice and its saved many bugs and man hours.

You are kidding, right?

You must have a superhuman team. The only languages I could even remotely imagine to get by without tests are Haskell/ML.

Even then, logic mistakes can and do still happen.

You're not alone in feeling this way. Greg Wilson organized 'Software Carpentry' [0] to teach researchers the basics of software engineering. They regularly do workshops (list/schedule at site).

[0] https://software-carpentry.org/

> Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories.

Funny, that sounds exactly like my IT Dept.'s approach to coding.

I work in Operations and was exposed to their approach when helping them translate some business rules into SQL for the warehouse management system. During the course of interaction the IT Director lost the code we had worked on together twice, keeps the requirements docs in his email etc. etc.

Me: your engineering practices are terrible

He: We don't have time to do it properly

Me: but somehow we have time to do it three times

I told my manager but no-one really understands or cares enough. The Warehousing Director calling out the IT Director about his approach to IT - where would that all end ?

I remember carefully crafting a software framework for my Ph.D. I got it running, it worked perfectly, it didn't deliver any advantages vs state of the art. I think it took me 7 mths to write. I then slapped together another one, and that took me 3 mths, it worked (just about), it produced some decent results, and I could move onto the next thing. The software I was producing now was not good but this was in fact excellent experience for research software development as as my work progressed into more innovative areas I found that many, many ideas didn't work out at all, so the priority was to be able to develop code really really fast. I suspect that this is what is going on with most research code - write it really fast, not really good.

Give me a break, you are complaining. I don't expect academics to follow enterprise standards. They are more rooted in theory than practice. I expect shitty code, uncommented code, no unit tests. You know that thing SV calls the real MVP? Yup, that's what I expect from academia, the most basic Proof of concept to demonstrate that an idea is possible. Correct is cherished, beauty is not. Note, the correctness is enough for best case/data at hand. What's more important and useful to me and society at large is the papers that follow.

Then there's the group of folks who tend to read those papers and turn it into products. Yes, those are the ones I expect to have repos, unit tests, build systems and all that.

How did you get into it? I think something like that would be my dream job. Yeah, I'm not normal :)

I'm with you. This is the second time I'm reading about this niche and would love to find out more info about getting into the field.

I used to work as a scientific/medical translator and one of the things I loved about my work then was that with each new project, I had to learn enough about a new topic/subfield to become a bit of an pseudo-expert in it. I greatly enjoyed that research and would love to do some work in software development that would require the same type of constant learning with each new project. Diving into some convoluted, 20-year-old Fortran driving highly-specialized scientific software actually sounds kind of exciting to me (yeah, I'm not normal either :) ).

Is there much demand for this kind of work, I wonder? I'm guessing the best way to get your foot in the door is to be in academia, and those days are long passed for me.

Fortran story time:

As an undergrad, I had a good friend who was doing his PhD in Atmospheric Physics. It turns out, most of that field works in Fortran '88. This is not a very useful language, seeing as it uses GOTO statements to function as a loop. Fortunately, it does have comments. My friend managed to sweet-talk an older PI into giving him the old code for use in nuclear blast atmospherics (Exp: say you nuked all of France, what happens to Greenland's ice). At about 3 am before a project was due in the morning, he was pulling through the spaghetti that was the code, tired, jittery, and over-caffeinated. In this mess of logic diagrams he had to draw out by hand, he finally got to somewhere he thought was going to really cement all the code together for him. He follows a GOTO statement, and there was only a set of another GOTO statements. This went on for about 30 (my recollection of his words) GOTO statements, all 'nested'. Eventually, he gets to one that only has a comment line: 'HAHA MADE YOU LOOK'.

His laptop was defenestrated and he had to buy a new one with me about a week later.

I think the issue is more one of funding than of a lack of demand - I suspect there's not that many grants that let you hire a programmer for this kind of thing.

I'd look into companies focused on SBIR grants. The SBIR program is a governmental program designed to spur R&D in small businesses. Over $2.5billion is awarded annually. For instance one grant deals with researching detecting and suppressing attacks on GPS receivers on missile systems.

You can staff projects off of grants, so it removes the need to sell the product. And the companies are probably going to be more academic, with a high percentage having graduate degrees. But companies will be under pressure to win grants. And because they have less of a need to sell a product, your work might not be widely used after the project ends. Also, the DoD is a major source of grants, so it would be helpful if you were comfortable with working on military research projects.

But it might be something to look into if you want to get into software development in an academic environment.

My previous job included this kind of work (along with coding up dirty prototypes and a bit of asset management). My job title eventually settled on "Research Engineer". The original job description was very broad, so I would suggest applying for any job postings by academic institutions that seem related.

Academia is being pushed more and more toward industry collaboration, so if you can get the exposure and connections it shouldn't be too hard to find the opportunities from the inside.

You really have to enjoy this kind of work and some of the frustrations that come with it. I did, but ultimately had to leave when funding for my group ran out. If life circumstances allowed me to work for less I might have stayed with them.

Mostly luck. I applied on a whim since my previous job (enterprise Java) was dull and tedious, and probably got the interview based on development experience and a math degree.

Reply on news.ycombinator.com