Hacker News Reader

Comment by londons_explore | Hacker News Reader

> Like most bioinformatics programs, AlphaFold 2 comes equipped with a “preprocessing pipeline”, which is the discipline’s lingo for “a Bash script that calls some other codes”.

Having Bioinformatics people requiring to stray a long way from their core competency to learn a scripting language from the 80's to write glue code seems... suboptimal. How many hours of expert time has been wasted figuring out how to split a string in bash?

Can us software people build a better tool to eliminate the need for this?

View on HN

There is a growing trend to include Docker (or Singularity, which is more compatible with HPC architectures common in bioinformatics) images alongside codes. In particular, AlphaFold 2 does provide a Dockerfile, and they even include a Python "launcher script" hiding all the details of running the code.

Sadly, this is very uncommon in the community. In a bioinformatics meeting, the sentence "I spent X days setting up Y software" will not raise many eyebrows

I work in power systems, and the situation is similar. Maybe worse because paper authors often come up with new computational techniques but don’t implement them in code (much less code with a dockerfile).

I look with much jealousy over at the computer science field where papers often include code, multiple versions under version control, automated tests, setup/docker scripts, and demonstration workflows and interfaces.

Sure there are better alternatives, but the advantage of bash / shell scripting is that it’s very easy to glue a whole collection of tools together, and that expertise in this transfers well between domains.

They probably could have achieved the same by invoking things in Python, but it would have been slower and not achieved a lot, other than “not using shell scripts”.

And once you go down the path of optimizing this enough, you’ll end up reinventing shell scripts altogether.

Well, AlphaFold 2 generates MSA by invoking things in Python: https://github.com/deepmind/alphafold/blob/main/alphafold/da.... So the article is actually mistaken on this point.

It’s sad. Bio people really need friends in software to boost their research. Unfortunately they do a codeacademy python course for 5 min and try to get their projects going. Sometimes they succeed, sometimes they fail. But they don’t really have much time to dedicate to properly learn software dev and it’s not what they are into anyway, it’s a necessity.

I think we could create something like a GitHub of bio projects that need help and people assist with hopes of getting their names in a paper

Pretty much anyone that works with data has to clean their data, learning to use the tools to do that is important. Whether that's bash, perl, R, python, ... doesn't really matter that much. If they already know bash then bash is a good tool since they can now focus on their data instead of wasting time on learning a new tool to do the same thing.

Bash is easier to explain and use than, eg, explaining ppl how to use Python subprocess Module Launch different apps, capture their output, etc.

I find astonishing how bad Python is as a bash replacement.

I often rather write an argument parser in bash than use Python if I have to invoke a bunch of commands.

Python is bad, but bash is worse as soon as you need any kind of logic.

> explaining ppl how to use Python subprocess Module Launch different apps, capture their output

There's no shame in using `os.system`.

Well, technically subprocess.check_call if you need to capture output.

I think the tools already exist just that people are conservative with their choices.

(Obligatory xkcd; you know the one)

Not really?

https://xkcd.com/927/

MLops is a pretty hot area right now. The industry is trying to figure out how to engineer these things in a robust way. It’s not ubiquitous at all yet. There are lots of tools that wrap K8s and help you train models, but trying in DataOps in a robust way…. I haven’t seen the definitive answer yet.

Seriously please tell me if you are founding this company so I can invest.

Among the bioinformatics folks I know, bash is already a core competency. If you’re using your average biologist as your mental model, you’re thinking of the wrong people.

* Can us software people build a better tool to eliminate the need for this?

Most probably, not. Bash is currently the sweet spot. It is actually the best tool for this job. Any other option comes with increased complexity, will make the whole software less stable.

From looking at the code, Bash looks pretty clean.

I also use Bash and AWK for preprocessing a lot.

I used to be that guy as well, till a college convinced me that anything I can do in Bash, Awk I could probably do easier in Perl. Then everyone sort of drifted to Python. I get that if you never used Perl it’s pointless to learn it if you’re already in the Python stack, but… damn Perl’s regular expressions and how it’s so baked into the syntax of the language makes using regex in Python seem like going back to the Stone Age.

> How many hours of expert time has been wasted figuring out how to split a string in bash?

Probably a good many more than would be needed to learn how to split a string in Python.

I don't see this as a bad thing. Shell scripts are a great way to prototype text processing pipelines using multiple smaller components.

I think DeepMind went one step further and solved the entire problem for them. They don't even have to touch their keyboards anymore

Reply on news.ycombinator.com