Hacker News Reader

Comment by bluedino | Hacker News Reader

I loved Python. It was easy to learn, very powerful, has libraries for everything...

Then I started supporting researchers and scientists who wrote "python code" to run simulations etc.

Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it.

But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting out random lines because they saw someone else "fix" something that way... and they all want Jupyter notebooks.

Now it's like an eternal September plus I get to deal with annoyingly slow package managers like Conda and rough academic projects with poor documentation and little testing.

View on HN

Having been in the same situation as you, you have to bring some technical leadership if you want to change things. And some empathy, these people are stuck in a local maximum, they try their best probably with limited tools and knowledge.

Conda is a shitshow for sure: teach them how things could be better.

Notebooks are a reproducibility nightmare: show them how nice eg. Pycharm is

People are unqualified to write O(n) code in less than O(n^3) time: identify skill gaps and get their management to sign off on some software development / python training.

If you can make a convincing case why your approach is better, you can make everyone's life better.

Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that.

Notebooks on the other hand are a god damned abomination. Obligatory past discussion;

https://news.ycombinator.com/item?id=17856700

Couldn't you just do that with pip?

No conda is for dependencies of your python environment. You need python itself, shell utilities, and some dependencies are dynamically linked. Think package manager for the `--prefix` compilation flag AND `LD_LIBRARY_PATH`

conda can also be used to manage python packages, but its dependency solver is very slow and sometimes completely fails.

Once you have created a conda environment, it is usually much more practical to install packages with pip or mamba.

> Notebooks on the other hand are a god damned abomination.

I completely disagree. Notebooks are an excellent way of doing literate programming, because they go beyond plain text, and integrate richly formatted text, tables, figures, videos with code. This is excellent where the human is following a line of reasoning, and offloading the computation to the computer. The reasoning is explained via the non-code, made concrete via the code cells, and its effects shown via the code cells.

Of course, people can abuse the conveniences of the medium to write really bad code, and follow some really bad practices. But that doesn't mean the medium is bad. In fact, I think if you forced people to stop using notebooks, their productivity would drop by quite a bit. Notebooks are tool for thought [1].

The solution is to teach them better practices, and provide better tools within the notebook interface that will help people write better code.

[1] https://numinous.productions/ttft/

That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.

To make this more clear as I have had the same experience and want more people to enjoy it: Mamba is more or less a drop-in replacement for the conda tool with the guts rewritten in C++ and as a result is much faster to resolve dependencies -- I want to say an order of magnitude faster or more. It's even noticeably faster at downloading and installing. You just

  conda install -c conda-forge mamba

in your base environment and then you can start running mamba instead of conda.

Yeah I was looking for this comment. +1 for mamba

The situations such as improving package management in python is a problem but fundamentally in any software stack you will eventually find bad code. Especially as the code base gets larger and people move on.

Instead you can try to fight it by teaching the rest of the group how to write more idiomatic code but that only goes so far. Also training and documentation can go out the window when the objective is to ship something that works once.

Reply on news.ycombinator.com