Hacker News Reader

Comment by GNU_James | Hacker News Reader

Friendly reminder that entire scientific community runs on Python 2 legacy code and nobody will rewrite it to Python 3.

not in my neck of science python (biomed/neuroscience; computer vision). We've been substantially more aggressive about moving to 3 than other python neighborhoods! Look at the top projects at https://python3statement.org/, for example.

Hell, numpy 1.20.x is dropping support for anything below 3.7!

I started doing science python in 2015 on 3.5.x, never had to touch 2.x at all.

That's why people move to crappy conda / docker. To keep those Python 2 environments running. There no new people to rewrite tons of software.

There's another reason for using containers or pretested distributions. I just wrote an article for LWN about SciPy¹. To get what was supposed to be a clean install of the newest release, I created a virtual environment and used Pip to install the SciPy components. It was broken out of the box. IPython did not work because a module that it depends on, that does tab completion, had been recently upgraded and was incompatible with it. I mean it crashed frequently, not just that it didn’t do tab completion. To get a working system, I had to install an older version of the library. So people seek solutions that avoid the dependency hell that comes from maintainers releasing things at will, with nobody testing the combination of things that they nevertheless market as “SciPy”, which is supposed to be a bunch of stuff that works together. And whenever I install anything from the Python world, I am amazed when it actually works without the need to spend hours with Google or staring at sources trying to resolve conflicts. One of the many nice things about Julia is that their package system works.

So yes, once you manage to put together a working installation, you naturally want to encase that precious thing in the bubble of some kind of container, to protect it from an upgrade of some piece that will break everything. If you work with a lot of Python things, you may have a dozen copies of the same libraries, and a handful of Python binaries, taking up space on your drive. The concept of dynamic linking was supposed to save us from this. But the haphazard habits endemic to the Python community seem to have left us with the worst of all possible worlds.

[1] https://lwn.net/SubscriberLink/842964/f7e2e5006c5efe36/

Docker is the best thing to ever happen to scientific computing yet there is so much griping from the old guard. Trying to make a hermetic, CI/CD, within-machine-epsilon reproducible computer vision pipeline opened my eyes to just how un-reproducible your average machine learning project really is. Especially those coming from academia.

"Oh the PhD pinned to a nightly version of some dep and the AUC score plummets when you try to upgrade and customer wants this latest model in production on airgapped machines last week? Aight, stick their Conda env in a container, call it with `docker run`, and sort it out later"

Horrifying? Yes. Solving the problem on short notice? Also yes.

Plenty of more pedestrian uses but that was an actual situation where docker's reproducibility saved my bacon.

Yes. When your calculation uses dozens of files and a change in any of them might change the results, I don’t see how there is any alternative to some form of containerization. Even if you could generate a statically linked binary, if it’s science you want to distribute the sources for everything as well.

I notice that most people seem to reach for Docker automatically, though. There are other technologies, such as systemd containers, that might be better in some situations.

Same in my neck of the woods (engineering R&D). Python is very popular and everybody I know is using Python 3 unless they are supporting an outdated codebase, where they are usually in process of migrating.

I work on a physics project. The data acquisition system was made on a tiny budget and most of the code is python 2.5.2. It hasn’t been touched in 12 years. We’re stuck with Windows 7 running on some machines because there are no Python 2.5 libraries that support NI-DAQmx and NI does not have any Traditional NI DAQ releases in windows 10. Obviously good solutions involve porting to linux and airgapping the DAQ, but the simplest solution I like is updating the codebase. I’m one engineer with loads of other responsibilities. Python is popular with scientists. I don’t begrudge the scientists. I begrudge the awful decision by the python developers to disregard backwards compatibility.

No one will do it until it becomes painful enough to not do it. Killing support in all of the tools is the only way to make it happen.

No one will do it after support in the tools is killed off either. People will use the last versions of the tools from wherever they can get them, and workarounds for any issues that are discovered will become part of the folklore surrounding the code.

Thats fine too, at least its no longer bogging down software maintainers since they can just tell people they do not support py2

Sweeping cat puke under the rug is not the same as cleaning cat puke off the floor.

Having software maintainers to tell their users “get bent” does not fix the user’s bugs or security exploits.

Software was supposed to make people's lives easier, no to be a form of torture!

At some point it will be selectively rewritten if it's important enough.

I love/hate this type of tautological yet provoking statement

In what sense is it tautological? I can think of a lot of things that are important that won’t get done, healthcare improvements being one. Software on the other hand is typically fluid enough that someone will rewrite it.

Because the bar can be effectively slid around justifying that it was/wasn’t important enough if it does/doesn’t get rewritten.

It’s as meaningless as telling someone they will achieve their dream if they want it bad enough.

I think it’s slightly more meaningful, because not all dreams can be achieved. If software isn’t getting ported from Python 2 to Python 3, it almost certainly isn’t because it’s too hard to do.

If it wasn't selectively rewritten, it wasn't important enough (to be selectively rewritten).

This is absolutely untrue.

I mean, huge chunks of the scientific community run on MATLAB or Fortran. (Also... are you actually managing to get new students to install Python 2 on their laptops, somehow? And no enterprising undergrad is just like "I will just make this work on Python 3 because that's easier"?)

Ehh, I absolutely did that. Heck, for a class I reverse engineered a robot's bluetooth control api because coding up computer vision was easier in python than in matlab (which had a provided "SDK" that some TA had written in years prior).

I spent less time doing that issue than it took to debug a missing comma on the end of a line in some matlab code (for a different assignment in a later class) that led to some incorrect matrix operation and completely failing code. I do not like matlab.

I think it’s a matter of experience. I’ve used MATLAB for nearly a decade and can whip up optimized bit banging, data acquisition, and signal processing code in a matter of minutes. When I try the same in python I spend days trying to figure out what the “right” solution to something is, try 3-4 options, find the issues with each, and by that point I’m burnt out and tell myself it isn’t worth it to try to replace MATLAB.

and MATLAB and Fortran are fully supported, so I'm not sure what the problem is?

That’s absolutely not true though.

Reply on news.ycombinator.com