Escaping from Anaconda's Stranglehold on macOS | Hacker News Reader

Escaping from Anaconda's Stranglehold on macOS

paulromer.net

143 pointssurprisetalk281 comments

> 10. What Changes When You Are Free

> You will not be forced to work inside something known as a “virtual environment.”

Oof, this terrible advice cancels out an otherwise reasonable post. Beginners who don't know what they're doing are the last people who should be `pip install -r requirements.txt`-ing into the system Python the way this article is recommending. That's not only going to make working on multiple projects nearly impossible (especially for the kind of beginning students who get recommended Anaconda, which are almost always the Data Science-y crowd using NumPy, Pandas, Scikit, etc, which are notoriously finicky with version conflicts), but it stands a good chance of breaking other Python-based system utilities in completely opaque ways. This sort of advice can fubar a naive user's entire workflow.

I know virtualenvs suck to explain to people, but in my opinion it needs to be done before you ever tell them about `pip install`.

Yes. I recently had to debug my gf’s entire work computer setup getting fubar’d from installing a pip package at a user level that conflicted with the name of some package used internally by the 10k+/seat/year software the company devices have installed. Proximal cause? The installer “helpfully” failed over to installing at the user level when she didn’t have permissions to write to the place she told it to. Root cause of course is that million dollar fancy fancy enterprise software somehow being unable to give itself an isolated python package namespace, or provide any relevant errors when conflicts occurred.

The python packaging system has to be used in case studies of worst design decisions of all time.

> package used internally by the 10k+/seat/year software the company devices have installed

Sounds like a skill issue that shouldn’t be seen in “10k+/seat/year software.” I wonder what other fails are in it.

Well it constantly crashes and requires every shop to develop their own half-broken way of interfacing between it and other industry-standard software, but at least ~half the pixels you’ve seen in any modern movie or tv show have been touched by it at one point or another. So that’s interesting.

Thats extremely fascinating. What's it used for?

By the books Digital Compositing, but Editors also gravitate towards it for generalist work.

Python has dogshit package management that makes java look well thought out and sane.

PYTHON @&$!!

The Python community needs to solve this ASAP. This almost weekly pain point has turned a language I used to love in college into one of my most despised languages. The fact that the ML community uses this broken platform is infuriating.

Make a Python version 4 that focuses only on fixing the packaging. 100% per-project hermeticy. No global packages whatsoever. Solve just this issue and bring the entire ecosystem on board. Kill all the various virtualenvs, the anacondas, global packages. All of it.

Learn from Rust/Cargo. That project does it mostly right (sans lack of namespaces and reproducible builds).

It took 10 years to force a breaking migration from Python 2 to Python 3 in order to handle Unicode...

I have resigned myself to repeatedly smashing my keyboard on the desk until it rains keys in blind rage induced frustration to let off steam and then calmly creating an issue in the appropriate repo instead of allowing even a modicum of hope that this will ever be fixed systemically.

Wouldn’t this end up like the XKCD standards cartoon. You’d have yet another Python package system and even more fragmentation.

https://xkcd.com/927/

If Python users weren't comfortable living in that reality already, they wouldn't still be Python users.

As far as I'm concerned, this is the fix: https://github.com/astral-sh/uv

It's never going to be solved. The mess is deeply ingrained in the Python development culture.

https://peps.python.org/pep-0582/

> that makes java look well thought out and sane

That's not hard. Java's package management is well thought out and sane.

Well, it's still PATH based, which is much worse than virtual environments (directory scoped) or Nix (well-versioned sharing).

Python's package management makes JavaScript look well thought out and sane.

The bar is so low, you have to excavate it from a landfill of rotten node_modules...

Node.js has (or at least had before the ES6 mess) the sanest and best thought out package manager out there.

NPM’s original sin was making package installation and general management so painless that folks installed micro packages for everything. Can’t really fault the software for that.

The issue was, as you say, the introduction of ESM. It used to be that you required modules one way and one way only (yes there was AMD for advanced use cases, but it was an add-on), then people felt the need to “standardize” that, no we have this mess of ESM and CJS.

I don't see what's wrong with micropackages.

Massive attack surface. That's true as a security risk, but also as a dependency risk (eg. left-pad).

If you don’t vendor your dependencies. Which is a poor practice that is commonly associated with NPM, but is by no means a requirement of the technology.

Especially if…, I should say. Even vendored dependencies are a risk as NPM commits the additional sin of allowing the act of pulling a package onto the local machine for inspection to execute arbitrary code in the form of “postinstall” hooks.

Review and pin your direct dependencies. With transitive dependencies it doesn't differ from trusting large dependencies in general.

The alternative to micropackages has significant downsides. Pulling in extra surface and rolling your own buggy implementations while waiting for some commitee to bikeshed years on the implementation.

Making the right thing easy rather than the wrong thing hard is a lot better approach.

I actually like Python's package management (with just pip and venv). It's different to all other modern solutions, taking a per-shell rather than per-project approach, but that doesn't mean worse.

The advantage is that it's less "magic" than, say, npm. No need for rules saying how to find node_modules, you just have a PYTHONPATH env variable.

The Rust and Java approach is to do everything through a build tool's CLI, and I can't complain about that. It's probably the best compromise: Less magic than npm, and more user-friendly than Python.

What magic is there for finding node_modules? It's in the same directory as package.json. Very similar (though simpler) to how Python modules work.

There's also usually one global package dir, but that's almost exclusively used for runnable binaries, and not even really needed when there's npx.

If you think Python packaging has less magic, you probably don't know it very deeply.

I lost all hope for Python package management to ever get better when PEP 582 got rejected for some stupid tribal reasons.

https://peps.python.org/pep-0582/

I was talking more about the package lookup when running code, via e.g. `node script.js`. I think it looks in all parent directories of the CWD, or maybe of the script? It's not too complicated, but it is "more magic" IMO.

Actually building Python packages is pretty complex, but that's the case for JS too. Java avoids this by distributing compiled libraries.

It looks up from script.js'd directory or the earliest parent with node_modules. Not CWD. A lot like "import somemodule" in script.py tries to import somemodule.py file from the directory script.py is in.

Traversing to the parent is especially nice for scripts. In python having scripts outside the module directory is quite painful.

Gyp is a lot easier than Python setup.py. The "easy" Java packages are comparable to pure Python/pure JS packages. With e.g. C bindings JNI/Java packaging is a horrid pain.

Sigh https://dublog.net/blog/so-many-python-package-managers/

I have never broken my Python installation on Windows (official installer) despite recklessly installing anything. If I had broken it, I would have just uninstalled and reinstalled.

While not familiar with macOS (topic of this article), I think the Mac installer works the same.

On Debian of course you can cause great damage because the system Python is used for critical system functions. This is silly, I think the system python should be an isolated install in /usr/sbin. Better yet, move back to Perl for the system.

You won’t break a machine by installing Python packages on an OS that never uses Python for anything.

This is a good point and something I hadn't thought about. I'm also a happy Windows user, and have never had a problem with Python installations. But like you say, Python isn't baked into the system.

I do use one piece of commercial software that includes Python. You can see the installer saying "installing Python." I suppose it should fall on the OS and software vendors to not abuse the infrastructure... "to whom much is given, much is expected," but maybe too much to expect.

I remember that, in the past at least, one could completely destroy a Red Hat install by messing too much with the system Python.

I guess "Python is my operating system" has its unintended consequences. ;-)

To be clear, there's absolutely nothing broken about the system Python in the article. There is just a shell alias causing the anaconda version of Python to be launched instead when you type `python3` at the command prompt...

I teach using python. You should try explaining virtual environments to students, most of them with no programming experience, or any real notion of the state of a computer system. I do, because we recommend students use them. Every class consists of endless debugging of student systems.

After that you might understand the OP's viewpoint.

After that I wonder why people think Python is a good choice to teach as a first experience in programming. Virtual environments is actually my biggest gripe with Python.

I think it is interesting to consider why Python is so popular despite such fiascos. The answers can be very informative but can also be giant red flags.

Because every option sucks for teaching.

C/C++? Full of footguns. Java/C#? Too complicated for beginners. Pascal? Outdated. Etc.

I personally prefer C to teach first-year CS students but just as the lesser evil. A good first programming language is sorely lacking.

(Note that I'm talking about the imperative programming paradigm. The debate on whether one should start with functional programming is outside the scope of this comment).

It's weird you throw out Java because they are the teaching language. They're the stick everything else is measured against. CS101 is a sea of Eclipse. Having a programming environment where all the major structures are discoverable through menus is actually pretty useful.

Java is popular as a first language, but I think it's just due to its (bygone?) general popularity. I don't think a language that forces you into a one-class-per-file mindset from day one has any business being used for teaching; students should start from the basics ("commands"/function calls, if, while, for, ...) and then discover what more complex concepts do within that framework.

Java's tooling is nice, but frankly Turbo Pascal (or something at its level) is enough for beginners.

You can teach imperative programming with OCaml or Racket, too. :)

C is indeed pretty evil, though slightly less though with modern sanitisers, so you can get a better error message than just a segfault (or silently doing the wrong thing).

Pascal isn't really more outdated than C. Especially if you use Delphi?

Python is actually fine, you can get pretty far with just the standard library, and the libraries that you can install with your Linux distribution's package manager (eg via Pacman). I do agree that package management with Python is pretty bad out-of-the-box.

> Because every option sucks for teaching.

The answer is really quite simple: F#

How in the world is c# too complicated? Console app, top level statements, vs code. As easy as Python. Easier because your env isn’t messed up.

If they can’t manage that they have no business doing data science or something similar to begin with

You sound like you have been doing this for way too long to know what it takes to teach beginners.

Not for complete but for beginners lightly familiar with programming it is overall not difficult to teach the basics, often easier than to stubborn seniors who want to do it their way only :)

    var start = "commtext c00\">";
    var end = "</div>";

    using var http = new HttpClient();
    var page = await http.GetStringAsync("https://news.ycombinator.com/item?id=41425416");

    var commStart = page.IndexOf(start) + start.Length;
    var commEnd = page.IndexOf(end, commStart);

    Console.WriteLine(page[commStart..commEnd]);

    var host = WebApplication
        .CreateBuilder()
        .Build();

    host.MapGet("/", () => "Hello World");
    host.Run("http://+:8080");

Not at all. I still teach interns and until 2 years ago complete beginners.

I can teach them a console app with top level statements and have it running before they have installed a Python virtual environment.

It’s not just the packaging situation that makes Python a bad first language. Whenever I write in Python, I find it completely impossible to stop myself from constantly making basic, beginner-level programming mistakes. Every time I miss having a compiler and strict typing that will yell at me when I’ve done something stupid.

Modern linters and type-checkers for Python come pretty close to something usable for these situations. But it's certainly something they tacked on to the language afterwards.

I like the recent addition of (proper) pattern matching to Python.

Yea, the only way to get close to programming efficiently in Python is to turn MyPy and Pylint up to 11.

I switched to Rust full time about 6 months ago and recently had to write a somewhat long Python script for something else... it felt awfully quiet without the compiler yelling at me, but that also meant I was making the same silly mistakes over and over and over and over

We use it because of other people use it. I would gladly switch to Julia for scientific commuting if I could.

I think docker would be a much better alternative. You can just hand students a pre-built docker with everything installed and they are free to mess up the system without any consequences.

But then you're explaining virtualization and networking to your students, how to manage interactions in and out ("I wrote a program in Txtedit, how can I run it?"), when it's started and when it's not etc.

I could see giving them a remote access to a prebuilt env (which can be containerized) to be simpler to explain.

In finding it very difficult to see what exactly is the problem with teaching a minimum of computer and operating system fundamentals to people that have to learn programming.

Not every person wants to become a software engineer, anymore than every person wants to learn to become a plumber or car mechanic. The vast majority, and by vast majority I mean more than 95% of students who are taught to code are training to be scientists, mathematicians, engineers, economists, accountants, or some such. They need to run simulations to understand their field. Any pain in the way of that is a failure of those who create software for a living.

If you're going to be spending a substantial part of your waking life working on a tool, spending a few hours to get the basics right is not an unreasonable ask. And as it turns out, all non-trivial programming languages that have some sort of packaging system or module system have some version of the problems involved.

You don't see people complaining that musical instruments are unreasonably complex when that complexity usually gets solved after a couple months of training, or when someone who wants to basic woodworking has to have at least a passing knowledge of the different types of hardwoods, MDFs, and essential joinery techniques.

Hate to say, but it definitely sounds like skills issues.

I'd argue it's a lot more than "minimum". You need them to be comfortable with the host system, and also manage the running container, which will have a different OS most of the time, and the virtualization mechanism (docker here).

Simple things like having a student access another student's web server becomes overly complicated, as you're dealing with 4 systems talking to/through each other.

But the fundamental fact is that "accessing someone else's web server" is not that trivial of an affair and has never been when you get down and dirty into the details of networking.

Giving access to a debug server on a local network is a single command line when running natively to the host.

Sure making them understand everything is a journey, but just spawning the server is fairly quick and painless.

Might as well use Guix at that point, since that kind of OS-based dependency management seems to be the future.

Not sure how helpful this advice is for students I'm in my late 30's have programming experience but had never touched Python until about 6 months ago.

What made Virtual environments work for me was switching to a different Editor. I use Visual Studio Code currently it works well with Virtual Environments.

When you first create a new python file in a given "project folder" it prompts you to create a new Venv and when you switch project folders it remembers and restores the Venv for each project.

One of my work colleagues pointed me to VSCode - it streamlined a lot of python things for me.

If your students are disciplined about creating a new folder for each project managing virtual environments vscode could help them.

Only issue I have is my work office has a corporate proxy setup and pip needs certificate to connect (and if I work remotely I have to turn this setting off) I wrote a shell script to toggle between the two proxy settings. Not sure if university will have the same issue proxy issues but if so this would certainly be a pain point for many students.

This is a suggestion that deserves serious consideration. I started using VS Code after I had learned all the basics so I don't know what it feels like to be a new learner who is getting started by using VS Code. But it sounds like it worked for you. (And on campus, the proxy should not be an issue.)

It does answer the question of which text editor to use when the time comes to teach them about editing text files. For a course, it helps that it is free and available on both macOS and Windows.

I have a vague recollection that at some point I was testing without an install of Python and after I selected the Python extension for VS Code, it offered to install one for me. (I don't recall if this was on macOS or Windows and memory could be playing a trick on me.) But in any case, the people behind VS Code do seem to be trying very hard to make it easy for someone who is getting started.

I am a little uneasy about the nudges to use Copilot, but on balance, it might offer a better path for students who are getting started.

Then why teach Python? All of these issues are because Python is a ridiculously poorly designed language which is still having new features added to it. It's generally a mess.

The issues are with one particular facet of poor design and management. The rest of it is manageable, if distasteful, and worth the effort for the ecosystem (in general).

There really is more than one particular face of poor design in Python. I honestly remain confused why people use and love Python so much. It very much says something.

Oh, there are many. There are just comparatively few that you can point at as being quite as directly responsible for dev misery as thé packaging system.

Lol. I see where you're coming from.

> You should try explaining virtual environments to students

can you? Because I can't.

e.g. "What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, pipenv, etc?" ( https://stackoverflow.com/questions/41573587/what-is-the-dif... )

what is the difference between conda and anaconda?

etc etc

That's exactly GP's point.

Reckon they'd understand using something like asdf instead?

https://asdf-vm.com

The projects I work with have moved to it because it handles multiple languages (not just Python). It's made things easy to keep "the correct version of a language" that each project needs.

So far so good, anyway. :)

You’re swapping something that can be feasibly debugged or at least easily burnt and reformed for something that’s impossible to untangle once they’ve done two projects with different or conflicting deps into the base environment.

What I do on every dev machine is build Python from source and install it to ~/.localpython and put it in my path.

Then I can pip install away care-free, and quickly run scripts / repls without having to activate an environment.

Why does it need to be built from source for that?

It doesn't, but it's typically the simplest (for me). It also means I get exactly the version I want compiled with the best optimizations for that machine.

Thanks, that makes sense.

Typically building from source is done to include optional components such as with the ./configure --with-pydebug option. With many projects make doc is an option because so many prefer to skip that part of the build in favor of online documentation.

Building Python is so quick and easy I’ve built an interpreter per project in some circumstances. Sometimes more if I want to make sure it’s compatible with multiple Python versions.

> I know virtualenvs suck to explain to people, but in my opinion it needs to be done

This is because Python doesn't have versioned imports, which means you can't have multiple versions of the same package in the same environment, but I like to dream about a world where this isn't the case. If instead of import foo we had import foo@x.y.z#optional-checksum, the Python world would be massively improved. It seems like it would be such a simple change too.

Virtualenvs solve not having separate workspace-local package installations, not a lack of versioned imports. Versioned imports are not a good solution to separate installations of packages: code is harder to upgrade, cluttered, and encouraged to depend on specifics rather than contracts. There’s a reason every major language localizes their version pinning into a per-project dependencies file (which can be anywhere on the spectrum between “contracts only semver ranges” and “checksummed/vendored lockfile”). And that’s before considering the troublesome behaviors that emerge when you permit importing multiple versions of the same library in the same program, if you want that too.

> And that’s before considering the troublesome behaviors that emerge when you permit importing multiple versions of the same library in the same program, if you want that too.

Rust has versioned imports. So you can import different versions of the same module (at least in your transitive dependencies).

Yep. And while that’s not something I’d call a mistake per se, it can be troublesome. What if a crate you depend on at different versions provides access to global state? What if a transitive embeds definitions from that crate into public contracts consumed by code using a different version?

Rust offers mitigations or means of detection for those cases, but they still require thought and troubleshooting when they occur. Given Python’s lack of static typing and more-likely-to-be-nonexpert user base and usage patterns, I suspect that troublesomeness would not be a value-add for the Python platform.

Well, for what it's worth, Rust's linter clippy likes to warn you about using different versions of the same crate. I think the warning is mostly formulated in terms of helping you reduce compile times, but your concerns about mutability might also be justified.

IMO what needs to happen is python needs to get rid of system installs and only work via venv. It should check the current folder for some .venv or something and auto-config itself to the current directory (or complain that you need to config with a messages like "python not configured for current folder. Run 'source bin/activate' to configure. Honestly thought that could be so much better.

The entire problem of python an venv IMO comes from it not being the default.

Python developers agree. See PEP 668 – Marking Python base environments as “externally managed” – https://peps.python.org/pep-0668/

More recent document

https://packaging.python.org/en/latest/specifications/extern...

Hmm that still recommends that distros allow admins to install to /usr/local, albeit in such a way that it at least can't break the OS.

IMO the idea that a 'Linux admin' is better informed than a 'Linux user' is increasingly anachronistic. In most cases the admin is just the user running sudo. I'd suggest that such functionality should be enabled by installing some kind of OS package rather than being the default

Yet another hack on hack. Venv was the wrong fix/hack and now the package management is cascading into even more madness.

This is insane. Apparently (at least on Debian) this can be circumvented by putting this in ~/.config/pip/pip.conf:

  [global]
  break-system-packages = true

I would have preferred single-version-externally-managed to keep fond memories of setuptools alive.

It becomes increasingly impossible to track down home directory pollution and config files in Python. Next step will be a Python registry on Linux. How about:

  regedit VENV=/home/sjw/venv42 KEYWORD=single-version-externally-managed DWORD=0xbadbee

This is already the case on Arch, where pip will yell at you if you try to install anything to either the user or system score.

That depends very much on what you're doing. For production I'd like to use the OS package manager to install my Python dependencies, and move the responsibility of patching to the OS.

For workstations, absolutely, go with virtual environments, it's the only way to go. One concern I've seen from some in the machine learning space is that rebuilding a virtual environment, for example if macOS upgrades the Python version, takes hours. That can be solved by using pyenv, then you can have multiple versions of Python and be free of Anaconda. I primarily use pyenv to be sure that I have the same Python version that ships with the server OS on my laptop.

I have been out of the loop with Python for a couple years at this point, but how could this get so bad?

If you are looking into a system you are unfamiliar with, where do you look first?... pip? pipx? homebrew?... or is it in anaconda? pyenv?... must be in the os package manager... apt? pacman?

Honestly, Maven and NPM look great compared to this mess.

NPM to me is the great example of a package manager that worse than anything the Python community has come up with, but that's subjective I think.

Python isn't as bad as people make it out to be. There are some issue that you will run into if your project/code base becomes really large, but that's not an issue for most people. The vast majority can get just use python -mvenv .venv to set up a virtual environment and install in that using pip.

Next step up is you need specific versions of Python, so you switch to pyenv, which functions mostly like the built in virtualenv.

Then you have the special cases where you need to lock dependencies much hard than pip can do and you use poetry, pip is to slow so you use pipx. Those are edge cases to be honest. That's not to say that they aren't solving very real problem, but mostly you don't need it.

It would be great if there was one tools that could do it all, lock the Python version, do quick dependency resolution and lock them down.

So far Python has opted to split the problem: Tools for locking the Python version, tools for creating separate environments and tools for managing packages. There's a lot of crossover in the first two, but you can pretty much mix and match virtual environment tools and package managers anyway you like. I think that's pretty unique.

It's definitely unique, but the "out of the box" experience is suffering for it in my opinion.

I actually recently had to fix something small in a Python project, pip refused to work because of homebrew, homebrew didn't have the dependencies and directed me to pipx, pipx finally worked - It was a strange experience.

And for the record NPM mostly has a bad reputation because of its past... nowadays it's perfectly usable and can lock dependencies and node version "out of the box".

> It's definitely unique, but the "out of the box" experience is suffering for it

The tooling that ships with Python could be much better, I'd agree with that. You can go pretty far with venv and pip, but just the python3 -mvenv .venv isn't exactly a great example of user friendliness at work.

What's your problem with NPM? It just works, it's easy to package for, it handles multiple version dependencies elegantly, with pnpm it doesn't trash your disk and is really fast.

(P)NPM and CJS is the best package/dependency management there currently is.

I always set:

    export PIP_REQUIRE_VIRTUALENV=true

I don't see how it could be correct to say that I'm recommending that students pip install into the system Python.

On macOS, there hasn't been a system Python since 12.3 came out in Jan of 2022. And as far as I know, there was never a system Python on Windows.

> On macOS, there hasn't been a system Python since 12.3 came out in Jan of 2022

It may not be pre-installed, but if you install Xcode it will also install python3 in /usr/bin/ for you...

Xcode users should probably be able to fix breakage and that /usr/bin/python is not used by MacOS for critical tasks.

The assertion that seemed incorrect was that I was recommending an option that would install libraries to the system Python.

Xcode does put a binary with the name python3 into `/usr/bin`. This is not a system Python, but set that aside.

Can you explain how someone might pip install libraries to this instance of Python?

Once Xcode's python is installed, `pip3 install ...` will install libraries to that python installation (assuming you haven't aliased pip3 to another installation, as it appears the anaconda installer did)

The install by xcode of a python3 binary in `/usr/bin` does not come with its own copy of pip3.

So either your command will fail to install or it will use a version of pip that is on PATH but was installed by some other version of python. So in either case you can't install a library into what you are mistakenly calling a system python.

So the original assertion was not true and you have not been able to make up an ex post justification for it.

I have a `/usr/bin/pip3`, and it appears to have been installed at the same time as the rest of XCode. Not sure what's up with your install:

  % ls -la /usr/bin/(pip3|python3|clang)
  -rwxr-xr-x  77 root  wheel  119008  4 Aug 12:31 /usr/bin/clang
  -rwxr-xr-x  77 root  wheel  119008  4 Aug 12:31 /usr/bin/pip3
  -rwxr-xr-x  77 root  wheel  119008  4 Aug 12:31 /usr/bin/python3

Yes, you are correct. pip3 is installed by XCode.

I was having trouble understanding the scenario you seem to have in mind because it never occurred to me that someone would try to run Python without doing an install from python.org as I explicitly recommend.

If someone does install a version of Python from python.org, it puts the bin folder for that version first on PATH via this line in .zprofile:

PATH="/Library/Frameworks/Python.framework/Versions/3.11/bin:${PATH}"

It also puts a symlink for python3 and pip3 into the `/usr/local/bin` folder that comes ahead of `/usr/bin`.

So if the user runs

`pip3 install ...`

it will not find the version in `/usr/bin` because there will be two other directories ahead of it on the path that have an instance of `pip3`.

As an aside, if they do the improbable and run something like

`/usr/bin/pip3 install ...`

or if they do not have any Python from python.org installed and run

`pip3 install`

what they will end up with is a user-install that puts libraries under their `~/Library` directory because (at least on an Apple Silicon mac) pip can't write to `/usr/bin`. This fallback to a user-install is confusing to people who encounter it, but it is very different from "installing to the system python."

To summarize, it is exceedingly unlikely that a student who is not comfortable running commands from the terminal is going make the mistake you seem to be worried about and end up with libraries in the user-install location:

1. They do not install an official Python even though that is exactly what I recommend.

2. They do install XCode or XCode Command Line tools, then try to use `pip3 install ...`

You are wrong: this was not a reasonable post.

This seems very "Anaconda's a mess, so here's this one weird trick" then suggests a foot-gun.

Arguably step one is: https://docs.anaconda.com/anaconda/install/uninstall/

Or for miniconda: https://gist.github.com/pulkitgangwar/421a1af800c5a9c6d4a77b...

Then, adopt `uv` instead, for all your Pythons: https://docs.astral.sh/uv/

The 2024-08-24 update finally does "all the things" — unified Python packaging: https://astral.sh/blog/uv-unified-python-packaging

. . .

PS. I noticed this original article's subsequent blog post gets into "env as code" but for "one tool to rule them all" one may want to consider a tool such as `mise` instead: https://mise.jdx.dev/about.html

> Then, adopt `uv` instead, for all your Pythons: https://docs.astral.sh/uv/

uv looks great, but its coming from a VC-backed company with no apparent way to monetise. Discussion here:

https://news.ycombinator.com/item?id=35617198

It's licensed MIT and Apache 2.0. The community can fork it if it ever becomes an issue.

Yet.

You have no idea what Astral’s other (future) plans include. For example, what if they are working on all this tooling so they can start providing “verified, reproducible Python packages” as a service, to mitigate against supply chain attacks?

Telling people not to use Astral’s products because they are VC-backed is silly, especially since everything they’ve done so far is permissively licensed.

I think your suggestions are for people who know more the students I meet in the classroom.

I appreciate all the comments. I agree with most, even when they disagree with me or each other. I especially agree with the questions several people raise: What it is that we should teach? In which order?

I think there is a consensus about what a student should end up knowing. Even if they are not going to become developers, they should be able to edit text files; they should be able to run commands from the terminal; they should understand PATH. They should create a virtual environment every time they start a new project.

Where we might disagree is how to get them to the point where they know all those things.

I offered to write this post for a colleague who is now facing the issue I faced when I taught last spring. My only goal was to help the students who can't get started running Python from python.org. When I say that they don't know what an editor is or how to use the command line, I just taking those as the facts on the ground. What I didn't say (at least not very clearly) is that they need to learn these other things. I did hint in the end that some of these probably need to come before learning to use a virtual environment. I know this is controversial.

Because the first post was narrowly focused, I wrote a subsequent blog post called "Environment as Code" that is more specific about the goal and offers a specific sequence to follow to get there.

If you have any reactions, I'd be interested to hear them. In a deep sense, I think that the issue here is how to free students from the GUI. If we can do this, it will change how they interact with the computer for the rest of their lives. If there is a better way, I'll be happy to support it.

Are you familiar with Software Carpentry? If so, what do you think about it? -> https://www.ub.uio.no/english/libraries/dsc/carpentry-uio/

If you visit the link https://swcarpentry.github.io/python-novice-inflammation/ which offers the introduction to Python, you'll find this strange statement.

> Although one can install a plain-vanilla Python and all required libraries by hand, we recommend installing Anaconda ...

In a novice course, if the official version of Python and its tools all work, why is this organization recommending a product from a for-profit organization that requires acceptance of some complex and potentially costly provisions in its terms of use?

As much as I would rather not associate with these people (to stay polite), especially in an educational context, note that you do see people recommending the use of VSCode / Github / Windows / MacOS here all the time, all of which are orders of magnitude worse than Anaconda.

If they are unable to use a shell, don't understand environments etc. I would push them onto some specific IDE/plugin combination that creates a new virtualenv, handles PATH etc. for every new project.

Dealing with environments and understanding how different parts of the filesystem relate to each other is its own pretty steep learning curve. You wouldn't want them to get tripped up on that while they are learning to program, so I think I would opt to teach the two as entirely different concepts and not mix them at first.

No idea if an appropriate IDE/plugin combination exists! Surely there is one.

One suggestion that someone endorsed in another comment is to use VS Code. This would not be my preferred choice, but it would be useful to run an experiment in which students who are new to code start by installing it and following its suggestions about how to configure a working environment. One advantage with this approach is that it introduces students to a text editor.

A second alternative is the Idle environment that is included with every official install of Python from python.org. I have not used it much and I've never tried to teach a course with it. But it comes from the most trustworthy source in this complicated environment.

I'd be very interested in hearing about experience using these to start learning from the very beginning or using them to teach a course for people who are just getting started.

I agree with most of your "Environment as Code" post. Virtual environments are overhyped and even experienced developers frequently have to fix their venv or repair their conda install. You get to depend on a tool that's not necessary and distracts from the original purpose of using the software.

On Unix, it is even trivial to have parallel installations.

Build two pythons:

  ./configure --prefix=/home/foo/a && make && make install
  make distclean
  ./configure --prefix=/home/foo/b && make && make install

Install packages:

  /home/foo/a/bin/python -m pip install bar
  /home/foo/b/bin/python -m pip install quux

This is completely isolated. You can do the same by using the Windows installer to install to different directories. If an installation breaks, remove it and reinstall.

My experience is that people who recommend various environment software often like strict bureaucratic procedures, or have a financial interest in pushing the software or are simply not experienced and do what they are told.

I agree. It would be better if people understood the costs and benefits of virtual environments and used costs and benefits to guide decisions about how and when to use them.

The general observation I would add is that these change with the level of experience of the person writing the code. I found it helpful and harmless to avoid them when I was experimenting. Now, I use them automatically.

I know that it made people angry, but I think it was reasonable to say that if someone does not know how to edit a file and is not comfortable using the terminal, they are not ready for virtual environments.

Re separation, another option is the one I recommend in the "Environment as Code" post: Right now, I'd suggest installing

- Python3.12

- Python3.11

- Python3.10

Then use edits to `.zprofile` to specify which will be used in any terminal session. It does require the use of an editor to make changes to `.zprofile`, which is where the official versions put the lines that add things to the user PATH. I think it is very helpful to get people familiar with how profiles set PATH for any terminal session and how easy it is to control by commenting out or commenting back in lines in `.zprofile`.

Later, one can introduce them to `.zshrc`.

You can just keep Anaconda and just use it as a pyenv replacement, e.g.:

  conda create -n "myenv" python=3.8
  conda activate myenv

Then just use pip in your environment, if you don’t like conda packages. Not sure what the paragraph about being able to install multiple Python versions after getting rid of Anaconda is about.

I had to do this, using Anaconda as a pyenv replacement, because I worked with people with many levels of IT bureaucracy. They couldn't just go to python.org, download an official release, and use it within their user profile. They were only allowed to use Anaconda because it was on a list of approved software.

Same here. It’s not that bad of a solution when you think about it.

Anaconda (and miniconda) is a very good solution if you need a container-like environment. I can easily go back to old projects with different python versions and do maintenance in them with no hassle.

pyproject.toml is a better solution

Does it handle binary dependencies and Python ABI changes well? Does it isolate them from almost the entire operating system? Conda does those.

Conda packages compiled such that the search paths of the binaries are not using the OS's (which why the Linux DE Qt theme doesn't work for Spyder).

Conda also comes with well optimized binaries for high performance compute which is absolutely a must for modern data science.

Use docker/podman if you are worried about ABI changes and isolating them from the entire operating system.

Most people's problems getting their Python toolchains to work optimally are caused by using operating systems that don't come with build utils. That's a cultural problem solved by using an operating system with a culture of distributing those tools.

Or use conda.

Or use debian

yes, because installing random python libraries to system python on Debian never broke any system.

Use poetry/Debian. Standard Python tooling is acceptable in 2024; using some busted 3-plus-year-old "supported" environment with a package manager with a really busted constraint solver, which doesn't even come bundled with compilers, is unnecessary.

I've installed plenty of compilers using conda - how long ago was it that you last tried it? even ROS is available in conda now, instead of requiring specific obsolete versions of Ubuntu.

That's installing another 3.8 rather than using the system one, isn't it?

Yes, you can specify any Python version available in the official channel: https://anaconda.org/main/python/files

If you have license issues with the official Anaconda repos (talked about in other comments), switch to the conda-forge channel instead, it doesn’t have these restrictions: https://anaconda.org/conda-forge/python/files

This would be the sane advice.

No, because half the students in your 200 person class don’t have anaconda installed and now you are managing students in yet another state of python environment management with TAs who are economics grad students, not python experts

> Not sure what the paragraph about being able to install multiple Python versions after getting rid of Anaconda is about.

The sentence says "install and run". The problem is with "run."

On macOS, most students who install Anaconda accept the default, which is to autoactivate the base conda environment every time someone starts a terminal session. As a result, the instructions for starting an official Python don't work.

If you use Anaconda on Windows, you will not be able to test the effects of autoactivate.

I did not know that (only know it from Windows), thanks for the explanation.

It looks like

  conda config --set auto_activate_base false

would be another fix for that. Or adjusting .condarc manually if you insist on not using the CLI. However I think teaching students about (virtual) environments will require them to know at least some CLI basics anyway.

Yes, this would be another way to escape from the conda base environment.

And as I've indicated in other responses, I absolutely agree that students need to learn how to edit files and run commands from the terminal. The only question concerns the order in which to teach these basics.

But please understand the facts. The majority of students I've encounter who have Anaconda installed on a Mac, had totally given up on the possibility of running an official version of Python. This, by the way, is a big indictment of the Anaconda Navigator, which encourages students to keep using a GUI instead of mastering files and shell commands.

Part of why I recommend the basics:

- an official python - pip and pypi.org - venv from the standard library

is that it gets them out of this pattern of this dependency on a GUI. This, by the way, also gives me some hesitation about VS Code. It too reinforces the GUI as the way to set up a working environment.

If your students are in a software engineering or engineering adjacent context, they should have some idea as to how the things they use are put together and how to adjust them.

Hyper-specialisation to the point where you "know python" but do not "know environment" (be it environment variables, virtual environments, shells or operating systems) seems like a rather pointless exercise.

If, however, someone wants to scope their class to teaching only a single thing and dismiss everything else and stick knowledge in a silo, why not provide them with a preconfigured option? This is an avenue that works with many languages, with or without GUIs, and with many steps or very few:

Maybe a Docker Container and a free download of Rancher Desktop (for any major OS) is a good option. You give them 1 link and 1 command (or just a picture of the GUI!) and you're good to go.

If that doesn't work because you're using some sort of fancy editor that doesn't work with containers, the fancy editor (like PyCharm Community) usually comes with a built in option to spawn a managed python environment just for your project. So that's a second great option to "solve" this problem that shouldn't exist in the first place.

If both of these options are too 'big', there is the super simple option of not involving the local computer at all. Stick them in a webbrowser. From CodeSpaces to just Python playgrounds, there are a ton of indestructible options to pick from. Works for other languages as well.

All of this will allow anyone to not learn anything about where a language and runtime sits or how to make it do what you need it to do, and instead you can write a bunch of code that doesn't interact with the world at all, but still get a certificate that says that some course was followed.

I collaborate with people that use macOS while I use Linux. For reproducible environments, Nix has served us well.

Despite the myth, it is not hard to use unless you need to package something complicated or messy. Then it may become hard.

The advantage of Nix is total version reproducibility and declarativeness. It's quite reassuring to know things are exactly the same on all computers.

I am fully on board with having reproducible declarative setups, line Nix or Guix offer. I want to note though, that the versions and original code may be the same, but it can still happen that something behaves differently on another OS. The code of some version of a package can intentionally do things differently on another OS. So you are not 100% safe from MacOS disturbing things.

I wish though, that at my job people were as far as using Nix oder Guix though. Still a long way off of that. Wish people would explore more on and off the job, to realize the potential.

Sure, but if you run the same Nix flake on two macOS machines with the same architecture, it will have the same behavior.

Most of package managers don't offer that guarantee. Only Guix has a similar level of assurance?

Nix openly recognizes outputs are different per platform, x86_64-linux is different from aarch64-darwin. Impossible not to, as ultimately architectures may behave differently.

What we did at our job was I set up a nix config for myself, because I wanted it. When new people onboarded, they had two options: install this list of software or install nix and it’ll all get installed automatically. Enough people chose to use the nix config over time that it became the de facto standard.

I like this strategy. Nix can be tricky to explain, so sometimes showing is much easier than telling.

Although Nix seems to be becoming more mainstream, I still have yet to encounter a (forgive me for lack of a better word) normie that uses it.

I feel somewhat blessed that I get to work on tools at a company that can support a dedicated tools team. We are free to use the best tool for the job without needing to worry about popularity contests, and nix has truly been a game changer for us.

I think it is already seeing some adoption at normal & boring companies, which is a sign of maturity.

Frankly, the biggest problem is that it needs some better docs, i.e. some serious corporate funding to get them written.

It’s not a myth, Nix is incredibly complicated to get into and is actively hostile towards its users with the abysmal DSL and documentation. Every company we’ve tried to introduce it in has failed to adopt it primarily due to its incredibly steep learning curve

It is quite complicated to understand or to set up, yes, but I agree that it is easy to use. We use it at work for our dev environments. Like two or three of us are able to actually add packages, adjust nix configs, etc. Everyone else just runs `direnv allow` and automatically gets everything they need to build and run out software.

If you don't have enough manpower who can control and collaborate on the development of the environment, the project will not succeed to be adopted.

Programming language-specific dependency tools succeeded due to their user friendliness, however abysmal they are in multi-language / OS dependent environments (and I despise them for that reason). Cargo.toml, go.mod, conan.txt, Dockerfile etc. all are can be understood by basically all members of a dev team. You cannot say the same for Nix for most devs in a team.

Our direct experience counteracts this point. We have docs on how to add nix dependencies, and a file where they’re defined. For anything already packaged in nix, everyone can figure out how to add a line in that file to add something to everyone’s dev environment. We have over 3 years only wound up having to write our own derivations for two or three things.

We still use language-specific tools for those dependencies, though, since that is what people are familiar with. I think this strikes a fine balance: nix describes the “operating system” that we all work in, and then Cargo, npm, etc lockfiles describe each language’s dependencies.

> It is quite complicated to understand or to set up, yes, but I agree that it is easy to use.

The problem is that everything works--until it doesn't. And then nobody knows how to fix it.

You wind up needing a single person who serves as local "nix tech support" who then hates life because he's a high end devloper dealing with nix n00bs all day long.

Is Nix still hard to uninstall on macOS? That's what scared me off but it's been years now...

this installer comes with an uninstaller https://install.determinate.systems/

Thanks for the link. Apparently there were issues even as recently as 4 months ago, specifically conflicts between the determinate installer and nix-darwin though I'm not sure they are intended to be used together.

https://news.ycombinator.com/item?id=40158558#40159195

The issue is just that Nix-Darwin doesn't 'own' your system, and it's conservative about blowing away existing files when installing files that it has generated. So if it detects that some file it wants to install conflicts with an existing one, it throws its hands up and tells you to deal with it before it will proceed.

As it happens, /etc/nix/nix.conf is one file where such a potential conflict can occur. As an optimization, I think it will (sometimes? always? idr) replace a file if the existing one and its intended replacement have the same contents. The defaults /etc/nix/nix.conf in the mainline Nix installer and the defaults for /etc/nix/nix.conf in Nix-Darwin are the same, so it doesn't complain there. But the Determinate Nix Installer ships a nix.conf with different settings, so Nix-Darwin refuses to replace it, to avoid clobbering your (the Determinate Nix Installer's) settings 'changes'.

There's no real compatibility issue, and the Determinate Nix Installer is doubtless intended to be used with any Nix-based software, including Nix-Darwin, Home Manager, etc. Nix-Darwin installation is just a bit hairy, as it's unfortunately always been.

NixOS is a better experience in that respect, and you should consider running NixOS in a VM if you want to get a feel for that kind of systemwide, declarative Nix configuration without worrying about installing it against a foreign base system. (If you do learn your way around nixos-rebuild, you won't need the Nix-Darwin installer anyway.)

I've had nix disappear out from under me a couple times on Mac, enough for me to start drifting away from home manager. Sucks cuz I like the tooling!

But seriously, let's say you have a Python project (like a Django application with a couple requirements in your `requirements.txt`), what is the least disruptive way to get it working? Many things I've looked at involved doing one-time transformations of requirements files or otherwise easy-to-desync things...

This is really elegant. What a nice solution; single-step and reversible.

It’s also excellent documentation, written by someone who clearly knows their audience. It keeps all operations in GUI-land, which most users consider more safe. It avoids almost all technical explanations.

Folks responding with “Terminal is easier” are missing context. You’re not the audience. The fact that you can come up with seven solutions for this proves that :)

> It keeps all operations in GUI-land, which most users consider more safe.

Which is unfortunately exactly what enables so many tech support scams, so I'm not sure it's a pattern worth reinforcing.

In addition to that, while it might still be helpful for completely non-technical users, I really don't think "avoiding the terminal and edits to files" is a desirable goal for anyone studying either Python or data analytics.

Notoriously, people get to desire their own goals, and there’s little that we can do about it. Plenty of people desire to learn and use Python without the terminal. That’s 100% possible in a professional environment today, never mind a classroom.

Most people learn computers because they want to get better at a task, like programming. The point of this post is to help people get unstuck so they can begin learning at all. Adding more gates in front isn’t going to help.

"If you teach a course that requires Python"

"Many of these students have not used the command line."

I'd suggest that we are failing to teach students learning to code, the basics of how to use computers. Pre-requisite should be a class on how to use the command line.

I thought a class of entirely non-tech art students how to use the commandline to run their python code within an hour and that includes distractions like people using multiple different operating systems and explai ing how paths work on each. Tech inclined students get the gist of the whole thing in less than 15 minutes.

The CLI is literally just: You write a command and a thing happens. That isn't a thing that people get hung up on. The main weirdness you need to tell them about is how to select/copy/paste and use the history. Throw in a lesson about how paths work. Sure the commands need to be remembered but for a start you basically just need pwd, cd and python/python3.

You are not doing your students a service if you try to teach them programming by shielding them from anything that could give them a feel for the systems they develope for.

If anything I think a pure CLI environment would be easier for a lot of people than the incomprehensible skeumorphism-less jumble of controls that a lot of GUIs have turned into. Everything is built in text and done one command at a time, so there's much less mystery compared to trying to figure out what's even a button or not.

I suspect it depends upon the person. Plenty of people have difficulty remembering commands, and only the clever ones in that population will create a cheat sheet to help them along until they do remember them. On the other hand, they may have no trouble remembering where something is or what it looks like.

That said, I do resent people who claim CLIs are harder because of that. What works for one person may not work for another.

Idk. I am from a generation where my 12 year old peers would blindly type SMS on their Nokia number pads below the table.

I think we massively underestimate what people are able to do – if motivated. Making sure they are motivated is our task. The CLI is awesome and not just for full on nerds.

I don't understand the mentality myself, but some people just turn off their brain immediately if you say "command line", but somehow they see the arcane key combinations that you have to remember and do in the right sequence as more of an exciting challenge.

Well, some people or *cough* hackers feel extra special when pointing out how crazy complicated it is.

When I shown the CLI I make sure to switch white on dark text to dsrk on white text. First it is more readable that way on a projector, second people are less likely to switch their brains off. A bit like xkcds idea to make graphs look hand drawn to reach people better.

do they? can you give an example?

A lot of students arrive at university these days without understanding what files and folders are: https://www.theverge.com/22684730/students-file-folder-direc...

I am aware of that. I see it as my task to make sure they leave without that material gap in their knowledge.

Somw of them might know more about files than they wish they would, e.g. the group whom I showed how to manually read broadcast wav file using a hex editor. Of course as an example to figure out where the actual data of an audio recording goes and what role metadata plays.

Most of the kids the author is talking about aren't going to ever code for a living - they're going to code to get their job done. It's a very different mindset.

By that logic... we might as well stop teaching math because kids are never going to become mathematicians.

are you thinking arithmetic, trigonometry, geometry, analysis or statistics?

statistics is not mathematics -- real

Never took it, huh.

two second search -- "Statistics is considered a mathematical science that is distinct from mathematics, though it does use mathematical methods:"

"statistics arguably is not a branch of mathematics. It is a mathematical science, built upon the mathematical discipline of probability. Some ways in which mathematics and Statistics differ include: Statistics often does not produce definitive conclusions whereas mathematics usually does."

In other two second searches: Lab Leak is Real, Ivermectin cures COVID, 9-11 was an inside job.

LLM's give you what you want to hear.

Now get some live quotes, both for and against, from actual working mathematicians.

There are a lot of definitive conclusions about lines | planes | surfaces that "best fit", by many metrics, sampled data, etc.

this has nothing to do with LLMs at all.. you are uninformed about the old feud between mathematics and statistics so you claim this is not true.

anyway, on reflection, the word "is" does not fit here.. statistics and mathematics are not disjoint, but no, statistics is not a field of mathematics.. ask around among people with serious graduate studies in mathematics and they will fill you in.. Maybe it overlaps into the academic practice and their department structure too.

> you are uninformed about the old feud between mathematics and statistics

A little knowledge is often dangerous. Sometimes it makes one look like a fool.

> this has nothing to do with LLMs at all

They are plugged into the backend of many search engines these days and they excel at picking out the nature of the pattern | result that a questioner wants and feeding back what they want to hear.

> ask around among people with serious graduate studies in mathematics and they will fill you in

I checked in with Terence Tao who I first met at an Australian math club get together in the mid 1980s .. he pointed out his 2007 paper on The Dantzig selector (a novel statistical estimator for linear regression) and seems fine accepting statistics as part of mathematics.

I'd hate to cite myself given my rather dull work, but I feel that Terrence surely qualifies as someone "with serious graduate studies in mathematics" given his Fields Medal and UCLA professorship and all that jazz.

Do you have someone more qualified in mind?

    Emmanuel Candes, Terence Tao "The Dantzig selector: Statistical estimation when p is much larger than n," The Annals of Statistics, Ann. Statist. 35(6), 2313-2351, (December 2007)

hah! certainly respect to Terence Tao and his Fields Medal (!!!!) .. probably not worth beating into the ground .. '-)

> in the mid 1980s

$\tau as \epsilon$!

Historically, it took some time for statistics to become mathemathised - they started out as letters to the higher state bureaucracy level with the modern era equivalent of bullet points, then some tables with still mostly words in them, IIRC mathematisation mostly started in the mid-19th century, and maybe only grew much heavier once physicists started to adopted some of these techniques ?

But I feel all this discussion has gone off topic that was the teaching of statistics in our post- (post-?) modern era.

That's right, statistics is done with a Ouija board, not mathematics.

statistics really is not mathematics according to serious math people. It feels like "eternal September" with your replies.

are these "serious math people" in the room with you now? how do they write their statistical operators, in cuneiform?

Unluckily this inability of seeing with each others' eyes - and the unwilling to do so: 'it is easy!' kind of never asked for flow of oppinions, ... daaaa, it is always easy AFTER you know that -, this inability makes the software products that damn shitshow for the user.

Beyond the attention seeking 'tips of the day' kind of popups in random places and stages that derails you all the time (I mean: ALL the time) but at least so tiny little fragments of wisdom that you reamin the same clueless, well inside the error of measurement in cluelesness. Only good for putting The Product into the focus (Am I nice, eh? What a cool feature I have, eh?! Brilliant am I, eh?!) instead of the need of the users.

The audience is people learning a programming language. It’s too much to ask of them to open up a file written in a different programming language and comment out a single line?

I work in HPC, and for all the Python programmers I come across, I'm surprised at how few of them know details about package management etc.

Of course I'm quite a bit biased because the majority of these people are coming to me for help...

Back to Anaconda. It's great system, and the problem is people learn how to use ML or whatever using Anaconda, and then when they get into the workplace, you need a license.

I really wish organizations would just buy licenses for the users that want them, it's so much of a pain to work around not having Anaconda. Their prebuilt packages are pretty helpful and time-saving, even thought you "don't need it" and you can get along with using just pip, but you'll have to provide workarounds for projects that use Conda.

The Anaconda Distribution has pretty good coverage of what scientific/technical Python users require. I'd wager that they rarely need to concern themselves about package management.

I had a pretty rough time with Anaconda Team Edition when it was rolled out by my employer. The security team required that any package with a high CVE score be filtered out of the on-prem repository. Fair enough, but it just led to packages and dependencies being constantly broken. IIRC, it wasn't possible to give normal users the ability to view CVE scores, which just compounded the issue.

Nobel laureate Paul Romer sharing some Python advice is a little strange!

Whatever project solves Python dependency management deserves to get a Nobel in Economics for saving humanity billion of hours per year.

Usually I simply use pip and poetry. Haven't run into an issue in a long time. Privately I use Guix often, when it has the required packages. I find the dependency management mostly to be already solved. Perhaps I am not riding along the edge enough, to see things broken.

Poetry does a lot of good stuff: lock file, dev dependencies (and other customizable groups), easy to use virtual env, dependencies pulled directly from git.

I do not recommend pip in serious projects, as it lacks the above. Here's things that I've seen:

- add dependencies and not save them in the requirements file - forget to activate cenv and install packages globally - different package versions across deployments due to absence of lock file - main dependencies mixed with dev dependencies

Pip has gone down the shithole with its preaching about venvs. It's my computer. I'll install to site-packages if I damn well want to.

What I usually do is make a venv using pip, then use that venv to install poetry in it, because I don't want to clutter my system with outdated poetry package. Then use Poetry to install packages inside the venv. So the pip part is merely for bootstrapping.

You shouldn't have to launch a sub-shell to run a script interpreter. That is exemplary of how nobody in the Python leadership can fix their perpetual packaging f-ups of the last two decades and are now just sticking their collective heads in the sand.

Just invoke the python interpreter from its venv path. You don't need the sub-shell.

Anaconda’s hidden license for institutions is a dark pattern time bomb for scientific computing.

Can you elaborate? Is there a link?

A news on the same: https://www.theregister.com/2024/08/08/anaconda_puts_the_squ...

> use by individual hobbyists, students, universities, non-profit organizations, or businesses with less than 200 employees is allowed, and all other usage is considered commercial and thus requires a business relationship with Anaconda

Wow this is so deliberately ambiguous about universities with more than 200 employees. Shameful.

I posted an update about this: https://www.linkedin.com/posts/pzwang_hi-everyone-recently-t...

TLDR - we are working to clean up this language to leave it clear that educational institutions are exempt, and that these commercial terms do not apply to third-party channels hosted at anaconda.org (which includes conda-forge).

You’ve had the existing language up for years. Your licensing regime is a dark pattern torpedoing anything Good about Anaconda. Anaconda is threatening academic researchers for back usage in something those researchers thought was free.

Show us don’t tell us. I’d love to not have to continue the rip and replace job ahead.

Edit: this linkedin response sums it up well. Not certain what guarantee you will make to research institutions’ leaders that is going to lift those blocks.

However, please tell this to your sales/legal teams: the nature of their approaches has been somewhat poisoning the well. If leadership’s first encounter with a piece of software is: "you are in legal trouble", the reaction you're going to get is: "remove and block that legally dangerous piece of software at once", rather than "oh yes we should license it". We do buy licenses for software, we view it as giving back, but how the offering is first presented matters.

Thank you. I do hope you reflect on how ambiguous the language was allowed to be for years.

But this doesn't apply to miniconda if you use the channel conda-forge, so I'm not sure there's much need for a fuss here.

source: https://www.anaconda.com/blog/is-conda-free

We've been hit by this.

When I talked to them four years ago, they agreed we were good to use it for free, no problem.

The dollar figure they're asking for would make it the single most expensive software product we would be licensing in our enterprise, by a lot. The deadline is absurdly soon for such a big deal. And they opened discussion in an incredibly hostile manner and have made no attempt to work with us.

So, I'm helping lead the effort to completely purge them from our ecosystem. On the one hand, I'm sad because their stuff is pretty good. On the other hand, their behavior is bad and the product isn't better by the amount they're asking for.

Good riddance.

> And they opened discussion in an incredibly hostile manner and have made no attempt to work with us.

With $$ for unlicensed period too?

Just switch to pixi.

Miniforge [1] provides almost all the same benefits and features without the licensing headaches. The commercial/sign-up nagging is gone, and the GUI launcher is the only thing I've noticed is missing, but I don't use that as it's extraordinarily slow.

[1] https://github.com/conda-forge/miniforge

Pixi takes it a step further and provides for project management rather than just environment management.

As expected, I see comments here like "Avoiding sh overcomplicates this." It's true. I found the shell-avoiding instructions no better, and possibly worse, than shell commands.

But: what's sauce for the end user goose is sauce for the developer gander. Developers are users. Give developers good visual tools. Sadly I don't think there is a visual front end to git,, for example, that is any good among the handful I have tried and the dozens that attempt to make git safe, discoverable, and visually obvious.

Wish there was something like magit outside of emacs that I could recommend to people.

tig?

This may be a backwards way to go about it, especially with an end goal of learning Git, but I feel that having learned Mercurial first helped me to more easily move to Git. Ultimately, it was using BitBucket's Mercurial repositories, and then Atlassian stating they were discontinuing Mercurial that forced me to jump to Git. Wanting to keep my Mercurial history forced me to learn parts of Git that I otherwise probably never would have touched. It seems Git has (at least for the present and near future) won the SCM wars for popularity. Working with mostly front end developers unfamiliar with code tools made me the go to guy at work, so I had no choice but to learn as much as I could about handling conflicts, as well as things most people rarely think about like a project's .gitignore and .gitattributes

I’m not sure about what the problem here is. If the target user cannot be bothered to learn about “the shell” why should they care about the Python vendor? And the procedure shown is not easy peasy for a newbie.

TFA is for students who are (presumably) very early in their programming careers.

Python is often billed as a good language for beginners because it does not require the author to keep track of curly braces or line-ending semicolons, and it lets the author focus less on syntax and more on reasoning with code.

Wouldn't it be nice if the beginner could focus on learning Python instead of having to learn stupid minutiae that only exist because we're still trying to maintain backwards compatibility with the VT220?

But you can learn Python with Anaconda. At the point you are sophisticated enough to outgrow it, wouldn’t you be better offering a drink or two to your sysadmin friend and have them fix it?

As an analogy- It may be just as easy to do, but I’ll never bother to change the spark plugs in my car - I’ll have someone knowledgeable do it. Do I think it’s hard? No. Do I think I have better ways to spend my time? I do.

Is dragging and dropping the .zshrc file the easiest way to run anaconda on a mac? It sounds incredibly inconvenient to me, and considering it is intended for people who are clueless about terminal and .zshrc, it may result in a mess with different .zshrc versions in case one installs anything that runs through terminal and changes .zshrc, which can be a lot of things.

At some point they'll have to learn about the file system, shell, network. Otherwise there is little point in learning Python.

Octave, Matlab, Mathematica, Julia, R are superior for small programs with just a couple of functions.

Escaping the stranglehold of Python would be the next project!

Well said. I am grateful that my employer licenses Matlab, Simulink, and a lot of toolboxes and blocksets. The IDE and especially the documentation help me get stuff done. We've had waves of Python as a Matlab killer first with Enthought and then Anaconda as the repository. I gave Python an honest try but was spending all my time searching for help on package after package with poor to no documentation and trying to debug code with substandard tools.

Matlab is a categorical loss due to its proprietary character, R debatable, Julia, maybe yes. Octave, while at least not proprietary, still surely not, if the task is anything but some matrix multiplication or matrices using program.

julia has an unacceptable overhead when running small programs. it works well for longer running workloads or extended interactive sessions, but for learning you want to be able to write and execute small scripts without feeling the perceptible lag every time. google up "time to first plot" for more on this.

It' s quite acceptable with 1.10.

>At some point they'll have to learn about the file system, shell, network. Otherwise there is little point in learning Python.

Sure, but I would argue that overwhelming someone with all those topics at the beginning is counterproductive.

I think it's possible that learning programming could be the gateway to learning those other things, otherwise they're of little use. Many people don't even need to know the filesystem. They store their stuff in Google Docs or somewhere like that. If they use mostly canned software (such as EPIC or SAP) for their jobs, then their data are stored in some magical special unknown place.

I learned programming (in BASIC, in 1981) before I heard about the filesystem or shell. Making more complex programs do interesting work was my reason for learning more about the innards of computer systems.

Probably because of anacondas licensing change...

Avoiding sh overcomplicates this. Just give them a one liner: mv ~/.zshrc ~/._zshrc and vice versa. Pasting a cmd is not rocket science.

Man you say that but many people simply do not understand that a command even is. They don't know to hit return after pasting or how much of the command to copy. Or they'll paste it in a text editor.

There's just a lot of people that did not grow up using computers the same way as many here. Whether it's they grew up on mobile devices or have only ever used a GUI.

I don't think it's reasonable to expect a developer to not understand command lines but plenty do not. There's whole populations of very junior programmers that have zero experience on the command line. Hell I know people with decades of Unix experience that don't know basic CLI stuff or reach for the GUI to install packages or manage things.

then they need to stop what they are doing, and learn how to use the system they are running first. there's no value in skipping ahead of basic "power user" learning for your preferred os. if you can't run basic commands, you dont' need to be writing python programs. if your educational institution is not providing you basic knowledge on how to use windows, macos, or linux, they are unfortunately failing you.

This PoV of software is why Excel (or Microsoft products in general) almost always wins against any database or specialized tool for creators, engineers, accountants alike.

macOS (and the entire Unix universe) is a shitty environment for people who rarely / don't need terminal but whose job could be improved by a lot if a scripting language is used.

Microsoft used to champion intermediate users who need more power than mindlessly clicking web pages. They are now turning their products yet another copy of the other systems where you are presented with a cliff as the learning curve.

I disagree completely that a simple programming environment should require terminal access. Terminals are unforgiving for mistakes that a beginner makes. Most of the terminal capabilities should be done in GUIs. One should be able to access all files, edit them, change the OS environment variables and create programming projects in GUIs. We achieved that in 90s already. Why should we go back?

> Hell I know people with decades of Unix experience that don't know basic CLI stuff or reach for the GUI to install packages or manage things.

What does "Unix experience" even mean if you can't sling the basics in the shell?

It’s always baffled me that the default configuration of conda takes the liberty of contaminating the default shell environment. It feels incredibly bold to assume that because I install conda I want it to exist in every context in which I use my computer. It’s the terminal equivalent to an adware “toolbar” installing itself in your browser.

Thanks for making this point. People who work on Windows may not realize how aggressive and disruptive Anaconda is on macOS. If you open a terminal and run the `lipo` command that macOS gives you to get information about a file, you will be running a version of that command that Anaconda installed, not the one that Apple manages and updates. And `lipo` is just one of many such examples.

Based on limited testing on Windows, Anaconda does not seem to do anything comparable on that OS.

Not only that, installation edits some of your dot files without ever telling you or asking permission.

I always appreciate a post about a simple fix to a problem a lot of folks run into thats not esoteric and lays it out simply :)

As engineer of many many years now all I can say with certainty is that confusion, failure, research and eventually small successes have been the central theme of working with computers. I personally love the struggle and mystery but its not for everyone. At the end of the day its about what you want to get out of it. If you want to write programs without learning about the terminal go for it, but I would say there are better gilded cages with GUIs that help you get a lot more done within that constraint.

If you want a deeper understanding, I think its more helpful to admit you can't cheat the process of learning. Slow is the only speed.

Alas, the audience targeted by the post would most likely have preferred a video of the instructions rather than text. For tasks related to configuring a visual computing environment (such as the desktop), this would also make some sense.

Might be a lost cause in the tiktok era, but the text / command line / unix pipes paradigm of computing is extremely powerful and it would be a blessing if educators would find way to make it attractive to students...

That website has both.

Relatedly, this seems to be the best advice for installing and using Python on t he Mac. Probably applies to Windows as well.

https://blog.glyph.im/2023/08/get-your-mac-python-from-pytho...

Why not teach students how to work with their .zshrc file and become better computer users? I've never understood the aversion in academia to teaching people how computers work. This could be "fixed" with a simple sed line or `vim/emacs/nano ~.zshrc` rather than a multi step process of dragging files around and futzing around in the UI. I see this all the time in higher ed, so much wasted time and effort coming up with long complicated procedures that appeal to people unwilling to learn the basics of computers rather than just teaching effective computing usage. Especially now in an era where computers are a tool almost everyone needs to know how to use to succeed in the workforce, it just doesn't make sense. Learning how to be effective in Unix should be a required 100 level class for almost every major by now.

What aversion ? Generalist basic computer usage courses are mandatory in undergrad, and this includes basic Linux command line usage and shell scripting.

But those might be too late if the students run into these issues during their very first weeks. And Python is pretty decent for Hello World kinds of programming-adjacent courses.

What is an “official version of python”?

Anaconda is a distribution, which to my understanding, includes an official version of Python. Anaconda includes its own way of doing things to solve a lot of common problems people face.

Personally, I don’t use Anaconda any longer because I ran into so many issues over so many years with conda, their package manager. I’m also almost competent enough to deal with a lot of the problems that anaconda tries to solve. This likely isn’t the case for beginners.

Virtual environments are great and, I’d argue, what people should be using 99% of the time.

What do you use in place of Anaconda ?

Homebrew, pip, venv

^ seconding this

This seems like a lot of work to bypass something in the zshrc file vs. doing something in a terminal emulator? I would assume Python developers are somewhat forced to use a command line anyway.

As described on https://docs.python.org/3/using/mac.html , python on MacOS both ships with an IDE and the ability to run scripts without using the command line.

macOS doesn’t ship with a version of Python anymore?

Seems a complete farce to learn Python without learning how environments work seen as the way it's designed it's impossible to use for more than a single project without them.

Contrast the python experience with R, MATLAB, Mathematica, (even Julia). That should be what python ecosystem should aspire to. Blaming students for not understanding command line, virtual env, package management versioning etc.. and screwing around with magic incantations (to them) is just a sad reflection of how unfriendly this frankensystem is. Doubly sad that python is the default way to learn anything nowadays.

Python's package management isn't great, but at least it exists. R is absolutely awful in this regard, 99% of packages don't even specify versions for their dependencies.

Try installing a specific version of an R package from more than 2 years ago, it's an exercise in frustration,

This is insane lmao. Why not just edit out the line that added into your .zshrc (or by extension the .bashrc) instead of running through all these complicated GUI specific things just to basically rename the file so that ZSH doesn't load it when it runs a shell???

This is exactly why you should learn to use your tools properly, learn just a tiny tiny tiny bit of posix shell (which works for ALL posix shells sh, tcsh, bash, zsh etc...) so that you can understand what the heck is going on.

Like seriously, this is day one stuff. Your shell loads a configuration file and you can set programs to run on start, or configure shell options, set aliases etc... This is nuts man.

in fact I was so baffled that I wrote my own article response https://sweetbbak.github.io/post/response-to-mac-anaconda/

While I agree in principle, this is a professor sharing as foolproof a method as he can find to get many students, whose existing knowledge, setup, and CLI avoidance he can't control, to the meat of the course experience he has designed for them.

FYI, I offered a comment above that might clarify. There are two questions:

1) How can we get a specific group of students who feel helpless started on the path to learning?

2) What should we teach students?

In the post linked here, I tried to be clear that these are instructions for a very specific group of students who are stuck and don't know what to do. I agree that they are not the right instructions to offer to the typical reader of this site.

The comment and the second post that it references address what we should teach.

I’m so happy with MacPorts I find all this discussion weird.

True, MacPorts isn’t the official Python installer, but macOS doesn’t have an official Python anyway.

This seems to come from a place that "learning virtual environments is too much work". I do not agree with that premise. Having a lab where you introduce environments and the command line (which you can do in a single class for the types of students Paul is talking about) is not a big deal and to me is a good use of time.

If you give a man a fish, he eats for a day. If you teach a man to fish, he eats forever.

These instructions are a fish, and don't really any is the core principles and knowledge that a developer should really know.

Yes, it fixes the problem but doesn't really address the "Why?" part at all.

Install orbstack, problem solved. Want isolation per project / a “virtual environment”? https://github.com/jrz/container-shell

Well he didn't win a Nobel Prize in Python best practices, that's for sure.

Install python from source into a directory that does not need sudo. Use poetry init and poetry install to do the rest. Poetry manages the environment itself. I have been using this setup for a while and it works quite well.

This is typical solve-one-problem-with-another approach. Anaconda may be bloated but it is really straightforward to use from the navigator and yes you can have many different Python environments installed at the same time.

Pixi works on MacOS. It's way better than anything else at this point.

I still don't really get what conda is or does. AFAIK it ensures ABI compatibility between Python packages that have C extensions? Does manylinux and the limited API not take care of this?

Way back Anaconda was an accessible way to get Numpy running without having to scrounge up a Fortran compiler for one’s MacOS version.

What if you want to ensure that the same packages are available, built in a similar way, between your Mac and some linux servers? What if you need to share or ensure that your projects work between Linux and Windows? What if you are supporting a lot of less-sophisticated users across a number of different OSes, or even different versions of the same OS?

There are a ton of subtleties in the build toolchain even within a single OS type, and these will lead to downstream frustrations with packages that just bundle pre-compiled versions of C/C++ libraries.

The conda approach treats the underlying libraries as first-class citizens in the package ecosystem; tracks their interdependencies; and most packages are built in such a way that they are relocatable on your filesystem and don't require system privileges to install. conda and the various packages in the conda universe (whether official ones from Anaconda or the community-built ones in conda-forge) all make it so that this baseline hard problem is mostly solved across all major OSes, and solved in a relatively consistent way.

You can think of conda as a cross-platform, cross-architecture, multi-language, userspace package manager. It grew into this because numerical Python's package ecosystem is so horribly complex and the users span such a huge set of install environments, that conda ended up having to be a generic rpm/brew/apt kind of thing.

Not https://en.m.wikipedia.org/wiki/Anaconda_(installer)

> Many of these students have not used the command line. Many do not understand the difference between a text editor and a word processor.

After taking a course using Python?

I use mise for version and virtualenv. It works.

python -m venv .venv source .venv/bin/activate

i've never understood the reasoning for anything but that

Indeed, it is high time for a new Fedora installer for Macs, the Anaconda monopoly needs to be broken! ;-)

Clickbait title (“stranglehold”, really?), and also really bad advice.

Terminal can be scary, but “open terminal from Applications/Utilities” and typing “rm ~/.zsh” is objectively easier. Also that’s not even the best solution, just a far better way to do this solution.

It sounds like they want to be able to restore it too. And the Terminal is harder to use than a graphical step by step guide.

This guide requires holding down arcane secret key combination, this is not remotely normal way to do anything, and prevents people from learning basic system commands every programmer needs to know.

Also, “mv ~/.zsh ~/zsh-anaconda”.

His audience is not programmers.

It quite literally is for students learning Python.

He is an economics professor, not a CS professor. His students may be going into jobs that require programming, but that doesn’t mean they are “programmers” or that the purpose of the class is to teach Python.

Pythons and anacondas are snakes which catch and kill their prey by strangling them and inducing cardiac arrest. It’s a pun.

Come on, putting “macOS” and “stranglehold” in a title is not merely a (bad) pun; it plays into article titles which adhere to common complaints, and topical complaints about macOS/Apple specifically, in a clickbaity way.

TIL : Shift, Command, . to reveal hidden files!

reminds me of instructions for using curl|bash.

do people really want to do things they don't understand at all?

cant you just do 'conda deactivate' until its gone?

The truest XKCD ever: https://imgs.xkcd.com/comics/python_environment_2x.png

Jesus Christ. Teaching students to program in Python by "incantation" should be an explicit non-goal. How bloody hard is it to explain what an rc file is, and how to edit one?

My takeaway from this article is that Paul Romer doesn’t know how to use the environment feature in anaconda. If he has a problem with dependency resolution speed, ‘conda install mamba’ will solve that.

But really, for his use cases, he should be using Julia (for DSGE modeling) or else R for stats.

Still, I applaud his willingness to try new things, if more economists were like him, we might finally break the stranglehold of stata and matlab in econ.

Reply on news.ycombinator.com