Doing Python development under Mac OS | Hacker News Reader

Doing Python development under Mac OS

peter-whittaker.com

I think it's better to create a virtualenv for each project and install tox and other project dependencies there.

In addition, I'm not sure tox-pyenv is required. I use tox with multiple pyenv-installed Python versions and have never used it.

Base:

    - install homebrew
    - brew install openssl readline sqlite3 xz zlib # recommended
    - brew install pyenv
    - pyenv install <version>  # for one or more versions

Project:

    - create/clone project directory
    - cd project directory
    - pyenv local x.y.z x.y.z ...  # include each version the project supports
    - python3 -m venv .venv   # create virtualenv for project using one of the pyenv versions
    - install the project into the virtualenv
    - install any additional dependencies into this virtualenv too, if necessary

Edit: I'd also add that if you do need a tool to be installed globally, it's better to use pipx or pip with --user. It's generally not a good idea to install things into a system-level Python.

https://pipxproject.github.io/pipx/

This is why I started using Conda recently. It works quite well to create isolated python environments, having many prebuilt packages, and most importantly, system level dependencies. So, you can install some of those, and for anything that is not in conda-forge, pip install works and builds.

just be careful. I don't know if things have improved in conda-land, but when I last used it a few years ago I table flipped and quit conda (and, ultimately, python) when the system-level dependency management got so broken that I was forced to reinstall linux from scratch and start COMPLETELY over (it had broken things outside of its domain). Tensorflow being hard-pinned to CUDA versions probably didn't help.

I've been using conda for quite some time, maybe version 0, I forget, and this has never happened to me. I hope you submitted an issue.

Having gone through the nightmare of setting up tensor flow with cuda, I can't help but think you're letting that color your view of python as a whole.

Tensorflow, cuda, and nvidia were a nightmare that took me 5 days to resolve on ubuntu 18 since it was my first time setting it up ever. There was literally no working guides or people posting with my exact issue, but tons of posts about other people having a nightmare of a time.

In the end I figured out I had to go find this one file/library nvidia had moved around between versions of drivers/cuda even though documentation suggested they hadn't and I was using the versions that were supposed to work fine. In the end, fixing it was either by setting an environment variable because the default path was wrong or editing the path in a config file. I'm not remembering which now.

I dunno, I think the problem is not having a project-specific location for your deps. Obviously you want global deps for a console scripting language, but if you are doing any sort of bigger project, having a global deps location (which is implicitly resolved by the language, no less) is unacceptable for reproducible development for larger projects today. The env hack that python managers like conda and venv use is, IMO, extremely dangerous, because it's hiding state.

These days I use languages that don't do this for my daily drivers, it's better for my sanity.

I've had a better experience using Conda for just Python and Pip, and anything Pip doesn't handle as I wish, then using Pip for the bulk of the dependency tree. Pip's new resolver should make this even better.

Have you tried poetry? It's pretty good.

Not yet. I'm happy enough with my setup.

Some people I work with have had major issues with Conda packaging. Sometimes it works, sometimes not. This can be especially tricky when you’re working with system libraries with competing versions. I still have no idea how a particular set of tools has ever run, given their configuration. The exact same Conda packages work on a local Mac, but fail on a Linux server. It can be frustrating. Because of this, I try to avoid it as much as possible.

I think Conda is trying too hard to hit the middle ground between full containers (Docker or Singularity) and virtualenv’s. If your program is too complicated to be managed with a venv, it’s better to package it up as a full container and be done with it.

Also, I don’t think Conda would have helped here when the problem is lack of specific Python versions installed at the system level. On a Mac (and everywhere, really), I try my best to avoid the main system Python install and use a user-local (compiled) install when I need a specific version that isn’t available by default.

> Yamale’s tox.ini, for example, depends upon two different versions of Python, 3.6 and 3.8.

What?

It seems to me that this awful design choice is root of the problem, and not MacOS at all.

Tox is designed for environment isolation, and primarily for testing against multiple release targets.

It's not an awful choice, when it's exactly what you want to do. The dependency is that tox will run the tests against a Python 3.6 target environment and a Python 3.8 target environment.

The dual requirement is asking for test assurance that Yamale will work with 3.6 or 3.8.

This just means that tox is configured to run the project's test suite against those two Python versions.

Ah, then move along.

For complicated situations like this, nothing beats MacPorts. Where brew is generally based on recipes (read: collection of bash scripts) that may or may not play well together, MacPorts is a package system with well-defined dependencies. The drawback to MacPorts is that everything is built from source, and nothing previously installed on the system is utilized -- so installing Python for the first time may take 30-60 minutes. But the next time is 5 minutes, and you can install all the versions you want, where they'll live side-by-side in peace and love and harmony.

MacPorts is slower, but offers peace of mind -- 'cause when I'm trying to develop something, the last thing I want to worry about is dependency issues (gross)

I've seen MacPorts uses pre-built binaries, at least in some cases. Not sure if this applies to the Python ports.

MacPorts has binary packages. Homebrew uses Ruby not bash. MacPorts uses Tcl. MacPorts just has better QA.

I don't understand. I've been using MacPorts for 16 years and I've never had to wait more than a few minutes for a Python install to complete. Most packages are pre-built. I can even get clang without having to compile. Plus one for MacPorts though. It rarely lets me down and unlike Homebrew stays in /opt/local instead of polluting /usr.

The author concludes the best solution is using brew to install different Python dependencies for different packages. I had a similar solution when I started using a Mac for work back in 2017, but around six months ago I found another way:

1. Install Docker CE on the Mac.

2. Start a container for your work, e.g.,

  docker run python:3.8

3. When you need to get back into it, use

  docker exec -it <first_three_container_characters> ipython

4. If the container is shut down from a reboot or something, start it with

  docker start <first_three_container_characters>

What's really cool about this is `bash` can be run instead of `ipython` to get a Bash prompt and install any Linux dependency. In theory, at some point if one project has conflicting dependencies with another project a new container can be spun up, but this is not something that's come up in practice yet.

This has worked so well for me these past few months, I actually replicated it on my Linux computer. I took a few extra convenience steps like mounting `/home/` and aliasing `ipython` to a command that starts the container before executing the command, but other than that it's been a pretty seamless experience and I never have to worry about OS-level Python installation stuff any more. I use a scipy image with Jupyter Lab in a similar way for my graphical notebook needs.

The only time I would recommend against this is when performance is an issue (like training ML models), but at work we use AWS machines with their own Docker environments for that. For everything else, Mac, Linux or maybe even Windows, Docker is the only way I now recommend firing up Python.

And then enjoy the jet engine that your macbook fans turn into, because holy shit Docker is bad at resource management.

This depends on your base image and what it's doing. If you're running like a large elasticsearch cluster.. you're still running a large elasticsearch cluster. Docker overhead should generally be minimal even with the VM that's running, but make sure to use alpine/slim images.

Agree. I have never enjoyed running docker on my laptop. If I have to use Docker, I use it on a remote Linux server. I had good experience with Vagrant if I have to use a linux environment locally. The only good thing with Docker on macOS is that you don't need to sudo it.

Is the problem that Docker is bad at resource management, or that this is just what happens when you're actually running an entire second OS under the hood?

I appreciate Docker's approach, but it's definitely a bit of a nuclear option (at least on non-Linux systems).

Surely it's an admission of failure if one has to resort to Docker just to get a scripting language to function on a local machine? It's only a matter of time before someone finds a use for Kubernetes on a local machine.

I highly recommend nix for this type of problem, both on linux and macos. You can make all kinds of dependencies available simultaneously in the same file system. You don't have to run vms, or copy in and out of containers. The build cache works great and is much better than a linear Dockerfile. Tweaking dependencies at any level is pretty easy.

It's been a while since I last ventured into nix with python, but when I last tried it seemed way less user friendly compared to the current python solutions like poetry[1].

With poetry I can `poetry add foo` and `foo` will be added as a dependency and importantly all its dependencies will be locked in the `poetry.lock` file.

With Nix either you write all your definitions from scratch, which feels like maintaining a lock file by hand, or you specify a package like `numpy` in your dependencies that is defined in the nix packages and hope that the next time you update nix packages nothing breaks.

Also if you want a package that isn't in nix packages you have to write the definition yourself, which isn't a big deal if you start a project off with nix and gradually add dependencies as you need them, but if you are trying to migrate your project with 200+ dependencies it feels really tedious.

[1]: https://python-poetry.org/

I suspect that the terrible installation/package management/distribution experience is going to be the thing that ultimately drives Python into irrelevance. Beginners are not going to want to figure this stuff out when Go/Rust/etc. make it so easy to get started and share your work.

I absolutely hate the way Go does packages. I've heard they've improved it, so maybe I need to take another look.

I dunno, I've been programming Go for 9 years and have had far fewer problems with dependencies than I ever did with pip, gem, or npm. I don't think it's perfect by any means, but none of them are.

Small Rust projects are... urgh. I spend more time downloading packages, and cleaning up the sometimes hundreds of megabytes of random crap, than I do developing. Easily finished small python projects in less time than the download-compile-cycle of Rust.

My python projects usually rely almost entirely on the same 3 or 4 already downloaded libraries, plus the large stdlib. They run 3.8. That experience is not comparable, and not in favour of Rust. Maybe it's because I don't use whatever Tox is, but I don't. So no. Not beginner friendly.

https://media.tenor.com/images/af0610f778ecdc1b7ab5a750c2d98...

Python used to be my tool of choice for all of my small programming tasks, but nowadays I find messing with Python to be an absolutely massive time-waster for me. It used to be my favorite programming language. Now I avoid it entirely because of this nonsense.

Where did you go from Python?

I just write everything in Go nowadays, since I didn't have any hard Python requirements like numpy to worry about.

Cool, thanks.

I'm quite confused as python development under OSX is extremely simple:

1. Install pyenv [1] and the specific version you plan to use.

2. Use poetry [2] for dependency management, which pins the python version to the one defined, ideally the one you've set via pyenv.

[1]: https://github.com/pyenv/pyenv

[2]: https://python-poetry.org/

Assuming this combination of tools does work, it may be that it's just too hard to discover, given the large number of competing approaches described in the article and these comments.

Python development on a Mac works pretty well if you install it from the installer that you get from python.org, at least it has in my experience.

I used to play this game but realized pyenv / nvm / etc. are all just anti-patterns nowadays. Use docker ( or some other containerized solution ) to run code in your local directory in a specific environment.

I think the author is doing the best they can considering all of these tools and resources aren't updated to this pattern yet, but I'd hate to see this become the new normal for any developer entering into the field. What a way to scare off new dev friends!

I was able to use venv and install tox and tox-pyenv locally to use tox. I also installed pyenv with brew. I had many problem with scientific packages in python projects mostly, the incompatibility between different versions of libraries, I use Conda for those projects, and venv for other projects not dependent on numpy and scikit-learn.

This is why I gave up on playing super-nerd and just moving all development to docker-compose.

I'm in Windows / Linux, my wife is on MacOS. She wanted to help with some python programming. Yep - playing super nerd is LAME. I thought getting environment setup to match mine would be pip requirements.txt but no go.

Really facinating. I think things like go / rust etc? may work better - you compile everything and compile to binary on platform, so don't need to build and carry around environments everywhere.

I moved to docker deploys / dockerfiles locally for pretty much everything. Totally overkill, but when I switched to working from home was so glad I could just get up and go without getting my overall system sorted out.

I haven't totally figured out the WSL2 python remote interpreter / debugger code flow yet however, so still using docker. I'm sure WSL2 will or is an option.

I've been tempted to just vnc or similar into a cloud instance, but my local machine is to fast by comparison for the price.

Using PyCharm with docker-compose as the interpreter works like a dream (relative to Python development). No more dependency problems and then you can also have a full setup with postgres, Celery, redis, etc. all available on the local machine.

My mac is a couple years old and I don't have the fan of death problem everyone seems to complain about. The best thing is allowing my team to know with confidence we're developing against the exact same environment.

Reply on news.ycombinator.com