Changes to Homebrew adversely affect Python development
justinmayer.com
justinmayer.com
Other replies here appear to assume basic compilation issues, but the issues I've seen on various GitHub projects are mostly related to these more insidious C toolchain challenges. They're not M1-specific, but they are manifesting on M1 machines.
https://github.com/pyenv/pyenv/issues/1643#issuecomment-7293...
For installing Python on the M1 Apple Silicon. This will install x86_64 versions, but those will work just fine.
If you're having trouble installing something with brew on M1, run arch x86_64 brew instead of brew to do a Rosetta installation.
My current Mac is going to be my last. Linux on Thinkpad is next
https://github.com/pyenv/pyenv
What's nice is you can find analog programs for a lot of different languages and platforms including
Node: https://github.com/ekalinin/nodeenv
Ruby: https://github.com/rbenv/rbenv
Terraform: https://github.com/tfutils/tfenvI agree that we could definitely improve our documentation on what people can and should expect from a Homebrewed version of Python; I'll look into opening a PR that improves these docs tonight.
The decision to remove build options was informed (in part) by user experience: for every one user who was using them correctly to configure a package slightly differently for their uses, there were fifteen users following outdated SO answers and reporting their own breakages to us. Like every other package manager that distributes binary builds, we made a decision to improve the 99% use case.
I wish more people try out MacPorts.
I honestly put its popularity down to it being written in Ruby, which made it the default choice for the large population of Ruby programmers on the Mac.
When I did the same for some homebrew formula I got a response a few hours later.
For me homebrew has a much more responsive community and due to their use of Github also a lower bar for contributions. Both might add to homebrew popularity.
MacPorts uses GitHub too now. We maintain much more ports than Homebrew, and few of our committers have time to regularly go through the PR queue, so our community seems inactive. But actually, we just need more hands on PR management.
One important distinction is that most active ports in MacPorts have specific maintainers, and we usually put off merging PRs until the port maintainer approves. With more than 20,000 ports, it hard for the committers to test every change thoroughly, so we rely on our port maintainers to resolve issues and review complex PRs whenever possible.
In Homebrew, any maintainer could review and merge a PR instantly, combined with fewer ports to maintain, average time to merge would be much shorter.
The pain is a bit lost to memory, but even as a long-time MacPorts user I understand why people wanted to switch. If everything has to be built from source you also have to build all your build-time dependencies from source, even weird ones for building docs or whatever. The behavior of MacPorts then (and still unchanged today) is to upgrade to the latest versions of all dependencies when installing or upgrading a requested package. So even if you had zlib sitting around if an update was pushed to zlib you're going to be building a new one. A lot of packages were forced to autoreconf because the Mac was still gaining popularity and a lot of source distributions didn't have ./configure scripts that supported this or that Mac-specific or MacPorts-specific functionality. So now you have to download and install whatever autotools you need to do autoreconf. And of course MacPorts ships a separate package for autoconf 1.3, 1.4, 1.5, and individual packages can depend on specific major versions of autoconf, so even if you had an up-to-date autoconf 1.4 you're still getting an upgrade to autoconf 1.3 if your package happened to ask for it.
It was ugly. Homebrew's initial insistence on requiring the re-use of system packages cut down on flexibility (you can't get a new zlib, you have to wait for Apple to ship one) but greased a lot of wheels people did care about. But as time has gone by both systems have improved on their faults and are in a better place than before. This particular bug sucks for the people affected but I think that the Homebrew people will eventually figure this one out, even if it means making their system more MacPorts-like. Like they have been for the past ten years. ;)
I used to use ports back in the days and appreciated it coming from FreeBSD previously but at some point most things I wanted were simply only available in brew, at that time brew and ports didn’t play well together on the same system so sometime around SL/Lion I switched to brew completely.
For macOS on Intel, MacPorts does not play well with Homebrew because it took /usr/local, so ports built from source might use Homebrew files and break.
But "brew" & "homebrew" is just a more charming and hipster name for a newbie to write in his blog for other newbies to follow, repeat eleventy-seven-hundred times.
I also never understood the concerns about correctness. Very few people want homebrew to be an apt/rpm replacement for servers that will provide stability for decades. Most Macs are effectively single-user systems, and if something goes wrong then nuking the whole homebrew directory and starting over is not a big deal.
> Homebrew’s Python is not for you. It exists to serve Homebrew, or more accurately, Homebrew’s other formulae. The primary purpose of Homebrew’s Python formula is to enable other Python-dependent Homebrew packages to work.
If you want to develop Python (likely Ruby/ PHP/ whatever), you are best keeping your dev environment separate from the interpreters installed for the purpose of running apps managed by Home-brew (or apt/ rpm/ etc).
Homebrew's guarantees are no different from those of apt or any other system package manager; the only difference is that we don't have a stable distribution version to lock our Python or Ruby versions against. Individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them.
Thanks again for bringing it to our attention!
Maybe. I wasn't privy to any discussions around making Python keg-only, but my experience with other formulae that are keg-only is that users will force-link keg-only packages into `/usr/local` anyways. What's more, there are probably enough people out there who do just want a barebones Python 3 installation and don't really care if it gets upgraded underneath them, and we'd like to provide that by default.
The Python ecosystem already contains mature tooling on both Linux and macOS for managing Python installations that are isolated from Homebrew and other system-level package managers, like pyenv. In general, I'd recommend using pyenv in parallel with Homebrew; that will get you the best of both worlds.
I really like Homebrew but its dependency management is horrible. The moment I saw that dyld "image not found" error message I didn't even have to finish reading the article to know what happened.
Python does not explicitly set this to what it wants (off), so if you didn't happen to have "set enable-bracketed-paste off" in your ~/.inputrc, pasting got messed up.
This isn't just a Homebrew thing. It also hit Arch Linux, and probably others.
As part of trying to figure out who to file a bug report with, I installed Apple's Python3 and I installed the Mac version of Python3 from python.org. It worked fine in the later two, so I filed the bug with Homebrew [1] (and they figured out that it is really a Python bug and they submitted a pull request upstream to have Python explicitly turn off bracketed paste).
The python.org Python3 was easy to install, and was the same version the Homebrew installed.
This raises a question. The article talks about alternate systems to manage Python installations instead of using Homebrew or using Apple's Python. But why do I want any of them? The python.org install seemed fine. Why not just use that and not have to deal with any third party Python installation managers?
MacPorts allows you to install multiple Python versions at the same time and I personally have several venvs with Python 2.7, 3.6, 3.7, 3.8, 3.9... all running without a glitch.
When you switch from Mac/Windows to Linux, you think "wow, this OS was actually made for software development. Everything just works. I don't have to resolve a library version conflict every 3 days."
If you can live with the rough edges, I find the general experience with applications or my printer suddenly being incompatible under MacOS for "reasons" is much less of an issue under Linux.
I think another big challenge for Linux is battery life and sleep/wake on laptops. Last time I considered replacing Mac OS with Ubuntu on my laptop, I got scared away by reports that battery life would decrease by half.
I'm excited to be able to upgrade to the M1 series Mac's as it sounds like I might actually get battery life, though I have the 2019 16" Pro and can't justify the upgrade just yet, even if that thing overheats with just a Slack video.
For others similarly curious, it appears setting [`$HOMEBREW_NO_AUTO_UPDATE=1`](https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#...) is the magic sauce (or the perhaps more accurate link: https://github.com/Homebrew/brew/blob/2.7.7/Library/Homebrew... )
Also, when using pyenv, I almost always set `env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -v ${pylatest}` (macOS) or `env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v ${pylatest}` (Linux). (I also set some CFLAGS, but these are personal preference.) I've had plenty of situations where not having the shared/framework build has screwed something up (and in fact that was my original reason for switching from Homebrew or package manager-installed Python years ago), and I've never had issues where the shared build screwed something up.
[1]: https://asdf-vm.com/
In my experience they work great, you can install multiple versions, and even pip3 seems to just work with them.
I'm not a heavy Python user, but ever since I started using the official installers it just works.
I could be mistaken, but I didn't find any documentation on how to do it and the fact that other people had projects to try and do it mean that I wasnt alone on finding this a bit of a mess.
Granted, I've only lived in these tools just long enough to install what I need, so I'm sure I'm naively overlooking a lot of nuance.
Sort of analogous to the way the JVM is distributed usually as separate packages with the JRE and the JDK.
I don't write very much prod-grade Python, so I just recreate the venv with my requirements.txt instead.
Last time I used Python on macOS and Windows I manually installed the official version from python.org and used pip and virtualenv. It wasn't too bad but it seems things have changed in the last 6 years or so? Or is this still the most flexible way to go?
I would be grateful for any guidance.
Otherwise if you're trying to use some python project with particular dependencies pyenv is a good way to get a specific python version. But if the project has a lot of native dependencies (think scientific computing with lots of fortran, C++, etc. code to compile) then you're almost certainly better off with conda as a higher-level entire binary system manager.
For managing virtual environments on my preferred shell (Fish), I use (and maintain) VirtualFish [2].
For managing project dependencies, I activate environments via VirtualFish and then use Poetry [3] to update the dependencies within the environments.
[1]: https://github.com/danhper/asdf-python
Similarly I'm sure that making `brew install X` implicitly perform `brew upgrade` simplifies things a lot for them, but it's been a major source of hassle for me and it's making me consider switching away from homebrew.
I can't tell if that would influence the behavior you're experiencing, but it sounds like it makes the brew command more deterministic
I have successfully confused people who sought meaning in 'asdf' when the string is absolutely meaningless, actually being the left-hand home keys.
'asdf' could be interpreted as "totally blew off that 'naming' thing".
As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
So, giving up (and re-installing everything from scratch or rebuilding virtualenvs, using several third party tools (pipx on top of pip, pipenv and/or poetry in lieu/in place of virtualenv) is your argument to how "it's not complicated" and people merely "arbitrarily complicate their personal workflows"?
Oh, the irony!
As to why there are two tools, it's because they have separate goals: pipx provides a robust way to manage global installs of Python utilities in per-tool virtualenvs while tools like poetry or pipenv are aimed at projects you're developing and handle things like automating the process of installing updates & updating the corresponding metadata files so you can ship clean updates. All of them use virtualenvs and pip underneath — it's just a modern workflow.
The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber. Learning to use a tool every decade or so is a LOT less time consuming than doing this by hand. The average poster in this thread has already spent more time on this topic than my projects have needed for the mechanics of installing Python packages in any given year.
That we can automate our way out of a mess doesn't mean there is a mess.
Even more so if the automation involves extra inconvenience (like, from writing the automation to waiting to rebuild), which with a different design in the Python packaging wouldn't be needed.
>The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber.
I can find and install updates just fine in other package managers for other languages, without messing my setup.
I can also move my whole environment to any path I like (with just copying).
> I can find and install updates just fine in other package managers for other languages, without messing my setup.
Just like Python, in other words? (remember, this thread is not about installing package updates)
> I can also move my whole environment to any path I like (with just copying).
The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime. A compiled language which produces a static binary won't need that, of course, but that doesn't avoid the need to periodically recompile to stay current with dependencies and language changes. In all of these cases, automation makes the process fast and easy.
No, for very specific reasons. Historical cruft, too many competing tools (from pip to conda, and from venv to poetry, not to mention setuptools), conceptual garbage like wheels, pths, etc., non-movable virtual environments, and so on.
>The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime.
And yet node handles this much more elegantly (with C++/gyp modules). And minor releases should absolutely no need this.
People are trying to fix it, and their are some great packages out there that make it easier. Pyenv, Poetry, this asdf tool the fine article mentions, all seem like good solutions.
So? You can still create nice tools for it, even decades after a language has been created...
At the end of the day, they're people volunteering their time for little to no benefit. Of course they should put themselves above their users; their users aren't even paying them for said time.
If you don't like their decisions, you're free to criticize them, but to imply that (rightly) putting themselves above their users is a bad thing is an entirely different beast when they don't owe anything to their users.
At the end of the day, if you don't like their decisions, you're free to work towards becoming a maintainer and make your voice heard, or fork it and do whatever you want and think is right. But nobody owes you or any other user anything unless you explicitly are paying them for the time and effort.
The only drawback is that the Homebrew organization won’t support, update, or build bottles for, custom taps.
The only difference is that Homebrew doesn't have a stable system distribution to pin to (we're not Apple, and we have no say over macOS's development), so we make upgrade decisions for packages like Python based on their stability with our packaged ecosystem. It's a poor practice to use your system Python distribution for any sort of virtualized development environment, and Homebrew behaves identically to every other system package manager in that regard.
Please keep in mind that we're a community of volunteers answering to millions of users. Your idea of "user hostile" is our idea of "simple for the sake of preserving our sanity."
And yeah, it's the homebrew maintainer's project, but when you're in a position to help people or make their life worse, it's immoral to do the latter.