Makefile Tricks for Python Projects
ricardoanderegg.com
ricardoanderegg.com
> I work using git worktrees to have multiple branches checked out at the same time. By using this trick, I can run different commands and make sure the imported package is from that branch, instead of having to python3 -m pip install -e . around all the packages
For most usecases, you can wire up your imports and script entry points in poetry, and manage venvs there too. Or just use pyenv-virtualenv if you’re on MacOS to auto-use your per-project venv.
Again, the author's requirements here are quite esoteric in my experience, and I’d really encourage folks to avoid this particular can of worms if at all possible.
Just chuck the venv name into a .python-version file then pyenv will do that for you, on any OS.
I agree my setup is not simple (or even recommended as the default), that's why I call them "tricks".
> you can wire up your imports and script entry points in poetry, and manage venvs there too. Or just use pyenv-virtualenv if you’re on MacOS to auto-use your per-project venv.
As someone who has tried most of the current Python tools (including poetry and pyenv), I wouldn't say that "wiring up" imports and entry points is much simpler or footgun-free. It also requires using specific tools, which may not be possible depending on the project.
Also, multiple venvs is not equivalent, since you still need to manage those venvs and install the packages (+ dependencies). The `PYTHONPATH` trick can save a lot of time when the dependency tree is complex, and you just want a single venv.
For example, a workflow like:
git worktree add -b feature_one
#
# edit some files
#
BRANCH=feature_one make run-service
# in a different terminal, assuming `master` is also checked-out as a worktree
BRANCH=master make run-service
Again, I know this is a hack and it can be very project-specific. But it has worked great for me, and maybe it can be helpful to others.
It's a shame that Python venv workflow is so opaque and unfriendly to new users that it requires extra automation tools.
Its quite sad to see the rationale as making it too complicated when "upgrading to venvs" when it should kill them outright. Why using python depends at all on weird shell semantics is beyond me.
A big part of that is due to an endless stream of people who couldn't be bothered to learn what a venv was or how to use it, but... yeah, that's what work is like sometimes. The makefile pretty much ended those issues, absolutely worth the effort put into it.
* It can't run tasks in parallel
* It doesn't handle file dependencies. i.e: it's just a task runner, not a build tool.
* It's another tool you have to install, make comes pre-installed in most UNIX-like operating systems.
MacOS make is outdated (or not GNU flavor not sure) and, for example, doesn’t support parallel tasks by default or some more advanced recipes.
Asking people to use gmake on a standard Makefile is much more difficult than just asking to install Justfile runner.
> or not GNU flavor not sure
On my macOS laptop
which make
# /usr/bin/make
/usr/bin/make --version
# GNU Make 3.81
I haven't verified this. I've checked `make --help` using the built-in make on my Mac (GNU Make 3.81), it mentions the `--jobs` option to run multiple jobs in parallel.
Cannot provide precise issues with (as it was more then a year) but I encountered 3-4 problems after having complete Makefile which was kind of downer because I had to rewrite it to support “vanilla” make.
In the end I dropped parallel went with Justfile and it took fraction of time (but truth be told I had dynamic PHONY targets which aren’t super easy to setup in Make).
"There should be one-- and preferably only one --obvious way to do it."
> The reason to use $$(some-command) instead of the built-in function $(shell some-command) is that the expression will be evaluated every time it is called. [...] When using $(shell ...) the expression gets evaluated only once and reused across all the recipes.
> doit is under active development. Version 0.36.0 released on 2022-04.
For the immense convenience it brings, the risk is very low.
I'm guessing it could stay that way for 10 years and still be perfectly usable.
> doit core features are quite stable. If there is no recent development, it does NOT mean the project is not being maintained... The project has 100% unit-test code coverage.
This is something that a dev should do when evaluating a new dependency even when it claims to support the version relevant to you. Sadly, many do not.
I've looked into pydoit multiple times and I would love to try it. There are a few reasons that make me go back to make (no pun intended!).
* It comes pre-installed on most UNIX operating systems. Yes, macOS ships an older version, but it works in most cases. This makes bootstrapping projects easier. * I like how simple it's to get started with a Makefile when you just need a few tasks. Although I admit, big Makefiles can become hard to maintain.
I've recently experimented with using Makefiles as an ETL orchestrator, and it's becoming quite messy. I believe pydoit is the perfect replacement candidate. I just haven't had time to try it.
Edit:
Another thing I like about Makefiles is that you can keep the "hackiness" of shell scripts, manipulate the environment [0], etc. I guess you can do the same with other tools, but I find it easier to reason about those useful "hacks" in Makefiles.
[0]: https://ricardoanderegg.com/posts/makefile-python-project-tr...
You could say the Makefile also serves as a form of (not great) documentation.
I code in more than a few languages and frameworks, but I don't always memorize the magic build commands for each one.
In my make file, I simply create `make build` and it installs npm, pip,
I run `make clean` and node_modules is deleted or DerivedData are gone.
Otherwise, I'd dump this documentation in an unstructured ReadMe.md.