Show HN: Hatch 1.0.0 – Modern, extensible Python project management | Hacker News Reader

Show HN: Hatch 1.0.0 – Modern, extensible Python project management

github.com

69 pointsofek44 comments

Meanwhile, the last few years I've consolidated on poetry and been pretty happy with it. How does hatch compare to poetry?

Ultimately, I don't really care what I use! But python's woeful package management and distribution is actually a factor in my decision criteria for when and how I use it.

If I control the system end to end? Sure, I don't mind messing with python and building something I deploy. If not, I don't want to deal with it or the headaches that come with the gazillion ways the runtime can be installed/packaged. In this situation I've been defaulting to go even though I'm slightly less productive in it.

I just want a community accepted standard solution that gives me an easy way to offer a pip installable package with its dependencies, or a way to package the runtime easily (cross platform) and move it around with my code so I can forget altogether about what's lurking underneath all that.

> How does hatch compare to poetry?

Poetry is mainly used for managing an _application_ and its dependencies whereas Hatch is more agnostic to the project type and offers plugin-based functionality for the entire workflow (versioning, tox-like environments, publishing) so you can easily build things other than wheel/sdist, test in a Docker container, etc.

Hatch also strictly adheres to standards and eagerly adopts whatever behavior new PEPs dictate while Poetry has a persistent unwillingness to adopt new standards if they are deemed suboptimal (see comments on PEP 621 [1] and PEP 665 [2])

As such, locking support is temporarily blocked https://ofek.dev/hatch/latest/meta/faq/#libraries-vs-applica...

You can continue using other tools like Poetry at the same time https://ofek.dev/hatch/latest/meta/faq/#interoperability

By the way I very much appreciate the eye for design/UX of Poetry's creator, we also share strong opinions on pipenv :)

[1]: https://discuss.python.org/t/5243/12

[2]: https://github.com/python-poetry/poetry/issues/4710#issuecom...

> Poetry is mainly used for managing an _application_ and its dependencies

I'm surprised to hear that view, since when I looked into poetry it seemed more designed for libraries than for (web) applications. I'm thinking specifically about this thread about whether it would be desirable to have a config file option for the --no-root command line flag: https://github.com/python-poetry/poetry/issues/2458

Poetry does several things by default that are bad for libraries - see https://iscinumpy.dev/post/poetry-versions/ for some details.

Oh yeah, that takes me back, that’s my issue right there. I can’t believe how I had to get so heavy handed in the comments to get people to stop debating the validity of the idea. Hadn’t worked on the project that needed it in a while so I forgot it wasn’t fixed yet, which is a bit disappointing considering.

Hey, thanks for taking the time to respond. I also wanted to say thanks for hatch, I can see this problem space is something you care a lot about! I will definitely give it a go at the next opportunity and see how I can fit it into my workflow.

I have to say I do like a lot of the design decisions that have gone into hatch. Regardless, it is great to have more compelling options being worked on, and eventually something will get it "right" enough for the important use cases that it becomes a de facto standard and then hopefully a "blessed" option that everyone can rally around.

Thanks for the kind words!

Meanwhile I've been using vanilla pip+virtualenv with possibly a Makefile and direnv (.envrc) for convenience for years and have zero issues with it.

When I occasionally encounter a version compatibility issue, I pin the offender in setup.py, done.

If reproducible builds/distributions are important, then I add a Dockerfile.

I will say that I primarily develop applications, not libraries meant for wider consumption. If I do create a library, it's for my own narrow use case. It seems like some of the value in these modern frameworks is reducing complexity over multiple supported versions and platforms.

I'm not saying my workflow is in the mainstream necessarily, but the standard community tooling has been fine for me.

Similarly I consolidated my Makefile process into a CI. Having a workflow that is: 1. Set up venv of this version or greater 2. Install packs from requirements.txt

I think the repeatability and simplicity combo of this is my favorite approach. I'm all for standard tooling wherever possible.

Came here to say the same thing. Set up a Makefile to build my project using pip&virtualenv, and make .deb packages using FPM. It's totally old-skool but it Just Works(TM).

Use pip-compile[1] to generate a lock file and pip to install it. Update said lock file periodically to pull in package updates.

Simple and effective in my experience.

How is this falling short? Why such a push for different tooling? Why the continued complaints about Python packaging being substandard? Honest questions because I've been using this for web and cli applications for years without issue.

I also use pipx[2] to install Python based tools onto my system so they have their own virtualenv.

1: https://github.com/jazzband/pip-tools

2: https://github.com/pypa/pipx

pip-compile does not use the new Pip dependency resolver, and as a result I have gotten in situations where Pip could resolve dependencies, but pip-compile would need manual pinning of many dependencies to do the same. (on one project at work it became untenable and people started updating the lockfile manually)

That's helpful to know. Looks like an update to use the new resolver is close:

https://github.com/jazzband/pip-tools/pull/1539

pip-compile only generates it for your current platform and python version. Poetry generates a lock file for all platforms and python versions.

This is true.

However if the transitive-closure in the dependencies contains an sdist package anywhere in the tree, written in a way that wasn't designed to be imported on all platforms (this is rare, but does happen), then it isn't possible to resolve cross platform.

The prevalence of wheels does alleviate this to some extent. I think it is one of the early design decisions that pip can only install/resolve for the current platform and this will be very difficult to unwind. There are movements slowly in a direction that may one day make it possible to statically resolve python packages and remove dynamic dependency metadata or builds happening during dependency resolution.

https://peps.python.org/pep-0643/ will help the situation eventually

Quick update: Hatch will likely be adopted by PyPA in the next week

- https://mail.python.org/archives/list/pypa-committers@python...

Curious to check it out. I’m also on poetry currently, which is fine 90% of the time. The 10% of the time I wish for a better system: crazy-long waits for dependency resolution, strange errors when I have to remove the lock file and reinstall everything. A look at the huge number of issues on their repo makes me question the wisdom of relying on poetry.

> A look at the huge number of issues on their repo makes me question the wisdom of relying on poetry.

That can also be a good sign. Lots of people using it, lots of people complaining about it. Assuming these issues are tended to (over time), poetry can only come out better, mainly more stable.

From what I understand, Hatch fixes a core failure of Poetry & PDM: if you have a tool that is not a library (say Black), it should not be able to collide with another tool you are using (say TensorFlow). For a while, black and TensorFlow could not be installed in the same environment due to bad upper caps (In Python, never upper cap without proof it fails or will fail, see https://iscinumpy.dev/post/bound-version-constraints/) - and that's fine, TensorFlow and Black don't use each other. But Poetry/PDM will fail (and this is why Poetry's solve times can be insane). Hypermodern-python still tries to jam everything into a massive Poetry dev environment, which can create multi-hour solves if you add things like TensorFlow.

Today we've been "fixing" this by using nox or tox, and avoiding adding everything into a big "dev" environment. But the draw of Poetry/PDM was supposed to be "A single tool". Very excited to see where this goes! I've already enjoyed hatchling, which can be used on it's own with other systems. :) (Except Poetry, which doesn't play well with standards or anyone else).

All of the new shiny tools are incapable of working with mutlirepos ... yes i know i can do [1]. But true editing on multi repos at the same time i can only do with zc.buildout [2] Still not perfect but it does the job.

[1]

  $ cat req.txt
  requests
  git+ssh://git@gitlab.com/foo/bar/amber.git@0.4.0
  git+ssh://git@gitlab.com/foo/bar/rebam.git@0.4.0

[2]

  $ cat buildout.cfg
  [buildout]
  extends = versions.cfg
  extensions = mr.developer
  auto-checkout = \*
  develop = .
  show-picked-versions = true
  update-versions-file = versions.cfg
  sources-dir = git-sources

  parts = py

  eggs = 
      amber
      rebam
      hammer

  [sources]
  amber = git git@gitlab.com/foo/bar/amber.git@0.4.0
  rebma = git git@gitlab.com/foo/bar/rebma.git@0.4.0

  [py]
  recipe = zc.recipe.egg
  eggs =
      ${buildout:eggs}
  interpreter = py-backend
  dependent-scripts = true

The included project templating functionality looks interesting. Being able to standardize scaffolding and project management under a single tool is definitely a value-add.

I'm interested to see how Hatch develops, especially in the currently quite fragmented mess that is {pipenv, poetry, conda, pyflow, ...}.

I see that tox switched at the beginning of March, which is a realtively high-profile project. I'm curious.

I'm using pipenv and poetry, with a slight preference towards poetry.

Strangely enough the more I think about this, I kind of want hatch, PDM and poetry to have a baby in some unholy union and give me everything:

I want the automatic virtualenv management from poetry with the lockfile for dependency management and resolution

BUT

with the option to use a vendor dir like PDM without virtualenvs

BUT

with hatch's awesome environment management and matrix functionality for tests/docs etc.

Then, I'd love to work out how I can bundle a python runtime against my package so I can ship a single distributable archive that can be placed anywhere to run predictably. Does anyone have any suggestions for something that solves this last part?

Does this generate a cross-platform lockfile? Besides usability, that is probably the most important feature from Poetry for me but I didn't see any details in this.

The most annoying anti-feature in poetry is inability to override dependencies. In cargo where there is less need (more consistent use of semver, allow multiple copies of deps) they still allow this but not poetry.

It does not generate lockfiles: https://ofek.dev/hatch/latest/meta/faq/#libraries-vs-applica...

Shameless plug: I use my own template, which organises things as runnable projects.

https://github.com/tpapastylianou/self-contained-runnable-py...

It serves my purposes very well (which is creating projects that represent standalone experiments).

Sharing in case someone else here finds it useful.

More recently I've modified this a bit to also generate nice html reports straight from the __main__.py file, independently of the underlying python code, and use this as lab books (where each lab book contains a single analysis and its report). I'll upload this template separately when I find the time.

What are folks using for scaffolding these days? I'm familiar with cookiecutter[0] and recently discovered copier[1], and while copier allows for updating a project when the templates update, it makes it hard to have a suite of mostly-overlapping templates in one repository (e.g. mytemplate-library and mytemplate-app with common files symlinked from top level).

Any chance of hatch covering all of this?

[0] https://github.com/cookiecutter/cookiecutter

[1] https://github.com/copier-org/copier

Pip and Docker and i never had any issues with python packaging.

>Docker

It's like paying someone else to do the thing

Works fine if you’re distributing an app, but you can’t distribute a library this way. That’s where something like this is invaluable (although I haven’t used it, I went down this path and landed on Poetry)

Ah that I totally understand.

Distribute a library by uploading to pypi, and/or cloning from version control. Setup.py still works.

How do you keep your builds reproducible? I understand that once it's built, a docker image makes that specific build immutable and you can distribute it without worry. However, if you were to run the build stage again i.e. rebuild the docker image and redownload the python packages, without changing any package versions, the resulting docker image *may* be quite different from the previous docker image, even though you haven't changed a single line in the entire repository. This is because the dependencies of dependencies may change. So while this works for you, it's not a general solution for the community. Tools like poetry have solved this problem to a large extent.

Not GP, but I solve this problem by pip freezing a requirements.txt with specific package versions and using that when I do a pip install. I don’t use docker, but probably should.

This is pretty reproducible across my own, co-workers, and external collaborators to get the exact version.

Well the way I mostly solve it is by using the cache from docker. As well as versioning important packages. As well as multiple requirements files.

However, i feel like if with every small change to your libraries your code breaks there would be something wrong haha.

Also I'm not saying it's a perfect solutions. Only that I had a lot of python apps this was never any issue (an d I had a lot of issues)

If you are building a library though, i totally get the point of using

I dropped python (for personal use) back in python 2 vs 3 debacle. To this day, python still doesn't have acceptable package management, which is a shame because python 3.10 is pretty great. At work I throw the python trash into a Dockerfile and/or with poetry. Hopefully python will evolve to use hashes soon...

Pipenv and Poetry are both pretty solid. I'm assuming when you say Python doesn't have acceptable package management, you mean the tools distributed with the language aren't acceptable?

One thing I was hoping to see is tight pyenv integration. That's the one thing I miss about Pipenv since I switched to poetry; always some finagling to do to get the right python version in use. Is there a reason these newer tools avoid touching that? Is it to retain parity on Windows, or something else?

I'm curious to hear why you switched from Pipenv to Poetry. I've always used Pipenv just because it was the first solution I came across and it fits my needs well. It seems Poetry has been gaining momentum although both still seem quite popular.

I switched when Pipenv had a really long gap in releases and one of the creators was embroiled in controversy. Probably read about poetry here, the toml config was clearly the future direction of the ecosystem, so I tried it out. Very similar ergonomics, great docs, and the solver was both faster and more reliable so I never looked back.

I also got my then-company to make the switch across a dozen projects or so and multiple teams, but it didn't take much convincing; "Pipenv seems to be unmaintained" did a lot of the convincing for me.

how does it compare to pipenv?

Where does the difference between the PyPI release, which is 1.0.0, and the GitHub release, which is supposedly 0.23.0, come from?

Hatch is the CLI, Hatchling is the minimalist build backend

Reply on news.ycombinator.com