Hacker News Reader

Comment by woodruffw | Hacker News Reader

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly.

Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is:

    python -m venv env && . env/bin/activate
    python -m pip install .[dev]

(We provide a `make dev` target that does that for you as well.)

Similarly, to build distributions, all you need is `build`:

    python -m build

[1]: https://github.com/pypa/pip-audit

View on HN

Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way".

If you go to Python.org and follow through to the beginners guide [0] this is what's suggested:

> There are several methods to install additional Python packages:

> Packages can be installed via the standard Python distutils mode (python setup.py install).

> Many packages can also be installed via the setuptools extension or pip wrapper, see https://pip.pypa.io/.

That is so out of date and fundamentally confuses people coming to Python for the first time. How is pip secondary, and no mention of venv?

The PyPA need to get buy in from Python Core to put one tool front and centre first. It needs to be like Rust with Cargo, literally the first thing you learn to use and core to all beginners guides.

That's not to diminish the work of PyPA, you are all amazing I just want your work to be more obvious!

0: https://docs.python.org/3/using/mac.html#installing-addition...

The future is here, it’s just inconsistently distributed. There’s also the problem of python.org not wanting to pick winners, i.e. not promoting activestate vs conda vs pypa.

https://packaging.python.org/en/latest/tutorials/installing-...

https://packaging.python.org/en/latest/tutorials/packaging-p...

Which is just, utterly utterly ridiculous. The Python packaging story will continue to be a mess right up until this particular inconsistency is resolved.

Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or haven't been studying python packaging for decades).

I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1].

Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particularly in that respect!

[1]: https://packaging.python.org/en/latest/tutorials/packaging-p...

[2]: https://packaging.python.org/en/latest/guides/distributing-p...

No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it.

To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run <official python tool to init a package>, that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command.

That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.

Yep. There being no documentation to really empower someone to work out what to use, to me very directly says “there’s some political spat going on that I’m being exposed to here”.

that "political spat" was actually targeted harassment towards a former PyPA member the last time someone actively working on packaging.python.org tried to be remotely opinionated. Since this ended with them stepping away and no one else has volunteered, it remains unopinionated.

Sounds like the steering council or whatever that runs python nowadays needs to step up. If people are being targeted for trying to improve packaging then the steering council should make a decision and direct all commentary, etc. to them, not individuals documenting best practices.

The whole 'let the community figure it out' seems to have failed and is causing nothing but more confusion and now attacks on people. The council needs to step up and say, "this is how python packaging will work, period. End of story, end of debate. There is no more discussion on this, the decision is final. All other python packaging tools now are non-standard and not recommended for use anymore".

That was the one good thing a BDFL model for leadership could achieve, making a hard decision in the face of many strong opinions.

Edit: My sibling comment has another answer, which makes me believe I'm lacking context. So I've removed this comment, in the interest of not offering opinions outside of what I know.

A few weeks ago, I was attempting to help introduce someone to Python. I felt embarrassed at trying to paper-over how complicated and confusing the initial configuration is for a newbie. No officially endorsed best-practices to which I could point. Half a dozen tribes with their own competing solutions and guidance.

A definitive statement from Python.org as to The Way would go so far.

AFAIK in the python 2 era it was the de-facto standard (just swap "python -m venv" with "virtualenv"). All these new tools have just made it more complicated and it's not clear to me what they gain, seems to me like they've simply succeeded in convincing people it's complicated by obscuring what's actually going on.

Correct. And most people back then were just using `pip install -r requirements.txt`, including different files for dev/test/prod as needed.

I worked at one company that used buildout back then, which seemed relatively rare, but buildout was more flexible and allowed you to do things like installing system packages or running your own scripts. We used it to pull down config files that weren't checked into the repo but still wanted to be shared, helping setup the local developer environment, and a few other things.

First time I hear PyPA, and looking up their website [1]: It is not a good sign if a packaging authority has a broken logo link on their mainpage [2].

[1]: https://www.pypa.io/ [2]: https://pypi.org/static/images/logo-large.svg/

So, the project you linked has a pyproject.toml file uses flit:

    requires = ["flit_core >=3.2,<4"]
    build-backend = "flit_core.buildapi"

Python.org defaults to using hatchling in the docs[1].

PyPA has a sample repository using setuptools[2]. This project is linked from Python.org on this page.[3]

Python.org recommends using setuptools under packaging recommendations as well, with no mention of hatchling or flit even on that page.[4]

No wonder so many devs are frustrated about this. I've been writing mostly Python for work and fun the past 11 years and I'm f*cking confused what tools to use. When I start a new side project I don't know whether a tool is going to exist and still work in a few years, because there's 10 tools trying to do the same things and it's not clear what the community is actually getting behind.

[1]: https://packaging.python.org/en/latest/tutorials/packaging-p...

[2]: https://github.com/pypa/sampleproject/blob/main/pyproject.to...

[3]: https://packaging.python.org/en/latest/guides/distributing-p...

[4]: https://packaging.python.org/en/latest/guides/tool-recommend...

That's still about 4 times as much command as I'd consider normal/reasonable, compared to say "mvn install".

That repository doesn't seem to pin dependency versions. How do you integrate that in this workflow?

You could use `pip-compile` if you want full pinning. That's what we do on another project -- we use GitHub Actions with `pip-compile` to provide a fully frozen copy of the dependency tree for users who'd like that[1].

In the context of `pip-audit`, that makes a little less sense: most of our dependencies are semantically versioned, and we'd rather users receive patches and fixes to our subdependencies automatically, rather than having to wait for us to release a corresponding fix version. Similarly, we expect users to install `pip-audit` into pre-existing virtual environments, meaning that excessive pinning will produce overly conservative dependency conflict errors.

[1]: https://github.com/sigstore/sigstore-python/tree/main/instal...

Or if you don't want to install something else and are willing to just use version numbers (instead of also hashes like pip-compile in that link), "pip freeze" is built in.

The tricky thing with `pip freeze` is that it dumps your environment, not your resolved set: your environment also contains things like your `pip` and `setuptools` versions, any developing tooling you have, and potentially your global environmental state (if the environment has global access and you forget to pass `--local` to `pip freeze`).

In other words, it's generally a superset of the resolutions collected by `pip-compile`. This may or may not be what you want, or what your users expect!

By default (at least in python3 nowadays) it also excludes pip, setuptools, distribute, and wheel; you need "--all" to include them in the output.

Oh, that's news to me! I stand corrected.

(The point about other development tooling is, I believe, still accurate -- if you e.g. have `black` installed, `pip freeze` will show it.)

Which is a huge limitation of many of the other tools. I have some beef with poetry, but it did at least get one thing correct: differentiating between the code required libraries and the development tooling (pytest, black, etc). There are hacky workarounds to accomplish this with other tools, but codifying this differentiation is incredibly valuable.

With pip-tools you can use a requirements.txt for production and a requirements-dev.txt for your development environment. The latter imports the former.

where is the

    install .[dev]

format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools

I found one mention in the docs but no more.

PEP-508[0] explains the grammar for "extras":

> Optional components of a distribution may be specified using the extras field:

  identifier_end = letterOrDigit | (('-' | '_' | '.' )* letterOrDigit)
  identifier    = letterOrDigit identifier_end*
  name          = identifier
  extras_list   = identifier (wsp* ',' wsp* identifier)*
  extras        = '[' wsp* extras_list? wsp* ']'

as well as explaining their behavior, albeit briefly:

> Extras union in the dependencies they define with the dependencies of the distribution they are attached to.

The resolution on . is explained by the pip documentation[1]:

> pip looks for packages in a number of places: on PyPI (if not disabled via --no-index), in the local filesystem, and in any additional repositories specified via --find-links or --index-url. There is no ordering in the locations that are searched. Rather they are all checked, and the “best” match for the requirements (in terms of version number - see PEP 440 for details) is selected.

[0]: https://peps.python.org/pep-0508/#grammar

[1]: https://pip.pypa.io/en/stable/cli/pip_install/#finding-packa...

you may have done

   pip install jupyter[notebook]

this is the same thing, just for . (current directory) and dev variant.

Yep. The key difference is that the former is specified in PEP 508, while `.` and its variants are a convenience feature that pip (and maybe some other tools) provide.

yeah but I'm missing data on how variant interacts with various requirements defined in setup.cfg (testing dependencies for instance)

It’s a pip-ism; as far as I know, it’s not defined in any PEP. It should be in their documentation, however.

gonna dig deeper there then, thanks

Reply on news.ycombinator.com