Hacker News Reader

Comment by legostormtroopr | Hacker News Reader

Pipenv is a really interesting development for Python, and I'm glad that someone was working to improve dependency locking for Python.

However, Kenneth abused his position with PyPA (and quickly bumped a what is a beta product to version 18) to imply Pipenv was more stable, more supported and more official than it really was.

And worse still, for anyone saying "but ts open source, you get what you pay for", Kenneth as former Python Overlord at Heroku, encouraged Heroku to place Pipenv above Pip as the default Python package manager in the Python buildpack. This decision impacted paying customers and the Python buildpack used a broken version of Pipenv for a long time. So long, most people I know just went back to Pip.

Then, lastly, when people complained he had a tizzy at reddit and twitter and got PyPA to help backtrack and say "no we didn't support it, nope, its just a thing that happened", all while the main Pipenv Github repository was held under the PyPA GitHub Org.

View on HN

Sometimes, improvements don't happen in a straight line.

There's been a lot of work on Pipenv over the the last 6 months, predominantly by Dan Ryan and Tzu-Ping Chung, and it's getting stronger and stronger with each release.

If you've gone back to using pip I'd encourage you to give Pipenv another try. Introducing a lockfile is a big step forward for Python dependency management, and the team working on Pipenv are committed and doing a great job.

> Sometimes, improvements don't happen in a straight line.

I don't deny that, what I am (and the article is) saying is that we were sold on Pipenv being "the officially recommended Python packaging tool from Python.org".

And PyPA didn't refute it, and Heroku didn't refute it, so the community bought it.

Yes, introducing a Lockfile is huge, and it was massively needed, and thats why when we were told "heres the official way to do it", we got excited. Then we got daily breaking updates, rude issue close messages, and a giant backtrack of "its free and still under development, why do you expect so much from us?"

Another reason Heroku didn't refute Pipenv is that Pipenv's creator leads Python related things at Heroku.

Not anymore, I don’t believe.

Pipenv tries to upgrade all the versions of everything in your lockfile whenever you add a new package (not just dependencies of the package), and there's no way to disable this behavior. Right now. That's the tip of the iceberg.

You might want to check out https://github.com/pypa/pipenv/pull/3304, which Dan is actively working on to solve that.

Poor control over this sort of behavior makes Cocoapods very frustrating to deal with.

> Sometimes, improvements don't happen in a straight line.

I don't think that the parent disagreed with that. The point, as I understood it, was that this beta stage improvement was marketed as being ready. IOW, if pipenv was not Kenneth's project, it likely would have evolved in, to use your phrase, a straighter line.

Pipenv is designed solely for packaging applications, per the maintainers' admission. It's not suited for libraries and is not designed to be. If you want to replace pip, you should have a look at Poetry.

> It's not suited for libraries and is not designed to be.

That's kind of the problem. Why on earth are libraries and apps getting a different treatment? The JS ecosystem manages to have one tool for apps and libraries. One too for installing and publishing. All of it with lockfile support, workspace/"virtualenv" support, etc. And somehow, it's not confusing.

Adding one more tool to the stack is a really funky step forward. Yes, it brings lockfiles. Cool, although we already kind of had those (--freeze). Packaging in Python is a mess and I'm more and more in the camp that as long as Pipenv keeps flouting itself as "the better solution", all the while not covering all the basic use cases, we've gone backwards and are in even more of a mess.

And it isn't only Javascript that manages to have a single tool for this. Clojure has a single tool. Java has a single tool. Rust has a single tool.

This is a solved problem across a variety of popular and mainstream programming languages. I don't mean to suggest that the problem isn't complicated, but this isn't a problem that doesn't have a wealth of previously written solutions to look at for inspiration.

What's the "one tool" for Java? Is it Maven? Ivy? sbt? Gradle? You may say some of those are "build tools" rather than dependency tools, but I don't see how it's different than what we are discussing in Python.

My Java app declares its dependencies in a build.sbt file using Scala syntax and has them cached in an Ivy directory. Yours declares them in a pom file using XML syntax and has them cached in a Maven directory. Neither of us even tries to do lockfiles and instead just has the CI server build an "uber jar" with all the dependencies bundled in.

We don't need lockfiles in Java land because we generally use version ranges very carefully and rely on package developers following semver - and we certainly don't use "just whatever the latest release is, dude" like shown in the example Pipfile: https://pipenv.readthedocs.io/en/latest/basics/#example-pipf... (the asterisks)

I get why Python needs lockfiles, but goddamn, that need is a symptom of the mess of managing Python dependencies.

There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8

In a Maven pom.xml I can easily exclude or override the conflicting dependency manually if needed, but I can't in a Pipfile.

rely on package developers following semver

Well, that seems naive. Almost 1/4th of Maven Central libraries broke binary compatibility in patch updates: https://avandeursen.com/2014/10/09/semantic-versioning-in-ma...

Netflix, at least, doesn't agree that Java doesn't need lockfiles: https://github.com/nebula-plugins/gradle-dependency-lock-plu...

> We don't need lockfiles in Java land because we generally use version ranges very carefully

Yes, which is part of the reason why lots of Java projects almost never update their dependencies, because no one remembers why that version was chosen. Splitting what you want from what you have is important to communicate this.

> There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8

I'm not sure what you expect it would be able to do here...

Enforce Flask-wtf 0.8 and see if it works.

With my very small sample size, maven and gradle's lack of version locking just means it gets tacked on in terrible ways when needed.

People use sbt for Java? Thats’s...surprising.

Most shops I know use one of maven or gradle. I can’t think of any other serious contenders in Java (if you still use ant: please consider alternatives).

Right, exactly! It bugs me that we have a wealth of examples and could have just said: "Look. Here's yarn. Do that, but for Python. Don't try to be too creative."

Poetry looks like it did just that though and I'm warming up to it at a very high speed.

Clojure has two tools.. designed to be used as the single tool though.

Boot, and Lein. I'm partial to boot lately.

libraries and apps _should_ get different treatment. A library often has to coexist with other unknown code (open world assumption), an application you're controlling is effectively existing in one specific universe: the one you define (closed world assumption).

You wouldn't write your application to support 5 versions of Django, but you _probably_ would do so for a library.

That said, I do basically agree about `pip` existing already. We could have built a tool to manipulate `requirements.txt` files instead of introducing another format and a toolchain that is _much_ slower and brittler. Though ultimately at this point Python packaging woes feel like they are at a much lower level (the fact that libraries end up being "installed" mean that preparing all your dependencies to go out to multiple servers is a mess).

> We could have built a tool to manipulate `requirements.txt` files

There's pip-compile from https://pypi.org/project/pip-tools/ that does exactly that. Pipenv uses its resolving mechanism if I'm not mistaken. It produces standard req.txt file with versions pinned and supports separate dev requirements. It had some bug with upgrades last time I checked though, not sure whether it's resolved, currently considering using it for projects at work.

Have used pip-tools for a long time... its very uncomplicated and just makes things pretty darn simple.

Since its all just requirements files with pip-tools, its been fairly commitment free, in that I haven't had to substantially rework any major bits of workflow to use it - nor would I to remove it. Not sure I could say the same thing about pipenv and/or poetry.

An internal application can exist in one specific universe. Most of the time.

The "library" workflow works for applications too. Put your direct dependencies in setup.py. Build wheels of everything and upload them to an internal PyPI server. Pin everything in requirements.txt.

Pipenv keeps flouting itself

That would be weird. Perhaps you mean 'flaunting'?

Yes, typo :)

I already managed my python deps with a lockfile - requirements.txt and requirements-frozen.txt, which about three lines of shell script took care of for me. From the article, it doesn't sound like pipenv buys me much on top of that.

It's a "standard" way to do those scripts. It gives you the NodeJS + NPM way's "npm i", by creating a virtualenv and pip installing into it. It helps with specifying prod and dev dependencies too - if I remember correctly.

I just switched away today. It still sucks

We’re giving it an honest try, but we still get burned regularly. I’m pretty disappointed.

> Introducing a lockfile is a big step forward for Python dependency management

Huh? I'm not really familiar with the state of dependency management for Python/dynamic languages but... there's much more out there beyond just lockfiles. I'm a bit appalled Python is so far behind.

Much more for what? In this context, "lockfile" means a file listing all your project dependencies with exact versions pinned. (Not semaphore-files or anything like that. It's not even a python-specific term, e.g. npm uses package-lock.json for the same purpose.) The need for something to store all dependencies with exact versions pinned exists in any language and infrastructure, are there any better solutions than store them in a file? Files are nice, they are VCS-friendly and everything.

> Much more for what?

Much more in the field of build tooling/package management. Pinning versions is fine, but dependency resolution is another legitimate choice.

I'm very glad we have the wheel and ensurepip now.

Yet, I think PyPA has not been taking the best decisions regarding Python packaging.

Your Kenneth story is not the only "weird event" in their history.

E.G:

Did you know that we don't need "pyproject.toml" at all ? That there is already a production ready plain text standard to replace setup.py ?

Did you know that this standard has been perfectly working for TWO YEARS with regular setuptools, is incredibly simple to use and completly compatible with the standard "setup.py stuff" workflow (and hence the whole legacy tool stack) ?

Yep. And nobody talks about it.

Let me (re)introduce...

Setup.cfg !

Oh, I know... Most people believes it's a useless file.

After all, the Python documentation seldom states to use it and for only one tiny option:

https://docs.python.org/3.7/distutils/configfile.html

But no. Setup.cfg is awesome !!

Put one line in setup.py:

    import setuptools; setuptools.setup()

And you can now put all the rest - yes, everything - in setup.cfg. It's perfectly documented:

https://setuptools.readthedocs.io/en/latest/setuptools.html#...

Not only it has been working since 2016, but it has fantastic goodies:

    version = attr: src.__version__

will include the version from __init__.py

     "options.data_files"

replaces the MANIFEST

     "license =  file: LICENCE.txt"

loads the licence from any text file

Try it, it just works. And you can "python setup.py sdist upload" as usual with it. You don't need any new tool.

Now why did the PyPA decide to forget about this and create a new format ? The short explanation in PEP 518 is a bad joke. And why does nobody talks about it ?

When I asked the PyPA, they told me they were too invested in the new project to stop now. I don't like this answer at all: we suffered enough with python packaging during the "distutils, eggs, etc" fiasco.

setup.cfg works. It works now. It's nice. It's compatible. It does what we need.

Use it. Talk about it. Write about it.

Make sure a lot of people knows so that tool makers and PyPA finally acknowledge that there is not need for the XKCD comics about standard to be true again.

I love setup.cfg, I've used it for years and indeed pyproject.toml is useless given that setup.cfg has existed for frickin ever at this point (and is supported by a multitude of tools). TOML is nice but its support isn't even in the stdlib which makes it very awkward to use for a core file like that.

Here are some examples of my libs/apps using it in the real world, if someone needs references for how to use setup.cfg with an empty or near-empty setup.py:

https://github.com/jleclanche/python-bna/blob/master/setup.c...

https://github.com/dj-stripe/dj-stripe/blob/master/setup.cfg

https://github.com/jazzband/django-oauth-toolkit/blob/master...

https://github.com/jazzband/django-push-notifications/blob/m...

Edit: I see you mention attr: src.__version__. I personally prefer doing it the other way around, with the version defined in setup.cfg and a pkg_resources snippet in __init__.py (https://github.com/HearthSim/python-hearthstone/blob/master/...).

To be honest I wish __version__ were automatically defined like that (but more reliably). Do you know if this was discussed in a PEP?

I don't understand the pyproject.toml hate. pyproject.toml exists specifically so you can specify your build system. Without it, you are basically forced to use setuptools/distutils as is currently the case. Hence, pyproject.toml and setup.cfg aren't at all in conflict.

- there is nothing you can do with pyproject.toml that you can't with setup.cfg. E.G: you are not forced to use setuptools to use setup.cfg. Any tool supporting pyproject.toml could support setup.cfg as easily, since it's a documented plain text format. It's a political decision.

- there are things you can't do with pyproject.toml you can with setup.cfg. E.G: you can't use pyproject.toml with legacy tools, or with just a fresh python setup. This may change in the future, but would requires a lot of effort because changing setuptools is a very tedious process.

- resources (time, people, money, documentation, public attention, communication, etc) invested in creating and supporting pyproject.toml could be invested in improving setup.cfg and its ecosystem support. E.G: Why does poetry support pyproject.toml and not setup.cfg ? No technical reason. Why does nobody knows this easy way to package python libs ? No technical reason reason either.

So not only the new format brings nothing on the table, but it is also a setback, AND add clutter to a situation that was just begining to be solved. It's not just poor engineering, it's poor manners really.

I've been coding in Python for 15 years. I've lived this: https://stackoverflow.com/a/14753678

Stop the pain.

The format of setup.cfg is whatever configparser accepts, which is different in different versions of Python.

First, it managed to work fine for setuptools for 2 years accross all recent python versions. That's because the differences are minor and edge cases. Try to use pyproject.toml in most CI toolchain just for fun...

Second, the format of setup.cfg is defined in a documentation already, so there is a reference outside of configparser. Yes, the low level format is not explicitly defined (although it is implicitly): so let's do define it instead of creating a new one.

Third, it's still a much easier and saner task to rafine the definition of setup.cfg than to create a new standard. I don't even understand how this is controversial, espacially among people in the computing world, where we had those kind of problems for decades and we know the pros and cons, and consequence of this.

The "I add my little format because it's pure and better and current things suck" falacy is such a stereotype we should all be able to recognize it from miles away from nowaday.

Using pyproject.toml just requires a recent version of pip. Which CI tools can't handle that?

I wouldn't call comments an edge case. The distutils documentation has a definition for comments, but I think it actually just uses configparser. setuptools just uses configparser. The pbr documentation has a slightly different definition, but I wouldn't be surprised if it just uses configparser too.

They also have different definitions of non-string types.

Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?

> Using pyproject.toml just requires a recent version of pip. Which CI tools can't handle that?

The most used one in the world: https://github.com/travis-ci/dpl/issues/822

Also the last time I used tox, anything complex didn't work either.

> Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?

Well the current PEP decided to turn a packaging situation that was stable into one that was not, again, after 15 years of mess with many versions of things. So you tell me.

Check the usage stats I posted in an other comment to see the problem.

Besides, yes, we do make compromise on best practices to allow peaceful transition all the time in Python. `async/await` allowing to be a variable silently. Non utf8 defaut encoding argument in open() in windows. Then... we fix it later.

Because I think you conveniently skip a lot of things I wrote in my comments. I clearly state that we would and should consider setup.cfg as a version 1 of the format. Then we would increment on that. I gave a detailed procedure on one way to do that, and there are others.

The point is, all your concerned can be addressed with a progressive transition, starting from setup.cfg. Actually we could even end up with a toml format in setup.cfg, __on the long run__, that matches exactly the current one.

While you addressed non of ours concerns. Just reject them. No will to even recognize there is a problem. It's insulting, really.

We did that during the 2/3 transition. Didn't work so well, did it ?

> we should all be able to recognize it from miles away from nowaday.

Oh but we do. Then we rationalize it away, because "this time...". Like we do for Big Rewrites.

It might have something to do with the fact that programming is mostly a craft you learn by doing it, so we overvalue "doing it again" because that's how we usually get better.

They're not in conflict. The problem is pyproject.toml is superfluous. See my comment here: https://news.ycombinator.com/item?id=18614058

I like the way you do it too, and I don't think it needs any improvements besides being shared more.

  > Use it. Talk about it. Write about it.

I have a project that converts basic setup.py files to setup.cfg files [1].

Still happily using plain setuptools for library development and pip-tools for application development.

[1]: https://github.com/gvalkov/setuptools-py2cfg

Very cool. I'll share that.

setup.cfg lets you configure setuptools with declarative syntax. pyproject.toml lets you replace setuptools with something else. The PEP explains why they didn't just reuse setup.cfg.[1]

[1] https://www.python.org/dev/peps/pep-0518/#sticking-with-setu...

The problem is that this is yet another boilerplate file.

The reasoning is good, but we were just arriving to the point that every Python tool out there is either compatible with tox.ini, setup.cfg, or both (much like the JS ecosystem has tools reading from package.json).

Now we have both Pipfile and pyproject.toml on top of it!

For a language that prides itself on its stability and backwards compatibility (especially when compared to the JS ecosystem), we churn through boilerplate files harder than Google churns through instant messaging apps.

> The point of pyproject.toml is to allow other libraries to replace setuptools.

This can be done with setup.cfg. Setuptools is only a backend supporting it. You can create other ones. Poetry and pipenv could support it in a week in their authors decided so.

> The reasons for not using setup.cfg are explained in the PEP.[1]

Those are not reasons, those are excuses. Let me quote it:

>> There are two issues with setup.cfg used by setuptools as a general format. One is that they are .ini files which have issues as mentioned in the configparser discussion above.

Not only setup.cfg does the job with the current limitations of the ini format (while pyproject.toml still doesn't with its fancy one), but python projects are not so complex they require such a rich content.

Besides, nothing prevent PyPA to says that setup.cfg format now has a version header, with the current setup.cfg being headerless version 1, then make the header mandatory for version 2 and increments it to move toward TOML if we ever reach a limitation. That's how formats grow everywhere else in the world.

>> The other is that the schema for that file has never been rigorously defined and thus it's unknown which format would be safe to use going forward without potentially confusing setuptools installations.

That's incredibly dishonest, since I gave a link to a complete documentation of the format in my previous post. Besides, it's better to actually refine the existing standard if you ever find it lacking than recreating one from scratch. While there are good reasons to do so, the later is rarely a rational engineering decision, and most often driven by ego.

>> While keeping with traditional thanks to setup.py, it does not necessarily match what the file may contain in the future

So ? How is that a problem ? A standard is not meant to be set in stone. It evolves. But it can't do so if everytime one has an itch, one reinvents the wheel.

The "existing standard" is whatever configparser accepts. configparser is part of the standard library. Different versions accept different things. The setuptools documentation covers higher-level things like key names. That isn't what they're trying to standardize.

The last sentence you quoted explains why they picked "pyproject" instead of "setup". It isn't why they picked TOML.

See my answer to your other comment.

Also "higher-level things like key names" is half of the standard.

Besides, picking a new (even if better) serialization format is not good reason to create a whole new standard with names, convention, tooling, etc., as explained earlier.

There are sane ways to make the existing system evolves and improves incrementally, using the legacy standards that benefits from the existing situation, and allow the improvements from the new one. All that without the madness of messing with the entire community once again after 15 years of instable package management.

Yeah it's less sexy that having to create your new baby, yes it's less fun than using that new shinny format (and I say that while I __love__ TOML), and less it's less satisfying than having your name as the creator of a whole new jesus-format-saver. But that's the mature and professional things to do.

Nothing from setuptools is being standardized. Tools can use any keys they want in any format they want in any file they want.

The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.

The only new tooling for TOML is a small library. A new INI dialect would need one too.

> Nothing from setuptools is being standardized. Tools can use any keys they want in any format they want in any file they want.

No, if you use any key, it won't work with setuptools.setup(), and just like a python code that doesn't run with cPython will never be popular, it will not be used.

Also, if you look at how poetry use pyproject.tml, they just create a custom section. So basically, they don't use your standard.

> The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.

That's kinda my point for comments and comments. Standardize the status quo, then increment from that. Not sexy. Not pure. Welcome to the real life.

Didn't you learn anything from the distutils/distribute/setuptool mess ? From the Python 2 / Python 3 breakage ?

And could you address any of my concerns instead of just attacking ? Because I'm trying to address yours with solutions. You just write short busts of "no, it's bad, we are good". That's not really giving me trust in your decisions, and it __lowers__ my confidence in pyproject.toml because the people defending it basically are not behaving like engineers trying to solve a problem, but as salesmen trying to only defend their product.

> The only new tooling for TOML is a small library. A new INI dialect would need one too.

But but we can start from a standard that works now, is used already, and is compatible with existing stacks. Instead of arriving with the theorical untested, incompatible best thing that add a layer on top of the mess.

Because I hate the "this ship has sailed" argument about pyproject.toml, here are some github usage stats:

- setup.py: 1,259,007 results (https://github.com/search?q=filename%3Asetup.py)

- setup.cfg: 165,716 results (https://github.com/search?q=filename%3Asetup.cfg)

- pyproject.toml: 2,137 results (https://github.com/search?q=filename%3Apyproject.toml)

Also, remember that setup.cfg is completly compatible with setup.py, the migration is painless. All the legacy tools work. Not the case with pyproject.toml.

setup.cfg isn't always used as a setup.py replacement. I use it for configuring flake8, wheel, pytest and coverage: https://github.com/Kwpolska/python-project-template/blob/mas...

This is interesting. Somehow I missed this, using setup.cfg make much more sense than putting everything into python code. I wonder if pip-tools will allow compiling install_requires and test_requires in setup.cfg into nice requirements.txt and dev-requirements.txt without too much magic...

Your latest blog post say good things about poetry and bad things about pyproject.toml.

However the default in poetry seems to be pyproject.toml... I'm confused.

Because I'm not dogmatic, I can regognize good tools, even if I disagree with the underlying political decision.

Computing is not black and white, and perfect purity is only nice in "fizz buzz".

Now to be extra fun, poetry uses a custom section ([tool.poetry]) in pyproject.toml, not really the standard itself. What does that say about this format ?

oh, nevermind... I'll stick to manual venv/pip:

  ptest poetry add requests

[UnicodeDecodeError] 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Author of Poetry here!

I've never seen that error before.

Which version of Python do you use?

And feel free to create an issue on the issue tracker: https://github.com/sdispater/poetry/issues

pyproject debates aside, I love your work. I think poetry is a beautiful piece of software, the source code is very easy to read and the 2/3 compat is well done (although I would not use assert to check things).

Looks like a locale problem. `locale-gen en_US.UTF-8` should fix it.

Power tends to corrupt.

Reply on news.ycombinator.com