brettcannon

There's a couple of reasons for the current situation in terms of the plethora of tools.

One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).

Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.

Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.

And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).

And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.

Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.

It was on purpose because TOML has not reached 1.0 yet and breaking people using a module in the stdlib for parsing TOML because TOML itself changed would be an absolute mess. Plus not a single packaging project said it was an issue to either depend on or vendor a TOML parser.

Once TOML hits 1.0 I plan to work on getting a TOML parser into Python's stdlib.

(disclaimer: I am one of the co-authors of PEP 518 that apparently handled things in way that was "a little absurd" .)

I'm not sure how old you are, but if you don't know the name "Borland" you might not realize how novel it is to have a compiler on every major OS. While Python's age means it has been around to see a lot of changes occur, that also means it had pre-existing practices and code to also consider whenever a shift in the industry occurred. Plus in hindsight things look obvious, but during the transition it isn't obvious what will stick and what won't.

It's pretty remarkable that things have continued to function since my 17 years with Python alone have seen things like:

- Compilers becoming available on all OSs - Linux becoming a thing - Laptops becoming common - Smartphones - Web-based email clients

As I have said a few times in comments on this thread, it's fine to feel like there's room for improvement, but trying to "stick it" to Python and its packaging ecosystem in some way for not being perfect is pointless for you and demotivating for those who are actually in a position to try to make things better for you (and even if you aren't a Python user you should care to some extent about it as Instagram runs on Python and thus it is helping to deliver you those cat photos while you're stuck at home ).

I think something people are forgetting when comparing Python's packaging system to other languages and runtimes is how old Python and its ecosystem is. For instance, Python's public release predates Linux's public release(Feb 1991 versus Aug 1991). The first release of setuptools was 2004 and it isn't like people weren't sharing Python code even before then (https://github.com/pypa/setuptools/commit/8423e1ed14ac1691c2...). That's over 5 years before npm was released (https://en.wikipedia.org/wiki/Npm_(software)). That means npm got to learn from Python and any other project that predated it. It also means they didn't have Python's momentum to contend with (I certainly never expected Python to become _this_ popular, and that makes it very hard to change as you literally have tens of millions of people to re-educate when you change something).

So while it's fine to say you wished Python's packaging approach was as good as another languages, please do so know that it is not fair to say that if one community got something to work for their needs it means Python should have been able to as well; it's not an apples-to-apples comparison considering the trees were quite possibly planted at different times and are different varietals.

The specific restriction from my understanding is that a US non-profit that accepts money for a specific thing must spend every penny of that money only on that thing. So if the PSF was given $1,000,000 to solve a specific problem and while trying to do it they go bankrupt they still couldn't touch that money to stay afloat. So non-profits avoid that situation since they typically have enough to survive a year or so, but not enough to be able to restrict what they might need to do with their cash.

There's a rough list at https://wiki.python.org/psf/Fundable%20Packaging%20Improveme.... There's no official roadmap as getting a group of volunteers to agree to a list of priorities would be near impossible. :)

If there's a specific pain point your employer would like to see fixed and is willing to donate money to help resolve that pain point then please go to https://donate.pypi.org/.

> First of all, as an aside, thank you for all the work you do on Python!

Welcome!

> is Don still there?

Yep, working on the data science features of the extension.

> on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so.

As I said there have been donations for PyPI and pip just got its big donation. The CZI donation was also much bigger than pip and covered a lot of the science ecosystem.

> Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented?

There isn't a need for the PSF to have a business subsidiary to accomplish that as people can donate money for such things if they truly wanted to. It's also really hard to manage due to tax laws in the US around non-profits and such. Plus there would be start-up capital costs, etc. just like any other business. I just don't think the risk of trying to get it set up would lead to any benefit when people have always had the option to donate directly to see if something could be done.

If people want to help out financially they can go to https://donate.pypi.org for packaging donations and https://www.python.org/psf/donations/python-dev/ for the language and CPython. And if someone wanted to fund a specific thing I'm sure the PSF is happy to discuss such an idea to see how feasible it would be.

Do note that Python the language is developed by a separate group as Python the packaging ecosystem. So even when Guido was a BDFL he completely stayed out of the packaging situation.

And there is a packaging BDFL, but there's a severe lack of time and effort in the form of volunteers to tackle a lot of the projects, big or small. Plus rolling these things out takes years and it takes even longer to gain traction as people are often reluctant to change their build tools once things work.

As an example of timelines, take what my blog post covers. PEP 518 and the concept of pyproject.toml was conceived in 2016, four years ago. Getting PEP 517 finalized and it all worked into pip didn't happen until pip 19 in January 2019, over a year ago (https://pip.pypa.io/en/stable/news/#id203). And a year later there's still enough of a need to educate folks that I wrote this blog post to help get the word out.

IOW the timelines are huge and the pay-off is way down the line. And that's if you manage to be motivated enough to do the work. Burn-out is a huge problem in the packaging world. I mean reading that you think "it is a fucking mess and it's unacceptable" doesn't motivate me and others to keep trying to improve things if that's how us volunteers will be treated for years to come while we try to rectify the situation. Add on to the fact that we aren't getting paid for this and it makes the idea of pulling out my laptop tonight to work on some packaging stuff feel rather pointless.

So please be understanding that people are working on things as best they can and that overt negativity about it all will work against you getting what you want.

People would love to see that happen, but it requires time and effort from people to build such a solution. Some people are trying, e.g. https://pyoxidizer.readthedocs.io/ and https://briefcase.readthedocs.io/ as well as stalwarts like PyInstaller and such.

But as I said, it takes time and effort and there's only so much of that when it's being done by volunteers in their evenings and weekends.

And as an aside, I would suggest teaching them `python -m pip` over `pip3`: https://snarky.ca/why-you-should-use-python-m-pip/.

It's all about capacity, not will. The PyPA as a whole knows what is lacking, but when you're all just a bunch of volunteers it's hard to tackle massive projects that take potentially years to complete and then even longer to see gain traction in the community when you are doing it all in your spare time. And that's if you don't burn out before you finish (e.g. calling projects "garbage" is not motivating to the few people volunteering their time to try and fix the situation while keeping what is there afloat and running).

> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded.

You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies.

This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies.

Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...).

Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3.

But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!

I personally don't think that comic is fair to hold against pip or the packaging ecosystem. I did a separate post on deconstructing it at https://snarky.ca/deconstructing-xkcd-com-1987/.

Raymond came up with the idea, PyPy implemented it, and then INADA Naoki implemented it for CPython.

Poetry tries to help you manage your whole project while flit is just for building your library (and optionally uploading; you can always use twine to do the upload). It's all-in-one versus one-job-one-tool.

If you use a grepping tool like ripgrep it will automatically ignore everything in your `.gitignore`, so at least for that specific case you're covered.

Please file issues as you find regressions at https://github.com/microsoft/python-language-server as the team is working diligently to fix such issues.

The language server team has been working hard to fix the high CPU and memory leaks as recently as this week. Please do try it out on occasion to see if your issue has been resolved as the team is working hard to improve it.

What specifically are you looking for in terms of pipenv? We actually supported pipenv environments before PyCharm, so if you have specific needs please open an issue at https://github.com/microsoft/vscode-python/issues .

If you specifically want Python support for e.g. coverage.py, then please open an issue at https://github.com/microsoft/vscode-python/issues .

Please file a bug at https://github.com/microsoft/vscode-python/issues

GitHub is planning to keep Atom going: https://www.reddit.com/r/AMA/comments/8pc8mf/im_nat_friedman...

The key thing to know is that the Microsoft Python Language Server -- long, official name I know, hence why we just call it MPLS internally ;) -- comes from our Python workload for Visual Studio. It was an extremely tough call to make because David Halter has done a great job with Jedi and running that project. But in the end we decided that if we were going to need to keep the IntelliSense engine in Visual Studio going then we should get more out of that investment by turning it into a language server and making it available in VS Code.

But I will say (and continue to say) that we are extremely grateful to David Halter and everyone who has worked on Jedi. It gave the extension the initial IntelliSense support it needed in order to be successful. And we currently have no plans on removing Jedi support for those that prefer it (from the extension's perspective we're actually trying to treat the language server as yet another project we have integrated support for, but where we have a direct line to when a bug or feature crops up that we would like to see addressed ;) .

The language server currently only supports 64-bit Linux, so if you have that on your Pi then it should be faster.

If you look under the Visual Studio 2017 instructions it mentions where to copy those files into the extension so they will get used.

The protocol itself is open source: https://github.com/Microsoft/language-server-protocol . I'm sure the folks in charge of it would be happy to talk to PyCharm if they wanted to contribute.

The community edition is free, the professional edition is not: https://www.jetbrains.com/pycharm/features/editions_comparis...

He fought for it because he thought it was the right call. The role of a BDFL is to make those kinds of calls for the benefit of the language and community. It's really tough to fight the "tyranny of the majority" and go with what you think is right versus what everyone is screaming and yelling at you about.

28 if you want the exact number. :) Guido started developing Python in December 1989 and went public with it on Usenet in February 1991 (Unicode 1.0 was standardized in October later that year to give perspective of how far back that was in the tech world).

We are acutely aware of why Guido is retiring sooner than any of us expected; it has actually been discussed already on the mailing list as to what went wrong with the PEP 572 discussion and how we could potentially fix it. So there's no lack of understanding of why this is occurring now and we have already begun to think about how to address the issue.

Many of us have also been thinking about what we would do when this day arrived for years (Guido has hinted at retiring previously), so there's a lot of pent-up thought to get out there while we all start to think about what comes next to a project some of us have dedicated decades to. The conversation has been thoughtful and not a single person has said "I should be the next BDFL". All names put forward by anyone for any position has been by another.

IOW the insinuation that any of us who are trying to grapple with this are trying to vie for power while ignoring why this occurred does not seem like a fair assessment to me. This kind of unnecessary negativity and accusation is why Guido is retiring early.

Please realize that the concept of virtual environments in Python predate things like npm, so there are lessons that were learned later on that no one knew about. Also realize that the things you list all layer on top of each other so you're listing lower-level libraries next to higher-level ones.

E.g. pipenv <- pew, pyenv <- virtualenv/venv, so that list could be likened to complaining that Python, C, assembly, and CPU microcode is "too many". Sure, you could argue they are usable independent of each other, but you aren't about to say that there are too many pieces of tech there and we should stop and sending raw electricity to the CPU.

I think that list could legitimately be cut down to pipenv and conda (maybe pipsi, but that's just for installing CLI tools and isn't for development). Everything else is lower-level than what most people will need for app development.