97 karma · joined October 30, 2019
Sometimes a PR either merits limited input or the situation doesn't merit a thorough and thoughtful review, and in those cases a simple "lgtm" is acceptable. But I don't think that diminishes the value of thoughtful non-in-person code review.
They've done it this time by making poor architectural decisions ("Isolated builds should install the newest setuptools") and then add in poor library maintenance decisions ("We'll remove this feature used by thousands of packages that are still in use as active dependencies today"). Possibly each of these decisions were fine in a vacuum, but when you maintain a system that people depend upon like this, you can't simply push this stuff out without thinking about it. And if you do decide to do those things, you can't just merge the code and call it a day without keeping an eye on things and figuring out if you need to yank the package immediately! This isn't rocket science, everyone else developing important libraries in Python world has mostly figured this stuff out. In classic pypa form, it sounds like there was a deprecation warning but it only showed up if you ran the deprecated command explicitly, while the simple presence of this command causes package installs to fail. You have to at least warn on the things that will trigger the error!
These days I try to rely on the absolute minimum number of packages possible, in order to minimize my pypi exposure. That's probably good dev practice anyway, but is really disappointing that I can't rely on third party libraries basically at all in Python, when the vast pypi package repository is supposed to be a big selling point of the language. But as a responsible developer I must minimize my pip / setuptools surface area, as it's the most dangerous and unreliable part of the language. Even wrapper tools are not safe, as you see in the thread.
>Twitter’s API service limits tweets for non-paying users to 1,500 a month — not enough for many emergency accounts. And while a small fee for the platform's 'Blue Check' service will increase that ceiling for individual users, the cost of an enterprise account has reportedly climbed into the tens of thousands of dollars.
If you just sit around letting your current company take advantage of you, and complaining about not getting a big enough raise you definitely won't find those opportunities though. You have to interview a lot.
edit: changed "much" to "slightly"
Though I will say, while most have been fixable in a few minutes, some have been a real chore to fix. Sometimes an innocuous looking error balloons into several hours of reconciling obscure type system behavior errors once you start fixing it. Regardless, it's a small price to pay for proper type checking in Python. I've more than made up the lost time in detecting bugs before they ship.
But the big culprit is typeshed. Something will get new/fixed annotations and suddenly you aren't handling all possible return types in your callers, or whatever.
The authors don't seem to ever discuss the fact that mypy version changes frequently make previously-passing code fail type checks.
I don't think I have ever upgraded mypy and not found new errors. Usually they're correct, sometimes incorrect, but it's a fact of being a mypy user. Between mypy itself and typeshed changes, most mypy upgrades are going to involve corresponding code changes. The larger your code base and the more complicated your types are, the worse it'll be, but it's basically an ever-present issue for any program interesting enough to really benefit from a type checker.
How many of those repositories were "type-correct" but only on particular versions of mypy? I bet it's a lot!
Maybe I'm spoiled from working with cargo and npm (I have almost no ruby experience so I can't comment there), but both of them have way fewer such version conflicts in my experience. Obviously there are tradeoffs and I don't want the node_modules experience for my users, but often it seems that would be a much better experience than pip for everyone. With either of those, I just "npm install" or "cargo install" and all my dependencies end up there working.
You can generate a requirements.txt file using "pip freeze" on a functioning system, but then you have to figure out a way to point users at it instead of using "pip install myapp". Also you might have to do it for each OS since windows vs mac vs linux can have different package dependencies specified, and even if you don't do that, a dependency doing it means you have to account for it.
You can copy+paste the "pip freeze" output into your setup.py and add quotes+commas, but then you're back to breaking side-by-side packages.
So what am I, a developer trying to distribute my command-line application to less-technical users, supposed to do? Distribute two entirely different packages, "myapp-locked" and "myapp"? Tell people to install from a copy+pasted "requirements.txt" file? I've started distributing docker containers that have the application installed via the requirements.txt method, which is fucking stupid but at least the users of that complain less about versioning issues... until the day someone yanks a package I guess.
This gets much much worse when a new version of Python comes out and we don't support it yet (because of the build system issues I mentioned). I spent several weeks teaching people how to uninstall 3.8 and install 3.7 before we finally got a functioning package out for 3.8.
That said... if you work on software that is distributed to less-technical users and have any number of dependencies, python package management is a nightmare. Specifying dependencies is just a minefield of bad results.
- If you specify a version that's too unbounded, users will often find themselves unable to install previous versions of your software with a simple `pip install foo==version`, because some dependency has revved in some incompatible way, or even worse specified a different dependency version that conflicts with another dependency. pip does a breadth-first search on dependencies and will happily resolve totally incompatible dependencies when a valid satisfying dependency exists.[1]
- If you specify a version with strict version bounds to avoid that problem, users will whine about not getting the newest version/conflicting packages that they also want to install. Obviously you just ignore them or explain it, but it's much more of a time sink than anyone wants.
- In theory you can use virtualenvs to solve that problem, but explaining how those work to a frustrated Windows user who just spent hours struggling to get Python installed and into their `PATH` is no fun for anyone. Python's made great strides here with their Windows installers, but it's frankly still amateur hour over there.
- Binary packages are hell. Wheels were supposed to make Conda obsolete but as a packager, it's no fun at all to have to build binary wheels for every Python version/OS/bitness combination. `manylinux` and the decline of 32-bit OSes has helped here, but it's still super painful. Having a hard time tracking down a Windows machine in your CI env that supports Python 3.9? Too bad, no wheels for them. When a user installs with the wrong version, Python spits out a big ugly error message about compilers because it found the sdist instead of a wheel. It's super easy as a maintainer to just make a mistake and not get a wheel uploaded and cut out some part of your user base from getting a valid update, and screw over everyone downstream.
- Heaven help you if you have to link with any C libraries you don't have control over and have shitty stability policies (looking at you, OpenSSL[2]). Users will experience your package breaking because of simple OS updates. Catalina made this about a million times worse on macos.
- Python has two setup libraries (`distutils` and `setuptools`) and on a project of any real complexity you'll find yourself importing both of them in your setup.py file. I guess I should be grateful it's just the two of them.
- Optional dependencies are very poorly implemented. It still isn't possible to say "users can opt-in to just a specific dependency, but by default get all options". This is such an obvious feature, instead you're supposed to write a post-install hook or something into distutils.
- Sometimes it feels like nobody in the python packaging ecosystem has ever written a project using PEP420 namespaces. It's been, what, 8 years now? and we're just starting to get real support. Ridiculous.
I could go on about this for days. Nothing makes me feel more like finding a new job in a language with a functioning dependency manager than finding out that someone updated a dependency's dependency's dependency and therefore I have to spend half my day tracking down obscure OS-specific build issues to add version bounds instead of adding actual features or fixing real bugs. I have to put tons of dependencies' dependencies into my package's setup.py, not because I care about the version, but because otherwise pip will just fuck it up every time for some percentage of my users.
[1] I am told that this is "in progress", and if you look at pip's codebase the current code is indeed in a folder marked "legacy".
[2] I 100% understand the OpenSSL team's opinion on this and as an open source maintainer I even support it to some degree, but man oh man is it a frustrating situation to be in from a user perspective. Similarly, as someone who cares about security, I understand Apple's perspective on the versioned dylib matter, but that doesn't make it suck any less to develop against.
I think a lot of this has to do with the effort you put into community management. We put tons of energy into making it easy to make good contributions: PR/issue templates, we're on slack, we respond to emails, we make it as easy as possible to get your code in as long as it passes tests. It's a lot of effort and I can't imagine doing it for something that was just a personal project, but I think that's how the bigger projects deal with it.