“Only five people maintain Python-pip” | Hacker News Reader

“Only five people maintain Python-pip”

twitter.com

108 pointssandGorgon50 comments

At Google there is an internal document called "No Heroes". It basically says that if your load is too high - let the damn thing fail. Perhaps it's not important, and only you think it is. Perhaps the higher ups don't realize it's important and they need to be reminded of it. Reliable infrastructure cannot depend on heroic actions of a small group of people, and especially on actions of a single person. Let it fail - the world will take notice. Or not, in which case you should move to something more useful.

I would love to read this or have it available somewhere. It sounds like something a lot of people could benefit from.

If it is at all possible to post it somewhere?

Indeed, it sounds something that should have been part of their now-famous SRE book :)

Is it breaking any obvious NDA or can you share it?

Not affiliated any more, so I don't even have it. But yeah documents you write on payroll are corporate intellectual property (same as code is) so I couldn't just share it even if I wrote it myself.

I wish they added it to the SRE book. But it's not a hard sell if someone already believes that you need to distribute computation/data for higher reliability. Same goes for labor/know-how.

Sounds familiar to the bus factor. You want to keep it high in a project, wikipedia has some hints about it.

Why is this not part of the SRE hype train?!

> It basically says that if your load is too high - let the damn thing fail. Perhaps it's not important, and only you think it is. Perhaps the higher ups don't realize it's important and they need to be reminded of it.

I do have friends at google who can get away with this approach, but have more who feel they would be fired for doing this.

> Google there is an internal document

They're not getting away with it. That's the way it be.

Your other friends need to take away the "internal document" part from this story not the "Let things fail" part. They're not there yet.

The longer I've been around open source, the less it's seemed like some ideological adventure where we extend human knowledge and capabilities for its own sake, while looking more and more like a set of intentionally confusing layers of abstraction and indirection around companies with plenty of money to spend on more engineering hours contriving to buy those hours well at below minimum wage by selling people on the above ideological vision.

Also, I'm going to guess that's something like five active contributors signed up, not five full-time engineers, meaning the actual number once you do the math for availability and such is probably closer to half an engineer. Could be five, but Heartbleed kinda put paid to reasoning like "pip is important, surely it has the funding for dedicated contributors who aren't distracted by something else being their real job."

What solution do you propose then ?

Money is always decoupled from value ( Warren Buffet paraphrased ).

Sometimes people who add negative value like the oil men in the Permian basin get paid 100x then your pip developer.

Screw that, large amount of the surplus generated by technology companies ends up in the hand of land owners instead of stockholders ! let alone the engineers building it - who might be riding on the work of open source volunteers. Its turtle all the way down.

If you are a volunteer, you are forgoing payment and are subjecting yourself to exploitation by others by definition.

Unless you have a mechanism for enforcement you are shit out of luck.

Isn't this the point of "copyleft"? Create a valuable product, require that when someone builds on it, that what they create also be free to use.

Open source software fails like this even if it's valuable because fixes and improvements don't necessarily make it out into the open, whereas Free/libre software, if it's valuable, gets stronger over time because fixes and improvements must be available to everyone.

Adding "fixes and improvements" to software haphazardly does not, in general, make it stronger. Quite often it can be the opposite.

Software projects need someone to maintain its course, make sure changes match the purpose of the software, and most importantly, reject changes that create a burden on the project.

Not only that, but onboarding yourself into a new project/codebase can take a significant amount of effort.

The point of copyleft is more ideological; it prevents others from using your software to restrict user freedoms, thereby limiting the spread of harmful, nonfree software. Copyleft is primarily intended to protect users.

Hold the phone. When you say “oil men” are you referring to the entrepreneurs or the workers? 100x s/w engineer salary sounds like a big deal..

"pip developer" probably means relative to the specific funding for pip development - very little - and not a full-time software engineering position.

I would guess most open source contributions are motivated by the author’s business need, not ideology.

For me it's both. I'm not going to work on a project for the sake of open source, but ideology is a big part of why I choose to release my code under an open source license. In fact, if I need something that I think would be useful to the community (or myself on other projects), I'll build it on my own time and then do bugfixes and whatnot on company time.

People are all over the spectrum, with people like Stallman and developers at the FSF working mostly on ideological grounds and companies like Oracle only contributing when it would clearly be beneficial to the bottom line.

I used to think I would love to work on OSS if I didn't need money, but I've found that I just like working on stuff, and OSS isn't as much of a priority anymore.

Five doesn't sound so bad. There are lots of FOSS projects with fewer or just one people maintaining them.

Besides that though, I wondered how much maintenance pip even needs. Shouldn't it be kinda stable?

But looking at https://github.com/pypa/pip/graphs/commit-activity it doesn't seem so.

Reading about the attacks on the SKS keyserver network, I get that zero people have maintained SKS.

Personally I'm more concerned about the attitude and atmosphere that implies that 5 maintainers are too few for a mature project with relatively limited scope. I'm more and more beginning to think how can we reduce the ridiculous amount of useless churn in software.

The problem is exactly that pip is neither mature nor has limited scope. Python packaging is constantly evolving, and pip needs to keep up with it. And you’d be very surprised if you ever look under the hood how incomplete the implementation actually is.

Python packaging is constantly evolving

I think there is a reasonable argument that that has been part of the problem with Python packaging.

Although I think five may be quite good comparatively, my thanks to the pip developers.

And if five is not a good number, this is indirectly one of the reasons that I enjoy using python. It has a 'batteries included' release, where I often write code without feeling like I need to $ pip install anything at all. If the 3rd party package delivery system is fragile, that isn't good - but it doesn't represent a dire crisis for me as a user of python.

Not my experience with Python

I wind up having to install as many packages as dealing with a Node JS project

Genuinely curious: what is your experience with python? What have you used it to do that requires so many packages?

Thank you pip people.

Let's nominate them for PSF fellowship https://wiki.python.org/moin/PythonSoftwareFoundation/Fellow...

Most commenters are rightly worried about the “bus factor” element, but I’m just amazed at the returns that you can generate these days with so little manpower.

Five people literally enable entire ecosystems of developers across the world, multiplying their productivity by ungodly amounts and indirectly generating billions of dollars of value. Isn’t that extraordinary?

This is not unique to Python-pip. A bunch of popular packages are maintained by very few people. But the good thing is that it does not really matter than much with FOSS projects: as long as the source remains available, anyone can pick it up at some point and improve on it - or you can even contract people to work on it when needed to.

It does matter. Witness the recent hacks because maintainers are overwhelmed.

There's a fine line between "there are so few maintainers that nobody could spare the attention for this" and "there are so many maintainers that this slipped through the cracks".

Really, it's less of a line than a region of negative width. Number of maintainers is not a good indicator for the security position of the project.

It's all fun and games until your heart bleeds

Yes, but after HeartBleed you had resources quickly shifted to support maintainers so it's not really a good example.

Or get hit by a (hypothetical) bus.

https://en.wikipedia.org/wiki/Bus_factor

Read the thread. Feel like it's fine. What's the number people would be happy with? 10? 50? 500?

The other argument is the more people you have the greater the risk of a compromise - sometimes less is more.

That is probably why it works so well to be honest.

Isn't there mainly one person responsible for writing Rust's Cargo? I vaguely remember them giving a talk.

Why guess when you can find the truth in 20 seconds of digging?

https://github.com/rust-lang/cargo/graphs/contributors

There's a dead comment that posted this link with a sarcastic tone, but I think it's worth seeing:

https://github.com/rust-lang/cargo/graphs/contributors

Maybe 2/3 people actively doing most of the contributions with sporadic other contributions.

It looks like I was right, until about 2016, Alex Crichton was the only big contributor.

How open are they to receiving help? Gatekeeping is often a big problem in open source software.

Very open - Python is one of the better communities out there for openness and inclusion. The pip folks are lovely people and would welcome your help - https://pip.pypa.io/en/latest/development/.

I've put in a couple of PRs to pip and they've been very responsive and professional.

I can't speak for PIP specifically, but something I've noticed in Python (just bc that's where I work) is that lots of projects would greatly benefit from more handholding for new contributors.

Obviously it's a question of limited resources, but I wonder why projects don't offer a clear path to growing as a contributor. For example, once you triage 3 tickets and close 5 documentation tix, you get assigned a mentor and a few features specifically set aside for mentor+mentee to build together. Then use these not as methods of gatekeeping (let anyone contribute as much as they want), but as a way to pull in a few new people at a time.

I also think that many projects would benefit from having formal "apprentices". Being an apprentice for React would be an amazing thing to have on your resume, and would cost no money for the project.

Wow! Five people dedicate time to maintain pip for no profit!

It's pretty amazing like the post states. After openssl's heartbleed I remember hearing about a critical FOSS fund to help with things like this. Much like how wikipedia is funded,it makes sense to fund critical projects like pip and npm.

Npm isn't FOSS.

It's owned by a private entity and it's server side code isn't available to my knowledge.

The situation is largely similar for rubygems, where a mere couple of people maintain the thing, and not even full time!

> 2-3 devs at 5 hours per week. [0]

The Ruby Together initiative aims to make the situation better through recurring donations.

[0]: https://rubytogether.org/

Well, thank you pip people! <3

At least pip had a release in May this year.

It's even worse for pipenv. It was last released in November last year and there are a lot of bugs fixed on master, but there is never a new release.

Reply on news.ycombinator.com