Urllib3 in 2022
sethmlarson.dev
sethmlarson.dev
It would be good to see more large companies making these sorts of donations. It's not like Spotify is unique in its usage of Urllib.
Why should I personally pay for the tools I use at work to make my employer money? Should I also buy my own work laptop? My own software licenses? It doesn't make sense.
Then there's also the tax issue, as others have pointed out. I'd be giving post-tax money in a lot of instances, whereas for Google it would have much more favorable tax treatment.
But the biggest problem is that me personally making $1000/year of donations is pissing in the ocean. It accomplishes nothing. A program where every engineer is doing so accomplishes a lot more.
Wait til you hear about the amount of tools trades people have to buy for their jobs!
(Also worth pointing out that some of those tradesmen are legitimate contractors, and own their own tools in the same way that a legitimate software engineer contractor would own their own computer and IntelliJ license.)
Your argument could be used to say that OP should pay out of pocket for server hardware to run their programs that Google uses, which doesn't make sense.
My argument cannot be used for server hardware, because, after all, Google has to pay for that.
Same for this. One individual elective donation to an open source project doesn't do much. A larger system though can do a lot.
Maybe rephrasing my argument makes it more palatable to you: Shouldn't we, the guild of software engineers, take care of our own and pay our sub guild of open source developers? Why would we leave that to faceless companies?
It also amounts to a $1k pay cut for employees, which no one will be happy with. We already capture a small amount of our value add to the company in terms of wages; this would just make that a little worse. And no, there is no fixed amount that goes to salary plus open source contributions, necessitating the same overall net sum.
It is not really different from whether the company pays for your office space, laptop, and health care, or makes you pay for those out of pocket.
One of Andrew Yang's policies when he was running for president was to give every American $100 to allocate to whatever non-profits they wanted. This is really the kind of policy we need to drive open source. I'd love to open source more software myself, but right now the thought have having 25 million people using my code commercially and making maybe 5k a year if I'm lucky isn't super appealing.
You will see a cottage industry of OSS projects that serve only serve the purpose of parting people from their money. This is already something we see in the non profit space. But there the extent of the fraud and the damage it causes is limited because there isn’t enough money to attract scammers en masse.
This led one candidate to hire people to solicit/collect these democracy vouchers. He raised over 350k from democracy vouchers.
When the election came around, far fewer people voted for him than contributed democracy vouchers.
And the program administrative and overhead costs are quite high, approaching the level of total funds distributed to candidates.
Just pointing out that handing out money isn't free (although it is one of the true core competencies of the US federal government), and can have unintended unproductive side effects.
Saying a person should be in the race shouldn't mean that you definitely think they should win it, especially in a FPTP system.
This sounds like what you'd expect an actually successful/winning candidate to do as well (explore all net positive fundraising avenues). The issue is, this guy wasn't actually a good candidate, and didn't excel at anything else.
It's still a complicated system so it'd would take serious buy in. Not to mention the drama donations generate internally and publicly. But it's a noble goal.
$1000 per engineer grant might still be easier and less fragile
The system would only be there to bunch up "votes" so if multiple people choose one project it gets paid out in one bulk payment w/ a fallback to other projects if that payment fails.
It seems like Google is slightly better about this in the Java world, but for the languages I mostly use Google is more guilty of "Not invented here" than other places I have worked. It's possible most of the code in third_party is written by Google, btw.
Anyway, here's the list of dependencies for my project. There's a lot of third-party stuff in it: https://github.com/google/nomulus/blob/master/dependencies.g...
Generally, I miss 99% of the internal stuff at Google. Prometheus/Grafana is very disappointing compared to Monarch/Panopticon, for example. I also like Go a lot more than Java. So generally, I had no complaints about NIH when I was at Google. Whether it's the best use of their money is up for debate, but I never found them hostile towards open source. (I was also a third_party reviewer. Lots of people using open-source libraries at Google.)
Maybe I'm just older / have been in more diverse parts of industry, but I've literally worked at employers where open source software was verboten (this used to be quite common even). Compared to that, my use of dozens of open source libraries on my current project at Google is open source heaven. Sure, a smaller startup might be using even more open source libraries because they don't have the resources of Google to write stuff like Monarch/Panopticon for internal use, but to my eye that's orthogonal on the issue of open source friendliness.
Surely some MBA people are the ones driving currently, so you're probably not wrong, but you and I might share different outlooks on anything turning around. I believe we're expected to be grateful that there was any donation at all.
This is what Tidelift is trying to do, the biggest barrier to more projects getting the amount of funding that we do from them is the number of businesses wanting that sort of relationship with their open source dependencies. The amount we receive from Tidelift is not typical for most projects since we're a medium complexity project that is a part of almost every Python project dependency list.
You can dual-license to avoid comercial exploitation. Why then set an expectation on how much should be donated for complete open source projects?
See: https://liberapay.org (and also patreon.com has a number of projects)
The idea is that people that can give would give, and those that can't won't. And who can give can also choose how much they think the project is valuable and how much maintenance it needs. Hopefully that would be covering the needs of the developers at a fair salary. It would be nice if open source projects were more transparent about this as well: "I need X/month to reach a fair salary and good maintenance, and could expand, hire devs and add functionality for Y/month." (milestones in patreon partially fulfill that role).
The problem is when there are others that rely on the work as well. How much should each give, such that the system would be fair? From a theoretical perspective, I think the money should be always coming from the source with the lowest marginal counterfactual return. That is, the organizations who have funds going to investors and other orgs with the lowest ROI should instead divert those funds to [something else], where [something else] is OSS in this case, until an equilibrium is reached and everywhere is operating at equal and optimal ROI. In practice things are not so easy. And this analysis (and financial analysis) tends to only value economic ROI, where we should be thinking of social ROI as well as environmental ROI.
But to give a rule of thumb, for now, I think it would be reasonable to pay
(a) if it is less than a % of the (internal) project cost: a % of the budget of the OSS project, depending on how valuable it is internally (say, 50% if their budget is low, or just 1% if their budget is high);
(b) a % of the (internal) project cost otherwise (e.g. at most 5% of project cost).
Ideally there would be some kind of system or framework to streamline this sort of evaluation and allocation to the devs.
I actually think a whole complementary economic system (i.e. enhancing capitalism or socialism) should be developed around this idea. We should be paying whomever is providing value to society, according to their needs to provide this value (and also give them a good life of course! -- and provide a reward/incentive to do valuable things, even ones that don't currently have an incentive). So some kind of organizations, that could be tied to companies or governments too, would be responsible for evaluating on a reasonably objective basis which projects need money and then allocating it (sort of as an outsourcing of the resource allocation job to specialized entities). But meanwhile individual and voluntary giving is basically that without outsourcing. I try to do this personally through Effective Altruism (which is essentially just that: give effectively) and giving what I can to Open Source and other impactful causes.
https://open.spotify.com/episode/0erUH7oqqbW5HDUjcnK6cb?si=q...
It also seems like they have mostly pivoted away towards software supply-chain analytics.
The labor cost is often huge. If you don't prevent fraud at the $500 level you'll have two problems: Lots of repeated fraud at the $500 level, and people trying to get away with fraud at the $5000, $50000, all the way up to FTX levels of fraud. So you need multiple people in different depts to agree to fund something while other people watch over to make sure the people being funded are not the spouses or shell companies of the approvers. Meanwhile there's paperwork costs to make sure the donation paperwork gets to accounting so accounting gets their tax deduction.
At a "really big company" the labor cost of paying an invoice might be $2000. Now does the average programmer get more than $2K of annual value out of urllib3, given the alternatives (which could involve up to architecting around needing it?)
Before you laugh and say "no way" to the $2K figure, imagine spending half of an hour long team meeting of ten software devs getting paid $100/hr but overhead etc costs the company $150/hr for them to attend that meeting, that's $750 of labor costs just to have the dev team possibly agree to support urllib3. Then its got to go up the chain to someone with signing approval, someone has to chase the paperwork to make sure it happens, frankly $2K is on the very low side of the cost of getting an invoice paid.
Commercial non-free software is "put up with" where its not that some rando piece of required software costs $50 thus the cost of the company including labor and overhead is $2050 and they're happy to pay it, its more similar to if you don't dot all the Is and cross all the Ts that software will cost $10M in some crazy lawsuit, and $2050 is cheaper than $10M, so they pay up or they'll get a huge lawsuit or more likely someone will get fired after the audit, and I'd rather pay $2050 for something worth $50 if it prevents me from getting fired and better yet its not my $2050. But nobody never got fired for not paying for urllib3, so nobody's paying urllib3.
Companies will pay for support and I could see a large company paying $10K/yr for a written guarantee if a dev at that company runs into a use problem or bug with urllib3 then an actual dev from the project will help them up to twenty hours per year or similar, even if no one asks for help. In contrast feel good donations are very expensive so they don't happen often, so trying to run a business off feel good donations doesn't work too well..
In a big company it's often a hassle to pay something to a "new vendor". It usually takes a long time.
> If you don't prevent fraud at the $500 level you'll have two problems
I had a colleague who used to work in the Auditing section. He had various stories of people committing fraud (stories of 50k USD+). There's a cost to trying to prevent all cases of fraud and that might hurt more than to try and prevent everything.
I've also heard cases (not from the auditing guy) where a vendor notified that we paid them 1000 times too much (caused by an input error with the , and the .). That happened despite several layers of checks.
> At a "really big company" the labor cost of paying an invoice might be $2000.
At a "really big company" if something is a good idea then it'll get done. And those handling administrative tasks are often "outsourced" to cheaper countries, e.g. India/Philippines/China/etc.
There have been several donations to "good causes". Those were handled without trying to figure out the cost of someone working in an entirely different part of the world and company. This as it's not like someone will specifically will be hired, or it really matters.
They usually made an event out of an donation (e.g. handing out something in person in an office). Funny enough the event surrounding that was often poorly attended.
> that's $750 of labor costs just to have the dev team possibly agree to support urllib3
You're not talking about a donation any more.
I could create a tool - or a suite of tools - to parse e.g. a Python, PHP, Javascript (Node), Java, or other codebase and determine the dependencies. Hell, the requirements.txt file would probably be enough. Maybe I'll manually see how they use Linux, Firefox, LibreOffice, Anki, Inkscape, etc too. The company then cuts me - a reputable business that distributes funds fairly to open source projects - a check and I divvy it up, keep n% for myself, and distribute it to the projects and to the FSF.
With enough clients some open source projects could see significant funds increases, and the companies would be alleviated of responsibility and worry. I could even issue "badges" to significant donors.
If that's the case all those credit card bills from corporate trips to other offices must have costed A LOT, like billions. Maybe even hundreds of billions.
That would be nice, but I think it would be even better if these companies allowed their employees to contribute. And were open about their policies.
Maybe there should be a ranking site where employees can report anonymously about company policies. And companies can respond with their official policies.
The ranking could be something like this
0 - can't contribute at all
1 - can contribute in spare time with approval
2 - can contribute in spare time without approval
3 - can contribute on company time with approval
4- can contribute on company time without approval.
YMMV if you’re in another jurisdiction.
(Just in case this comes across as snarky, it’s a legitimate question.)
This space is a little messy for python. The old urllib ships with python, but it doesn't support some normal expected things like HTTP/1.1 pipelined requests. It adds a "Connection: close" header to each request.
https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
Here's one discussion about it in the requests repo https://github.com/psf/requests/issues/2424
Coding against a pile of external python modules is perfectly reasonable when you're building something "app scale" -- i.e. something that is going to be spread across many files and installed in a container or venv.
However, when writing something at "script scale" I just don't want to deal with all of that. I want to write something that I can deploy as a single file and not end up dealing with missing python dependencies every time. This means I'm using the old urllib and such more often than I'd like.
It's a shame that there doesn't seem to be much flow of functionality into stdlib at all any more. For my needs, I wish "requests" and "yaml" were in that set, although I'm sure other people have their own opinions on that.
I could look for myself, but it might be more fun as an open discussion topic.
import json
import urllib3
http = urllib3.PoolManager()
data = {'key': 'value'}
encoded_data = json.dumps(data).encode('utf-8')
r = http.request(
'POST',
'http://httpbin.org/post',
body=encoded_data,
headers={'Content-Type': 'application/json'}
)
import requests
requests.post('http://httpbin.org/post', json={'key': 'value'}) import urllib3
urllib3.request("POST", "https://httpbin.org/post", json={"key": "value"})Yes, I just didn't in my toy example. That's the response object, which will typically include all the response info, plus reponse.request, which has request info.
> and does it give you the same information to help debug that http.request gives?
I'm not positive, but I'd bet it's similar.
Its interesting to compare to other industries. "In the old days" the way to make money during a gold rush was not to try to mine gold after the good stuff was already dug up, but to sell shovels to wanna-be miners. Apparently, software is nearly the opposite of that, LOL.
See also: cloud computing
urllib2 was a standard library replacement for urllib in Python 2. Python 3 further replaced this with a new base urllib. urllib3 is a completely separate, unofficial, alternative library.
The original author of the package likely didn't know it'd grow to be the most downloaded package at the time!
Universal, that's what the U in urllib is for, right? If you make specific derivatives, it's not universal anymore. Something fundamental is lost.
Full backward compatibility is all we need. Keep it simple please.
Your entire business is build around web scraping and using libraries to pull data from other sources. Common, you can do better!
Thank you Indeed (and others on the list, especially Spotify).
Edit: toned down a bit.
This is the Copenhagen Interpretation of Morality.
Thanks for posting this.
I wouldn't be surprised if that effort was driven by a single employee.
Is that a mistake? Should I switch to urllib3?
It's something to be aware of should the above change I suppose:)