To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load.
It's an incredibly entitled way to think about things and I would have a real hard time employing someone who's first response was like this.
Cocoapods uses GitHub. No abuse here.
Using a GitHub repo as a high traffic code CDN and keeping 5+ cores pegged while being the single biggest consumer of resources across the whole platform could be reasonably defined as an abuse of the service.
Primary definitions on Google:
"abuse" verb: use (something) to bad effect
"abuse" noun: the improper use of something
I don't know... I'm sure they'd get some, CocoaPods has a very large user base after all. But if GitHub laid out the reasons like they laid out the options to start this thread, I think they'd weather the storm fine and diffuse some people who show up to be angry.
I could see why GH would drop them and think they'd be well within their right and in good moral standing in my book. I just don't think they would unless the maintainers became incredibly hostile or proved unable to fix the problem or even band-aid it. GH just seems too culturally invested in making things like this work to a satisfactory conclusion. I have a feeling that the CocoaPods team will be schooled in a lot of things directly from the GitHub team as they work to resolve the performance issues, just look how informative the initial post was.
It's abuse. Wasn't intentional, but at the scale that Cocoapods is running it's abuse.
Discounting the fact that CocoaPods is used by millions of iOS developers...
If CocoaPod was run by a business that was charging those millions of developers for their services, it would be reasonable to expect that business to pay for a real CDN.
They're not, so that's not a reasonable expectation. But it's no more reasonable to make these demands of GitHub. Giving out a free product doesn't mean that you're required to give it out unconditionally, or in unlimited amounts, or forever. In the end, GitHub owns the infrastructure and services it's providing and can do what it wants with it.
Just about every other package manager for every other language (Pip, CPAN, Hackage, etc.) uses its own infrastructure.
I've run tor exit nodes and repo hosting when I worked for ISPs and Datacenters while at the same time shuting customers down who do the same. The difference being that I had that conversation with my boss and said, 'this will violate our normal terms of service but I would like to do this.' The boss is of course more willing to make that concession when he can walk down the hall and say, 'uh, we have extremely high usage and today, can you shut down the repos until we can get another link installed?'
If you want, you can email me (link in profile). Thanks!
Everything you need is online but you'd have to find it for yourself and make decisions about the best way for you to do it.
I certainly don't think the barb about your willingness to employ people who write things on Github issues threads that you disagree with is helping anyone understand any part of this situation. I understand the urge to find ways to be emphatic about how much you disagree with things, and I often find myself compelled to write lines like that, but I think they're virtually always a bad idea.
this is what we used to call the "good" problem (things breaking because they are successful), but that doesn't make it cheap or easy to fix. the other stuff they're talking about in the thread will alleviate some pain and buy some time, but it won't solve the fundamental problem if CocoaPods continues to get "big" (imagine apt or yum trying to run like this).
i understand their want to maintain the simple and coherent workflow... if i was writing a package dist sys, i would love to have it work off a standard git repo. maybe this is something that could be solved with a plugin architecture like the large binaries stuff so that developers could continue with their preferred workflow but end-users could take advantage of a CDN-like system for distribution.
This is mine as well, but it's also troubling to me, given that the repo in question is meant to be a package management system; it means there are fundamental holes in the user's understanding of packaging systems.
My mentor has a lot of contempt for the bevy of packaging solutions that people come up with - invariable people look at the old ones, think they're too complex and wrong, write up new code that is Slick(tm) and Fast(tm) and Cool(tm) and they are... until they hit scale. Whether that scale is number of users, or serving multiple environments, or serving a great many packages of different versions... the lack of domain knowledge in the design stage will cause huge amounts of issues.
It seems to be one of HN's go-to insults. "Look at this person's behavior, I would never hire them," as if everyone wants to work at your startup.
Do you really think that when people say "I would never hire that person" that there is an implication of "everyone wants to be hired by me (and by extension my company)?"
Nailed it. Just because someone, a team, or a company has hiring criteria doesn't mean they assume everyone wants to work at their company. It means they have an idea of who they are looking for.
its the difference between buying a packet of your favorite snacks and telling all your friends what your favorite snack is..
you probably expect them to like the same snack.
Additionally what if they inform me about my snack with information that means I can't morally choose it anymore, or that it's dangerous to my health? I now have the opportunity to switch my viewpoint, or reduce the weight that it has in my criteria.
It begins the discussion if you as the person starting the thread are interested in having it and not just looking to be agreed with. I, whether I'm in the minority or not, am always looking to start the dialog. Being agreed with is boring.
I wonder how many times CocoaPods has ruined someone's day/night on some GH team. I wonder how many dinners some mom or dad has missed with their kids because their service alarms are going apeshit. I don't think it's hyperbole to say that if you are a top 10 repo at Github, you are responsible for ruining individuals days and taking time away from their families if you are hammering the system.
Now, these are entirely my opinion and I'm not saying alloy is bad at what they do. I'm saying that is a collection of attitudes that I'm not going to put on my team.
Let's think this through and ask ourselves a few question.
1. Did they go out there way to do damage? 2. Are they responsible for deciding the infrastructure and how well it can handle with load? 3. Did they force said people to work at GitHub? 4. Is the open source culture and hosting a major part of GitHub business plan? 5. Are they responsible for staffing to ensure people are scheduled to work when work needs some?
That's selfish to the point that it could be a textbook example of an externality. Fortunately, like you said, things got agreeable by the end with Alloy taking simple steps he was given to make thjngs better for everyone.
That's actually not true.
https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... is the answer we are talking about, aren't we? What alloy is doing there is:
1. Thanking mhagger for the response
2. Asking for additional explanations
3. He explains then why the project is taking the route they took, the benefits for them. Explaining alone does not mean unwillingness to change. It just makes sure it is clear why it is like it currently is. Yes, that mentions the time and money benefit, but so what – it's honest, and it is valid to limit expectations on what is possible now.
4. He then indeed asks for a discussion on how to improve the current system without changing it completely. But this does not mean not investing time, to the contrary: He actively invites a continuation of a discussion and already there makes clear that he is indeed willing to work on a better solution, and that is the core point making this a constructive answer.
How the discussion continues in my eyes clearly shows that the negative interpretation of this first answer, in your comment and this thread, is wrong. That's not someone blocking change, not even at first, that is someone asking for clarification and even clearer tasks to do. That's not a bad thing at all.
"That's actually not true."
"taking the route they took, the benefits for them... Yes, that mentions the time and money benefit, but so what – it's honest, and it is valid to limit expectations on what is possible now."
So, it was true. Then the dialog continued from there with Alloy paying attention to the situation. Alloy refused to do any signiciant work on their end for their maximal benefit in a disruptive situation from a free service. Even admitted they were using a service for something it's not designed for but didn't care about moving to a more appropriate one. Hence, my calling it selfish.
Eventually, others had invested their own time and energy into the problem enough to come up with some simple recommendations for Alloy that take very little effort on his part. Alloy summarized those and agreed to attempt them. Thread was closed before we could see where that went.
Given above, I stand by my claim that he was a selfish individual pushing his own liabilities onto others wherever possible. Even in how it was remedied was mostly on others. On other hand, he might make a good capitalist w/ that level of exploitation and externalizing. :)
> I.e. I’d like us to continue this discussion, at first, from the notion of us maintaining the existing architecture. Where things are absolutely impossible, it would be great if you can include more links to docs/source that explain why things are impossible.
Notice the "at first", notice also the following proposed action of working with a snapshot.
You made clear you were expecting another reaction. And I see why. Still, I think you miss how much good faith was contained in this response. It is a bit sad this gets overlooked. People forget text is not that easy to interpret.
You're right that text is not easy to interpret. For instance, there were two interpretations of my text: a literal and precise one that focuses on how much effort I say he would commit; an interpretation that realizes I was speaking figuratively with hyperbole. It was the latter. The message was a counterpoint supporting that he was selfish rather than a precise statement of how selfish he was. You would be 100% right if we were talking literally about him such as in a court filing or HR report.
"Notice the "at first", notice also the following proposed action of working with a snapshot. You made clear you were expecting another reaction. And I see why. Still, I think you miss how much good faith was contained in this response."
This is possible. Let me re-read his post first. Alright, done. Here's a re-review.
His first response starts with thanks and statements that show either (a) an incredibly joyous and friendly personality or (b) brown-nosing of a salesman before a pitch. Horizontal line. Unclear on some things. Asks for more information. We then get to the reasons:
1. Did no work on syncing data to reduce funded development hours.
2. Don't want to operate a repo due to reduced effort or funding.
3. Easier for their users and adoption.
These are all self-centered. Honest as you said but already support my claim of selfishness. Let's keep looking. Upon a suggestion of other packaging systems, vaguely claims they are using a "smarter" method then reiterates HR and funding justifications above. Ignores alternatives in next sentence to reiterate their existing, strange, and broken solution with a dismissal about having to build a cathedral rather than just using existing solutions.
So, Alloy already laid a foundation of total selfishness in terms of time, funding, and design inflexibility. At this point, Alloy is interested in solutions that totally maintain their existing design and lack of commitment to anything else. Offers to make a few simple changes that "would still use your resources." Asks for information that basically leads to those in recommendations that they begin to apply.
So, re-reading his post, it comes off as incredibly selfish using text that's not hard to interpret. He clearly believes their design works, won't be changed unless forced, changes must take little effort from them, they must not use their funding, and must specifically use GitHub's resources. My claim of selfish and externalizing is fully supported at this point. I think the other commenter's claim of being "myopic" about what he's doing in the project is accurate, too.
Now granted, I'd try to give them a chance to step away from that statement and let them show me that they are interested in understanding the issue and interested in reducing their impact on the product. I'm okay if they don't yet know HOW, but if they basically just throw up their hands, say it's someone else's problem, and leave it at that then no. In fact, hell no. I'm not interested. It's shows a level of entitlement and a lack of interest in their craft and I will not subject my team to someone like that.
The Git deep v. shallow issue just puts a band-aid on the CPU problem, but it doesn't do anything about the terabytes of bandwidth per week (it'll be worse), and it won't do anything about GitHub's claim that Git is not meant to be used as a CDN and doesn't scale well.
They've become a big project that warrants thinking about revenue or organization strategy, but they're delaying it by externalizing their costs. Cases like these can pressure GitHub into rethinking the leniency of their policies.
I also think that if you're in the top-5 resource consuming group, more sympathy would go your direction if you were a paying customer, but they've indicated no interest.
And it had this passive agressive ring to it, with the hand clapping and the hurray in the beginning and the stone walling in effect.
...only to have someone come up and act like a paying customer whose expectations weren't being met. He answered suggestions by saying something that comes down to "I don't understand, can you repeat please", and never quite grasped that if he wants a better experience for his users, he also needs to work for it.
The introduction to the response, in typical douche manager style, was the cherry on top.
I took that emojis to be the exact opposite of the almost sarcastic tone I think you're interpreting it to have.
I guess even with emojis, it's hard not to make tone ambiguous.
...probably especially with emoji.
fwiw: "I would have a really hard time working for someone whose first inclination was always towards criticism over accommodation or compassion." But then I also acknowledge there may be a whole bunch of other stuff going on here behind the scenes. ;)
My biggest problem is that I can't imagine a time, even if I don't understand the problem, where I would say that I'm not interested in essentially not abusing a free system that can't handle my load. He basically said he had better things to spend his time or money on. That shows a motivation/attitude that would likely be there regardless of the issue or if they understood it. That's his knee jerk reaction, which I would say is probably his most honest, and it seems very selfish and not empathitic to the GH team at all.
And what the hell was with the quotes around "free"? Are you paying? No? Then there's no quotes about it.
> To be blunt, you're abusing the shit out of SOMEONE ELSE’S
> product that you're not even paying for.
I <3 GitHub, but for their own business/values/whatever reason they choose to host open source for free. It’s not like these people have found a loophole and are getting a paid service for free.AFAIC, that makes every free user a customer. They may not be a paying customer, but it's GitHub’s choice to be in the free hosting business.
From @mhagger's measured and thoughtful reply: "We understand that part of the CocoaPods workflow is that its end users (i.e., not just the people contributing to CocoaPods/Specs) fetch regularly from GitHub..."
I guess I’m saying that I don’t see CocoaPods as being a “bad actor” so much as the extreme tail of a distribution.
The one that CP's usage most likely confronts is G.12 - found here: (https://help.github.com/articles/github-terms-of-service/)
> If your bandwidth usage significantly exceeds the average bandwidth usage (as determined solely by GitHub) of other GitHub customers, we reserve the right to immediately disable your account or throttle your file hosting until you can reduce your bandwidth consumption.
But I think you may be reading his tone more negatively than necessary. What I see is him starting off by expressing gratitude and then switching voices to communicate very explicitly what the needs and desires of his stakeholders are. He's simply trying to reflect that as clearly as possible and discover the additional context. This was clearly effective, as you can see from the rest of the conversation that with all the information out there, everybody comes to a mutually acceptable consensus. Problem solved!
What we've ultimately got here is a free service built on a free service. CocoaPods has nothing but the time and effort of volunteers. GitHub has input of resources from the commercial side of their business. Both sides clearly want to preserve the utility of this end-to-end workflow in a more sustainable way.
"Falling over yourself" is a subjective amount of effort, but clearly CocoaPods has tried to minimize their impact on GitHub. As it turns out, the attempted optimization of shallow fetching backfired, but that's not from lack of regard for the resources they rely on. What was missing was exactly the context the Github employees provided.
Honestly, I think people are offended second-hand by a perceived lack of groveling on CocoaPods part, and to me, that's way overblown.
that being said, i do believe it could help cocoapod's use case since the fetches are done automatically (as i understand it)
It is also being used in scripts, etc. Since from the user's point of view it's a no-op if there are no updates, there is no reason not to do it on a schedule.
In the medium to long-term I'd like to consider Homebrew running `brew update` automatically before you `brew install` (https://github.com/Homebrew/homebrew/issues/38818). For us to ever be able to do that `brew update`'s no-op case needs to be extremely fast.
My comment isn't to pass judgement on your suggestion, but if you took a look at homebrew itself you'd be able to make better non-generic suggestions.
Re: your suggestion - It's a generically seemingly good thing to separate out fetch from install, but as a user of homebrew, it's not very applicable because when homebrewing things you're likely already connected to the internet, and it's hard to predict when you want to brew install something before hand. If you have the internet capacity to fetch something, it'd be just as easy to brew install it there on the spot.
By extension, it's important to run brew update before installing just to make sure the package index is up to date, so I agree with the dev above, integrating brew update step before brew install would be a good thing - except - perhaps print out on console the exact version number that's going to be installed. Current behaviour does put the version # in the file name of the package being installed, but it could be listed in a more obvious way.
Often times I do a brew info, find the version and details on it before brew install. If the installation step then installs a new version (because of the brew update step), then it's a bit strange that I didn't get the version I was intending to get.
I am guilty of the same, but in my case, I have an installation I can actually check. You can read the source code if you don't have a mac to work with, but the important thing is this already exists
from the brew man page:
brew(1)
fetch [--force] [-v] [--devel|--HEAD] [--deps] [--build-from-source|--force-bottle] formulae
Download the source packages for the given formulae. For tarballs, also print SHA-1 and SHA-256 checksums.
Is this classic "I have an opinion and all of my opinions are great, so I must pontificate!"?
Maybe you should just trust that the developers of the only successful package manager for OS X have some idea of what their users want and need... and that as someone who has NEVER used their software, and not a seasoned veteran of any sort of similar projects, your opinion counts WAY less than any of their actual users.
EDIT: the upside is that cocoapods will have to either rethink there architecture in order to eat less resources or move to their own paid infrastructure because their package manager will soon be less than functional given the aggressive rate limiting github is performing.
I'd like to see both happen:
* CocoaPods refactoring to be more efficient
* GitHub providing open source projects the option to buy reserved capacity if they're using excessive resources (versus just saying "No").
> GitHub providing open source projects
> the option to buy reserved capacity.
I have no affiliation with GitHub, but I'd guess that if you were paying for one of their $200/month organization plans[1] you'd be having a very different conversation with them about rate limiting.I'm just pointing out that the feature you're wishing exists very likely already exists in practice. Unless GitHub is stupid they aren't going to be complaining about you pegging 5 CPU cores for $200/month.
GitHub people are truly going above and beyond in service even when barely warranted. I'll give them that.