GitHub monoculture
nedbatchelder.com
nedbatchelder.com
I was curious, so I went to bitbucket.org to try to find coverage.py. Experience was awkward to say the least:
1) Bitbucket homepage does not have a search bar, nor is there a link from the homepage to the search [1]
2) Guessing that the URL is /search puts you in the team profile for account `search`. finally you see what looks like a search bar at top [2]
3) Searching for coverage.py doesn't show `ned / coverage.py` in the first page! There are lesser-known forks of coverage.py that show up higher than the original project. [3]
I'm sure these will improve over time, but discoverability is still incredibly difficult on bitbucket.
[1] http://i.imgur.com/soRXB98.png [2] http://i.imgur.com/z5YmcL2.png [3] http://i.imgur.com/uhg8rbc.png
Well it's more that they accepted reality (git and github had the network effect) and moved on.
The number of programers in the world is going to follow an exponential curve. By 2032 there will be more programmers than all the atoms in all the stars of the known universe.
pip home coverage
In my admittedly anecdotal experience, GitHub projects are way more likely to have a clear overview with a link to a GitHub Pages site with real docs, they're more likely to define the rules/procedures for sending in contributions, and they're more likely to have an active community of contributors. I know I'm not going to have to hunt for the source link in a paragraph somewhere, because if I know the repo name and author I can clone the source.
The way I'd host my projects would be:
1. I have my own server where I have a git repo, the issues repo, and maybe a wiki repo.
2. My server would host semi-statically generated project browsers; source code browsers, issue browsers, etc. Unlike when I was running Trac, I'd have no need to allow people access to my server, no need to worry about spammers, pages would be low-latency/cachable so that I can allow crawlers as well even on a very small linode host.
3. all of my wiki/issues/git would mirror out to github, bitbucket, and whoever else wants to participate, and updates on those sites mirror back. I already do this now with my git repos using WSGI services and hooks.
4. when someone wants to post/update an issue, they link out to one of those services, or even run a local command line/GUI based issue system (wouldn't that be great??), and everything synchronizes back.
The only missing link is a decent distributed issue system everyone can get behind. Building out the glue to sync between services is not that hard of a task.
This is definitely the biggest problem, there are a bunch of "distributed" trackers living inside the repository, but so far I've seen nothing which worked well (even ignoring the lack of integration with gh/bb/whatever).
There's the second issue of contribution workflow though, it's not really possible to have a dozen different places where contributions can be proposed, discussed, hashed out and possibly rejected; with the sub-problem of tools integration (e.g. integration bots or CI runs on contribution proposals)
I have seen file(s) in the repo. But branching a big makes little sense
I have seen git-notes, which seems good, but nothing survived across rebases.
There is a zombie StackOverflow thread on this but I would love a good solution -
Well that's the big nut to crack isn't it?
Worse is, when answering to the question what do you use for version control software, I hear quite often people saying "github" and not git, and not understanding the difference between the two.
Remember at work when Github goes down everyone is just sitting there twiddling their thumbs for hours. As in "yeah my commit is ready just waiting to push it" or "waiting for the issue to be close, then we can test".
It is ironic a bit isn't it. One of Git's major selling points was -- we are tired of one centralized version system, you shouldn't need a single point of failure or be tied to the network in order to develop. And then years later we have one giant global single point of failure across the world. There is some irony there.
(Yeah, yeah, I know I can commit locally. But at some point you'll need to update issues, and publish your changes. This is more a of a generalized point. It would be the same if bitbucket went down and so on. We never it seems have gotten to the mythical peer 2 peer internet of things -- it just doesn't work it seems).
The second part is solved. You just need a second git server. Maybe Bitbucket, or maybe your own. We just never do it though, for reasons I don't know either. I don't do it, even if I know how. There are multiple tools to push to different servers automatically, but we still go the comfortable route. Maybe it's the psychological thing that "if Github goes down it won't be for long, is it really going to affect me?".
Because of that I don't agree the tools are the problem though. Or even that they are part of some irony. Git delivers on its promise. People just don't use it like that.
The issues part or the dependency with Github to test/deploy software is more of a vendor lock in situation. A lot of people depend on it being up to work, but that's the convenience charge since Github isn't open source. Not much different from people depending on Heroku to be up for their application to function.
For instance, have a Github repo but also a personal Git server and Bitbucket. Whenever you do `git push`, it'd update all three. And all three would be in sync.
You can create a single remote called "all" that has all your servers URLs. Then you can just push to that one, like "git push all", and all your servers will update. Pretty much the simplest, you just need to edit the .git/config.
You can set up a mirror too ("git remote add foobar --mirror=push ..."), but I've never done that.
Or make a hook so that whenever you push to a remote it also pushes to another.
There's also mr[1], which I've heard of but not used.
There are ways aplenty, we just neglect to use them.
However... having multiple remotes is a very easy way to get multiple distributed backups of your code. (You should do a "git fsck" on all your mirrors at least semi-regularly. The reason has to do with how push and --mirror work.)
However, there is something to be said about mirroring canonical repositories for accessibility.
If github goes down and your devs are twiddling their thumbs because they can't push, you have larger problems.
Yeah, I have been using it for about 5 years. I know how git works
> Many popular projects push their code to multiple services
What does that mean "multiple services"? Multiple machines?
You can reference (and thus comment on) and close issues from your commit messages.
> and publish your changes.
That does not require sitting on your ass and doing nothing though.
But of course it is their fault for not having anything else to do and so someone on HN tells them that, "Well trust me this is totally fine, you guys and gals just need to do something else at the moment. Don't think of it as a bug, it is a feature built it to help you multitask..."
git remote add bitbucket git@bitbucket.org:username/reponame.git
git remote add github git@github.com:username/reponame.git
One could imagine a package you install on your droplet, ec2, linode, what-have-you, which would create a 'site' for your project which included a Wiki, a user management package with role management, automated source code backup to Glacier, automated mirror management to choice of mirrors. Install, set up your various keys (aws, ssh to mirrors, etc) and then push to it your source and make it the origin master.
You make that and then people might start creating free floating git repos, except that it will cost them some small money (like $25/month) which would discourage many people.
Something like Fossil's model seems better, where issue tracker and wiki are instead built into the repository. Sure, essentially you're still running a little mini web server on the local machine, but the presentation is completely different (a single binary rather than a huge Ruby webapp with many dependencies).
The problem isn't really that GitHub goes down, but that we can't mutate or make use of certain pieces of state while it is down (mostly issues/wiki), and nobody can see our changes until it comes back up.
Actually Git comes with all the tools necessary to send pure data in the form of changesets by mail, its just that its usability sucks unless you have a working sendmail/mutt config, which 99% of devs don't bother with any more.
That seems the more interesting thing to fix, rather than just having another self-hosted SourceForge clone or whatever
https://github.com/gollum/gollum / git@github.com:gollum/gollum.wiki.git
As such, you can clone wikis, work on them locally, and push changes later just like any other git repo.
Usability is the huge selling point of Github, so any replacement would need to be as slick
Google is not efficient at searching for code. If each creator spins up their own project sites using Gitlab or other tool, there needs to be some way of searching the network of Gitlab sites if we want users to actually switch.
There are a couple of other things about GitHub specifically, and the presumption in some quarters that everyone uses it, that put me off.
Firstly, they create their own world where lots of things aren't quite like normal Git. Some are simple terminology changes, like using "fork" instead of "clone". Some are more intrusive, like tying the pull request mechanism to the GitHub review tool.
Secondly, GitHub terms only let you have one unpaid account. If I want to maintain some personal repos, but also some with one of my professional hats on, it costs me money. If I want to keep my work on a major project separate to a few sidelines, it costs me money.
Thirdly, GitHub is an on-line service, and like anything else in the cloud, that means it comes with unknown security, privacy and reliability implications. For a big open source project, that might or might not be a problem, but I don't know why anyone would expect me to host my own projects there instead of, I don't know, in a Git repository on a server at home or at work, or something crazy like that.
So no, sorry, I won't fork you on GitHub. Nor will I contribute a patch via a pull request. Nor will I raise my helpful bug report on your project if you require me to submit it via GitHub. And I guess I'm lucky that I don't apply for jobs as an employee any more, because I am able to show approximately none of the best professional quality code I write without violating confidentiality agreements, and certainly none of it is in a GitHub repo for public inspection.
I am happy to support open source projects when I have a bit of time free, and I'm happy to demonstrate my credentials and abilities to prospective clients, but GitHub is a hoop that is rarely worth jumping through IME.
There is no pull request mechanism not tied to a site, unless you want to send an email with a public git URL yourself. Which you can still do, assuming the project you are working with has specified that. But why should you be able to dictate to other people that they do it this way? Github gives you the public URL, if you want it to, and lots of people do.
Hosting your own source repository also has unknown security, privacy and reliability implications. (Or known ones that are often not very good)
It doesn't really matter to Github users if you don't want to use Github. Doesn't make a lot of sense that you would decline to make bug reports just because the project used Github. That sounds like you are boycotting those projects for some moral reason.
Perhaps unintentionally, I think you have just made my first point better than anything I wrote before.
But why should you be able to dictate to other people that they do it this way?
I wouldn't presume to dictate anything to anyone. I'm just explaining why I find it frustrating that so many people now assume others will work their way, as if using GitHub is some sort of universal norm.
Hosting your own source repository also has unknown security, privacy and reliability implications.
So the cloud guys keep saying. And yet we keep hearing about downtime and security breaches in cloud services, while the servers and normal Git repos I use regularly all seem to work just fine. I guess the guys running all those projects must be some kind of sysadmin demigods or something.
It doesn't really matter to Github users if you don't want to use Github.
It does if they want me (or anyone with similar feelings; let's not make this personal) to contribute to their projects.
Doesn't make a lot of sense that you would decline to make bug reports just because the project used Github.
I don't. I decline to make bug reports when making the bug report requires the use of GitHub, which is probably a more dangerous trend that I have observed with increasing frequency in recent months.
That sounds like you are boycotting those projects for some moral reason.
I have no idea where you got that from. I'm not "boycotting" those projects at all. But since I can't create a free GitHub account just to contribute to them without affecting anything else I might do there, and since participating via GitHub is significantly more hassle than contributing to other projects I have supported in various ways, I choose to spend my available time where I can help other projects more efficiently.
What they do allow, though, is adding new "organizations". These are effectively groups that get top level URLs just like users (https://github.com/galvanix/ is one my business partner and I set up).
When you create a repo you can put it in your main account or in any of the organizations you are part of.
If your organization doesn't host any private repos then there is no cost.
Now, this isn't completely disconnected from your main account—you still do everything in that name, but it's close.
> …because I am able to show approximately none of the best professional quality code I write without violating confidentiality agreements, and certainly none of it is in a GitHub repo for public inspection.
Well, ain't that the truth for most of us! :-)
Since git has largely won the dvcs war and I'm a one man developer having to inter-operate with others it makes sense for me to suck it up and use git myself.
I really dislike git though.
As one example mentioned by the gp, the "detached heads" are a breeze: Just click the Lost Heads checkbox and they all show up in the visual log like any other commit. It was when I saw this that the concept of detached heads made sense to me.
Of course, when I've suggested SmartGit to people, a few have replied, "Oh no, I would never use a GUI for this, it's command line only for me."
To each his own, but I don't understand that. To me, source control is an inherently visual activity just like text editing, and a visual tool is a good match.
So for anyone who is willing to consider a visual tool, I highly recommend it.
I'm mostly ok now though since I rammed enough of git's weirdness into my head that I can work with it (that and phpstorm/pycharm have nice basic git support).
Which is kinda the problem we're discussing here. Git is far less usable without Github and yet all the workflow niceties that we've come to rely upon are locked up in Github's proprietary implementation. The more developers come to rely upon Github features, the more locked into Github we are and the more difficult it would be if Github were to go away or become obnoxious.
Of course, from an "all your eggs in one basket" or business monopolization standpoint, I agree with the author.
But selfishly speaking, I like Github and find that it adds more value to me the more it becomes a monoculture.
I was hoping for something like Diaspora (https://joindiaspora.com/) so we could use our choice of repo or self-hosting but still interact with a global community.
Bitbucket distinguishes itself from Github by making itself free for small teams only, but allowing you free private repositories. Seems like they are more interested in businesses and proprietary code.
That's likely true. However, it ignores whether or not those issues are outweighed by the substantial benefits to a de facto place that everyone goes, which are generally very large.
Github has produced a near monopoly on hosting for popular open source projects because it is so liked (as OP himself admits). Soon enough something better will come along that makes Github look like SourceForge and then we will all migrate our projects over there.
I use a mixture of bitbucket and github for different projects, not wanting all my eggs in one basket, really. I find them pretty much equal in many respects - except being able to have private repos on bitbucket is very convenient, and github pages is also very useful. I wish bitbucket had something similar.
https://bitbucket.org/dfairhead/streetsign-server in case you're interested.
Bitbucket is often slower than github, but this has not impact for my small projects. And I like to have some free repository that remain private.
There is inherently less cognitive overhead to use something you are already familiar with. Same reason why people wouldn't give Bing the time of day.
All things being equal, and you just had to choose a public place for a remote repo, Github should be your first choice unless you have a very good reason go use something else.
I'd rather not, nor expect consumers of my code, to have to learn a completely different UI than what they are used to in order to use or contribute to my public repo.
- I think it's healthy to always look at the different tools in the space to witness first hand if you're missing out on something
- There's some product integration in the works, so, it only makes sense to get curious about how existing users of the platform may feel once it is there.
So far it's going well, but it's early days.
Github is interesting is that the code is shown front-and-center whenever you visit a project page. Sure there is the Readme.md and wiki stuff you can do, but it makes browsing code and project discovery very easy. Github also made project creation and management super easy and lightweight. The number of questions you need to answer to setup a repo is so minimal. It gets out of your way when setting up a project.
I still feel like Google Code has some things going for it. The biggest being it's defect tracker compared to Github.
If the project sounds interesting I download it and look at the code locally. The few times I look at the code online is in response to like a stackoverflow question if I don't have the code locally already and I want to look up something.
But that's just me. It's not that big a deal for me to scroll down. If everyone else goes to code more than the readme under it then fine
Unless, however, I really do want to view a file online (say because I want to check the API of some command I’m using and it is not documented otherwise) in which case I really appreciate that it is so easy to find. I really don’t understand why most other web frontends (Bitbucket, Gitweb, GitLab) make the commit log the first thing to see on the summary page. These mean nothing to me from an outside perspective.
a) is github monoculture is bad?
I really disagree with author's view on the emphasize on public github project in job interview process is misplaced. Yes, it will not eval how good you are at your previous job, but the problem is it is very very hard to find how good you are at previous job. Doing open source project at spare time is a positive sign of interest in programming. If in the interview process you said I don't have a github account, but I put all my repo at abc.com, I am pretty sure it will work, too.
And I don't care the monoculture if its service is the best in this field. Why use the second best if the best is possble? I would like to be an earlier adoptor if there are better service, even if there are currently no user.
b) how to disrupt github?
That I have no idea. What github does not provide but is really needed? I do not know. Would love to hear ideas.
There are certainly areas that github could use some improvement. One of the basic features that I see as currently missing is actual side-by-side Diffs like those available in Chromium Code Reviews[1]. This makes it so much easier to review code changes and thus improves the experience of collaboratively working on a project. The Github folks are smart people, so I'm sure they know this and yet years after launch this is still not available in Github while they continue adding things like improved map diffs [2] which while nice features on their own, it only serves to show that a good code diff experience is not as important to them now.
Secondly, I find Github search to be lacking in terms of fulfilling its potential. It seems we all agree that Github is the number one place most people put their public code on. This is great, we have one central public repository. And yet searching for, finding and discovering good code on Github is not as easy as it should be. I argue that it should be easy to find solutions to programming problems that others have already tackled. Imagine being able to type "compute sha256, c++" and seeing relevant code snippets right there. I am not saying this is an easy problem to solve, but I think it's definitely worthwhile one to try. Who knows, may be someone will build an awesome code search engine on top of github.
[1] - https://codereview.chromium.org/261573002/diff/60001/google_...
[2] - https://github.com/blog/1772-diffable-more-customizable-maps
[1]: https://chrome.google.com/webstore/detail/mnkacicafjlllhcedh...
Every industry that's been disrupted was filled with companies that were coasting or in decline. GitHub has a ways to go before it ends up there.
Remember, GitHub disrupted SourceForge because it wasn't improving. Stack Overflow disrupted Experts Exchange for similar reasons.
GitHub is the only place where you can easily read, search, and immediately get the commit history for a file in one place extremely quickly.
my first though was - but Ned is caught in the far more worrying search monoculture.
And my second thought was the other monocultures to fear more - antibiotics, wheat and soya...
In the end, contributing to someone else's code is hard - it requires not merely coding skills but agreements on standards, willingness to compromise, and willingness never to do so, it requires humility and determination. Anything that makes that easier I welcome - I suspect githubs biggest issue is not having people start repos, it's having > 1 person work in the same repo.
They can solve that, more than happy to live with that monoculture
They are a choice. The author even mentioned Bitbucket. The fact that Github has become the dominant player doesn't make it not a choice. There is no one preventing a competitor from emerging and working to compete with Github (the same way Git emerged and enabled Github to exist in the first place). Being frustrated that people default to Github doesn't mean the choice to use something else doesn't exist. It does, most people have just decided not to.
Setting up a GitHub profile and posting a few quips on Twitter is almost effortless. So that's what we use. If someone comes a long and makes a competing product that is just as easy to use, people will switch. It won't be much of a problem.
If whoever comes along with a competing product runs it in a centralized manner like GitHub, then they'll create the same issue the OP has with GitHub. If they offer a decentralized, easy to use solution, they need to also offer a way for the person looking for a project to search the decentralized net. Without a unified search, nobody is going to oust GitHub from their spot.
This is a technical problem.
There is no way to efficiently look for code across the various repository sites. Too often a search on google turns up nothing, while searching on github (or pypi or another repository of software libraries) finds exactly what you are looking for.
Until there is an effective way to search for code, people will be left no other choice than to search on a few sites (github is by far the largest, so most people will check there, and some may not check anywhere else!).
To the author: if this bothers you, fix it, but don't act like it's a behavioral issue. (Also, bitbucket has been dead for like 5 years, it's a backwater and it's not going to come back.)
I want to find code based on "it's simple, you know my kind of simple, like made it into the BSD manual simple, but I also need it to be in python or bash because I am going to need to back on it but not spend so long on it that it becomes full time (leaves out C++ and haskell)
Ok now it must work with rabbit MQ and preferably install via pipy and run with decent test coverage using py.test and
you get the idea.
Anyway no search system does that - what does that is newsletters and emails and recommendation and ... social interactions
There are a plethora of choices, and you can always use Google to find projects. However, it happens to be that GitHub is the best choice, it's a free choice, and there's really no reason NOT to use it other than some stupid prejudice that putting all your code in one place is a bad idea.
Keep backups, and of course make sure GitHub isn't the only place your code exists (since you're using Git, that's not the case). and I don't really see a problem here.
Your website demonstrates who you say you are. Github work demonstrates who you actually are.
I'd hire those people any day over somebody with numerous GitHub repos filled with quasi-maintained JavaScript libraries or numerous forks of code they didn't write.
There are other things than code to define who you are; and even if you're only look for code, that code need not live on Github.