GitHub major issues - repos have "lost" commits and site is erroring
support.github.com
support.github.com
For private repositories gitolite is really awesome, but I think there should be a company about it. What I mean is that github or some other company should offer hosted gitolite instances, in the following form:
You pay something like price_of_small_linode_instace+price_of_service, to get:
A virtual machine with gitolite up and running, with a good web interface and online documentation. I think that the web interface part is the piece this company should not opensource. With this web interface you can create repositories and users, and modify the gitolite settings in general.
A backup service, in some other different VM and network, so that if something goes wrong with your gitolite VM you can ask a free (and automated) restore, possibly with hour granularity for the latest 24 hours, and less granularity for bigger periods and so forth.
Note that this gives the user virtually infinite repositories... if they don't upload movies to their instance, as there is a lot of space even in a small linode instance for git repositories.
This way this is a "pure product" as there is no need to repair or install manually virtual machines, and the company can focus into evolving the offering, developing a better version of gitolite, of the web interface, and so forth.
p.s. I mentioned linode but can well be ec2 or whatever is better suited for this stuff.
Simpler reaction: Github:FI is already priced at a rounding error for the kinds of companies from which most of the revenue in this space will originate.
1) We are so small that we don't even think twice and buy some private repos on GitHub. (i.e. no market there)
2) We are so big that we can't possibly even fathom the idea of having our code on someone else's server so GitHub.com is out of the question. Sadly GitHub:Fi is per/user and because we are really big that means we have lots of devs so it is too expensive in most cases. (i.e. enterprise sale yuck)
3) For medium sized companies that either have some devs (more then 5) or a ton of repos or large repos they want an alternative to github. Many are slapping up gitolite which while good is a far cry from github.
Like I mentioned I already have found companies that want this solution and they are using GitHaven. The real question is if this market is too small or should I just move on to another project, open source GitHaven and let everyone (who wants to) slap it up as a frontend to their existing gitolite installs?
You can try to avoid these schemes, but you have to do it and not go out of business.
The problem with Github:Fi is the value proposition is extremely weak when it comes to meeting enterprise needs.
- They have an issue tracker that doesn't support custom fields. This is critical for companies with 500+ developers and/or very diverse product lines.
- They have a wiki but you can get confluence for $12,000 for the entire company which is more feature rich. It has decent searching.
- They have blogging but this is of no value in the enterprise world. The majority of employees really don't have the time or care about blogging internally.
- They have Gist which offers no real value for the same reason why blogging isn't that big of a deal.
When it's all over, their only strong value proposition for the enterprise world is their intuitive push and pull interface. But the problem with this is they are competing directly with free solutions like:
- gitorious
- gitolite + internal web interface
- gerrit
I'm obviously speculating here, but based on how their pricing model changed:
old: 1000 per user (initial purchase) + 200 something a year per user subscription cost
new: 250 per user if less than 25 users and 200 per user if more than 25 users
I think they realized they over estimated their value proposition initially. If Github wants to seriously break into the enterprise world, they are either going to have to lower their per seat cost and/or strengthen their value proposition.
How many of these companies do you think have developers using git in the first place (As opposed to subversion or [wait while I assume a look of sufficient disgust] visual source safe)?
"Sorry guys, there was a Javascript error displaying an error when there was none. We've deployed a fix for this and everything should be good to go."
The GitHub guy saying it was a "JavaScript display error" is incorrect though. I was pulling a repo and getting an old version.
EDIT: I'm not getting the javaScript display error, but I'm still missing my commit, so I guess these errors are unrelated :/
Now, for some people, I can see how waiting is an easier solution. I personally just don't like it.
(of course, this is reffering to private repos. public repos are still king on github)
Shit happens, servers go down, that's why you also have a remote repo hosted on Linode, and X, and Y too.
They key problem, frankly, is that GitHub conflates two entirely unrelated things: a nice UI and social features, and a hosted version of your repo. I love the idea of outsourcing a nice UI and having cool social features, and /maybe/ to make those features work they need to have a mirror of my repository (I'm not convinced), but when people go to pull it the URL listed should be the actual upstream "I own the DNS on this and feel I can make this stable in the long term", not the GitHub mirror.
If you have a million people `pull`ing from your repo, of course you should have be hosting your own public access point. But, in 80% of cases, people can't be bothered to figure out how to set up Gitosis, pay for slices, mess with DNS, etc. just to host a repo.
To put it another way, see: Heroku vs. EC2
In this case, Github is very responsive about outages and clearly strives to eliminate or reduce them as much as possible.
You can't really fault Github for individual teams not opting to host their code in more than one spot online, even if Github doesn't offer the capability for users to use their own domain name for seamless switching of git hosts.
Does Github encourage keeping everything centered at Github? Perhaps implicitly. But they certainly don't lock anyone's data in, so blaming them for their customers opting to NOT put their code anywhere besides Github seems unfair.
However, people are now going to copy/paste the GitHub repository URL and use that to clone my repository, and that URL is going to end up as a large number of peoples' origins. Even worse, that URL may end up in third party projects as a submodule (which is much more difficult to retroactively change).
Again: the problem here is not that GitHub is somehow encouraging people to keep things at GitHub "centrally": it isn't, and the goal is not to have your data in multiple places.
In fact, that's what you need to /avoid/: there should be a single URL for "this is the git repository that we consider to be the official, canonical source for our (distributed) contributions to this project".
That URL should be one that you feel comfortable you can maintain for a long time, as that URL can end up baked into a lot of things. Some of them are theoretically easy to change (the million users who are pulling from that URL, assuming they know how to do that without just re-cloning), and some of them aren't (usages of your project as submodules in other peoples' projects).
To quickly put this in another, maybe simpler manner: the problem isn't that people aren't choosing to /also/ put their code in places other than GitHub, it is that putting you code /also/ in GitHub undermines your git repository URL.
ssh:// and git:// are more difficult, but project contributers with commit access can just ping you on irc to see what's up with the repo and where to push to today.
And my model for working on git projects is not distributed. We use a centralized model (which is totally viable and one of the many uses) instead of pull requests and branches. On small projects, I find this faster and easier for people to use.
Yes, I have backups, but when my centralized repo goes down, it's annoying to tell people to start pushing and pulling to a USB drive. Workflow is the issue, not losing work.
Github actually has a program where if you tell them you are a student/professor you can get free private repo's.
The main point of Github is the social aspect; git just makes that easier.
They were other public git hosts before, but they didn't get the traction of GitHub because they didn't offer the same magic as The Place to share code.
It's git hub for a reason.
I, for one, am a positively mediocre sysadmin. And the amount of time I have to spend waiting for GitHub to fix bugs is almost certainly far, far, FAR less than the amount of time I'd spend scrambling (but not waiting!) to fix my own.
This is also how I feel about Heroku.
For whatever reason, I prefer the helplessness I feel while waiting for them to fix stuff to the helplessness I feel when I'm in "OH SHIT MY SERVER IS DOWN AND I DIDN'T EVEN DO ANYTHING AND WHY IS THE DATABASE NOT RESPONDING IT RESPONDS WHEN I QUERY IT MANUALLY AAAUUGGHHH!!!" mode.
Although I do try to keep myself sharp by doing things by hand in other areas of my engineering activities.
But seriously, if you use git over ssh there is no sys admin work other than running apt-get upgrade
I've done this several times over the last 5 years and am now finally moving everything to specialized hosting services. Just hosting a web site, photo repo, mail and svn on a machine (vps) is enough to make it a serious hassle Moving all those to specialized services costs the same, is less work and more reliable.
You can get 5 years security updates support with Ubuntu Server LTS and you can set up safe unattended updates[1] with email notification if anything ever requires your attention. If you set your backups right as well, you can get away even with catastrophic hardware failures.
Remember, this is not some gigantic scale, it shouldn't be that difficult.
[1] https://help.ubuntu.com/10.04/serverguide/C/automatic-update...
Famous last words...
I'm not saying it can't be done, I'm just saying that my experience has been different. Another example, I used to have my own installation for our websites and bug tracking and time sheet management software, on a locally-hosted machine with apache and mysql and a bunch of other software. The number of hours I've spend migrating data between versions, tweaking mod_rewrite rules, setting up backups for various databases and other data, making usage statistics etc. - I don't even want to think about it.
Recently I moved to a shared hosting server. The yearly cost of that is covered with one hours of my hourly rate (and I'm not even expensive). I can set up 50 or so different applications with a few clicks in a web UI, and upgrades are handled by it, too. Backups are taken care of, I have a web ui for dns, ssl, everything. It was like a breath of fresh air.
To each his own and each situation is different, of course. But there's something to be said for division of labor.
A one-man operation doing sysadmin, development, everything? I'd be looking for hosted solutions too for my clients - it makes my life easier.
But as a company, with a development and operations staff? I'd be wary.
It seems (from prev. post as well) like you were trying to stay on the cutting edge functionality wise. OTOH I was arguing for running stable and maintained Linux distribution.
I'd certainly agree that the initial investment is pretty big—not only it takes some time, but you need non-trivial specific knowledge about Linux administration—and for that reason alone I'd advocate using maintained hosting. My only beef with your comment was that, in theory (and in my experience), a properly configured VPS running stable software shouldn't require nowhere near the level of maintenance you seem to be suggesting.
Actually your home desktop box should be much more difficult to upgrade than a simple generic server box with generic virtualized hardware drivers.
There are valid reasons to use service providers or maintained hosting but properly configured (and backed up) VPS running stable (& maintained) software can get you a long way.
1) get a distro with 5 years support.
2) when it's out of date, buy a new $20/month vps, install git and ssh. Move your repo and turn off the old server. This might take 2 hours, every 5 years. Not a big deal.
1) You need to keep current on your security patches.
2) You need to upgrade your OS when it reaches end-of-life.
3) You need to make backups, and more importantly, verify that those backups will actually restore. (For git, this is most critical if you have a large team and dozens of repos.)
You can ignore a server for years, but eventually you'll get compromised or loose a hard drive. (If you use RAID, you'll eventually lose or an entire RAID array. Not fun at all, nor cheap.)
(Part of me is wondering if many of the more controversial discussions on this site are between people who have sysadmin experience (and considered it valuable) and people who don't.)
I've done a fair bit of sysadmin work over the years, mostly in self-defense, because I want fewer crises when a critical development server eats itself.
Lots of people can figure out how to install Ubuntu, or rent a Linode. But if they don't master upgrades, patches and backups, they'll eventually end up paying a real sysadmin a lot of money at the worst possible moment.
point 3 isn't that necessary if all you're doing is git over ssh. All the people that have a checked out copy have a backup of your repo. Also, if you're only doing git over ssh then it's not hard to make the box relatively secure. ssh will be hte only open port, and it won't be on the standard port.
What gives you that idea? I run several cronjobs that deal with their service, and it's down a lot compared to other similar services. My jobs deal with that gracefully, but if I wanted to I could easily set up a more reliable service than GitHub if my only goal was to host *.git repositories, not have any of the other nice things they provide.
And the fact that they have somewhere north of 5 terabytes of active data, and their job queue is handling many hundreds of thousands of jobs a day, and it pretty much all happens extremely fast with nary a hiccup. I know there are outages, but I'm pushing up changes to them something like 10 hours a day every day, and I've never noticed them.
PS: We love github very much. its made it so easy to collaborate everyone.
My team uses dozens of private repos hundreds of times a day and everything's been smooth for the past few months, aside from the disruptions noted on their status blog.
Do I wait? Do I just move stuff to my own server? Go back to Gitorious?
It's not that I can't keep working, but the main reason I use Github is to make my OSS projects available. For other things I run my own server and my own git host. Having a project unavailable to people for days, when I'm trying to get bug fixes and such out, is not so good.
I believe the easiest way to build a successful business (software and/or service business anyway) is to respond promptly. Even if you don't have an answer yet. If you do, your customers will stick with you through almost anything.
edit: that makes no sense
http://www.anchor.com.au/blog/2009/09/github-designing-succe...
Now, Github probably wouldn't encourage total reliance on their service. Git is fundamentally distributed, so this doesn't have to be a problem. Still, the Github service is an integral part of many developers' workflow, and I do agree with the above commenter that I'm not sold on the idea of trusting remote servers with integral steps in my workflow.
For open-projects, it's a FANTASTIC platform... no doubt about it.
For a corporate closed project, I honestly would not use github - not because of any feature or lack of them, but because it's a hosted service. I want control over where my code sits, period - if only for the same reason as most hosted services - a subpoena or similar court order can be served against the provider with no notification to the owner of the data. This is the #1 reason one should be careful basing one's business on hosted services.
This is the only reason I decided to use Github after avoiding Sourceforge and their ilk for the past decade and a half.
But why github? Why not gitorious.org?
If github goes away, I still have my code, but I lose any issues people have filed on the site, and I lose being able to easily check on forks of my projects where people might be doing interesting stuff.
I had been using gitorius.org, but moved to github for all the things other than git that makes one public git host different from another.
It'd be nice of all those related items were also in a repo, easy to pull.
I retract the 'very' in my original parent comment, replacing it with 'usually'.
SVN and CVS are missing this key feature, which is why the sf.net hacking is scary.
I should be push a new release right from my browser.