Github and Engineyard part ways
engineyard.com
engineyard.com
My blog post is direct and to the point. There's honestly no mystery about this, I laid out all the details, possibly excepting one.
We designed our infrastructure offering 3 years ago. We knew that the vast majority of websites would produce nearly read-only file I/O. GFS's less than stellar write performance isn't a problem in that typical case.
Along comes Github, and they have an entirely different disk I/O profile to the rest of our customers. Github was built on a shared-something architecture because GFS made it quick and easy to get up and running, developing features, and attracting users. This is a good thing!
Github has been very vocal about their dislike of the GFS filesystem we use for shared filesystem access. GFS doesn't scale forever, and we've never suggested it does. It could scale far larger than it has at Github, but fizx hit the nail on the head: we weren't willing to do it for free, and Github was unwilling to pay our price.
We warned Github many, many moons ago that given their growth rate, in order to scale their application smoothly and inexpensively, a shared-nothing architecture was eventually going to be needed. We offered to help them with that architecture, as we saw Github running wonderfully atop a cloud service such as EC2.
Rather than do the re-architecture now, they've chosen instead to move to a vendor who can provide them a high performance, high availability, non-commodity, proprietary network file server infrastructure. I suspect it will work well for them, and we'll all enjoy a faster Github. :-)
From my perspective, I really want everyone to understand that there's no bad blood on the EY side, and hopefully none on the Github side. Business is business, decisions need to be made each and every day.
GitHub has been working on a re-architecture since our last major feature release, which was GitHub Issues in April.
And we can't wait to roll it out.
So, the new Github architecture is shared nothing?
This is a pet peeve of mine -- what does "shared nothing" even mean?
A database, a "NoSQL" key store, and memcached are all sharing something, and that something is a contention point.
Specifically in this context, how do you implement transactional SCM semantics without sharing something (such as a distributed lock system?).
The bad part about shared nothing is quite simple, it's damn hard to do, and harder to get right. One of the biggest problems is handling conflicting data between nodes. You either have to handle the conflict with your application, or your database needs to automatically choose the best copy (based on date etc).
Are we really to believe that EY would have facilitated GitHub's growth had the repos remained overwhelmingly Ruby?
Which had me honestly scratching my head. Then again, I prefer Mercurial and Python, so to each his own I suppose.
I'm also pleased that github is upgrading since I've noticed gradually decreasing performance for about a year now which prompted me to setup my own private repo server (but I missed the github front-end). Kudos to github for doing something about it.
It sounds odd Engine Yard hosts an entire for-profit company for free just in exchange for some complimentary accounts.
My imagination tells me GitHub is tired of being slow and Engine Yard couldn't do anything more to help them. Instead of the news being "GitHub Leaves Engine Yard Because It's Too Slow" the news is "Engine Yard Kicks GitHub To The Curb Because GitHub has Non-Ruby Repos."
I don't see either headline being positive towards Engine Yard.
Well, sort of. It's really a value trade-off. EY offers its customers something that would cost them $50 a month, so it's costing Github (customers) * $50 to offer that. They exchange the lost value for added value in hosting (i.e., they're "losing" a few thousand a month in free accounts for our customers, so we'll offer them free hosting to continue to offer that).
It's simple barter.
I dunno, it could go either way for me: "Engineyard drops Github for being too costly" vs "Github needs better hosting, so we're dropping Engineyard"
This is another failure of Freemium. GH is only able to provide such large repos because they get free infrastructure.
I can't really blame EY for bailing. The partnership is no longer equitable. GH is getting something for nothing. I shouldn't say nothing, it's something, but it's not enough.
It's just a conclusion that makes the most sense for both sides of the arrangement.
This really shows poorly on the guys at Github. It sounds like EY went out of their way to accommodate the growth rate (I have yet to hear the Github guys dispute that) and just because EY was asking to be paid for all the hard work, Github was unwilling to, for once, be paid...Github ditched them.
One of the things I appreciate about EngineYard, and the Ruby development world in general, is that they appreciate the benefit of helping to incubate open-source and small startup operations. It's about re-investing back into the community. They did this when they helped incubate a startup company with some great programming minds behind it, which they knew would assist in the rapid development of open-source Ruby libraries.
But it seems GitHub hasn't absorbed any of that mindset by their move to ditch their supporter as soon as they're in a position to show appreciation for the help by actually paying for the service they receive for once.
For all we know it could have been equally lopsided in the other direction: if EngineYard was sending them tons of traffic through free accounts, it could have proved costly in terms of support and negative effect on paying customers.
We of course don't know.
FWIW, I first heard about Engine Yard through GitHub, and I'm sure I'm not the only one.
After reading more into the thread and the issue being mainly with GFS, then perhaps it won't be that difficult to improve performance at Rackspace because they'll control the hardware and be able to use any file system they want.
Knowing why they can't run at EY clears up a lot of the questions. The scalability at EY is simply too limiting for GH, paid or not. It probably doesn't make sense for EY to install a new file system simply because GH wants it and GH probably can't afford, or doesn't want to afford the amount that it'd take for EY to move all their customers to a more scalable file system.
If you look beyond all the drama here, this really is a rational decision on both sides. It's a testament to the community here that two groups parting ways in such a dramatic way can both participate in this very comment thread together and with civility. It speaks volumes and I think everyone involved has handled themselves very well.
I can't imagine this is a pleasurable experience for either group, but when one party wants something the other is unwilling to provide and in this case, it's true of both EY (different file system) and GH (compensation for the change) it's rare for both to understand the other side's perspective and avoid irrational name calling and bickering. I'm glad this breakup hasn't gone down that route.
It seems like it's more about the fit between the companies, than any particular performance issues.
why? its fairly high quality, targeted advertising with a built-in demonstration.
If you think about it, Github is one of the few sites that actually directly uses the filesystem heavily. Everyone else hits scaling issues on the DB first.
The sad thing of all of this is it's not really a matter
of scaling, and it never has been. Our bottleneck has
always been the file system. GFS just... sucks. I'm sorry,
but I have to say it. Case in point, your graph. The first
rebuild I ran timed out because of GFS. The second one ran
fine, took maybe a minute to process, if that. GFS impacts
everything... gem build failures due to cloning... GFS.
Network graphs taking long time to build... GFS. Caching
jobs not completing... GFS. I think you see where I'm
going here. There's no plans to deploy the new code to the
live servers, and I think the reason is that we're afraid
it'll make GFS performance worse, not better. But on the
new servers where we don't have to fight GFS, it's
amazing.Anyway there seems to be plenty of airchair quarterback on this one. The real story is that we can't afford to host them for free anymore.
But throwing everything on github is a lot easier than using bzr, so I do that. There is no bzrhub, and I don't have commit-access to GNU Emacs.
They offered a great deal of value-added services, but we found that it would be cheaper to go with the second-most expensive option (Amazon EC2) while also hiring a qualified full-time sysadmin.
Whether this was a wise choice or not given our experience with EC2 is up for debate. But even if we had gone with Engineyard and had no problems at all it would be hard to justify a couple developer's salaries for it.
Various hosts offered greater degrees of managed services at various prices. None really offered the flexibility that EC2 offered, or offered that much over simply colocating our own servers. With less traffic or fewer servers I can see many of these hosts being a great deal.
There also were several places that gave us completely screwy quotes that I can imagine were only meant to trick CEOs who don't know any better. One example is a place that offered us the use of an HTTP load balancer for the "low, low price" of $1000 a month. A lot of places offered expensive managed services, but when talking to the people who would serve us it was clear they didn't know what they were talking about. We did not seriously consider these places.
To Engineyard's credit, they were only managed host who sounded like they had knowledge that would be really valuable to us. They also made clear that they would devote significant resources to getting our site working smoothly. We would have paid a premium for this service, but not the premium they asked.
EC2 offers an amazing amount of flexibility that we found highly valuable. A sysadmin and developer spent most of a day diagnosing an issue with a DB server that ended up being a flaky drive. In EC2, we would have simply killed the server, fired up a new instance, waited an hour for its replication to catch up, and returned it to the rotation.
To return to EC2 we would have to drastically reduce the load on our MySQL instances, since we found that EBS volumes had somewhat unpredictable and not that great performance. Reducing our reliance on MySQL is something we're doing anyways, since on its current course it would have been difficult and expensive to continue scaling up in any data center.
With EY, you get EC2 + several heavily experienced sysadmins who work every day on the exact stack that you're working with...they know it in and out like the back of their hand + the EY stack which has been battle-tested for several years now in all sorts of unique situations + performance that you couldn't replicate yourself without all that experience.
Again, not attacking your choice at all, I'm just saying that its hard to appreciate all that you're getting until you actually use the service and once you are, its easy to justify the costs.
I've also heard great things about the Porsche 911 turbo from everyone who drives one. We were expecting a very expensive quote, but we were all surprised by how high it actually was.
This seems... odd to me. It doesn't feel like the right boundary between businesses. From a hosting provider, I expect good, steady service, reboots, a root console, and that they'll fix anything that's on their end (hardware, for instance).
Patching ssh is development work, and is something I would expect to pay for to meet specific goals, but not something that comes as part of my hosting package. I mean, what if you are just cruising along, and don't need any deep hacking over a couple of months. Is your money being wasted? With actual developers, I could redeploy them to do other things. Can I do that with EY?
They seem like really good, sharp guys, but I don't quite get the business, I guess.
> if you hire a lawyer on retainer and don't ask enough questions?
At a certain point, yes.
We're working really hard to make Ruby on Rails succeed. This is just one of the many ways that we've pitched in to help it do so.
If, during that downtime, when the "Twitter's problems are Ruby on Rails" FUD was running high, would anyone believe that a Github scale failure wasn't a Ruby on Rails problem? We weren't willing to test it, so we solved the problem, i.e. we put our money where our mouth is -- That Ruby on Rails scales just fine, the bottlenecks are generally elsewhere.
I suppose what seems confusing to me is that there are a lot of things you could do to help your customers succeed, but many of them are fairly expensive - such as cutting your rates, or doing high end development work.
With more basic hosting, say EC2, I know that Amazon isn't going to do beans for me, so I'm on my own. I know exactly what the price does and doesn't include, and what I have to provide myself. EY seems fuzzier... it's almost like having an extra developer on staff in terms of talent, but it is and it isn't: you can't tell that person to go off and do something else.
Say I move one of my sites to EY; is your "whatever it takes" attitude going to include fixing up my ugly design/graphics work?:-)
If you operated a site that EngineYard saw as a valuable resource for the Ruby community, I'm sure they'd help you out in whatever way possible as well.
It's the Perl community who just seem to get it done. Amazon and BBC iPlayer have more traffic than most of us, I bet!
Um, that crown unquestionably goes to a big chunk of the python community right now. Reddit alone provides a steady stream of both extreme fawning over python and vitriol targeted at ruby, and your comment and its upvotes demonstrate that it's present here, as well.
It's also quite a colossal double standard to complain about Ruby-specific hosts considering the existence of GAE, all the django-targeting hosting companies http://www.google.com/search?q=django+hosting and all the companies providing language-specific services for PHP, Java and every other mainstream language.
What "extreme focus on the language"?
Simply mentioning "Ruby" in the context of a Ruby-only hosting company's decisions does not constitute "extreme focus" any more than mentioning "Python" or, more recently, "Java" when discussing the Google App Engine. It's completely uncontroversial to anyone who doesn't have an axe to grind.
Also, the services provided by the companies (or as you put it, "the products being made") are Ruby and Git hosting, with EY also maintaining several Ruby implementations. I'm not sure how you could say anything about EY's "products being made" without mentioning Ruby.
Edit Woops. Looks like I missed the key part of a sentence there. Sorry folks.
s/working directly with/delaying/
http://support.github.com/discussions/site/721-network-feed-...
"Provider" really sounds like they are talking about infrastructure
"However, in the end, the best arrangement for them was moving."
good luck
But, drop in "Ruby" and "Git" and see it as front page news on HN with a discussion going off on tangents.
Uhh... can't think of anything off the top of my head that I really need or want though... OH, wait, I know, how about Bespin integration?
I expect prices will rise soon as well. I'm not sure anyone has mentioned this yet, even though it seems inevitable. Still, I think GitHub is fantastic and I'm planning to increase my usage.
EDIT: N/M, found the answer.
Does this mean I can have a free GitHub account because I'm also a Rackspace customer :)