Twitter Said To Be Abandoning Ruby on Rails
techcrunch.com
techcrunch.com
I've always thought RoR's tight coupling with traditional SQL would be a serious scaling issue. RDBMSs in the traditional sense just dont scale easily. Google figured this out a while ago (hence BigTable). And RoR reads and writes soo much that you are left with a burden on the DB that might have been better mitigated by doing more of the processing client side and persisting with a simplified cloud-based service rather than a single, central RDBMS.
Not to stir up a hornets nest, but I really think this is a limitation of RoR that should be addressed in the future. Abstract out the SQL so other methods of persistence are possible.
EDIT: I am not trolling RoR. I am simply proposing that it might be better to be able to use it with something other than traditional SQL for persistence of data, so that traditional RDBMS can be replaced with a simplified cloud-based service such as SimpleDB.
I speculate this might be possible now using Ruby -> JRuby -> JPA -> SimpleJPA (or any other MySQL/SimpleDB wrapper) -> SimpleDB. I dont want to even get into how much work that would be though. It needs to be easier I think.
Startup idea maybe?
edit: their best bet is to try to find someone who is burned out from working on wall street applications.
Which is exactly what they're doing:
Having said that, it would still be fun to rewrite Twitter in Erlang... :)
One of our Oracle databases sits on a 24 core itanium box
That sort of hardware + Oracle can munch through some serious work - perhaps that is the database solution, but I believe it doesn't come cheap!
The new rewrite will give the new architect time to vest, until it fails, like big rewrites usually do. If you're not sure how to fix things, the big rewrite is a perfect solution, really.
I don't see that as the problem.
Since their requirements seem like they're pretty static now, I would think writing a backend in C wouldn't be out of the question.
i am really surprised how facebook, so quikly found the skillset to build a 10,000 server site and how come, twitter didnt! just why ...
its the skillset, but again why was facebook successful in aquiring the skillset when twitter wasnt! ... why is twitter considering a rewrite and facebook didnt ... they just grew !!
In my experience, that's really the only way to do software. It's completely unrealistic to think you'll get it right the first time, and yet completely destructive to go and start from scratch. So you pick your worst problem, rewrite it, then hook it back into the working system without touching the rest. Then pick your next-worst problem, rewrite it, etc.
With their recent fundraising, they might even be able to run a couple parallel fixup projects in competition, and let the best win.
When I wonder why Facebook doesn't have problem scaling their chat? Oh, erlang + flat files.
I can't wait for Twitter to move to Java or PHP so people finally STFU about Rails.
Twitter moving away from rails doens't make rails look bad, it makes everyone at Twitter look like the biggest jackasses in history.
"We can't get this wrench to hit this nail hard enough, so lets buy another brand of wrench instead of a hammer and see if that works."
All tools tend to look like hammers when all problems are seen as a subset of nails.
A quick Google of "erlang facebook chat flat file" returns your initial post, would love to read more about that if have anything.
http://twitter.com/ev/statuses/801530348
(Ironically, Twitter was down when I first tried to get the URL for this.)
(With an HTTP Status header being dumped to the screen instead of, you know, sent as a header.)
I hate it when non-programmers write about programming! It's a giant festival of stupid.
But I don't think it's ridiculous to assume that the stability and operational functionality of a website is primarily dependent on the web framework the website uses. The data center is probably the second to blame, but I don't think that Twitter's problems are related to continual issues with their hosting (though i'm sure some are). I doubt the blame lies on Debian, or whatever their using.
Handing a sniper rifle to someone does not make them a sniper. Snipers have personal qualities and training that make them what they are; their weapons are just tools that help them realize their inherent abilities as well as possible.
Why are you and other TechCrunch people so eager to assign blame when you don't actually know anything about what you're assigning blame for?
Groping an elephant seems like a bad idea even for the sighted.
Elephants are a dear and treasured species. This continual assault by blind people and various other gropers have too often left our Loxodonta Africanan friends cowering in terror.
As for RoR, I'll simply quote a good friend of mine. "RoR rocks! Scaling isn't that big a deal. Took us over six months to get the stack just right for a recent customer."
Ugh.
My previous company (a web testing company) did a load test for a large tax return company. When testing started, their app could support 3 simultaneous users. When the bottlenecks were removed, one by one, the same app (framework, codebase, language and all) supported north of 10,000 users. As you test, there is usually something like a database lock problem, high cpu from a loop that needs to be optimized, bad query/data layout, memory usage maxing out, or something you are running out of, one way or the other.
The people who point out that twitter is not a standard web app and might be better served by messaging have a point also.
This is a good point. But, in the interests of balance, the twitter situation could also be compared to arming a sniper with a broadsword and then being surprised he fails. The choice of analogy makes a surprising difference in how you think about the situation.
"Why are you and other TechCrunch people so eager to assign blame when you don't actually know anything about what you're assigning blame for?"
I would rephrase the question to "what do you and the other tech crunch folks know that gives you confidence in your statements about decisions at twitter".
Point being no one knows what the folks at tech crunch know (or not).
It doesn't matter what he says. All that matters is that you notice.
After 30 minutes, you are the #1 response to the #2 thread.
So people click through.
Cha-ching.
You could probably rewrite the front end in applescript and you'd see little difference, and if you did just throw more servers at it. The problem is probably that theres a DB at the heart of all the message crazyness, you can't do that and expect to scale.
Surely they are just rewriting the slower core areas of the API in something faster/more distributed. There is no reason they'd have to rewrite the entire thing from scratch.
"This is exactly what makes Rails a ghetto. A bunch of half-trained former PHP morons who never bother to sit down and really learn the computer science they were too good to study in college. [...]
After this, I started thinking, since it was the first realization that I picked the absolute last wrong community to make my living in. They were all pricks, morons, assholes, and arrogant fucks who didn’t care about the art or the craft.
They were all a bunch of little ghetto fabulous princesses trying to make it in this tiny little Rails world and not enough brains between them all to make it happen."
and about the framework
"Now, DHH tells me that he’s got 400 restarts a mother fucking day. That’s 1 restart about ever 4 minutes bitches. These restarts went away after I exposed bugs in the GC and Threads which Mentalguy fixed with fastthread (like a Ninja, Mentalguy is awesome).
If anyone had known Rails was that unstable they would have laughed in his face. Think about it further, this means that the creator of Rails in his flagship products could not keep them running for longer than 4 minutes on average."
Although he does add that
" [...] Rails is a pretty nice idea, and it demolished the Java world I hated so much."
For Twitter, I think some parts of the API need to be in C. But the whole service? I doubt it.
They say most of their traffic is API requests, and I would expect that they would want to optimize that porition of their system. Rails is composed of many components, many of which you don't need for a REST service that talks json/xml. Blaine and other twitters have talked about their experimentments: other app servers (Thin), queueing systems (before building starling), ...
Sounds to me like Mr Arrington has an axe to grind and is putting that ahead of objective journalism.
Daniel
Has ror become a cult or something? It's just a language, and if it doesn't fit well with the application, best to move on and try something else.
Anyway, if the problem happens to be that the Rails frontend is too slow, speeding that up isn't going to buy them anything long-term. Eventually the database is going to bog them down even if the web GUI is hand-optimized assembly.
JMHO, obviously I have no idea what the twitter codebase looks like or what specific problems they are having. I also don't use Ruby or Rails, but I doubt that's their problem.
What they should do instead is to slowly rework their code, instead of doing a massive rewrite.
The reason you'd ditch Java or PHP is because they are lacking a lot of higher-order constructs. (PHP epsecially. Statically-scoped variables? First class functions? An object system that isn't a hilarious joke? Unicode support? PHP doesn't have 'em.)