Twitter jilts Ruby for Scala | Hacker News Reader

Twitter jilts Ruby for Scala

theregister.co.uk

85 pointsastrec78 comments

The very title of this article shows the religious nature of much of the hype around dynamic languages. Why is using the right tool for the job "jilting" another language which was never designed or meant to be suitable for the same job?

I am curious though whether they tried running their Ruby code using JRuby ... if the bottlenecks were (as stated in the article) in the ruby VM they might have got the same result just by switching VMs (pure speculation).

And ...

> "The biggest selling point for certain organizations is that Scala is 100 per cent compatible with your existing JVM code.

No, it's not. There are problems with quite important features like annotations etc. which are now very important in the java EE space but Scala cannot do them. This was one reason I've used Groovy so far in a lot of situations.

Having said that - I'm seriously looking at Scala because groovy has plenty of faults: the complete lack of any static type checking at compile time and very poor IDE support and error reporting all weigh down on the productivity benefits to the point where it's sometimes a net negative. Scala could end up being the best of all worlds.

>Why is using the right tool for the job "jilting" another language which was never designed or meant to be suitable for the same job?

That would be a question for the journalist who made it up ;) I am sure the answer was "cause it sounded cool !".

that's pretty much how "the Reg" operates.

Without regard to the tone of the article though, Scala and Lift are worth spending some time with.

Lift still has a ways to go in terms of providing a turnkey stack in the same vein as a Rails or Merb stack. If you thought putting together a Ruby dev environment had a lot of pieces, get ready for Lift...not so much Lift's fault as its the nature of the heavyweight JVM toolset world.

I love scala, as a language, and love rails, as a framework which doesn't make me thing to much - I can see how twitter would like the combo too. Even with the toolchain annoyances I think scala is pretty productive, at least it is for me.

What doesn't work with annotations in Scala? The documentation looks like Scala supports them. http://www.scala-lang.org/node/106

Scala doesn't support nested annotations. That will most likely change with 2.8.0 when Scala gets optional and default arguments (nested annotations being a convenient side-effect).

That said, most real-world uses of nested annotations in Java can also be configured with XML (I know, yuck). Or you can always write a wrapper interface in Java and use that from Scala, since Scala supports mixed compilation projects. Needing nested annotations in Scala won't be a show-stopper, just inconvenient.

I think anyone who has any depth of experience with Ruby can attest to the flaws it has data-intensive tasks. Thats why many of us just drop down into another language when we need to get something done efficiently. Ruby's clarity and conciseness make a lot of sense when writing views, controllers, and models, but don't have the same effectiveness in a background process that needs to be written and forgotten about.

Unless there are some major changes in Ruby (MORE than 1.9), I believe that it will be relegated to simple web applications, and never really be good at processing tons of data.

And honestly, isn't Ruby really just about prototyping? I admit, its great for startups who have nil for cash that need to get something written QUICK (to get money from investors), but once you cross that threshold, why not write it over with a language designed for efficiency?

I'm not a ruby dev, but could you be more specific? Assuming you are refering to syntax, not speed: how is it good for prototyping but not "data-intensive tasks"

One example: Ruby has slow IO and text-processing capabilities. I sometimes write pipeline tasks in Ruby or Perl. i.e. while line = STDIN.gets; puts transform(line); end. Typical disk bandwidth is 2MB/s in Ruby, and 25-120MB/s in Perl, Java, Awk, or C.

Beyond that, a lot of the metaprogramming features that make Ruby cool are also expensive when done repetitively.

Also, Ruby's threads REALLY suck. And it has impossible to track down memory leaks in the interpreter. Ask mojombo about fun with those.

All of the Ruby runtimes are unreliable (buggy) and very slow. Ruby syntax is perfectly good for "data-intensive tasks", but the current runtimes are not.

How does JRuby's implementation compare to Scala's, though? I've been pleased overall with Scala, but I've not had much JRuby experience, so I'm curious.

JRuby's a pretty solid implementation. It's generally faster than MRI 1.8.x, and has the benefit of quality threading. I think Scala is probably ahead, because Scala has the benefit of being-more Java-like than JRuby, and is therefore easier to implement.

For my serious apps, I typically write the front end in JRuby on Rails, and migrate code back into Java as it stabilizes and/or needs more performance. Works great for me!

I'm doing this at my current job as well. Jruby is great!

Data-intensive tasks that you do every day, or every hour, or every minute, require efficient execution above all else.

Prototyping tasks require efficient programing (as in, the act of putting the program together) above all else.

Ruby excels at the latter, but not (yet) at the former.

This really hit home for us when we started messing with TweetRex. It was my first project in Ruby and things seemed to go well -- the language has nice syntax, good selection of libraries, the prototype was done fairly quickly...

...and then I had to rewrite 80% of it in C++ because of the horrible threading, slow string manipulation and high memory usage.

It was still an interesting experiment, and I intend to continue to use Ruby for stuff where performance doesn't really matter, but I was surprised to learn so vividly that the Ruby bashing that I'd assumed was oversold was in fact pretty much spot on.

Seems like a perfectly reasonable approach. They still are going to continue using Ruby on the front end of the site. It's just the backend message queuing, and data operational stuff that they're moving to Scala.

I probably would have just used JRuby for this stuff, but Scala seems fine to me. And if it works for them, thats great. But I hardly see this a "jilt" against Ruby.

JRuby is what, a few times faster than Ruby 1.8.x? In a situation where Ruby was too slow, even after optimization by experienced Ruby coders, I don't see how switching to JRuby would have helped. Is there a crucial performance difference (threading maybe, or libraries) that would have made JRuby scale better than Ruby in this case?

I use JRuby to run a web application that I first coded in Ruby and it works pretty well even though it reserves and consumes more memory. Having a dedicated server for JRuby applications instead of running such applications on your own desktop alongside other applications slightly helps with the feeling of speed of the JRuby application, as it does not compete for CPU time with the other desktop applications you have like the browser, when you can feel that JRuby requires more work than Ruby for doing similar workloads. :-)

I know that the pros of JRuby for running Ruby applications mean things like being able to run multiple instances of Ruby from the same process, being able to run JRuby applications in a standard Java web-server be it as Servlet or a more high-level installation, and so on.

Also, Java has had a lot experience in running server applications so Java does help with maximizing the use of the resources available, be it 100% of CPU time, gigs of memory, multiple CPUs, and so on. Java has those famous garbage collectors and HotSpot technologies. JRuby can enjoy all of that so it does improve things over pure Ruby. Let's say, Java doesn't break a sweat under heavy load, or at least it shouldn't. Whereas with pure Ruby all bets are off.

"And he's pitching his new Scala book. And his new Scala book is published by O'Reilly. And O'Reilly runs the Web 2.0 Expo. ®"

Timing is priceless.

So what? He likes Scala. Likes it enough to write a book in his spare time, of which he has precious little since he's, oh yeah, a full-time developer at one of the biggest, fastest growing sites on the face of God's green earth.

So it's a (mild) conflict-of-interest (really, it's just a promotional opportunity). Doesn't make the arguments any less valid or his book any less worth picking up. Nice thing about code is that it's like science - every claim can be tested ad infinitum.

And if the Scala reworking goes poorly, he presumably suffers, too.

One idle thought: It'd be hard to eliminate the factor that they might be a better Scala dev team than a Ruby dev team. That's still, of course, a perfectly good reason to go to Scala.

This quote:

"The trouble, he insisted, is that the so-called Web 2.0 languages aren't always as efficient as they need to be, especially in an economic environment threatening to bring Web 2.0 to its knees."

Is particularly dire. Sure, so if we all switch to C or machine code we'll be able to skip through the recession, right? Seriously, less drama and more reporting please.

Carpenter jilts saw for hammer

[EDIT: dear downmodders, the intention here is that languages are tools, and you pick the one which fits your current needs best. What's good for Twitter isn't necessarily best for another app, and Scala <=> Ruby is meaningless]

HN is getting lamer than ever with this downmodding of anybody who doesn't conform. It would be interesting to know the overlap of opinion-based down-modders and Reddit/Digg bashers.

"Investors now want to know that you're not going to be paying tons and tons of money for servers because you decided to build in a stack that isn't as efficient as possible," he said.

I've never used Scala and I'm a Ruby dev but I'm still not sure I totally agree with that. Scala is probably great and very efficient but you probably don't need to be thinking at Twitter's level out of the gate.

It's also the case that headcount is far more expensive than servers.

A really talented sysadmin with good tools may be able to manage 100 servers; a great developer can optimize your algorithms to get an order-of-magnitude reduction in the number of servers you need.

Eventually, more servers == more headcount.

No other implication about Scala intended (Alex is very credible), but optimizing for lower hardware footprint at the expense of your hiring pool (by choosing a very niche language) seems like a bad trade.

Won't every new developer who works on this codebase lose a week to two weeks becoming facile in Scala? Assume a 6 person dev team, and a conservative fully loaded $180k for a valley dev, and that's $43,000 out the window.

How big is the codebase for twitter? Are you limited by devs or production resources?

You can also swap in Java components when you need to so, depending on the architecture, you may only need a small Scala team. Java devs are everywhere. You also win by acquiring easy access to Java libraries.

Not saying Scala is the right choice... but there are big advantages besides "Scala is lighter than Ruby" -- and Scala is a lot lighter.

But their move to Scala meant the Fail Whale went away and their investors were happy enough to sink many more millions into the company. Would it be possible to run a high-volume messaging system in Ruby? Perhaps. But why fuck around when the people bankrolling your company are getting angry?

Hey you know what? Not making a case for Ruby here. Just picking one very small part of Alex's argument and critiquing it.

On the other hand, using an elegant, smooth, well-designed language like Scala is a perk that helps you hire excellent devs for less money. Plus, if you want to transition a bunch of experienced Ruby devs to a faster language without losing most of them, you'd better not pick Java or C++.

That's kind of a straw-man argument, isn't it? Nobody is moving from high-level languages back to C or even Java. The talk here is of moving from "early adopter" to "innovator" on the adoption curve. And I'm just making the point that when you do that, you necessarily either contract your candidate pool, or add ramp-up overhead to new hires.

I'm defending their choice to move to Scala (a "niche" language) after running into performance problems with Ruby. I don't think it's a straw man to mention Java and C++ as representative of the non-niche languages that perform significantly better than Ruby. There are plenty of faster language platforms they can move to, but which ones offer a powerful and elegant language? As far as I know, you either pick a "niche" language that few developers know or you pick an old and clunky language like Java or C++. In this case, you have to weigh the ability to hire new people, which would be better for the more popular languages, against the ability to retain your existing Ruby developers, which is better with an elegant, powerful language.

Once upon a time, people made the same comments about Ruby and Rails: "it's a fringe language", "the tools are immature", "you shouldn't use it because you can't find Ruby coders", etc., etc.

I find it honestly hilarious to hear it suggested that one of the early high-volume Rails shops should stick with "safe" language choices because they won't be able to hire enough developers to work with a new language.

I've read through the sources to Kestrel, and found them quite easy to grok. If anything, most real-world Scala code seems to be essentially very clean Java, with much of the syntactic cruft (redundant type declarations, inner classes, 'public static final' declarations, etc.) removed. Any experienced, effective developer is going to spend far more time getting to know your in-house APIs and coding style than they are the language.

Again: not an anti-Scala person, just turning the economics of this around in my head. That said:

Ruby is our house language. We've never hired anyone that came in the door knowing Ruby. We have a very high bar for hiring (typical first project: "write a programmable symbolic debugger for closed-source binaries on platform Y"). It takes people at least a week to become facile with Ruby --- a famously easy language to learn. Reading isn't writing.

If you have a house style, easily figure another week to pick up the right idioms, so that your code survives in the repository and isn't just refactored out.

It might cut down on the volumes of resumes, but once culled down to actual potentials, I think the effect of the "niche language" is minimal. I spend my time hopping between a bunch of languages and I'm equally mediocre in all of them, but my friend who does the same is amazing in whatever language is thrown at him. I have yet to meet a really excellent programmer that can only program in one language (they might not be equally proficient in all of them).

When hiring for the small percentage of excellent programmers, I don't see language as a barrier. Sure, there will be spin up time, but you'll have that regardless. No one is experienced with every library, framework, or configuration. Adding in a new syntax might compound the issue, but I think that there is the definite potential for long-term productivity gains once the team is up to speed.

Still, I worry that despite Ruby's niceness, a lot of time is spent developing around it's issues. Whenever I read about deploying Ruby, it just sounds scary. Memory munching Mongrels that have to be restarted on a regular basis? It sounds like a science in itself, whereas tomcat just runs and runs and runs. Less headaches == less headcount.

Thankfully these issues have been completely solved by passenger/mod_rails. Deploying Rails apps is now trivial and stability (at least outside of extreme loads) isn't an issue.

If you know and like Tomcat, may I suggest you have a look at JRuby deployment for Rails apps? With recent versions of jruby-rack and the warbler gem, it's as simple as listing your dependencies in a config file, and then running 'rake war' from the top level of a Rails app to get a shiny new WAR file ready to drop into your Tomcat deployment directory.

I think that users care about speed as much as anything, and being fast cheaply is a competitive advantage.

Having 2 RoR products in the time it takes you to build 1 Scala product is the true competitive advantage.

Please read the original article. Nowhere in it (or the original talk) does Alex advocate using Scala instead of Rails for the things that Rails does well -- namely, interactive web page rendering.

Within that niche, Rails does a fine job. For infrastructure and middleware code, however, Rails has little to offer other than the Ruby language, which has serious issues when applied to time + memory-intensive problem domains.

I'm a huge fan of Ruby, and a (perhaps somewhat begrudging) user of Rails, but that doesn't mean I think they're the right solution to every problem.

"The trouble, he insisted, is that the so-called Web 2.0 languages aren't always as efficient as they need to be, especially in an economic environment threatening to bring Web 2.0 to its knees."

I'm saying that is false, over-optimization is what brings startups to their knees.

Why is it "over-optimization" to use language with a genuinely well-implemented runtime?

If there's anything I've noticed over the years when building websites, it's that customers care a ton about speed. I'd argue that it's the most important feature, followed by design. The more responsive your interface, the better. Now, you shouldn't sacrifice key features to speed things up, and you shouldn't optimize your code from the get-go, but your product had better be fast.

Google won because of good search results, but I remember that the first thing about the search engine that impressed me when I first tried it in 2000 was its responsiveness.

Speed is very important. However: how much time does server spend running your Ruby code? I think I saw somewhere it is about 5% of the request time. There is DB, network latency, all the client side cruft (twitter did not get even assets caching right yet). Article says they are leaving Ruby in place for the front end, so most likely they do understand where bottleneck is, and where it has nothing to do with a language.

I just hope that other won't jump to porting their code without understanding the reasons. It should be pretty embarrassing to spend six month porting the codebase just to find out that reducing time server spends in your app code from 5% to 1% did not impact the overall speed of your site.

It would be interesting to see more details about those queue processing problems they have though.

But we've already been over this: It's not Ruby, it's Rails that's slow.

http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...

Scala is 900x faster than Ruby for some workloads. Ruby is not a fast language. It works for web stuff because every language waits for IO as quickly as the others.

But Twitter isn't a normal website and Ruby was the reason why it kept failing. That doesn't mean your website will fail because of Ruby, but Twitter had too much background messaging work for Ruby to keep up with. The fact that simply replacing the code with Scala made huge improvements should demonstrate that fact. They didn't change what was being done, just what language was doing it.

/agree the bottleneck for any website is ALWAYS the database, never the language. If you have requests that are bottlenecking at the language, nothing is going to solve that.

Some of us have business models which do not require traffic the size of nation states and, accordingly, aren't exactly that intensive on the computational resources.

My side business runs off of a $20 a month VPS. My day job routinely charges millions for software that runs on one to three commodity boxes.

You'll forgive me if I don't consider "Has difficulty running Twitter" to be a knock against a language. I don't, after all, have to run Twitter.

And in case you're wondering.. no, this isn't an April Fools joke.

In this case, it'd've been a joke cued up several months ago, when this story first broke.

I think it's important to remember that Twitter had this problem because they were ULTRA successful to the degree of being a household name, a new cultural lingo, a game changing tech of enormous value, etc.

Yes, different languages and techs for different advantages and purposes.

"Just" developing a web app with models and views and controllers is enough to create the user experience smash hits are made off. Scaling issues are big, but certainly fall in the class of problems we want to have at some point.

> Scaling issues are big, but certainly fall in the class of problems we want to have at some point.

Well actually with a big of good design from start you may avoid this problem from the start with little efforts. You'll have to scale anyway but it will be a matter of buying more hardware and not redesigning the system.

Although the Reg sensationalizes the story, it's interesting to see how Ruby is in limbo. How much damage has been done to Merb by the 1.8 to 1.9 fiasco?

I agree that the Ruby 1.8 to 1.9 transition has caused headaches. But the 1.9 version does have some interesting features that have been long being cooked to get to this point, and perhaps more interesting is that the 1.9 version has motivated even more developments in Ruby-land as found in MacRuby that uses the 1.9 version or in yarv2llvm that recompiles the bytecode to llvm or something.

Let's say that many folks might have had higher expectations that have been hard to meet given the reality of it all.

The Merb guys helped with further explorations themselves which probably helped with making Rails advance more decisively, given the promissed "merging" of the Merb ideals with Rails.

Also, with things like Passenger, the deployment of Ruby met an all-time high in features.

I don't understand a lot this approach. Basically the front-end is almost always not the problem, but the Db / backend is. And this should be written in C probably. If you write a queue system in Ruby and it's slow where is the news?

The article title is misleading since Ruby continues to be used for its core business that is the front end.

Sheesh, from the comments you'd think The Register is known for level-headed journalism.

It is a snarky, piss-taking, grease-trap that mixes news with commentary with little to no discretion or fear of violating "journalistic integrity." Once you understand that, it becomes a great source of news and entertainment.

Once you understand that, it becomes a great source of news and entertainment.

Or rather, a great site to ignore.

Following Twitter's issue blog, most of their problems appear database-related. I wonder whether a shift to Scala would have any real effect.

Threading.

Hmm, I could see that. I just wonder how many of their problems actually come down to thread issues.

May just have to wait and see whether their new "beautiful code" does any better. :)

it seems that the article ignores the trade off in cost between efficient code and development time.

Scala tries to be pretty good at both. Java like speeds, with type-inferencing, first class functions, and a more flexible syntax to improve programming productivity.

Maybe the article ignores that trade off, but Twitter is not really a "startup" anymore, so they are correctly leaving the startup mindset behind and beginning to analyze where they are wasting resources and what they can do about it.

If the expected total cost of rewriting and maintaining the codebase (over and above the cost of maintaining the current code, of course) to get an N% efficiency increase is less than the amount you will save on hardware, then it's worthwhile to do.

For a relatively simple application, which Twitter is, I'm unsurprised that a re-write/port is a fairly low cost ordeal, and might provide enough savings over time to be worthwhile.

I also take issue with the implicit claim that RoR is automatically more productive than Scala + Lift - Lift might be a little younger than Rails, true, but it's still a pretty solid piece of work, especially if you don't have to beat against the edges too much. And if you put some experienced Scala people up against similarly skilled Ruby folk in a "code race," I'd expect it to be a pretty fair match. This is not Java we're talking about, Scala can be extremely powerful in the hands of someone that knows what they're doing, and I imagine the Twitter guys have enough of those people that they'll do just fine.

That is so funny to choice JVM-based stuff is case when you need an accept/select/read/write hardcore (message passing) along with queue processing. Of course, you can do networking from JVM, and it works well in clean cisco-branded qos-enabled networks. =)

If one had had an experience of using java-based network tools over cellular or wireless networks, one knows.

At the same time, Erlang is around for a decade.

Unfounded claims, no research, nothing. After years of suffering websites running on JVM web stacks, I am very sceptic of this Scala for the web rant. Plus he extrapolates Ruby to Python and PHP just because. The JVMs (there are several) mostly trade-off memory for speed. Go check the Language Shootout page. It doesn't scale in my book.

So unless Scala has some secret ingredient to use the JVM more efficiently, this looks to me as yet another fanboy rant.

First of all, I don't think it's fair to accuse the Twitter dev team of working without real experience and numbers to back up their decisions. They have a huge volume of data being generated by their service, and have weathered the whole "Rails doesn't scale" flame-fest with relative grace and good humor. IMHO, that entitles them to talk fairly freely about the relative scalability of different languages and frameworks.

Secondly, any "ranting" you hear in this article is the product of El Reg, not Alex. He mostly offered a calm, positive presentation about the improved speed (with relatively little, if any, tradeoff in developer productivity) that came from using Scala instead of Ruby for critical infrastructure services.

Regardless, most Java web applications' performance issues stem from framework-itis, not from the underlying runtime. By itself, Tomcat (or another modern Java servlet container) is capable of some pretty impressive throughput -- perhaps not on par with a lean, pure C server like nginx, but not nearly as limited in functionality and extensibility, either.

No matter what you think about Java the language, you at least have to admit that HotSpot blows MRI, YARV, the Python VM, and Parrot out of the water when it comes to raw performance on low-level code. That doesn't save you from bad implementation ideas, but it does at least give you the possibility of getting fast, efficient code, if you have a decent compiler and work fairly close to the metal.

Using Scala is one way to produce that sort of low-level implementation code without wanting to gouge your eyes out (which is how most Java source tends to make me feel, anyway).

No, the JVMs use memory like crazy themselves, it's not just the frameworks. As I said, check out the Language Shootout. Java, in almost every test, is several times bigger than most other languages.

The rest of your comment, you change what I said so it's not worth addressing.

Java scales just fine thanks. FWIW, Mibbit handles a good number more messages per day that twitter. On a single VPS with 1.4GB ram, running Java.

  > Java scales just fine thanks. FWIW, Mibbit handles a good number more messages
  > per day that twitter. On a single VPS with 1.4GB ram, running Java.

I wasn't talking about that. Message passing is trivial and scale on any language, or at least it should be. Well, unless you do Java and XML. That doesn't scale, from experience. Unless something changed in the last couple of years I'm not aware of. (Just taking your red herring further, to show it doesn't make sense and it has nothing to do with what I said.)

Third time lucky, I hope. My point is I seriously doubt Scala is much better at scaling than Ruby, Python, or PHP. And the author seems to have based his opinion on his experience refactoring Twitter components previously coded in Ruby in Scala. Caveat emptor, I wasn't at the conference and this rant is second hand via a questionable source, The Register.

> My point is I seriously doubt Scala is much better at scaling than Ruby

I spoke with some people at Twitter the other day. They said that they were switching to Scala just to annoy alecco on Hacker News. The performance and scaling advantages are entirely negligible.

Careful here: are you _really_ suggesting that Twitter would do a full rewrite without making sure that it would increase throughput first?

I don't get the sense that they are that inexperienced. If they think a switch to Scala will handle more volume, I'd be really shocked if they didn't have some solid evidence to back that up.

  > Careful here: are you _really_ suggesting that Twitter would do a full rewrite
  > without making sure that it would increase throughput first?

No. I say this Scala vs. Ruby/Python/PHP rant sounds farfetched and unfounded.

Seeing the April 1st date of the article it was just a little bit harder to believe they would write it like that. As folks have taken it seriously...

The Ruby community was formed without being forced into it necessarily. Academics did not force people into Ruby. Businesses did not force people into Ruby. Even Ruby on Rails was created because DHH was not forced to use PHP and instead tried to make things work in this new called Ruby.

Scala has its followers but Academics push it and business types can push it as well even though many business types still prefer Java and C#. Also, Scala is being primarily used for server-side stuff which is a very crowded market already and many developers are just a little bit too tired of having to interface with complex server-side stuff while trying to innovate on the client-side.

Thing is, as a guy who really digs Ruby, Scala does not worry me one bit. I could always try and use a hyper-advanced Scala implementation or application in the future given I can make choices at one point and then have to revisit said choices in the future given the changes brought by new developments.

Let me tell you a little story. When C# was being first introduced in its very early versions, I was very much playing Ultima Online on pirate servers. A new pirate server was being coded in C# and it was becoming rather popular quite fast and the single developer (or was it?) was having quite a lot of fun with it. Nobody told him to use C# for that kind of thing I am sure, it was just him putting his passion forward with the means he had at the time. C# has grown since then, and many pros and hobbyists use it to create all kinds of little programs on Windows.

Scala? Scala has an uphill battle, and many other languages have had their own uphill battles throughout the years. It's tough.

Uh, who cares?

Some day you will realize that programming languages are tools for getting work done, not religions.

Reply on news.ycombinator.com