Obie Fernandez: My Reasoned Response about Scala at Twitter | Hacker News Reader

Obie Fernandez: My Reasoned Response about Scala at Twitter

blog.obiefernandez.com

45 pointstortilla44 comments

Hmmm, the Twitter guys say Scala helps them write "reliable, high performance code" that is "correct and maintainable," and Obie Fernandez basically blasts them for needing that help. Their problem is that they wrote a large codebase in Ruby and ended up creating so many bugs that they had to add a bunch of run-time checks to guard against them. It seems reasonable that they would want to choose a language that helped them with that, rather than sitting around flagellating themselves because they failed to get the job done in Ruby, which is apparently what this writer wants them to do.

I think you're missing the point. You can't bash Ruby because you wrote shitty code in Ruby. What do you want to bet I can write a shitty message queue in Scala? Does that mean Scala is bad? "Reliable, high performance, correct and maintainable" have nothing to do with the language you've chosen.

""" "Reliable, high performance, correct and maintainable" have nothing to do with the language you've chosen."""

I totally don't agree with this. Sure, it all comes down to good architecture and common sense but in reality some languages are simply not the right solution for interesting backend stuff.

Ruby is great for web apps and it is a clear winner there, but I would never ever use it for anything long-lived and high performance on the backend or 'service' side. As the Twitter guys discovered, Scala on top of the JVM is a much better choice for that.

You're dead wrong. If the code you write in Ruby tends to be shittier than the code you write in Scala, then you're a damn fool if you continue to write your code in Ruby.

"Reliable, high performance, correct and maintainable" has _tons_ to do with choice of language. Maybe not everything, but it's ridiculous to claim that there's no correlation between any of those factors and the language you use - there are various tradeoffs with different languages, and again, you'd be a fool to ignore them. There's a reason people don't code Web 2.0 stuff in Assembly, and there's a reason people don't code graphics drivers in Python.

That said, what is best for Twitter is probably not best for you. Your Ruby code probably _is_ better than your Scala code would be, and maybe it runs more than fast enough for your purposes. That's fine. Twitter has blown up practically from nothing into one of the stars of the Internet, so obviously they have different priorities. I doubt many people have ever had to deal with scaling issues like they have, so their experiences are pretty much irrelevant to the rest of us.

You can't bash Ruby because you wrote shitty code in Ruby.

If by "bashing" you mean claiming that Ruby was less suitable than Scala, why not? When a language isn't suitable for a particular application, that manifests as shitty code. Needing a bunch of hand-coded runtime type checks to prevent new bugs from creeping into the code suggests that a statically typed language would be more appropriate.

Of course, an infinitely intelligent programmer could have written Twitter in assembly, Brainfuck, or even C. There exists a positive integer N such that Ruby would have worked if all the Twitter developers had IQ >= N. (Please, nobody think too hard about that statement. I'm just making a point.)

So maybe they were just too stupid to use Ruby. But in that case, they were right to switch to another language, and they were right to report that Scala helped them succeed where they failed with Ruby.

In my opinion this is a clear ad-hominen attack from Obie against Alex, and one that should not be appreciated.

I am not a fan of scala, but I say lets wait and see what happens, as we all should know by now, in theory Ruby should scale as much as scala, but maybe in practice we come up with different results! As is there is a difference between what you should be able to do, and what you can really do.

Alex is obviously trying what he thinks he can, not what other thinks he should be able to do! which is of course just a theory

Obie's appeal to emotion while flying the flag of rationality does us all an intellectual disservice. The article contains many accusations, but few reasoned arguments to support them, and could not be any more antithetical to Alex Payne's appeal for calm, reasoned discussion.

"Alex is indeed scapegoating Ruby for Twitter's shitty legacy codebase"

"... frankly the timing of his essay made me throw up a little in my mouth. The nausea only grew this morning when I checked on how much it's being re-tweeted."

Truth in labeling: the headline is sarcastic. It's not really a 'reasoned response' when, by the third paragraph, it reaches for the new gross-out hyperbolic cliche, "made me throw up a little in my mouth".

Scaling from one end of the performance/team size/code size scale to another using a single language is a very difficult problem no matter what language you choose. Things like static type checks that get in your way at one end of the spectrum become much more useful at the other end, and different languages fit different problem domains; when your problem space essentially changes on you, the tools you chose initially might no longer be the best choice. That doesn't mean the tools themselves are broken, just that they're no longer a fit.

Obie took the easy way and simply said 'oh it must be the crappy Twitter code'.

Too bad he is ignoring the very valid points about Ruby's slow runtime, memory usage because you are forced to a per-process mode, lack of proper threading and usage of dynamic languages in large systems.

A per-process model for Ruby doesn't have to use a fuckton of memory -- with fork() & COW, the processes could easily share.

Unfortunately Ruby has an awful mark-and-sweep garbage collector that writes to nearly every page on every collection, making COW useless.

Forgive me as I'm unfamiliar with lower-level details like this, but isn't this exactly what Ruby Enterprise Edition is meant to solve?

'REE' - that is a joke right?

No it's not and it works quite well.

This problem has been solved well over a year by Phusion Passenger (www.modrails.com) and Ruby Enterprise Edition (www.rubyenterpriseedition.com). It saves you tons of memory.

There is a rule of thumb in UI design wherein if someone blames operator error, that's shorthand for system failure. Obie's call that Alex's code base is bad is classic blame the operator behaviour.

Seriously, is there really a need to be such a dick about these things? Maybe he's right, maybe the other guy's right, maybe they're both talking about totally different things and just not noticing because they're too busy arguing.

There may well be things world for which Ruby is not the best language. Or maybe even there aren't. Why does it matter so much if someone says that there are? Even if you believe that there aren't? Are they stopping you from doing what you think is the right thing?

I kind of feel like some people have this fear that someone preaching The Wrong Way will convert everyone and leave them all alone. This fear seems to rob them of all civility. Even the discussion here seems to have degenerated to a shouting match in some parts.

Guys, discussion is good and fun, but don't get so upset about it. No one's going to take your toys away from you :)

Wow, what a douchebag.

Yes. Language warring with extra nerd-rage.

People need to remember that twitter started as a side-project at odeo. Odeo is a Rails app (right?). These guys just used what they knew to get twitter going: ruby.

No big deal and no surprise at all that ruby (for several reasons) is not the first choice for such a high volume messaging back-end. Who cares.

Ruby is a beautiful language. But your pet langugage can't serve all needs, get over it.

Scala is a fantastic language that is capable of handling the requirements of twitter's back-end problem domain. Its my understanding that twitter played with quite a few alternatives before settling on scala (erlang based solutions being one). They went with scala. Cool.

I hope MagLev does great things for the Ruby community. I like the language, but keep away from it because of lack of a formal spec and the slow implementation. I suspect most potential users have the same reasons for not giving Ruby a go.

Agree.

"I suspect most potential users have the same reasons for not giving Ruby a go."

Reasons I've heard for not using Ruby:

- Already know and like <Python|Lisp|Blub>

- It's like Python, but Python has more robust libs, so I'm learning Python

- Dynamic typing leads to serious bugs in non-trivial apps

- It's slow

- I hate the syntax

- I'd like to use it, but my boss insists on <Java|PHP|Blub>

Slow isn't usually the first concern I hear. Besides, JRuby and Ruby 1.9 alleviate the speed issue; JRuby alleviates the library concern.

Obie is a smart man. He knows when to assert himself into the situation. He has achieved his goal: advertising himself/his business. It doesn't matter what he says regarding this drama as long as it's not "don't do Ruby/Rails".

Meanwhile, the C# developers go all the way laughing to the bank. ;-)

Here's hoping that Scala will help to fix the Java issues. However, given the similar way such languages work, I fear many developers will prefer to use something else, really, like PHP, C#, Python, JavaScript, ActionScript...

If Scala means more open source developments, there are many ways to go about that.

Celebrities and their audiences, huh? Both the Scala guy and the JRuby guy are celebrities in their own rights.

Here's how worried Ruby is that Scala is gaining traction:

  dewd@senna:~/in_motion/ruby$ ls -l
  total 32
  drwxr-xr-x  5 dewd dewd 4096 2009-03-29 15:44 ironruby
  drwxr-xr-x 18 dewd dewd 4096 2009-04-05 15:23 jruby
  drwxr-xr-x 20 dewd dewd 4096 2009-04-05 03:05 macruby-experimental
  drwxr-xr-x 16 dewd dewd 4096 2009-03-25 12:13 rubinius
  drwxr-xr-x 25 dewd dewd 4096 2009-04-05 16:18 ruby
  drwxr-xr-x 18 dewd dewd 4096 2009-04-05 16:27 ruby-1.8.6
  drwxr-xr-x 18 dewd dewd 4096 2009-04-05 16:27 ruby-1.8.7
  drwxr-xr-x 24 dewd dewd 4096 2009-03-06 02:58 ruby-1.9.1

As the Ruby community has grown throughout the years, there have been some controversies among Ruby users every now and again. For example, now that there are many Specs and Tests to run when testing a Ruby implementation, there has been the implementation of a minimalistic and fast runner for them which was not meant to be fully compatible with RSpec and Test/Unit, which has caused some discontentment at times... :-)

JRuby too is not without its own love/hate feedbacks among potential users. ;-)

That said, Ruby and its ecosystem is not "too big to fail" yet. Other languages and systems like Java and C# are "too big to fail" and it can be hard to compete against them with other less-supported tools. Scala builds on top of Java and to some extent it works great that way, right? Ruby "loves" the Java world in the JRuby form and it works quite OK that way, without the static typing and absolute performance of Scala.

Scala and Ruby belong to different typing systems, static and dynamic, and may well be extremes on each end of that spectrum. Ruby has had a "motto" of encouraging "duck-typing" as in "if it walks like a duck, quacks like duck, then it's a duck", so no need to check the is_a?(Duck) with Ruby. It kind of betrays the way Ruby is "meant to be" to do stuff like that. If big systems weren't to be built in Ruby, Ruby on Rails is that exception, that wonder of a miracle, really. Ruby on Rails with all kinds of plugins and stuff can be quite a featureful framework... And yet, it's constantly under development and the (lack of) typing system hasn't killed it just yet...

In celebrities wars, we are just fans really...

Sorry, but I have to dismiss this as Obie protecting his economic interests as an owner of a Rails consultancy. I can only guess that he sees negative Ruby coverage like this as a threat to new Ruby development, and thus the future of his consultancy.

With few exceptions, Ruby applications running on MRI will be wholly unsuited to high throughput environments. This is true regardless of how good your Ruby code is. Saying that using Object#kind_of? is bad Ruby coding may be true but it's mostly orthogonal to the speed of the code. Even the most aesthetically pleasing, extensible, and clean Ruby code will be slow.

After a certain threshold of requests per second the bottleneck will almost always become Ruby. After all, it might not even matter how aggressive your caching strategy is if it can take upwards of 50ms to even talk to your cache. MRI Ruby's IO subsystem and its threading model absolutely hamstring its suitability for even IO bound applications.

The good news is that Ruby will improve in the future. For better or worse, Engine Yard hitched its wagon to the 1.8.x generation of Ruby. Engine Yard's customers will be using the 1.8.x series of interpreters for quite some time to come. And Engine Yard has the engineering know how and an economic interest on improving that generation of Ruby.

But in the meantime those of us building applications that need to make efficient use of a given set of hardware are left with a choice. Do we fight with our platform tooth and nail to try and wring even an acceptable amount throughput out of it? Or do we change our platform to best suit the problems in front of us?

Sorry, but I have to dismiss this as Obie protecting his economic interests as an owner of a Rails consultancy

This was my exact thought. After reading his claims to Alex's "agenda", I looked at the sidebar and saw that he's written four (obnoxiously-covered) Ruby books. Who really has the agenda here?

Near as I can tell, Alex's entire argument was "Twitter is really big and ugly; we need a stricter type system; Ruby doesn't have one." That seems perfectly legitimate to me.

He has only written one of those books (The Rails Way) the others are just books in the series. I don't think he choose the covers either.

He also states at the end:

"Have I been guilty of this stuff in my own past as a Rails advocate with a strong agenda? Yeah, probably so. Which is why I know it when I see it. :)"

Also, as Alex himself alluded in his original piece, why would someone who presumably has a stake in a company that shrugs off half-billion dollar acquisition offers be trying anything devious to promote a technical book?

Fun? Personal brand? I mean, I make more than enough to get by on but I still promote my book every change I get. :)

You dismiss anyone who's job applies to the topic at hand? Shouldn't you dismiss Alex because he's writing a book for O'Reilly on Scala? Should the only people allowed to discuss this be the Erlang people? Just asking because I'd bet they would say that Kestrel is a crappy version of RabbitMQ.

Please try to pay closer attention. That was my conclusion after reading what he said. He claims good Ruby code would have helped Twitter. My experience and apparently the experience of Alex and his team prove otherwise.

And as for Kestrel, you can call it a crappy version of X all you care to, and it may be depending upon how requirements for X and Kestrel differ. One thing to keep in mind is that essentially the Twitter engineers are doing the software engineering equivalent of rebuilding a jet aircraft in midflight.

Part of the point of having a modular architecture is to try and set up API's that act as firebreaks for intra-module changes. So if you need to switch out your message queue for performance reasons, rewriting a brand new one that conforms to the API the rest of your system expects might be the best solution at the moment. Otherwise by breaking compatibility you can set off cascading changes which require rewrites all across the system.

I didn't realize that anecdotes proved things. You should pay closer attention to the things you say.

I don't believe Engine Yard has hitched it's wagon. Quite the opposite considering they fund a great deal of work on the Rubinius ruby implementation effort.

Generalizations like "ruby falls down at high throughput" are fun to echo but a poor basis for planning architecture. The constraints of the tools you're using matters, but so does the overall pattern of how you're applying them.

Efficiency is a property of languages. Stability and Scalability are properties of architectures. Design is the process of fitting one to the other, and requires real actual thought, work and measurement, not just repeating chestnuts.

Not necessarily. Sometimes, the very structure (syntactic I mean) of language makes ceratin kinds of abstractions (behavior abtractions as well) veryhard to implement, which is effectively same as having no way to express them. I mean, for example - there is no way, absolutely no way to write Lapack benchmark in Ruby to be as fast as C one in reasonable amount of time (you can always emit machine code and run afterwards, but that is cheating). Or less controversial example: Java VM has no tail recursion optimisation, and for example Clojure does not have either. That means you have no way to use certain class of programmin techniques with it, As a conclusion: languages largely prescribe the way to implement architecture with them.

By the way for me it is also important that the particular Language has non-gpl translators.That is also can contribute to the choice of the language for particular task, and consequently the final architecture.

I think we're in agreement without quite realizing it.

These are examples of efficiency, not scalability. This distinction isn't quite binary, and Ahmdal's law is a good approximation. But for the discussion at hand we can generally ignore the serially dependent portion of the workload.

Also, I'm speaking of systems architecture, not program architecture, so whether tail calls are interpreted or compiled doesn't really apply.

To make my point more clear: I'm absolutely certain you could write twitter in pure ruby, even at it's current traffic level. It's also likely this wouldn't make sense, as the economics would be wrong. But the economics would be due to efficiency * hardware costs as compared to programmer staff costs, not any limitation of scalability.

I would grant you that there are a few exceptional cases where the language really does constrain the system's architecture (non turing languages and jailed languages with limited connectivity options) but those are never the languages being discussed in "does it scale" debates.

I have to disagree. Let us take an extreme example - Erlang. It has very interesting design, which enables you to create highly scalable but quite ineffecient program. There is something in Erlang, that affects _system_ architectures that you can create with it. The notion of scalabiliy - it is how it scales, how fast the performance degrades with number of users, and for some paradigms there is a necessity of support of them in the language. erlang exists only because people wants scalable solutions even though quite slow. From what I said it seems for me that it is obvious that some _languages_ do scale and some do not.

Two parts here. 1. Scripting languages might be too slow for Twitter? If they say so, I'll believe them -- the right tool for the job.

2. Too large and sloppily written code base, so there are problems with keeping the APIs between modules clean? That is a risk with loosely typed languages.

I don't get why this is a problem.

Other people has written large systems with Perl/Python/PHP/Ruby/etc and gotten around that problem. Also, I assume Ruby (like all other scripting languages) has some simple module to assert parameter types -- it's only needed for the externally called methods anyway.

Edit: Another thing hit me, regarding what I comment on. The memory usage should be different for long running Scala and Java code. Has this been tested and tuned..?

> Stability and Scalability are properties of architectures

But the architecture starts at the virtual machine level. Without a decent VM you'll have to put more effort into building a scalable architecture.

True story: the team working on Photoshop Express tried Rails for displaying a 404 Not Found page (containing some dynamic content). Because the interface was already in tests by a lot of people, they couldn't even display a static rhtml template ... Mongrel was segfaulting because of the high loads. Now, this may be an incident that got fixed since then (last Summer), I wouldn't know since I'm not working with Rails ... but it surely sends shivers down my spine.

what's the deal with questioning people's "agenda" when you could just stick with a technical argument? unless you don't have one.

I don't see how you're scapegoating Ruby when you say that you have bugs that could have been prevented entirely by having a type system that lets you check types at compile time.

Because blaming a dynamic language for being dynamic because your code sucks is scapegoating.

If you're passing an invoice to a function that expects a customer, you don't fix it by inspecting the type inside the function; you let it blow up and you fix the bug where the hand off is done wrong.

You don't litter up a program in a dynamic language with manually written type checks. If you're reinventing a static type system, you're doing something wrong.

"You don't litter up a program in a dynamic language with manually written type checks. If you're reinventing a static type system, you're doing something wrong"

Yes: using a dynamic language

Reply on news.ycombinator.com