BERT and Ernie: Scaling your Ruby site with Erlang | Hacker News Reader

BERT and Ernie: Scaling your Ruby site with Erlang

rubyconf.org

17 pointsfogus11 comments

There are some other really interesting looking talks at RubyConf this year also: http://rubyconf.org/pages/program. I only wish that it hadn't sold out so quickly.

I think the problem with solutions like these it usually comes down throwing down money at more development (Erlang is hard) or throwing down money on more servers (delayed_job, Amazon SQS, etc.).

I usually prefer money on servers since it's faster to develop for my team. But I guess for the functionally gifted, this is a great solution.

We were constrained by network our file system and could not add more machines. We needed an RPC solution to shard the data and move away from our network file system.

delayed_job, Amazon SQS, etc are not RPC systems. You can't block waiting on a response like you can with messaging systems.

For instance, delayed_job makes an RPC call that asks "how much disk space is this repository using?" It waits until it gets a response from an Ernie handler then continues with its background job.

I wouldn't say Erlang is a "hard" language, especially once you understand it. Sure, there aren't a whole hell of a lot of people that are proficient in it, but I think that's just because it's a little obscure. To someone with a little experience with functional programming, Erlang should make a lot of sense.

Agreed. I'm learning it now, and while there are some new ideas in the concurrency model, it doesn't strike me as hard to learn or use the way (say) Haskell and C++ are. There's a fairly small number of core elements, and the language design as a whole strongly encourages you to make things loudly crash when anything is wrong, rather than passing on corrupt data. That helps immensely with debugging...

Also, I found that a lot of concepts from Prolog carry over nicely. Erlang was initially implemented in Prolog, and that's where it gets pattern matching and immutable variables from. (Along with several stylistic / syntactic quirks.)

Yeah I'm sure everyone has their own opinions on how to solve this problem (scaling in general), this is just my approach. I'm sure your approach is just as, or more, valid.

Why would you choose an approach like this instead of using tmm1-amqp + rabbitmq? A message queue seems like it would work well, and is the logical step up from delayed_job.

You could use BERT with amqp, as it's just a data format (like JSON). Ernie and BERT-RPC are libraries which build on that.

Ok, but why did you decide to build something rather than use an approach like I described? Is it because you needed synchronous requests, not all of them were async?

Definitely needed fast synchronous requests. I know you can fake this in RabbitMQ but am not sure why we didn't go with it... guess you'll have to wait for Tom's presentation :)

Not really understanding why a bizarre homebrew solution was better than Thrift...?

Reply on news.ycombinator.com