Did you evaluate EventMachine? If yes would you please share your findings.
We're using it for handling jobs from Resque, which reads and writes from our database and do a ping to a 3rd party server.
But. The amount of bugs in the em-enabled libraries is just amazing. For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers. Also Resque blocks by default, so of course doing a non-blocking and non-forking version was a priority.
The worst thing here are the error messages and backtraces. There are none (DeadFiberException for failing assert) and with hacks you can get out a bit more information if the job crashes.
Now thinking this again, it would be a good idea to write it again with Node.js. Just because it's still more mature compared to EventMachine, and all of it's libraries are reactor core friendly by default.
That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.
The thing is - my first thought would have been to leave the ORM alone and focus on the DB side (like more sophisticated connection pooling/pgbouncer, etc. ). Which is why I'm interested in what went wrong in the ORM that made it screw up when used in a non-blocking kind of a setup.
I will do a blog post about this when I finally have some time (christmas holidays, maybe). Em-activerecord worked very nice first, but when I hit it with thousands of concurrent jobs, some workers just dropped dead saying MySQL couldn't answer. This was annoying, because the workers didn't really fail in Resque, just failed to do their job and if this was in production it would've cost us thousands.
So, my own ORM is just a database superclass with em-connection-pool, configuration, openstruct and couple of class and instance methods (insert, update, find, query). And now this thing is really really fast, using lot less of sql connections (delayed job had 300 connections, now we'll need only ~40 connections) and scales really well.
The thing here is, that when I'm not using Rails at all, I don't know why I should use evented Ruby instead of Node.
You can probably use both node and ruby. From what ive seen sites that are using node use it more for the backend to serve ajax then for serving the web pages. Although i have seen few site with very ajax based interfaces that apparently use just node. It really depends on your ui i guess. Good luck with whatever you choose though!
This seems like exactly what would hit anybody who is trying to build a worker model for processing data... hell even for sending emails. It would be interesting to see where you are pushing the limits - it could even be something deep like
This problem is noted by the developers of em-activerecord and em-synchrony a several months ago already. There seems to be no progress, which might lead to a harder problem or that people are not using these libraries in bigger services.
But my point was, that Node.js seems to be much much more finished product compared to EventMachine for example. With EM you have to live without most of the Rails and for example testing the EM code with unit tests is not so nice job to do...
What I liked the most with EM are the Ruby fibers. The best solution for hiding the callbacks and writing nice and readable code by far. Too bad the fibers are a bit of a ghetto still, like somebody said earlier.