Hacker News Reader

Comment by boundlessdreamz | Hacker News Reader

A noob question: If I have an expensive background job (say pdf generation) is there a way to make it async and use it with Goliath ?

View on HN

That's a great question actually.. In any evented app, blocking your "reactor" is a big performance problem, since everyone will be waiting for you to complete that operation before anything else can happen. In general, you want to turn any CPU intensive work into an IO-bound operation, where the reactor is "waiting" for the IO notification that the computation is complete.

Now.. How you actually achieve that is a whole different story. You could, in theory, throw a job into some external work queue and poll that, or if your runtime permits, spawn some threadpool and periodically check that, or.. spawn a process and wait on that. In other words, it all depends on the actual operation.

In the case of PDF generation, if you rely on some external tool, you could use a mechanism like EM.system('shell cmd') to spawn a process and wait for it to return you the data.

If you're going to be forking off processes to do work for you, I think it's a bad idea to hide the implicit state by using EventMachine to try to manage the pipe I/O and the process state. The reason for that is, sidecar processes screw up, backlog, crash, and eventually start consuming resources you want to track. You end up reinventing the same wheel the Github people did with Resque. Better to reify all those processes from the start with a real queue.

I only make this pedantic comment because EventMachine makes it really easy to start down the path of "just event the process management and I/O", and it seems like you're almost always better off not doing that.

This used to be a trickier question, but especially in the ruby world it has a very straightforward answer right now: Resque. Redis is especially amenable to evented clients. So, while there probably are reasonable ways to do an async fork+monitor-a-pipe, the simpler and sounder way to do it in 2011 would be to queue up a job in Resque (or any other "blpop jobs off a Redis list" scheme).

You misunderstood the question. I specifically say "background job". I'm using delayed_job / resque already. But if the results of the job is needed client side immediately then I've to poll the server periodically and fetch the results when done.

What I asked was whether these types of jobs can be done without the server blocking. igrigorik's answered that. The queue might still be needed but the polling can happen from inside the server code rather than from browser (the server will hold a connection from browser, keep polling queue and on success return the response)

You're right. It was a noob question. It is a bad idea to hold open an HTTP connection waiting for a long-running job to complete. Good luck with your design, though.

Why would it be a bad idea ? Event servers do not to fall over if there are lot of clients holding http connections open. This is used quite widely. http://en.wikipedia.org/wiki/Push_technology#Long_polling

Long-polling, in app designs based from the start on long-polling, is a fine thing (or at least, I'll stipulate that).

Connections that you hold open while forking off and waiting for a process that will take many seconds to complete is just bad UX. There are better ways to do it.

You'll excuse me if after arguing with my response to your one-sentence "noob question: how do I make PDF generation asynchronous from an evented Ruby webserver" I am not chomping at the bit to get into a long architecture debate with you. I meant it: good luck with your design. Sorry I couldn't be more helpful.

You can use rpc + message queues for that.

client request1 --> web server --> rpc server --> work queue

client request1 <-- web server <-- rpc server <-- done queue

Of course you need to create consumers to operate on the work queue, process the jobs and put them in a different queue (eg. 'done queue') signaling the rpc server that you finished the job so that the rpc server in turn will reply back to the web server.

Reply on news.ycombinator.com