Announcing DJJob, a PHP port of delayed_job | Hacker News Reader

Announcing DJJob, a PHP port of delayed_job

seatgeek.com

38 pointsericwaller13 comments

I'm the author of Delayed Job. The popularity of it really surprises me to no end, and always brings a smile to my face. DJ took about 15 minutes to code up originally and is a product of one of those software rage moments ("why don't people queue like this??!").

I put it on GitHub because Jeremy Kemper really wanted to have a look at it. I thought the idea behind it was so naive that it would have occurred to everyone who ever needed a queue and had discarded the approach for some reason.

Little did I know that it was a fairly novel approach and that it would become one of my most popular libraries, right after ActiveMerchant and Liquid.

Thank you for the PHP port!

Great to hear from you. I remember being impressed by how short and straightforward delayed_job was when I first explored job queues. That's actually one of the only reasons I even considered building/porting something myself.

You're welcome, and thank you for delayed_job!

Also the php PECL gearman driver (http://pecl.php.net/package/gearman) works well with the C-based gearmand server (http://gearman.org/). We do some asynchronous image processing with PHP gearman workers, and it's been a very stable solution for us.

Gearman is especially handy if you expose your application to it. I've got a wrapper that allows the app to push any function call out to the workers for async handling. So your image processing (or whatever heavy/blocking task) can be dispatched from within your standard libraries without having to maintain a separate set of code.

And the post here sounds very NIH for not even mentioning Gearman in its survey of options.

If you like Resque, too:

http://github.com/chrisboulton/php-resque

you need to strip out the implementation specific classes such as propel and the mysql-specific calls to make this useful to anybody else

for eg. an in-memory sqlite db would be ideal for this purpose (and a lot faster)

Absolutely, dropping Propel is TODO #1. I just figured I'd throw it on github and see if there was any interest first.

In your example, does the call:

$worker = new DJWorker($options); $worker->start();

Actually: 1. block until all jobs stored in the database have been completed 2. simply run a single job and return 3. return immediately and run all the jobs on a separate thread in the background

I'm wondering where in a web application would be the best place to call $worker->start()?

I think the point is to run the worker outside of a web request and using a cron job

Or using the command line version of PHP and GNU Screen (I haven't fully studied the code, but it looks similar to a queue system that I hacked together a couple of days ago where the worker is wrapped in a while(TRUE) loop).

In the interest of some outside figures, how have you found this to scale? I implemented a similar solution as more of a stopgap until I had the time to get RabbitMQ installed with a "proper" worker dispatcher (more moving parts but less stress on the DB). I mostly ask because we implemented it for our notification system where we had to generate a send large batches of custom emails which was bogging down the page (and hard to debug where in the chain we were getting failed messages).

Not sure who your question was directed to, but I've implemented a similar system in the past for a website with 10m+ users and 15+ web servers. It scaled just fine, we were sending close to 150K emails a day, among other background jobs for reporting, data cleanup, automated spam checks, etc

Made something very similar to this in PHP about a year ago. It's buried in the Drupal project called Boost. It's the multi process crawler. It supports up to 8 processes crawling your site getting the cache warm. Gets call on cron & then 15 seconds after drupal's cron finished it starts up all 8 "threads". I've used it for image processing as well. Very stable will run for days if you give it a big enough queue (millions of pages to crawl). http://drupal.org/project/boost

Reply on news.ycombinator.com