Comment by meskyanichi | Hacker News Reader

Comment by meskyanichi | Hacker News Reader

Parent Full thread

Hey. I was actually also using SimpleWorker prior to HireFire, but I find that setting it up is a bit hard / takes long since you have to set isolated code / credentials / etc to their servers in order to process background jobs. Also, the jobs aren't guaranteed to run immediately after they are queued, may take up to 10 minutes on the cheapest plan. When deployed to Heroku it's easy since Heroku will just use the same compiled slug it uses for Dyno's, no additional code/configuration required, it just works. I also found that on SimpleWorker a job that took only 3 seconds on Heroku, would run for 30-90 seconds on SimpleWorker (no idea why).

That said. There are some advantages when using SimpleWorker such as the web interface, scheduling recurring jobs, and other useful options. Especially when you aren't running on Heroku, and memory is an issue if you have a small VPS. In that case, offloading background jobs to SimpleWorker is ideal. But I find that HireFire will make Heroku Workers cheaper than SimpleWorker due to the fact that jobs appear to take (a lot) longer to finish on SimpleWorker than on Heroku (probably due to the start up time? I'm not sure why).

Couldn't agree more. The setup time of SimpleWorker is alot ( especially when dealing with specific gems ) and the fact that for DB work you have to create a separate non concurrent connection to the DB per job makes those times you talk about a reality. Ok, you could use the 'expensive jobs' but Heroku's DJ's win hands down in all tests I've done. Plus who doesn't like the handle_asynchronously :function.

Exactly. :) The DB connection is also quite frustrating depending on the location of the DB. MongoDB with MongoHQ or MongoMachine is fine since it's external already. But otherwise you have to know how to configure for example your MySQL PostgreSQL instances to allow incoming connections from external locations, and also becomes potentially insecure depending on your sysadmin skills.

I find Heroku + DJ + HireFire so much simpler, either using that handle_asynchronous method, or just called delay.my_method(*args) is brilliant. No need to think about isolated code or credentials due to the environment variables that are already set on Heroku.

I should look in to integrating Navvy and Resque as well.

Hey Guys, I'm a co-creator/founder of SimpleWorker.com. Can't say enough how appreciative we are of the feedback both good and bad. We've been experiencing pretty strong growth and through this are identifying the sticking points and working through them as fast as we can.

The isolated code issue has been a big issue, and we're addressing this through calls like merge_gem(), and soon merge_mailer(). It's a challenging task, and we see the gaps, but we're improving every day.

Runtime: this is also a top priority. We're working with each and every customer to ensure that their workers execute quickly. 3 seconds vs 30-60 is totally unacceptable and we won't leave you hanging here.

Queue speed: we're prototyping alternatives to http queuing which will dramatically speed up queue times.

This is just the tip of the iceberg for roadmap... Great work on HireFire. I see cases for both. Potentially massive parallelization on SimpleWorker (hundreds if not thousands of concurrent jobs), advanced scheduling needs, etc.

Anyways - we're listening - so any and all feedback is well received. If anybody has needs outside HireFire, in conjunction with, or otherwise, shoot me a line and we'll get you up and running at no cost to start.

Chad Arimura chad[at]simpleworker.com

http://www.simpleworker.com

Hey Chad, great to hear! I also (despite having created HireFire) still have use-cases for SimpleWorker, however in my case they will likely be for my VPS' rather than on Heroku. On Heroku (even though there isn't a UI, unless you use Resque), it's already a partially managed service to host workers, it runs from the same compiled slug as the dyno's use so all the credentials / env variables are all shared among dyno's and workers, no need to re-apply configuration and such.

I think for me the hardest part was getting it to work the first time, waiting quite a while for the jobs to run to see if I managed to get it to work. The merge_gem is a huge improvement, though, handling credentials and re-connecting databases and other things are quite a hassle to set up (imo). I have to say that once it's all set up, and if the 30-60 start-up issue is gone, that it does seems like a beast for processing a lot of jobs concurrently and is exciting.

I'm definitely still looking forward to seeing future progress of SimpleWorker. So please do keep up the good work!

Reply on news.ycombinator.com